August 2016 - commits - mailman3.cels.anl.gov

[mpich] MPICH primary repository branch, master, updated. v3.2-408-g1b68d68
by noreply＠mpich.org 29 Aug '16

29 Aug '16

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "MPICH primary repository". The branch, master has been updated via 1b68d685084037ff3b9061f9eaab38ca7459f9d7 (commit) from f1193cd4b9302776cb983333e458b71729945e9f (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- http://git.mpich.org/mpich.git/commitdiff/1b68d685084037ff3b9061f9eaab38ca7… commit 1b68d685084037ff3b9061f9eaab38ca7459f9d7 Author: Ken Raffenetti <raffenet(a)mcs.anl.gov> Date: Mon Aug 29 16:57:41 2016 -0500 CH4/Portals4: Fix compile issues No reviewer. diff --git a/src/mpid/ch4/netmod/portals4/ptl_datatype.h b/src/mpid/ch4/netmod/portals4/ptl_datatype.h index 679a48b..95f346a 100644 --- a/src/mpid/ch4/netmod/portals4/ptl_datatype.h +++ b/src/mpid/ch4/netmod/portals4/ptl_datatype.h @@ -11,12 +11,12 @@ static inline int MPIDI_NM_type_free_hook(MPIR_Datatype * datatype_p) { - return; + return 0; } static inline int MPIDI_NM_type_create_hook(MPIR_Datatype * datatype_p) { - return; + return 0; } static inline void MPIDI_NM_type_dup_hook(MPIR_Datatype * old_datatype_p, diff --git a/src/mpid/ch4/netmod/portals4/ptl_progress.h b/src/mpid/ch4/netmod/portals4/ptl_progress.h index 50163cb..5972225 100644 --- a/src/mpid/ch4/netmod/portals4/ptl_progress.h +++ b/src/mpid/ch4/netmod/portals4/ptl_progress.h @@ -29,7 +29,7 @@ static inline int MPIDI_PTL_am_handler(ptl_event_t * e) in_data = p_data = (e->start + (e->mlength - data_sz)); int handler_id = e->hdr_data >> 56; - MPIDI_PTL_global.am_handlers[handler_id] (e->start, + MPIDI_PTL_global.am_handlers[handler_id] (handler_id, e->start, &p_data, &data_sz, &is_contig, &cmpl_handler_fn, &rreq); ----------------------------------------------------------------------- Summary of changes: src/mpid/ch4/netmod/portals4/ptl_datatype.h | 4 ++-- src/mpid/ch4/netmod/portals4/ptl_progress.h | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) hooks/post-receive -- MPICH primary repository

1 0

[mpich] MPICH primary repository branch, master, updated. v3.2-407-gf1193cd
by noreply＠mpich.org 29 Aug '16

29 Aug '16

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "MPICH primary repository". The branch, master has been updated via f1193cd4b9302776cb983333e458b71729945e9f (commit) via 95c5bd34622ae139c1fd8629a1748f227af02aaf (commit) from 5a02fae9ef24a26627a32987ea738df5ea858929 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- http://git.mpich.org/mpich.git/commitdiff/f1193cd4b9302776cb983333e458b7172… commit f1193cd4b9302776cb983333e458b71729945e9f Author: Ken Raffenetti <raffenet(a)mcs.anl.gov> Date: Sun Aug 28 22:28:12 2016 -0500 CH4/OFI: Remove always_inline hack This block was used to fool compilers into inlining functions that were defined in other files. It is not safe and prone to warnings. Signed-off-by: Charles J Archer <charles.j.archer(a)intel.com> diff --git a/src/mpid/ch4/netmod/ofi/ofi_impl.h b/src/mpid/ch4/netmod/ofi/ofi_impl.h index 95e3117..cd80910 100644 --- a/src/mpid/ch4/netmod/ofi/ofi_impl.h +++ b/src/mpid/ch4/netmod/ofi/ofi_impl.h @@ -17,16 +17,6 @@ #include "ch4_impl.h" #include "ofi_iovec_util.h" -/* Tag the prototypes with always_inline to force object allocation */ -/* routines to inline This allows the library, compiled without */ -/* ipo/pgo enabled to inline MPI layer functions */ -__ALWAYS_INLINE__ MPIR_Request *MPIR_Request_create(MPIR_Request_kind_t kind); -__ALWAYS_INLINE__ void *MPIR_Handle_obj_alloc(MPIR_Object_alloc_t *); -__ALWAYS_INLINE__ void *MPIR_Handle_obj_alloc_unsafe(MPIR_Object_alloc_t *); -__ALWAYS_INLINE__ void MPIR_Handle_obj_free(MPIR_Object_alloc_t *, void *); -__ALWAYS_INLINE__ void *MPIR_Handle_get_ptr_indirect(int, MPIR_Object_alloc_t *); -__ALWAYS_INLINE__ MPIDII_av_entry_t *MPIDIU_comm_rank_to_av(MPIR_Comm * comm, int rank); - #define MPIDI_OFI_DT(dt) ((dt)->dev.netmod.ofi) #define MPIDI_OFI_OP(op) ((op)->dev.netmod.ofi) #define MPIDI_OFI_COMM(comm) ((comm)->dev.ch4.netmod.ofi) http://git.mpich.org/mpich.git/commitdiff/95c5bd34622ae139c1fd8629a1748f227… commit 95c5bd34622ae139c1fd8629a1748f227af02aaf Author: Ken Raffenetti <raffenet(a)mcs.anl.gov> Date: Sun Aug 28 22:27:46 2016 -0500 CH4: Remove unnecessary forward decl Signed-off-by: Charles J Archer <charles.j.archer(a)intel.com> diff --git a/src/mpid/ch4/include/mpidpre.h b/src/mpid/ch4/include/mpidpre.h index 9990fb1..1a045ca 100644 --- a/src/mpid/ch4/include/mpidpre.h +++ b/src/mpid/ch4/include/mpidpre.h @@ -38,10 +38,6 @@ typedef int MPID_Progress_state; #define CH4_COMPILE_TIME_ASSERT(expr_) \ do { switch(0) { case 0: case (expr_): default: break; } } while (0) -/* Forward declaration of MPIR_Win so that we can refer to it in this file */ -struct MPIR_Win; -typedef struct MPIR_Win MPIR_Win; - typedef enum { MPIDI_PTYPE_RECV, MPIDI_PTYPE_SEND, ----------------------------------------------------------------------- Summary of changes: src/mpid/ch4/include/mpidpre.h | 4 ---- src/mpid/ch4/netmod/ofi/ofi_impl.h | 10 ---------- 2 files changed, 0 insertions(+), 14 deletions(-) hooks/post-receive -- MPICH primary repository

1 0

[mpich] MPICH primary repository branch, master, updated. v3.2-405-g5a02fae
by noreply＠mpich.org 26 Aug '16

26 Aug '16

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "MPICH primary repository". The branch, master has been updated via 5a02fae9ef24a26627a32987ea738df5ea858929 (commit) via 0841f2bd47d6f0134c4881e9b9e3f3dba3404f55 (commit) from 398f4af12cf6d05176886515c8f42df870c92631 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- http://git.mpich.org/mpich.git/commitdiff/5a02fae9ef24a26627a32987ea738df5e… commit 5a02fae9ef24a26627a32987ea738df5ea858929 Author: Ken Raffenetti <raffenet(a)mcs.anl.gov> Date: Fri Aug 26 10:07:41 2016 -0500 maint/version.m4: Update version info and CHANGES Prep for 3.3a1 release. Signed-off-by: Pavan Balaji <balaji(a)anl.gov> diff --git a/CHANGES b/CHANGES index ca19721..fd3c771 100644 --- a/CHANGES +++ b/CHANGES @@ -1,4 +1,26 @@ =============================================================================== + Changes in 3.3 +=============================================================================== + + # CH4 Device: A new device layer implementation designed for low software + overheads. CH4 has experimental support for OFI and UCX network libraries, + and POSIX shared memory. Thanks to Intel, Mellanox, and RIKEN AICS for + participating in the CH4 coding effort. + + # Removed the PAMI device and poe PMI client. + + # Several other minor bug fixes, memory leak fixes, and code cleanup. + + A full list of changes is available at the following link: + + http://git.mpich.org/mpich.git/shortlog/v3.2..v3.3a1 + + A full list of bugs that have been fixed is available at the + following link: + + https://trac.mpich.org/projects/mpich/query?status=closed&group=resolution&… + +=============================================================================== Changes in 3.2 =============================================================================== diff --git a/maint/version.m4 b/maint/version.m4 index d3e271e..7957dbd 100644 --- a/maint/version.m4 +++ b/maint/version.m4 @@ -14,7 +14,7 @@ # changing this by playing with diversions, but then we would probably be # playing with autotools-fire. -m4_define([MPICH_VERSION_m4],[3.2])dnl +m4_define([MPICH_VERSION_m4],[3.3a1])dnl m4_define([MPICH_RELEASE_DATE_m4],[unreleased development copy])dnl # For libtool ABI versioning rules see: @@ -35,6 +35,6 @@ m4_define([MPICH_RELEASE_DATE_m4],[unreleased development copy])dnl # libmpi so version only includes functionality defined in the MPI # standard, and does not include MPIX_ functions and C++ bindings. -m4_define([libmpi_so_version_m4],[13:0:1])dnl +m4_define([libmpi_so_version_m4],[0:0:0])dnl [#] end of __file__ http://git.mpich.org/mpich.git/commitdiff/0841f2bd47d6f0134c4881e9b9e3f3dba… commit 0841f2bd47d6f0134c4881e9b9e3f3dba3404f55 Author: Ken Raffenetti <raffenet(a)mcs.anl.gov> Date: Fri Aug 26 10:31:18 2016 -0500 mpid: Remove PAMI device Remove unsupported pamid device, including poe PMI client. Signed-off-by: Pavan Balaji <balaji(a)anl.gov> diff --git a/README.vin b/README.vin index 40b6e72..c845b17 100644 --- a/README.vin +++ b/README.vin @@ -577,33 +577,6 @@ channel can be configured using the following option: --with-device=ch3:sock - -pamid device -************ -This is the device used on the IBM Blue Gene/Q system. The following -configure options can be used: - - ./configure --host=powerpc64-bgq-linux \ - --with-device=pamid:BGQ \ - --with-file-system=bg+bglockless - -The Blue Gene/Q cross compilers must either be in the $PATH, or -explicitly specified using environment variables, before configure. -For example: - - PATH=$PATH:/bgsys/drivers/ppcfloor/gnu-linux/bin - -or - - CC=/bgsys/drivers/ppcfloor/gnu-linux/bin/powerpc64-bgq-linux-gcc - CXX=... - ... - -There are several other configure options that are specific to building -on a Blue Gene/Q system. See the wiki page for more information: - - https://wiki.mpich.org/mpich/index.php/BGQ - ------------------------------------------------------------------------- 5. Alternate Process Managers diff --git a/src/mpid/Makefile.mk b/src/mpid/Makefile.mk index 84122ef..247a485 100644 --- a/src/mpid/Makefile.mk +++ b/src/mpid/Makefile.mk @@ -13,5 +13,4 @@ noinst_HEADERS += \ include $(top_srcdir)/src/mpid/ch3/Makefile.mk include $(top_srcdir)/src/mpid/ch4/Makefile.mk -include $(top_srcdir)/src/mpid/pamid/Makefile.mk include $(top_srcdir)/src/mpid/common/Makefile.mk diff --git a/src/mpid/pamid/.gitattributes b/src/mpid/pamid/.gitattributes deleted file mode 100644 index 3d45ef0..0000000 --- a/src/mpid/pamid/.gitattributes +++ /dev/null @@ -1,15 +0,0 @@ -* whitespace=trail-space,space-before-tab -*.[ch] whitespace=trail-space,space-before-tab,cr-at-eol,tab-in-indent -Makefile.sm whitespace=trail-space,space-before-tab,cr-at-eol - - -# * `blank-at-eof` treats blank lines added at the end of file as an error (enabled by default). -# * `blank-at-eol` treats trailing whitespaces at the end of the line as an error (enabled by default). -# * `trailing-space` is a short-hand to cover both `blank-at-eol` and `blank-at-eof`. -# * `space-before-tab` treats a space character that appears immediately before a tab character in the initial indent part of the line as an error (enabled by default). -# * `cr-at-eol` treats a carriage-return at the end of line as part of the line terminator, i.e. with it, `trailing-space` does not trigger if the character before such a carriage-return is not a whitespace (not enabled by default). - -# * `indent-with-non-tab` treats a line that is indented with 8 or more space characters as an error (not enabled by default). -# * `tab-in-indent` treats a tab character in the initial indent part of the line as an error (not enabled by default). - -# * `tabwidth=<n>` tells how many character positions a tab occupies; this is relevant for `indent-with-non-tab` and when git fixes `tab-in-indent` errors. The default tab width is 8. Allowed values are 1 to 63. diff --git a/src/mpid/pamid/.gitignore b/src/mpid/pamid/.gitignore deleted file mode 100644 index f6f157a..0000000 --- a/src/mpid/pamid/.gitignore +++ /dev/null @@ -1,3 +0,0 @@ -/autom4te.cache -/configure -Makefile.in diff --git a/src/mpid/pamid/Makefile.mk b/src/mpid/pamid/Makefile.mk deleted file mode 100644 index f33946c..0000000 --- a/src/mpid/pamid/Makefile.mk +++ /dev/null @@ -1,43 +0,0 @@ -# begin_generated_IBM_copyright_prolog -# -# This is an automatically generated copyright prolog. -# After initializing, DO NOT MODIFY OR MOVE -# --------------------------------------------------------------- -# Licensed Materials - Property of IBM -# Blue Gene/Q 5765-PER 5765-PRP -# -# (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved -# US Government Users Restricted Rights - -# Use, duplication, or disclosure restricted -# by GSA ADP Schedule Contract with IBM Corp. -# -# --------------------------------------------------------------- -# -# end_generated_IBM_copyright_prolog -# -*- mode: makefile-gmake; -*- - -# note that the includes always happen but the effects of their contents are -# affected by "if BUILD_PAMID" -if BUILD_PAMID - -noinst_HEADERS += \ - src/mpid/pamid/include/mpidi_thread.h \ - src/mpid/pamid/include/mpidimpl.h \ - src/mpid/pamid/include/mpidi_mutex.h \ - src/mpid/pamid/include/mpidpost.h \ - src/mpid/pamid/include/mpidi_constants.h \ - src/mpid/pamid/include/mpidi_externs.h \ - src/mpid/pamid/include/mpidi_util.h \ - src/mpid/pamid/include/mpidi_hooks.h \ - src/mpid/pamid/include/mpidi_macros.h \ - src/mpid/pamid/include/mpidi_datatypes.h \ - src/mpid/pamid/include/mpidi_prototypes.h \ - src/mpid/pamid/include/pamix.h \ - src/mpid/pamid/include/mpidpre.h \ - src/mpid/pamid/include/mpidi_platform.h - -include_HEADERS += src/mpid/pamid/include/mpix.h - -include $(top_srcdir)/src/mpid/pamid/src/Makefile.mk - -endif BUILD_PAMID diff --git a/src/mpid/pamid/cross/bgq8 b/src/mpid/pamid/cross/bgq8 deleted file mode 100644 index 7cecbb1..0000000 --- a/src/mpid/pamid/cross/bgq8 +++ /dev/null @@ -1,27 +0,0 @@ -# begin_generated_IBM_copyright_prolog -# -# This is an automatically generated copyright prolog. -# After initializing, DO NOT MODIFY OR MOVE -# --------------------------------------------------------------- -# Licensed Materials - Property of IBM -# Blue Gene/Q 5765-PER 5765-PRP -# -# (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved -# US Government Users Restricted Rights - -# Use, duplication, or disclosure restricted -# by GSA ADP Schedule Contract with IBM Corp. -# -# --------------------------------------------------------------- -# -# end_generated_IBM_copyright_prolog -CROSS_F77_SIZEOF_INTEGER=4 -CROSS_F77_SIZEOF_REAL=4 -CROSS_F77_SIZEOF_DOUBLE_PRECISION=8 -CROSS_F90_ADDRESS_KIND=8 -CROSS_F90_OFFSET_KIND=8 -CROSS_F90_INTEGER_KIND=8 -CROSS_F90_REAL_MODEL=6,37 -CROSS_F90_DOUBLE_MODEL=15,307 -CROSS_F90_INTEGER_MODEL_MAP={9,4,4}, -CROSS_F77_TRUE_VALUE=1 -CROSS_F77_FALSE_VALUE=0 diff --git a/src/mpid/pamid/cross/pe4 b/src/mpid/pamid/cross/pe4 deleted file mode 100644 index 2fc1d9e..0000000 --- a/src/mpid/pamid/cross/pe4 +++ /dev/null @@ -1,27 +0,0 @@ -# begin_generated_IBM_copyright_prolog -# -# This is an automatically generated copyright prolog. -# After initializing, DO NOT MODIFY OR MOVE -# --------------------------------------------------------------- -# Licensed Materials - Property of IBM -# Blue Gene/Q 5765-PER 5765-PRP -# -# (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved -# US Government Users Restricted Rights - -# Use, duplication, or disclosure restricted -# by GSA ADP Schedule Contract with IBM Corp. -# -# --------------------------------------------------------------- -# -# end_generated_IBM_copyright_prolog -CROSS_F77_SIZEOF_INTEGER=4 -CROSS_F77_SIZEOF_REAL=4 -CROSS_F77_SIZEOF_DOUBLE_PRECISION=8 -CROSS_F90_ADDRESS_KIND=4 -CROSS_F90_OFFSET_KIND=8 -CROSS_F90_INTEGER_KIND=4 -CROSS_F90_REAL_MODEL=6,37 -CROSS_F90_DOUBLE_MODEL=15,307 -CROSS_F90_INTEGER_MODEL_MAP={9,4,4}, -CROSS_F77_TRUE_VALUE=1 -CROSS_F77_FALSE_VALUE=0 \ No newline at end of file diff --git a/src/mpid/pamid/cross/pe4-aix b/src/mpid/pamid/cross/pe4-aix deleted file mode 100644 index f3478d1..0000000 --- a/src/mpid/pamid/cross/pe4-aix +++ /dev/null @@ -1,28 +0,0 @@ -# begin_generated_IBM_copyright_prolog -# -# This is an automatically generated copyright prolog. -# After initializing, DO NOT MODIFY OR MOVE -# --------------------------------------------------------------- -# Licensed Materials - Property of IBM -# Blue Gene/Q 5765-PER 5765-PRP -# -# (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved -# US Government Users Restricted Rights - -# Use, duplication, or disclosure restricted -# by GSA ADP Schedule Contract with IBM Corp. -# -# --------------------------------------------------------------- -# -# end_generated_IBM_copyright_prolog -CROSS_F77_SIZEOF_INTEGER=4 -CROSS_F77_SIZEOF_REAL=4 -CROSS_F77_SIZEOF_DOUBLE_PRECISION=8 -CROSS_F90_ADDRESS_KIND=4 -CROSS_F90_OFFSET_KIND=8 -CROSS_F90_INTEGER_KIND=4 -CROSS_F90_REAL_MODEL=6,37 -CROSS_F90_DOUBLE_MODEL=15,307 -CROSS_F90_INTEGER_MODEL_MAP={9,4,4}, -CROSS_F77_TRUE_VALUE=1 -CROSS_F77_FALSE_VALUE=0 -CROSS_ALIGN_DOUBLE_EXCEPTION="four" diff --git a/src/mpid/pamid/cross/pe8 b/src/mpid/pamid/cross/pe8 deleted file mode 100644 index 866fdd1..0000000 --- a/src/mpid/pamid/cross/pe8 +++ /dev/null @@ -1,27 +0,0 @@ -# begin_generated_IBM_copyright_prolog -# -# This is an automatically generated copyright prolog. -# After initializing, DO NOT MODIFY OR MOVE -# --------------------------------------------------------------- -# Licensed Materials - Property of IBM -# Blue Gene/Q 5765-PER 5765-PRP -# -# (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved -# US Government Users Restricted Rights - -# Use, duplication, or disclosure restricted -# by GSA ADP Schedule Contract with IBM Corp. -# -# --------------------------------------------------------------- -# -# end_generated_IBM_copyright_prolog -CROSS_F77_SIZEOF_INTEGER=4 -CROSS_F77_SIZEOF_REAL=4 -CROSS_F77_SIZEOF_DOUBLE_PRECISION=8 -CROSS_F90_ADDRESS_KIND=8 -CROSS_F90_OFFSET_KIND=8 -CROSS_F90_INTEGER_KIND=8 -CROSS_F90_REAL_MODEL=6,37 -CROSS_F90_DOUBLE_MODEL=15,307 -CROSS_F90_INTEGER_MODEL_MAP={9,4,4}, -CROSS_F77_TRUE_VALUE=1 -CROSS_F77_FALSE_VALUE=0 \ No newline at end of file diff --git a/src/mpid/pamid/cross/pe8-aix b/src/mpid/pamid/cross/pe8-aix deleted file mode 100644 index 8380c89..0000000 --- a/src/mpid/pamid/cross/pe8-aix +++ /dev/null @@ -1,28 +0,0 @@ -# begin_generated_IBM_copyright_prolog -# -# This is an automatically generated copyright prolog. -# After initializing, DO NOT MODIFY OR MOVE -# --------------------------------------------------------------- -# Licensed Materials - Property of IBM -# Blue Gene/Q 5765-PER 5765-PRP -# -# (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved -# US Government Users Restricted Rights - -# Use, duplication, or disclosure restricted -# by GSA ADP Schedule Contract with IBM Corp. -# -# --------------------------------------------------------------- -# -# end_generated_IBM_copyright_prolog -CROSS_F77_SIZEOF_INTEGER=4 -CROSS_F77_SIZEOF_REAL=4 -CROSS_F77_SIZEOF_DOUBLE_PRECISION=8 -CROSS_F90_ADDRESS_KIND=8 -CROSS_F90_OFFSET_KIND=8 -CROSS_F90_INTEGER_KIND=8 -CROSS_F90_REAL_MODEL=6,37 -CROSS_F90_DOUBLE_MODEL=15,307 -CROSS_F90_INTEGER_MODEL_MAP={9,4,4}, -CROSS_F77_TRUE_VALUE=1 -CROSS_F77_FALSE_VALUE=0 -CROSS_ALIGN_DOUBLE_EXCEPTION="four" diff --git a/src/mpid/pamid/include/.gitignore b/src/mpid/pamid/include/.gitignore deleted file mode 100644 index ea8dee9..0000000 --- a/src/mpid/pamid/include/.gitignore +++ /dev/null @@ -1 +0,0 @@ -/mpid_config.h.in diff --git a/src/mpid/pamid/include/mpidi_constants.h b/src/mpid/pamid/include/mpidi_constants.h deleted file mode 100644 index 25c7c5a..0000000 --- a/src/mpid/pamid/include/mpidi_constants.h +++ /dev/null @@ -1,128 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file include/mpidi_constants.h - * \brief ??? - */ -/* - * (C) 2001 by Argonne National Laboratory. - * See COPYRIGHT in top-level directory. - */ - - -#ifndef __include_mpidi_constants_h__ -#define __include_mpidi_constants_h__ - -enum - { - /* N is "number of operations" and P is number of ranks */ - MPIDI_VERBOSE_NONE = 0, /**< Do not print any verbose information */ - MPIDI_VERBOSE_SUMMARY_0 = 1, /**< Print summary information on rank 0. O(1) lines printed. */ - MPIDI_VERBOSE_SUMMARY_ALL = 2, /**< Print summary information on all ranks. O(P) lines printed. */ - MPIDI_VERBOSE_DETAILS_0 = 2, /**< Print detailed information on rank 0. O(N) lines printed. */ - MPIDI_VERBOSE_DETAILS_ALL = 3, /**< Print detailed information on all ranks. O(P*N) lines printed. */ - }; - - -/** - * \defgroup Allgather(v) optimization datatype info - * \{ - */ -enum - { - MPID_SEND_CONTIG = 0, /**< Contiguous send buffer */ - MPID_RECV_CONTIG = 1, /**< Contiguous recv buffer */ - MPID_RECV_CONTINUOUS = 2, /**< Continuous recv buffer */ - MPID_LARGECOUNT = 3, /**< Total send count is "large" */ - MPID_MEDIUMCOUNT = 4, /**< Total send count is "medium" */ - MPID_ALIGNEDBUFFER = 5, /**< Buffers are 16b aligned */ - }; - -enum - { - MPID_ALLGATHER_PREALLREDUCE = 0, - MPID_ALLGATHERV_PREALLREDUCE = 1, - MPID_ALLREDUCE_PREALLREDUCE = 2, - MPID_BCAST_PREALLREDUCE = 3, - MPID_SCATTERV_PREALLREDUCE = 4, - MPID_GATHER_PREALLREDUCE = 5, - MPID_NUM_PREALLREDUCES = 6, - }; - -enum /* The type of protocol selected */ - { - MPID_COLL_NOQUERY = 0, - MPID_COLL_QUERY = 1, - /* Can we cache stuff? If not set to ALWAYS_QUERY */ - MPID_COLL_ALWAYS_QUERY = 2, - MPID_COLL_CHECK_FN_REQUIRED = 3, - MPID_COLL_USE_MPICH = 4, - MPID_COLL_NOSELECTION = 5, - MPID_COLL_OPTIMIZED = 6, - }; - -enum - { - MPID_COLL_OFF = 0, - MPID_COLL_ON = 1, - MPID_COLL_FCA = 2, /* Selecting these is fairly easy so special case */ - MPID_COLL_CUDA = 3, /* This is used to enable PAMI geometry but sets default to MPICH */ - }; -/** \} */ - - -enum -{ -MPID_EPOTYPE_NONE = 0, /**< No epoch in affect */ -MPID_EPOTYPE_LOCK = 1, /**< MPI_Win_lock access epoch */ -MPID_EPOTYPE_START = 2, /**< MPI_Win_start access epoch */ -MPID_EPOTYPE_POST = 3, /**< MPI_Win_post exposure epoch */ -MPID_EPOTYPE_FENCE = 4, /**< MPI_Win_fence access/exposure epoch */ -MPID_EPOTYPE_REFENCE = 5, /**< MPI_Win_fence possible access/exposure epoch */ -MPID_EPOTYPE_LOCK_ALL = 6, /**< MPI_Win_lock_all access epoch */ -}; - -enum -{ - MPID_AUTO_SELECT_COLLS_NONE = 0, - MPID_AUTO_SELECT_COLLS_BARRIER = 1, - MPID_AUTO_SELECT_COLLS_BCAST = ((int)((MPID_AUTO_SELECT_COLLS_BARRIER << 1) & 0xFFFFFFFF)), - MPID_AUTO_SELECT_COLLS_ALLGATHER = ((int)((MPID_AUTO_SELECT_COLLS_BCAST << 1) & 0xFFFFFFFF)), - MPID_AUTO_SELECT_COLLS_ALLGATHERV = ((int)((MPID_AUTO_SELECT_COLLS_ALLGATHER << 1) & 0xFFFFFFFF)), - MPID_AUTO_SELECT_COLLS_ALLREDUCE = ((int)((MPID_AUTO_SELECT_COLLS_ALLGATHERV << 1) & 0xFFFFFFFF)), - MPID_AUTO_SELECT_COLLS_ALLTOALL = ((int)((MPID_AUTO_SELECT_COLLS_ALLREDUCE << 1) & 0xFFFFFFFF)), - MPID_AUTO_SELECT_COLLS_ALLTOALLV = ((int)((MPID_AUTO_SELECT_COLLS_ALLTOALL << 1) & 0xFFFFFFFF)), - MPID_AUTO_SELECT_COLLS_EXSCAN = ((int)((MPID_AUTO_SELECT_COLLS_ALLTOALLV << 1) & 0xFFFFFFFF)), - MPID_AUTO_SELECT_COLLS_GATHER = ((int)((MPID_AUTO_SELECT_COLLS_EXSCAN << 1) & 0xFFFFFFFF)), - MPID_AUTO_SELECT_COLLS_GATHERV = ((int)((MPID_AUTO_SELECT_COLLS_GATHER << 1) & 0xFFFFFFFF)), - MPID_AUTO_SELECT_COLLS_REDUCE_SCATTER = ((int)((MPID_AUTO_SELECT_COLLS_GATHERV << 1) & 0xFFFFFFFF)), - MPID_AUTO_SELECT_COLLS_REDUCE = ((int)((MPID_AUTO_SELECT_COLLS_REDUCE_SCATTER << 1) & 0xFFFFFFFF)), - MPID_AUTO_SELECT_COLLS_SCAN = ((int)((MPID_AUTO_SELECT_COLLS_REDUCE << 1) & 0xFFFFFFFF)), - MPID_AUTO_SELECT_COLLS_SCATTER = ((int)((MPID_AUTO_SELECT_COLLS_SCAN << 1) & 0xFFFFFFFF)), - MPID_AUTO_SELECT_COLLS_SCATTERV = ((int)((MPID_AUTO_SELECT_COLLS_SCATTER << 1) & 0xFFFFFFFF)), - MPID_AUTO_SELECT_COLLS_TUNE = 0x80000000, - MPID_AUTO_SELECT_COLLS_ALL = 0xFFFFFFFF, -}; - -enum /* PAMID_COLLECTIVES_MEMORY_OPTIMIZED levels */ - -{ - MPIR_OPT_LVL_IRREG = 1, /**< Do not optimize irregular communicators */ - MPIR_OPT_LVL_NONCONTIG = 2, /**< Disable some non-contig collectives */ -}; -#endif diff --git a/src/mpid/pamid/include/mpidi_datatypes.h b/src/mpid/pamid/include/mpidi_datatypes.h deleted file mode 100644 index b716fc9..0000000 --- a/src/mpid/pamid/include/mpidi_datatypes.h +++ /dev/null @@ -1,658 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file include/mpidi_datatypes.h - * \brief ??? - */ -/* - * (C) 2001 by Argonne National Laboratory. - * See COPYRIGHT in top-level directory. - */ - - -#ifndef __include_mpidi_datatypes_h__ -#define __include_mpidi_datatypes_h__ - -#ifdef MPIDI_STATISTICS -#include <pami_ext_pe.h> -#endif -#include "mpidi_constants.h" -#include "mpidi_platform.h" -#include "pami.h" -#include "mpidi_trace.h" - -#include "opa_primitives.h" - - - -#if (MPIU_HANDLE_ALLOCATION_METHOD == MPIU_HANDLE_ALLOCATION_THREAD_LOCAL) && defined(__BGQ__) -struct MPIR_Request; -typedef struct -{ - struct MPIR_Request * head; - size_t count; -} MPIDI_RequestHandle_t; -#endif - -#define MPIDI_PT2PT_LIMIT_SET(is_internal,is_immediate,is_local,value) \ - MPIDI_Process.pt2pt.limits_lookup[is_internal][is_immediate][is_local] = value\ - -typedef struct -{ - unsigned remote; - unsigned local; -} MPIDI_remote_and_local_limits_t; - -typedef struct -{ - MPIDI_remote_and_local_limits_t eager; - MPIDI_remote_and_local_limits_t immediate; -} MPIDI_immediate_and_eager_limits_t; - -typedef struct -{ - MPIDI_immediate_and_eager_limits_t application; - MPIDI_immediate_and_eager_limits_t internal; -} MPIDI_pt2pt_limits_t; - -/** - * \brief MPI Process descriptor - * - * This structure contains global configuration flags. - */ -typedef struct -{ - unsigned avail_contexts; - union - { - unsigned limits_array[8]; - unsigned limits_lookup[2][2][2]; - MPIDI_pt2pt_limits_t limits; - } pt2pt; - unsigned disable_internal_eager_scale; /**< The number of tasks at which point eager will be disabled */ -#if TOKEN_FLOW_CONTROL - unsigned long long mp_buf_mem; - unsigned long long mp_buf_mem_max; - unsigned is_token_flow_control_on; -#endif -#if (MPIDI_STATISTICS || MPIDI_PRINTENV) - unsigned mp_infolevel; - unsigned mp_statistics; /* print pamid statistcs data */ - unsigned mp_printenv; ; /* print env data */ - unsigned mp_interrupts; ; /* interrupts */ -#endif -#ifdef RDMA_FAILOVER - unsigned mp_s_use_pami_get; /* force the PAMI_Get path instead of PAMI_Rget */ -#endif - -#if (MPIU_HANDLE_ALLOCATION_METHOD == MPIU_HANDLE_ALLOCATION_THREAD_LOCAL) && defined(__BGQ__) - MPIDI_RequestHandle_t request_handles[MPIDI_MAX_THREADS]; -#endif - -#if QUEUE_BINARY_SEARCH_SUPPORT - unsigned queue_binary_search_support_on; -#endif - -#if CUDA_AWARE_SUPPORT - unsigned cuda_aware_support_on; -#endif - - unsigned verbose; /**< The current level of verbosity for end-of-job stats. */ - unsigned statistics; /**< The current level of stats collection. */ - unsigned rma_pending; /**< The max num outstanding requests during an RMA op */ - unsigned shmem_pt2pt; /**< Enable optimized shared memory point-to-point functions. */ - unsigned smp_detect; - pami_geometry_t world_geometry; - - struct - { - unsigned collectives; /**< Enable optimized collective functions. */ - unsigned subcomms; /**< Enable hardware optimized subcomm's */ - unsigned select_colls; /**< Enable collective selection */ - unsigned auto_select_colls; /**< Enable automatic collective selection */ - unsigned memory; /**< Enable memory optimized subcomm's - See MPIR_OPT_LVL_xxxx */ - unsigned num_requests; /**< Number of requests between flow control barriers */ - } - optimized; - - struct - { - volatile unsigned active; /**< Number of contexts with active async progress */ - unsigned mode; /**< 0 == 'disabled', 1 == 'locked', 2 == 'trigger' */ - } - async_progress; - - struct - { - struct - { - unsigned requested; /**< 1 == application requests context post */ - unsigned active; /**< 1 == context post is currently required */ - } context_post; - } perobj; /**< This structure is only used in the 'perobj' mpich lock mode. */ - - unsigned mpir_nbc; /**< Enable MPIR_* non-blocking collectives implementations. */ - int numTasks; /* total number of tasks on a job */ - unsigned typed_onesided; /**< Enable typed PAMI calls for derived types within MPID_Put and MPID_Get. */ -#ifdef DYNAMIC_TASKING - struct MPIDI_PG_t * my_pg; /**< Process group I belong to */ - int my_pg_rank; /**< Rank in process group */ -#endif -} MPIDI_Process_t; - - -enum - { - MPIDI_Protocols_Short, - MPIDI_Protocols_ShortSync, - MPIDI_Protocols_Eager, - MPIDI_Protocols_RVZ, - MPIDI_Protocols_Cancel, - MPIDI_Protocols_Control, - MPIDI_Protocols_WinCtrl, - MPIDI_Protocols_WinAccum, - MPIDI_Protocols_RVZ_zerobyte, - MPIDI_Protocols_WinGetAccum, - MPIDI_Protocols_WinGetAccumAck, -#ifdef DYNAMIC_TASKING - MPIDI_Protocols_Dyntask, - MPIDI_Protocols_Dyntask_disconnect, -#endif - MPIDI_Protocols_WinAtomic, - MPIDI_Protocols_WinAtomicAck, - MPIDI_Protocols_COUNT, - }; - - -/** - * \brief This defines the type of message being sent/received - * mpid_startall() invokes the correct start based on the type of the request - */ -typedef enum - { - MPIDI_REQUEST_PTYPE_RECV, - MPIDI_REQUEST_PTYPE_SEND, - MPIDI_REQUEST_PTYPE_BSEND, - MPIDI_REQUEST_PTYPE_SSEND, - } MPIDI_REQUEST_PTYPE; - - -typedef enum - { - MPIDI_CONTROL_SSEND_ACKNOWLEDGE, - MPIDI_CONTROL_CANCEL_REQUEST, - MPIDI_CONTROL_CANCEL_ACKNOWLEDGE, - MPIDI_CONTROL_CANCEL_NOT_ACKNOWLEDGE, - MPIDI_CONTROL_RENDEZVOUS_ACKNOWLEDGE, - MPIDI_CONTROL_RETURN_TOKENS, - } MPIDI_CONTROL; - - -/** \brief Request completion actions */ -typedef enum - { - MPIDI_CA_COMPLETE, - MPIDI_CA_UNPACK_UEBUF_AND_COMPLETE, /**< Unpack uebuf, then complete. */ - } MPIDI_CA; - - -/** - * \brief MPIDI_Message_match contains enough information to match an - * MPI message. - */ -typedef struct -{ - int tag; /**< match tag */ - int rank; /**< match rank */ - int context_id; /**< match context */ -#ifdef OUT_OF_ORDER_HANDLING - int seqno; /**< match seqno */ -#endif -} MPIDI_Message_match; - - -/** - * \brief MPID pt2pt message header - */ -typedef struct -{ - MPI_Request req; /**< peer's request handle */ - unsigned MPItag; /**< match tag */ - unsigned MPIrank; /**< match rank */ - uint16_t MPIctxt; /**< match context */ - - union { - uint16_t flags; - struct { - unsigned control:3; /**< message type for control protocols */ - unsigned isSync:1; /**< set for sync sends */ - unsigned isRzv :1; /**< use pt2pt rendezvous */ - unsigned noRDMA:1; /**< msg sent via shm or mem reg. fails */ - unsigned reserved:6; /**< unused bits */ - unsigned tokens:4; /** tokens need to be returned */ - } __attribute__ ((__packed__)); - }; - -#ifdef OUT_OF_ORDER_HANDLING - unsigned MPIseqno; /**< match seqno */ -#endif -#if TOKEN_FLOW_CONTROL - unsigned alltokens; /* control:MPIDI_CONTROL_RETURN_TOKENS */ -#endif -} MPIDI_MsgInfo; - -/** \brief Full Rendezvous msg info to be set as two quads of unexpected data. */ -typedef struct -{ - MPIDI_MsgInfo msginfo; - pami_memregion_t memregion; -#ifdef RDMA_FAILOVER - uint32_t memregion_used; -#endif - void * data; - size_t length; -} MPIDI_MsgEnvelope; - -/** \brief This defines the portion of MPIR_Request that is specific to the Device */ -struct MPIDI_Request -{ - struct MPIR_Request *next; /**< Link to next req. in queue */ - struct MPIDU_Datatype*datatype_ptr; /**< Info about the datatype */ - pami_work_t post_request; /**< */ - - MPIDI_MsgEnvelope envelope; - - void *userbuf; /**< User buffer */ - MPI_Aint userbufcount; /**< Userbuf data count */ - MPI_Datatype datatype; /**< Data type of message */ - pami_task_t peer_pami; /**< The other guy's rank (in PAMI) */ - unsigned peer_comm; /**< The other guy's rank (in the orig communicator) */ - unsigned cancel_pending:16; /**< Cancel status */ - unsigned uebuf_malloc:16; /**< does uebuf require free() */ - - unsigned uebuflen; /**< Length (bytes) of uebuf */ - void *uebuf; /**< Unexpected buffer */ - - MPIDI_REQUEST_PTYPE ptype; /**< The persistent msg type */ - MPIDI_CA ca; /**< Completion action */ - pami_memregion_t memregion; /**< Rendezvous recv memregion */ -#ifdef OUT_OF_ORDER_HANDLING - struct MPIR_Request *prev; /**< Link to prev req. in queue */ - void *nextR; /** < pointer to next recv for the out-of-order list, the out-of-order list is a list per source */ - void *prevR; /** < pointer to prev recv for the out-of-order list, the out-of-order list is a list per source */ - struct MPIR_Request *oo_peer; /** < pointer to the matched post recv request to complete in the out-of-order case */ -#endif -#ifdef RDMA_FAILOVER - uint32_t memregion_used:16; - uint32_t shm:16; -#endif -#ifdef MPIDI_TRACE - int partner_id; - int idx; - int PR_idx; -#endif - struct MPIDI_Win_request *win_req; /* anchor of request based rma handle so as to free it properly when wait is called */ -}; - -typedef void* fast_query_t; -/** \brief This defines the portion of MPIR_Comm that is specific to the Device */ -struct MPIDI_Comm -{ - pami_geometry_t geometry; /**< Geometry component for collectives */ - pami_geometry_t parent; /**< The parent geometry this communicator came from */ - pami_algorithm_t *coll_algorithm[PAMI_XFER_COUNT][2]; - pami_metadata_t *coll_metadata[PAMI_XFER_COUNT][2]; - char coll_count[PAMI_XFER_COUNT][2]; - pami_algorithm_t user_selected[PAMI_XFER_COUNT]; - /* no way to tell if user_selected[] is NULL */ - /* could probably union these two though? */ - char user_selected_type[PAMI_XFER_COUNT]; - pami_metadata_t user_metadata[PAMI_XFER_COUNT]; - char last_algorithm[100]; - char preallreduces[MPID_NUM_PREALLREDUCES]; - /* \todo Need to figure out how to deal with algorithms above the pami level */ - char allgathers[4]; - char allgathervs[4]; - char scattervs[2]; - char optgather, optscatter, optreduce; - unsigned num_requests; - /* These need to be freed at geom destroy, so we need to store them - * inside the communicator struct until destroy time rather than - * allocating pointers on the stack - */ - /* For create_taskrange */ - pami_geometry_range_t range; - /* For create_tasklist/endpoints if we ever use it */ - pami_task_t *tasks; - pami_endpoint_t *endpoints; - /* There are some protocols where the optimized protocol always works and - * is the best performance */ - /* Assume we have small vs large cutoffs vs medium for some protocols */ - pami_algorithm_t opt_protocol[PAMI_XFER_COUNT][2]; - int must_query[PAMI_XFER_COUNT][2]; - pami_metadata_t opt_protocol_md[PAMI_XFER_COUNT][2]; - int cutoff_size[PAMI_XFER_COUNT][2]; - /* Our best allreduce protocol always works on - * doubles and sum/min/max. Since that is a common - * occurance let's cache that protocol and call - * it without checking. Any other dt/op must be - * checked */ - pami_algorithm_t cached_allreduce; - pami_metadata_t cached_allreduce_md; - int query_cached_allreduce; - - union tasks_descrip_t { - /* For create_taskrange */ - pami_geometry_range_t *ranges; - /* For create_tasklist/endpoints if we ever use it */ - pami_task_t *tasks; - pami_endpoint_t *endpoints; - } tasks_descriptor; -#ifdef DYNAMIC_TASKING - int local_leader; - long long world_intercomm_cntr; - int *world_ids; /* ids of worlds that composed this communicator (inter communicator created for dynamic tasking */ -#endif - fast_query_t collsel_fast_query; -}; - - -typedef struct -{ - pami_work_t state; - pami_xfer_t *coll_struct; -} MPIDI_Post_coll_t; - - -/** \brief Forward declaration of the MPIR_Comm structure */ -struct MPIR_Comm; -/** \brief Forward declaration of the MPIR_Win structure */ -struct MPIR_Win; -/** \brief Forward declaration of the MPIR_Group structure */ -struct MPIR_Group; - -typedef enum - { - MPIDI_REQUEST_LOCK, - MPIDI_REQUEST_LOCKALL, - } MPIDI_LOCK_TYPE_t; - -struct MPIDI_Win_lock -{ - struct MPIDI_Win_lock *next; - unsigned rank; - MPIDI_LOCK_TYPE_t mtype; /* MPIDI_REQUEST_LOCK or MPIDI_REQUEST_LOCKALL */ - int type; - void *flagAddr; -}; -struct MPIDI_Win_queue -{ - struct MPIDI_Win_lock *head; - struct MPIDI_Win_lock *tail; -}; - -typedef enum { - MPIDI_ACCU_ORDER_RAR = 1, - MPIDI_ACCU_ORDER_RAW = 2, - MPIDI_ACCU_ORDER_WAR = 4, - MPIDI_ACCU_ORDER_WAW = 8 -} MPIDI_Win_info_accumulate_ordering; - -typedef enum { - MPIDI_ACCU_SAME_OP, - MPIDI_ACCU_SAME_OP_NO_OP -} MPIDI_Win_info_accumulate_ops; - -typedef struct MPIDI_Win_info_args { - int no_locks; - MPIDI_Win_info_accumulate_ordering accumulate_ordering; - MPIDI_Win_info_accumulate_ops accumulate_ops; /* default is same_op_no_op */ - int same_size; - int alloc_shared_noncontig; -} MPIDI_Win_info_args; - -typedef struct workQ_t { - void *msgQ; - int count; -} workQ_t; - - -/** - * \brief Collective information related to a window - * - * This structure is used to share information about a local window with - * all nodes in the window communicator. Part of that information includes - * statistics about RMA operations during access/exposure epochs. - * - * The structure is allocated as an array sized for the window communicator. - * Each entry in the array corresponds directly to the node of the same rank. - */ -typedef struct MPIDI_Win_info -{ - void * base_addr; /**< Node's exposure window base address */ - struct MPIR_Win * win; - uint32_t disp_unit; /**< Node's exposure window displacement units */ - pami_memregion_t memregion; /**< Memory region descriptor for each node */ - uint32_t memregion_used; - MPI_Aint base_size; /**< Node's exposure window base size in bytes */ -} MPIDI_Win_info; - -typedef pthread_mutex_t MPIDI_SHM_MUTEX; - -typedef struct MPIDI_Win_shm_ctrl_t -{ - MPIDI_SHM_MUTEX mutex_lock; /* shared memory windows -- lock for */ - /* accumulate/atomic operations */ - OPA_int_t active; - int shm_count; -} MPIDI_Win_shm_ctrl_t; - -typedef struct MPIDI_Win_shm_t -{ - int allocated; /* flag: TRUE iff this window has a shared memory - region associated with it */ - void *base_addr; /* base address of shared memory region */ - MPI_Aint segment_len; /* size of shared memory region */ - union - { - uint32_t shm_id; /* shared memory id - sysv */ - char shm_key[64]; /* shared memory key - posix */ - }; - MPIDI_Win_shm_ctrl_t *ctrl; -} MPIDI_Win_shm_t; - -/** - * \brief Structure of PAMI extensions to MPIR_Win structure - */ -struct MPIDI_Win -{ - struct MPIDI_Win_info *info; /**< allocated array of collective info */ - MPIDI_Win_info_args info_args; - void ** shm_base_addrs; /* base address shared by all process in comm */ - MPIDI_Win_shm_t *shm; /* shared memory info */ - workQ_t work; - int max_ctrlsends; - struct MPIDI_Win_sync - { -#if 0 - /** \todo optimize some of the synchronization assertion */ - uint32_t assert; /**< MPI_MODE_* bits asserted at epoch start */ -#endif - - volatile int origin_epoch_type; /**< curretn epoch type for origin */ - volatile int target_epoch_type; /**< curretn epoch type for target */ - - /* These fields are reset by the sync functions */ - uint32_t total; /**< The number of PAMI requests that we know about (updated only by calling thread) */ - volatile uint32_t started; /**< The number of PAMI requests made (updated only in the context_post callback) */ - volatile uint32_t complete; /**< The number of completed PAMI requests (only updated by the done callbacks) */ - - struct MPIDI_Win_sync_pscw - { - struct MPIR_Group * group; - volatile unsigned count; - } sc, pw; - struct MPIDI_Win_sync_lock - { - struct - { - volatile unsigned locked; - volatile unsigned allLocked; - } remote; - struct - { - struct MPIDI_Win_queue requested; - int type; - unsigned count; - } local; - } lock; - } sync; - int request_based; /* flag for request based rma */ - struct MPIR_Request *rreq; /* anchor of MPIR_Request for request based rma */ -}; - -/** - * \brief Structures and typedefs for collective selection extensions in PAMI - */ - -typedef void* advisor_t; -typedef void* advisor_table_t; -typedef void* advisor_attribute_name_t; - -typedef union -{ - size_t intval; - double doubleval; - const char * chararray; - const size_t * intarray; -} advisor_attribute_value_t; - -typedef struct -{ - advisor_attribute_name_t name; - advisor_attribute_value_t value; -} advisor_configuration_t; - -typedef struct { - pami_xfer_type_t *collectives; - size_t num_collectives; - size_t *procs_per_node; - size_t num_procs_per_node; - size_t *geometry_sizes; - size_t num_geometry_sizes; - size_t *message_sizes; - size_t num_message_sizes; - int iter; - int verify; - int verbose; - int checkpoint; -} advisor_params_t; - -typedef enum -{ - COLLSEL_ALGO = 0, /* 'Always works' PAMI algorithm */ - COLLSEL_QUERY_ALGO, /* 'Must query' PAMI algorithm */ - COLLSEL_EXTERNAL_ALGO, /* External algorithm */ -} advisor_algorithm_type_t; - -/* External algorithm callback function */ -typedef pami_result_t (*external_algorithm_fn)(pami_xfer_t *, void *); - -typedef struct -{ - external_algorithm_fn callback; - void *cookie; -} external_algorithm_t; - -typedef struct -{ - union - { - pami_algorithm_t internal;/* PAMI Algorithm */ - external_algorithm_t external;/* External Algorithm */ - } algorithm; - pami_metadata_t *metadata; - advisor_algorithm_type_t algorithm_type; -} advisor_algorithm_t; - - -typedef pami_result_t (*pami_extension_collsel_init) (pami_client_t, - advisor_configuration_t [], - size_t, - pami_context_t [], - size_t, - advisor_t *); - -typedef pami_result_t (*pami_extension_collsel_destroy) (advisor_t *); - -typedef int (*pami_extension_collsel_initialized) (pami_client_t, advisor_t *); - -typedef pami_result_t (*pami_extension_collsel_table_load) (advisor_t, - char *, - advisor_table_t *); - -typedef pami_result_t (*pami_extension_collsel_get_collectives) (advisor_table_t, - pami_xfer_type_t **, - unsigned *); - -typedef pami_result_t (*pami_extension_collsel_register_algorithms) (advisor_table_t, - pami_geometry_t, - pami_xfer_type_t, - advisor_algorithm_t *, - size_t); - -typedef pami_result_t (*external_geometry_create_fn)(pami_geometry_range_t* task_slices, - size_t slice_count, - pami_geometry_t *geometry, - void **cookie); - -typedef pami_result_t (*external_geometry_destroy_fn)(void *cookie); - -typedef pami_result_t (*external_register_algorithms_fn)(void *cookie, - pami_xfer_type_t collective, - advisor_algorithm_t **algorithms, - size_t *num_algorithms); - -typedef struct -{ - external_geometry_create_fn geometry_create; - external_geometry_destroy_fn geometry_destroy; - external_register_algorithms_fn register_algorithms; -} external_geometry_ops_t; - -typedef pami_result_t (*pami_extension_collsel_table_generate) (advisor_t, - char *, - advisor_params_t *, - external_geometry_ops_t *, - int); - -typedef pami_result_t (*pami_extension_collsel_query_create) (advisor_table_t advisor_table, - pami_geometry_t geometry, - fast_query_t *query); - -typedef pami_result_t (*pami_extension_collsel_query_destroy) (fast_query_t *query); - -typedef int (*pami_extension_collsel_advise) (fast_query_t fast_query, - pami_xfer_type_t xfer_type, - size_t message_size, - advisor_algorithm_t algorithms_optimized[], - size_t max_algorithms); - - - -#endif diff --git a/src/mpid/pamid/include/mpidi_externs.h b/src/mpid/pamid/include/mpidi_externs.h deleted file mode 100644 index 774e5c4..0000000 --- a/src/mpid/pamid/include/mpidi_externs.h +++ /dev/null @@ -1,43 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file include/mpidi_externs.h - * \brief ??? - */ -/* - * (C) 2001 by Argonne National Laboratory. - * See COPYRIGHT in top-level directory. - */ - - -#ifndef __include_mpidi_externs_h__ -#define __include_mpidi_externs_h__ - - -extern pami_client_t MPIDI_Client; -extern pami_context_t MPIDI_Context[]; - -extern MPIDI_Process_t MPIDI_Process; - -extern advisor_table_t MPIDI_Collsel_advisor_table; -extern pami_extension_t MPIDI_Collsel_extension; -extern advisor_params_t MPIDI_Collsel_advisor_params; -extern char *MPIDI_Collsel_output_file; -extern pami_extension_collsel_advise MPIDI_Pamix_collsel_advise; - -#endif diff --git a/src/mpid/pamid/include/mpidi_hooks.h b/src/mpid/pamid/include/mpidi_hooks.h deleted file mode 100644 index 54e47af..0000000 --- a/src/mpid/pamid/include/mpidi_hooks.h +++ /dev/null @@ -1,66 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file include/mpidi_hooks.h - * \brief ??? - */ -/* - * (C) 2001 by Argonne National Laboratory. - * See COPYRIGHT in top-level directory. - */ - - -#ifndef __include_mpidi_hooks_h__ -#define __include_mpidi_hooks_h__ - - -struct MPID_VCR_t { - pami_task_t taskid; -#ifdef DYNAMIC_TASKING - int pg_rank; /** rank in process group **/ - struct MPIDI_PG *pg; /** process group **/ -#endif -}; -typedef struct MPID_VCR_t * MPID_VCR ; -typedef struct MPIDI_VCRT * MPID_VCRT; - - -typedef size_t intptr_t; - -#define MPID_Irsend MPID_Isend -#define MPID_Rsend MPID_Send -#define MPID_Rsend_init MPID_Send_init - - -/** \brief Our progress engine does not require state */ -#define MPID_PROGRESS_STATE_DECL - -/** \brief This defines the portion of MPIR_Request that is specific to the Device */ -#define MPID_DEV_REQUEST_DECL struct MPIDI_Request mpid; - -/** \brief This defines the portion of MPIR_Comm that is specific to the Device */ -#define MPID_DEV_COMM_DECL struct MPIDI_Comm mpid; - -/** \brief This defines the portion of MPIR_Win that is specific to the Device */ -#define MPID_DEV_WIN_DECL struct MPIDI_Win mpid; - -#define MPID_Comm_create_hook(a) ({ int MPIDI_Comm_create (MPIR_Comm *comm); MPIDI_Comm_create (a); }) -#define MPID_Comm_free_hook(a) ({ int MPIDI_Comm_destroy(MPIR_Comm *comm); MPIDI_Comm_destroy(a); }) - - -#endif diff --git a/src/mpid/pamid/include/mpidi_macros.h b/src/mpid/pamid/include/mpidi_macros.h deleted file mode 100644 index 9fd3cb4..0000000 --- a/src/mpid/pamid/include/mpidi_macros.h +++ /dev/null @@ -1,217 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file include/mpidi_macros.h - * \brief ??? - */ -/* - * (C) 2001 by Argonne National Laboratory. - * See COPYRIGHT in top-level directory. - */ - - -#ifndef __include_mpidi_macros_h__ -#define __include_mpidi_macros_h__ - -#include "mpidi_datatypes.h" -#include "mpidi_externs.h" - -#define TOKEN_FLOW_CONTROL_ON (TOKEN_FLOW_CONTROL && MPIU_Token_on()) - -#ifdef TRACE_ON -#ifdef __GNUC__ -#define TRACE_ALL(fd, format, ...) fprintf(fd, "%s:%u (%d) " format, __FILE__, __LINE__, MPIR_Process.comm_world->rank, ##__VA_ARGS__) -#define TRACE_OUT(format, ...) TRACE_ALL(stdout, format, ##__VA_ARGS__) -#define TRACE_ERR(format, ...) TRACE_ALL(stderr, format, ##__VA_ARGS__) -#else -#define TRACE_OUT(format...) fprintf(stdout, format) -#define TRACE_ERR(format...) fprintf(stderr, format) -#endif -#else -#define TRACE_OUT(format...) -#define TRACE_ERR(format...) -#endif - -#if TOKEN_FLOW_CONTROL -#define MPIU_Token_on() (MPIDI_Process.is_token_flow_control_on) -#else -#define MPIU_Token_on() (0) -#endif - -/** - * \brief Gets significant info regarding the datatype - * Used in mpid_send, mpidi_send. - */ -#define MPIDI_Datatype_get_info(_count, _datatype, \ -_dt_contig_out, _data_sz_out, _dt_ptr, _dt_true_lb) \ -({ \ - if (HANDLE_GET_KIND(_datatype) == HANDLE_KIND_BUILTIN) \ - { \ - (_dt_ptr) = NULL; \ - (_dt_contig_out) = TRUE; \ - (_dt_true_lb) = 0; \ - (_data_sz_out) = (_count) * \ - MPIDU_Datatype_get_basic_size(_datatype); \ - } \ - else \ - { \ - MPIDU_Datatype_get_ptr((_datatype), (_dt_ptr)); \ - (_dt_contig_out) = (_dt_ptr)->is_contig; \ - (_dt_true_lb) = (_dt_ptr)->true_lb; \ - (_data_sz_out) = (_count) * (_dt_ptr)->size; \ - } \ -}) - -/** - * \brief Gets data size of the datatype - */ -#define MPIDI_Datatype_get_data_size(_count, _datatype, \ -_data_sz_out) \ -({ \ - if (HANDLE_GET_KIND(_datatype) == HANDLE_KIND_BUILTIN) \ - { \ - (_data_sz_out) = (_count) * \ - MPIDU_Datatype_get_basic_size(_datatype); \ - } \ - else \ - { \ - MPIDU_Datatype*_dt_ptr; \ - MPIDU_Datatype_get_ptr((_datatype), (_dt_ptr)); \ - (_data_sz_out) = (_count) * (_dt_ptr)->size; \ - } \ -}) - -/* Add some error checking for size eventually */ -#define MPIDI_Update_last_algorithm(_comm, _name) \ -({ strncpy( (_comm)->mpid.last_algorithm, (_name), strlen((_name))+1); }) - - -/** - * \brief Macro for allocating memory - * - * \param[in] count Number of elements to allocate - * \param[in] type The type of the memory, excluding "*" - * \return Address or NULL - */ -#define MPL_calloc0(count, type) \ -({ \ - size_t __size = (count) * sizeof(type); \ - type* __p = MPL_malloc(__size); \ - MPID_assert(__p != NULL); \ - if (__p != NULL) \ - memset(__p, 0, __size); \ - __p; \ -}) - -#define MPIU_TestFree(p) \ -({ \ - if (*(p) != NULL) \ - { \ - MPL_free(*(p)); \ - *(p) = NULL; \ - } \ -}) - - -#define MPID_VCR_GET_LPID(vcr, index) \ -({ \ - vcr[index]->taskid; \ -}) - -#define MPID_VCR_GET_LPIDS(comm, taskids) \ -({ \ - int i; \ - taskids=MPL_malloc((comm->local_size)*sizeof(pami_task_t)); \ - MPID_assert(taskids != NULL); \ - for(i=0; i<comm->local_size; i++) \ - taskids[i] = comm->vcr[i]->taskid; \ -}) -#define MPID_VCR_FREE_LPIDS(taskids) MPL_free(taskids) - -#define MPID_GPID_Get(comm_ptr, rank, gpid) \ -({ \ - gpid[1] = MPID_VCR_GET_LPID(comm_ptr->vcr, rank); \ - gpid[0] = 0; \ - MPI_SUCCESS; /* return success from macro */ \ -}) - - -static inline void -MPIDI_Context_post(pami_context_t context, - pami_work_t * work, - pami_work_function fn, - void * cookie) -{ -#if (MPICH_THREAD_GRANULARITY == MPICH_THREAD_GRANULARITY__POBJ) - /* It is possible that a work function posted to a context may attempt to - * initiate a communication operation and, if context post were disabled, that - * operation would be performed directly on the context BY TAKING A LOCK that - * the is already held by the thread that is advancing the context. This will - * result in a hang. - * - * A solution would be to identify all code flows where this situation might - * occur and then change the code to avoid taking a lock that is already held. - * - * Another solution is to always disable the "non-context-post" configuration - * when compiled with per-object locking. This would only occur if the - * application requested !MPI_THREAD_MULTIPLE and the "pretend single threaded - * by disabling async progress, context post, and multiple contexts" optimization - * was in effect. - */ - pami_result_t rc; - rc = PAMI_Context_post(context, work, fn, cookie); - MPID_assert(rc == PAMI_SUCCESS); -#else /* (MPICH_THREAD_GRANULARITY != MPICH_THREAD_GRANULARITY__POBJ) */ - /* - * It is not necessary to lock the context before access in the "global" - * mpich lock mode because all threads, application and async progress, - * must first acquire the global mpich lock upon entry into the library. - */ - fn(context, cookie); -#endif -} - -#if (MPICH_THREAD_GRANULARITY == MPICH_THREAD_GRANULARITY__POBJ) -#define MPIDI_Send_post(__func, __req) \ -({ \ - pami_context_t context = MPIDI_Context_local(__req); \ - \ - if (likely(MPIDI_Process.perobj.context_post.active > 0)) \ - { \ - pami_result_t rc; \ - rc = PAMI_Context_post(context, \ - &(__req)->mpid.post_request, \ - __func, \ - __req); \ - MPID_assert(rc == PAMI_SUCCESS); \ - } \ - else \ - { \ - PAMI_Context_lock(context); \ - __func(context, __req); \ - PAMI_Context_unlock(context); \ - } \ -}) -#else /* (MPICH_THREAD_GRANULARITY != MPICH_THREAD_GRANULARITY__POBJ) */ -#define MPIDI_Send_post(__func, __req) \ -({ \ - __func(MPIDI_Context[0], __req); \ -}) -#endif /* #if (MPICH_THREAD_GRANULARITY == MPICH_THREAD_GRANULARITY__POBJ) */ - -#endif diff --git a/src/mpid/pamid/include/mpidi_mutex.h b/src/mpid/pamid/include/mpidi_mutex.h deleted file mode 100644 index 0fdb0f4..0000000 --- a/src/mpid/pamid/include/mpidi_mutex.h +++ /dev/null @@ -1,374 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file include/mpidi_mutex.h - * \brief ??? - */ -/* - * (C) 2001 by Argonne National Laboratory. - * See COPYRIGHT in top-level directory. - */ -#ifndef __include_mpidi_mutex_h__ -#define __include_mpidi_mutex_h__ - -#include <opa_primitives.h> -#include <mpiimpl.h> -#include <malloc.h> - -#define MPIDI_THREAD_ID() Kernel_ProcessorID() - -#if MPIDI_MUTEX_L2_ATOMIC - - - -#define MUTEX_FAIL 0x8000000000000000UL - -#include <spi/include/kernel/location.h> -#include <spi/include/kernel/memory.h> -#include <spi/include/l2/atomic.h> - - -#define MPIDI_MAX_MUTEXES 16 -typedef struct -{ - uint64_t counter; - uint64_t bound; -} MPIDI_Mutex_t; - -extern MPIDI_Mutex_t * MPIDI_Mutex_vector; -extern uint32_t MPIDI_Mutex_counter[MPIDI_MAX_THREADS][MPIDI_MAX_MUTEXES]; -int MPIDI_Mutex_initialize(); - - -/** - * \brief Try to acquire a mutex identified by an index. - * \param[in] m Index of the mutex - * \return 0 Lock successfully acquired - * \return 1 Lock was not acquired - */ -static inline int -MPIDI_Mutex_try_acquire(unsigned m) -{ - -#if MPIDI_MUTEX_RECURSIVE - size_t tid = MPIDI_THREAD_ID(); - MPID_assert(m < MPIDI_MAX_MUTEXES); - if (MPIDI_Mutex_counter[tid][m] >= 1) { - ++MPIDI_Mutex_counter[tid][m]; - return 0; - } -#endif - - MPIDI_Mutex_t *mutex = &(MPIDI_Mutex_vector[m]); - size_t rc = L2_AtomicLoadIncrementBounded(&mutex->counter); - - if (rc == MUTEX_FAIL) - return 1; - -#if MPIDI_MUTEX_RECURSIVE - MPIDI_Mutex_counter[tid][m] = 1; -#endif - return 0; /* Lock succeeded */ -} - - -/** - * \brief Acquire a mutex identified by an index. - * \param[in] m Index of the mutex - * \return 0 Lock successfully acquired - * \return 1 Fail - */ -static inline int -MPIDI_Mutex_acquire(unsigned m) -{ -#if MPIDI_MUTEX_RECURSIVE - size_t tid = MPIDI_THREAD_ID(); - MPID_assert(m < MPIDI_MAX_MUTEXES); - - if (unlikely(MPIDI_Mutex_counter[tid][m] >= 1)) { - ++MPIDI_Mutex_counter[tid][m]; - return 0; - } -#endif - - MPIDI_Mutex_t *mutex = &(MPIDI_Mutex_vector[m]); - size_t rc = 0; - do { - rc = L2_AtomicLoadIncrementBounded(&mutex->counter); - } while (rc == MUTEX_FAIL); - -#if MPIDI_MUTEX_RECURSIVE - MPIDI_Mutex_counter[tid][m] = 1; -#endif - return 0; -} - - -/** - * \brief Release a mutex identified by an index. - * \param[in] m Index of the mutex - * \return 0 Lock successfully released - * \return 1 Fail - */ -static inline int -MPIDI_Mutex_release(unsigned m) -{ -#if MPIDI_MUTEX_RECURSIVE - size_t tid = MPIDI_THREAD_ID(); - MPID_assert(m < MPIDI_MAX_MUTEXES); - /* Verify this thread is the owner of this lock */ - MPID_assert(MPIDI_Mutex_counter[tid][m] > 0); - - --MPIDI_Mutex_counter[tid][m]; - MPID_assert(MPIDI_Mutex_counter[tid][m] >= 0); - if (unlikely(MPIDI_Mutex_counter[tid][m] > 0)) - return 0; /* Future calls will release the lock to other threads */ -#endif - - /* Wait till all the writes in the critical sections from this - thread have completed and invalidates have been delivered */ - //OPA_read_write_barrier(); - - /* Release the lock */ - L2_AtomicStore(&(MPIDI_Mutex_vector[m].counter), 0); - - return 0; -} - - -#define MPIDI_Mutex_sync() OPA_read_write_barrier() - - - -#elif MPIDI_MUTEX_LLSC - - - -#include <spi/include/kernel/location.h> - -#define MPIDI_MAX_MUTEXES 16 -typedef OPA_int_t MPIDI_Mutex_t; -extern MPIDI_Mutex_t MPIDI_Mutex_vector [MPIDI_MAX_MUTEXES]; -extern uint32_t MPIDI_Mutex_counter[MPIDI_MAX_THREADS][MPIDI_MAX_MUTEXES]; - -/** - * \brief Initialize a mutex. - * - * In this API, mutexes are acessed via indices from - * 0..MPIDI_MAX_MUTEXES. The mutexes are recursive - */ -static inline int -MPIDI_Mutex_initialize() -{ - size_t i, j; - for (i=0; i<MPIDI_MAX_MUTEXES; ++i) { - OPA_store_int(&(MPIDI_Mutex_vector[i]), 0); - } - - for (i=0; i<MPIDI_MAX_MUTEXES; ++i) { - for (j=0; j<MPIDI_MAX_THREADS; ++j) { - MPIDI_Mutex_counter[j][i] = 0; - } - } - - return 0; -} - - -/** - * \brief Try to acquire a mutex identified by an index. - * \param[in] m Index of the mutex - * \return 0 Lock successfully acquired - * \return 1 Lock was not acquired - */ -static inline int -MPIDI_Mutex_try_acquire(unsigned m) -{ - register int old_val; - size_t tid = MPIDI_THREAD_ID(); - - MPID_assert(m < MPIDI_MAX_MUTEXES); - - if (MPIDI_Mutex_counter[tid][m] >= 1) { - ++MPIDI_Mutex_counter[tid][m]; - return 0; - } - - MPIDI_Mutex_t *mutex = &(MPIDI_Mutex_vector[m]); - old_val = OPA_LL_int(mutex); - if (old_val != 0) - return 1; /* Lock failed */ - - int rc = OPA_SC_int(mutex, 1); /* returns 0 when SC fails */ - - if (rc == 0) - return 1; /* Lock failed */ - - MPIDI_Mutex_counter[tid][m] = 1; - return 0; /* Lock succeeded */ -} - - -/** - * \brief Acquire a mutex identified by an index. - * \param[in] m Index of the mutex - * \return 0 Lock successfully acquired - * \return 1 Fail - */ -static inline int -MPIDI_Mutex_acquire(unsigned m) -{ - register int old_val; - size_t tid = MPIDI_THREAD_ID(); - - MPID_assert(m < MPIDI_MAX_MUTEXES); - - if (unlikely(MPIDI_Mutex_counter[tid][m] >= 1)) { - ++MPIDI_Mutex_counter[tid][m]; - return 0; - } - - MPIDI_Mutex_t *mutex = &(MPIDI_Mutex_vector[m]); - do { - do { - old_val = OPA_LL_int(mutex); - } while (old_val != 0); - - } while (!OPA_SC_int(mutex, 1)); - - MPIDI_Mutex_counter[tid][m] = 1; - return 0; -} - - -/** - * \brief Release a mutex identified by an index. - * \param[in] m Index of the mutex - * \return 0 Lock successfully released - * \return 1 Fail - */ -static inline int -MPIDI_Mutex_release(unsigned m) -{ - size_t tid = MPIDI_THREAD_ID(); - MPID_assert(m < MPIDI_MAX_MUTEXES); - /* Verify this thread is the owner of this lock */ - MPID_assert(MPIDI_Mutex_counter[tid][m] > 0); - - --MPIDI_Mutex_counter[tid][m]; - MPID_assert(MPIDI_Mutex_counter[tid][m] >= 0); - if (unlikely(MPIDI_Mutex_counter[tid][m] > 0)) - return 0; /* Future calls will release the lock to other threads */ - - /* Wait till all the writes in the critical sections from this - thread have completed and invalidates have been delivered */ - //OPA_read_write_barrier(); - - /* Release the lock */ - OPA_store_int(&(MPIDI_Mutex_vector[m]), 0); - - return 0; -} - - -#define MPIDI_Mutex_sync() OPA_read_write_barrier() - - - -#else - - - -extern pthread_mutex_t MPIDI_Mutex_lock; - -static inline int -MPIDI_Mutex_initialize() -{ - int rc; - - pthread_mutexattr_t attr; - rc = pthread_mutexattr_init(&attr); - MPID_assert(rc == 0); -#if !defined(__AIX__) - extern int pthread_mutexattr_settype(pthread_mutexattr_t *__attr, int __kind) __THROW __nonnull ((1)); -#else - extern int pthread_mutexattr_settype(pthread_mutexattr_t *__attr, int __kind); -#endif -#ifndef __PE__ -#if (MPICH_THREAD_GRANULARITY == MPICH_THREAD_GRANULARITY__POBJ) - rc = pthread_mutexattr_settype(&attr, PTHREAD_MUTEX_RECURSIVE_NP); -#else /*(MPICH_THREAD_GRANULARITY == MPICH_THREAD_GRANULARITY__POBJ)*/ - rc = pthread_mutexattr_settype(&attr, PTHREAD_MUTEX_ERRORCHECK_NP); -#endif /*(MPICH_THREAD_GRANULARITY == MPICH_THREAD_GRANULARITY__POBJ)*/ -#else /* __PE__ */ -#if !defined(__AIX__) - rc = pthread_mutexattr_settype(&attr, PTHREAD_MUTEX_RECURSIVE_NP); -#else - rc = pthread_mutexattr_settype(&attr, PTHREAD_MUTEX_RECURSIVE); -#endif -#endif - MPID_assert(rc == 0); - - rc = pthread_mutex_init(&MPIDI_Mutex_lock, &attr); - MPID_assert(rc == 0); - - return 0; -} - - -static inline int -MPIDI_Mutex_try_acquire(unsigned m) -{ - int rc; - rc = pthread_mutex_trylock(&MPIDI_Mutex_lock); - MPID_assert( (rc == 0) || (rc == EBUSY) ); - /* fprintf(stderr, "%s:%u (rc=%d)\n", __FUNCTION__, __LINE__, rc); */ - return rc; -} - - -static inline int -MPIDI_Mutex_acquire(unsigned m) -{ - int rc; - /* fprintf(stderr, "%s:%u\n", __FUNCTION__, __LINE__); */ - rc = pthread_mutex_lock(&MPIDI_Mutex_lock); - /* fprintf(stderr, "%s:%u (rc=%d)\n", __FUNCTION__, __LINE__, rc); */ - MPID_assert(rc == 0); - return rc; -} - - -static inline int -MPIDI_Mutex_release(unsigned m) -{ - int rc; - rc = pthread_mutex_unlock(&MPIDI_Mutex_lock); - /* fprintf(stderr, "%s:%u (rc=%d)\n", __FUNCTION__, __LINE__, rc); */ - MPID_assert(rc == 0); - return rc; -} - - -#define MPIDI_Mutex_sync() - - -#endif - - -#endif diff --git a/src/mpid/pamid/include/mpidi_platform.h b/src/mpid/pamid/include/mpidi_platform.h deleted file mode 100644 index cd95e72..0000000 --- a/src/mpid/pamid/include/mpidi_platform.h +++ /dev/null @@ -1,196 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file include/mpidi_platform.h - * \brief ??? - */ - - -#ifndef __include_mpidi_platform_h__ -#define __include_mpidi_platform_h__ - -/* Allow MPICH to detect local tasks */ -#define MPID_USE_NODE_IDS -typedef int32_t MPID_Node_id_t; - -/* Default values */ - -#define MPIDI_MAX_CONTEXTS 64 -/** This is not the real value, but should default to something larger than PAMI_DISPATCH_SEND_IMMEDIATE_MAX */ -#define MPIDI_SHORT_LIMIT 555 -/** This is set to 4 BGQ torus packets (+1, because of the way it is compared) */ -#define MPIDI_EAGER_LIMIT 2049 -/** This is set to 0 which effectively disables the eager protocol for local transfers */ -#define MPIDI_EAGER_LIMIT_LOCAL 0 -/** This is set to 'max unsigned' which effectively never disables internal eager at scale */ -#define MPIDI_DISABLE_INTERNAL_EAGER_SCALE ((unsigned)-1) - -/* Default features */ -#define USE_PAMI_RDMA 1 -#define USE_PAMI_CONSISTENCY PAMI_HINT_ENABLE -#undef OUT_OF_ORDER_HANDLING -#undef DYNAMIC_TASKING -#undef RDMA_FAILOVER -#undef QUEUE_BINARY_SEARCH_SUPPORT - -#define ASYNC_PROGRESS_MODE_DEFAULT 0 - -/* - * The default behavior is to disable (ignore) the 'internal vs application' and - * the 'local vs remote' point-to-point eager limits. - */ -#define MPIDI_PT2PT_EAGER_LIMIT(is_internal,is_local) \ -({ \ - MPIDI_Process.pt2pt.limits_lookup[0][0][0]; \ -}) - -/* - * The default behavior is to disable (ignore) the 'internal vs application' and - * the 'local vs remote' point-to-point short limits. - */ -#define MPIDI_PT2PT_SHORT_LIMIT(is_internal,is_local) \ -({ \ - MPIDI_Process.pt2pt.limits_lookup[0][1][0]; \ -}) - - - -#ifdef __BGQ__ -#undef MPIDI_EAGER_LIMIT_LOCAL -#define MPIDI_EAGER_LIMIT_LOCAL 4097 -#undef MPIDI_EAGER_LIMIT -#define MPIDI_EAGER_LIMIT 4097 -#undef MPIDI_DISABLE_INTERNAL_EAGER_SCALE -#define MPIDI_DISABLE_INTERNAL_EAGER_SCALE (512*1024) -#define MPIDI_MAX_THREADS 64 -#define MPIDI_MUTEX_L2_ATOMIC 1 -#define MPIDI_OPTIMIZED_COLLECTIVE_DEFAULT 1 - -#define PAMIX_IS_LOCAL_TASK -#define PAMIX_IS_LOCAL_TASK_STRIDE (4) -#define PAMIX_IS_LOCAL_TASK_SHIFT (6) -#define MPIDI_SMP_DETECT_DEFAULT 1 -#define TOKEN_FLOW_CONTROL 0 -#define CUDA_AWARE_SUPPORT 0 - -/* - * Enable both the 'internal vs application' and the 'local vs remote' - * point-to-point eager limits. - */ -#undef MPIDI_PT2PT_EAGER_LIMIT -#define MPIDI_PT2PT_EAGER_LIMIT(is_internal,is_local) \ -({ \ - MPIDI_Process.pt2pt.limits_lookup[is_internal][0][is_local]; \ -}) - -/* - * Enable both the 'internal vs application' and the 'local vs remote' - * point-to-point short limits. - */ -#undef MPIDI_PT2PT_SHORT_LIMIT -#define MPIDI_PT2PT_SHORT_LIMIT(is_internal,is_local) \ -({ \ - MPIDI_Process.pt2pt.limits_lookup[is_internal][1][is_local]; \ -}) - - -#undef ASYNC_PROGRESS_MODE_DEFAULT -#define ASYNC_PROGRESS_MODE_DEFAULT 1 - -static const char _ibm_release_version_[] = "V1R2M0"; -#endif - -#ifdef __PE__ - -/* - * This 'maximum contexts' define needs to be changed when mpich on PE - * will support multiple contexts. Currently the PE PAMI allows multiple - * contexts, but the PE mpich code is not set up to use them. - */ -#undef MPIDI_MAX_CONTEXTS -#define MPIDI_MAX_CONTEXTS 1 - -#undef USE_PAMI_CONSISTENCY -#define USE_PAMI_CONSISTENCY PAMI_HINT_DISABLE -#undef MPIDI_SHORT_LIMIT -#define MPIDI_SHORT_LIMIT 256 - sizeof(MPIDI_MsgInfo) -#undef MPIDI_EAGER_LIMIT -#define MPIDI_EAGER_LIMIT 65536 -#undef MPIDI_EAGER_LIMIT_LOCAL -#define MPIDI_EAGER_LIMIT_LOCAL 1048576 -#define OUT_OF_ORDER_HANDLING 1 -#define MPIDI_STATISTICS 1 -#define MPIDI_PRINTENV 1 -#define MPIDI_OPTIMIZED_COLLECTIVE_DEFAULT 0 -#undef USE_PAMI_RDMA -#define RDMA_FAILOVER -#define MPIDI_BANNER 1 -#define MPIDI_NO_ASSERT 1 -#define TOKEN_FLOW_CONTROL 1 -#define DYNAMIC_TASKING 1 -#define CUDA_AWARE_SUPPORT 1 - -/* 'is local task' extension and limits */ -#define PAMIX_IS_LOCAL_TASK -#define PAMIX_IS_LOCAL_TASK_STRIDE (1) -#define PAMIX_IS_LOCAL_TASK_SHIFT (0) -#define MPIDI_SMP_DETECT_DEFAULT 1 -/* - * Enable only the 'local vs remote' point-to-point eager limits. - */ -#undef MPIDI_PT2PT_EAGER_LIMIT -#define MPIDI_PT2PT_EAGER_LIMIT(is_internal,is_local) \ -({ \ - MPIDI_Process.pt2pt.limits_lookup[0][0][is_local]; \ -}) - -/* - * Enable only the 'local vs remote' point-to-point short limits. - */ -#undef MPIDI_PT2PT_SHORT_LIMIT -#define MPIDI_PT2PT_SHORT_LIMIT(is_internal,is_local) \ -({ \ - MPIDI_Process.pt2pt.limits_lookup[0][1][is_local]; \ -}) - - -#undef ASYNC_PROGRESS_MODE_DEFAULT -#define ASYNC_PROGRESS_MODE_DEFAULT 2 - -/* When the Pok build team extracts this file from CMVC, %W% will expand to */ -/* a string with the current release, for example ppe_rbarlx. */ -/* If this file is cloned from GIT then %W% will not be expanded. The */ -/* banner code has accounted for this situation. */ -static const char _ibm_release_version_[] = "%W%"; - -#endif - -#if TOKEN_FLOW_CONTROL -#define BUFFER_MEM_DEFAULT (1<<26) /* 64MB */ -#define BUFFER_MEM_MAX (1<<26) /* 64MB */ -#define ONE_SHARED_SEGMENT (1<<28) /* 256MB */ -#define EAGER_LIMIT_DEFAULT 65536 -#define MAX_BUF_BKT_SIZE (1<<18) /* Max eager_limit is 256K */ -#define MIN_BUF_BKT_SIZE (64) -#define TOKENS_BIT (4) /* 4 bits piggy back to sender */ - /* should be consistent with tokens - defined in MPIDI_MsgInfo */ -#define TOKENS_BITMASK ((1 << TOKENS_BIT)-1) -#endif - -#endif diff --git a/src/mpid/pamid/include/mpidi_prototypes.h b/src/mpid/pamid/include/mpidi_prototypes.h deleted file mode 100644 index fc683ba..0000000 --- a/src/mpid/pamid/include/mpidi_prototypes.h +++ /dev/null @@ -1,460 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file include/mpidi_prototypes.h - * \brief ??? - */ -/* - * (C) 2001 by Argonne National Laboratory. - * See COPYRIGHT in top-level directory. - */ - - -#ifndef __include_mpidi_prototypes_h__ -#define __include_mpidi_prototypes_h__ - -#if CUDA_AWARE_SUPPORT -#include <cuda_runtime_api.h> -#endif - - -/** - * \addtogroup MPID_RECVQ - * \{ - */ -void MPIDI_Recvq_init(); -void MPIDI_Recvq_finalize(); -int MPIDI_Recvq_FU (int s, int t, int c, MPI_Status * status); -MPIR_Request * MPIDI_Recvq_FDUR (MPI_Request req, int source, int tag, int context_id); -int MPIDI_Recvq_FDPR (MPIR_Request * req); -#ifndef OUT_OF_ORDER_HANDLING -MPIR_Request * MPIDI_Recvq_FDP_or_AEU(MPIR_Request *newreq, int s, int t, int c, int * foundp); -MPIR_Request * MPIDI_Recvq_FDU (int source, int tag, int context_id, int * foundp); -MPIR_Request * MPIDI_Recvq_AEU (MPIR_Request *newreq, int source, int tag, int context_id); -#else -MPIR_Request * MPIDI_Recvq_FDP_or_AEU(MPIR_Request *newreq, int s, pami_task_t ps, int t, int c, int sq, int * foundp); -MPIR_Request * MPIDI_Recvq_FDU (int source, pami_task_t pami_source, int tag, int context_id, int * foundp); -MPIR_Request * MPIDI_Recvq_AEU (MPIR_Request *newreq, int source, pami_task_t pami_source, int tag, int context_id, int msg_seqno); -#endif -void MPIDI_Recvq_DumpQueues (int verbose); -#ifdef OUT_OF_ORDER_HANDLING -void MPIDI_Recvq_enqueue_ool (pami_task_t s, MPIR_Request * r); -void MPIDI_Recvq_insert_ool (MPIR_Request *q,MPIR_Request *e); -#endif -/** \} */ - -void MPIDI_Buffer_copy(const void * const sbuf, - MPI_Aint scount, - MPI_Datatype sdt, - int * smpi_errno, - void * const rbuf, - MPI_Aint rcount, - MPI_Datatype rdt, - intptr_t * rsz, - int * rmpi_errno); - -pami_result_t MPIDI_Send_handoff (pami_context_t context, void * sreq); -pami_result_t MPIDI_Ssend_handoff(pami_context_t context, void * sreq); -pami_result_t MPIDI_Isend_handoff(pami_context_t context, void * sreq); -pami_result_t MPIDI_Isend_handoff_internal(pami_context_t context, void * sreq); - -void MPIDI_RecvMsg_procnull(MPIR_Comm * comm, - unsigned is_blocking, - MPI_Status * status, - MPIR_Request ** request); -void MPIDI_RecvMsg_Unexp(MPIR_Request * rreq, void * buf, MPI_Aint count, MPI_Datatype datatype); - -/** - * \defgroup MPID_CALLBACKS MPID callbacks for communication - * - * These calls are used to manage message asynchronous start and completion - * - * \addtogroup MPID_CALLBACKS - * \{ - */ -void MPIDI_SendDoneCB (pami_context_t context, - void * clientdata, - pami_result_t result); - -void MPIDI_RecvShortAsyncCB(pami_context_t context, - void * cookie, - const void * _msginfo, - size_t msginfo_size, - const void * sndbuf, - size_t sndlen, - pami_endpoint_t sender, - pami_recv_t * recv); -void MPIDI_RecvShortSyncCB (pami_context_t context, - void * cookie, - const void * _msginfo, - size_t msginfo_size, - const void * sndbuf, - size_t sndlen, - pami_endpoint_t sender, - pami_recv_t * recv); -void MPIDI_RecvCB (pami_context_t context, - void * cookie, - const void * _msginfo, - size_t msginfo_size, - const void * sndbuf, - size_t sndlen, - pami_endpoint_t sender, - pami_recv_t * recv); -void MPIDI_RecvRzvCB (pami_context_t context, - void * cookie, - const void * _msginfo, - size_t msginfo_size, - const void * sndbuf, - size_t sndlen, - pami_endpoint_t sender, - pami_recv_t * recv); -void MPIDI_RecvRzvCB_zerobyte (pami_context_t context, - void * cookie, - const void * _msginfo, - size_t msginfo_size, - const void * sndbuf, - size_t sndlen, - pami_endpoint_t sender, - pami_recv_t * recv); -void MPIDI_RecvDoneCB (pami_context_t context, - void * clientdata, - pami_result_t result); -void MPIDI_RecvDoneCB_mutexed(pami_context_t context, - void * clientdata, - pami_result_t result); -void MPIDI_RecvRzvDoneCB (pami_context_t context, - void * cookie, - pami_result_t result); -void MPIDI_RecvRzvDoneCB_zerobyte (pami_context_t context, - void * cookie, - pami_result_t result); -#ifdef DYNAMIC_TASKING -void MPIDI_Recvfrom_remote_world (pami_context_t context, - void * cookie, - const void * _msginfo, - size_t msginfo_size, - const void * sndbuf, - size_t sndlen, - pami_endpoint_t sender, - pami_recv_t * recv); -void MPIDI_Recvfrom_remote_world_disconnect (pami_context_t context, - void * cookie, - const void * _msginfo, - size_t msginfo_size, - const void * sndbuf, - size_t sndlen, - pami_endpoint_t sender, - pami_recv_t * recv); -#endif -#ifdef OUT_OF_ORDER_HANDLING -void MPIDI_Recvq_process_out_of_order_msgs(pami_task_t src, pami_context_t context); -int MPIDI_Recvq_search_recv_posting_queue(int src, int tag, int context_id, - MPIR_Request **handleptr ); -#endif - -void MPIDI_Callback_process_unexp(MPIR_Request *newreq, - pami_context_t context, - const MPIDI_MsgInfo * msginfo, - size_t sndlen, - pami_endpoint_t senderendpoint, - const void * sndbuf, - pami_recv_t * recv, - unsigned isSync); -void MPIDI_Callback_process_trunc(pami_context_t context, - MPIR_Request *rreq, - pami_recv_t *recv, - const void *sndbuf); -void MPIDI_Callback_process_userdefined_dt(pami_context_t context, - const void * sndbuf, - size_t sndlen, - MPIR_Request * rreq); -/** \} */ - - -/** \brief Acknowledge an MPI_Ssend() */ -void MPIDI_SyncAck_post(pami_context_t context, MPIR_Request * req, unsigned rank); -pami_result_t MPIDI_SyncAck_handoff(pami_context_t context, void * inputReq); -/** \brief This is the general PT2PT control message call-back */ -void MPIDI_ControlCB(pami_context_t context, - void * cookie, - const void * _msginfo, - size_t msginfo_size, - const void * sndbuf, - size_t sndlen, - pami_endpoint_t sender, - pami_recv_t * recv); -void -MPIDI_WinControlCB(pami_context_t context, - void * cookie, - const void * _control, - size_t size, - const void * sndbuf, - size_t sndlen, - pami_endpoint_t sender, - pami_recv_t * recv); -void -MPIDI_WinAtomicCB(pami_context_t context, - void * cookie, - const void * _control, - size_t size, - const void * sndbuf, - size_t sndlen, - pami_endpoint_t sender, - pami_recv_t * recv); -void -MPIDI_WinAtomicAckCB(pami_context_t context, - void * cookie, - const void * _control, - size_t size, - const void * sndbuf, - size_t sndlen, - pami_endpoint_t sender, - pami_recv_t * recv); - - -void -MPIDI_WinGetAccumCB(pami_context_t context, - void * cookie, - const void * _control, - size_t size, - const void * sndbuf, - size_t sndlen, - pami_endpoint_t sender, - pami_recv_t * recv); -void -MPIDI_WinGetAccumAckCB(pami_context_t context, - void * cookie, - const void * _control, - size_t size, - const void * sndbuf, - size_t sndlen, - pami_endpoint_t sender, - pami_recv_t * recv); - -/** \brief Helper function to complete a rendevous transfer */ -pami_result_t MPIDI_RendezvousTransfer(pami_context_t context, void* rreq); -pami_result_t MPIDI_RendezvousTransfer_SyncAck(pami_context_t context, void* rreq); -pami_result_t MPIDI_RendezvousTransfer_zerobyte(pami_context_t context, void* rreq); - - -int MPIDI_Comm_create (MPIR_Comm *comm); -int MPIDI_Comm_destroy (MPIR_Comm *comm); -void MPIDI_Coll_comm_create (MPIR_Comm *comm); -void MPIDI_Coll_comm_destroy (MPIR_Comm *comm); -void MPIDI_Env_setup (); -void MPIDI_Comm_world_setup (); - -pami_result_t MPIDI_Comm_create_from_pami_geom(pami_geometry_range_t *task_slices, - size_t slice_count, - pami_geometry_t *geometry, - void **cookie); -pami_result_t MPIDI_Comm_destroy_external(void *comm_ext); -pami_result_t MPIDI_Register_algorithms_ext(void *cookie, - pami_xfer_type_t collective, - advisor_algorithm_t **algorithms, - size_t *num_algorithms); -int MPIDI_collsel_pami_tune_parse_params(int argc, char ** argv); -void MPIDI_collsel_pami_tune_cleanup(); -#if CUDA_AWARE_SUPPORT -int CudaMemcpy( void* dst, const void* src, size_t count, int kind ); -int CudaPointerGetAttributes( struct cudaPointerAttributes* attributes, const void* ptr ); -const char * CudaGetErrorString( int error); -#endif -inline bool MPIDI_enable_cuda(); -inline bool MPIDI_cuda_is_device_buf(const void* ptr); -void MPIDI_Coll_Comm_create (MPIR_Comm *comm); -void MPIDI_Coll_Comm_destroy(MPIR_Comm *comm); -void MPIDI_Comm_coll_query (MPIR_Comm *comm); -void MPIDI_Comm_coll_envvars(MPIR_Comm *comm); -void MPIDI_Comm_coll_select(MPIR_Comm *comm); -void MPIDI_Coll_register (void); - -int MPIDO_Bcast(void *buffer, int count, MPI_Datatype dt, int root, MPIR_Comm *comm_ptr, int *mpierrno); -int MPIDO_Bcast_simple(void *buffer, int count, MPI_Datatype dt, int root, MPIR_Comm *comm_ptr, int *mpierrno); -int MPIDO_CSWrapper_bcast(pami_xfer_t *bcast, void *comm); -int MPIDO_Ibcast(void *buffer, int count, MPI_Datatype datatype, int root, MPIR_Comm *comm_ptr, MPIR_Request **request); -int MPIDO_Barrier(MPIR_Comm *comm_ptr, int *mpierrno); -int MPIDO_Barrier_simple(MPIR_Comm *comm_ptr, int *mpierrno); -int MPIDO_CSWrapper_barrier(pami_xfer_t *barrier, void *comm); -int MPIDO_Ibarrier(MPIR_Comm *comm_ptr, MPIR_Request **request); - -int MPIDO_Allreduce(const void *sbuffer, void *rbuffer, int count, - MPI_Datatype datatype, MPI_Op op, MPIR_Comm *comm_ptr, int *mpierrno); -int MPIDO_Allreduce_simple(const void *sbuffer, void *rbuffer, int count, - MPI_Datatype datatype, MPI_Op op, MPIR_Comm *comm_ptr, int *mpierrno); -int MPIDO_CSWrapper_allreduce(pami_xfer_t *allreduce, void *comm); -int MPIDO_Iallreduce(const void *sbuffer, void *rbuffer, int count, - MPI_Datatype datatype, MPI_Op op, MPIR_Comm *comm_ptr, - MPIR_Request ** request); -int MPIDO_Reduce(const void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, - MPI_Op op, int root, MPIR_Comm *comm_ptr, int *mpierrno); -int MPIDO_Reduce_simple(const void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, - MPI_Op op, int root, MPIR_Comm *comm_ptr, int *mpierrno); -int MPIDO_CSWrapper_reduce(pami_xfer_t *reduce, void *comm); -int MPIDO_Ireduce(const void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, - MPI_Op op, int root, MPIR_Comm *comm_ptr, MPIR_Request **request); -int MPIDO_Allgather(const void *sendbuf, int sendcount, MPI_Datatype sendtype, - void *recvbuf, int recvcount, MPI_Datatype recvtype, - MPIR_Comm *comm_ptr, int *mpierrno); -int MPIDO_Allgather_simple(const void *sendbuf, int sendcount, MPI_Datatype sendtype, - void *recvbuf, int recvcount, MPI_Datatype recvtype, - MPIR_Comm *comm_ptr, int *mpierrno); -int MPIDO_CSWrapper_allgather(pami_xfer_t *allgather, void *comm); -int MPIDO_Iallgather(const void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, - int recvcount, MPI_Datatype recvtype, MPIR_Comm *comm_ptr, - MPIR_Request **request); - -int MPIDO_Allgatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype, - void *recvbuf, const int *recvcounts, const int *displs, - MPI_Datatype recvtype, MPIR_Comm * comm_ptr, int *mpierrno); -int MPIDO_Allgatherv_simple(const void *sendbuf, int sendcount, MPI_Datatype sendtype, - void *recvbuf, const int *recvcounts, const int *displs, - MPI_Datatype recvtype, MPIR_Comm * comm_ptr, int *mpierrno); -int MPIDO_CSWrapper_allgatherv(pami_xfer_t *allgatherv, void *comm); -int MPIDO_Iallgatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype, - void *recvbuf, const int *recvcounts, const int *displs, - MPI_Datatype recvtype, MPIR_Comm * comm_ptr, - MPIR_Request ** request); - -int MPIDO_Gather(const void *sendbuf, int sendcount, MPI_Datatype sendtype, - void *recvbuf, int recvcount, MPI_Datatype recvtype, - int root, MPIR_Comm * comm_ptr, int *mpierrno); -int MPIDO_Gather_simple(const void *sendbuf, int sendcount, MPI_Datatype sendtype, - void *recvbuf, int recvcount, MPI_Datatype recvtype, - int root, MPIR_Comm * comm_ptr, int *mpierrno); -int MPIDO_CSWrapper_gather(pami_xfer_t *gather, void *comm); -int MPIDO_Igather(const void *sendbuf, int sendcount, MPI_Datatype sendtype, - void *recvbuf, int recvcount, MPI_Datatype recvtype, - int root, MPIR_Comm * comm_ptr, MPIR_Request **request); - -int MPIDO_Gatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype, - void *recvbuf, const int *recvcounts, const int *displs, MPI_Datatype recvtype, - int root, MPIR_Comm * comm_ptr, int *mpierrno); -int MPIDO_Gatherv_simple(const void *sendbuf, int sendcount, MPI_Datatype sendtype, - void *recvbuf, const int *recvcounts, const int *displs, MPI_Datatype recvtype, - int root, MPIR_Comm * comm_ptr, int *mpierrno); -int MPIDO_CSWrapper_gatherv(pami_xfer_t *gatherv, void *comm); -int MPIDO_Igatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype, - void *recvbuf, const int *recvcounts, const int *displs, MPI_Datatype recvtype, - int root, MPIR_Comm * comm_ptr, MPIR_Request **request); - -int MPIDO_Scan(const void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, - MPI_Op op, MPIR_Comm * comm_ptr, int *mpierrno); -int MPIDO_Scan_simple(const void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, - MPI_Op op, MPIR_Comm * comm_ptr, int *mpierrno); -int MPIDO_CSWrapper_scan(pami_xfer_t *scan, void *comm); -int MPIDO_Iscan(const void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, - MPI_Op op, MPIR_Comm * comm_ptr, MPIR_Request **request); - -int MPIDO_Exscan(const void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, - MPI_Op op, MPIR_Comm * comm_ptr, int *mpierrno); -int MPIDO_Exscan_simple(const void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, - MPI_Op op, MPIR_Comm * comm_ptr, int *mpierrno); -int MPIDO_Iexscan(const void *sendbuf, void *recvbuf, int count, MPI_Datatype datatype, - MPI_Op op, MPIR_Comm * comm_ptr, MPIR_Request **request); - -int MPIDO_Scatter(const void *sendbuf, int sendcount, MPI_Datatype sendtype, - void *recvbuf, int recvcount, MPI_Datatype recvtype, - int root, MPIR_Comm * comm_ptr, int *mpierrno); -int MPIDO_Scatter_simple(const void *sendbuf, int sendcount, MPI_Datatype sendtype, - void *recvbuf, int recvcount, MPI_Datatype recvtype, - int root, MPIR_Comm * comm_ptr, int *mpierrno); -int MPIDO_CSWrapper_scatter(pami_xfer_t *scatter, void *comm); -int MPIDO_Iscatter(const void *sendbuf, int sendcount, MPI_Datatype sendtype, - void *recvbuf, int recvcount, MPI_Datatype recvtype, - int root, MPIR_Comm * comm_ptr, MPIR_Request **request); - -int MPIDO_Scatterv(const void *sendbuf, const int *sendcounts, const int *displs, - MPI_Datatype sendtype, - void *recvbuf, int recvcount, MPI_Datatype recvtype, - int root, MPIR_Comm * comm_ptr, int *mpierrno); -int MPIDO_Scatterv_simple(const void *sendbuf, const int *sendcounts, const int *displs, - MPI_Datatype sendtype, - void *recvbuf, int recvcount, MPI_Datatype recvtype, - int root, MPIR_Comm * comm_ptr, int *mpierrno); -int MPIDO_CSWrapper_scatterv(pami_xfer_t *scatterv, void *comm); -int MPIDO_Iscatterv(const void *sendbuf, const int *sendcounts, const int *displs, - MPI_Datatype sendtype, - void *recvbuf, int recvcount, MPI_Datatype recvtype, - int root, MPIR_Comm * comm_ptr, MPIR_Request **request); - -int MPIDO_Alltoallv(const void *sendbuf, const int *sendcounts, const int *senddispls, - MPI_Datatype sendtype, - void *recvbuf, const int *recvcounts, const int *recvdispls, - MPI_Datatype recvtype, - MPIR_Comm *comm_ptr, int *mpierrno); -int MPIDO_Alltoallv_simple(const void *sendbuf, const int *sendcounts, const int *senddispls, - MPI_Datatype sendtype, - void *recvbuf, const int *recvcounts, const int *recvdispls, - MPI_Datatype recvtype, - MPIR_Comm *comm_ptr, int *mpierrno); -int MPIDO_CSWrapper_alltoallv(pami_xfer_t *alltoallv, void *comm); -int MPIDO_Ialltoallv(const void *sendbuf, const int *sendcounts, const int *senddispls, - MPI_Datatype sendtype, - void *recvbuf, const int *recvcounts, const int *recvdispls, - MPI_Datatype recvtype, - MPIR_Comm *comm_ptr, MPIR_Request **request); - -int MPIDO_Alltoall(const void *sendbuf, int sendcount, MPI_Datatype sendtype, - void *recvbuf, int recvcount, MPI_Datatype recvtype, - MPIR_Comm *comm_ptr, int *mpierrno); -int MPIDO_Alltoall_simple(const void *sendbuf, int sendcount, MPI_Datatype sendtype, - void *recvbuf, int recvcount, MPI_Datatype recvtype, - MPIR_Comm *comm_ptr, int *mpierrno); -int MPIDO_CSWrapper_alltoall(pami_xfer_t *alltoall, void *comm); -int MPIDO_Ialltoall(const void *sendbuf, int sendcount, MPI_Datatype sendtype, - void *recvbuf, int recvcount, MPI_Datatype recvtype, - MPIR_Comm *comm_ptr, MPIR_Request **request); - -int MPIDO_Ialltoallw(const void *sendbuf, const int *sendcounts, const int *senddispls, - const MPI_Datatype * sendtypes, - void *recvbuf, const int *recvcounts, const int *recvdispls, - const MPI_Datatype * recvtypes, - MPIR_Comm *comm_ptr, MPIR_Request **request); - -int MPIDO_Reduce_scatter(const void *sendbuf, void *recvbuf, int *recvcounts, MPI_Datatype datatype, - MPI_Op op, MPIR_Comm *comm_ptr, int *mpierrno); - -int MPIDO_Reduce_scatter_block(const void *sendbuf, void *recvbuf, int recvcount, - MPI_Datatype datatype, MPI_Op op, MPIR_Comm *comm_ptr, int *mpierrno); - -int MPIDO_Ireduce_scatter_block(const void *sendbuf, void *recvbuf, int recvcount, - MPI_Datatype datatype, MPI_Op op, MPIR_Comm *comm_ptr, - MPIR_Request **request); - -int MPIDO_Ireduce_scatter(const void *sendbuf, void *recvbuf, const int *recvcounts, - MPI_Datatype datatype, MPI_Op op, MPIR_Comm *comm_ptr, MPIR_Request **request); - -int MPIDI_Datatype_to_pami(MPI_Datatype dt, - pami_type_t *pdt, - MPI_Op op, - pami_data_function *pop, - int *musupport); - -int MPIDI_Dtpami_to_dtmpi(pami_type_t pdt, - MPI_Datatype *dt, - pami_data_function pop, - MPI_Op *op); -void MPIDI_Op_to_string(MPI_Op op, char *string); -pami_result_t MPIDI_Pami_post_wrapper(pami_context_t context, void *cookie); - - -void MPIDI_NBC_init (); - - -#endif diff --git a/src/mpid/pamid/include/mpidi_thread.h b/src/mpid/pamid/include/mpidi_thread.h deleted file mode 100644 index f543565..0000000 --- a/src/mpid/pamid/include/mpidi_thread.h +++ /dev/null @@ -1,184 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file include/mpidi_thread.h - * \brief ??? - * - */ -/* - * (C) 2001 by Argonne National Laboratory. - * See COPYRIGHT in top-level directory. - */ -#include "mpidi_mutex.h" - - -#ifndef __include_mpidi_thread_h__ -#define __include_mpidi_thread_h__ - - -/** - * ****************************************************************** - * \brief Mutexes for thread/interrupt safety - * ****************************************************************** - */ - -/* This file is included by mpidpre.h, so it is included before mpiimplthread.h. - * This is intentional because it lets us override the critical section macros */ - -#define MPID_DEVICE_DEFINES_THREAD_CS 1 - - -#if (MPICH_THREAD_LEVEL != MPI_THREAD_MULTIPLE) -#error MPICH_THREAD_LEVEL should be MPI_THREAD_MULTIPLE -#endif - -#define MPIU_THREAD_CS_INIT ({ MPIDI_Mutex_initialize(); }) -#define MPIU_THREAD_CS_FINALIZE - -#define MPIU_THREADSAFE_INIT_DECL(_var) static volatile int _var=1 -#define MPIU_THREADSAFE_INIT_BLOCK_BEGIN(_var) \ - MPIU_THREAD_CS_ENTER(INITFLAG,); \ - if (_var) \ - { -#define MPIU_THREADSAFE_INIT_CLEAR(_var) _var=0 -#define MPIU_THREADSAFE_INIT_BLOCK_END(_var) \ - } \ - MPIU_THREAD_CS_EXIT(INITFLAG,) - - -#define MPIU_THREAD_CS_ENTER(name,_context) MPIU_THREAD_CS_##name##_ENTER(_context) -#define MPIU_THREAD_CS_EXIT(name,_context) MPIU_THREAD_CS_##name##_EXIT (_context) -#define MPIU_THREAD_CS_YIELD(name,_context) MPIU_THREAD_CS_##name##_YIELD(_context) -#define MPIU_THREAD_CS_SCHED_YIELD(name,_context) MPIU_THREAD_CS_##name##_SCHED_YIELD(_context) -#define MPIU_THREAD_CS_TRY(name,_context) MPIU_THREAD_CS_##name##_TRY(_context) - -#if MPICH_THREAD_GRANULARITY == MPICH_THREAD_GRANULARITY__GLOBAL - -#define MPIDI_CS_ENTER(m) ({ if (MPIR_ThreadInfo.isThreaded) { MPIDI_Mutex_acquire(m); } }) -#define MPIDI_CS_EXIT(m) ({ if (MPIR_ThreadInfo.isThreaded) { MPIDI_Mutex_sync(); MPIDI_Mutex_release(m); } }) -#define MPIDI_CS_YIELD(m) ({ if (MPIR_ThreadInfo.isThreaded) { MPIDI_Mutex_sync(); MPIDI_Mutex_release(m); MPIDI_Mutex_acquire(m); } }) -#define MPIDI_CS_TRY(m) ({ (0==MPIDI_Mutex_try_acquire(m)); }) -#define MPIDI_CS_SCHED_YIELD(m) ({ if (MPIR_ThreadInfo.isThreaded) { MPIDI_Mutex_sync(); MPIDI_Mutex_release(m); sched_yield(); MPIDI_Mutex_acquire(m); } }) - -/* There is a single, global lock, held for the duration of an MPI call */ -#define MPIU_THREAD_CS_ALLFUNC_ENTER(_context) MPIDI_CS_ENTER(0) -#define MPIU_THREAD_CS_ALLFUNC_EXIT(_context) MPIDI_CS_EXIT (0) -#define MPIU_THREAD_CS_ALLFUNC_YIELD(_context) MPIDI_CS_YIELD(0) -#define MPIU_THREAD_CS_ALLFUNC_SCHED_YIELD(_context) MPIDI_CS_SCHED_YIELD(0) -#define MPIU_THREAD_CS_ALLFUNC_TRY(_context) MPIDI_CS_TRY(0) -#define MPIU_THREAD_CS_INIT_ENTER(_context) MPIDI_Mutex_acquire(0) -#define MPIU_THREAD_CS_INIT_EXIT(_context) MPIDI_Mutex_release(0) - -#define MPIU_THREAD_CS_CONTEXTID_ENTER(_context) -#define MPIU_THREAD_CS_CONTEXTID_EXIT(_context) -#define MPIU_THREAD_CS_CONTEXTID_YIELD(_context) MPIDI_CS_YIELD(0) -#define MPIU_THREAD_CS_CONTEXTID_SCHED_YIELD(_context) MPIDI_CS_SCHED_YIELD(0) -#define MPIU_THREAD_CS_HANDLEALLOC_ENTER(_context) -#define MPIU_THREAD_CS_HANDLEALLOC_EXIT(_context) -#define MPIU_THREAD_CS_HANDLE_ENTER(_context) -#define MPIU_THREAD_CS_HANDLE_EXIT(_context) -#define MPIU_THREAD_CS_INITFLAG_ENTER(_context) -#define MPIU_THREAD_CS_INITFLAG_EXIT(_context) -#define MPIU_THREAD_CS_MEMALLOC_ENTER(_context) -#define MPIU_THREAD_CS_MEMALLOC_EXIT(_context) -#define MPIU_THREAD_CS_MPI_OBJ_ENTER(_context) MPIDI_CS_ENTER(5) -#define MPIU_THREAD_CS_MPI_OBJ_EXIT(_context) MPIDI_CS_EXIT(5) -#define MPIU_THREAD_CS_MSGQUEUE_ENTER(_context) -#define MPIU_THREAD_CS_MSGQUEUE_EXIT(_context) -#define MPIU_THREAD_CS_PAMI_ENTER(_context) -#define MPIU_THREAD_CS_PAMI_EXIT(_context) -#define MPIU_THREAD_CS_ASYNC_ENTER(_context) MPIDI_CS_ENTER(8) -#define MPIU_THREAD_CS_ASYNC_EXIT(_context) MPIDI_CS_EXIT (8) - -#elif MPICH_THREAD_GRANULARITY == MPICH_THREAD_GRANULARITY__POBJ - -#define MPIDI_CS_ENTER(m) \ - do { \ - if (likely(MPIR_ThreadInfo.isThreaded)) { \ - MPIDI_Mutex_acquire(m); \ - } \ - } while (0) - -#define MPIDI_CS_EXIT(m) \ - do { \ - if (likely(MPIR_ThreadInfo.isThreaded)) { \ - MPIDI_Mutex_sync(); \ - MPIDI_Mutex_release(m); \ - } \ - } while (0) - -#define MPIDI_CS_YIELD(m) \ - do { \ - if (likely(MPIR_ThreadInfo.isThreaded)) { \ - MPIDI_Mutex_sync(); \ - MPIDI_Mutex_release(m); \ - MPIDI_Mutex_acquire(m); \ - } \ - } while (0) - -#define MPIDI_CS_TRY(m) \ - do { \ - if (likely(MPIR_ThreadInfo.isThreaded)) { \ - MPIDI_Mutex_try_acquire(m); \ - } \ - } while (0) - -#define MPIDI_CS_SCHED_YIELD(m) \ - do { \ - if (likely(MPIR_ThreadInfo.isThreaded)) { \ - MPIDI_Mutex_sync(); \ - MPIDI_Mutex_release(m); \ - sched_yield(); \ - MPIDI_Mutex_acquire(m); \ - } \ - } while (0) - -#define MPIU_THREAD_CS_ALLFUNC_ENTER(_context) -#define MPIU_THREAD_CS_ALLFUNC_EXIT(_context) -#define MPIU_THREAD_CS_ALLFUNC_YIELD(_context) -#define MPIU_THREAD_CS_ALLFUNC_SCHED_YIELD(_context) -#define MPIU_THREAD_CS_ALLFUNC_TRY(_context) (0) - -#define MPIU_THREAD_CS_INIT_ENTER(_context) MPIDI_CS_ENTER(0) -#define MPIU_THREAD_CS_INIT_EXIT(_context) MPIDI_CS_EXIT(0) - -#define MPIU_THREAD_CS_CONTEXTID_ENTER(_context) MPIDI_CS_ENTER(0) -#define MPIU_THREAD_CS_CONTEXTID_EXIT(_context) MPIDI_CS_EXIT (0) -#define MPIU_THREAD_CS_CONTEXTID_YIELD(_context) MPIDI_CS_YIELD(0) -#define MPIU_THREAD_CS_CONTEXTID_SCHED_YIELD(_context) MPIDI_CS_SCHED_YIELD(0) -#define MPIU_THREAD_CS_HANDLEALLOC_ENTER(_context) MPIDI_CS_ENTER(1) -#define MPIU_THREAD_CS_HANDLEALLOC_EXIT(_context) MPIDI_CS_EXIT (1) -#define MPIU_THREAD_CS_HANDLE_ENTER(_context) MPIDI_CS_ENTER(2) -#define MPIU_THREAD_CS_HANDLE_EXIT(_context) MPIDI_CS_EXIT (2) -#define MPIU_THREAD_CS_INITFLAG_ENTER(_context) MPIDI_CS_ENTER(3) -#define MPIU_THREAD_CS_INITFLAG_EXIT(_context) MPIDI_CS_EXIT (3) -#define MPIU_THREAD_CS_MEMALLOC_ENTER(_context) MPIDI_CS_ENTER(4) -#define MPIU_THREAD_CS_MEMALLOC_EXIT(_context) MPIDI_CS_EXIT (4) -#define MPIU_THREAD_CS_MPI_OBJ_ENTER(context_) MPIDI_CS_ENTER(5) -#define MPIU_THREAD_CS_MPI_OBJ_EXIT(context_) MPIDI_CS_EXIT (5) -#define MPIU_THREAD_CS_MSGQUEUE_ENTER(_context) MPIDI_CS_ENTER(6) -#define MPIU_THREAD_CS_MSGQUEUE_EXIT(_context) MPIDI_CS_EXIT (6) -#define MPIU_THREAD_CS_PAMI_ENTER(_context) MPIDI_CS_ENTER(7) -#define MPIU_THREAD_CS_PAMI_EXIT(_context) MPIDI_CS_EXIT (7) -#define MPIU_THREAD_CS_ASYNC_ENTER(_context) MPIDI_CS_ENTER(8) -#define MPIU_THREAD_CS_ASYNC_EXIT(_context) MPIDI_CS_EXIT (8) - -#endif /* MPICH_THREAD_GRANULARITY == MPICH_THREAD_GRANULARITY__GLOBAL */ - - -#endif /* !MPICH_MPIDTHREAD_H_INCLUDED */ diff --git a/src/mpid/pamid/include/mpidi_trace.h b/src/mpid/pamid/include/mpidi_trace.h deleted file mode 100644 index a921e68..0000000 --- a/src/mpid/pamid/include/mpidi_trace.h +++ /dev/null @@ -1,266 +0,0 @@ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file include/mpidi_trace.h - * \brief record trace info. for pt2pt comm. - */ -/* - * - * - */ - - -#ifndef __include_mpidi_trace_h__ -#define __include_mpidi_trace_h__ - -#include <sys/time.h> -#include <sys/param.h> - -#ifdef MPIDI_TRACE -#define N_MSGS 1024 -#define SEQMASK N_MSGS-1 -typedef struct { - void *req; /* address of request */ - void *bufadd; /* user's receive buffer address */ - uint msgid; /* msg seqno. */ - unsigned short ctx; /* mpi context id */ - unsigned short dummy; /* reserved */ - uint nMsgs; /* highest msg seqno that arrived in order */ - int tag; - int len; - int rsource; /* source of the message arrived */ - int rtag; /* tag of a received message */ - int rlen; /* len of a received message */ - int rctx; /* context of a received message */ - void * matchedHandle; /* a message with multiple handles */ - union { - uint flags; - struct { -#ifdef __BIG_ENDIAN__ - uint posted:1; /* has the receive posted */ - uint rzv:1; /* rendezvous message ? */ - uint sync:1; /* synchronous message? */ - uint sendAck:1; /* send ack? */ - uint sendFin:1; /* send complete info? */ - uint HH:1; /* header handler */ - uint ool:1; /* the msg arrived out of order */ - uint matchedInOOL:1;/* found a match in out of order list */ - uint comp_in_HH:4; /* the msg completed in header handler */ - uint comp_in_HHV_noMatch:1;/* no matched in header handler EA */ - uint sync_com_in_HH:1; /* sync msg completed in header handler*/ - uint matchedInHH:1; /* found a match in header haldner */ - uint matchedInComp:1;/* found a match in completion handler */ - uint matchedInUQ:2; /* found a match in unexpected queue */ - uint matchedInUQ2:2;/* found a match in unexpected queue */ - uint matchedInWait:1;/* found a match in MPI_Wait() etc. */ - uint ReadySend:1; /* a ready send messsage */ - uint persist:1; /* persist communication */ - uint reserve:1; - uint reserve1:8; -#else - uint reserve1:8; - uint reserve:1; - uint persist:1; /* persist communication */ - uint ReadySend:1; /* a ready send messsage */ - uint matchedInWait:1;/* found a match in MPI_Wait() etc. */ - uint matchedInUQ2:2;/* found a match in unexpected queue */ - uint matchedInUQ:2; /* found a match in unexpected queue */ - uint matchedInComp:1;/* found a match in completion handler */ - uint matchedInHH:1; /* found a match in header haldner */ - uint sync_com_in_HH:1; /* sync msg completed in header handler*/ - uint comp_in_HHV_noMatch:1;/* no matched in header handler EA */ - uint comp_in_HH:4; /* the msg completed in header handler */ - uint matchedInOOL:1;/* found a match in out of order list */ - uint ool:1; /* the msg arrived out of order */ - uint HH:1; /* header handler */ - uint sendFin:1; /* send complete info? */ - uint sendAck:1; /* send ack? */ - uint sync:1; /* synchronous message? */ - uint rzv:1; /* rendezvous message ? */ - uint posted:1; /* has the receive posted */ -#endif - }f; - }fl; -} recv_status; - -typedef struct { - void *req; /* address of request */ - void *bufaddr; /* address of user's send buffer */ - int dest; /* destination of a message */ - int rank; /* rank in a communicator */ - int mode; /* protocol used */ - uint msgid; /* message sequence no. */ - unsigned short sctx; /* context id */ - unsigned short dummy; - int tag; /* tag of a message */ - int len; /* lengh of a message */ - union { - uint flags; - struct { -#ifdef __BIG_ENDIAN__ - uint blocking:1; /* blocking send ? */ - uint sync:1; /* sync message */ - uint sendEnvelop:1; /* envelop send? */ - uint sendShort:1; /* send immediate */ - uint sendEager:1; /* eager send */ - uint sendRzv:1; /* send via renzdvous protocol */ - uint memRegion:1; /* memory is registered */ - uint use_pami_get:1;/* use only PAMI_Get() */ - uint NoComp:4; /* no completion handler */ - uint sendComp:1; /* send complete */ - uint recvAck:1; /* recv an ack from the receiver */ - uint recvFin:1; /* recv complete information */ - uint complSync:1; /* complete sync */ - uint ReadySend:1; /* ready send */ - uint reqXfer:1; /* request message transfer */ - uint persist:1; /* persistent communiation */ - uint reserve:5; - uint reserve1:8; -#else - uint reserve1:8; - uint reserve:5; - uint persist:1; /* persistent communiation */ - uint reqXfer:1; /* request message transfer */ - uint ReadySend:1; /* ready send */ - uint complSync:1; /* complete sync */ - uint recvFin:1; /* recv complete information */ - uint recvAck:1; /* recv an ack from the receiver */ - uint sendComp:1; /* send complete */ - uint NoComp:4; /* no completion handler */ - uint use_pami_get:1;/* use only PAMI_Get() */ - uint memRegion:1; /* memory is registered */ - uint sendRzv:1; /* send via renzdvous protocol */ - uint sendEager:1; /* eager send */ - uint sendShort:1; /* send immediate */ - uint sendEnvelop:1; /* envelop send? */ - uint sync:1; /* sync message */ - uint blocking:1; /* blocking send ? */ -#endif - }f; - }fl; -} send_status; - -typedef struct { - void *req; /* address of a request */ - void *bufadd; /* address of user receive buffer */ - int src_task; /* source PAMI task id */ - int rank; /* rank in a communicator */ - int tag; /* tag of a posted recv */ - int count; /* count of a specified datattype */ - int datatype; - int len; /* length of a receive message */ - uint nMsgs; /* no. of messages have been received */ - uint msgid; /* msg seqno of the matched message */ -#ifdef __BIG_ENDIAN__ - uint sendCtx:16; /* context of incoming msg */ - uint recvCtx:16; /* context of a posted receive */ -#else - uint recvCtx:16; /* context of a posted receive */ - uint sendCtx:16; /* context of incoming msg */ -#endif - union { - uint flags; - struct { -#ifdef __BIG_ENDIAN__ - uint lw:4; /* use lw protocol immediate send */ - uint persist:4; /* persistent communication */ - uint blocking:2; /* blocking receive */ - uint reserve:6; - uint reserve1:16; -#else - uint reserve1:16; - uint reserve:6; - uint blocking:2; /* blocking receive */ - uint persist:4; /* persistent communication */ - uint lw:4; /* use lw protocol immediate send */ -#endif - }f; - }fl; -} posted_recv; - - -typedef struct MPIDI_Trace_buf { - recv_status *R; /* record incoming messages */ - posted_recv *PR; /* record posted receive */ - send_status *S; /* send messages */ - int totPR; /* total no. of poste receive */ -} MPIDI_Trace_buf_t; - -MPIDI_Trace_buf_t *MPIDI_Trace_buf; - - - -#define MPIDI_SET_PR_REC(rreq,buf,ct,dt,pami_id,rank,tag,comm,is_blk) { \ - int idx,src,seqNo,x; \ - if (pami_id != MPI_ANY_SOURCE) \ - src=pami_id; \ - else { \ - src= MPIR_Process.comm_world->rank; \ - } \ - MPIDI_Trace_buf[src].totPR++ ; \ - seqNo=MPIDI_Trace_buf[src].totPR; \ - idx = (seqNo & SEQMASK); \ - memset(&MPIDI_Trace_buf[src].PR[idx],0,sizeof(posted_recv)); \ - MPIDI_Trace_buf[src].PR[idx].src_task= pami_id; \ - MPIDI_Trace_buf[src].PR[idx].rank = rank; \ - MPIDI_Trace_buf[src].PR[idx].bufadd = buf; \ - MPIDI_Trace_buf[src].PR[idx].msgid = seqNo; \ - MPIDI_Trace_buf[src].PR[idx].count = ct; \ - MPIDI_Trace_buf[src].PR[idx].datatype = dt; \ - MPIDI_Trace_buf[src].PR[idx].tag=tag; \ - MPIDI_Trace_buf[src].PR[idx].sendCtx=comm->context_id; \ - MPIDI_Trace_buf[src].PR[idx].recvCtx=comm->recvcontext_id;\ - MPIDI_Trace_buf[src].PR[idx].fl.f.blocking=is_blk; \ - rreq->mpid.PR_idx=idx; \ -} - -#define MPIDI_GET_S_REC(dd,sreq,ctx,isSync,dataSize) { \ - send_status *sstatus; \ - int seqNo=sreq->mpid.envelope.msginfo.MPIseqno; \ - int idx = (seqNo & SEQMASK); \ - sreq->mpid.partner_id=dd; \ - memset(&MPIDI_Trace_buf[dd].S[idx],0,sizeof(send_status));\ - sstatus=&MPIDI_Trace_buf[dd].S[idx]; \ - sstatus->req = (void *)sreq; \ - sstatus->tag = sreq->mpid.envelope.msginfo.MPItag; \ - sstatus->dest = sreq->mpid.peer_pami; \ - sstatus->rank = sreq->mpid.peer_comm; \ - sstatus->msgid = seqNo; \ - sstatus->fl.f.sync = isSync; \ - sstatus->sctx = ctx; \ - sstatus->tag = sreq->mpid.envelope.msginfo.MPItag; \ - sstatus->len= dataSize; \ - sreq->mpid.idx=idx; \ -} - - -#define TRACE_SET_S_BIT(dd,ii,mbr) MPIDI_Trace_buf[(dd)].S[(ii)].mbr=1; -#define TRACE_SET_R_BIT(dd,ii,mbr) MPIDI_Trace_buf[(dd)].R[(ii)].mbr=1; -#define TRACE_SET_S_VAL(dd,ii,mbr,val) MPIDI_Trace_buf[(dd)].S[(ii)].mbr=val; -#define TRACE_SET_R_VALX(dd,rr,mbr,val) { \ - pami_task_t dd1; \ - if (dd < 0) \ - dd1=rr->mpid.partner_id; \ - else \ - dd1=dd; \ - MPIDI_Trace_buf[(dd1)].R[(rr->mpid.PR_idx)].mbr=val; \ -} -#define TRACE_SET_R_VAL(dd,ii,mbr,val) MPIDI_Trace_buf[(dd)].R[(ii)].mbr=val; -#define TRACE_SET_REQ_VAL(ww,val1) ww=val1; -#define TRACE_MEMSET_R(tt,nbr,str) (memset(&MPIDI_Trace_buf[tt].R[(nbr & SEQMASK)],0,sizeof(str))); -#define TRACE_MEMSET_S(tt,nbr,str) (memset(&MPIDI_Trace_buf[tt].S[(nbr & SEQMASK)],0,sizeof(str))); -#else -int recv_status; -int send_status; -int posted_recv; -#define MPIDI_SET_PR_REC(rreq,buf,ct,dt,pami_id,rank,tag,comm,is_blk) -#define MPIDI_GET_S_REC(dest,sreq,ctx,isSync,dataSize) -#define TRACE_SET_S_BIT(dd,ii,mbr) -#define TRACE_SET_R_BIT(dd,ii,mbr) -#define TRACE_SET_S_VAL(dd,ii,mbr,val) -#define TRACE_SET_R_VALX(dd,rr,mbr,val) -#define TRACE_SET_R_VAL(dd,ii,mbr,val) -#define TRACE_SET_REQ_VAL(ww,val1) -#define TRACE_MEMSET_R(tt,nbr,str) - -#endif /* MPIDI_TRACE */ -#endif /* include_mpidi_trace_h */ diff --git a/src/mpid/pamid/include/mpidi_util.h b/src/mpid/pamid/include/mpidi_util.h deleted file mode 100644 index 5a010c4..0000000 --- a/src/mpid/pamid/include/mpidi_util.h +++ /dev/null @@ -1,187 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file include/mpidi_util.h - * \brief ??? - */ -/* - * (C) 2001 by Argonne National Laboratory. - * See COPYRIGHT in top-level directory. - */ - - -#ifndef __include_mpidi_util_h__ -#define __include_mpidi_util_h__ - - -#if (MPIDI_STATISTICS || MPIDI_PRINTENV || MPIDI_BANNER) -#include <pami_ext_pe.h> -#include <sys/time.h> -#include <sys/param.h> - -#ifndef MAXHOSTNAMELEN -#ifdef LINUX -#define MAXHOSTNAMELEN 64 -#else -#define MAXHOSTNAMELEN 256 -#endif -#endif - -extern int MPIDI_atoi(char* , unsigned int* ); -extern int MPIDI_Banner(char *); -typedef enum {IS_IP, IS_US} MPIDI_TransportType; -typedef enum {NO_STRIPING=0, IS_PACKET_STRIPING} MPIDI_StripingType; - -typedef struct { - char version[128]; /* M19 */ - char hostname[MAXHOSTNAMELEN+1]; -/* Start job info */ - unsigned int partition; - int pid; - int this_task; - int nprocs; - int mp_statistics; - char nnodes[8]; - char tasks_node[8]; - char euilib[8]; - char window_id[128]; - char network_id[32]; - char adapter_type[32]; - char device[8]; - char protocol[16]; - char libpath[MAXPATHLEN]; /* size=BUFSIZE (pm_common.c) */ - char directory[FILENAME_MAX+1]; /* size = 256 */ - int mode_64; - int threaded; - int copySendBufSize:16; - char thread_scope[8]; - char cpu_use[10]; - char adapter_use[10]; - char clock_source[8]; - char priority[24]; - MPIDI_TransportType transport_type; - MPIDI_StripingType strip_type; - int use_rdma; - char *network_string; - int max_pkts_out; - long rdma_min_msg_size; - int timeout; - int interrupts; - uint polling_interval; - unsigned long buffer_mem; - long long buffer_mem_max; - int eager_limit; - int use_token_flow_control; - char wait_mode[8]; - int use_shmem; - uint retransmit_interval; - int shmem_pt2pt; - int shared_mem_pg_size; - char mem_affinity[8]; - int single_thread; - char checkpoint[8]; /* NA */ - char gang_scheduler[10]; /* NA */ - int instances; - char striping_type[40]; - int ack_thresh; - int recv_fifo_sz; - int max_pkt_size; - int rexmit_buf_size; - int rexmit_buf_cnt; - int max_atom_size; - char bulk_min_msg_size[16]; - char use_bulk_xfer[8]; - char user_rdma_avail[16]; - char user_rdma_total[16]; - char debug_notimeout[8]; - char develop[16]; - char stdinmode[12]; - char stdoutmode[12]; - int statistics; - int service_variables; - int rc_max_qp; /* M32 */ - char rc_qp_use_lmc[8]; /* M32 */ - char rc_qp_use_lru[8]; /* M32 */ - char rc_qp_init_setup[8]; /* M32 */ -} MPIDI_printenv_t; - - - -typedef struct { - long sends; /* Count of sends initiated */ - long sendsComplete; /* Count of sends completed (msg sent) */ - long sendWaitsComplete; /* Count of send waits completed - (blocking & nonblocking) */ - long recvs; /* Count of recvs initiated */ - long recvWaitsComplete; /* Count of recv waits complete */ - long earlyArrivals; /* Count of msgs received for which - no receive was posted */ - long earlyArrivalsMatched; /* Count of early arrivals for which - a posted receive has been found */ - long lateArrivals; /* Count of msgs received for which a recv - was posted */ - long unorderedMsgs; /* Total number of out of order msgs */ - long buffer_mem_hwmark; - long pamid_reserve_10; - long pamid_reserve_9; - long pamid_reserve_8; - long pamid_reserve_7; - long pamid_reserve_6; - long pamid_reserve_5; - long pamid_reserve_4; - long pamid_reserve_3; - long pamid_reserve_2; - long pamid_reserve_1; -} MPIX_stats_t; - -extern MPIDI_printenv_t *mpich_env; -extern MPIX_stats_t *mpid_statp; -extern int prtStat; -extern int prtEnv; -extern void set_mpich_env(int *,int*); -extern void MPIDI_open_pe_extension(); -extern void MPIDI_close_pe_extension(); -extern MPIDI_Statistics_write(FILE *); - -#if CUDA_AWARE_SUPPORT -int (*pamidCudaMemcpy)( void* dst, const void* src, size_t count, int kind ); -int (*pamidCudaPointerGetAttributes)( struct cudaPointerAttributes* attributes, const void* ptr ); -const char* (*pamidCudaGetErrorString)( int error ); -extern void * pamidCudaPtr; -#endif -/************************************************************* - * MPIDI_STATISTICS - *************************************************************/ -/* It is not necessary to do fetch_and_and. Statistics */ -/* should be taken when holding a lock. */ -#ifdef MPIDI_STATISTICS - #ifdef _AIX_ - #ifndef __64BIT__ - #define MPID_NSTAT(cmd) if (prtStat) fetch_and_add(&(cmd),1); - #else - #define MPID_NSTAT(cmd) if (prtStat) fetch_and_addlp(&(cmd),1); - #endif - #else /* Linux */ - #define MPID_NSTAT(cmd) if (prtStat) (cmd)++; - #endif /* End Linux */ -#else /* MPIDI_STATISTICS not set*/ - #define MPID_NSTAT(cmd) -#endif /* MPIDI_STATISTICS */ -#endif /* MPIDI_PRINTENV || MPIDI_STATISTICS */ - -#endif /* __include_mpidi_util_h__ */ diff --git a/src/mpid/pamid/include/mpidimpl.h b/src/mpid/pamid/include/mpidimpl.h deleted file mode 100644 index 553b644..0000000 --- a/src/mpid/pamid/include/mpidimpl.h +++ /dev/null @@ -1,197 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file include/mpidimpl.h - * \brief API MPID additions to MPI functions and structures - */ - -#ifndef __include_mpidimpl_h__ -#define __include_mpidimpl_h__ - -#include <mpiimpl.h> -#include "mpid_sched.h" -#include "pamix.h" -#include <mpix.h> - -extern int MPIDI_PAMID_Timer_is_ready; - -static inline void MPIDI_coll_check_in_place(void* src, void** dst) -{ - if(MPI_IN_PLACE == PAMI_IN_PLACE) - *dst = src; - else - { - if(src == PAMI_IN_PLACE) - *dst = MPI_IN_PLACE; - else - *dst = src; - } -} - -#ifdef DYNAMIC_TASKING - -#define MPIDI_MAX_KVS_VALUE_LEN 4096 - -typedef struct MPIDI_PG -{ - /* MPIU_Object field. MPIDI_PG_t objects are not allocated using the - MPIU_Object system, but we do use the associated reference counting - routines. Therefore, handle must be present, but is not used - except by debugging routines */ - MPIR_OBJECT_HEADER; /* adds handle and ref_count fields */ - - /* Next pointer used to maintain a list of all process groups known to - this process */ - struct MPIDI_PG * next; - - /* Number of processes in the process group */ - int size; - - /* VC table. At present this is a pointer to an array of VC structures. - Someday we may want make this a pointer to an array - of VC references. Thus, it is important to use MPIDI_PG_Get_vc() - instead of directly referencing this field. */ - MPID_VCR vct; - - /* Pointer to the process group ID. The actual ID is defined and - allocated by the process group. The pointer is kept in the - device space because it is necessary for the device to be able to - find a particular process group. */ - void * id; - - /* Replacement abstraction for connection information */ - /* Connection information needed to access processes in this process - group and to share the data with other processes. The items are - connData - pointer for data used to implement these functions - (e.g., a pointer to an array of process group info) - getConnInfo( rank, buf, bufsize, self ) - function to store into - buf the connection information for rank in this process - group - connInfoToString( buf_p, size, self ) - return in buf_p a string - that can be sent to another process to recreate the - connection information (the info needed to support - getConnInfo) - connInfoFromString( buf, self ) - setup the information needed - to implement getConnInfo - freeConnInfo( self ) - free any storage or resources associated - with the connection information. - - See ch3/src/mpidi_pg.c - */ - void *connData; - int (*getConnInfo)( int, char *, int, struct MPIDI_PG * ); - int (*connInfoToString)( char **, int *, struct MPIDI_PG * ); - int (*connInfoFromString)( const char *, struct MPIDI_PG * ); - int (*freeConnInfo)( struct MPIDI_PG * ); -} -MPIDI_PG_t; - -typedef int (*MPIDI_PG_Compare_ids_fn_t)(void * id1, void * id2); -typedef int (*MPIDI_PG_Destroy_fn_t)(MPIDI_PG_t * pg); - - -typedef MPIDI_PG_t * MPIDI_PG_iterator; - -typedef struct conn_info { - int rem_world_id; - int ref_count; - int *rem_taskids; /* The last member of this array is -1 */ - struct conn_info *next; -}conn_info; - -/* link list of transaciton id for all active remote connections in my world */ -typedef struct transactionID_struct { - long long tranid; - int *cntr_for_AM; /* Array size = TOTAL_AM */ - struct transactionID_struct *next; -}transactionID_struct; - -/*-------------------------- - BEGIN MPI PORT SECTION - --------------------------*/ -/* These are the default functions */ -int MPIDI_Comm_connect(const char *, struct MPIR_Info *, int, struct MPIR_Comm *, struct MPIR_Comm **); -int MPIDI_Comm_accept(const char *, struct MPIR_Info *, int, struct MPIR_Comm *, struct MPIR_Comm **); - -int MPIDI_Comm_spawn_multiple(int, char **, char ***, int *, struct MPIR_Info **, - int, struct MPIR_Comm *, struct MPIR_Comm **, int *); - - -typedef struct MPIDI_Port_Ops { - int (*OpenPort)( struct MPIR_Info *, char *); - int (*ClosePort)( const char * ); - int (*CommAccept)( const char *, struct MPIR_Info *, int, struct MPIR_Comm *, - struct MPIR_Comm ** ); - int (*CommConnect)( const char *, struct MPIR_Info *, int, struct MPIR_Comm *, - struct MPIR_Comm ** ); -} MPIDI_PortFns; - - -#define MPIDI_VC_add_ref( _vc ) \ - do { MPIR_Object_add_ref( _vc ); } while (0) - -#define MPIDI_PG_add_ref(pg_) \ -do { \ - MPIR_Object_add_ref(pg_); \ -} while (0) -#define MPIDI_PG_release_ref(pg_, inuse_) \ -do { \ - MPIR_Object_release_ref(pg_, inuse_); \ -} while (0) - -#define MPIDI_VC_release_ref( _vc, _inuse ) \ - do { MPIR_Object_release_ref( _vc, _inuse ); } while (0) - - -/* Initialize a new VC */ -int MPIDI_PG_Create_from_string(const char * str, MPIDI_PG_t ** pg_pptr, - int *flag); -int MPIDI_PG_Get_size(MPIDI_PG_t * pg); -#define MPIDI_PG_Get_size(pg_) ((pg_)->size) -#endif /** DYNAMIC_TASKING **/ - - -static inline pami_endpoint_t MPIDI_Task_to_endpoint(pami_task_t task, size_t offset) -{ - pami_endpoint_t ep; - pami_result_t rc; - rc = PAMI_Endpoint_create(MPIDI_Client, task, 0, &ep); -#if ASSERT_LEVEL > 0 - if(rc != PAMI_SUCCESS) - MPID_Abort (NULL, 0, 1, "MPIDI_Task_to_endpoint: Invalid task/offset. No endpoint found"); -#endif - return ep; -} - -int -MPIDI_Win_set_info(MPIR_Win *win, - MPIR_Info *info); - -MPI_Aint MPID_Aint_add(MPI_Aint base, MPI_Aint disp); -MPI_Aint MPID_Aint_diff(MPI_Aint addr1, MPI_Aint addr2); - -int MPIDI_Progress_register_hook(int (*progress_fn)(int*), int *id); -int MPIDI_Progress_deregister_hook(int id); -int MPIDI_Progress_activate_hook(int id); -int MPIDI_Progress_deactivate_hook(int id); - -#define MPID_Progress_register_hook(fn_, id_) MPIDI_Progress_register_hook(fn_, id_) -#define MPID_Progress_deregister_hook(id_) MPIDI_Progress_deregister_hook(id_) -#define MPID_Progress_activate_hook(id_) MPIDI_Progress_activate_hook(id_) -#define MPID_Progress_deactivate_hook(id_) MPIDI_Progress_deactivate_hook(id_) -#endif diff --git a/src/mpid/pamid/include/mpidpost.h b/src/mpid/pamid/include/mpidpost.h deleted file mode 100644 index a9ae835..0000000 --- a/src/mpid/pamid/include/mpidpost.h +++ /dev/null @@ -1,104 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file include/mpidpost.h - * \brief The trailing device header - * - * This file is included after the rest of the headers - * (mpidimpl.h, mpidpre.h, and mpiimpl.h) - */ - -#ifndef __include_mpidpost_h__ -#define __include_mpidpost_h__ - -#include <mpidu_datatype.h> -#include "mpidi_prototypes.h" -#include "mpidi_macros.h" - -#include "../src/mpid_progress.h" -#include "../src/mpid_request.h" -#include "../src/mpid_recvq.h" -#include "../src/pt2pt/mpid_isend.h" -#include "../src/pt2pt/mpid_send.h" -#include "../src/pt2pt/mpid_irecv.h" - -#ifdef DYNAMIC_TASKING -#define MPID_ICCREATE_REMOTECOMM_HOOK(_p,_c,_np,_gp,_r) \ - MPID_PG_ForwardPGInfo(_p,_c,_np,_gp,_r) -#else /* ! DYNAMIC_TASKING */ -/* If DYNAMIC_TASKING is not defined, PAMID does not provide its own GPID routines, - so provide one here. **/ - -/* FIXME: A temporary version for lpids within my comm world */ -static inline int MPID_GPID_GetAllInComm( MPIR_Comm *comm_ptr, int local_size, - int local_gpids[], int *singlePG ) -{ - int i; - int *gpid = local_gpids; - - for (i=0; i<comm_ptr->local_size; i++) { - *gpid++ = 0; - (void)MPID_VCR_Get_lpid( comm_ptr->vcr[i], gpid ); - gpid++; - } - *singlePG = 1; - return 0; -} - -/* FIXME: A temp for lpids within my comm world */ -static inline int MPID_GPID_ToLpidArray( int size, int gpid[], int lpid[] ) -{ - int i; - - for (i=0; i<size; i++) { - lpid[i] = *++gpid; gpid++; - } - return 0; -} -/* FIXME: for MPI1, all process ids are relative to MPI_COMM_WORLD. - For MPI2, we'll need to do something more complex */ -static inline int MPID_VCR_CommFromLpids( MPIR_Comm *newcomm_ptr, - int size, const int lpids[] ) -{ - MPIR_Comm *commworld_ptr; - int i; - - commworld_ptr = MPIR_Process.comm_world; - /* Setup the communicator's vc table: remote group */ - MPID_VCRT_Create( size, &newcomm_ptr->vcrt ); - MPID_VCRT_Get_ptr( newcomm_ptr->vcrt, &newcomm_ptr->vcr ); - for (i=0; i<size; i++) { - /* For rank i in the new communicator, find the corresponding - rank in the comm world (FIXME FOR MPI2) */ - /* printf( "[%d] Remote rank %d has lpid %d\n", - MPIR_Process.comm_world->rank, i, lpids[i] ); */ - if (lpids[i] < commworld_ptr->remote_size) { - MPID_VCR_Dup( commworld_ptr->vcr[lpids[i]], - &newcomm_ptr->vcr[i] ); - } - else { - /* We must find the corresponding vcr for a given lpid */ - /* FIXME: Error */ - return 1; - /* MPID_VCR_Dup( ???, &newcomm_ptr->vcr[i] ); */ - } - } - return 0; -} -#endif /* DYNAMIC_TASKING */ -#endif diff --git a/src/mpid/pamid/include/mpidpre.h b/src/mpid/pamid/include/mpidpre.h deleted file mode 100644 index 6df54a6..0000000 --- a/src/mpid/pamid/include/mpidpre.h +++ /dev/null @@ -1,94 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file include/mpidpre.h - * \brief The leading device header - * - * This file is included at the start of the other headers - * (mpidimpl.h, mpidpost.h, and mpiimpl.h). It generally contains - * additions to MPI objects. - */ - -#ifndef __include_mpidpre_h__ -#define __include_mpidpre_h__ - -#include <errno.h> -#include <stdio.h> -#include <assert.h> -#include <pthread.h> -#include <unistd.h> - -#include "mpiimpl.h" -#include "mpid_datatype_fallback.h" -#include <pami.h> - -/* provides "pre" typedefs and such for NBC scheduling mechanism */ -#include "mpidu_pre.h" - -/** \brief Creates a compile error if the condition is false. */ -#define MPID_assert_static(expr) ({ switch(0){case 0:case expr:;} }) -#define MPID_assert_always(x) assert(x) /**< \brief Tests for likely problems--always active */ -#define MPID_abort() assert(0) /**< \brief Always exit--usually implies missing functionality */ -#if ASSERT_LEVEL==0 -#define MPID_assert(x) -#else -#define MPID_assert(x) assert(x) /**< \brief Tests for likely problems--may not be active in performance code */ -#endif - -/* The timer code is allowed to return "NOT_INITIALIZED" before the - * device is initialized. Once the device is initialized, it must - * always return SUCCESS, so the upper layers do not need to check for - * the return code. */ -#define MPID_TIMER_SUCCESS MPL_TIMER_SUCCESS -#define MPID_TIMER_ERR_NOT_INITIALIZED MPL_TIMER_ERR_NOT_INITIALIZED - -typedef double MPID_Time_t; - -int MPID_Wtime(MPID_Time_t *tval); -int MPID_Wtick(double *wtick); -int MPID_Wtime_diff(MPID_Time_t *t1, MPID_Time_t *t2, double *diff); -int MPID_Wtime_todouble(MPID_Time_t *t, double *val); -int MPID_Wtime_acc(MPID_Time_t *t1, MPID_Time_t *t2, MPID_Time_t *t3); -int MPID_Wtime_init(void); - -#include "mpidi_platform.h" - -#include "mpidi_constants.h" -#include "mpidi_datatypes.h" -#include "mpidi_externs.h" -#include "mpidi_hooks.h" -#include "mpidi_thread.h" -#include "mpidi_util.h" - -#ifdef __BGQ__ -#define MPID_HANDLE_NUM_INDICES 256 -#endif /* __BGQ__ */ - -#define MPID_MAX_SMP_BCAST_MSG_SIZE (16384) -#define MPID_MAX_SMP_REDUCE_MSG_SIZE (16384) -#define MPID_MAX_SMP_ALLREDUCE_MSG_SIZE (16384) -#ifdef MPID_DEV_DATATYPE_DECL -#error 'Conflicting definitions of MPID_DEV_DATATYPE_DECL' -#else -#define MPID_DEV_DATATYPE_DECL void *device_datatype; -#endif -#define MPIDU_Dev_datatype_commit_hook(ptr) MPIDI_PAMI_datatype_commit_hook(ptr) -#define MPIDU_Dev_datatype_destroy_hook(ptr) MPIDI_PAMI_datatype_destroy_hook(ptr) -#define MPIDU_Dev_datatype_dup_hook(ptr) MPIDI_PAMI_datatype_dup_hook(ptr) - -#endif diff --git a/src/mpid/pamid/include/mpix.h b/src/mpid/pamid/include/mpix.h deleted file mode 100644 index 6940ad9..0000000 --- a/src/mpid/pamid/include/mpix.h +++ /dev/null @@ -1,298 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file include/mpix.h - * \brief PAMI device extensions to the MPI Spec - * - * These functions generally use MPI functions and internal APIs to - * expose extra information relating to the specific system on which - * the job is running. This may allow certain hardware specific - * optimizations to be made. - */ - -#ifndef __include_mpix_h__ -#define __include_mpix_h__ - -#if defined(__cplusplus) -extern "C" { -#endif - - - /** - * \brief Print the current system stack - * - * The first frame (this function) is discarded to make the trace look nicer. - */ - void MPIX_Dump_stacks(); - - void MPIX_Progress_poke(); - - /** - * \brief Wait for network to quiesce - * - * \praram[in] timeout Maximum time, Seconds, to wait. 0 for internal default - * \retval MPI_SUCCESS Network appears to be quiesced - * \retval MPI_ERR_PENDING Network did not quiesce - * \retval MPI_ERR_OTHER Encounter error(s), network state unknown - */ - int MPIX_Progress_quiesce(double timeout); - -#define MPIX_TORUS_MAX_DIMS 5 /* This is the maximum physical size of the torus */ - typedef struct - { -/* These fields will be used on all platforms. */ - unsigned prank; /**< Physical rank of the node (irrespective of mapping) */ - unsigned psize; /**< Size of the partition (irrespective of mapping) */ - unsigned ppn; /**< Processes per node */ - unsigned coreID; /**< Process id; values monotonically increase from 0..63 */ - - unsigned clockMHz; /**< Frequency in MegaHertz */ - unsigned memSize; /**< Size of the core memory in MB */ - -/* These fields are only set on torus platforms (i.e. Blue Gene) */ - unsigned torus_dimension; /**< Actual dimension for the torus */ - unsigned Size[MPIX_TORUS_MAX_DIMS]; /**< Max coordinates on the torus */ - unsigned Coords[MPIX_TORUS_MAX_DIMS]; /**< This node's coordinates */ - unsigned isTorus[MPIX_TORUS_MAX_DIMS]; /**< Do we have wraparound links? */ - -/* These fields are only set on systems using Blue Gene IO psets. */ - unsigned rankInPset; - unsigned sizeOfPset; - unsigned idOfPset; - } MPIX_Hardware_t; - - /** - * \brief Determine the rank-in-COMM_WORLD of the process associated with rank-in-comm - * - * Fortran interface: - * MPIX_COMM_RANK2GLOBAL (INTEGER comm, INTEGER crank, INTEGER grank, INTEGER ierr) - * - * \param[in] comm The communicator associated with the input rank - * \param[in] crank The rank-in-comm - * \param[out] grank The rank-in-COMM_WORLD (AKA Global rank) - * \return MPI_SUCCESS on success, an error on failure detection. - */ - int MPIX_Comm_rank2global(MPI_Comm comm, int crank, int *grank); - - /** - * \brief Fill in an MPIX_Hardware_t structure - * \param[in] hw A pointer to an MPIX_Hardware_t structure to be filled in - */ - int MPIX_Hardware(MPIX_Hardware_t *hw); - - - /* These functions only exist on torus platforms (i.e. Blue Gene) */ - - /** - * \brief Determine the number of physical hardware dimensions - * \param[out] numdimensions The number of torus dimensions - * \note This does NOT include the core+thread ID, so if you plan on - * allocating an array based on this information, you'll need to - * add 1 to the value returned here - */ - int MPIX_Torus_ndims(int *numdim); - /** - * \brief Convert an MPI rank into physical coordinates plus core ID - * \param[in] rank The MPI Rank - * \param[out] coords An array of size hw.torus_dimensions+1. The last - * element of the returned array is the core+thread ID - */ - int MPIX_Rank2torus(int rank, int *coords); - /** - * \brief Convert a set of coordinates (physical+core/thread) to an MPI rank - * \param[in] coords An array of size hw.torus_dimensions+1. The last element - * should be the core+thread ID (0..63). - * \param[out] rank The MPI rank cooresponding to the coords array passed in - */ - int MPIX_Torus2rank(int *coords, int *rank); - - /** - * \brief Optimize/deoptimize a communicator by adding/stripping - * platform specific optimizations (i.e. class routes support - * for efficient bcast/reductions). - * \param[in] comm MPI communicator - * \param[in] optimize Optimize(1) or deoptimize(0) the communicator - */ - int MPIX_Comm_update(MPI_Comm comm, int optimize); - - /** - * \brief Return the most recently used collective protocol name - * param[in] comm The communicator that collective was issued on - * param[out] protocol Storage space for the string name - * param[in] length Length available for the string name. - * Note: Max internal length is 100 - */ - int MPIX_Get_last_algorithm_name(MPI_Comm comm, char *protocol, int length); - - /** - * \brief Create a communicator such that all nodes in the same - * communicator are served by the same I/O node - * - * Fortran interface: - * MPIX_PSET_SAME_COMM_CREATE(INTEGER pset_comm, INTEGER ierr) - * - * \note This is a collective operation on MPI_COMM_WORLD - * - * \param [out] pset_comm The new communicator - * - * \return MPI status code - */ - int MPIX_Pset_same_comm_create (MPI_Comm *pset_comm); - - /** - * \brief Create a communicator such that all nodes in the same - * communicator are served by a different I/O node - * - * Fortran interface: - * MPIX_PSET_DIFF_COMM_CREATE(INTEGER pset_comm, INTEGER ierr) - * - * \note This is a collective operation on MPI_COMM_WORLD - * - * \param [out] pset_comm The new communicator - * - * \return MPI status code - */ - int MPIX_Pset_diff_comm_create (MPI_Comm *pset_comm); - - /** - * \brief Create a communicator such that all nodes in the same - * communicator are served by the same I/O node - * - * Fortran interface: - * MPIX_PSET_SAME_COMM_CREATE (INTEGER parent_comm, INTEGER pset_comm, INTEGER ierr) - * - * \note This is a collective operation on the parent communicator. - * - * \param [in] parent_comm The parent communicator - * \param [out] pset_comm The new communicator - * - * \return MPI status code - */ - int MPIX_Pset_same_comm_create_from_parent (MPI_Comm parent_comm, MPI_Comm *pset_comm); - - /** - * \brief Create a communicator such that all nodes in the same - * communicator are served by a different I/O node - * - * Fortran interface: - * MPIX_PSET_DIFF_COMM_CREATE (INTEGER parent_comm, INTEGER pset_comm, INTEGER ierr) - * - * \note This is a collective operation on the parent communicator - * - * \param [in] parent_comm The parent communicator - * \param [out] pset_comm The new communicator - * - * \return MPI status code - */ - int MPIX_Pset_diff_comm_create_from_parent (MPI_Comm parent_comm, MPI_Comm *pset_comm); - - /** - * \brief Retrieve the identifier of the io node associated with the local compute node. - * - * The I/O node identifier is a unique number, yet it is not a monotonically - * increasing integer; such as a rank in a communicator. Multiple ranks, and - * multiple compute nodes, can be associated with the same I/O link. - * - * Fortran interface: - * MPIX_IO_NODE_ID (INTEGER io_node_id) - * - * \note On BG/Q two 'bridge' compute nodes are connected to each io node; - * each unique io node identifier will be associated with at most two - * different io link identifiers. - * - * \return I/O node identifier - */ - int MPIX_IO_node_id (); - - /** - * \brief Retrieve the identifier of the io link associated with the local compute node. - * - * The I/O link identifier is a unique number, yet it is not a monotonically - * increasing integer; such as a rank in a communicator. Multiple ranks, and - * multiple compute nodes, can be associated with the same I/O link. - * - * Fortran interface: - * MPIX_IO_LINK_ID (INTEGER io_link_id) - * - * \return I/O link identifier - */ - int MPIX_IO_link_id (); - - /** - * \brief Retrieve the number of hops to the io node associated with the local compute node. - * - * The distance to the I/O node is the number of hops on the torus from the - * local compute node to the associated I/O node. The 'torus' vs 'mesh' - * attribute of each dimention is considered when determining the distance. - * - * Fortran interface: - * MPIX_IO_DISTANCE (INTEGER io_distance) - * - * \note On BG/Q the 'bridge' compute nodes are those nodes that are closest - * to the I/O node and will have a distance of '1'. - * - * \return number of hops to the I/O node - */ - int MPIX_IO_distance (); - - /** - * \brief Retrieve information about the I/O node associated with the - * local compute node. - * - * \deprecated This function has been replaced with MPIX_IO_link_id() and - * MPIX_IO_distance() - * - * \see MPIX_IO_link_id - * \see MPIX_IO_distance - * \see MPIX_IO_node_id - * - * \param [out] io_node_route_id The unique I/O node route identifier - * \param [out] distance_to_io_node The number of hops to the I/O node - */ - void MPIX_Pset_io_node (int *io_node_route_id, int *distance_to_io_node); - - /** - * \brief Create a Cartesian communicator that exactly matches the partition - * - * This is a collective operation on MPI_COMM_WORLD, and will only run - * successfully on a full partition job (no -np) - * - * The communicator is created to match the size of each dimension, the - * physical coords on each node, and the torus/mesh link status. - * - * Because of MPICH dimension ordering, the associated arrays (i.e. coords, - * sizes, and periods) are in [a, b, c, d, e, t] order. Consequently, when - * using the default ABCDET mapping, the rank in cart_comm will match the rank - * in MPI_COMM_WORLD. However, when using a non-default mapping or a mapfile - * the ranks will be different. - * - * Fortran interface: - * MPIX_CART_COMM_CREATE (INTEGER cart_comm, INTEGER ierr) - * - * \param [out] cart_comm The new Cartesian communicator - * - * \return MPI_SUCCESS or MPI_ERR_TOPOLOGY - */ - int MPIX_Cart_comm_create (MPI_Comm *cart_comm); - - -#if defined(__cplusplus) -} -#endif - -#endif diff --git a/src/mpid/pamid/include/pamix.h b/src/mpid/pamid/include/pamix.h deleted file mode 100644 index 80ef77a..0000000 --- a/src/mpid/pamid/include/pamix.h +++ /dev/null @@ -1,147 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file include/pamix.h - * \brief Extensions to PAMI - */ - - -#ifndef __include_pamix_h__ -#define __include_pamix_h__ - -#include <pami.h> -#include <mpidi_platform.h> -#if defined(__cplusplus) -extern "C" { -#endif - -typedef pami_result_t (*node_info_fn)(pami_task_t task, - uint32_t *node_id, - uint32_t *offset, - uint32_t *max_nodes); -typedef struct -{ - pami_extension_t progress; - - struct - { - pami_extension_t extension; - pami_result_t status; - uint8_t * base; - uintptr_t stride; - uintptr_t bitmask; - node_info_fn node_info; - } is_local_task; - -#if defined(__BGQ__) - pami_extension_t torus; -#endif -} pamix_extension_info_t; - -extern pamix_extension_info_t PAMIX_Extensions; - -void -PAMIX_Initialize(pami_client_t client); - -void -PAMIX_Finalize(pami_client_t client); - -pami_configuration_t -PAMIX_Client_query(pami_client_t client, - pami_attribute_name_t name); - -void -PAMIX_Dispatch_set(pami_context_t context[], - size_t num_contexts, - size_t dispatch, - pami_dispatch_callback_function fn, - pami_dispatch_hint_t options, - size_t * immediate_max); - -pami_task_t -PAMIX_Endpoint_query(pami_endpoint_t endpoint); - - -typedef void (*pamix_progress_function) (pami_context_t context, void *cookie); -#define PAMIX_CLIENT_ASYNC_GUARANTEE 1016 -#define ASYNC_PROGRESS_ALL 0x1111 -typedef enum -{ - PAMIX_PROGRESS_ALL = 0, - PAMIX_PROGRESS_RECV_INTERRUPT = 1, - PAMIX_PROGRESS_TIMER = 2, - PAMIX_PROGRESS_EXT = 1000 -} pamix_progress_t; - -void -PAMIX_Progress_register(pami_context_t context, - pamix_progress_function progress_fn, - pamix_progress_function suspend_fn, - pamix_progress_function resume_fn, - void * cookie); -void -PAMIX_Progress_enable(pami_context_t context, - pamix_progress_t event_type); - -void -PAMIX_Progress_disable(pami_context_t context, - pamix_progress_t event_type); - -pami_result_t -PAMIX_is_local_task_get_node_info(pami_task_t task, - uint32_t *node_id, - uint32_t *offset, - uint32_t *max_nodes); - -#ifdef __BGQ__ - -typedef struct -{ - size_t dims; - size_t *coord; - size_t *size; - size_t *torus; -} pamix_torus_info_t; - -const pamix_torus_info_t * PAMIX_Torus_info(); -int PAMIX_Task2torus(pami_task_t task_id, size_t coords[]); -int PAMIX_Torus2task(size_t coords[], pami_task_t* task_id); - -#endif - -#ifdef PAMIX_IS_LOCAL_TASK -#if defined(PAMIX_IS_LOCAL_TASK_STRIDE) && defined(PAMIX_IS_LOCAL_TASK_SHIFT) -#define PAMIX_Task_is_local(task_id) \ - (((1UL << PAMIX_IS_LOCAL_TASK_SHIFT) & \ - *(PAMIX_Extensions.is_local_task.base + \ - task_id * PAMIX_IS_LOCAL_TASK_STRIDE)) >> PAMIX_IS_LOCAL_TASK_SHIFT) -#else -#define PAMIX_Task_is_local(task_id) \ - ((PAMIX_Extensions.is_local_task.base && \ - (PAMIX_Extensions.is_local_task.bitmask & \ - *(PAMIX_Extensions.is_local_task.base + \ - task_id * PAMIX_Extensions.is_local_task.stride))) > 0) -#endif /* PAMIX_IS_LOCAL_TASK_STRIDE && PAMIX_IS_LOCAL_TASK_SHIFT */ -#else -#define PAMIX_Task_is_local(task_id) (0) -#endif /* PAMIX_IS_LOCAL_TASK */ - -#if defined(__cplusplus) -} -#endif -#endif diff --git a/src/mpid/pamid/src/Makefile.mk b/src/mpid/pamid/src/Makefile.mk deleted file mode 100644 index e2468f3..0000000 --- a/src/mpid/pamid/src/Makefile.mk +++ /dev/null @@ -1,73 +0,0 @@ -# begin_generated_IBM_copyright_prolog -# -# This is an automatically generated copyright prolog. -# After initializing, DO NOT MODIFY OR MOVE -# --------------------------------------------------------------- -# Licensed Materials - Property of IBM -# Blue Gene/Q 5765-PER 5765-PRP -# -# (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved -# US Government Users Restricted Rights - -# Use, duplication, or disclosure restricted -# by GSA ADP Schedule Contract with IBM Corp. -# -# --------------------------------------------------------------- -# -# end_generated_IBM_copyright_prolog -# -*- mode: makefile-gmake; -*- - -# note that the includes always happen but the effects of their contents are -# affected by "if BUILD_PAMID" -if BUILD_PAMID - -#AM_CPPFLAGS += -I$(top_srcdir)/src/mpid/pamid/src/include \ -# -I$(top_builddir)/src/mpid/pamid/src/include - -noinst_HEADERS += \ - src/mpid/pamid/src/mpid_request.h \ - src/mpid/pamid/src/mpid_progress.h \ - src/mpid/pamid/src/mpid_recvq.h - - -include $(top_srcdir)/src/mpid/pamid/src/coll/Makefile.mk -include $(top_srcdir)/src/mpid/pamid/src/comm/Makefile.mk -include $(top_srcdir)/src/mpid/pamid/src/misc/Makefile.mk -include $(top_srcdir)/src/mpid/pamid/src/mpix/Makefile.mk -include $(top_srcdir)/src/mpid/pamid/src/onesided/Makefile.mk -include $(top_srcdir)/src/mpid/pamid/src/pamix/Makefile.mk -include $(top_srcdir)/src/mpid/pamid/src/pt2pt/Makefile.mk -include $(top_srcdir)/src/mpid/pamid/src/dyntask/Makefile.mk - - -mpi_core_sources += \ - src/mpid/pamid/src/mpid_buffer.c \ - src/mpid/pamid/src/mpidi_bufmm.c \ - src/mpid/pamid/src/mpid_finalize.c \ - src/mpid/pamid/src/mpid_init.c \ - src/mpid/pamid/src/mpid_iprobe.c \ - src/mpid/pamid/src/mpid_probe.c \ - src/mpid/pamid/src/mpid_progress.c \ - src/mpid/pamid/src/mpid_recvq.c \ - src/mpid/pamid/src/mpid_request.c \ - src/mpid/pamid/src/mpid_time.c \ - src/mpid/pamid/src/mpid_vc.c \ - src/mpid/pamid/src/mpidi_env.c \ - src/mpid/pamid/src/mpidi_util.c \ - src/mpid/pamid/src/mpidi_mutex.c \ - src/mpid/pamid/src/mpid_mrecv.c \ - src/mpid/pamid/src/mpid_mprobe.c \ - src/mpid/pamid/src/mpid_imrecv.c \ - src/mpid/pamid/src/mpid_improbe.c \ - src/mpid/pamid/src/mpid_aint.c \ - src/mpid/pamid/src/mpidi_nbc_sched.c \ - src/mpid/pamid/src/mpidi_pami_datatype.c - -if QUEUE_BINARY_SEARCH_SUPPORT -mpi_core_sources += \ - src/mpid/pamid/src/mpid_recvq_mmap.cpp -endif QUEUE_BINARY_SEARCH_SUPPORT - - - -endif BUILD_PAMID - diff --git a/src/mpid/pamid/src/coll/Makefile.mk b/src/mpid/pamid/src/coll/Makefile.mk deleted file mode 100644 index adf7131..0000000 --- a/src/mpid/pamid/src/coll/Makefile.mk +++ /dev/null @@ -1,47 +0,0 @@ -# begin_generated_IBM_copyright_prolog -# -# This is an automatically generated copyright prolog. -# After initializing, DO NOT MODIFY OR MOVE -# --------------------------------------------------------------- -# Licensed Materials - Property of IBM -# Blue Gene/Q 5765-PER 5765-PRP -# -# (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved -# US Government Users Restricted Rights - -# Use, duplication, or disclosure restricted -# by GSA ADP Schedule Contract with IBM Corp. -# -# --------------------------------------------------------------- -# -# end_generated_IBM_copyright_prolog -# -*- mode: makefile-gmake; -*- - -# note that the includes always happen but the effects of their contents are -# affected by "if BUILD_PAMID" -if BUILD_PAMID - -include $(top_srcdir)/src/mpid/pamid/src/coll/barrier/Makefile.mk -include $(top_srcdir)/src/mpid/pamid/src/coll/bcast/Makefile.mk -include $(top_srcdir)/src/mpid/pamid/src/coll/allreduce/Makefile.mk -include $(top_srcdir)/src/mpid/pamid/src/coll/allgather/Makefile.mk -include $(top_srcdir)/src/mpid/pamid/src/coll/allgatherv/Makefile.mk -include $(top_srcdir)/src/mpid/pamid/src/coll/scatterv/Makefile.mk -include $(top_srcdir)/src/mpid/pamid/src/coll/scatter/Makefile.mk -include $(top_srcdir)/src/mpid/pamid/src/coll/gather/Makefile.mk -include $(top_srcdir)/src/mpid/pamid/src/coll/alltoall/Makefile.mk -include $(top_srcdir)/src/mpid/pamid/src/coll/alltoallv/Makefile.mk -include $(top_srcdir)/src/mpid/pamid/src/coll/gatherv/Makefile.mk -include $(top_srcdir)/src/mpid/pamid/src/coll/scan/Makefile.mk -include $(top_srcdir)/src/mpid/pamid/src/coll/reduce/Makefile.mk -include $(top_srcdir)/src/mpid/pamid/src/coll/alltoallw/Makefile.mk -include $(top_srcdir)/src/mpid/pamid/src/coll/exscan/Makefile.mk -include $(top_srcdir)/src/mpid/pamid/src/coll/ired_scat_block/Makefile.mk -include $(top_srcdir)/src/mpid/pamid/src/coll/ired_scat/Makefile.mk -include $(top_srcdir)/src/mpid/pamid/src/coll/red_scat/Makefile.mk - -mpi_core_sources += \ - src/mpid/pamid/src/coll/coll_utils.c - - -endif BUILD_PAMID - diff --git a/src/mpid/pamid/src/coll/allgather/Makefile.mk b/src/mpid/pamid/src/coll/allgather/Makefile.mk deleted file mode 100644 index db3c098..0000000 --- a/src/mpid/pamid/src/coll/allgather/Makefile.mk +++ /dev/null @@ -1,30 +0,0 @@ -# begin_generated_IBM_copyright_prolog -# -# This is an automatically generated copyright prolog. -# After initializing, DO NOT MODIFY OR MOVE -# --------------------------------------------------------------- -# Licensed Materials - Property of IBM -# Blue Gene/Q 5765-PER 5765-PRP -# -# (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved -# US Government Users Restricted Rights - -# Use, duplication, or disclosure restricted -# by GSA ADP Schedule Contract with IBM Corp. -# -# --------------------------------------------------------------- -# -# end_generated_IBM_copyright_prolog -# -*- mode: makefile-gmake; -*- - -# note that the includes always happen but the effects of their contents are -# affected by "if BUILD_PAMID" -if BUILD_PAMID - - -mpi_core_sources += \ - src/mpid/pamid/src/coll/allgather/mpido_allgather.c \ - src/mpid/pamid/src/coll/allgather/mpido_iallgather.c - - -endif BUILD_PAMID - diff --git a/src/mpid/pamid/src/coll/allgather/mpido_allgather.c b/src/mpid/pamid/src/coll/allgather/mpido_allgather.c deleted file mode 100644 index 38b46bd..0000000 --- a/src/mpid/pamid/src/coll/allgather/mpido_allgather.c +++ /dev/null @@ -1,820 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/coll/allgather/mpido_allgather.c - * \brief ??? - */ - -/* #define TRACE_ON */ -#include <mpidimpl.h> - - -static void allred_cb_done(void *ctxt, void *clientdata, pami_result_t err) -{ - int *active = (int *)clientdata; - (*active)--; -} - -static void allgather_cb_done(void *ctxt, void *clientdata, pami_result_t err) -{ - int *active = (int *)clientdata; - (*active)--; -} - - -/* ****************************************************************** */ -/** - * \brief Use (tree) MPIDO_Allreduce() to do a fast Allgather operation - * - * \note This function requires that: - * - The send/recv data types are contiguous - * - Tree allreduce is availible (for max performance) - * - The datatype parameters needed added to the function signature - */ -/* ****************************************************************** */ - -#define MAX_ALLGATHER_ALLREDUCE_BUFFER_SIZE (1024*1024*2) - -int MPIDO_Allgather_allreduce(const void *sendbuf, - int sendcount, - MPI_Datatype sendtype, - void *recvbuf, - int recvcount, - MPI_Datatype recvtype, - MPI_Aint send_true_lb, - MPI_Aint recv_true_lb, - size_t send_size, - size_t recv_size, - MPIR_Comm * comm_ptr, - int *mpierrno) - -{ - int rc, i; - char *startbuf = NULL; - char *destbuf = NULL; - const int rank = comm_ptr->rank; - - startbuf = (char *) recvbuf + recv_true_lb; - destbuf = startbuf + rank * send_size; - - if (sendbuf != MPI_IN_PLACE) - { - char *outputbuf = (char *) sendbuf + send_true_lb; - memcpy(destbuf, outputbuf, send_size); - } - - /* TODO: Change to PAMI */ - /*Do a convert and then do the allreudce*/ - if ( recv_size <= MAX_ALLGATHER_ALLREDUCE_BUFFER_SIZE && - (send_size & 0x3)==0 && /*integer/long allgathers only*/ - (sendtype != MPI_DOUBLE || recvtype != MPI_DOUBLE)) - { - double *tmprbuf = (double *)MPL_malloc(recv_size*2); - if (tmprbuf == NULL) - goto direct_algo; /*skip int to fp conversion and go to direct - algo*/ - - double *tmpsbuf = tmprbuf + (rank*send_size)/sizeof(int); - int *sibuf = (int *) destbuf; - - memset(tmprbuf, 0, rank*send_size*2); - memset(tmpsbuf + send_size/sizeof(int), 0, - (recv_size - (rank + 1)*send_size)*2); - - for(i = 0; i < (send_size/sizeof(int)); ++i) - tmpsbuf[i] = (double)sibuf[i]; - /* Switch to comm->coll_fns->fn() */ - rc = MPIDO_Allreduce(MPI_IN_PLACE, - tmprbuf, - recv_size/sizeof(int), - MPI_DOUBLE, - MPI_SUM, - comm_ptr, - mpierrno); - - sibuf = (int *) startbuf; - for(i = 0; i < (rank*send_size/sizeof(int)); ++i) - sibuf[i] = (int)tmprbuf[i]; - - for(i = (rank+1)*send_size/sizeof(int); i < recv_size/sizeof(int); ++i) - sibuf[i] = (int)tmprbuf[i]; - - MPL_free(tmprbuf); - return rc; - } - - direct_algo: - - memset(startbuf, 0, rank * send_size); - memset(destbuf + send_size, 0, recv_size - (rank + 1) * send_size); - - if (sendtype == MPI_DOUBLE && recvtype == MPI_DOUBLE) - /* Switch to comm->coll_fns->fn() */ - rc = MPIDO_Allreduce(MPI_IN_PLACE, - startbuf, - recv_size/sizeof(double), - MPI_DOUBLE, - MPI_SUM, - comm_ptr, - mpierrno); - else - /* Switch to comm->coll_fns->fn() */ - rc = MPIDO_Allreduce(MPI_IN_PLACE, - startbuf, - recv_size/sizeof(int), - MPI_UNSIGNED, - MPI_BOR, - comm_ptr, - mpierrno); - - return rc; -} - - -/* ****************************************************************** */ -/** - * \brief Use (tree/rect) MPIDO_Bcast() to do a fast Allgather operation - * - * \note This function requires one of these (for max performance): - * - Tree broadcast - */ -/* ****************************************************************** */ -int MPIDO_Allgather_bcast(const void *sendbuf, - int sendcount, - MPI_Datatype sendtype, - void *recvbuf, - int recvcount, - MPI_Datatype recvtype, - MPI_Aint send_true_lb, - MPI_Aint recv_true_lb, - size_t send_size, - size_t recv_size, - MPIR_Comm * comm_ptr, - int *mpierrno) -{ - int i, np, rc = 0; - MPI_Aint extent; - const int rank = comm_ptr->rank; - - np = comm_ptr ->local_size; - MPIDU_Datatype_get_extent_macro(recvtype, extent); - - MPIR_Ensure_Aint_fits_in_pointer ((MPIR_VOID_PTR_CAST_TO_MPI_AINT recvbuf + - np * recvcount * extent)); - if (sendbuf != MPI_IN_PLACE) - { - void *destbuf = recvbuf + rank * recvcount * extent; - MPIR_Localcopy(sendbuf, - sendcount, - sendtype, - destbuf, - recvcount, - recvtype); - } - -/* this code should either abort on first error or somehow aggregate - * error codes, esp since it calls internal routines */ - for (i = 0; i < np; i++) - { - void *destbuf = recvbuf + i * recvcount * extent; - /* Switch to comm->coll_fns->fn() */ - rc = MPIDO_Bcast(destbuf, - recvcount, - recvtype, - i, - comm_ptr, - mpierrno); - } - - return rc; -} - -/* ****************************************************************** */ -/** - * \brief Use (tree/rect) MPIDO_Alltoall() to do a fast Allgather operation - * - * \note This function requires that: - * - The send/recv data types are contiguous - * - DMA alltoallv is availible (for max performance) - * - The datatype parameters needed added to the function signature - */ -/* ****************************************************************** */ -int MPIDO_Allgather_alltoall(const void *sendbuf, - int sendcount, - MPI_Datatype sendtype, - void *recvbuf, - int recvcount, - MPI_Datatype recvtype, - MPI_Aint send_true_lb, - MPI_Aint recv_true_lb, - size_t send_size, - size_t recv_size, - MPIR_Comm * comm_ptr, - int *mpierrno) -{ - int i, rc; - void *a2a_sendbuf = NULL; - char *destbuf=NULL; - char *startbuf=NULL; - const int size = comm_ptr->local_size; - const int rank = comm_ptr->rank; - - int a2a_sendcounts[size]; - int a2a_senddispls[size]; - int a2a_recvcounts[size]; - int a2a_recvdispls[size]; - - for (i = 0; i < size; ++i) - { - a2a_sendcounts[i] = send_size; - a2a_senddispls[i] = 0; - a2a_recvcounts[i] = recvcount; - a2a_recvdispls[i] = recvcount * i; - } - if (sendbuf != MPI_IN_PLACE) - { - a2a_sendbuf = (char *)sendbuf + send_true_lb; - } - else - { - startbuf = (char *) recvbuf + recv_true_lb; - destbuf = startbuf + rank * send_size; - a2a_sendbuf = destbuf; - a2a_sendcounts[rank] = 0; - - a2a_recvcounts[rank] = 0; - } - - - /* Switch to comm->coll_fns->fn() */ - rc = MPIDO_Alltoallv((const void *)a2a_sendbuf, - a2a_sendcounts, - a2a_senddispls, - MPI_CHAR, - recvbuf, - a2a_recvcounts, - a2a_recvdispls, - recvtype, - comm_ptr, - mpierrno); - - return rc; -} - - - - -int -MPIDO_Allgather(const void *sendbuf, - int sendcount, - MPI_Datatype sendtype, - void *recvbuf, - int recvcount, - MPI_Datatype recvtype, - MPIR_Comm * comm_ptr, - int *mpierrno) -{ -#ifndef HAVE_PAMI_IN_PLACE - if (sendbuf == MPI_IN_PLACE) - { - MPID_Abort (NULL, 0, 1, "'MPI_IN_PLACE' requries support for `PAMI_IN_PLACE`"); - return -1; - } -#endif - /* ********************************* - * Check the nature of the buffers - * ********************************* - */ - const struct MPIDI_Comm* const mpid = &(comm_ptr->mpid); - int config[6], i; - MPIDU_Datatype* dt_null = NULL; - MPI_Aint send_true_lb = 0; - MPI_Aint recv_true_lb = 0; - int comm_size = comm_ptr->local_size; - size_t send_bytes = 0; - size_t recv_bytes = 0; - volatile unsigned allred_active = 1; - volatile unsigned allgather_active = 1; - pami_xfer_t allred; - const int rank = comm_ptr->rank; - int queryreq = 0; -#if ASSERT_LEVEL==0 - /* We can't afford the tracing in ndebug/performance libraries */ - const unsigned verbose = 0; -#else - const unsigned verbose = (MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_ALL) && (rank == 0); -#endif - const int selected_type = mpid->user_selected_type[PAMI_XFER_ALLGATHER]; - - for (i=0;i<6;i++) config[i] = 1; - const pami_metadata_t *my_md = (pami_metadata_t *)NULL; - - - allred.cb_done = allred_cb_done; - allred.cookie = (void *)&allred_active; - /* Pick an algorithm that is guaranteed to work for the pre-allreduce */ - /* TODO: This needs selection for fast(er|est) allreduce protocol */ - allred.algorithm = mpid->coll_algorithm[PAMI_XFER_ALLREDUCE][0][0]; - allred.cmd.xfer_allreduce.sndbuf = (void *)config; - allred.cmd.xfer_allreduce.stype = PAMI_TYPE_SIGNED_INT; - allred.cmd.xfer_allreduce.rcvbuf = (void *)config; - allred.cmd.xfer_allreduce.rtype = PAMI_TYPE_SIGNED_INT; - allred.cmd.xfer_allreduce.stypecount = 6; - allred.cmd.xfer_allreduce.rtypecount = 6; - allred.cmd.xfer_allreduce.op = PAMI_DATA_BAND; - - char use_tree_reduce, use_alltoall, use_bcast, use_pami, use_opt; - char *rbuf = NULL, *sbuf = NULL; - - const char * const allgathers = mpid->allgathers; - use_alltoall = allgathers[2]; - use_tree_reduce = allgathers[0]; - use_bcast = allgathers[1]; - use_pami = (selected_type == MPID_COLL_USE_MPICH) ? 0 : 1; - use_opt = use_alltoall || use_tree_reduce || use_bcast || use_pami; - - - TRACE_ERR("flags before: b: %d a: %d t: %d p: %d\n", use_bcast, use_alltoall, use_tree_reduce, use_pami); - if(!use_opt) - { - if(unlikely(verbose)) - fprintf(stderr,"Using MPICH allgather algorithm\n"); - TRACE_ERR("No options set/available; using MPICH for allgather\n"); - MPIDI_Update_last_algorithm(comm_ptr, "ALLGATHER_MPICH"); -#if CUDA_AWARE_SUPPORT - if(MPIDI_Process.cuda_aware_support_on) - { - MPI_Aint sdt_extent,rdt_extent; - MPIDU_Datatype_get_extent_macro(sendtype, sdt_extent); - MPIDU_Datatype_get_extent_macro(recvtype, rdt_extent); - char *scbuf = NULL; - char *rcbuf = NULL; - int is_send_dev_buf = MPIDI_cuda_is_device_buf(sendbuf); - int is_recv_dev_buf = MPIDI_cuda_is_device_buf(recvbuf); - if(is_send_dev_buf) - { - scbuf = MPL_malloc(sdt_extent * sendcount); - cudaError_t cudaerr = CudaMemcpy(scbuf, sendbuf, sdt_extent * sendcount, cudaMemcpyDeviceToHost); - if (cudaSuccess != cudaerr) - fprintf(stderr, "cudaMemcpy failed: %s\n", CudaGetErrorString(cudaerr)); - } - else - scbuf = sendbuf; - if(is_recv_dev_buf) - { - rcbuf = MPL_malloc(rdt_extent * recvcount); - if(sendbuf == MPI_IN_PLACE) - { - cudaError_t cudaerr = CudaMemcpy(rcbuf, recvbuf, rdt_extent * recvcount, cudaMemcpyDeviceToHost); - if (cudaSuccess != cudaerr) - fprintf(stderr, "cudaMemcpy failed: %s\n", CudaGetErrorString(cudaerr)); - } - else - memset(rcbuf, 0, rdt_extent * recvcount); - } - else - rcbuf = recvbuf; - int cuda_res = MPIR_Allgather(scbuf, sendcount, sendtype, rcbuf, recvcount, recvtype, comm_ptr, mpierrno); - if(is_send_dev_buf)MPL_free(scbuf); - if(is_recv_dev_buf) - { - cudaError_t cudaerr = CudaMemcpy(recvbuf, rcbuf, rdt_extent * recvcount, cudaMemcpyHostToDevice); - if (cudaSuccess != cudaerr) - fprintf(stderr, "cudaMemcpy failed: %s\n", CudaGetErrorString(cudaerr)); - MPL_free(rcbuf); - } - return cuda_res; - } - else -#endif - return MPIR_Allgather(sendbuf, sendcount, sendtype, - recvbuf, recvcount, recvtype, - comm_ptr, mpierrno); - } - if ((sendcount < 1 && sendbuf != MPI_IN_PLACE) || recvcount < 1) - return MPI_SUCCESS; - - /* Gather datatype information */ - MPIDI_Datatype_get_info(recvcount, - recvtype, - config[MPID_RECV_CONTIG], - recv_bytes, - dt_null, - recv_true_lb); - - send_bytes = recv_bytes; - rbuf = (char *)recvbuf+recv_true_lb; - - sbuf = PAMI_IN_PLACE; - if(sendbuf != MPI_IN_PLACE) - { - MPIDI_Datatype_get_info(sendcount, - sendtype, - config[MPID_SEND_CONTIG], - send_bytes, - dt_null, - send_true_lb); - sbuf = (char *)sendbuf+send_true_lb; - } - else - if(unlikely(verbose)) - fprintf(stderr,"allgather MPI_IN_PLACE buffering\n"); - - - /* verify everyone's datatype contiguity */ - /* Check buffer alignment now, since we're pre-allreducing anyway */ - /* Only do this if one of the glue protocols is likely to be used */ - if(use_alltoall || use_tree_reduce || use_bcast) - { - config[MPID_ALIGNEDBUFFER] = - !((long)sendbuf & 0x0F) && !((long)recvbuf & 0x0F); - - /* #warning need to determine best allreduce for short messages */ - if(mpid->preallreduces[MPID_ALLGATHER_PREALLREDUCE]) - { - TRACE_ERR("Preallreducing in allgather\n"); - MPIDI_Post_coll_t allred_post; - MPIDI_Context_post(MPIDI_Context[0], &allred_post.state, - MPIDI_Pami_post_wrapper, (void *)&allred); - - MPID_PROGRESS_WAIT_WHILE(allred_active); - } - - - use_alltoall = allgathers[2] && - config[MPID_RECV_CONTIG] && config[MPID_SEND_CONTIG];; - - /* Note: some of the glue protocols use recv_bytes*comm_size rather than - * recv_bytes so we use that for comparison here, plus we pass that in - * to those protocols. */ - use_tree_reduce = allgathers[0] && - config[MPID_RECV_CONTIG] && config[MPID_SEND_CONTIG] && - config[MPID_RECV_CONTINUOUS] && (recv_bytes*comm_size%sizeof(unsigned)) == 0; - - use_bcast = allgathers[1]; - - TRACE_ERR("flags after: b: %d a: %d t: %d p: %d\n", use_bcast, use_alltoall, use_tree_reduce, use_pami); - } - if(use_pami) - { - TRACE_ERR("Using PAMI-level allgather protocol\n"); - pami_xfer_t allgather; - allgather.cb_done = allgather_cb_done; - allgather.cookie = (void *)&allgather_active; - allgather.cmd.xfer_allgather.rcvbuf = rbuf; - allgather.cmd.xfer_allgather.sndbuf = sbuf; - allgather.cmd.xfer_allgather.stype = PAMI_TYPE_BYTE; - allgather.cmd.xfer_allgather.rtype = PAMI_TYPE_BYTE; - allgather.cmd.xfer_allgather.stypecount = send_bytes; - allgather.cmd.xfer_allgather.rtypecount = recv_bytes; - if(selected_type == MPID_COLL_OPTIMIZED) - { - if((mpid->cutoff_size[PAMI_XFER_ALLGATHER][0] == 0) || - (mpid->cutoff_size[PAMI_XFER_ALLGATHER][0] > 0 && mpid->cutoff_size[PAMI_XFER_ALLGATHER][0] >= send_bytes)) - { - allgather.algorithm = mpid->opt_protocol[PAMI_XFER_ALLGATHER][0]; - my_md = &mpid->opt_protocol_md[PAMI_XFER_ALLGATHER][0]; - queryreq = mpid->must_query[PAMI_XFER_ALLGATHER][0]; - } - else - { - return MPIR_Allgather(sendbuf, sendcount, sendtype, - recvbuf, recvcount, recvtype, - comm_ptr, mpierrno); - } - } - else - { - allgather.algorithm = mpid->user_selected[PAMI_XFER_ALLGATHER]; - my_md = &mpid->user_metadata[PAMI_XFER_ALLGATHER]; - queryreq = selected_type; - } - - if(unlikely( queryreq == MPID_COLL_ALWAYS_QUERY || - queryreq == MPID_COLL_CHECK_FN_REQUIRED)) - { - metadata_result_t result = {0}; - TRACE_ERR("Querying allgather protocol %s, type was: %d\n", - my_md->name, - selected_type); - if(my_md->check_fn == NULL) - { - /* process metadata bits */ - if((!my_md->check_correct.values.inplace) && (sendbuf == MPI_IN_PLACE)) - result.check.unspecified = 1; - if(my_md->check_correct.values.rangeminmax) - { - if((my_md->range_lo <= recv_bytes) && - (my_md->range_hi >= recv_bytes)) - ; /* ok, algorithm selected */ - else - { - result.check.range = 1; - if(unlikely(verbose)) - { - fprintf(stderr,"message size (%zu) outside range (%zu<->%zu) for %s.\n", - recv_bytes, - my_md->range_lo, - my_md->range_hi, - my_md->name); - } - } - } - } - else /* calling the check fn is sufficient */ - result = my_md->check_fn(&allgather); - TRACE_ERR("bitmask: %#X\n", result.bitmask); - result.check.nonlocal = 0; /* #warning REMOVE THIS WHEN IMPLEMENTED */ - if(result.bitmask) - { - if(unlikely(verbose)) - fprintf(stderr,"Query failed for %s. Using MPICH allgather\n", - my_md->name); - MPIDI_Update_last_algorithm(comm_ptr, "ALLGATHER_MPICH"); - return MPIR_Allgather(sendbuf, sendcount, sendtype, - recvbuf, recvcount, recvtype, - comm_ptr, mpierrno); - } - if(my_md->check_correct.values.asyncflowctl && !(--(comm_ptr->mpid.num_requests))) - { - comm_ptr->mpid.num_requests = MPIDI_Process.optimized.num_requests; - int tmpmpierrno; - if(unlikely(verbose)) - fprintf(stderr,"Query barrier required for %s\n", my_md->name); - MPIDO_Barrier(comm_ptr, &tmpmpierrno); - } - } - - if(unlikely(verbose)) - { - unsigned long long int threadID; - MPL_thread_id_t tid; - MPL_thread_self(&tid); - threadID = (unsigned long long int)tid; - fprintf(stderr,"<%llx> Using protocol %s for allgather on %u\n", - threadID, - my_md->name, - (unsigned) comm_ptr->context_id); - } - TRACE_ERR("Calling PAMI_Collective with allgather structure\n"); - MPIDI_Post_coll_t allgather_post; - MPIDI_Context_post(MPIDI_Context[0], &allgather_post.state, MPIDI_Pami_post_wrapper, (void *)&allgather); - - MPIDI_Update_last_algorithm(comm_ptr, my_md->name); - MPID_PROGRESS_WAIT_WHILE(allgather_active); - TRACE_ERR("Allgather done\n"); - return PAMI_SUCCESS; - } - - if(use_tree_reduce) - { - if(unlikely(verbose)) - fprintf(stderr,"Using protocol GLUE_ALLREDUCE for allgather\n"); - TRACE_ERR("Using allgather via allreduce\n"); - MPIDI_Update_last_algorithm(comm_ptr, "ALLGATHER_OPT_ALLREDUCE"); - return MPIDO_Allgather_allreduce(sendbuf, sendcount, sendtype, - recvbuf, recvcount, recvtype, - send_true_lb, recv_true_lb, send_bytes, recv_bytes*comm_size, comm_ptr, mpierrno); - } - if(use_alltoall) - { - if(unlikely(verbose)) - fprintf(stderr,"Using protocol GLUE_BCAST for allgather\n"); - TRACE_ERR("Using allgather via alltoall\n"); - MPIDI_Update_last_algorithm(comm_ptr, "ALLGATHER_OPT_ALLTOALL"); - return MPIDO_Allgather_alltoall(sendbuf, sendcount, sendtype, - recvbuf, recvcount, recvtype, - send_true_lb, recv_true_lb, send_bytes, recv_bytes*comm_size, comm_ptr, mpierrno); - } - - if(use_bcast) - { - if(unlikely(verbose)) - fprintf(stderr,"Using protocol GLUE_ALLTOALL for allgather\n"); - TRACE_ERR("Using allgather via bcast\n"); - MPIDI_Update_last_algorithm(comm_ptr, "ALLGATHER_OPT_BCAST"); - return MPIDO_Allgather_bcast(sendbuf, sendcount, sendtype, - recvbuf, recvcount, recvtype, - send_true_lb, recv_true_lb, send_bytes, recv_bytes*comm_size, comm_ptr, mpierrno); - } - - /* Nothing used yet; dump to MPICH */ - if(unlikely(verbose)) - fprintf(stderr,"Using MPICH allgather algorithm\n"); - TRACE_ERR("Using allgather via mpich\n"); - MPIDI_Update_last_algorithm(comm_ptr, "ALLGATHER_MPICH"); - return MPIR_Allgather(sendbuf, sendcount, sendtype, - recvbuf, recvcount, recvtype, - comm_ptr, mpierrno); -} - - -int -MPIDO_Allgather_simple(const void *sendbuf, - int sendcount, - MPI_Datatype sendtype, - void *recvbuf, - int recvcount, - MPI_Datatype recvtype, - MPIR_Comm * comm_ptr, - int *mpierrno) -{ -#ifndef HAVE_PAMI_IN_PLACE - if (sendbuf == MPI_IN_PLACE) - { - MPID_Abort (NULL, 0, 1, "'MPI_IN_PLACE' requries support for `PAMI_IN_PLACE`"); - return -1; - } -#endif - /* ********************************* - * Check the nature of the buffers - * ********************************* - */ - const struct MPIDI_Comm* const mpid = &(comm_ptr->mpid); - MPIDU_Datatype* dt_null = NULL; - void *snd_noncontig_buff = NULL, *rcv_noncontig_buff = NULL; - MPI_Aint send_true_lb = 0; - MPI_Aint recv_true_lb = 0; - int snd_data_contig = 1, rcv_data_contig = 1; - size_t send_size = 0; - size_t recv_size = 0; - MPIDU_Segment segment; - volatile unsigned allgather_active = 1; - const int rank = comm_ptr->rank; - const int size = comm_ptr->local_size; -#if ASSERT_LEVEL==0 - /* We can't afford the tracing in ndebug/performance libraries */ - const unsigned verbose = 0; -#else - const unsigned verbose = (MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_ALL) && (rank == 0); -#endif - - const pami_metadata_t *my_md; - - char *rbuf = NULL, *sbuf = NULL; - - - if ((sendcount < 1 && sendbuf != MPI_IN_PLACE) || recvcount < 1) - return MPI_SUCCESS; - - /* Gather datatype information */ - MPIDI_Datatype_get_info(recvcount, - recvtype, - rcv_data_contig, - recv_size, - dt_null, - recv_true_lb); - - send_size = recv_size; - - if(MPIDI_Pamix_collsel_advise != NULL && mpid->collsel_fast_query != NULL) - { - advisor_algorithm_t advisor_algorithms[1]; - int num_algorithms = MPIDI_Pamix_collsel_advise(mpid->collsel_fast_query, PAMI_XFER_ALLGATHER, send_size, advisor_algorithms, 1); - if(num_algorithms) - { - if(advisor_algorithms[0].algorithm_type == COLLSEL_EXTERNAL_ALGO) - { - return MPIR_Allgather(sendbuf, sendcount, sendtype, - recvbuf, recvcount, recvtype, - comm_ptr, mpierrno); - } - else if(advisor_algorithms[0].metadata && advisor_algorithms[0].metadata->check_correct.values.asyncflowctl && !(--(comm_ptr->mpid.num_requests))) - { - comm_ptr->mpid.num_requests = MPIDI_Process.optimized.num_requests; - int tmpmpierrno; - if(unlikely(verbose)) - fprintf(stderr,"Query barrier required for %s\n", advisor_algorithms[0].metadata->name); - MPIDO_Barrier(comm_ptr, &tmpmpierrno); - } - } - } - - rbuf = (char *)recvbuf+recv_true_lb; - - if(!rcv_data_contig) - { - rcv_noncontig_buff = MPL_malloc(recv_size * size); - rbuf = rcv_noncontig_buff; - if(rcv_noncontig_buff == NULL) - { - MPID_Abort(NULL, MPI_ERR_NO_SPACE, 1, - "Fatal: Cannot allocate pack buffer"); - } - if(sendbuf == MPI_IN_PLACE) - { - sbuf = PAMI_IN_PLACE; - size_t extent; - MPIDU_Datatype_get_extent_macro(recvtype,extent); - MPIR_Localcopy(recvbuf + (rank*recvcount*extent), recvcount, recvtype, - rcv_noncontig_buff + (rank*recv_size), recv_size,MPI_CHAR); - } - } - - if(sendbuf != MPI_IN_PLACE) - { - MPIDI_Datatype_get_info(sendcount, - sendtype, - snd_data_contig, - send_size, - dt_null, - send_true_lb); - - sbuf = (char *)sendbuf+send_true_lb; - - if(!snd_data_contig) - { - snd_noncontig_buff = MPL_malloc(send_size); - sbuf = snd_noncontig_buff; - if(snd_noncontig_buff == NULL) - { - MPID_Abort(NULL, MPI_ERR_NO_SPACE, 1, - "Fatal: Cannot allocate pack buffer"); - } - DLOOP_Offset last = send_size; - MPIDU_Segment_init(sendbuf, sendcount, sendtype, &segment, 0); - MPIDU_Segment_pack(&segment, 0, &last, snd_noncontig_buff); - } - } - else - sbuf = PAMI_IN_PLACE; - - TRACE_ERR("Using PAMI-level allgather protocol\n"); - pami_xfer_t allgather; - allgather.cb_done = allgather_cb_done; - allgather.cookie = (void *)&allgather_active; - allgather.cmd.xfer_allgather.rcvbuf = rbuf; - allgather.cmd.xfer_allgather.sndbuf = sbuf; - allgather.cmd.xfer_allgather.stype = PAMI_TYPE_BYTE;/* stype is ignored when sndbuf == PAMI_IN_PLACE */ - allgather.cmd.xfer_allgather.rtype = PAMI_TYPE_BYTE; - allgather.cmd.xfer_allgather.stypecount = send_size; - allgather.cmd.xfer_allgather.rtypecount = recv_size; - allgather.algorithm = mpid->coll_algorithm[PAMI_XFER_ALLGATHER][0][0]; - my_md = &mpid->coll_metadata[PAMI_XFER_ALLGATHER][0][0]; - - TRACE_ERR("Calling PAMI_Collective with allgather structure\n"); - MPIDI_Post_coll_t allgather_post; - MPIDI_Context_post(MPIDI_Context[0], &allgather_post.state, MPIDI_Pami_post_wrapper, (void *)&allgather); - TRACE_ERR("Allgather %s\n", MPIDI_Process.context_post.active>0?"posted":"invoked"); - - MPIDI_Update_last_algorithm(comm_ptr, my_md->name); - MPID_PROGRESS_WAIT_WHILE(allgather_active); - if(!rcv_data_contig) - { - MPIR_Localcopy(rcv_noncontig_buff, recv_size * size, MPI_CHAR, - recvbuf, recvcount, recvtype); - MPL_free(rcv_noncontig_buff); - } - if(!snd_data_contig) MPL_free(snd_noncontig_buff); - TRACE_ERR("Allgather done\n"); - return MPI_SUCCESS; -} - - -int -MPIDO_CSWrapper_allgather(pami_xfer_t *allgather, - void *comm) -{ - int mpierrno = 0; - MPIR_Comm *comm_ptr = (MPIR_Comm*)comm; - MPI_Datatype sendtype, recvtype; - void *sbuf; - MPIDI_coll_check_in_place(allgather->cmd.xfer_allgather.sndbuf, &sbuf); - int rc = MPIDI_Dtpami_to_dtmpi( allgather->cmd.xfer_allgather.stype, - &sendtype, - NULL, - NULL); - if(rc == -1) return rc; - - rc = MPIDI_Dtpami_to_dtmpi( allgather->cmd.xfer_allgather.rtype, - &recvtype, - NULL, - NULL); - if(rc == -1) return rc; - - rc = MPIR_Allgather(sbuf, - allgather->cmd.xfer_allgather.stypecount, sendtype, - allgather->cmd.xfer_allgather.rcvbuf, - allgather->cmd.xfer_allgather.rtypecount, recvtype, - comm_ptr, &mpierrno); - if(allgather->cb_done && rc == 0) - allgather->cb_done(NULL, allgather->cookie, PAMI_SUCCESS); - return rc; - -} - diff --git a/src/mpid/pamid/src/coll/allgather/mpido_iallgather.c b/src/mpid/pamid/src/coll/allgather/mpido_iallgather.c deleted file mode 100644 index 1f3f3ab..0000000 --- a/src/mpid/pamid/src/coll/allgather/mpido_iallgather.c +++ /dev/null @@ -1,73 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/coll/allgather/mpido_iallgather.c - * \brief ??? - */ - -/*#define TRACE_ON */ -#include <mpidimpl.h> - -int -MPIDO_Iallgather(const void *sendbuf, - int sendcount, - MPI_Datatype sendtype, - void *recvbuf, - int recvcount, - MPI_Datatype recvtype, - MPIR_Comm * comm_ptr, - MPIR_Request ** request) -{ - /*if (unlikely((data_size == 0) || (user_selected_type == MPID_COLL_USE_MPICH)))*/ - { - /* - * If the mpich mpir non-blocking collectives are enabled, return without - * first constructing the MPIR_Request. This signals to the - * MPIR_Iallgather_impl() function to invoke the mpich nbc implementation - * of MPI_Iallgather(). - */ - if (MPIDI_Process.mpir_nbc != 0) - return 0; - - /* - * MPIR_* nbc implementation is not enabled. Fake a non-blocking - * MPIR_Iallgather() with a blocking MPIR_Allgather(). - */ - if(unlikely(MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_ALL && comm_ptr->rank == 0)) - fprintf(stderr,"Using MPICH iallgather algorithm\n"); - - int mpierrno = 0; - int rc = MPIR_Allgather_impl(sendbuf, sendcount, sendtype, - recvbuf, recvcount, recvtype, - comm_ptr, &mpierrno); - - /* - * The blocking allgather has completed - create and complete a - * MPIR_Request object so the MPIR_Iallgather_impl() function does not - * perform an additional iallgather. - */ - MPIR_Request * mpid_request = MPID_Request_create_inline(); - mpid_request->kind = MPIR_REQUEST_KIND__COLL; - *request = mpid_request; - MPIDI_Request_complete_norelease_inline(mpid_request); - - return rc; - } - - return 0; -} diff --git a/src/mpid/pamid/src/coll/allgatherv/Makefile.mk b/src/mpid/pamid/src/coll/allgatherv/Makefile.mk deleted file mode 100644 index c8895ca..0000000 --- a/src/mpid/pamid/src/coll/allgatherv/Makefile.mk +++ /dev/null @@ -1,30 +0,0 @@ -# begin_generated_IBM_copyright_prolog -# -# This is an automatically generated copyright prolog. -# After initializing, DO NOT MODIFY OR MOVE -# --------------------------------------------------------------- -# Licensed Materials - Property of IBM -# Blue Gene/Q 5765-PER 5765-PRP -# -# (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved -# US Government Users Restricted Rights - -# Use, duplication, or disclosure restricted -# by GSA ADP Schedule Contract with IBM Corp. -# -# --------------------------------------------------------------- -# -# end_generated_IBM_copyright_prolog -# -*- mode: makefile-gmake; -*- - -# note that the includes always happen but the effects of their contents are -# affected by "if BUILD_PAMID" -if BUILD_PAMID - - -mpi_core_sources += \ - src/mpid/pamid/src/coll/allgatherv/mpido_allgatherv.c \ - src/mpid/pamid/src/coll/allgatherv/mpido_iallgatherv.c - - -endif BUILD_PAMID - diff --git a/src/mpid/pamid/src/coll/allgatherv/mpido_allgatherv.c b/src/mpid/pamid/src/coll/allgatherv/mpido_allgatherv.c deleted file mode 100644 index 1df9876..0000000 --- a/src/mpid/pamid/src/coll/allgatherv/mpido_allgatherv.c +++ /dev/null @@ -1,921 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/coll/allgatherv/mpido_allgatherv.c - * \brief ??? - */ - -/* #define TRACE_ON */ -#include <mpidimpl.h> - -static void allgatherv_cb_done(void *ctxt, void *clientdata, pami_result_t err) -{ - int *active = (int *)clientdata; - (*active)--; -} - -static void allred_cb_done(void *ctxt, void *clientdata, pami_result_t err) -{ - int *active = (int *)clientdata; - (*active)--; -} - - -/* ****************************************************************** */ -/** - * \brief Use (tree) MPIDO_Allreduce() to do a fast Allgatherv operation - * - * \note This function requires that: - * - The send/recv data types are contiguous - * - The recv buffer is continuous - * - Tree allreduce is availible (for max performance) - */ -/* ****************************************************************** */ -#define MAX_ALLGATHERV_ALLREDUCE_BUFFER_SIZE (1024*1024*2) -int MPIDO_Allgatherv_allreduce(const void *sendbuf, - int sendcount, - MPI_Datatype sendtype, - void *recvbuf, - const int *recvcounts, - int buffer_sum, - const int *displs, - MPI_Datatype recvtype, - MPI_Aint send_true_lb, - MPI_Aint recv_true_lb, - size_t send_size, - size_t recv_size, - MPIR_Comm * comm_ptr, - int *mpierrno) -{ - int start, rc, i; - int length; - char *startbuf = NULL; - char *destbuf = NULL; - const int rank = comm_ptr->rank; - TRACE_ERR("Entering MPIDO_Allgatherv_allreduce\n"); - - startbuf = (char *) recvbuf + recv_true_lb; - destbuf = startbuf + displs[rank] * recv_size; - - if (sendbuf != MPI_IN_PLACE) - { - char *outputbuf = (char *) sendbuf + send_true_lb; - memcpy(destbuf, outputbuf, send_size); - } - - //printf("buffer_sum %d, send_size %d recv_size %d\n", buffer_sum, - // (int)send_size, (int)recv_size); - - /* TODO: Change to PAMI */ - /*integer/long/double allgathers only*/ - /*Do a convert and then do the allreudce*/ - if ( buffer_sum <= MAX_ALLGATHERV_ALLREDUCE_BUFFER_SIZE && - (send_size & 0x3)==0 && (recv_size & 0x3)==0) - { - double *tmprbuf = (double *)MPL_malloc(buffer_sum*2); - if (tmprbuf == NULL) - goto direct_algo; /*skip int to fp conversion and go to direct - algo*/ - - double *tmpsbuf = tmprbuf + (displs[rank]*recv_size)/sizeof(int); - int *sibuf = (int *) destbuf; - - memset(tmprbuf, 0, displs[rank]*recv_size*2); - start = (displs[rank] + recvcounts[rank]) * recv_size; - length = buffer_sum - (displs[rank] + recvcounts[rank]) * recv_size; - memset(tmprbuf + start/sizeof(int), 0, length*2); - - for(i = 0; i < (send_size/sizeof(int)); ++i) - tmpsbuf[i] = (double)sibuf[i]; - - /* Switch to comm->coll_fns->fn() */ - rc = MPIDO_Allreduce(MPI_IN_PLACE, - tmprbuf, - buffer_sum/sizeof(int), - MPI_DOUBLE, - MPI_SUM, - comm_ptr, - mpierrno); - - sibuf = (int *) startbuf; - for(i = 0; i < (displs[rank]*recv_size/sizeof(int)); ++i) - sibuf[i] = (int)tmprbuf[i]; - - for(i = start/sizeof(int); i < buffer_sum/sizeof(int); ++i) - sibuf[i] = (int)tmprbuf[i]; - - MPL_free(tmprbuf); - return rc; - } - - direct_algo: - - start = 0; - length = displs[rank] * recv_size; - memset(startbuf + start, 0, length); - - start = (displs[rank] + - recvcounts[rank]) * recv_size; - length = buffer_sum - (displs[rank] + - recvcounts[rank]) * recv_size; - memset(startbuf + start, 0, length); - - TRACE_ERR("Calling MPIDO_Allreduce from MPIDO_Allgatherv_allreduce\n"); - /* Switch to comm->coll_fns->fn() */ - rc = MPIDO_Allreduce(MPI_IN_PLACE, - startbuf, - buffer_sum/sizeof(unsigned), - MPI_UNSIGNED, - MPI_BOR, - comm_ptr, - mpierrno); - - TRACE_ERR("Leaving MPIDO_Allgatherv_allreduce\n"); - return rc; -} - -/* ****************************************************************** */ -/** - * \brief Use (tree/rect) MPIDO_Bcast() to do a fast Allgatherv operation - * - * \note This function requires one of these (for max performance): - * - Tree broadcast - * - Rect broadcast - * ? Binomial broadcast - */ -/* ****************************************************************** */ -int MPIDO_Allgatherv_bcast(const void *sendbuf, - int sendcount, - MPI_Datatype sendtype, - void *recvbuf, - const int *recvcounts, - int buffer_sum, - const int *displs, - MPI_Datatype recvtype, - MPI_Aint send_true_lb, - MPI_Aint recv_true_lb, - size_t send_size, - size_t recv_size, - MPIR_Comm * comm_ptr, - int *mpierrno) -{ - const int rank = comm_ptr->rank; - TRACE_ERR("Entering MPIDO_Allgatherv_bcast\n"); - int i, rc=MPI_ERR_INTERN; - MPI_Aint extent; - MPIDU_Datatype_get_extent_macro(recvtype, extent); - - if (sendbuf != MPI_IN_PLACE) - { - void *destbuffer = recvbuf + displs[rank] * extent; - MPIR_Localcopy(sendbuf, - sendcount, - sendtype, - destbuffer, - recvcounts[rank], - recvtype); - } - - TRACE_ERR("Calling MPIDO_Bcasts in MPIDO_Allgatherv_bcast\n"); - for (i = 0; i < comm_ptr->local_size; i++) - { - void *destbuffer = recvbuf + displs[i] * extent; - /* Switch to comm->coll_fns->fn() */ - rc = MPIDO_Bcast(destbuffer, - recvcounts[i], - recvtype, - i, - comm_ptr, - mpierrno); - } - TRACE_ERR("Leaving MPIDO_Allgatherv_bcast\n"); - - return rc; -} - -/* ****************************************************************** */ -/** - * \brief Use (tree/rect) MPIDO_Alltoall() to do a fast Allgatherv operation - * - * \note This function requires that: - * - The send/recv data types are contiguous - * - DMA alltoallv is availible (for max performance) - */ -/* ****************************************************************** */ - -int MPIDO_Allgatherv_alltoall(const void *sendbuf, - int sendcount, - MPI_Datatype sendtype, - void *recvbuf, - int *recvcounts, - int buffer_sum, - const int *displs, - MPI_Datatype recvtype, - MPI_Aint send_true_lb, - MPI_Aint recv_true_lb, - size_t send_size, - size_t recv_size, - MPIR_Comm * comm_ptr, - int *mpierrno) -{ - TRACE_ERR("Entering MPIDO_Allgatherv_alltoallv\n"); - size_t total_send_size; - char *startbuf; - char *destbuf; - int i, rc; - int my_recvcounts = -1; - void *a2a_sendbuf = NULL; - const int size = comm_ptr->local_size; - int a2a_sendcounts[size]; - int a2a_senddispls[size]; - const int rank = comm_ptr->rank; - - total_send_size = recvcounts[rank] * recv_size; - for (i = 0; i < size; ++i) - { - a2a_sendcounts[i] = total_send_size; - a2a_senddispls[i] = 0; - } - if (sendbuf != MPI_IN_PLACE) - { - a2a_sendbuf = (char *)sendbuf + send_true_lb; - } - else - { - startbuf = (char *) recvbuf + recv_true_lb; - destbuf = startbuf + displs[rank] * recv_size; - a2a_sendbuf = destbuf; - a2a_sendcounts[rank] = 0; - my_recvcounts = recvcounts[rank]; - recvcounts[rank] = 0; - } - - TRACE_ERR("Calling alltoallv in MPIDO_Allgatherv_alltoallv\n"); - /* Switch to comm->coll_fns->fn() */ - rc = MPIDO_Alltoallv(a2a_sendbuf, - a2a_sendcounts, - a2a_senddispls, - MPI_CHAR, - recvbuf, - recvcounts, - displs, - recvtype, - comm_ptr, - mpierrno); - if (sendbuf == MPI_IN_PLACE) - recvcounts[rank] = my_recvcounts; - - TRACE_ERR("Leaving MPIDO_Allgatherv_alltoallv\n"); - return rc; -} - - -int -MPIDO_Allgatherv(const void *sendbuf, - int sendcount, - MPI_Datatype sendtype, - void *recvbuf, - const int *recvcounts, - const int *displs, - MPI_Datatype recvtype, - MPIR_Comm * comm_ptr, - int *mpierrno) -{ -#ifndef HAVE_PAMI_IN_PLACE - if (sendbuf == MPI_IN_PLACE) - { - MPID_Abort (NULL, 0, 1, "'MPI_IN_PLACE' requries support for `PAMI_IN_PLACE`"); - return -1; - } -#endif - TRACE_ERR("Entering MPIDO_Allgatherv\n"); - /* function pointer to be used to point to approperiate algorithm */ - - /* Check the nature of the buffers */ - MPIDU_Datatype*dt_null = NULL; - MPI_Aint send_true_lb = 0; - MPI_Aint recv_true_lb = 0; - size_t send_size = 0; - size_t recv_size = 0; - int config[6]; - int scount=sendcount; - - int i, rc, buffer_sum = 0; - const int size = comm_ptr->local_size; - char use_tree_reduce, use_alltoall, use_bcast, use_pami, use_opt; - char *sbuf, *rbuf; - const int rank = comm_ptr->rank; - const struct MPIDI_Comm* const mpid = &(comm_ptr->mpid); - int queryreq = 0; - -#if ASSERT_LEVEL==0 - /* We can't afford the tracing in ndebug/performance libraries */ - const unsigned verbose = 0; -#else - const unsigned verbose = (MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_ALL) && (rank == 0); -#endif - const int selected_type = mpid->user_selected_type[PAMI_XFER_ALLGATHERV_INT]; - - pami_xfer_t allred; - volatile unsigned allred_active = 1; - volatile unsigned allgatherv_active = 1; - pami_type_t stype, rtype; - int tmp; - const pami_metadata_t *my_md = (pami_metadata_t *)NULL; - - for(i=0;i<6;i++) config[i] = 1; - - allred.cb_done = allred_cb_done; - allred.cookie = (void *)&allred_active; - allred.algorithm = mpid->coll_algorithm[PAMI_XFER_ALLREDUCE][0][0]; - allred.cmd.xfer_allreduce.sndbuf = (void *)config; - allred.cmd.xfer_allreduce.stype = PAMI_TYPE_SIGNED_INT; - allred.cmd.xfer_allreduce.rcvbuf = (void *)config; - allred.cmd.xfer_allreduce.rtype = PAMI_TYPE_SIGNED_INT; - allred.cmd.xfer_allreduce.stypecount = 6; - allred.cmd.xfer_allreduce.rtypecount = 6; - allred.cmd.xfer_allreduce.op = PAMI_DATA_BAND; - - use_alltoall = mpid->allgathervs[2]; - use_tree_reduce = mpid->allgathervs[0]; - use_bcast = mpid->allgathervs[1]; - use_pami = selected_type != MPID_COLL_USE_MPICH; - - if((sendbuf != MPI_IN_PLACE) && (MPIDI_Datatype_to_pami(sendtype, &stype, -1, NULL, &tmp) != MPI_SUCCESS)) - use_pami = 0; - if(MPIDI_Datatype_to_pami(recvtype, &rtype, -1, NULL, &tmp) != MPI_SUCCESS) - use_pami = 0; - - use_opt = use_alltoall || use_tree_reduce || use_bcast || use_pami; - - if(!use_opt) /* back to MPICH */ - { - if(unlikely(verbose)) - fprintf(stderr,"Using MPICH allgatherv type %u.\n", - selected_type); - TRACE_ERR("Using MPICH Allgatherv\n"); - MPIDI_Update_last_algorithm(comm_ptr, "ALLGATHERV_MPICH"); -#if CUDA_AWARE_SUPPORT - if(MPIDI_Process.cuda_aware_support_on) - { - MPI_Aint sdt_extent,rdt_extent; - MPIDU_Datatype_get_extent_macro(sendtype, sdt_extent); - MPIDU_Datatype_get_extent_macro(recvtype, rdt_extent); - char *scbuf = NULL; - char *rcbuf = NULL; - int is_send_dev_buf = MPIDI_cuda_is_device_buf(sendbuf); - int is_recv_dev_buf = MPIDI_cuda_is_device_buf(recvbuf); - if(is_send_dev_buf) - { - scbuf = MPL_malloc(sdt_extent * sendcount); - cudaError_t cudaerr = CudaMemcpy(scbuf, sendbuf, sdt_extent * sendcount, cudaMemcpyDeviceToHost); - if (cudaSuccess != cudaerr) - fprintf(stderr, "cudaMemcpy failed: %s\n", CudaGetErrorString(cudaerr)); - } - else - scbuf = sendbuf; - size_t rtotal_buf; - if(is_recv_dev_buf) - { - //Since displs can be non-continous, we need to calculate max buffer size - int highest_displs = displs[size - 1]; - int highest_recvcount = recvcounts[size - 1]; - for(i = 0; i < size; i++) - { - if(displs[i]+recvcounts[i] > highest_displs+highest_recvcount) - { - highest_displs = displs[i]; - highest_recvcount = recvcounts[i]; - } - } - rtotal_buf = (highest_displs+highest_recvcount)*rdt_extent; - rcbuf = MPL_malloc(rtotal_buf); - if(sendbuf == MPI_IN_PLACE) - { - cudaError_t cudaerr = CudaMemcpy(rcbuf, recvbuf, rtotal_buf, cudaMemcpyDeviceToHost); - if (cudaSuccess != cudaerr) - fprintf(stderr, "cudaMemcpy failed: %s\n", CudaGetErrorString(cudaerr)); - } - else - memset(rcbuf, 0, rtotal_buf); - } - else - rcbuf = recvbuf; - int cuda_res = MPIR_Allgatherv(scbuf, sendcount, sendtype, rcbuf, recvcounts, displs, recvtype, comm_ptr, mpierrno); - if(is_send_dev_buf)MPL_free(scbuf); - if(is_recv_dev_buf) - { - cudaError_t cudaerr = CudaMemcpy(recvbuf, rcbuf, rtotal_buf, cudaMemcpyHostToDevice); - if (cudaSuccess != cudaerr) - fprintf(stderr, "cudaMemcpy failed: %s\n", CudaGetErrorString(cudaerr)); - MPL_free(rcbuf); - } - return cuda_res; - } - else -#endif - return MPIR_Allgatherv(sendbuf, sendcount, sendtype, - recvbuf, recvcounts, displs, recvtype, - comm_ptr, mpierrno); - } - - MPIDI_Datatype_get_info(1, - recvtype, - config[MPID_RECV_CONTIG], - recv_size, - dt_null, - recv_true_lb); - - if(sendbuf == MPI_IN_PLACE) - { - sbuf = PAMI_IN_PLACE; - if(unlikely(verbose)) - fprintf(stderr,"allgatherv MPI_IN_PLACE buffering\n"); - stype = rtype; - scount = recvcounts[rank]; - send_size = recv_size * scount; - } - else - { - MPIDI_Datatype_get_info(sendcount, - sendtype, - config[MPID_SEND_CONTIG], - send_size, - dt_null, - send_true_lb); - sbuf = (char *)sendbuf+send_true_lb; - } - - rbuf = (char *)recvbuf+recv_true_lb; - - if(use_alltoall || use_bcast || use_tree_reduce) - { - if (displs[0]) - config[MPID_RECV_CONTINUOUS] = 0; - - for (i = 1; i < size; i++) - { - buffer_sum += recvcounts[i - 1]; - if (buffer_sum != displs[i]) - { - config[MPID_RECV_CONTINUOUS] = 0; - break; - } - } - - buffer_sum += recvcounts[size - 1]; - - buffer_sum *= recv_size; - - /* disable with "safe allgatherv" env var */ - if(mpid->preallreduces[MPID_ALLGATHERV_PREALLREDUCE]) - { - MPIDI_Post_coll_t allred_post; - MPIDI_Context_post(MPIDI_Context[0], &allred_post.state, - MPIDI_Pami_post_wrapper, (void *)&allred); - - MPID_PROGRESS_WAIT_WHILE(allred_active); - } - - use_tree_reduce = mpid->allgathervs[0] && - config[MPID_RECV_CONTIG] && config[MPID_SEND_CONTIG] && - config[MPID_RECV_CONTINUOUS] && buffer_sum % sizeof(unsigned) == 0; - - use_alltoall = mpid->allgathervs[2] && - config[MPID_RECV_CONTIG] && config[MPID_SEND_CONTIG]; - - use_bcast = mpid->allgathervs[1]; - } - - if(use_pami) - { - pami_xfer_t allgatherv; - allgatherv.cb_done = allgatherv_cb_done; - allgatherv.cookie = (void *)&allgatherv_active; - if(selected_type == MPID_COLL_OPTIMIZED) - { - if((mpid->cutoff_size[PAMI_XFER_ALLGATHERV_INT][0] == 0) || - (mpid->cutoff_size[PAMI_XFER_ALLGATHERV_INT][0] > 0 && mpid->cutoff_size[PAMI_XFER_ALLGATHERV_INT][0] >= send_size)) - { - allgatherv.algorithm = mpid->opt_protocol[PAMI_XFER_ALLGATHERV_INT][0]; - my_md = &mpid->opt_protocol_md[PAMI_XFER_ALLGATHERV_INT][0]; - queryreq = mpid->must_query[PAMI_XFER_ALLGATHERV_INT][0]; - } - else - return MPIR_Allgatherv(sendbuf, sendcount, sendtype, - recvbuf, recvcounts, displs, recvtype, - comm_ptr, mpierrno); - } - else - { - allgatherv.algorithm = mpid->user_selected[PAMI_XFER_ALLGATHERV_INT]; - my_md = &mpid->user_metadata[PAMI_XFER_ALLGATHERV_INT]; - queryreq = selected_type; - } - - allgatherv.cmd.xfer_allgatherv_int.sndbuf = sbuf; - allgatherv.cmd.xfer_allgatherv_int.rcvbuf = rbuf; - - allgatherv.cmd.xfer_allgatherv_int.stype = stype; - allgatherv.cmd.xfer_allgatherv_int.rtype = rtype; - allgatherv.cmd.xfer_allgatherv_int.stypecount = scount; - allgatherv.cmd.xfer_allgatherv_int.rtypecounts = (int *) recvcounts; - allgatherv.cmd.xfer_allgatherv_int.rdispls = (int *) displs; - - if(unlikely (queryreq == MPID_COLL_ALWAYS_QUERY || - queryreq == MPID_COLL_CHECK_FN_REQUIRED)) - { - metadata_result_t result = {0}; - TRACE_ERR("Querying allgatherv_int protocol %s, type was %d\n", my_md->name, - selected_type); - if(my_md->check_fn == NULL) - { - /* process metadata bits */ - if((!my_md->check_correct.values.inplace) && (sendbuf == MPI_IN_PLACE)) - result.check.unspecified = 1; -/* Can't check ranges like this. Non-local. Comment out for now. - if(my_md->check_correct.values.rangeminmax) - { - MPI_Aint data_true_lb; - MPIDU_Datatype*data_ptr; - int data_size, data_contig; - MPIDI_Datatype_get_info(sendcount, sendtype, data_contig, data_size, data_ptr, data_true_lb); - if((my_md->range_lo <= data_size) && - (my_md->range_hi >= data_size)) - ; - else - { - result.check.range = 1; - if(unlikely(verbose)) - { - fprintf(stderr,"message size (%u) outside range (%zu<->%zu) for %s.\n", - data_size, - my_md->range_lo, - my_md->range_hi, - my_md->name); - } - } - } - */ - } - else /* calling the check fn is sufficient */ - result = my_md->check_fn(&allgatherv); - TRACE_ERR("Allgatherv bitmask: %#X\n", result.bitmask); - result.check.nonlocal = 0; /* #warning REMOVE THIS WHEN IMPLEMENTED */ - if(result.bitmask) - { - if(unlikely(verbose)) - fprintf(stderr,"Query failed for %s. Using MPICH allgatherv.\n", my_md->name); - MPIDI_Update_last_algorithm(comm_ptr, "ALLGATHERV_MPICH"); - return MPIR_Allgatherv(sendbuf, sendcount, sendtype, - recvbuf, recvcounts, displs, recvtype, - comm_ptr, mpierrno); - } - if(my_md->check_correct.values.asyncflowctl && !(--(comm_ptr->mpid.num_requests))) - { - comm_ptr->mpid.num_requests = MPIDI_Process.optimized.num_requests; - int tmpmpierrno; - if(unlikely(verbose)) - fprintf(stderr,"Query barrier required for %s\n", my_md->name); - MPIDO_Barrier(comm_ptr, &tmpmpierrno); - } - } - - if(unlikely(verbose)) - { - unsigned long long int threadID; - MPL_thread_id_t tid; - MPL_thread_self(&tid); - threadID = (unsigned long long int)tid; - fprintf(stderr,"<%llx> Using protocol %s for allgatherv on %u\n", - threadID, - my_md->name, - (unsigned) comm_ptr->context_id); - } - MPIDI_Post_coll_t allgatherv_post; - MPIDI_Context_post(MPIDI_Context[0], &allgatherv_post.state, - MPIDI_Pami_post_wrapper, (void *)&allgatherv); - - MPIDI_Update_last_algorithm(comm_ptr, my_md->name); - - TRACE_ERR("Rank %d waiting on active %d\n", rank, allgatherv_active); - MPID_PROGRESS_WAIT_WHILE(allgatherv_active); - - return PAMI_SUCCESS; - } - - /* TODO These need ordered in speed-order */ - if(use_tree_reduce) - { - if(unlikely(verbose)) - fprintf(stderr,"Using tree reduce allgatherv type %u.\n", - selected_type); - rc = MPIDO_Allgatherv_allreduce(sendbuf, sendcount, sendtype, - recvbuf, recvcounts, buffer_sum, displs, recvtype, - send_true_lb, recv_true_lb, send_size, recv_size, - comm_ptr, mpierrno); - MPIDI_Update_last_algorithm(comm_ptr, "ALLGATHERV_OPT_ALLREDUCE"); - return rc; - } - - if(use_bcast) - { - if(unlikely(verbose)) - fprintf(stderr,"Using bcast allgatherv type %u.\n", - selected_type); - rc = MPIDO_Allgatherv_bcast(sendbuf, sendcount, sendtype, - recvbuf, recvcounts, buffer_sum, displs, recvtype, - send_true_lb, recv_true_lb, send_size, recv_size, - comm_ptr, mpierrno); - MPIDI_Update_last_algorithm(comm_ptr, "ALLGATHERV_OPT_BCAST"); - return rc; - } - - if(use_alltoall) - { - if(unlikely(verbose)) - fprintf(stderr,"Using alltoall allgatherv type %u.\n", - selected_type); - rc = MPIDO_Allgatherv_alltoall(sendbuf, sendcount, sendtype, - recvbuf, (int *)recvcounts, buffer_sum, displs, recvtype, - send_true_lb, recv_true_lb, send_size, recv_size, - comm_ptr, mpierrno); - MPIDI_Update_last_algorithm(comm_ptr, "ALLGATHERV_OPT_ALLTOALL"); - return rc; - } - - if(unlikely(verbose)) - fprintf(stderr,"Using MPICH allgatherv type %u.\n", - selected_type); - TRACE_ERR("Using MPICH for Allgatherv\n"); - MPIDI_Update_last_algorithm(comm_ptr, "ALLGATHERV_MPICH"); - return MPIR_Allgatherv(sendbuf, sendcount, sendtype, - recvbuf, recvcounts, displs, recvtype, - comm_ptr, mpierrno); -} - -int -MPIDO_Allgatherv_simple(const void *sendbuf, - int sendcount, - MPI_Datatype sendtype, - void *recvbuf, - const int *recvcounts, - const int *displs, - MPI_Datatype recvtype, - MPIR_Comm * comm_ptr, - int *mpierrno) -{ -#ifndef HAVE_PAMI_IN_PLACE - if (sendbuf == MPI_IN_PLACE) - { - MPID_Abort (NULL, 0, 1, "'MPI_IN_PLACE' requries support for `PAMI_IN_PLACE`"); - return -1; - } -#endif - TRACE_ERR("Entering MPIDO_Allgatherv_optimized\n"); - /* function pointer to be used to point to approperiate algorithm */ - /* Check the nature of the buffers */ - MPIDU_Datatype*dt_null = NULL; - MPI_Aint send_true_lb = 0; - MPI_Aint recv_true_lb = 0; - size_t send_size = 0; - size_t recv_size = 0; - size_t rcvtypelen = 0; - int snd_data_contig = 0, rcv_data_contig = 0; - void *snd_noncontig_buff = NULL, *rcv_noncontig_buff = NULL; - int scount=sendcount; - - char *sbuf, *rbuf; - pami_type_t stype = NULL, rtype; - const int rank = comm_ptr->rank; - const int size = comm_ptr->local_size; - const struct MPIDI_Comm* const mpid = &(comm_ptr->mpid); -#if ASSERT_LEVEL==0 - /* We can't afford the tracing in ndebug/performance libraries */ - const unsigned verbose = 0; -#else - const unsigned verbose = (MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_ALL) && (rank == 0); -#endif - - int recvcontinuous=0; - size_t totalrecvcount=0; - int *lrecvdispls = NULL; /* possible local displs calculated for noncontinous */ - int *lrecvcounts = NULL;/* possible local counts calculated for noncontinous */ - const int *precvdispls = displs; /* pointer to displs to use as pami parmi */ - const int *precvcounts = recvcounts; /* pointer to counts to use as pami parmi */ - int inplace = sendbuf == MPI_IN_PLACE? 1 : 0; - - - volatile unsigned allgatherv_active = 1; - int recvok=PAMI_SUCCESS, sendok=PAMI_SUCCESS; - int tmp; - const pami_metadata_t *my_md; - - - MPIDI_Datatype_get_info(1, - recvtype, - rcv_data_contig, - rcvtypelen, - dt_null, - recv_true_lb); - - if(MPIDI_Pamix_collsel_advise != NULL && mpid->collsel_fast_query != NULL) - { - advisor_algorithm_t advisor_algorithms[1]; - int num_algorithms = MPIDI_Pamix_collsel_advise(mpid->collsel_fast_query, PAMI_XFER_ALLGATHERV_INT, rcvtypelen * recvcounts[0], advisor_algorithms, 1); - if(num_algorithms) - { - if(advisor_algorithms[0].algorithm_type == COLLSEL_EXTERNAL_ALGO) - { - return MPIR_Allgatherv(sendbuf, sendcount, sendtype, - recvbuf, recvcounts, displs, recvtype, - comm_ptr, mpierrno); - } - else if(advisor_algorithms[0].metadata && advisor_algorithms[0].metadata->check_correct.values.asyncflowctl && !(--(comm_ptr->mpid.num_requests))) - { - comm_ptr->mpid.num_requests = MPIDI_Process.optimized.num_requests; - int tmpmpierrno; - if(unlikely(verbose)) - fprintf(stderr,"Query barrier required for %s\n", advisor_algorithms[0].metadata->name); - MPIDO_Barrier(comm_ptr, &tmpmpierrno); - } - } - } - - - if(!inplace) - { - sendok = MPIDI_Datatype_to_pami(sendtype, &stype, -1, NULL, &tmp); - MPIDI_Datatype_get_info(sendcount, sendtype, snd_data_contig, send_size, dt_null, send_true_lb); - sbuf = (char *)sendbuf + send_true_lb; - if(!snd_data_contig || (sendok != PAMI_SUCCESS)) - { - stype = PAMI_TYPE_UNSIGNED_CHAR; - scount = send_size; - if(!snd_data_contig) - { - snd_noncontig_buff = MPL_malloc(send_size); - sbuf = snd_noncontig_buff; - if(snd_noncontig_buff == NULL) - { - MPID_Abort(NULL, MPI_ERR_NO_SPACE, 1, - "Fatal: Cannot allocate pack buffer"); - } - MPIR_Localcopy(sendbuf, sendcount, sendtype, - snd_noncontig_buff, send_size,MPI_CHAR); - } - } - } - else - sbuf = PAMI_IN_PLACE; - - recvok = MPIDI_Datatype_to_pami(recvtype, &rtype, -1, NULL, &tmp); - rbuf = (char *)recvbuf+recv_true_lb; - if(!rcv_data_contig || (recvok != PAMI_SUCCESS)) - { - rtype = PAMI_TYPE_UNSIGNED_CHAR; - totalrecvcount = recvcounts[0]; - recvcontinuous = displs[0] == 0? 1 : 0 ; - int i; - precvdispls = lrecvdispls = MPL_malloc(size*sizeof(int)); - precvcounts = lrecvcounts = MPL_malloc(size*sizeof(int)); - lrecvdispls[0]= 0; - lrecvcounts[0]= rcvtypelen * recvcounts[0]; - for(i=1; i<size; ++i) - { - lrecvdispls[i]= rcvtypelen * totalrecvcount; - totalrecvcount += recvcounts[i]; - if(displs[i] != (displs[i-1] + recvcounts[i-1])) - recvcontinuous = 0; - lrecvcounts[i]= rcvtypelen * recvcounts[i]; - } - recv_size = rcvtypelen * totalrecvcount; - TRACE_ERR("Pack receive rcv_contig %zu, recvok %zd, totalrecvcount %zu, recvcontinuous %zu, rcvtypelen %zu, recv_size %zu\n", - (size_t)rcv_data_contig, (size_t)recvok, (size_t)totalrecvcount, (size_t)recvcontinuous,(size_t)rcvtypelen, (size_t)recv_size); - - rcv_noncontig_buff = MPL_malloc(recv_size); - rbuf = rcv_noncontig_buff; - if(rcv_noncontig_buff == NULL) - { - MPID_Abort(NULL, MPI_ERR_NO_SPACE, 1, - "Fatal: Cannot allocate pack buffer"); - } - if(inplace) - { - size_t extent; - MPIDU_Datatype_get_extent_macro(recvtype,extent); - MPIR_Localcopy(recvbuf + displs[rank]*extent, recvcounts[rank], recvtype, - rcv_noncontig_buff + precvdispls[rank], precvcounts[rank],MPI_CHAR); - scount = precvcounts[rank]; - stype = PAMI_TYPE_UNSIGNED_CHAR; - sbuf = PAMI_IN_PLACE; - } - } - - - pami_xfer_t allgatherv; - allgatherv.cb_done = allgatherv_cb_done; - allgatherv.cookie = (void *)&allgatherv_active; - allgatherv.cmd.xfer_allgatherv_int.sndbuf = sbuf; - allgatherv.cmd.xfer_allgatherv_int.rcvbuf = rbuf; - allgatherv.cmd.xfer_allgatherv_int.stype = stype;/* stype is ignored when sndbuf == PAMI_IN_PLACE */ - allgatherv.cmd.xfer_allgatherv_int.rtype = rtype; - allgatherv.cmd.xfer_allgatherv_int.stypecount = scount; - allgatherv.cmd.xfer_allgatherv_int.rtypecounts = (int *) precvcounts; - allgatherv.cmd.xfer_allgatherv_int.rdispls = (int *) precvdispls; - allgatherv.algorithm = mpid->coll_algorithm[PAMI_XFER_ALLGATHERV_INT][0][0]; - my_md = &mpid->coll_metadata[PAMI_XFER_ALLGATHERV_INT][0][0]; - - TRACE_ERR("Calling allgatherv via %s()\n", MPIDI_Process.context_post.active>0?"PAMI_Collective":"PAMI_Context_post"); - MPIDI_Post_coll_t allgatherv_post; - MPIDI_Context_post(MPIDI_Context[0], &allgatherv_post.state, - MPIDI_Pami_post_wrapper, (void *)&allgatherv); - - MPIDI_Update_last_algorithm(comm_ptr, my_md->name); - - TRACE_ERR("Rank %d waiting on active %d\n", rank, allgatherv_active); - MPID_PROGRESS_WAIT_WHILE(allgatherv_active); - - if(!rcv_data_contig || (recvok != PAMI_SUCCESS)) - { - if(recvcontinuous) - { - MPIR_Localcopy(rcv_noncontig_buff, recv_size,MPI_CHAR, - recvbuf, totalrecvcount, recvtype); - } - else - { - size_t extent; - int i; - MPIDU_Datatype_get_extent_macro(recvtype,extent); - for(i=0; i<size; ++i) - { - char* scbuf = (char*)rcv_noncontig_buff+ precvdispls[i]; - char* rcbuf = (char*)recvbuf + displs[i]*extent; - MPIR_Localcopy(scbuf, precvcounts[i], MPI_CHAR, - rcbuf, recvcounts[i], recvtype); - TRACE_ERR("Pack recv src extent %zu, displ[%zu]=%zu, count[%zu]=%zu buf[%zu]=%u\n", - (size_t)extent, (size_t)i,(size_t)precvdispls[i],(size_t)i,(size_t)precvcounts[i],(size_t)precvdispls[i], *(int*)scbuf); - TRACE_ERR("Pack recv dest extent %zu, displ[%zu]=%zu, count[%zu]=%zu buf[%zu]=%u\n", - (size_t)extent, (size_t)i,(size_t)displs[i],(size_t)i,(size_t)recvcounts[i],(size_t)displs[i], *(int*)rcbuf); - } - } - MPL_free(rcv_noncontig_buff); - } - if(!snd_data_contig) MPL_free(snd_noncontig_buff); - if(lrecvdispls) MPL_free(lrecvdispls); - if(lrecvcounts) MPL_free(lrecvcounts); - - return MPI_SUCCESS; -} - - -int -MPIDO_CSWrapper_allgatherv(pami_xfer_t *allgatherv, - void *comm) -{ - int mpierrno = 0; - MPIR_Comm *comm_ptr = (MPIR_Comm*)comm; - MPI_Datatype sendtype, recvtype; - void *sbuf; - MPIDI_coll_check_in_place(allgatherv->cmd.xfer_allgatherv_int.sndbuf, &sbuf); - int rc = MPIDI_Dtpami_to_dtmpi( allgatherv->cmd.xfer_allgatherv_int.stype, - &sendtype, - NULL, - NULL); - if(rc == -1) return rc; - - rc = MPIDI_Dtpami_to_dtmpi( allgatherv->cmd.xfer_allgatherv_int.rtype, - &recvtype, - NULL, - NULL); - if(rc == -1) return rc; - - rc = MPIR_Allgatherv(sbuf, - allgatherv->cmd.xfer_allgatherv_int.stypecount, sendtype, - allgatherv->cmd.xfer_allgatherv_int.rcvbuf, - allgatherv->cmd.xfer_allgatherv_int.rtypecounts, - allgatherv->cmd.xfer_allgatherv_int.rdispls, recvtype, - comm_ptr, &mpierrno); - if(allgatherv->cb_done && rc == 0) - allgatherv->cb_done(NULL, allgatherv->cookie, PAMI_SUCCESS); - return rc; - - -} - diff --git a/src/mpid/pamid/src/coll/allgatherv/mpido_iallgatherv.c b/src/mpid/pamid/src/coll/allgatherv/mpido_iallgatherv.c deleted file mode 100644 index b7570cd..0000000 --- a/src/mpid/pamid/src/coll/allgatherv/mpido_iallgatherv.c +++ /dev/null @@ -1,77 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/coll/allgatherv/mpido_iallgatherv.c - * \brief ??? - */ - -/*#define TRACE_ON */ -#include <mpidimpl.h> - -int -MPIDO_Iallgatherv(const void *sendbuf, - int sendcount, - MPI_Datatype sendtype, - void *recvbuf, - const int *recvcounts, - const int *displs, - MPI_Datatype recvtype, - MPIR_Comm * comm_ptr, - MPIR_Request ** request) -{ - TRACE_ERR("Entering MPIDO_Iallgatherv\n"); - - - /*if (unlikely((data_size == 0) || (user_selected_type == MPID_COLL_USE_MPICH)))*/ - { - /* - * If the mpich mpir non-blocking collectives are enabled, return without - * first constructing the MPIR_Request. This signals to the - * MPIR_Iallgather_impl() function to invoke the mpich nbc implementation - * of MPI_Iallgather(). - */ - if (MPIDI_Process.mpir_nbc != 0) - return 0; - - /* - * MPIR_* nbc implementation is not enabled. Fake a non-blocking - * MPIR_Iallgather() with a blocking MPIR_Allgather(). - */ - if(unlikely(MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_ALL && comm_ptr->rank == 0)) - fprintf(stderr,"Using MPICH iallgather algorithm\n"); - - int mpierrno = 0; - int rc = MPIR_Allgatherv_impl(sendbuf, sendcount, sendtype, - recvbuf, recvcounts, displs, recvtype, - comm_ptr, &mpierrno); - - /* - * The blocking allgather has completed - create and complete a - * MPIR_Request object so the MPIR_Iallgather_impl() function does not - * perform an additional iallgather. - */ - MPIR_Request * mpid_request = MPID_Request_create_inline(); - mpid_request->kind = MPIR_REQUEST_KIND__COLL; - *request = mpid_request; - MPIDI_Request_complete_norelease_inline(mpid_request); - - return rc; - } - - return 0; -} diff --git a/src/mpid/pamid/src/coll/allreduce/Makefile.mk b/src/mpid/pamid/src/coll/allreduce/Makefile.mk deleted file mode 100644 index 875e2d9..0000000 --- a/src/mpid/pamid/src/coll/allreduce/Makefile.mk +++ /dev/null @@ -1,30 +0,0 @@ -# begin_generated_IBM_copyright_prolog -# -# This is an automatically generated copyright prolog. -# After initializing, DO NOT MODIFY OR MOVE -# --------------------------------------------------------------- -# Licensed Materials - Property of IBM -# Blue Gene/Q 5765-PER 5765-PRP -# -# (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved -# US Government Users Restricted Rights - -# Use, duplication, or disclosure restricted -# by GSA ADP Schedule Contract with IBM Corp. -# -# --------------------------------------------------------------- -# -# end_generated_IBM_copyright_prolog -# -*- mode: makefile-gmake; -*- - -# note that the includes always happen but the effects of their contents are -# affected by "if BUILD_PAMID" -if BUILD_PAMID - - -mpi_core_sources += \ - src/mpid/pamid/src/coll/allreduce/mpido_allreduce.c \ - src/mpid/pamid/src/coll/allreduce/mpido_iallreduce.c - - -endif BUILD_PAMID - diff --git a/src/mpid/pamid/src/coll/allreduce/mpido_allreduce.c b/src/mpid/pamid/src/coll/allreduce/mpido_allreduce.c deleted file mode 100644 index 48f0bfc..0000000 --- a/src/mpid/pamid/src/coll/allreduce/mpido_allreduce.c +++ /dev/null @@ -1,562 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/coll/allreduce/mpido_allreduce.c - * \brief ??? - */ - -/* #define TRACE_ON */ - -#include <mpidimpl.h> -/* -#undef TRACE_ERR -#define TRACE_ERR(format, ...) fprintf(stderr, format, ##__VA_ARGS__) -*/ -static void cb_allreduce(void *ctxt, void *clientdata, pami_result_t err) -{ - int *active = (int *) clientdata; - TRACE_ERR("callback enter, active: %d\n", (*active)); - MPIDI_Progress_signal(); - (*active)--; -} - -int MPIDO_Allreduce(const void *sendbuf, - void *recvbuf, - int count, - MPI_Datatype dt, - MPI_Op op, - MPIR_Comm *comm_ptr, - int *mpierrno) -{ -#ifndef HAVE_PAMI_IN_PLACE - if (sendbuf == MPI_IN_PLACE) - { - MPID_Abort (NULL, 0, 1, "'MPI_IN_PLACE' requries support for `PAMI_IN_PLACE`"); - return -1; - } -#endif - void *sbuf; - TRACE_ERR("Entering mpido_allreduce\n"); - pami_type_t pdt; - pami_data_function pop; - int mu; - int rc; -#ifdef TRACE_ON - int len; - char op_str[255]; - char dt_str[255]; - MPIDI_Op_to_string(op, op_str); - PMPI_Type_get_name(dt, dt_str, &len); -#endif - volatile unsigned active = 1; - pami_xfer_t allred; - pami_algorithm_t my_allred = 0; - const pami_metadata_t *my_md = (pami_metadata_t *)NULL; - int alg_selected = 0; - const int rank = comm_ptr->rank; - const struct MPIDI_Comm* const mpid = &(comm_ptr->mpid); - const int selected_type = mpid->user_selected_type[PAMI_XFER_ALLREDUCE]; -#if ASSERT_LEVEL==0 - /* We can't afford the tracing in ndebug/performance libraries */ - const unsigned verbose = 0; -#else - const unsigned verbose = (MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_ALL) && (rank == 0); -#endif - int queryreq = 0; - if(likely(dt == MPI_DOUBLE || dt == MPI_DOUBLE_PRECISION)) - { - rc = MPI_SUCCESS; - pdt = PAMI_TYPE_DOUBLE; - if(likely(selected_type == MPID_COLL_OPTIMIZED) && - (mpid->query_cached_allreduce != MPID_COLL_USE_MPICH)) - { - /* double protocol works on all message sizes */ - my_allred = mpid->cached_allreduce; - my_md = &mpid->cached_allreduce_md; - alg_selected = 1; - } - if(likely(op == MPI_SUM)) - pop = PAMI_DATA_SUM; - else if(likely(op == MPI_MAX)) - pop = PAMI_DATA_MAX; - else if(likely(op == MPI_MIN)) - pop = PAMI_DATA_MIN; - else - { - alg_selected = 0; - rc = MPIDI_Datatype_to_pami(dt, &pdt, op, &pop, &mu); - } - } - else rc = MPIDI_Datatype_to_pami(dt, &pdt, op, &pop, &mu); - - if(unlikely(verbose)) - fprintf(stderr,"allred rc %u,count %d, Datatype %p, op %p, mu %u, selectedvar %u != %u, sendbuf %p, recvbuf %p\n", - rc, count, pdt, pop, mu, - (unsigned)selected_type,MPID_COLL_USE_MPICH, sendbuf, recvbuf); - /* convert to metadata query */ - /* Punt count 0 allreduce to MPICH. Let them do whatever's 'right' */ - if(unlikely(rc != MPI_SUCCESS || (count==0) || - selected_type == MPID_COLL_USE_MPICH)) - { - if(unlikely(verbose)) - fprintf(stderr,"Using MPICH allreduce type %u.\n", - selected_type); - MPIDI_Update_last_algorithm(comm_ptr, "ALLREDUCE_MPICH"); -#if CUDA_AWARE_SUPPORT - if(MPIDI_Process.cuda_aware_support_on) - { - MPI_Aint dt_extent; - MPIDU_Datatype_get_extent_macro(dt, dt_extent); - char *scbuf = NULL; - char *rcbuf = NULL; - int is_send_dev_buf = MPIDI_cuda_is_device_buf(sendbuf); - int is_recv_dev_buf = MPIDI_cuda_is_device_buf(recvbuf); - if(is_send_dev_buf) - { - scbuf = MPL_malloc(dt_extent * count); - cudaError_t cudaerr = CudaMemcpy(scbuf, sendbuf, dt_extent * count, cudaMemcpyDeviceToHost); - if (cudaSuccess != cudaerr) - fprintf(stderr, "cudaMemcpy failed: %s\n", CudaGetErrorString(cudaerr)); - } - else - scbuf = sendbuf; - if(is_recv_dev_buf) - { - rcbuf = MPL_malloc(dt_extent * count); - if(sendbuf == MPI_IN_PLACE) - { - cudaError_t cudaerr = CudaMemcpy(rcbuf, recvbuf, dt_extent * count, cudaMemcpyDeviceToHost); - if (cudaSuccess != cudaerr) - fprintf(stderr, "cudaMemcpy failed: %s\n", CudaGetErrorString(cudaerr)); - } - else - memset(rcbuf, 0, dt_extent * count); - } - else - rcbuf = recvbuf; - int cuda_res = MPIR_Allreduce(scbuf, rcbuf, count, dt, op, comm_ptr, mpierrno); - if(is_send_dev_buf)MPL_free(scbuf); - if(is_recv_dev_buf) - { - cudaError_t cudaerr = CudaMemcpy(recvbuf, rcbuf, dt_extent * count, cudaMemcpyHostToDevice); - if (cudaSuccess != cudaerr) - fprintf(stderr, "cudaMemcpy failed: %s\n", CudaGetErrorString(cudaerr)); - MPL_free(rcbuf); - } - return cuda_res; - } - else -#endif - return MPIR_Allreduce(sendbuf, recvbuf, count, dt, op, comm_ptr, mpierrno); - } - - sbuf = (void *)sendbuf; - if(unlikely(sendbuf == MPI_IN_PLACE)) - { - if(unlikely(verbose)) - fprintf(stderr,"allreduce MPI_IN_PLACE buffering\n"); - sbuf = PAMI_IN_PLACE; - } - - allred.cb_done = cb_allreduce; - allred.cookie = (void *)&active; - allred.cmd.xfer_allreduce.sndbuf = sbuf; - allred.cmd.xfer_allreduce.stype = pdt; - allred.cmd.xfer_allreduce.rcvbuf = recvbuf; - allred.cmd.xfer_allreduce.rtype = pdt; - allred.cmd.xfer_allreduce.stypecount = count; - allred.cmd.xfer_allreduce.rtypecount = count; - allred.cmd.xfer_allreduce.op = pop; - - TRACE_ERR("Allreduce - Basic Collective Selection\n"); - - if(unlikely(!alg_selected)) /* Cached double algorithm not selected above */ - { - if(likely(selected_type == MPID_COLL_OPTIMIZED)) - { - if(mpid->query_cached_allreduce != MPID_COLL_USE_MPICH) - { /* try the cached algorithm first, assume it's always a query algorithm so query now */ - my_allred = mpid->cached_allreduce; - my_md = &mpid->cached_allreduce_md; - alg_selected = 1; - if(my_md->check_fn != NULL)/*This should always be the case in FCA.. Otherwise punt to mpich*/ - { - metadata_result_t result = {0}; - TRACE_ERR("querying allreduce algorithm %s\n", - my_md->name); - result = my_md->check_fn(&allred); - TRACE_ERR("bitmask: %#X\n", result.bitmask); - /* \todo Ignore check_correct.values.nonlocal until we implement the - 'pre-allreduce allreduce' or the 'safe' environment flag. - We will basically assume 'safe' -- that all ranks are aligned (or not). - */ - result.check.nonlocal = 0; /* #warning REMOVE THIS WHEN IMPLEMENTED */ - if(!result.bitmask) - ; /* ok, algorithm selected */ - else - { - alg_selected = 0; - if(unlikely(verbose)) - fprintf(stderr,"check_fn failed for %s.\n", my_md->name); - } - } - else /* no check_fn, manually look at the metadata fields */ - { - TRACE_ERR("Optimzed selection line %d\n",__LINE__); - /* Check if the message range if restricted */ - if(my_md->check_correct.values.rangeminmax) - { - int data_size; - MPIDI_Datatype_get_data_size(count, dt, data_size); - if((my_md->range_lo <= data_size) && - (my_md->range_hi >= data_size)) - ; /* ok, algorithm selected */ - else - { - if(unlikely(verbose)) - fprintf(stderr,"message size (%u) outside range (%zu<->%zu) for %s.\n", - data_size, - my_md->range_lo, - my_md->range_hi, - my_md->name); - alg_selected = 0; - } - } - /* \todo check the rest of the metadata */ - } - } - /* If we didn't use the cached protocol above (query failed?) then check regular optimized protocol fields */ - if(!alg_selected) - { - const int queryreq0 = mpid->must_query[PAMI_XFER_ALLREDUCE][0]; - const int queryreq1 = mpid->must_query[PAMI_XFER_ALLREDUCE][1]; - /* TODO this really needs to be cleaned up for BGQ and fca */ - if(queryreq0 == MPID_COLL_NOQUERY && - count <= mpid->cutoff_size[PAMI_XFER_ALLREDUCE][0]) - { - TRACE_ERR("Optimzed selection line %d\n",__LINE__); - my_allred = mpid->opt_protocol[PAMI_XFER_ALLREDUCE][0]; - my_md = &mpid->opt_protocol_md[PAMI_XFER_ALLREDUCE][0]; - alg_selected = 1; - } - else if(queryreq1 == MPID_COLL_NOQUERY && - count > mpid->cutoff_size[PAMI_XFER_ALLREDUCE][0]) - { - TRACE_ERR("Optimzed selection line %d\n",__LINE__); - my_allred = mpid->opt_protocol[PAMI_XFER_ALLREDUCE][1]; - my_md = &mpid->opt_protocol_md[PAMI_XFER_ALLREDUCE][1]; - alg_selected = 1; - } - else if(((queryreq0 == MPID_COLL_CHECK_FN_REQUIRED) || - (queryreq0 == MPID_COLL_QUERY) || - (queryreq0 == MPID_COLL_ALWAYS_QUERY)) && - ((mpid->cutoff_size[PAMI_XFER_ALLREDUCE][0] == 0) || - (count <= mpid->cutoff_size[PAMI_XFER_ALLREDUCE][0] && mpid->cutoff_size[PAMI_XFER_ALLREDUCE][0] > 0))) - { - TRACE_ERR("Optimzed selection line %d\n",__LINE__); - my_allred = mpid->opt_protocol[PAMI_XFER_ALLREDUCE][0]; - my_md = &mpid->opt_protocol_md[PAMI_XFER_ALLREDUCE][0]; - alg_selected = 1; - queryreq = queryreq0; - } - else if((queryreq1 == MPID_COLL_CHECK_FN_REQUIRED) || - (queryreq1 == MPID_COLL_QUERY) || - (queryreq1 == MPID_COLL_ALWAYS_QUERY)) - { - TRACE_ERR("Optimzed selection line %d\n",__LINE__); - my_allred = mpid->opt_protocol[PAMI_XFER_ALLREDUCE][1]; - my_md = &mpid->opt_protocol_md[PAMI_XFER_ALLREDUCE][1]; - alg_selected = 1; - queryreq = queryreq1; - } - TRACE_ERR("Alg selected: %d\n", alg_selected); - if(likely(alg_selected)) - { - if(unlikely((queryreq == MPID_COLL_CHECK_FN_REQUIRED) || - (queryreq == MPID_COLL_QUERY) || - (queryreq == MPID_COLL_ALWAYS_QUERY))) - { - TRACE_ERR("Optimzed selection line %d\n",__LINE__); - if(my_md->check_fn != NULL)/*This should always be the case in FCA.. Otherwise punt to mpich*/ - { - metadata_result_t result = {0}; - TRACE_ERR("querying allreduce algorithm %s\n", - my_md->name); - result = my_md->check_fn(&allred); - TRACE_ERR("bitmask: %#X\n", result.bitmask); - /* \todo Ignore check_correct.values.nonlocal until we implement the - 'pre-allreduce allreduce' or the 'safe' environment flag. - We will basically assume 'safe' -- that all ranks are aligned (or not). - */ - result.check.nonlocal = 0; /* #warning REMOVE THIS WHEN IMPLEMENTED */ - if(!result.bitmask) - ; /* ok, algorithm selected */ - else - { - alg_selected = 0; - if(unlikely(verbose)) - fprintf(stderr,"check_fn failed for %s.\n", my_md->name); - } - } - else /* no check_fn, manually look at the metadata fields */ - { - TRACE_ERR("Optimzed selection line %d\n",__LINE__); - /* Check if the message range if restricted */ - if(my_md->check_correct.values.rangeminmax) - { - int data_size; - MPIDI_Datatype_get_data_size(count, dt, data_size); - if((my_md->range_lo <= data_size) && - (my_md->range_hi >= data_size)) - ; /* ok, algorithm selected */ - else - { - if(unlikely(verbose)) - fprintf(stderr,"message size (%u) outside range (%zu<->%zu) for %s.\n", - data_size, - my_md->range_lo, - my_md->range_hi, - my_md->name); - alg_selected = 0; - } - } - /* \todo check the rest of the metadata */ - } - } - else - { - TRACE_ERR("Using %s for allreduce\n", my_md->name); - } - } - } - } - else - { - TRACE_ERR("Non-Optimzed selection line %d\n",__LINE__); - my_allred = mpid->user_selected[PAMI_XFER_ALLREDUCE]; - my_md = &mpid->user_metadata[PAMI_XFER_ALLREDUCE]; - if(selected_type == MPID_COLL_QUERY || - selected_type == MPID_COLL_ALWAYS_QUERY || - selected_type == MPID_COLL_CHECK_FN_REQUIRED) - { - TRACE_ERR("Non-Optimzed selection line %d\n",__LINE__); - if(my_md->check_fn != NULL) - { - /* For now, we don't distinguish between MPID_COLL_ALWAYS_QUERY & - MPID_COLL_CHECK_FN_REQUIRED, we just call the fn */ - metadata_result_t result = {0}; - TRACE_ERR("querying allreduce algorithm %s, type was %d\n", - my_md->name, - selected_type); - result = mpid->user_metadata[PAMI_XFER_ALLREDUCE].check_fn(&allred); - TRACE_ERR("bitmask: %#X\n", result.bitmask); - /* \todo Ignore check_correct.values.nonlocal until we implement the - 'pre-allreduce allreduce' or the 'safe' environment flag. - We will basically assume 'safe' -- that all ranks are aligned (or not). - */ - result.check.nonlocal = 0; /* #warning REMOVE THIS WHEN IMPLEMENTED */ - if(!result.bitmask) - alg_selected = 1; /* query algorithm successfully selected */ - else - if(unlikely(verbose)) - fprintf(stderr,"check_fn failed for %s.\n", my_md->name); - } - else /* no check_fn, manually look at the metadata fields */ - { - TRACE_ERR("Non-Optimzed selection line %d\n",__LINE__); - /* Check if the message range if restricted */ - if(my_md->check_correct.values.rangeminmax) - { - int data_size; - MPIDI_Datatype_get_data_size(count, dt, data_size); - if((my_md->range_lo <= data_size) && - (my_md->range_hi >= data_size)) - alg_selected = 1; /* query algorithm successfully selected */ - else - if(unlikely(verbose)) - fprintf(stderr,"message size (%u) outside range (%zu<->%zu) for %s.\n", - data_size, - my_md->range_lo, - my_md->range_hi, - my_md->name); - } - /* \todo check the rest of the metadata */ - } - } - else alg_selected = 1; /* non-query algorithm selected */ - - } - } - - if(unlikely(!alg_selected)) /* must be fallback to MPICH */ - { - if(unlikely(verbose)) - fprintf(stderr,"Using MPICH allreduce\n"); - MPIDI_Update_last_algorithm(comm_ptr, "ALLREDUCE_MPICH"); - return MPIR_Allreduce(sendbuf, recvbuf, count, dt, op, comm_ptr, mpierrno); - } - allred.algorithm = my_allred; - - if(unlikely(verbose)) - { - unsigned long long int threadID; - MPL_thread_id_t tid; - MPL_thread_self(&tid); - threadID = (unsigned long long int)tid; - fprintf(stderr,"<%llx> Using protocol %s for allreduce on %u\n", - threadID, - my_md->name, - (unsigned) comm_ptr->context_id); - } - - MPIDI_Post_coll_t allred_post; - MPIDI_Context_post(MPIDI_Context[0], &allred_post.state, - MPIDI_Pami_post_wrapper, (void *)&allred); - - MPID_assert(rc == PAMI_SUCCESS); - MPIDI_Update_last_algorithm(comm_ptr,my_md->name); - MPID_PROGRESS_WAIT_WHILE(active); - TRACE_ERR("allreduce done\n"); - return MPI_SUCCESS; -} - -int MPIDO_Allreduce_simple(const void *sendbuf, - void *recvbuf, - int count, - MPI_Datatype dt, - MPI_Op op, - MPIR_Comm *comm_ptr, - int *mpierrno) -{ -#ifndef HAVE_PAMI_IN_PLACE - if (sendbuf == MPI_IN_PLACE) - { - MPID_Abort (NULL, 0, 1, "'MPI_IN_PLACE' requries support for `PAMI_IN_PLACE`"); - return -1; - } -#endif - void *sbuf; - TRACE_ERR("Entering MPIDO_Allreduce_optimized\n"); - pami_type_t pdt; - pami_data_function pop; - int mu; - int rc; -#ifdef TRACE_ON - int len; - char op_str[255]; - char dt_str[255]; - MPIDI_Op_to_string(op, op_str); - PMPI_Type_get_name(dt, dt_str, &len); -#endif - volatile unsigned active = 1; - pami_xfer_t allred; - const pami_metadata_t *my_allred_md = (pami_metadata_t *)NULL; - const struct MPIDI_Comm* const mpid = &(comm_ptr->mpid); - MPIDU_Datatype*data_ptr; - MPI_Aint data_true_lb = 0; - int data_size, data_contig; - - MPIDI_Datatype_get_info(1, dt, - data_contig, data_size, data_ptr, data_true_lb); - - - if(MPIDI_Pamix_collsel_advise != NULL && mpid->collsel_fast_query != NULL) - { - advisor_algorithm_t advisor_algorithms[1]; - int num_algorithms = MPIDI_Pamix_collsel_advise(mpid->collsel_fast_query, PAMI_XFER_ALLREDUCE, data_size * count, advisor_algorithms, 1); - if(num_algorithms) - { - if(advisor_algorithms[0].algorithm_type == COLLSEL_EXTERNAL_ALGO) - { - return MPIR_Allreduce(sendbuf, recvbuf, count, dt, op, comm_ptr, mpierrno); - } - } - } - - - rc = MPIDI_Datatype_to_pami(dt, &pdt, op, &pop, &mu); - - /* convert to metadata query */ - /* Punt count 0 allreduce to MPICH. Let them do whatever's 'right' */ - if(unlikely(rc != MPI_SUCCESS || (count==0))) - { - MPIDI_Update_last_algorithm(comm_ptr, "ALLREDUCE_MPICH"); - return MPIR_Allreduce(sendbuf, recvbuf, count, dt, op, comm_ptr, mpierrno); - } - - if(!data_contig) - { - MPIDI_Update_last_algorithm(comm_ptr, "ALLREDUCE_MPICH"); - return MPIR_Allreduce(sendbuf, recvbuf, count, dt, op, comm_ptr, mpierrno); - } - - sbuf = (void *)sendbuf; - if(unlikely(sendbuf == MPI_IN_PLACE)) - { - sbuf = PAMI_IN_PLACE; - } - - allred.cb_done = cb_allreduce; - allred.cookie = (void *)&active; - allred.cmd.xfer_allreduce.sndbuf = sbuf; - allred.cmd.xfer_allreduce.stype = pdt; - allred.cmd.xfer_allreduce.rcvbuf = recvbuf; - allred.cmd.xfer_allreduce.rtype = pdt; - allred.cmd.xfer_allreduce.stypecount = count; - allred.cmd.xfer_allreduce.rtypecount = count; - allred.cmd.xfer_allreduce.op = pop; - allred.algorithm = mpid->coll_algorithm[PAMI_XFER_ALLREDUCE][0][0]; - my_allred_md = &mpid->coll_metadata[PAMI_XFER_ALLREDUCE][0][0]; - - MPIDI_Post_coll_t allred_post; - MPIDI_Context_post(MPIDI_Context[0], &allred_post.state, - MPIDI_Pami_post_wrapper, (void *)&allred); - - MPID_assert(rc == PAMI_SUCCESS); - MPIDI_Update_last_algorithm(comm_ptr,my_allred_md->name); - MPID_PROGRESS_WAIT_WHILE(active); - TRACE_ERR("Allreduce done\n"); - return MPI_SUCCESS; -} - -int -MPIDO_CSWrapper_allreduce(pami_xfer_t *allreduce, - void *comm) -{ - int mpierrno = 0; - MPIR_Comm *comm_ptr = (MPIR_Comm*)comm; - MPI_Datatype type; - MPI_Op op; - void *sbuf; - MPIDI_coll_check_in_place(allreduce->cmd.xfer_allreduce.sndbuf, &sbuf); - int rc = MPIDI_Dtpami_to_dtmpi( allreduce->cmd.xfer_allreduce.stype, - &type, - allreduce->cmd.xfer_allreduce.op, - &op); - if(rc == -1) return rc; - - rc = MPIR_Allreduce(sbuf, - allreduce->cmd.xfer_allreduce.rcvbuf, - allreduce->cmd.xfer_allreduce.rtypecount, - type, op, comm_ptr, &mpierrno); - if(allreduce->cb_done && rc == 0) - allreduce->cb_done(NULL, allreduce->cookie, PAMI_SUCCESS); - return rc; - -} - diff --git a/src/mpid/pamid/src/coll/allreduce/mpido_iallreduce.c b/src/mpid/pamid/src/coll/allreduce/mpido_iallreduce.c deleted file mode 100644 index d504b80..0000000 --- a/src/mpid/pamid/src/coll/allreduce/mpido_iallreduce.c +++ /dev/null @@ -1,73 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/coll/allreduce/mpido_iallreduce.c - * \brief ??? - */ - -/*#define TRACE_ON*/ - -#include <mpidimpl.h> - -int MPIDO_Iallreduce(const void *sendbuf, - void *recvbuf, - int count, - MPI_Datatype dt, - MPI_Op op, - MPIR_Comm *comm_ptr, - MPIR_Request **request) -{ - TRACE_ERR("Entering mpido_iallreduce\n"); - - /*if (unlikely((data_size == 0) || (user_selected_type == MPID_COLL_USE_MPICH)))*/ - { - /* - * If the mpich mpir non-blocking collectives are enabled, return without - * first constructing the MPIR_Request. This signals to the - * MPIR_Iallreduce_impl() function to invoke the mpich nbc implementation - * of MPI_Iallreduce(). - */ - if (MPIDI_Process.mpir_nbc != 0) - return 0; - - /* - * MPIR_* nbc implementation is not enabled. Fake a non-blocking - * MPIR_Iallreduce() with a blocking MPIR_Allreduce(). - */ - if(unlikely(MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_ALL && comm_ptr->rank == 0)) - fprintf(stderr,"Using MPICH iallreduce algorithm\n"); - - int mpierrno = 0; - int rc = MPIR_Allreduce_impl(sendbuf, recvbuf, count, dt, op, - comm_ptr, &mpierrno); - - /* - * The blocking allreduce has completed - create and complete a - * MPIR_Request object so the MPIR_Iallreduce_impl() function does not - * perform an additional iallreduce. - */ - MPIR_Request * mpid_request = MPID_Request_create_inline(); - mpid_request->kind = MPIR_REQUEST_KIND__COLL; - *request = mpid_request; - MPIDI_Request_complete_norelease_inline(mpid_request); - - return rc; - } - - return 0; -} diff --git a/src/mpid/pamid/src/coll/alltoall/Makefile.mk b/src/mpid/pamid/src/coll/alltoall/Makefile.mk deleted file mode 100644 index f82f25e..0000000 --- a/src/mpid/pamid/src/coll/alltoall/Makefile.mk +++ /dev/null @@ -1,30 +0,0 @@ -# begin_generated_IBM_copyright_prolog -# -# This is an automatically generated copyright prolog. -# After initializing, DO NOT MODIFY OR MOVE -# --------------------------------------------------------------- -# Licensed Materials - Property of IBM -# Blue Gene/Q 5765-PER 5765-PRP -# -# (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved -# US Government Users Restricted Rights - -# Use, duplication, or disclosure restricted -# by GSA ADP Schedule Contract with IBM Corp. -# -# --------------------------------------------------------------- -# -# end_generated_IBM_copyright_prolog -# -*- mode: makefile-gmake; -*- - -# note that the includes always happen but the effects of their contents are -# affected by "if BUILD_PAMID" -if BUILD_PAMID - - -mpi_core_sources += \ - src/mpid/pamid/src/coll/alltoall/mpido_alltoall.c \ - src/mpid/pamid/src/coll/alltoall/mpido_ialltoall.c - - -endif BUILD_PAMID - diff --git a/src/mpid/pamid/src/coll/alltoall/mpido_alltoall.c b/src/mpid/pamid/src/coll/alltoall/mpido_alltoall.c deleted file mode 100644 index a3354ae..0000000 --- a/src/mpid/pamid/src/coll/alltoall/mpido_alltoall.c +++ /dev/null @@ -1,449 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/coll/alltoall/mpido_alltoall.c - * \brief ??? - */ - -/* #define TRACE_ON */ - -#include <mpidimpl.h> - -static void cb_alltoall(void *ctxt, void *clientdata, pami_result_t err) -{ - int *active = (int *)clientdata; - TRACE_ERR("alltoall callback enter, active: %d\n", (*active)); - MPIDI_Progress_signal(); - (*active)--; -} - - -int MPIDO_Alltoall(const void *sendbuf, - int sendcount, - MPI_Datatype sendtype, - void *recvbuf, - int recvcount, - MPI_Datatype recvtype, - MPIR_Comm *comm_ptr, - int *mpierrno) -{ -#ifndef HAVE_PAMI_IN_PLACE - if (sendbuf == MPI_IN_PLACE) - { - MPID_Abort (NULL, 0, 1, "'MPI_IN_PLACE' requries support for `PAMI_IN_PLACE`"); - return -1; - } -#endif - TRACE_ERR("Entering MPIDO_Alltoall\n"); - volatile unsigned active = 1; - MPIDU_Datatype*sdt, *rdt; - pami_type_t stype, rtype; - MPI_Aint sdt_true_lb=0, rdt_true_lb; - MPIDI_Post_coll_t alltoall_post; - int snd_contig, rcv_contig, pamidt=1; - int sndlen ATTRIBUTE((unused)), rcvlen ATTRIBUTE((unused)); - int tmp; -#if ASSERT_LEVEL==0 - /* We can't afford the tracing in ndebug/performance libraries */ - const unsigned verbose = 0; -#else - const unsigned verbose = (MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_ALL) && (comm_ptr->rank == 0); -#endif - const struct MPIDI_Comm* const mpid = &(comm_ptr->mpid); - const int selected_type = mpid->user_selected_type[PAMI_XFER_ALLTOALL]; - - if(sendbuf != MPI_IN_PLACE) - { - MPIDI_Datatype_get_info(1, sendtype, snd_contig, sndlen, sdt, sdt_true_lb); - if(!snd_contig) pamidt = 0; - } - MPIDI_Datatype_get_info(1, recvtype, rcv_contig, rcvlen, rdt, rdt_true_lb); - if(!rcv_contig) pamidt = 0; - - /* Alltoall is much simpler if bytes are required because we don't need to - * malloc displ/count arrays and copy things - */ - - - /* Is it a built in type? If not, send to MPICH */ - if(sendbuf != MPI_IN_PLACE && (MPIDI_Datatype_to_pami(sendtype, &stype, -1, NULL, &tmp) != MPI_SUCCESS)) - pamidt = 0; - if(MPIDI_Datatype_to_pami(recvtype, &rtype, -1, NULL, &tmp) != MPI_SUCCESS) - pamidt = 0; - - if((selected_type == MPID_COLL_USE_MPICH) || - pamidt == 0) - { - if(unlikely(verbose)) - fprintf(stderr,"Using MPICH alltoall algorithm\n"); -#if CUDA_AWARE_SUPPORT - if(MPIDI_Process.cuda_aware_support_on) - { - MPI_Aint sdt_extent,rdt_extent; - MPIDU_Datatype_get_extent_macro(sendtype, sdt_extent); - MPIDU_Datatype_get_extent_macro(recvtype, rdt_extent); - char *scbuf = NULL; - char *rcbuf = NULL; - int is_send_dev_buf = MPIDI_cuda_is_device_buf(sendbuf); - int is_recv_dev_buf = MPIDI_cuda_is_device_buf(recvbuf); - if(is_send_dev_buf) - { - scbuf = MPL_malloc(sdt_extent * sendcount); - cudaError_t cudaerr = CudaMemcpy(scbuf, sendbuf, sdt_extent * sendcount, cudaMemcpyDeviceToHost); - if (cudaSuccess != cudaerr) - fprintf(stderr, "cudaMemcpy failed: %s\n", CudaGetErrorString(cudaerr)); - } - else - scbuf = sendbuf; - if(is_recv_dev_buf) - { - rcbuf = MPL_malloc(recvcount * rdt_extent); - if(sendbuf == MPI_IN_PLACE) - { - cudaError_t cudaerr = CudaMemcpy(rcbuf, recvbuf, recvcount * rdt_extent, cudaMemcpyDeviceToHost); - if (cudaSuccess != cudaerr) - fprintf(stderr, "cudaMemcpy failed: %s\n", CudaGetErrorString(cudaerr)); - } - else - memset(rcbuf, 0, recvcount * rdt_extent); - } - else - rcbuf = recvbuf; - int cuda_res = MPIR_Alltoall_intra(scbuf, sendcount, sendtype, rcbuf, recvcount, recvtype, comm_ptr, mpierrno); - if(is_send_dev_buf)MPL_free(scbuf); - if(is_recv_dev_buf) - { - cudaError_t cudaerr = CudaMemcpy(recvbuf, rcbuf, recvcount * rdt_extent, cudaMemcpyHostToDevice); - if (cudaSuccess != cudaerr) - fprintf(stderr, "cudaMemcpy failed: %s\n", CudaGetErrorString(cudaerr)); - MPL_free(rcbuf); - } - return cuda_res; - } - else -#endif - return MPIR_Alltoall_intra(sendbuf, sendcount, sendtype, - recvbuf, recvcount, recvtype, - comm_ptr, mpierrno); - - } - - pami_xfer_t alltoall; - pami_algorithm_t my_alltoall; - const pami_metadata_t *my_md = (pami_metadata_t *)NULL; - int queryreq = 0; - if(selected_type == MPID_COLL_OPTIMIZED) - { - TRACE_ERR("Optimized alltoall was pre-selected\n"); - my_alltoall = mpid->opt_protocol[PAMI_XFER_ALLTOALL][0]; - my_md = &mpid->opt_protocol_md[PAMI_XFER_ALLTOALL][0]; - queryreq = mpid->must_query[PAMI_XFER_ALLTOALL][0]; - } - else - { - TRACE_ERR("Alltoall was specified by user\n"); - my_alltoall = mpid->user_selected[PAMI_XFER_ALLTOALL]; - my_md = &mpid->user_metadata[PAMI_XFER_ALLTOALL]; - queryreq = selected_type; - } - char *pname = my_md->name; - TRACE_ERR("Using alltoall protocol %s\n", pname); - - alltoall.cb_done = cb_alltoall; - alltoall.cookie = (void *)&active; - alltoall.algorithm = my_alltoall; - if(sendbuf == MPI_IN_PLACE) - { - if(unlikely(verbose)) - fprintf(stderr,"alltoall MPI_IN_PLACE buffering\n"); - alltoall.cmd.xfer_alltoall.stype = rtype; - alltoall.cmd.xfer_alltoall.stypecount = recvcount; - alltoall.cmd.xfer_alltoall.sndbuf = PAMI_IN_PLACE; - } - else - { - alltoall.cmd.xfer_alltoall.stype = stype; - alltoall.cmd.xfer_alltoall.stypecount = sendcount; - alltoall.cmd.xfer_alltoall.sndbuf = (char *)sendbuf + sdt_true_lb; - } - alltoall.cmd.xfer_alltoall.rcvbuf = (char *)recvbuf + rdt_true_lb; - - alltoall.cmd.xfer_alltoall.rtypecount = recvcount; - alltoall.cmd.xfer_alltoall.rtype = rtype; - - if(unlikely(queryreq == MPID_COLL_ALWAYS_QUERY || - queryreq == MPID_COLL_CHECK_FN_REQUIRED)) - { - metadata_result_t result = {0}; - TRACE_ERR("querying alltoall protocol %s, query level was %d\n", pname, - queryreq); - if(my_md->check_fn == NULL) - { - /* process metadata bits */ - if((!my_md->check_correct.values.inplace) && (sendbuf == MPI_IN_PLACE)) - result.check.unspecified = 1; - if(my_md->check_correct.values.rangeminmax) - { - MPI_Aint data_true_lb ATTRIBUTE((unused)); - MPIDU_Datatype*data_ptr; - int data_size, data_contig ATTRIBUTE((unused)); - MPIDI_Datatype_get_info(sendcount, sendtype, data_contig, data_size, data_ptr, data_true_lb); - if((my_md->range_lo <= data_size) && - (my_md->range_hi >= data_size)) - ; /* ok, algorithm selected */ - else - { - result.check.range = 1; - if(unlikely(verbose)) - { - fprintf(stderr,"message size (%u) outside range (%zu<->%zu) for %s.\n", - data_size, - my_md->range_lo, - my_md->range_hi, - my_md->name); - } - } - } - } - else /* calling the check fn is sufficient */ - result = my_md->check_fn(&alltoall); - TRACE_ERR("bitmask: %#X\n", result.bitmask); - result.check.nonlocal = 0; /* #warning REMOVE THIS WHEN IMPLEMENTED */ - if(result.bitmask) - { - if(unlikely(verbose)) - fprintf(stderr,"Query failed for %s. Using MPICH alltoall.\n", pname); - MPIDI_Update_last_algorithm(comm_ptr, "ALLTOALL_MPICH"); - return MPIR_Alltoall_intra(sendbuf, sendcount, sendtype, - recvbuf, recvcount, recvtype, - comm_ptr, mpierrno); - } - if(my_md->check_correct.values.asyncflowctl && !(--(comm_ptr->mpid.num_requests))) - { - comm_ptr->mpid.num_requests = MPIDI_Process.optimized.num_requests; - int tmpmpierrno; - if(unlikely(verbose)) - fprintf(stderr,"Query barrier required for %s\n", my_md->name); - MPIDO_Barrier(comm_ptr, &tmpmpierrno); - } - } - - if(unlikely(verbose)) - { - unsigned long long int threadID; - MPL_thread_id_t tid; - MPL_thread_self(&tid); - threadID = (unsigned long long int)tid; - fprintf(stderr,"<%llx> Using protocol %s for alltoall on %u\n", - threadID, - my_md->name, - (unsigned) comm_ptr->context_id); - } - - MPIDI_Context_post(MPIDI_Context[0], &alltoall_post.state, - MPIDI_Pami_post_wrapper, (void *)&alltoall); - - TRACE_ERR("Waiting on active\n"); - MPID_PROGRESS_WAIT_WHILE(active); - - TRACE_ERR("Leaving alltoall\n"); - return PAMI_SUCCESS; -} - - -int MPIDO_Alltoall_simple(const void *sendbuf, - int sendcount, - MPI_Datatype sendtype, - void *recvbuf, - int recvcount, - MPI_Datatype recvtype, - MPIR_Comm *comm_ptr, - int *mpierrno) -{ -#ifndef HAVE_PAMI_IN_PLACE - if (sendbuf == MPI_IN_PLACE) - { - MPID_Abort (NULL, 0, 1, "'MPI_IN_PLACE' requries support for `PAMI_IN_PLACE`"); - return -1; - } -#endif - TRACE_ERR("Entering MPIDO_Alltoall_optimized\n"); - volatile unsigned active = 1; - void *snd_noncontig_buff = NULL, *rcv_noncontig_buff = NULL; - void *sbuf = NULL, *rbuf = NULL; - size_t send_size = 0; - size_t recv_size = 0; - MPIDU_Segment segment; - MPIDU_Datatype*sdt, *rdt; - MPI_Aint sdt_true_lb=0, rdt_true_lb; - MPIDI_Post_coll_t alltoall_post; - int sndlen, rcvlen, snd_contig = 1, rcv_contig = 1; - const int rank = comm_ptr->rank; - const int size = comm_ptr->local_size; -#if ASSERT_LEVEL==0 - /* We can't afford the tracing in ndebug/performance libraries */ - const unsigned verbose = 0; -#else - const unsigned verbose = (MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_ALL) && (comm_ptr->rank == 0); -#endif - - const struct MPIDI_Comm* const mpid = &(comm_ptr->mpid); - - MPIDI_Datatype_get_info(1, recvtype, rcv_contig, rcvlen, rdt, rdt_true_lb); - rbuf = (char *)recvbuf + rdt_true_lb; - recv_size = rcvlen * recvcount; - - if(MPIDI_Pamix_collsel_advise != NULL && mpid->collsel_fast_query != NULL) - { - advisor_algorithm_t advisor_algorithms[1]; - int num_algorithms = MPIDI_Pamix_collsel_advise(mpid->collsel_fast_query, PAMI_XFER_ALLTOALL, recv_size, advisor_algorithms, 1); - if(num_algorithms) - { - if(advisor_algorithms[0].algorithm_type == COLLSEL_EXTERNAL_ALGO) - { - return MPIR_Alltoall_intra(sendbuf, sendcount, sendtype, - recvbuf, recvcount, recvtype, - comm_ptr, mpierrno); - } - else if(advisor_algorithms[0].metadata && advisor_algorithms[0].metadata->check_correct.values.asyncflowctl && !(--(comm_ptr->mpid.num_requests))) - { - comm_ptr->mpid.num_requests = MPIDI_Process.optimized.num_requests; - int tmpmpierrno; - if(unlikely(verbose)) - fprintf(stderr,"Query barrier required for %s\n", advisor_algorithms[0].metadata->name); - MPIDO_Barrier(comm_ptr, &tmpmpierrno); - } - } - } - - if(sendbuf != MPI_IN_PLACE) - { - MPIDI_Datatype_get_info(1, sendtype, snd_contig, sndlen, sdt, sdt_true_lb); - sbuf = (char *)sendbuf + sdt_true_lb; - send_size = sndlen * sendcount; - if(!snd_contig) - { - snd_noncontig_buff = MPL_malloc(send_size*size); - sbuf = snd_noncontig_buff; - if(snd_noncontig_buff == NULL) - { - MPID_Abort(NULL, MPI_ERR_NO_SPACE, 1, - "Fatal: Cannot allocate pack buffer"); - } - DLOOP_Offset last = send_size*size; - MPIDU_Segment_init(sendbuf, sendcount*size, sendtype, &segment, 0); - MPIDU_Segment_pack(&segment, 0, &last, snd_noncontig_buff); - - } - } - - if(!rcv_contig) - { - rcv_noncontig_buff = MPL_malloc(recv_size*size); - rbuf = rcv_noncontig_buff; - if(rcv_noncontig_buff == NULL) - { - MPID_Abort(NULL, MPI_ERR_NO_SPACE, 1, - "Fatal: Cannot allocate pack buffer"); - } - if(sendbuf == MPI_IN_PLACE) - { - size_t extent; - MPIDU_Datatype_get_extent_macro(recvtype,extent); - MPIR_Localcopy(recvbuf + (rank*recvcount*extent), recvcount, recvtype, - rcv_noncontig_buff + (rank*recv_size), recv_size,MPI_CHAR); - } - } - - /* Alltoall is much simpler if bytes are required because we don't need to - * malloc displ/count arrays and copy things - */ - - - pami_xfer_t alltoall; -#ifdef TRACE_ON - const pami_metadata_t *my_alltoall_md; - my_alltoall_md = &mpid->coll_metadata[PAMI_XFER_ALLTOALL][0][0]; - TRACE_ERR("Using alltoall protocol %s\n", my_alltoall_md->name); -#endif - alltoall.cb_done = cb_alltoall; - alltoall.cookie = (void *)&active; - alltoall.algorithm = mpid->coll_algorithm[PAMI_XFER_ALLTOALL][0][0]; - alltoall.cmd.xfer_alltoall.stype = PAMI_TYPE_BYTE;/* stype is ignored when sndbuf == PAMI_IN_PLACE */ - alltoall.cmd.xfer_alltoall.stypecount = send_size; - alltoall.cmd.xfer_alltoall.sndbuf = sbuf; - - if(sendbuf == MPI_IN_PLACE) - { - alltoall.cmd.xfer_alltoall.sndbuf = PAMI_IN_PLACE; - } - alltoall.cmd.xfer_alltoall.rcvbuf = rbuf; - alltoall.cmd.xfer_alltoall.rtypecount = recv_size; - alltoall.cmd.xfer_alltoall.rtype = PAMI_TYPE_BYTE; - - MPIDI_Context_post(MPIDI_Context[0], &alltoall_post.state, - MPIDI_Pami_post_wrapper, (void *)&alltoall); - - TRACE_ERR("Waiting on active\n"); - MPID_PROGRESS_WAIT_WHILE(active); - - if(!rcv_contig) - { - MPIR_Localcopy(rcv_noncontig_buff, recv_size*size, MPI_CHAR, - recvbuf, recvcount*size, recvtype); - MPL_free(rcv_noncontig_buff); - } - if(!snd_contig) MPL_free(snd_noncontig_buff); - - TRACE_ERR("Leaving MPIDO_Alltoall_optimized\n"); - return MPI_SUCCESS; -} - - -int -MPIDO_CSWrapper_alltoall(pami_xfer_t *alltoall, - void *comm) -{ - int mpierrno = 0; - MPIR_Comm *comm_ptr = (MPIR_Comm*)comm; - MPI_Datatype sendtype, recvtype; - void *sbuf; - MPIDI_coll_check_in_place(alltoall->cmd.xfer_alltoall.sndbuf, &sbuf); - int rc = MPIDI_Dtpami_to_dtmpi( alltoall->cmd.xfer_alltoall.stype, - &sendtype, - NULL, - NULL); - if(rc == -1) return rc; - - rc = MPIDI_Dtpami_to_dtmpi( alltoall->cmd.xfer_alltoall.rtype, - &recvtype, - NULL, - NULL); - if(rc == -1) return rc; - - rc = MPIR_Alltoall_intra(sbuf, - alltoall->cmd.xfer_alltoall.stypecount, sendtype, - alltoall->cmd.xfer_alltoall.rcvbuf, - alltoall->cmd.xfer_alltoall.rtypecount, recvtype, - comm_ptr, &mpierrno); - if(alltoall->cb_done && rc == 0) - alltoall->cb_done(NULL, alltoall->cookie, PAMI_SUCCESS); - return rc; - -} - diff --git a/src/mpid/pamid/src/coll/alltoall/mpido_ialltoall.c b/src/mpid/pamid/src/coll/alltoall/mpido_ialltoall.c deleted file mode 100644 index 7ecfb2c..0000000 --- a/src/mpid/pamid/src/coll/alltoall/mpido_ialltoall.c +++ /dev/null @@ -1,76 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/coll/alltoall/mpido_ialltoall.c - * \brief ??? - */ - -/*#define TRACE_ON*/ - -#include <mpidimpl.h> - -int MPIDO_Ialltoall(const void *sendbuf, - int sendcount, - MPI_Datatype sendtype, - void *recvbuf, - int recvcount, - MPI_Datatype recvtype, - MPIR_Comm *comm_ptr, - MPIR_Request **request) -{ - TRACE_ERR("Entering MPIDO_Ialltoall\n"); - - /*if (unlikely((data_size == 0) || (user_selected_type == MPID_COLL_USE_MPICH)))*/ - { - /* - * If the mpich mpir non-blocking collectives are enabled, return without - * first constructing the MPIR_Request. This signals to the - * MPIR_Ialltoall_impl() function to invoke the mpich nbc implementation - * of MPI_Ialltoall(). - */ - if (MPIDI_Process.mpir_nbc != 0) - return 0; - - /* - * MPIR_* nbc implementation is not enabled. Fake a non-blocking - * MPIR_Ialltoall() with a blocking MPIR_Alltoall(). - */ - if(unlikely(MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_ALL && comm_ptr->rank == 0)) - fprintf(stderr,"Using MPICH blocking alltoall algorithm\n"); - - int mpierrno = 0; - int rc = MPIR_Alltoall_impl(sendbuf, sendcount, sendtype, - recvbuf, recvcount, recvtype, - comm_ptr, &mpierrno); - - /* - * The blocking allitoall has completed - create and complete a - * MPIR_Request object so the MPIR_Ialltoall_impl() function does not - * perform an additional ialltoall. - */ - MPIR_Request * mpid_request = MPID_Request_create_inline(); - mpid_request->kind = MPIR_REQUEST_KIND__COLL; - *request = mpid_request; - MPIDI_Request_complete_norelease_inline(mpid_request); - - return rc; - } - - TRACE_ERR("Leaving MPIDO_Ialltoall\n"); - return 0; -} diff --git a/src/mpid/pamid/src/coll/alltoallv/Makefile.mk b/src/mpid/pamid/src/coll/alltoallv/Makefile.mk deleted file mode 100644 index 4e95a3c..0000000 --- a/src/mpid/pamid/src/coll/alltoallv/Makefile.mk +++ /dev/null @@ -1,30 +0,0 @@ -# begin_generated_IBM_copyright_prolog -# -# This is an automatically generated copyright prolog. -# After initializing, DO NOT MODIFY OR MOVE -# --------------------------------------------------------------- -# Licensed Materials - Property of IBM -# Blue Gene/Q 5765-PER 5765-PRP -# -# (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved -# US Government Users Restricted Rights - -# Use, duplication, or disclosure restricted -# by GSA ADP Schedule Contract with IBM Corp. -# -# --------------------------------------------------------------- -# -# end_generated_IBM_copyright_prolog -# -*- mode: makefile-gmake; -*- - -# note that the includes always happen but the effects of their contents are -# affected by "if BUILD_PAMID" -if BUILD_PAMID - - -mpi_core_sources += \ - src/mpid/pamid/src/coll/alltoallv/mpido_alltoallv.c \ - src/mpid/pamid/src/coll/alltoallv/mpido_ialltoallv.c - - -endif BUILD_PAMID - diff --git a/src/mpid/pamid/src/coll/alltoallv/mpido_alltoallv.c b/src/mpid/pamid/src/coll/alltoallv/mpido_alltoallv.c deleted file mode 100644 index a66fdea..0000000 --- a/src/mpid/pamid/src/coll/alltoallv/mpido_alltoallv.c +++ /dev/null @@ -1,734 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/coll/alltoallv/mpido_alltoallv.c - * \brief ??? - */ -/* #define TRACE_ON */ - -#include <mpidimpl.h> - -static void cb_alltoallv(void *ctxt, void *clientdata, pami_result_t err) -{ - int *active = (int *)clientdata; - TRACE_ERR("alltoallv callback enter, active: %d\n", (*active)); - MPIDI_Progress_signal(); - (*active)--; -} - - -int MPIDO_Alltoallv(const void *sendbuf, - const int *sendcounts, - const int *senddispls, - MPI_Datatype sendtype, - void *recvbuf, - const int *recvcounts, - const int *recvdispls, - MPI_Datatype recvtype, - MPIR_Comm *comm_ptr, - int *mpierrno) -{ -#ifndef HAVE_PAMI_IN_PLACE - if (sendbuf == MPI_IN_PLACE) - { - MPID_Abort (NULL, 0, 1, "'MPI_IN_PLACE' requries support for `PAMI_IN_PLACE`"); - return -1; - } -#endif - TRACE_ERR("Entering MPIDO_Alltoallv\n"); - volatile unsigned active = 1; - void *snd_noncontig_buff = NULL, *rcv_noncontig_buff = NULL; - void *sbuf = NULL, *rbuf = NULL; - int recvok=PAMI_SUCCESS, sendok=PAMI_SUCCESS; - int sndtypelen, rcvtypelen, snd_contig=0, rcv_contig=0; - MPIDU_Datatype*sdt, *rdt; - pami_type_t stype, rtype; - MPI_Aint sdt_true_lb, rdt_true_lb; - MPIDI_Post_coll_t alltoallv_post; - int tmp; - const int rank = comm_ptr->rank; -#if ASSERT_LEVEL==0 - /* We can't afford the tracing in ndebug/performance libraries */ - const unsigned verbose = 0; -#else - const unsigned verbose = (MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_ALL) && (rank == 0); -#endif - const struct MPIDI_Comm* const mpid = &(comm_ptr->mpid); - const int selected_type = mpid->user_selected_type[PAMI_XFER_ALLTOALLV_INT]; - const int size = comm_ptr->local_size; - int sendcontinuous , recvcontinuous=0; - size_t recv_size=0, send_size=0; - size_t totalrecvcount=0; - int *lrecvdispls = NULL; /* possible local displs calculated for noncontinous */ - int *lsenddispls = NULL;/* possible local displs calculated for noncontinous */ - int *lrecvcounts = NULL;/* possible local counts calculated for noncontinous */ - int *lsendcounts = NULL;/* possible local counts calculated for noncontinous */ - const int *precvdispls = recvdispls; /* pointer to displs to use as pami parmi */ - const int *psenddispls = senddispls; /* pointer to displs to use as pami parmi */ - const int *precvcounts = recvcounts; /* pointer to counts to use as pami parmi */ - const int *psendcounts = sendcounts; /* pointer to counts to use as pami parmi */ - int inplace = sendbuf == MPI_IN_PLACE? 1 : 0; - if(selected_type == MPID_COLL_USE_MPICH) - { - if(unlikely(verbose)) - fprintf(stderr,"Using MPICH alltoallv algorithm\n"); - return MPIR_Alltoallv(sendbuf, sendcounts, senddispls, sendtype, - recvbuf, recvcounts, recvdispls, recvtype, - comm_ptr, mpierrno); - } - - if(!inplace) - { - sendok = MPIDI_Datatype_to_pami(sendtype, &stype, -1, NULL, &tmp); - MPIDI_Datatype_get_info(1, sendtype, snd_contig, sndtypelen, sdt, sdt_true_lb); - sbuf = (char *)sendbuf + sdt_true_lb; - if(!snd_contig || (sendok != PAMI_SUCCESS)) - { - stype = PAMI_TYPE_UNSIGNED_CHAR; - size_t totalsendcount = sendcounts[0]; - sendcontinuous = senddispls[0] == 0? 1 : 0 ; - int i; - psenddispls = lsenddispls = MPL_malloc(size*sizeof(int)); - psendcounts = lsendcounts = MPL_malloc(size*sizeof(int)); - lsenddispls[0]= 0; - lsendcounts[0]= sndtypelen * sendcounts[0]; - for(i=1; i<size; ++i) - { - lsenddispls[i]= sndtypelen * totalsendcount; - totalsendcount += sendcounts[i]; - if(senddispls[i] != (senddispls[i-1] + (sendcounts[i-1]*sndtypelen))) - sendcontinuous = 0; - lsendcounts[i]= sndtypelen * sendcounts[i]; - } - send_size = sndtypelen * totalsendcount; - TRACE_ERR("Pack receive sndv_contig %zu, sendok %zd, totalsendcount %zu, sendcontinuous %zu, sndtypelen %zu, send_size %zu\n", - (size_t)snd_contig, (size_t)sendok, (size_t)totalsendcount, (size_t)sendcontinuous, (size_t) sndtypelen, (size_t)send_size); - snd_noncontig_buff = MPL_malloc(send_size); - sbuf = snd_noncontig_buff; - if(snd_noncontig_buff == NULL) - { - MPID_Abort(NULL, MPI_ERR_NO_SPACE, 1, - "Fatal: Cannot allocate pack buffer"); - } - if(sendcontinuous) - { - MPIR_Localcopy(sendbuf, totalsendcount, sendtype, - snd_noncontig_buff, send_size,MPI_CHAR); - } - else - { - size_t extent; - MPIDU_Datatype_get_extent_macro(sendtype,extent); - for(i=0; i<size; ++i) - { - char* scbuf = (char*)sendbuf + senddispls[i]*extent; - char* rcbuf = (char*)snd_noncontig_buff + psenddispls[i]; - MPIR_Localcopy(scbuf, sendcounts[i], sendtype, - rcbuf, psendcounts[i], MPI_CHAR); - TRACE_ERR("Pack send src extent %zu, displ[%zu]=%zu, count[%zu]=%zu buf[%zu]=%u\n", - (size_t)extent, (size_t)i,(size_t)senddispls[i],(size_t)i,(size_t)sendcounts[i],(size_t)senddispls[i], *(int*)scbuf); - TRACE_ERR("Pack send dest displ[%zu]=%zu, count[%zu]=%zu buf[%zu]=%u\n", - (size_t)i,(size_t)psenddispls[i],(size_t)i,(size_t)psendcounts[i],(size_t)psenddispls[i], *(int*)rcbuf); - } - } - } - } - MPIDI_Datatype_get_info(1, recvtype, rcv_contig, rcvtypelen, rdt, rdt_true_lb); - recvok = MPIDI_Datatype_to_pami(recvtype, &rtype, -1, NULL, &tmp); - rbuf = (char *)recvbuf + rdt_true_lb; - if(!rcv_contig || (recvok != PAMI_SUCCESS)) - { - rtype = PAMI_TYPE_UNSIGNED_CHAR; - totalrecvcount = recvcounts[0]; - recvcontinuous = recvdispls[0] == 0? 1 : 0 ; - int i; - precvdispls = lrecvdispls = MPL_malloc(size*sizeof(int)); - precvcounts = lrecvcounts = MPL_malloc(size*sizeof(int)); - lrecvdispls[0]= 0; - lrecvcounts[0]= rcvtypelen * recvcounts[0]; - for(i=1; i<size; ++i) - { - lrecvdispls[i]= rcvtypelen * totalrecvcount; - totalrecvcount += recvcounts[i]; - if(recvdispls[i] != (recvdispls[i-1] + (recvcounts[i-1]*rcvtypelen))) - recvcontinuous = 0; - lrecvcounts[i]= rcvtypelen * recvcounts[i]; - } - recv_size = rcvtypelen * totalrecvcount; - TRACE_ERR("Pack receive rcv_contig %zu, recvok %zd, totalrecvcount %zu, recvcontinuous %zu, rcvtypelen %zu, recv_size %zu\n", - (size_t)rcv_contig, (size_t)recvok, (size_t)totalrecvcount, (size_t)recvcontinuous,(size_t)rcvtypelen, (size_t)recv_size); - rcv_noncontig_buff = MPL_malloc(recv_size); - rbuf = rcv_noncontig_buff; - if(rcv_noncontig_buff == NULL) - { - MPID_Abort(NULL, MPI_ERR_NO_SPACE, 1, - "Fatal: Cannot allocate pack buffer"); - } - /* need to copy it now if it's used for the send buffer and then do not do in place */ - if(inplace) - { - inplace = 0; - stype = PAMI_TYPE_UNSIGNED_CHAR; - size_t totalsendcount = recvcounts[0]; - sendcontinuous = recvdispls[0] == 0? 1 : 0 ; - int i; - psenddispls = lsenddispls = MPL_malloc(size*sizeof(int)); - psendcounts = lsendcounts = MPL_malloc(size*sizeof(int)); - lsenddispls[0]= 0; - lsendcounts[0]= rcvtypelen * recvcounts[0]; - for(i=1; i<size; ++i) - { - lsenddispls[i]= rcvtypelen * totalsendcount; - totalsendcount += recvcounts[i]; - if(recvdispls[i] != (recvdispls[i-1] + (recvcounts[i-1]*rcvtypelen))) - sendcontinuous = 0; - lsendcounts[i]= rcvtypelen * recvcounts[i]; - } - send_size = rcvtypelen * totalsendcount; - TRACE_ERR("Pack MPI_IN_PLACE receive sndv_contig %zu, sendok %zd, totalsendcount %zu, sendcontinuous %zu, rcvtypelen %zu, send_size %zu\n", - (size_t)snd_contig, (size_t)sendok, (size_t)totalsendcount, (size_t)sendcontinuous, (size_t) rcvtypelen, (size_t)send_size); - snd_noncontig_buff = MPL_malloc(send_size); - sbuf = snd_noncontig_buff; - if(snd_noncontig_buff == NULL) - { - MPID_Abort(NULL, MPI_ERR_NO_SPACE, 1, - "Fatal: Cannot allocate pack buffer"); - } - if(sendcontinuous) - { - MPIR_Localcopy(recvbuf, totalsendcount, recvtype, - snd_noncontig_buff, send_size,MPI_CHAR); - } - else - { - size_t extent; - MPIDU_Datatype_get_extent_macro(recvtype,extent); - for(i=0; i<size; ++i) - { - char* scbuf = (char*)recvbuf + recvdispls[i]*extent; - char* rcbuf = (char*)snd_noncontig_buff + psenddispls[i]; - MPIR_Localcopy(scbuf, recvcounts[i], recvtype, - rcbuf, psendcounts[i], MPI_CHAR); - TRACE_ERR("Pack send src extent %zu, displ[%zu]=%zu, count[%zu]=%zu buf[%zu]=%u\n", - (size_t)extent, (size_t)i,(size_t)recvdispls[i],(size_t)i,(size_t)recvcounts[i],(size_t)recvdispls[i], *(int*)scbuf); - TRACE_ERR("Pack send dest extent %zu, displ[%zu]=%zu, count[%zu]=%zu buf[%zu]=%u\n", - (size_t)extent, (size_t)i,(size_t)psenddispls[i],(size_t)i,(size_t)psendcounts[i],(size_t)psenddispls[i], *(int*)rcbuf); - } - } - } - } - - - pami_xfer_t alltoallv; - pami_algorithm_t my_alltoallv; - const pami_metadata_t *my_md = (pami_metadata_t *)NULL; - int queryreq = 0; - - if(selected_type == MPID_COLL_OPTIMIZED) - { - TRACE_ERR("Optimized alltoallv was selected\n"); - my_alltoallv = mpid->opt_protocol[PAMI_XFER_ALLTOALLV_INT][0]; - my_md = &mpid->opt_protocol_md[PAMI_XFER_ALLTOALLV_INT][0]; - queryreq = mpid->must_query[PAMI_XFER_ALLTOALLV_INT][0]; - } - else - { /* is this purely an else? or do i need to check for some other selectedvar... */ - TRACE_ERR("Alltoallv specified by user\n"); - my_alltoallv = mpid->user_selected[PAMI_XFER_ALLTOALLV_INT]; - my_md = &mpid->user_metadata[PAMI_XFER_ALLTOALLV_INT]; - queryreq = selected_type; - } - alltoallv.algorithm = my_alltoallv; - char *pname = my_md->name; - - - alltoallv.cb_done = cb_alltoallv; - alltoallv.cookie = (void *)&active; - /* We won't bother with alltoallv since MPI is always going to be ints. */ - if(inplace) - { - if(unlikely(verbose)) - fprintf(stderr,"alltoallv MPI_IN_PLACE buffering\n"); - alltoallv.cmd.xfer_alltoallv_int.stype = rtype; - alltoallv.cmd.xfer_alltoallv_int.sdispls = (int *) precvdispls; - alltoallv.cmd.xfer_alltoallv_int.stypecounts = (int *) precvcounts; - alltoallv.cmd.xfer_alltoallv_int.sndbuf = PAMI_IN_PLACE; - } - else - { - alltoallv.cmd.xfer_alltoallv_int.stype = stype; - alltoallv.cmd.xfer_alltoallv_int.sdispls = (int *) psenddispls; - alltoallv.cmd.xfer_alltoallv_int.stypecounts = (int *) psendcounts; - alltoallv.cmd.xfer_alltoallv_int.sndbuf = sbuf; - } - alltoallv.cmd.xfer_alltoallv_int.rcvbuf = rbuf; - - alltoallv.cmd.xfer_alltoallv_int.rdispls = (int *) precvdispls; - alltoallv.cmd.xfer_alltoallv_int.rtypecounts = (int *) precvcounts; - alltoallv.cmd.xfer_alltoallv_int.rtype = rtype; - - if(unlikely(queryreq == MPID_COLL_ALWAYS_QUERY || - queryreq == MPID_COLL_CHECK_FN_REQUIRED)) - { - metadata_result_t result = {0}; - TRACE_ERR("querying alltoallv protocol %s, type was %d\n", pname, queryreq); - if(my_md->check_fn == NULL) - { - /* process metadata bits */ - if((!my_md->check_correct.values.inplace) && (sendbuf == MPI_IN_PLACE)) - result.check.unspecified = 1; -/* Can't check ranges like this. Non-local. Comment out for now. - if(my_md->check_correct.values.rangeminmax) - { - MPI_Aint data_true_lb; - MPIDU_Datatype*data_ptr; - int data_size, data_contig; - MPIDI_Datatype_get_info(??, sendtype, data_contig, data_size, data_ptr, data_true_lb); - if((my_md->range_lo <= data_size) && - (my_md->range_hi >= data_size)) - ; - else - { - result.check.range = 1; - if(unlikely(verbose)) - { - fprintf(stderr,"message size (%u) outside range (%zu<->%zu) for %s.\n", - data_size, - my_md->range_lo, - my_md->range_hi, - my_md->name); - } - } - } -*/ - } - else /* calling the check fn is sufficient */ - result = my_md->check_fn(&alltoallv); - TRACE_ERR("bitmask: %#X\n", result.bitmask); - result.check.nonlocal = 0; /* #warning REMOVE THIS WHEN IMPLEMENTED */ - if(result.bitmask) - { - if(unlikely(verbose)) - fprintf(stderr,"Query failed for %s. Using MPICH alltoallv\n", pname); - MPIDI_Update_last_algorithm(comm_ptr, "ALLTOALLV_MPICH"); - return MPIR_Alltoallv(sendbuf, sendcounts, senddispls, sendtype, - recvbuf, recvcounts, recvdispls, recvtype, - comm_ptr, mpierrno); - } - if(my_md->check_correct.values.asyncflowctl && !(--(comm_ptr->mpid.num_requests))) - { - comm_ptr->mpid.num_requests = MPIDI_Process.optimized.num_requests; - int tmpmpierrno; - if(unlikely(verbose)) - fprintf(stderr,"Query barrier required for %s\n", pname); - MPIDO_Barrier(comm_ptr, &tmpmpierrno); - } - } - - if(unlikely(verbose)) - { - unsigned long long int threadID; - MPL_thread_id_t tid; - MPL_thread_self(&tid); - threadID = (unsigned long long int)tid; - fprintf(stderr,"<%llx> Using protocol %s for alltoallv on %u\n", - threadID, - my_md->name, - (unsigned) comm_ptr->context_id); - } - - MPIDI_Context_post(MPIDI_Context[0], &alltoallv_post.state, - MPIDI_Pami_post_wrapper, (void *)&alltoallv); - - TRACE_ERR("%d waiting on active %d\n", rank, active); - MPID_PROGRESS_WAIT_WHILE(active); - - if(!rcv_contig || (recvok != PAMI_SUCCESS)) - { - if(recvcontinuous) - { - MPIR_Localcopy(rcv_noncontig_buff, recv_size,MPI_CHAR, - recvbuf, totalrecvcount, recvtype); - } - else - { - size_t extent; - int i; - MPIDU_Datatype_get_extent_macro(recvtype,extent); - for(i=0; i<size; ++i) - { - char* scbuf = (char*)rcv_noncontig_buff+ precvdispls[i]; - char* rcbuf = (char*)recvbuf + recvdispls[i]*extent; - MPIR_Localcopy(scbuf, precvcounts[i], MPI_CHAR, - rcbuf, recvcounts[i], recvtype); - TRACE_ERR("Pack recv src extent %zu, displ[%zu]=%zu, count[%zu]=%zu buf[%zu]=%u\n", - (size_t)extent, (size_t)i,(size_t)precvdispls[i],(size_t)i,(size_t)precvcounts[i],(size_t)precvdispls[i], *(int*)scbuf); - TRACE_ERR("Pack recv dest extent %zu, displ[%zu]=%zu, count[%zu]=%zu buf[%zu]=%u\n", - (size_t)extent, (size_t)i,(size_t)recvdispls[i],(size_t)i,(size_t)recvcounts[i],(size_t)recvdispls[i], *(int*)rcbuf); - } - } - MPL_free(rcv_noncontig_buff); - } - if(!snd_contig || (sendok != PAMI_SUCCESS)) MPL_free(snd_noncontig_buff); - if(lrecvdispls) MPL_free(lrecvdispls); - if(lsenddispls) MPL_free(lsenddispls); - if(lrecvcounts) MPL_free(lrecvcounts); - if(lsendcounts) MPL_free(lsendcounts); - - TRACE_ERR("Leaving alltoallv\n"); - - - return 0; -} - - -int MPIDO_Alltoallv_simple(const void *sendbuf, - const int *sendcounts, - const int *senddispls, - MPI_Datatype sendtype, - void *recvbuf, - const int *recvcounts, - const int *recvdispls, - MPI_Datatype recvtype, - MPIR_Comm *comm_ptr, - int *mpierrno) -{ -#ifndef HAVE_PAMI_IN_PLACE - if (sendbuf == MPI_IN_PLACE) - { - MPID_Abort (NULL, 0, 1, "'MPI_IN_PLACE' requries support for `PAMI_IN_PLACE`"); - return -1; - } -#endif - TRACE_ERR("Entering MPIDO_Alltoallv_optimized\n"); - volatile unsigned active = 1; - int sndtypelen, rcvtypelen, snd_contig = 1, rcv_contig = 1; - MPIDU_Datatype*sdt, *rdt; - pami_type_t stype = NULL, rtype; - MPI_Aint sdt_true_lb = 0, rdt_true_lb; - MPIDI_Post_coll_t alltoallv_post; - void *snd_noncontig_buff = NULL, *rcv_noncontig_buff = NULL; - void *sbuf = NULL, *rbuf = NULL; - int recvok=PAMI_SUCCESS, sendok=PAMI_SUCCESS; - int tmp; - const int rank = comm_ptr->rank; - const int size = comm_ptr->local_size; -#if ASSERT_LEVEL==0 - /* We can't afford the tracing in ndebug/performance libraries */ - const unsigned verbose = 0; -#else - const unsigned verbose = (MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_ALL) && (rank == 0); -#endif - - int sendcontinuous , recvcontinuous=0; - size_t recv_size=0, send_size=0; - size_t totalrecvcount=0; - int *lrecvdispls = NULL; /* possible local displs calculated for noncontinous */ - int *lsenddispls = NULL;/* possible local displs calculated for noncontinous */ - int *lrecvcounts = NULL;/* possible local counts calculated for noncontinous */ - int *lsendcounts = NULL;/* possible local counts calculated for noncontinous */ - const int *precvdispls = recvdispls; /* pointer to displs to use as pami parmi */ - const int *psenddispls = senddispls; /* pointer to displs to use as pami parmi */ - const int *precvcounts = recvcounts; /* pointer to counts to use as pami parmi */ - const int *psendcounts = sendcounts; /* pointer to counts to use as pami parmi */ - int inplace = sendbuf == MPI_IN_PLACE? 1 : 0; - - - - - const struct MPIDI_Comm* const mpid = &(comm_ptr->mpid); - /* We don't pack and unpack in alltoallv as we do in alltoall because alltoallv has - more overhead of book keeping for sendcounts/displs and recvcounts/displs. Since, - alltoallv is non-rooted and all tasks has type info, decision to punt to mpich - will be the same on all tasks */ - - - /* Check if collsel has MPICH algorithm as the best performing one, if so, call MPICH now w/o doing any conversions */ - MPIDI_Datatype_get_info(1, recvtype, rcv_contig, rcvtypelen, rdt, rdt_true_lb); - if(MPIDI_Pamix_collsel_advise != NULL && mpid->collsel_fast_query != NULL) - { - advisor_algorithm_t advisor_algorithms[1]; - int num_algorithms = MPIDI_Pamix_collsel_advise(mpid->collsel_fast_query, PAMI_XFER_ALLTOALLV_INT, rcvtypelen * recvcounts[0], advisor_algorithms, 1); - if(num_algorithms) - { - if(advisor_algorithms[0].algorithm_type == COLLSEL_EXTERNAL_ALGO) - { - return MPIR_Alltoallv(sendbuf, sendcounts, senddispls, sendtype, - recvbuf, recvcounts, recvdispls, recvtype, - comm_ptr, mpierrno); - } - else if(advisor_algorithms[0].metadata && advisor_algorithms[0].metadata->check_correct.values.asyncflowctl && !(--(comm_ptr->mpid.num_requests))) - { - comm_ptr->mpid.num_requests = MPIDI_Process.optimized.num_requests; - int tmpmpierrno; - if(unlikely(verbose)) - fprintf(stderr,"Query barrier required for %s\n", advisor_algorithms[0].metadata->name); - MPIDO_Barrier(comm_ptr, &tmpmpierrno); - } - } - } - - /* Now do checks on send data and datatypes (contig and contin) and do necessary conversions */ - if(!inplace) - { - sendok = MPIDI_Datatype_to_pami(sendtype, &stype, -1, NULL, &tmp); - MPIDI_Datatype_get_info(1, sendtype, snd_contig, sndtypelen, sdt, sdt_true_lb); - sbuf = (char *)sendbuf + sdt_true_lb; - if(!snd_contig || (sendok != PAMI_SUCCESS)) - { - stype = PAMI_TYPE_UNSIGNED_CHAR; - size_t totalsendcount = sendcounts[0]; - sendcontinuous = senddispls[0] == 0? 1 : 0 ; - int i; - psenddispls = lsenddispls = MPL_malloc(size*sizeof(int)); - psendcounts = lsendcounts = MPL_malloc(size*sizeof(int)); - lsenddispls[0]= 0; - lsendcounts[0]= sndtypelen * sendcounts[0]; - for(i=1; i<size; ++i) - { - lsenddispls[i]= sndtypelen * totalsendcount; - totalsendcount += sendcounts[i]; - if(senddispls[i] != (senddispls[i-1] + (sendcounts[i-1]))) - sendcontinuous = 0; - lsendcounts[i]= sndtypelen * sendcounts[i]; - } - send_size = sndtypelen * totalsendcount; - TRACE_ERR("Pack receive sndv_contig %zu, sendok %zd, totalsendcount %zu, sendcontinuous %zu, sndtypelen %zu, send_size %zu\n", - (size_t)snd_contig, (size_t)sendok, (size_t)totalsendcount, (size_t)sendcontinuous, (size_t) sndtypelen, (size_t)send_size); - snd_noncontig_buff = MPL_malloc(send_size); - sbuf = snd_noncontig_buff; - if(snd_noncontig_buff == NULL) - { - MPID_Abort(NULL, MPI_ERR_NO_SPACE, 1, - "Fatal: Cannot allocate pack buffer"); - } - if(sendcontinuous) - { - MPIR_Localcopy(sendbuf, totalsendcount, sendtype, - snd_noncontig_buff, send_size,MPI_CHAR); - } - else - { - size_t extent; - MPIDU_Datatype_get_extent_macro(sendtype,extent); - for(i=0; i<size; ++i) - { - char* scbuf = (char*)sendbuf + senddispls[i]*extent; - char* rcbuf = (char*)snd_noncontig_buff + psenddispls[i]; - MPIR_Localcopy(scbuf, sendcounts[i], sendtype, - rcbuf, psendcounts[i], MPI_CHAR); - TRACE_ERR("Pack send src extent %zu, displ[%zu]=%zu, count[%zu]=%zu buf[%zu]=%u\n", - (size_t)extent, (size_t)i,(size_t)senddispls[i],(size_t)i,(size_t)sendcounts[i],(size_t)senddispls[i], *(int*)scbuf); - TRACE_ERR("Pack send dest displ[%zu]=%zu, count[%zu]=%zu buf[%zu]=%u\n", - (size_t)i,(size_t)psenddispls[i],(size_t)i,(size_t)psendcounts[i],(size_t)psenddispls[i], *(int*)rcbuf); - } - } - } - } - - recvok = MPIDI_Datatype_to_pami(recvtype, &rtype, -1, NULL, &tmp); - rbuf = (char *)recvbuf + rdt_true_lb; - if(!rcv_contig || (recvok != PAMI_SUCCESS)) - { - rtype = PAMI_TYPE_UNSIGNED_CHAR; - totalrecvcount = recvcounts[0]; - recvcontinuous = recvdispls[0] == 0? 1 : 0 ; - int i; - precvdispls = lrecvdispls = MPL_malloc(size*sizeof(int)); - precvcounts = lrecvcounts = MPL_malloc(size*sizeof(int)); - lrecvdispls[0]= 0; - lrecvcounts[0]= rcvtypelen * recvcounts[0]; - for(i=1; i<size; ++i) - { - lrecvdispls[i]= rcvtypelen * totalrecvcount; - totalrecvcount += recvcounts[i]; - if(recvdispls[i] != (recvdispls[i-1] + (recvcounts[i-1]))) - recvcontinuous = 0; - lrecvcounts[i]= rcvtypelen * recvcounts[i]; - } - recv_size = rcvtypelen * totalrecvcount; - TRACE_ERR("Pack receive rcv_contig %zu, recvok %zd, totalrecvcount %zu, recvcontinuous %zu, rcvtypelen %zu, recv_size %zu\n", - (size_t)rcv_contig, (size_t)recvok, (size_t)totalrecvcount, (size_t)recvcontinuous,(size_t)rcvtypelen, (size_t)recv_size); - rcv_noncontig_buff = MPL_malloc(recv_size); - rbuf = rcv_noncontig_buff; - if(rcv_noncontig_buff == NULL) - { - MPID_Abort(NULL, MPI_ERR_NO_SPACE, 1, - "Fatal: Cannot allocate pack buffer"); - } - /* need to copy it now if it's used for the send buffer and then do not do in place */ - if(inplace) - { - inplace = 0; - stype = PAMI_TYPE_UNSIGNED_CHAR; - size_t totalsendcount = recvcounts[0]; - sendcontinuous = recvdispls[0] == 0? 1 : 0 ; - int i; - psenddispls = lsenddispls = MPL_malloc(size*sizeof(int)); - psendcounts = lsendcounts = MPL_malloc(size*sizeof(int)); - lsenddispls[0]= 0; - lsendcounts[0]= rcvtypelen * recvcounts[0]; - for(i=1; i<size; ++i) - { - lsenddispls[i]= rcvtypelen * totalsendcount; - totalsendcount += recvcounts[i]; - if(recvdispls[i] != (recvdispls[i-1] + (recvcounts[i-1]))) - sendcontinuous = 0; - lsendcounts[i]= rcvtypelen * recvcounts[i]; - } - send_size = rcvtypelen * totalsendcount; - TRACE_ERR("Pack MPI_IN_PLACE receive sndv_contig %zu, sendok %zd, totalsendcount %zu, sendcontinuous %zu, rcvtypelen %zu, send_size %zu\n", - (size_t)snd_contig, (size_t)sendok, (size_t)totalsendcount, (size_t)sendcontinuous, (size_t) rcvtypelen, (size_t)send_size); - snd_noncontig_buff = MPL_malloc(send_size); - sbuf = snd_noncontig_buff; - if(snd_noncontig_buff == NULL) - { - MPID_Abort(NULL, MPI_ERR_NO_SPACE, 1, - "Fatal: Cannot allocate pack buffer"); - } - if(sendcontinuous) - { - MPIR_Localcopy(recvbuf, totalsendcount, recvtype, - snd_noncontig_buff, send_size,MPI_CHAR); - } - else - { - size_t extent; - MPIDU_Datatype_get_extent_macro(recvtype,extent); - for(i=0; i<size; ++i) - { - char* scbuf = (char*)recvbuf + recvdispls[i]*extent; - char* rcbuf = (char*)snd_noncontig_buff + psenddispls[i]; - MPIR_Localcopy(scbuf, recvcounts[i], recvtype, - rcbuf, psendcounts[i], MPI_CHAR); - TRACE_ERR("Pack send src extent %zu, displ[%zu]=%zu, count[%zu]=%zu buf[%zu]=%u\n", - (size_t)extent, (size_t)i,(size_t)recvdispls[i],(size_t)i,(size_t)recvcounts[i],(size_t)recvdispls[i], *(int*)scbuf); - TRACE_ERR("Pack send dest extent %zu, displ[%zu]=%zu, count[%zu]=%zu buf[%zu]=%u\n", - (size_t)extent, (size_t)i,(size_t)psenddispls[i],(size_t)i,(size_t)psendcounts[i],(size_t)psenddispls[i], *(int*)rcbuf); - } - } - } - } - - - - pami_xfer_t alltoallv; - alltoallv.algorithm = mpid->coll_algorithm[PAMI_XFER_ALLTOALLV_INT][0][0]; - - alltoallv.cb_done = cb_alltoallv; - alltoallv.cookie = (void *)&active; - if(inplace) - { - alltoallv.cmd.xfer_alltoallv_int.stype = rtype; - alltoallv.cmd.xfer_alltoallv_int.sdispls = (int *) precvdispls; - alltoallv.cmd.xfer_alltoallv_int.stypecounts = (int *) precvcounts; - alltoallv.cmd.xfer_alltoallv_int.sndbuf = PAMI_IN_PLACE; - } - else - { - alltoallv.cmd.xfer_alltoallv_int.stype = stype; - alltoallv.cmd.xfer_alltoallv_int.sdispls = (int *) psenddispls; - alltoallv.cmd.xfer_alltoallv_int.stypecounts = (int *) psendcounts; - alltoallv.cmd.xfer_alltoallv_int.sndbuf = sbuf; - } - alltoallv.cmd.xfer_alltoallv_int.rcvbuf = rbuf; - - alltoallv.cmd.xfer_alltoallv_int.rdispls = (int *) precvdispls; - alltoallv.cmd.xfer_alltoallv_int.rtypecounts = (int *) precvcounts; - alltoallv.cmd.xfer_alltoallv_int.rtype = rtype; - - - MPIDI_Context_post(MPIDI_Context[0], &alltoallv_post.state, - MPIDI_Pami_post_wrapper, (void *)&alltoallv); - - TRACE_ERR("%d waiting on active %d\n", rank, active); - MPID_PROGRESS_WAIT_WHILE(active); - - if(!rcv_contig || (recvok != PAMI_SUCCESS)) - { - if(recvcontinuous) - { - MPIR_Localcopy(rcv_noncontig_buff, recv_size,MPI_CHAR, - recvbuf, totalrecvcount, recvtype); - } - else - { - size_t extent; - int i; - MPIDU_Datatype_get_extent_macro(recvtype,extent); - for(i=0; i<size; ++i) - { - char* scbuf = (char*)rcv_noncontig_buff+ precvdispls[i]; - char* rcbuf = (char*)recvbuf + recvdispls[i]*extent; - MPIR_Localcopy(scbuf, precvcounts[i], MPI_CHAR, - rcbuf, recvcounts[i], recvtype); - TRACE_ERR("Pack recv src extent %zu, displ[%zu]=%zu, count[%zu]=%zu buf[%zu]=%u\n", - (size_t)extent, (size_t)i,(size_t)precvdispls[i],(size_t)i,(size_t)precvcounts[i],(size_t)precvdispls[i], *(int*)scbuf); - TRACE_ERR("Pack recv dest extent %zu, displ[%zu]=%zu, count[%zu]=%zu buf[%zu]=%u\n", - (size_t)extent, (size_t)i,(size_t)recvdispls[i],(size_t)i,(size_t)recvcounts[i],(size_t)recvdispls[i], *(int*)rcbuf); - } - } - MPL_free(rcv_noncontig_buff); - } - if(!snd_contig || (sendok != PAMI_SUCCESS)) MPL_free(snd_noncontig_buff); - if(lrecvdispls) MPL_free(lrecvdispls); - if(lsenddispls) MPL_free(lsenddispls); - if(lrecvcounts) MPL_free(lrecvcounts); - if(lsendcounts) MPL_free(lsendcounts); - - - TRACE_ERR("Leaving alltoallv\n"); - - - return MPI_SUCCESS; -} - -int -MPIDO_CSWrapper_alltoallv(pami_xfer_t *alltoallv, - void *comm) -{ - int mpierrno = 0; - MPIR_Comm *comm_ptr = (MPIR_Comm*)comm; - MPI_Datatype sendtype, recvtype; - void *sbuf; - MPIDI_coll_check_in_place(alltoallv->cmd.xfer_alltoallv_int.sndbuf, &sbuf); - int rc = MPIDI_Dtpami_to_dtmpi( alltoallv->cmd.xfer_alltoallv_int.stype, - &sendtype, - NULL, - NULL); - if(rc == -1) return rc; - - rc = MPIDI_Dtpami_to_dtmpi( alltoallv->cmd.xfer_alltoallv_int.rtype, - &recvtype, - NULL, - NULL); - if(rc == -1) return rc; - - rc = MPIR_Alltoallv(sbuf, - alltoallv->cmd.xfer_alltoallv_int.stypecounts, - alltoallv->cmd.xfer_alltoallv_int.sdispls, sendtype, - alltoallv->cmd.xfer_alltoallv_int.rcvbuf, - alltoallv->cmd.xfer_alltoallv_int.rtypecounts, - alltoallv->cmd.xfer_alltoallv_int.rdispls, recvtype, - comm_ptr, &mpierrno); - if(alltoallv->cb_done && rc == 0) - alltoallv->cb_done(NULL, alltoallv->cookie, PAMI_SUCCESS); - return rc; - -} - diff --git a/src/mpid/pamid/src/coll/alltoallv/mpido_ialltoallv.c b/src/mpid/pamid/src/coll/alltoallv/mpido_ialltoallv.c deleted file mode 100644 index a296664..0000000 --- a/src/mpid/pamid/src/coll/alltoallv/mpido_ialltoallv.c +++ /dev/null @@ -1,79 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/coll/alltoallv/mpido_alltoallv.c - * \brief ??? - */ -/*#define TRACE_ON*/ - -#include <mpidimpl.h> - -int MPIDO_Ialltoallv(const void *sendbuf, - const int *sendcounts, - const int *senddispls, - MPI_Datatype sendtype, - void *recvbuf, - const int *recvcounts, - const int *recvdispls, - MPI_Datatype recvtype, - MPIR_Comm *comm_ptr, - MPIR_Request **request) -{ - if(comm_ptr->rank == 0) - TRACE_ERR("Entering MPIDO_Ialltoallv\n"); - - /*if (unlikely((data_size == 0) || (user_selected_type == MPID_COLL_USE_MPICH)))*/ - { - /* - * If the mpich mpir non-blocking collectives are enabled, return without - * first constructing the MPIR_Request. This signals to the - * MPIR_Ialltoalliv_impl() function to invoke the mpich nbc implementation - * of MPI_Ialltoallv(). - */ - if (MPIDI_Process.mpir_nbc != 0) - return 0; - - /* - * MPIR_* nbc implementation is not enabled. Fake a non-blocking - * MPIR_Ialltoallv() with a blocking MPIR_Alltoallv(). - */ - if(unlikely(MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_ALL && comm_ptr->rank == 0)) - fprintf(stderr,"Using MPICH blocking alltoallv algorithm\n"); - - int mpierrno = 0; - int rc = MPIR_Alltoallv_impl(sendbuf, sendcounts, senddispls, sendtype, - recvbuf, recvcounts, recvdispls, recvtype, - comm_ptr, &mpierrno); - - /* - * The blocking alltoallv has completed - create and complete a - * MPIR_Request object so the MPIR_Ialltoallv_impl() function does not - * perform an additional ialltoallv. - */ - MPIR_Request * mpid_request = MPID_Request_create_inline(); - mpid_request->kind = MPIR_REQUEST_KIND__COLL; - *request = mpid_request; - MPIDI_Request_complete_norelease_inline(mpid_request); - - return rc; - } - - - TRACE_ERR("Leaving ialltoallv\n"); - return 0; -} diff --git a/src/mpid/pamid/src/coll/alltoallw/Makefile.mk b/src/mpid/pamid/src/coll/alltoallw/Makefile.mk deleted file mode 100644 index 4393952..0000000 --- a/src/mpid/pamid/src/coll/alltoallw/Makefile.mk +++ /dev/null @@ -1,28 +0,0 @@ -# begin_generated_IBM_copyright_prolog -# -# This is an automatically generated copyright prolog. -# After initializing, DO NOT MODIFY OR MOVE -# --------------------------------------------------------------- -# Licensed Materials - Property of IBM -# Blue Gene/Q 5765-PER 5765-PRP -# -# (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved -# US Government Users Restricted Rights - -# Use, duplication, or disclosure restricted -# by GSA ADP Schedule Contract with IBM Corp. -# -# --------------------------------------------------------------- -# -# end_generated_IBM_copyright_prolog -# -*- mode: makefile-gmake; -*- - -# note that the includes always happen but the effects of their contents are -# affected by "if BUILD_PAMID" -if BUILD_PAMID - - -mpi_core_sources += \ - src/mpid/pamid/src/coll/alltoallw/mpido_ialltoallw.c - - -endif BUILD_PAMID diff --git a/src/mpid/pamid/src/coll/alltoallw/mpido_ialltoallw.c b/src/mpid/pamid/src/coll/alltoallw/mpido_ialltoallw.c deleted file mode 100644 index 2569bfa..0000000 --- a/src/mpid/pamid/src/coll/alltoallw/mpido_ialltoallw.c +++ /dev/null @@ -1,79 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/coll/alltoallv/mpido_alltoallw.c - * \brief ??? - */ -/*#define TRACE_ON*/ - -#include <mpidimpl.h> - -int MPIDO_Ialltoallw(const void *sendbuf, - const int *sendcounts, - const int *senddispls, - const MPI_Datatype *sendtypes, - void *recvbuf, - const int *recvcounts, - const int *recvdispls, - const MPI_Datatype *recvtypes, - MPIR_Comm *comm_ptr, - MPIR_Request **request) -{ - if(comm_ptr->rank == 0) - TRACE_ERR("Entering MPIDO_Ialltoallw\n"); - - /*if (unlikely((data_size == 0) || (user_selected_type == MPID_COLL_USE_MPICH)))*/ - { - /* - * If the mpich mpir non-blocking collectives are enabled, return without - * first constructing the MPIR_Request. This signals to the - * MPIR_Ialltoallw_impl() function to invoke the mpich nbc implementation - * of MPI_Ialltoallw(). - */ - if (MPIDI_Process.mpir_nbc != 0) - return 0; - - /* - * MPIR_* nbc implementation is not enabled. Fake a non-blocking - * MPIR_Ialltoallw() with a blocking MPIR_Alltoallw(). - */ - if(unlikely(MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_ALL && comm_ptr->rank == 0)) - fprintf(stderr,"Using MPICH blocking alltoallw algorithm\n"); - - int mpierrno = 0; - int rc = MPIR_Alltoallw_impl(sendbuf, sendcounts, senddispls, sendtypes, - recvbuf, recvcounts, recvdispls, recvtypes, - comm_ptr, &mpierrno); - - /* - * The blocking alltoallw has completed - create and complete a - * MPIR_Request object so the MPIR_Ialltoallw_impl() function does not - * perform an additional ialltoallw. - */ - MPIR_Request * mpid_request = MPID_Request_create_inline(); - mpid_request->kind = MPIR_REQUEST_KIND__COLL; - *request = mpid_request; - MPIDI_Request_complete_norelease_inline(mpid_request); - - return rc; - } - - - TRACE_ERR("Leaving ialltoallw\n"); - return 0; -} diff --git a/src/mpid/pamid/src/coll/barrier/Makefile.mk b/src/mpid/pamid/src/coll/barrier/Makefile.mk deleted file mode 100644 index 3d49921..0000000 --- a/src/mpid/pamid/src/coll/barrier/Makefile.mk +++ /dev/null @@ -1,30 +0,0 @@ -# begin_generated_IBM_copyright_prolog -# -# This is an automatically generated copyright prolog. -# After initializing, DO NOT MODIFY OR MOVE -# --------------------------------------------------------------- -# Licensed Materials - Property of IBM -# Blue Gene/Q 5765-PER 5765-PRP -# -# (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved -# US Government Users Restricted Rights - -# Use, duplication, or disclosure restricted -# by GSA ADP Schedule Contract with IBM Corp. -# -# --------------------------------------------------------------- -# -# end_generated_IBM_copyright_prolog -# -*- mode: makefile-gmake; -*- - -# note that the includes always happen but the effects of their contents are -# affected by "if BUILD_PAMID" -if BUILD_PAMID - - -mpi_core_sources += \ - src/mpid/pamid/src/coll/barrier/mpido_barrier.c \ - src/mpid/pamid/src/coll/barrier/mpido_ibarrier.c - - -endif BUILD_PAMID - diff --git a/src/mpid/pamid/src/coll/barrier/mpido_barrier.c b/src/mpid/pamid/src/coll/barrier/mpido_barrier.c deleted file mode 100644 index 9e91334..0000000 --- a/src/mpid/pamid/src/coll/barrier/mpido_barrier.c +++ /dev/null @@ -1,146 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/coll/barrier/mpido_barrier.c - * \brief ??? - */ - -/* #define TRACE_ON */ - -#include <mpidimpl.h> - -static void cb_barrier(void *ctxt, void *clientdata, pami_result_t err) -{ - int *active = (int *) clientdata; - TRACE_ERR("callback. enter: %d\n", (*active)); - MPIDI_Progress_signal(); - (*active)--; -} - -int MPIDO_Barrier(MPIR_Comm *comm_ptr, int *mpierrno) -{ - TRACE_ERR("Entering MPIDO_Barrier\n"); - volatile unsigned active=1; - MPIDI_Post_coll_t barrier_post; - pami_xfer_t barrier; - pami_algorithm_t my_barrier; - const pami_metadata_t *my_md = (pami_metadata_t *)NULL; - int queryreq = 0; - const struct MPIDI_Comm* const mpid = &(comm_ptr->mpid); - const int selected_type = mpid->user_selected_type[PAMI_XFER_BARRIER]; -#if ASSERT_LEVEL==0 - /* We can't afford the tracing in ndebug/performance libraries */ - const unsigned verbose = 0; -#else - const unsigned verbose = (MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_ALL) && (comm_ptr->rank == 0); -#endif - - if(unlikely(selected_type == MPID_COLL_USE_MPICH)) - { - if(unlikely(verbose)) - fprintf(stderr,"Using MPICH barrier\n"); - TRACE_ERR("Using MPICH Barrier\n"); - return MPIR_Barrier(comm_ptr, mpierrno); - } - - barrier.cb_done = cb_barrier; - barrier.cookie = (void *)&active; - if(likely(selected_type == MPID_COLL_OPTIMIZED)) - { - TRACE_ERR("Optimized barrier (%s) was pre-selected\n", mpid->opt_protocol_md[PAMI_XFER_BARRIER][0].name); - my_barrier = mpid->opt_protocol[PAMI_XFER_BARRIER][0]; - my_md = &mpid->opt_protocol_md[PAMI_XFER_BARRIER][0]; - queryreq = mpid->must_query[PAMI_XFER_BARRIER][0]; - } - else - { - TRACE_ERR("Barrier (%s) was specified by user\n", mpid->user_metadata[PAMI_XFER_BARRIER].name); - my_barrier = mpid->user_selected[PAMI_XFER_BARRIER]; - my_md = &mpid->user_metadata[PAMI_XFER_BARRIER]; - queryreq = selected_type; - } - - barrier.algorithm = my_barrier; - /* There is no support for query-required barrier protocols here */ - MPID_assert(queryreq != MPID_COLL_ALWAYS_QUERY); - MPID_assert(queryreq != MPID_COLL_CHECK_FN_REQUIRED); - - if(unlikely(verbose)) - { - unsigned long long int threadID; - MPL_thread_id_t tid; - MPL_thread_self(&tid); - threadID = (unsigned long long int)tid; - fprintf(stderr,"<%llx> Using protocol %s for barrier on %u\n", - threadID, - my_md->name, - (unsigned) comm_ptr->context_id); - } - MPIDI_Context_post(MPIDI_Context[0], &barrier_post.state, - MPIDI_Pami_post_wrapper, (void *)&barrier); - - TRACE_ERR("advance spinning\n"); - MPIDI_Update_last_algorithm(comm_ptr, my_md->name); - MPID_PROGRESS_WAIT_WHILE(active); - TRACE_ERR("exiting mpido_barrier\n"); - return 0; -} - - -int MPIDO_Barrier_simple(MPIR_Comm *comm_ptr, int *mpierrno) -{ - TRACE_ERR("Entering MPIDO_Barrier_optimized\n"); - volatile unsigned active=1; - MPIDI_Post_coll_t barrier_post; - pami_xfer_t barrier; - pami_algorithm_t my_barrier; - const pami_metadata_t *my_barrier_md; - const struct MPIDI_Comm* const mpid = &(comm_ptr->mpid); - - barrier.cb_done = cb_barrier; - barrier.cookie = (void *)&active; - my_barrier = mpid->coll_algorithm[PAMI_XFER_BARRIER][0][0]; - my_barrier_md = &mpid->coll_metadata[PAMI_XFER_BARRIER][0][0]; - barrier.algorithm = my_barrier; - - - TRACE_ERR("%s barrier\n", MPIDI_Process.context_post.active>0?"posting":"invoking"); - MPIDI_Context_post(MPIDI_Context[0], &barrier_post.state, - MPIDI_Pami_post_wrapper, (void *)&barrier); - TRACE_ERR("barrier %s rc: %d\n", MPIDI_Process.context_post.active>0?"posted":"invoked", rc); - - TRACE_ERR("advance spinning\n"); - MPIDI_Update_last_algorithm(comm_ptr, my_barrier_md->name); - MPID_PROGRESS_WAIT_WHILE(active); - TRACE_ERR("Exiting MPIDO_Barrier_optimized\n"); - return 0; -} - -int -MPIDO_CSWrapper_barrier(pami_xfer_t *barrier, - void *comm) -{ - int mpierrno = 0; - MPIR_Comm *comm_ptr = (MPIR_Comm*)comm; - int rc = MPIR_Barrier(comm_ptr, &mpierrno); - if(barrier->cb_done && rc == 0) - barrier->cb_done(NULL, barrier->cookie, PAMI_SUCCESS); - return rc; - -} - diff --git a/src/mpid/pamid/src/coll/barrier/mpido_ibarrier.c b/src/mpid/pamid/src/coll/barrier/mpido_ibarrier.c deleted file mode 100644 index eb093fa..0000000 --- a/src/mpid/pamid/src/coll/barrier/mpido_ibarrier.c +++ /dev/null @@ -1,62 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/coll/barrier/mpido_ibarrier.c - * \brief ??? - */ - -/*#define TRACE_ON*/ - -#include <mpidimpl.h> - -static void cb_ibarrier(void *ctxt, void *clientdata, pami_result_t err) -{ - MPIR_Request *mpid_request = (MPIR_Request *) clientdata; - MPIDI_Request_complete_norelease_inline(mpid_request); -} - -int MPIDO_Ibarrier(MPIR_Comm *comm_ptr, MPIR_Request **request) -{ - TRACE_ERR("Entering MPIDO_Ibarrier\n"); - - /* - * There is actually no current pami optimization for this - * so just kick it back to MPICH if mpir_nbc is set, otherwise - * call the blocking MPIR_Barrier(). - */ - if (MPIDI_Process.mpir_nbc != 0) - return 0; - - /* - * MPIR_* nbc implementation is not enabled. Fake a non-blocking - * MPIR_Ibarrier() with a blocking MPIR_Barrier(). - */ - if(unlikely(MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_ALL && comm_ptr->rank == 0)) - fprintf(stderr,"Using MPICH barrier\n"); - TRACE_ERR("Using MPICH Barrier\n"); - - int mpierrno = 0; - int rc = MPIR_Barrier(comm_ptr, &mpierrno); - - MPIR_Request * mpid_request = MPID_Request_create_inline(); - mpid_request->kind = MPIR_REQUEST_KIND__COLL; - *request = mpid_request; - MPIDI_Request_complete_norelease_inline(mpid_request); - - return rc; -} diff --git a/src/mpid/pamid/src/coll/bcast/Makefile.mk b/src/mpid/pamid/src/coll/bcast/Makefile.mk deleted file mode 100644 index 3988631..0000000 --- a/src/mpid/pamid/src/coll/bcast/Makefile.mk +++ /dev/null @@ -1,30 +0,0 @@ -# begin_generated_IBM_copyright_prolog -# -# This is an automatically generated copyright prolog. -# After initializing, DO NOT MODIFY OR MOVE -# --------------------------------------------------------------- -# Licensed Materials - Property of IBM -# Blue Gene/Q 5765-PER 5765-PRP -# -# (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved -# US Government Users Restricted Rights - -# Use, duplication, or disclosure restricted -# by GSA ADP Schedule Contract with IBM Corp. -# -# --------------------------------------------------------------- -# -# end_generated_IBM_copyright_prolog -# -*- mode: makefile-gmake; -*- - -# note that the includes always happen but the effects of their contents are -# affected by "if BUILD_PAMID" -if BUILD_PAMID - - -mpi_core_sources += \ - src/mpid/pamid/src/coll/bcast/mpido_bcast.c \ - src/mpid/pamid/src/coll/bcast/mpido_ibcast.c - - -endif BUILD_PAMID - diff --git a/src/mpid/pamid/src/coll/bcast/mpido_bcast.c b/src/mpid/pamid/src/coll/bcast/mpido_bcast.c deleted file mode 100644 index d10a916..0000000 --- a/src/mpid/pamid/src/coll/bcast/mpido_bcast.c +++ /dev/null @@ -1,394 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/coll/bcast/mpido_bcast.c - * \brief ??? - */ - -/* #define TRACE_ON */ - -#include <mpidimpl.h> - -static void cb_bcast(void *ctxt, void *clientdata, pami_result_t err) -{ - int *active = (int *) clientdata; - TRACE_ERR("callback enter, active: %d\n", (*active)); - MPIDI_Progress_signal(); - (*active)--; -} - -int MPIDO_Bcast(void *buffer, - int count, - MPI_Datatype datatype, - int root, - MPIR_Comm *comm_ptr, - int *mpierrno) -{ - TRACE_ERR("in mpido_bcast\n"); - const size_t BCAST_LIMIT = 0x40000000; - int data_contig, rc; - void *data_buffer = NULL, - *noncontig_buff = NULL; - volatile unsigned active = 1; - MPI_Aint data_true_lb = 0; - MPIDU_Datatype*data_ptr; - MPIDU_Segment segment; - MPIDI_Post_coll_t bcast_post; - const struct MPIDI_Comm* const mpid = &(comm_ptr->mpid); - const int rank = comm_ptr->rank; -#if ASSERT_LEVEL==0 - /* We can't afford the tracing in ndebug/performance libraries */ - const unsigned verbose = 0; -#else - const unsigned verbose = (MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_ALL) && (rank == 0); -#endif - const int selected_type = mpid->user_selected_type[PAMI_XFER_BROADCAST]; - - /* Must calculate data_size based on count=1 in case it's total size is > integer */ - int data_size_one; - MPIDI_Datatype_get_info(1, datatype, - data_contig, data_size_one, data_ptr, data_true_lb); - /* do this calculation once and use twice */ - const size_t data_size_sz = (size_t)data_size_one*(size_t)count; - if(unlikely(verbose)) - fprintf(stderr,"bcast count %d, size %d (%#zX), root %d, buffer %p\n", - count,data_size_one, (size_t)data_size_one*(size_t)count, root,buffer); - if(unlikely( data_size_sz > BCAST_LIMIT) ) - { - void *new_buffer=buffer; - int c, new_count = (int)BCAST_LIMIT/data_size_one; - MPID_assert(new_count > 0); - - for(c=1; ((size_t)c*(size_t)new_count) <= (size_t)count; ++c) - { - if ((rc = MPIDO_Bcast(new_buffer, - new_count, - datatype, - root, - comm_ptr, - mpierrno)) != MPI_SUCCESS) - return rc; - new_buffer = (char*)new_buffer + (size_t)data_size_one*(size_t)new_count; - } - new_count = count % new_count; /* 0 is ok, just returns no-op */ - return MPIDO_Bcast(new_buffer, - new_count, - datatype, - root, - comm_ptr, - mpierrno); - } - - /* Must use data_size based on count for byte bcast processing. - Previously calculated as a size_t but large data_sizes were - handled above so this cast to int should be fine here. - */ - const int data_size = (int)data_size_sz; - - if(selected_type == MPID_COLL_USE_MPICH || data_size == 0) - { - if(unlikely(verbose)) - fprintf(stderr,"Using MPICH bcast algorithm\n"); - MPIDI_Update_last_algorithm(comm_ptr,"BCAST_MPICH"); - return MPIR_Bcast_intra(buffer, count, datatype, root, comm_ptr, mpierrno); - } - - data_buffer = (char *)buffer + data_true_lb; - - if(!data_contig) - { - noncontig_buff = MPL_malloc(data_size); - data_buffer = noncontig_buff; - if(noncontig_buff == NULL) - { - MPID_Abort(NULL, MPI_ERR_NO_SPACE, 1, - "Fatal: Cannot allocate pack buffer"); - } - if(rank == root) - { - DLOOP_Offset last = data_size; - MPIDU_Segment_init(buffer, count, datatype, &segment, 0); - MPIDU_Segment_pack(&segment, 0, &last, noncontig_buff); - } - } - - pami_xfer_t bcast; - pami_algorithm_t my_bcast; - const pami_metadata_t *my_md = (pami_metadata_t *)NULL; - int queryreq = 0; - - bcast.cb_done = cb_bcast; - bcast.cookie = (void *)&active; - bcast.cmd.xfer_broadcast.root = MPIDI_Task_to_endpoint(MPID_VCR_GET_LPID(comm_ptr->vcr, root), 0); - bcast.algorithm = mpid->user_selected[PAMI_XFER_BROADCAST]; - bcast.cmd.xfer_broadcast.buf = data_buffer; - bcast.cmd.xfer_broadcast.type = PAMI_TYPE_BYTE; - /* Needs to be sizeof(type)*count since we are using bytes as * the generic type */ - bcast.cmd.xfer_broadcast.typecount = data_size; - - if(selected_type == MPID_COLL_OPTIMIZED) - { - TRACE_ERR("Optimized bcast (%s) and (%s) were pre-selected\n", - mpid->opt_protocol_md[PAMI_XFER_BROADCAST][0].name, - mpid->opt_protocol_md[PAMI_XFER_BROADCAST][1].name); - - if(mpid->cutoff_size[PAMI_XFER_BROADCAST][1] != 0)/* SSS: There is FCA cutoff (FCA only sets cutoff for [PAMI_XFER_BROADCAST][1]) */ - { - if(data_size <= mpid->cutoff_size[PAMI_XFER_BROADCAST][1]) - { - my_bcast = mpid->opt_protocol[PAMI_XFER_BROADCAST][1]; - my_md = &mpid->opt_protocol_md[PAMI_XFER_BROADCAST][1]; - queryreq = mpid->must_query[PAMI_XFER_BROADCAST][1]; - } - else - { - return MPIR_Bcast_intra(buffer, count, datatype, root, comm_ptr, mpierrno); - } - } - - if(data_size > mpid->cutoff_size[PAMI_XFER_BROADCAST][0]) - { - my_bcast = mpid->opt_protocol[PAMI_XFER_BROADCAST][1]; - my_md = &mpid->opt_protocol_md[PAMI_XFER_BROADCAST][1]; - queryreq = mpid->must_query[PAMI_XFER_BROADCAST][1]; - } - else - { - my_bcast = mpid->opt_protocol[PAMI_XFER_BROADCAST][0]; - my_md = &mpid->opt_protocol_md[PAMI_XFER_BROADCAST][0]; - queryreq = mpid->must_query[PAMI_XFER_BROADCAST][0]; - } - } - else - { - TRACE_ERR("Bcast (%s) was specified by user\n", - mpid->user_metadata[PAMI_XFER_BROADCAST].name); - my_bcast = mpid->user_selected[PAMI_XFER_BROADCAST]; - my_md = &mpid->user_metadata[PAMI_XFER_BROADCAST]; - queryreq = selected_type; - } - - bcast.algorithm = my_bcast; - - if(unlikely(queryreq == MPID_COLL_ALWAYS_QUERY || - queryreq == MPID_COLL_CHECK_FN_REQUIRED)) - { - metadata_result_t result = {0}; - TRACE_ERR("querying bcast protocol %s, type was: %d\n", - my_md->name, queryreq); - if(my_md->check_fn != NULL) /* calling the check fn is sufficient */ - { - result = my_md->check_fn(&bcast); - result.check.nonlocal = 0; /* #warning REMOVE THIS WHEN IMPLEMENTED */ - } - else /* no check_fn, manually look at the metadata fields */ - { - TRACE_ERR("Optimzed selection line %d\n",__LINE__); - /* Check if the message range if restricted */ - if(my_md->check_correct.values.rangeminmax) - { - if((my_md->range_lo <= data_size) && - (my_md->range_hi >= data_size)) - ; /* ok, algorithm selected */ - else - { - result.check.range = 1; - if(unlikely(verbose)) - { - fprintf(stderr,"message size (%u) outside range (%zu<->%zu) for %s.\n", - data_size, - my_md->range_lo, - my_md->range_hi, - my_md->name); - } - } - } - /* \todo check the rest of the metadata */ - } - TRACE_ERR("bitmask: %#X\n", result.bitmask); - if(result.bitmask) - { - if(unlikely(verbose)) - fprintf(stderr,"Using MPICH bcast algorithm - query fn failed\n"); - MPIDI_Update_last_algorithm(comm_ptr,"BCAST_MPICH"); - return MPIR_Bcast_intra(buffer, count, datatype, root, comm_ptr, mpierrno); - } - if(my_md->check_correct.values.asyncflowctl && !(--(comm_ptr->mpid.num_requests))) - { - comm_ptr->mpid.num_requests = MPIDI_Process.optimized.num_requests; - int tmpmpierrno; - if(unlikely(verbose)) - fprintf(stderr,"Query barrier required for %s\n", my_md->name); - MPIDO_Barrier(comm_ptr, &tmpmpierrno); - } - } - - if(unlikely(verbose)) - { - unsigned long long int threadID; - MPL_thread_id_t tid; - MPL_thread_self(&tid); - threadID = (unsigned long long int)tid; - fprintf(stderr,"<%llx> Using protocol %s for bcast on %u\n", - threadID, - my_md->name, - (unsigned) comm_ptr->context_id); - } - - MPIDI_Context_post(MPIDI_Context[0], &bcast_post.state, MPIDI_Pami_post_wrapper, (void *)&bcast); - MPIDI_Update_last_algorithm(comm_ptr, my_md->name); - MPID_PROGRESS_WAIT_WHILE(active); - TRACE_ERR("bcast done\n"); - - if(!data_contig) - { - if(rank != root) - MPIR_Localcopy(noncontig_buff, data_size, MPI_CHAR, - buffer, count, datatype); - MPL_free(noncontig_buff); - } - - TRACE_ERR("leaving bcast\n"); - return 0; -} - - -int MPIDO_Bcast_simple(void *buffer, - int count, - MPI_Datatype datatype, - int root, - MPIR_Comm *comm_ptr, - int *mpierrno) -{ - TRACE_ERR("Entering MPIDO_Bcast_optimized\n"); - - int data_contig; - void *data_buffer = NULL, - *noncontig_buff = NULL; - volatile unsigned active = 1; - MPI_Aint data_true_lb = 0; - MPIDU_Datatype*data_ptr; - MPIDU_Segment segment; - MPIDI_Post_coll_t bcast_post; - const struct MPIDI_Comm* const mpid = &(comm_ptr->mpid); - const int rank = comm_ptr->rank; -#if ASSERT_LEVEL==0 - /* We can't afford the tracing in ndebug/performance libraries */ - const unsigned verbose = 0; -#else - const unsigned verbose = (MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_ALL) && (rank == 0); -#endif - - /* Must calculate data_size based on count=1 in case it's total size is > integer */ - int data_size_one; - MPIDI_Datatype_get_info(1, datatype, - data_contig, data_size_one, data_ptr, data_true_lb); - if(MPIDI_Pamix_collsel_advise != NULL && mpid->collsel_fast_query != NULL) - { - advisor_algorithm_t advisor_algorithms[1]; - int num_algorithms = MPIDI_Pamix_collsel_advise(mpid->collsel_fast_query, PAMI_XFER_BROADCAST, data_size_one * count, advisor_algorithms, 1); - if(num_algorithms) - { - if(advisor_algorithms[0].algorithm_type == COLLSEL_EXTERNAL_ALGO) - { - return MPIR_Bcast_intra(buffer, count, datatype, root, comm_ptr, mpierrno); - } - else if(advisor_algorithms[0].metadata && advisor_algorithms[0].metadata->check_correct.values.asyncflowctl && !(--(comm_ptr->mpid.num_requests))) - { - comm_ptr->mpid.num_requests = MPIDI_Process.optimized.num_requests; - int tmpmpierrno; - if(unlikely(verbose)) - fprintf(stderr,"Query barrier required for %s\n", advisor_algorithms[0].metadata->name); - MPIDO_Barrier(comm_ptr, &tmpmpierrno); - } - } - } - - const int data_size = data_size_one*(size_t)count; - - data_buffer = (char *)buffer + data_true_lb; - - if(!data_contig) - { - noncontig_buff = MPL_malloc(data_size); - data_buffer = noncontig_buff; - if(noncontig_buff == NULL) - { - MPID_Abort(NULL, MPI_ERR_NO_SPACE, 1, - "Fatal: Cannot allocate pack buffer"); - } - if(rank == root) - { - DLOOP_Offset last = data_size; - MPIDU_Segment_init(buffer, count, datatype, &segment, 0); - MPIDU_Segment_pack(&segment, 0, &last, noncontig_buff); - } - } - - pami_xfer_t bcast; - const pami_metadata_t *my_bcast_md; - - bcast.cb_done = cb_bcast; - bcast.cookie = (void *)&active; - bcast.cmd.xfer_broadcast.root = MPIDI_Task_to_endpoint(MPID_VCR_GET_LPID(comm_ptr->vcr, root), 0); - bcast.algorithm = mpid->coll_algorithm[PAMI_XFER_BROADCAST][0][0]; - bcast.cmd.xfer_broadcast.buf = data_buffer; - bcast.cmd.xfer_broadcast.type = PAMI_TYPE_BYTE; - /* Needs to be sizeof(type)*count since we are using bytes as * the generic type */ - bcast.cmd.xfer_broadcast.typecount = data_size; - my_bcast_md = &mpid->coll_metadata[PAMI_XFER_BROADCAST][0][0]; - - MPIDI_Context_post(MPIDI_Context[0], &bcast_post.state, MPIDI_Pami_post_wrapper, (void *)&bcast); - MPIDI_Update_last_algorithm(comm_ptr, my_bcast_md->name); - MPID_PROGRESS_WAIT_WHILE(active); - TRACE_ERR("bcast done\n"); - - if(!data_contig) - { - if(rank != root) - MPIR_Localcopy(noncontig_buff, data_size, MPI_CHAR, - buffer, count, datatype); - MPL_free(noncontig_buff); - } - - TRACE_ERR("Exiting MPIDO_Bcast_optimized\n"); - return 0; -} - - -int -MPIDO_CSWrapper_bcast(pami_xfer_t *bcast, - void *comm) -{ - int mpierrno = 0; - MPIR_Comm *comm_ptr = (MPIR_Comm*)comm; - MPI_Datatype type; - int rc = MPIDI_Dtpami_to_dtmpi( bcast->cmd.xfer_broadcast.type, - &type, - NULL, - NULL); - if(rc == -1) return rc; - - rc = MPIR_Bcast_intra(bcast->cmd.xfer_broadcast.buf, - bcast->cmd.xfer_broadcast.typecount, type, - bcast->cmd.xfer_broadcast.root, comm_ptr, &mpierrno); - if(bcast->cb_done && rc == 0) - bcast->cb_done(NULL, bcast->cookie, PAMI_SUCCESS); - return rc; -} - diff --git a/src/mpid/pamid/src/coll/bcast/mpido_ibcast.c b/src/mpid/pamid/src/coll/bcast/mpido_ibcast.c deleted file mode 100644 index 85134dd..0000000 --- a/src/mpid/pamid/src/coll/bcast/mpido_ibcast.c +++ /dev/null @@ -1,67 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/coll/bcast/mpido_ibcast.c - * \brief ??? - */ - -/*#define TRACE_ON*/ - -#include <mpidimpl.h> - -int MPIDO_Ibcast(void *buffer, - int count, - MPI_Datatype datatype, - int root, - MPIR_Comm *comm_ptr, - MPIR_Request **request) -{ - TRACE_ERR("in mpido_ibcast\n"); - - /* - * If the mpich mpir non-blocking collectives are enabled, return without - * first constructing the MPIR_Request. This signals to MPIR_Ibcast_impl - * to invoke the mpich nbc implementation of MPI_Ibcast(). - */ - if (MPIDI_Process.mpir_nbc != 0) - return 0; - - /* - * MPIR_* nbc implementation is not enabled. Fake a non-blocking - * MPIR_Ibcast() with a blocking MPIR_Bcast(). - */ - if(unlikely(MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_ALL && comm_ptr->rank == 0)) - fprintf(stderr,"Using MPICH bcast algorithm\n"); - - int mpierrno = 0; - int rc = MPIR_Bcast_intra(buffer, count, datatype, root, comm_ptr, &mpierrno); - - /* - * The blocking bcast has completed - create and complete a MPIR_Request - * object so the MPIR_Ibcast_impl() function does not perform the bcast. - */ - MPIR_Request * mpid_request = MPID_Request_create_inline(); - mpid_request->kind = MPIR_REQUEST_KIND__COLL; - *request = mpid_request; - MPIDI_Request_complete_norelease_inline(mpid_request); - - return rc; - - TRACE_ERR("leaving ibcast\n"); - return 0; -} diff --git a/src/mpid/pamid/src/coll/coll_utils.c b/src/mpid/pamid/src/coll/coll_utils.c deleted file mode 100644 index 50a3d55..0000000 --- a/src/mpid/pamid/src/coll/coll_utils.c +++ /dev/null @@ -1,405 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/coll/coll_utils.c - * \brief ??? - */ - -//#define TRACE_ON - -#include <mpidimpl.h> - -pami_result_t MPIDI_Pami_post_wrapper(pami_context_t context, void *cookie) -{ - TRACE_ERR("In post wrapper\n"); - TRACE_ERR("About to call collecive\n"); - PAMI_Collective(context, (pami_xfer_t *)cookie); - TRACE_ERR("Done calling collective\n"); - return PAMI_SUCCESS; -} - -/* some useful macros to make the comparisons less icky, esp given the */ -/* explosion of datatypes in MPI2.2 */ -#define isS_INT(x) ( (x)==MPI_INTEGER || \ - (x) == MPI_INT32_T || (x) == MPI_INTEGER4 || \ - (x) == MPI_INT) - -#define isUS_INT(x) ( (x) == MPI_UINT32_T || (x) == MPI_UNSIGNED) - -#define isS_SHORT(x) ( (x) == MPI_SHORT || (x) == MPI_INT16_T || \ - (x) == MPI_INTEGER2) - -#define isUS_SHORT(x) ( (x) == MPI_UNSIGNED_SHORT || (x) == MPI_UINT16_T) - -#define isS_CHAR(x) ( (x) == MPI_SIGNED_CHAR || (x) == MPI_INT8_T || \ - (x) == MPI_INTEGER1 || (x) == MPI_CHAR) - -#define isUS_CHAR(x) ( (x) == MPI_BYTE || \ - (x) == MPI_UNSIGNED_CHAR || (x) == MPI_UINT8_T ) - -/* sizeof(longlong) == sizeof(long) == sizeof(uint64) on bgq */ -#define isS_LONG(x) ( (x) == MPI_LONG || (x) == MPI_AINT ) - -#define isUS_LONG(x) ( (x) == MPI_UNSIGNED_LONG ) - -#define isS_LONG_LONG(x) ( (x) == MPI_INT64_T || (x) == MPI_OFFSET || \ - (x) == MPI_INTEGER8 || (x) == MPI_LONG_LONG || \ - (x) == MPI_LONG_LONG_INT ) - -#define isUS_LONG_LONG(x) ( (x) == MPI_UINT64_T || (x) == MPI_UNSIGNED_LONG_LONG ) - - - -#define isFLOAT(x) ( (x) == MPI_FLOAT || (x) == MPI_REAL) - -#define isDOUBLE(x) ( (x) == MPI_DOUBLE || (x) == MPI_DOUBLE_PRECISION) - -#define isLONG_DOUBLE(x) ( (x) == MPI_LONG_DOUBLE ) - -#define isLOC_TYPE(x) ( (x) == MPI_2REAL || (x) == MPI_2DOUBLE_PRECISION || \ - (x) == MPI_2INTEGER || (x) == MPI_FLOAT_INT || \ - (x) == MPI_DOUBLE_INT || (x) == MPI_LONG_INT || \ - (x) == MPI_2INT || (x) == MPI_SHORT_INT || \ - (x) == MPI_LONG_DOUBLE_INT ) - -#define isPAMI_LOC_TYPE(x) ( (x) == PAMI_TYPE_LOC_2INT || (x) == PAMI_TYPE_LOC_FLOAT_INT || \ - (x) == PAMI_TYPE_LOC_DOUBLE_INT || (x) == PAMI_TYPE_LOC_SHORT_INT || \ - (x) == PAMI_TYPE_LOC_LONG_INT || (x) == PAMI_TYPE_LOC_LONGDOUBLE_INT || \ - (x) == PAMI_TYPE_LOC_2FLOAT || (x) == PAMI_TYPE_LOC_2DOUBLE ) - -#define isBOOL(x) ( (x) == MPI_C_BOOL ) - -#define isLOGICAL(x) ( (x) == MPI_LOGICAL ) - -#define isSINGLE_COMPLEX(x) ( (x) == MPI_COMPLEX || (x) == MPI_C_FLOAT_COMPLEX) - -#define isDOUBLE_COMPLEX(x) ( (x) == MPI_DOUBLE_COMPLEX || (x) == MPI_COMPLEX8 ||\ - (x) == MPI_C_DOUBLE_COMPLEX) - -/* known missing types: MPI_C_LONG_DOUBLE_COMPLEX */ - - -#define MUSUPPORTED 1 -#define MUUNSUPPORTED 0 - -/* for easier debug */ -void MPIDI_Op_to_string(MPI_Op op, char *string) -{ - switch(op) - { - case MPI_SUM: strcpy(string, "MPI_SUM"); break; - case MPI_PROD: strcpy(string, "MPI_PROD"); break; - case MPI_LAND: strcpy(string, "MPI_LAND"); break; - case MPI_LOR: strcpy(string, "MPI_LOR"); break; - case MPI_LXOR: strcpy(string, "MPI_LXOR"); break; - case MPI_BAND: strcpy(string, "MPI_BAND"); break; - case MPI_BOR: strcpy(string, "MPI_BOR"); break; - case MPI_BXOR: strcpy(string, "MPI_BXOR"); break; - case MPI_MAX: strcpy(string, "MPI_MAX"); break; - case MPI_MIN: strcpy(string, "MPI_MIN"); break; - case MPI_MINLOC: strcpy(string, "MPI_MINLOC"); break; - case MPI_MAXLOC: strcpy(string, "MPI_MAXLOC"); break; - default: strcpy(string, "Other"); break; - } -} - - -int MPIDI_Datatype_to_pami(MPI_Datatype dt, - pami_type_t *pdt, - MPI_Op op, - pami_data_function *pop, - int *musupport) -{ - *musupport = MUSUPPORTED; - *pdt = PAMI_TYPE_NULL; - if(op != -1) - *pop = PAMI_DATA_NOOP; - if(isS_INT(dt)) *pdt = PAMI_TYPE_SIGNED_INT; - else if(isUS_INT(dt)) *pdt = PAMI_TYPE_UNSIGNED_INT; - else if(isFLOAT(dt)) *pdt = PAMI_TYPE_FLOAT; - else if(isDOUBLE(dt)) *pdt = PAMI_TYPE_DOUBLE; - else if(isLONG_DOUBLE(dt)) *pdt = PAMI_TYPE_LONG_DOUBLE; - else - { - *musupport = MUUNSUPPORTED; - if(isS_CHAR(dt)) *pdt = PAMI_TYPE_SIGNED_CHAR; - else if(isUS_CHAR(dt)) *pdt = PAMI_TYPE_UNSIGNED_CHAR; - else if(isS_SHORT(dt)) *pdt = PAMI_TYPE_SIGNED_SHORT; - else if(isUS_SHORT(dt)) *pdt = PAMI_TYPE_UNSIGNED_SHORT; - else if(isS_LONG(dt)) *pdt = PAMI_TYPE_SIGNED_LONG; - else if(isUS_LONG(dt)) *pdt = PAMI_TYPE_UNSIGNED_LONG; - else if(isS_LONG_LONG(dt)) *pdt = PAMI_TYPE_SIGNED_LONG_LONG; - else if(isUS_LONG_LONG(dt)) *pdt = PAMI_TYPE_UNSIGNED_LONG_LONG; - else if(isSINGLE_COMPLEX(dt)) *pdt = PAMI_TYPE_SINGLE_COMPLEX; - else if(isDOUBLE_COMPLEX(dt)) *pdt = PAMI_TYPE_DOUBLE_COMPLEX; - else if(isLOC_TYPE(dt)) - { - switch(dt) - { - case MPI_2INT: *pdt = PAMI_TYPE_LOC_2INT; break; - case MPI_FLOAT_INT: *pdt = PAMI_TYPE_LOC_FLOAT_INT; break; - case MPI_DOUBLE_INT: *pdt = PAMI_TYPE_LOC_DOUBLE_INT; break; - case MPI_SHORT_INT: *pdt = PAMI_TYPE_LOC_SHORT_INT; break; - - case MPI_LONG_INT: *pdt = PAMI_TYPE_LOC_LONG_INT; break; - case MPI_LONG_DOUBLE_INT: *pdt = PAMI_TYPE_LOC_LONGDOUBLE_INT; break; - default: - /* The following is needed to catch missing fortran bindings */ - if (dt == MPI_2REAL) - *pdt = PAMI_TYPE_LOC_2FLOAT; - else if (dt == MPI_2DOUBLE_PRECISION) - *pdt = PAMI_TYPE_LOC_2DOUBLE; - else if (dt == MPI_2INTEGER) - *pdt = PAMI_TYPE_LOC_2INT; - } - /* - * There are some 2-element types that PAMI doesn't support, so we - * need to bail on anything that's left of the LOC_TYPEs - */ - if(*pdt == PAMI_TYPE_NULL) return -1; - - if(op == -1) return MPI_SUCCESS; - if(op == MPI_MINLOC) - { - *pop = PAMI_DATA_MINLOC; - return MPI_SUCCESS; - } - if(op == MPI_MAXLOC) - { - *pop = PAMI_DATA_MAXLOC; - return MPI_SUCCESS; - } - if(op == MPI_REPLACE) - { - *pop = PAMI_DATA_COPY; - return MPI_SUCCESS; - } - else return -1; - } - else if(isLOGICAL(dt)) - { - *pdt = PAMI_TYPE_LOGICAL4; - if(op == -1) return MPI_SUCCESS; - if(op == MPI_LOR) - { - *pop = PAMI_DATA_LOR; - return MPI_SUCCESS; - } - if(op == MPI_LAND) - { - *pop = PAMI_DATA_LAND; - return MPI_SUCCESS; - } - if(op == MPI_LXOR) - { - *pop = PAMI_DATA_LXOR; - return MPI_SUCCESS; - } - if(op == MPI_REPLACE) - { - *pop = PAMI_DATA_COPY; - return MPI_SUCCESS; - } - return -1; - } - else if(isBOOL(dt)) - { - *pdt = PAMI_TYPE_LOGICAL1; - if(op == -1) return MPI_SUCCESS; - if(op == MPI_LOR) - { - *pop = PAMI_DATA_LOR; - return MPI_SUCCESS; - } - if(op == MPI_LAND) - { - *pop = PAMI_DATA_LAND; - return MPI_SUCCESS; - } - if(op == MPI_LXOR) - { - *pop = PAMI_DATA_LXOR; - return MPI_SUCCESS; - } - if(op == MPI_REPLACE) - { - *pop = PAMI_DATA_COPY; - return MPI_SUCCESS; - } - return -1; - } - } - - if(*pdt == PAMI_TYPE_NULL) return -1; - - if(op == -1) return MPI_SUCCESS; /* just doing DT conversion */ - - *pop = PAMI_DATA_NOOP; - switch(op) - { - case MPI_SUM: *pop = PAMI_DATA_SUM; return MPI_SUCCESS; break; - case MPI_PROD: *pop = PAMI_DATA_PROD; return MPI_SUCCESS; break; - case MPI_MAX: *pop = PAMI_DATA_MAX; return MPI_SUCCESS; break; - case MPI_MIN: *pop = PAMI_DATA_MIN; return MPI_SUCCESS; break; - case MPI_BAND: *pop = PAMI_DATA_BAND; return MPI_SUCCESS; break; - case MPI_BOR: *pop = PAMI_DATA_BOR; return MPI_SUCCESS; break; - case MPI_BXOR: *pop = PAMI_DATA_BXOR; return MPI_SUCCESS; break; - case MPI_LAND: - if(isLONG_DOUBLE(dt)) return -1; - *pop = PAMI_DATA_LAND; return MPI_SUCCESS; break; - case MPI_LOR: - if(isLONG_DOUBLE(dt)) return -1; - *pop = PAMI_DATA_LOR; return MPI_SUCCESS; break; - case MPI_LXOR: - if(isLONG_DOUBLE(dt)) return -1; - *pop = PAMI_DATA_LXOR; return MPI_SUCCESS; break; - case MPI_REPLACE: *pop = PAMI_DATA_COPY; return MPI_SUCCESS; break; - } - if(*pop == PAMI_DATA_NOOP) return -1; - - return MPI_SUCCESS; -} - -int MPIDI_Dtpami_to_dtmpi( pami_type_t pdt, - MPI_Datatype *dt, - pami_data_function pop, - MPI_Op *op) -{ - *dt = MPI_DATATYPE_NULL; - if(pop != NULL) - *op = MPI_OP_NULL; - - if(pdt == PAMI_TYPE_SIGNED_INT) *dt = MPI_INT; - else if(pdt == PAMI_TYPE_UNSIGNED_INT) *dt = MPI_UNSIGNED; - else if(pdt == PAMI_TYPE_FLOAT) *dt = MPI_FLOAT; - else if(pdt == PAMI_TYPE_DOUBLE) *dt = MPI_DOUBLE; - else if(pdt == PAMI_TYPE_LONG_DOUBLE) *dt = MPI_LONG_DOUBLE; - else if(pdt == PAMI_TYPE_SIGNED_CHAR) *dt = MPI_CHAR; - else if(pdt == PAMI_TYPE_UNSIGNED_CHAR) *dt = MPI_UNSIGNED_CHAR; - else if(pdt == PAMI_TYPE_BYTE) *dt = MPI_BYTE; - else if(pdt == PAMI_TYPE_SIGNED_SHORT) *dt = MPI_SHORT; - else if(pdt == PAMI_TYPE_UNSIGNED_SHORT) *dt = MPI_UNSIGNED_SHORT; - else if(pdt == PAMI_TYPE_SIGNED_LONG_LONG) *dt = MPI_LONG_LONG; - else if(pdt == PAMI_TYPE_UNSIGNED_LONG_LONG) *dt = MPI_UNSIGNED_LONG_LONG; - else if(pdt == PAMI_TYPE_SINGLE_COMPLEX) *dt = MPI_C_FLOAT_COMPLEX; - else if(pdt == PAMI_TYPE_DOUBLE_COMPLEX) *dt = MPI_C_DOUBLE_COMPLEX; - else if(isPAMI_LOC_TYPE(pdt)) - { - if(pdt == PAMI_TYPE_LOC_2INT) *dt = MPI_2INT; - else if(pdt == PAMI_TYPE_LOC_FLOAT_INT) *dt = MPI_FLOAT_INT; - else if(pdt == PAMI_TYPE_LOC_DOUBLE_INT) *dt = MPI_DOUBLE_INT; - else if(pdt == PAMI_TYPE_LOC_SHORT_INT) *dt = MPI_SHORT_INT; - else if(pdt == PAMI_TYPE_LOC_LONG_INT) *dt = MPI_LONG_INT; - else if(pdt == PAMI_TYPE_LOC_LONGDOUBLE_INT)*dt = MPI_LONG_DOUBLE_INT; - else if(pdt == PAMI_TYPE_LOC_2FLOAT) *dt = MPI_2REAL; - else if(pdt == PAMI_TYPE_LOC_2DOUBLE) *dt = MPI_2DOUBLE_PRECISION; - - if(pop == NULL) return MPI_SUCCESS; - if(pop == PAMI_DATA_MINLOC) - { - *op = MPI_MINLOC; - return MPI_SUCCESS; - } - if(pop == PAMI_DATA_MAXLOC) - { - *op = MPI_MAXLOC; - return MPI_SUCCESS; - } - if(pop == PAMI_DATA_COPY) - { - *op = MPI_REPLACE; - return MPI_SUCCESS; - } - else return -1; - } - else if(pdt == PAMI_TYPE_LOGICAL4) - { - *dt = MPI_LOGICAL; - if(pop == NULL) return MPI_SUCCESS; - if(pop == PAMI_DATA_LOR) - { - *op = MPI_LOR; - return MPI_SUCCESS; - } - if(pop == PAMI_DATA_LAND) - { - *op = MPI_LAND; - return MPI_SUCCESS; - } - if(pop == PAMI_DATA_LXOR) - { - *op = MPI_LXOR; - return MPI_SUCCESS; - } - if(pop == PAMI_DATA_COPY) - { - *op = MPI_REPLACE; - return MPI_SUCCESS; - } - return -1; - } - else if(pdt == PAMI_TYPE_LOGICAL1) - { - *dt = MPI_C_BOOL; - if(pop == NULL) return MPI_SUCCESS; - if(pop == PAMI_DATA_LOR) - { - *op = MPI_LOR; - return MPI_SUCCESS; - } - if(pop == PAMI_DATA_LAND) - { - *op = MPI_LAND; - return MPI_SUCCESS; - } - if(pop == PAMI_DATA_LXOR) - { - *op = MPI_LXOR; - return MPI_SUCCESS; - } - if(pop == PAMI_DATA_COPY) - { - *op = MPI_REPLACE; - return MPI_SUCCESS; - } - return -1; - } - - - if(*dt == MPI_DATATYPE_NULL) return -1; - - if(pop == NULL) return MPI_SUCCESS; /* just doing DT conversion */ - - *op = MPI_OP_NULL; - if(pop == PAMI_DATA_SUM) *op = MPI_SUM; - else if(pop ==PAMI_DATA_PROD) *op = MPI_PROD; - else if(pop ==PAMI_DATA_MAX) *op = MPI_MAX; - else if(pop ==PAMI_DATA_MIN) *op = MPI_MIN; - else if(pop ==PAMI_DATA_BAND) *op = MPI_BAND; - else if(pop ==PAMI_DATA_BOR) *op = MPI_BOR; - else if(pop ==PAMI_DATA_BXOR) *op = MPI_BXOR; - else if(pop ==PAMI_DATA_LAND) *op = MPI_LAND; - else if(pop ==PAMI_DATA_LOR) *op = MPI_LOR; - else if(pop ==PAMI_DATA_LXOR) *op = MPI_LXOR; - else if(pop ==PAMI_DATA_COPY) *op = MPI_REPLACE; - - if(*op == MPI_OP_NULL) return -1; - - return MPI_SUCCESS; -} - diff --git a/src/mpid/pamid/src/coll/exscan/Makefile.mk b/src/mpid/pamid/src/coll/exscan/Makefile.mk deleted file mode 100644 index 511bc9d..0000000 --- a/src/mpid/pamid/src/coll/exscan/Makefile.mk +++ /dev/null @@ -1,28 +0,0 @@ -# begin_generated_IBM_copyright_prolog -# -# This is an automatically generated copyright prolog. -# After initializing, DO NOT MODIFY OR MOVE -# --------------------------------------------------------------- -# Licensed Materials - Property of IBM -# Blue Gene/Q 5765-PER 5765-PRP -# -# (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved -# US Government Users Restricted Rights - -# Use, duplication, or disclosure restricted -# by GSA ADP Schedule Contract with IBM Corp. -# -# --------------------------------------------------------------- -# -# end_generated_IBM_copyright_prolog -# -*- mode: makefile-gmake; -*- - -# note that the includes always happen but the effects of their contents are -# affected by "if BUILD_PAMID" -if BUILD_PAMID - - -mpi_core_sources += \ - src/mpid/pamid/src/coll/exscan/mpido_iexscan.c - - -endif BUILD_PAMID diff --git a/src/mpid/pamid/src/coll/exscan/mpido_iexscan.c b/src/mpid/pamid/src/coll/exscan/mpido_iexscan.c deleted file mode 100644 index 2da2b08..0000000 --- a/src/mpid/pamid/src/coll/exscan/mpido_iexscan.c +++ /dev/null @@ -1,65 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/coll/exscan/mpido_iexscan.c - * \brief ??? - */ - -/*#define TRACE_ON */ -#include <mpidimpl.h> - -int MPIDO_Iexscan(const void *sendbuf, void *recvbuf, - int count, MPI_Datatype datatype, - MPI_Op op, MPIR_Comm * comm_ptr, MPIR_Request **request) -{ - /*if (unlikely((data_size == 0) || (user_selected_type == MPID_COLL_USE_MPICH)))*/ - { - /* - * If the mpich mpir non-blocking collectives are enabled, return without - * first constructing the MPIR_Request. This signals to the - * MPIR_Iexscan_impl() function to invoke the mpich nbc implementation - * of MPI_Iexscan(). - */ - if (MPIDI_Process.mpir_nbc != 0) - return 0; - - /* - * MPIR_* nbc implementation is not enabled. Fake a non-blocking - * MPIR_Iexscan() with a blocking MPIR_Exscan(). - */ - if(unlikely(MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_ALL && comm_ptr->rank == 0)) - fprintf(stderr,"Using MPICH blocking exscan algorithm\n"); - - int mpierrno = 0; - int rc = MPIR_Exscan_impl(sendbuf, recvbuf, count, datatype, op, comm_ptr, &mpierrno); - - /* - * The blocking exscan has completed - create and complete a - * MPIR_Request object so the MPIR_Iexscan_impl() function does not - * perform an additional iexscan. - */ - MPIR_Request * mpid_request = MPID_Request_create_inline(); - mpid_request->kind = MPIR_REQUEST_KIND__COLL; - *request = mpid_request; - MPIDI_Request_complete_norelease_inline(mpid_request); - - return rc; - } - - return 0; -} diff --git a/src/mpid/pamid/src/coll/gather/Makefile.mk b/src/mpid/pamid/src/coll/gather/Makefile.mk deleted file mode 100644 index 7b95dee..0000000 --- a/src/mpid/pamid/src/coll/gather/Makefile.mk +++ /dev/null @@ -1,30 +0,0 @@ -# begin_generated_IBM_copyright_prolog -# -# This is an automatically generated copyright prolog. -# After initializing, DO NOT MODIFY OR MOVE -# --------------------------------------------------------------- -# Licensed Materials - Property of IBM -# Blue Gene/Q 5765-PER 5765-PRP -# -# (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved -# US Government Users Restricted Rights - -# Use, duplication, or disclosure restricted -# by GSA ADP Schedule Contract with IBM Corp. -# -# --------------------------------------------------------------- -# -# end_generated_IBM_copyright_prolog -# -*- mode: makefile-gmake; -*- - -# note that the includes always happen but the effects of their contents are -# affected by "if BUILD_PAMID" -if BUILD_PAMID - - -mpi_core_sources += \ - src/mpid/pamid/src/coll/gather/mpido_gather.c \ - src/mpid/pamid/src/coll/gather/mpido_igather.c - - -endif BUILD_PAMID - diff --git a/src/mpid/pamid/src/coll/gather/mpido_gather.c b/src/mpid/pamid/src/coll/gather/mpido_gather.c deleted file mode 100644 index c99265d..0000000 --- a/src/mpid/pamid/src/coll/gather/mpido_gather.c +++ /dev/null @@ -1,642 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/coll/gather/mpido_gather.c - * \brief ??? - */ - -#include <mpidimpl.h> - -static void cb_gather(void *ctxt, void *clientdata, pami_result_t err) -{ - unsigned *active = (unsigned *)clientdata; - TRACE_ERR("cb_gather enter, active: %u\n", (*active)); - MPIDI_Progress_signal(); - (*active)--; -} - -static void cb_allred(void *ctxt, void *clientdata, pami_result_t err) -{ - unsigned *active = (unsigned *)clientdata; - TRACE_ERR("cb_allred preallred enter, active: %u\n", (*active)); - MPIDI_Progress_signal(); - (*active)--; -} -int MPIDO_Gather_reduce(void * sendbuf, - int sendcount, - MPI_Datatype sendtype, - void *recvbuf, - int recvcount, - MPI_Datatype recvtype, - int root, - MPIR_Comm * comm_ptr, - int *mpierrno) -{ - MPIDU_Datatype* data_ptr; - MPI_Aint true_lb ATTRIBUTE((unused)); - const int rank = comm_ptr->rank; - const int size = comm_ptr->local_size; - int rc, sbytes, rbytes; - int contig ATTRIBUTE((unused)); - char *tempbuf = NULL; - char *inplacetemp = NULL; - - MPIDI_Datatype_get_info(sendcount, sendtype, contig, - sbytes, data_ptr, true_lb); - - MPIDI_Datatype_get_info(recvcount, recvtype, contig, - rbytes, data_ptr, true_lb); - - if(rank == root) - { - tempbuf = recvbuf; - /* zero the buffer if we aren't in place */ - if(sendbuf != MPI_IN_PLACE) - { - memset(tempbuf, 0, sbytes * size * sizeof(char)); - memcpy(tempbuf+(rank * sbytes), sendbuf, sbytes); - } - /* copy our contribution temporarily, zero the buffer, put it back */ - /* this will likely suck */ - else - { - inplacetemp = MPL_malloc(rbytes * sizeof(char)); - memcpy(inplacetemp, recvbuf+(rank * rbytes), rbytes); - memset(tempbuf, 0, sbytes * size * sizeof(char)); - memcpy(tempbuf+(rank * rbytes), inplacetemp, rbytes); - MPL_free(inplacetemp); - } - } - /* everyone might need to speciifcally allocate a tempbuf, or - * we might need to make sure we don't end up at mpich in the - * mpido_() functions - seems to be a problem? - */ - - /* If we aren't root, malloc tempbuf and zero it, - * then copy our contribution to the right spot in the big buffer */ - else - { - tempbuf = MPL_malloc(sbytes * size * sizeof(char)); - if(!tempbuf) - return MPIR_Err_create_code(MPI_SUCCESS, - MPIR_ERR_RECOVERABLE, - __FUNCTION__, - __LINE__, - MPI_ERR_OTHER, - "**nomem", 0); - - memset(tempbuf, 0, sbytes * size * sizeof(char)); - memcpy(tempbuf+(rank*sbytes), sendbuf, sbytes); - } - /* Switch to comm->coll_fns->fn() */ - rc = MPIDO_Reduce(MPI_IN_PLACE, - tempbuf, - (sbytes * size)/4, - MPI_INT, - MPI_BOR, - root, - comm_ptr, - mpierrno); - - if(rank != root) - MPL_free(tempbuf); - - return rc; -} - -/* works for simple data types, assumes fast reduce is available */ - -int MPIDO_Gather(const void *sendbuf, - int sendcount, - MPI_Datatype sendtype, - void *recvbuf, - int recvcount, - MPI_Datatype recvtype, - int root, - MPIR_Comm *comm_ptr, - int *mpierrno) -{ -#ifndef HAVE_PAMI_IN_PLACE - if (sendbuf == MPI_IN_PLACE) - { - MPID_Abort (NULL, 0, 1, "'MPI_IN_PLACE' requries support for `PAMI_IN_PLACE`"); - return -1; - } -#endif - MPIDU_Datatype* data_ptr; - MPI_Aint true_lb ATTRIBUTE((unused)); - pami_xfer_t gather; - MPIDI_Post_coll_t gather_post; - int use_opt = 1, contig=0, send_bytes=-1, recv_bytes = 0; - const int rank = comm_ptr->rank; - const int size = comm_ptr->local_size; -#if ASSERT_LEVEL==0 - /* We can't afford the tracing in ndebug/performance libraries */ - const unsigned verbose = 0; -#else - const unsigned verbose = (MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_ALL) && (rank == 0); -#endif - const struct MPIDI_Comm* const mpid = &(comm_ptr->mpid); - const int selected_type = mpid->user_selected_type[PAMI_XFER_GATHER]; - - if (rank == root) - { - if (recvtype != MPI_DATATYPE_NULL && recvcount >= 0) - { - MPIDI_Datatype_get_info(recvcount, recvtype, contig, - recv_bytes, data_ptr, true_lb); - if (!contig || ((recv_bytes * size) % sizeof(int))) /* ? */ - use_opt = 0; - } - else - use_opt = 0; - } - - if ((sendbuf != MPI_IN_PLACE) && sendtype != MPI_DATATYPE_NULL && sendcount >= 0) - { - MPIDI_Datatype_get_info(sendcount, sendtype, contig, - send_bytes, data_ptr, true_lb); - if (!contig || ((send_bytes * size) % sizeof(int))) - use_opt = 0; - } - else - { - if(sendbuf == MPI_IN_PLACE) - send_bytes = recv_bytes; - if (sendtype == MPI_DATATYPE_NULL || sendcount == 0) - { - send_bytes = 0; - use_opt = 0; - } - } - - if(!mpid->optgather && - selected_type == MPID_COLL_USE_MPICH) - { - MPIDI_Update_last_algorithm(comm_ptr, "GATHER_MPICH"); - if(unlikely(verbose)) - fprintf(stderr,"Using MPICH gather algorithm (01) opt %x, selected type %d\n",mpid->optgather,selected_type); -#if CUDA_AWARE_SUPPORT - if(MPIDI_Process.cuda_aware_support_on) - { - MPI_Aint sdt_extent,rdt_extent; - MPIDU_Datatype_get_extent_macro(sendtype, sdt_extent); - MPIDU_Datatype_get_extent_macro(recvtype, rdt_extent); - char *scbuf = NULL; - char *rcbuf = NULL; - int is_send_dev_buf = MPIDI_cuda_is_device_buf(sendbuf); - int is_recv_dev_buf = (rank == root) ? MPIDI_cuda_is_device_buf(recvbuf) : 0; - if(is_send_dev_buf) - { - scbuf = MPL_malloc(sdt_extent * sendcount); - cudaError_t cudaerr = CudaMemcpy(scbuf, sendbuf, sdt_extent * sendcount, cudaMemcpyDeviceToHost); - if (cudaSuccess != cudaerr) - fprintf(stderr, "cudaMemcpy failed: %s\n", CudaGetErrorString(cudaerr)); - } - else - scbuf = sendbuf; - if(is_recv_dev_buf) - { - rcbuf = MPL_malloc(rdt_extent * recvcount); - if(sendbuf == MPI_IN_PLACE) - { - cudaError_t cudaerr = CudaMemcpy(rcbuf, recvbuf, rdt_extent * recvcount, cudaMemcpyDeviceToHost); - if (cudaSuccess != cudaerr) - fprintf(stderr, "cudaMemcpy failed: %s\n", CudaGetErrorString(cudaerr)); - } - else - memset(rcbuf, 0, rdt_extent * recvcount); - } - else - rcbuf = recvbuf; - int cuda_res = MPIR_Gather(scbuf, sendcount, sendtype, rcbuf, recvcount, recvtype, root, comm_ptr, mpierrno); - if(is_send_dev_buf)MPL_free(scbuf); - if(is_recv_dev_buf) - { - cudaError_t cudaerr = CudaMemcpy(recvbuf, rcbuf, rdt_extent * recvcount, cudaMemcpyHostToDevice); - if (cudaSuccess != cudaerr) - fprintf(stderr, "cudaMemcpy failed: %s\n", CudaGetErrorString(cudaerr)); - MPL_free(rcbuf); - } - return cuda_res; - } - else -#endif - return MPIR_Gather(sendbuf, sendcount, sendtype, - recvbuf, recvcount, recvtype, - root, comm_ptr, mpierrno); - } - if(mpid->preallreduces[MPID_GATHER_PREALLREDUCE]) - { - if(unlikely(verbose)) - fprintf(stderr,"MPID_GATHER_PREALLREDUCE opt %x, selected type %d, use_opt %d\n",mpid->optgather,selected_type, use_opt); - volatile unsigned allred_active = 1; - pami_xfer_t allred; - MPIDI_Post_coll_t allred_post; - allred.cb_done = cb_allred; - allred.cookie = (void *)&allred_active; - /* Guaranteed to work allreduce */ - allred.algorithm = mpid->coll_algorithm[PAMI_XFER_ALLREDUCE][0][0]; - allred.cmd.xfer_allreduce.sndbuf = (void *)PAMI_IN_PLACE; - allred.cmd.xfer_allreduce.stype = PAMI_TYPE_SIGNED_INT; - allred.cmd.xfer_allreduce.rcvbuf = (void *)&use_opt; - allred.cmd.xfer_allreduce.rtype = PAMI_TYPE_SIGNED_INT; - allred.cmd.xfer_allreduce.stypecount = 1; - allred.cmd.xfer_allreduce.rtypecount = 1; - allred.cmd.xfer_allreduce.op = PAMI_DATA_BAND; - - MPIDI_Context_post(MPIDI_Context[0], &allred_post.state, - MPIDI_Pami_post_wrapper, (void *)&allred); - MPID_PROGRESS_WAIT_WHILE(allred_active); - if(unlikely(verbose)) - fprintf(stderr,"MPID_GATHER_PREALLREDUCE opt %x, selected type %d, use_opt %d\n",mpid->optgather,selected_type, use_opt); - } - - if(mpid->optgather) - { - if(use_opt) - { - MPIDI_Update_last_algorithm(comm_ptr, "GLUE_REDUCDE"); - abort(); - /* GLUE_REDUCE ? */ - } - else - { - MPIDI_Update_last_algorithm(comm_ptr, "GATHER_MPICH"); - if(unlikely(verbose)) - fprintf(stderr,"Using MPICH gather algorithm (02) opt %x, selected type %d, use_opt %d\n",mpid->optgather,selected_type, use_opt); - return MPIR_Gather(sendbuf, sendcount, sendtype, - recvbuf, recvcount, recvtype, - root, comm_ptr, mpierrno); - } - } - - - pami_algorithm_t my_gather; - const pami_metadata_t *my_md = (pami_metadata_t *)NULL; - int queryreq = 0; - volatile unsigned active = 1; - - gather.cb_done = cb_gather; - gather.cookie = (void *)&active; - gather.cmd.xfer_gather.root = MPIDI_Task_to_endpoint(MPID_VCR_GET_LPID(comm_ptr->vcr, root), 0); - if(sendbuf == MPI_IN_PLACE) - { - if(unlikely(verbose)) - fprintf(stderr,"gather MPI_IN_PLACE buffering\n"); - gather.cmd.xfer_gather.stypecount = recv_bytes; - gather.cmd.xfer_gather.sndbuf = PAMI_IN_PLACE; - } - else - { - gather.cmd.xfer_gather.stypecount = send_bytes; - gather.cmd.xfer_gather.sndbuf = (void *)sendbuf; - } - gather.cmd.xfer_gather.stype = PAMI_TYPE_BYTE; - gather.cmd.xfer_gather.rcvbuf = (void *)recvbuf; - gather.cmd.xfer_gather.rtype = PAMI_TYPE_BYTE; - gather.cmd.xfer_gather.rtypecount = recv_bytes; - - /* If glue-level protocols are good, this will require some changes */ - if(selected_type == MPID_COLL_OPTIMIZED) - { - TRACE_ERR("Optimized gather (%s) was pre-selected\n", - mpid->opt_protocol_md[PAMI_XFER_GATHER][0].name); - my_gather = mpid->opt_protocol[PAMI_XFER_GATHER][0]; - my_md = &mpid->opt_protocol_md[PAMI_XFER_GATHER][0]; - queryreq = mpid->must_query[PAMI_XFER_GATHER][0]; - } - else - { - TRACE_ERR("Optimized gather (%s) was specified by user\n", - mpid->user_metadata[PAMI_XFER_GATHER].name); - my_gather = mpid->user_selected[PAMI_XFER_GATHER]; - my_md = &mpid->user_metadata[PAMI_XFER_GATHER]; - queryreq = selected_type; - } - - gather.algorithm = my_gather; - if(unlikely(queryreq == MPID_COLL_ALWAYS_QUERY || - queryreq == MPID_COLL_CHECK_FN_REQUIRED)) - { - metadata_result_t result = {0}; - TRACE_ERR("querying gather protocol %s, type was %d\n", - my_md->name, queryreq); - if(my_md->check_fn == NULL) - { - /* process metadata bits */ - if((!my_md->check_correct.values.inplace) && (sendbuf == MPI_IN_PLACE)) - result.check.unspecified = 1; - if(my_md->check_correct.values.rangeminmax) - { - /* Non-local decision? */ - if(((rank == root) && - (my_md->range_lo <= recv_bytes) && - (my_md->range_hi >= recv_bytes) - ) && - ((rank != root) && - (my_md->range_lo <= send_bytes) && - (my_md->range_hi >= send_bytes) - ) - ) - ; /* ok, algorithm selected */ - else - { - result.check.range = 1; - if(unlikely(verbose)) - { - fprintf(stderr,"message size (%u) outside range (%zu<->%zu) for %s.\n", - recv_bytes, - my_md->range_lo, - my_md->range_hi, - my_md->name); - } - } - } - } - else /* calling the check fn is sufficient */ - result = my_md->check_fn(&gather); - TRACE_ERR("bitmask: %#X\n", result.bitmask); - result.check.nonlocal = 0; /* #warning REMOVE THIS WHEN IMPLEMENTED */ - if(result.bitmask) - { - if(unlikely(verbose)) - fprintf(stderr,"query failed for %s\n", my_md->name); - MPIDI_Update_last_algorithm(comm_ptr, "GATHER_MPICH"); - return MPIR_Gather(sendbuf, sendcount, sendtype, - recvbuf, recvcount, recvtype, - root, comm_ptr, mpierrno); - } - if(my_md->check_correct.values.asyncflowctl && !(--(comm_ptr->mpid.num_requests))) - { - comm_ptr->mpid.num_requests = MPIDI_Process.optimized.num_requests; - int tmpmpierrno; - if(unlikely(verbose)) - fprintf(stderr,"Query barrier required for %s\n", my_md->name); - MPIDO_Barrier(comm_ptr, &tmpmpierrno); - } - } - - MPIDI_Update_last_algorithm(comm_ptr, - mpid->user_metadata[PAMI_XFER_GATHER].name); - - - if(unlikely(verbose)) - { - unsigned long long int threadID; - MPL_thread_id_t tid; - MPL_thread_self(&tid); - threadID = (unsigned long long int)tid; - fprintf(stderr,"<%llx> Using protocol %s for gather on %u\n", - threadID, - my_md->name, - (unsigned) comm_ptr->context_id); - } - - MPIDI_Context_post(MPIDI_Context[0], &gather_post.state, - MPIDI_Pami_post_wrapper, (void *)&gather); - - TRACE_ERR("Waiting on active: %d\n", active); - MPID_PROGRESS_WAIT_WHILE(active); - - TRACE_ERR("Leaving MPIDO_Gather\n"); - return 0; -} - - -int MPIDO_Gather_simple(const void *sendbuf, - int sendcount, - MPI_Datatype sendtype, - void *recvbuf, - int recvcount, - MPI_Datatype recvtype, - int root, - MPIR_Comm *comm_ptr, - int *mpierrno) -{ -#ifndef HAVE_PAMI_IN_PLACE - if (sendbuf == MPI_IN_PLACE) - { - MPID_Abort (NULL, 0, 1, "'MPI_IN_PLACE' requries support for `PAMI_IN_PLACE`"); - return -1; - } -#endif - MPIDU_Datatype* data_ptr; - MPI_Aint true_lb = 0; - pami_xfer_t gather; - MPIDI_Post_coll_t gather_post; - int success = 1, snd_contig = 1, rcv_contig = 1; - void *snd_noncontig_buff = NULL, *rcv_noncontig_buff = NULL; - void *sbuf = NULL, *rbuf = NULL; - int send_size = 0; - int recv_size = 0; - MPIDU_Segment segment; - const int rank = comm_ptr->rank; - const int size = comm_ptr->local_size; - const struct MPIDI_Comm* const mpid = &(comm_ptr->mpid); -#if ASSERT_LEVEL==0 - /* We can't afford the tracing in ndebug/performance libraries */ - const unsigned verbose = 0; -#else - const unsigned verbose = (MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_ALL) && (rank == 0); -#endif - - if(sendbuf != MPI_IN_PLACE) - { - MPIDI_Datatype_get_info(sendcount, sendtype, snd_contig, - send_size, data_ptr, true_lb); - if(MPIDI_Pamix_collsel_advise != NULL && mpid->collsel_fast_query != NULL) - { - advisor_algorithm_t advisor_algorithms[1]; - int num_algorithms = MPIDI_Pamix_collsel_advise(mpid->collsel_fast_query, PAMI_XFER_GATHER, send_size, advisor_algorithms, 1); - if(num_algorithms) - { - if(advisor_algorithms[0].algorithm_type == COLLSEL_EXTERNAL_ALGO) - { - return MPIR_Gather(sendbuf, sendcount, sendtype, - recvbuf, recvcount, recvtype, - root, comm_ptr, mpierrno); - } - else if(advisor_algorithms[0].metadata && advisor_algorithms[0].metadata->check_correct.values.asyncflowctl && !(--(comm_ptr->mpid.num_requests))) - { - comm_ptr->mpid.num_requests = MPIDI_Process.optimized.num_requests; - int tmpmpierrno; - if(unlikely(verbose)) - fprintf(stderr,"Query barrier required for %s\n", advisor_algorithms[0].metadata->name); - MPIDO_Barrier(comm_ptr, &tmpmpierrno); - } - } - } - - sbuf = (char *)sendbuf + true_lb; - if (!snd_contig) - { - snd_noncontig_buff = MPL_malloc(send_size); - sbuf = snd_noncontig_buff; - if(snd_noncontig_buff == NULL) - { - MPID_Abort(NULL, MPI_ERR_NO_SPACE, 1, - "Fatal: Cannot allocate pack buffer"); - } - DLOOP_Offset last = send_size; - MPIDU_Segment_init(sendbuf, sendcount, sendtype, &segment, 0); - MPIDU_Segment_pack(&segment, 0, &last, snd_noncontig_buff); - } - } - else - { - if(MPIDI_Pamix_collsel_advise != NULL && mpid->collsel_fast_query != NULL) - { - MPIDI_Datatype_get_info(recvcount, recvtype, rcv_contig, - recv_size, data_ptr, true_lb); - advisor_algorithm_t advisor_algorithms[1]; - int num_algorithms = MPIDI_Pamix_collsel_advise(mpid->collsel_fast_query, PAMI_XFER_GATHER, recv_size, advisor_algorithms, 1); - if(num_algorithms) - { - if(advisor_algorithms[0].algorithm_type == COLLSEL_EXTERNAL_ALGO) - { - return MPIR_Gather(sendbuf, sendcount, sendtype, - recvbuf, recvcount, recvtype, - root, comm_ptr, mpierrno); - } - else if(advisor_algorithms[0].metadata && advisor_algorithms[0].metadata->check_correct.values.asyncflowctl && !(--(comm_ptr->mpid.num_requests))) - { - comm_ptr->mpid.num_requests = MPIDI_Process.optimized.num_requests; - int tmpmpierrno; - if(unlikely(verbose)) - fprintf(stderr,"Query barrier required for %s\n", advisor_algorithms[0].metadata->name); - MPIDO_Barrier(comm_ptr, &tmpmpierrno); - } - - } - } - } - - rbuf = (char *)recvbuf + true_lb; - if (rank == root) - { - if (recvtype != MPI_DATATYPE_NULL && recvcount >= 0) - { - MPIDI_Datatype_get_info(recvcount, recvtype, rcv_contig, - recv_size, data_ptr, true_lb); - rbuf = (char *)recvbuf + true_lb; - if (!rcv_contig) - { - rcv_noncontig_buff = MPL_malloc(recv_size * size); - rbuf = rcv_noncontig_buff; - if(rcv_noncontig_buff == NULL) - { - MPID_Abort(NULL, MPI_ERR_NO_SPACE, 1, - "Fatal: Cannot allocate pack buffer"); - } - if(sendbuf == MPI_IN_PLACE) - { - size_t extent; - MPIDU_Datatype_get_extent_macro(recvtype,extent); - MPIR_Localcopy(recvbuf + (rank*recvcount*extent), recvcount, recvtype, - rcv_noncontig_buff + (rank*recv_size), recv_size,MPI_CHAR); - } - } - } - else - success = 0; - } - - if(!success) - { - MPIDI_Update_last_algorithm(comm_ptr, "GATHER_MPICH"); - return MPIR_Gather(sendbuf, sendcount, sendtype, - recvbuf, recvcount, recvtype, - root, comm_ptr, mpierrno); - } - - - const pami_metadata_t *my_gather_md; - volatile unsigned active = 1; - - gather.cb_done = cb_gather; - gather.cookie = (void *)&active; - gather.cmd.xfer_gather.root = MPIDI_Task_to_endpoint(MPID_VCR_GET_LPID(comm_ptr->vcr, root), 0); - gather.cmd.xfer_gather.stypecount = send_size;/* stypecount is ignored when sndbuf == PAMI_IN_PLACE */ - gather.cmd.xfer_gather.sndbuf = (void *)sbuf; - if(sendbuf == MPI_IN_PLACE) - { - gather.cmd.xfer_gather.sndbuf = PAMI_IN_PLACE; - } - gather.cmd.xfer_gather.stype = PAMI_TYPE_BYTE;/* stype is ignored when sndbuf == PAMI_IN_PLACE */ - gather.cmd.xfer_gather.rcvbuf = (void *)rbuf; - gather.cmd.xfer_gather.rtype = PAMI_TYPE_BYTE; - gather.cmd.xfer_gather.rtypecount = recv_size; - gather.algorithm = mpid->coll_algorithm[PAMI_XFER_GATHER][0][0]; - my_gather_md = &mpid->coll_metadata[PAMI_XFER_GATHER][0][0]; - - MPIDI_Update_last_algorithm(comm_ptr, - my_gather_md->name); - - TRACE_ERR("%s gather\n", MPIDI_Process.context_post.active>0?"Posting":"Invoking"); - MPIDI_Context_post(MPIDI_Context[0], &gather_post.state, - MPIDI_Pami_post_wrapper, (void *)&gather); - - TRACE_ERR("Waiting on active: %d\n", active); - MPID_PROGRESS_WAIT_WHILE(active); - - if(!rcv_contig) - { - MPIR_Localcopy(rcv_noncontig_buff, recv_size*size, MPI_CHAR, - recvbuf, recvcount*size, recvtype); - MPL_free(rcv_noncontig_buff); - } - if(!snd_contig) MPL_free(snd_noncontig_buff); - - TRACE_ERR("Leaving MPIDO_Gather_optimized\n"); - return MPI_SUCCESS; -} - -int -MPIDO_CSWrapper_gather(pami_xfer_t *gather, - void *comm) -{ - int mpierrno = 0; - MPIR_Comm *comm_ptr = (MPIR_Comm*)comm; - MPI_Datatype sendtype, recvtype; - void *sbuf; - MPIDI_coll_check_in_place(gather->cmd.xfer_gather.sndbuf, &sbuf); - int rc = MPIDI_Dtpami_to_dtmpi( gather->cmd.xfer_gather.stype, - &sendtype, - NULL, - NULL); - if(rc == -1) return rc; - - rc = MPIDI_Dtpami_to_dtmpi( gather->cmd.xfer_gather.rtype, - &recvtype, - NULL, - NULL); - if(rc == -1) return rc; - - rc = MPIR_Gather(sbuf, - gather->cmd.xfer_gather.stypecount, sendtype, - gather->cmd.xfer_gather.rcvbuf, - gather->cmd.xfer_gather.rtypecount, recvtype, - gather->cmd.xfer_gather.root, comm_ptr, &mpierrno); - - if(gather->cb_done && rc == 0) - gather->cb_done(NULL, gather->cookie, PAMI_SUCCESS); - return rc; - -} - diff --git a/src/mpid/pamid/src/coll/gather/mpido_igather.c b/src/mpid/pamid/src/coll/gather/mpido_igather.c deleted file mode 100644 index f55eea4..0000000 --- a/src/mpid/pamid/src/coll/gather/mpido_igather.c +++ /dev/null @@ -1,72 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/coll/gather/mpido_igather.c - * \brief ??? - */ - -#include <mpidimpl.h> - -int MPIDO_Igather(const void *sendbuf, - int sendcount, - MPI_Datatype sendtype, - void *recvbuf, - int recvcount, - MPI_Datatype recvtype, - int root, - MPIR_Comm *comm_ptr, - MPIR_Request **request) -{ - /*if (unlikely((data_size == 0) || (user_selected_type == MPID_COLL_USE_MPICH)))*/ - { - /* - * If the mpich mpir non-blocking collectives are enabled, return without - * first constructing the MPIR_Request. This signals to the - * MPIR_Igather_impl() function to invoke the mpich nbc implementation - * of MPI_Igather(). - */ - if (MPIDI_Process.mpir_nbc != 0) - return 0; - - /* - * MPIR_* nbc implementation is not enabled. Fake a non-blocking - * MPIR_Igather() with a blocking MPIR_Gather(). - */ - if(unlikely(MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_ALL && comm_ptr->rank == 0)) - fprintf(stderr,"Using MPICH blocking gather algorithm\n"); - - int mpierrno = 0; - int rc = MPIR_Gather_impl(sendbuf, sendcount, sendtype, - recvbuf, recvcount, recvtype, - root, comm_ptr, &mpierrno); - - /* - * The blocking gather has completed - create and complete a - * MPIR_Request object so the MPIR_Igather_impl() function does not - * perform an additional igather. - */ - MPIR_Request * mpid_request = MPID_Request_create_inline(); - mpid_request->kind = MPIR_REQUEST_KIND__COLL; - *request = mpid_request; - MPIDI_Request_complete_norelease_inline(mpid_request); - - return rc; - } - - return 0; -} diff --git a/src/mpid/pamid/src/coll/gatherv/Makefile.mk b/src/mpid/pamid/src/coll/gatherv/Makefile.mk deleted file mode 100644 index ee401df..0000000 --- a/src/mpid/pamid/src/coll/gatherv/Makefile.mk +++ /dev/null @@ -1,30 +0,0 @@ -# begin_generated_IBM_copyright_prolog -# -# This is an automatically generated copyright prolog. -# After initializing, DO NOT MODIFY OR MOVE -# --------------------------------------------------------------- -# Licensed Materials - Property of IBM -# Blue Gene/Q 5765-PER 5765-PRP -# -# (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved -# US Government Users Restricted Rights - -# Use, duplication, or disclosure restricted -# by GSA ADP Schedule Contract with IBM Corp. -# -# --------------------------------------------------------------- -# -# end_generated_IBM_copyright_prolog -# -*- mode: makefile-gmake; -*- - -# note that the includes always happen but the effects of their contents are -# affected by "if BUILD_PAMID" -if BUILD_PAMID - - -mpi_core_sources += \ - src/mpid/pamid/src/coll/gatherv/mpido_gatherv.c \ - src/mpid/pamid/src/coll/gatherv/mpido_igatherv.c - - -endif BUILD_PAMID - diff --git a/src/mpid/pamid/src/coll/gatherv/mpido_gatherv.c b/src/mpid/pamid/src/coll/gatherv/mpido_gatherv.c deleted file mode 100644 index 5832468..0000000 --- a/src/mpid/pamid/src/coll/gatherv/mpido_gatherv.c +++ /dev/null @@ -1,573 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/coll/gather/mpido_gatherv.c - * \brief ??? - */ - -/* #define TRACE_ON */ -#include <mpidimpl.h> - -static void cb_gatherv(void *ctxt, void *clientdata, pami_result_t err) -{ - unsigned *active = (unsigned *)clientdata; - TRACE_ERR("cb_gatherv enter, active: %u\n", (*active)); - (*active)--; -} - -int MPIDO_Gatherv(const void *sendbuf, - int sendcount, - MPI_Datatype sendtype, - void *recvbuf, - const int *recvcounts, - const int *displs, - MPI_Datatype recvtype, - int root, - MPIR_Comm * comm_ptr, - int *mpierrno) - -{ -#ifndef HAVE_PAMI_IN_PLACE - if (sendbuf == MPI_IN_PLACE) - { - MPID_Abort (NULL, 0, 1, "'MPI_IN_PLACE' requries support for `PAMI_IN_PLACE`"); - return -1; - } -#endif - TRACE_ERR("Entering MPIDO_Gatherv\n"); - int i; - int contig ATTRIBUTE((unused)), rsize ATTRIBUTE((unused)), ssize ATTRIBUTE((unused)); - int pamidt = 1; - MPIDU_Datatype*dt_ptr = NULL; - MPI_Aint send_true_lb, recv_true_lb; - char *sbuf, *rbuf; - pami_type_t stype, rtype; - int tmp; - volatile unsigned gatherv_active = 1; - const int rank = comm_ptr->rank; - const int size = comm_ptr->local_size; -#if ASSERT_LEVEL==0 - /* We can't afford the tracing in ndebug/performance libraries */ - const unsigned verbose = 0; -#else - const unsigned verbose = (MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_ALL) && (rank == 0); -#endif - const struct MPIDI_Comm* const mpid = &(comm_ptr->mpid); - const int selected_type = mpid->user_selected_type[PAMI_XFER_GATHERV_INT]; - - /* Check for native PAMI types and MPI_IN_PLACE on sendbuf */ - /* MPI_IN_PLACE is a nonlocal decision. We will need a preallreduce if we ever have - * multiple "good" gathervs that work on different counts for example */ - if((sendbuf != MPI_IN_PLACE) && (MPIDI_Datatype_to_pami(sendtype, &stype, -1, NULL, &tmp) != MPI_SUCCESS)) - pamidt = 0; - if(MPIDI_Datatype_to_pami(recvtype, &rtype, -1, NULL, &tmp) != MPI_SUCCESS) - pamidt = 0; - - if(pamidt == 0 || selected_type == MPID_COLL_USE_MPICH) - { - if(unlikely(verbose)) - fprintf(stderr,"Using MPICH gatherv algorithm\n"); - TRACE_ERR("GATHERV using MPICH\n"); - MPIDI_Update_last_algorithm(comm_ptr, "GATHERV_MPICH"); -#if CUDA_AWARE_SUPPORT - if(MPIDI_Process.cuda_aware_support_on) - { - MPI_Aint sdt_extent,rdt_extent; - MPIDU_Datatype_get_extent_macro(sendtype, sdt_extent); - MPIDU_Datatype_get_extent_macro(recvtype, rdt_extent); - char *scbuf = NULL; - char *rcbuf = NULL; - int is_send_dev_buf = MPIDI_cuda_is_device_buf(sendbuf); - int is_recv_dev_buf = (rank == root) ? MPIDI_cuda_is_device_buf(recvbuf) : 0; - if(is_send_dev_buf) - { - scbuf = MPL_malloc(sdt_extent * sendcount); - cudaError_t cudaerr = CudaMemcpy(scbuf, sendbuf, sdt_extent * sendcount, cudaMemcpyDeviceToHost); - if (cudaSuccess != cudaerr) - fprintf(stderr, "cudaMemcpy failed: %s\n", CudaGetErrorString(cudaerr)); - } - else - scbuf = sendbuf; - size_t rtotal_buf; - if(is_recv_dev_buf) - { - //Since displs can be non-continous, we need to calculate max buffer size - int highest_displs = displs[size - 1]; - int highest_recvcount = recvcounts[size - 1]; - for(i = 0; i < size; i++) - { - if(displs[i]+recvcounts[i] > highest_displs+highest_recvcount) - { - highest_displs = displs[i]; - highest_recvcount = recvcounts[i]; - } - } - rtotal_buf = (highest_displs+highest_recvcount)*rdt_extent; - rcbuf = MPL_malloc(rtotal_buf); - if(sendbuf == MPI_IN_PLACE) - { - cudaError_t cudaerr = CudaMemcpy(rcbuf, recvbuf, rtotal_buf, cudaMemcpyDeviceToHost); - if (cudaSuccess != cudaerr) - fprintf(stderr, "cudaMemcpy failed: %s\n", CudaGetErrorString(cudaerr)); - } - else - memset(rcbuf, 0, rtotal_buf); - } - else - rcbuf = recvbuf; - int cuda_res = MPIR_Gatherv(scbuf, sendcount, sendtype, rcbuf, recvcounts, displs, recvtype, root, comm_ptr, mpierrno); - if(is_send_dev_buf)MPL_free(scbuf); - if(is_recv_dev_buf) - { - cudaError_t cudaerr = CudaMemcpy(recvbuf, rcbuf, rtotal_buf, cudaMemcpyHostToDevice); - if (cudaSuccess != cudaerr) - fprintf(stderr, "cudaMemcpy failed: %s\n", CudaGetErrorString(cudaerr)); - MPL_free(rcbuf); - } - return cuda_res; - } - else -#endif - return MPIR_Gatherv(sendbuf, sendcount, sendtype, - recvbuf, recvcounts, displs, recvtype, - root, comm_ptr, mpierrno); - } - - MPIDI_Datatype_get_info(1, recvtype, contig, rsize, dt_ptr, recv_true_lb); - rbuf = (char *)recvbuf + recv_true_lb; - sbuf = (void *) sendbuf; - - pami_xfer_t gatherv; - - gatherv.cb_done = cb_gatherv; - gatherv.cookie = (void *)&gatherv_active; - gatherv.cmd.xfer_gatherv_int.root = MPIDI_Task_to_endpoint(MPID_VCR_GET_LPID(comm_ptr->vcr, root), 0); - gatherv.cmd.xfer_gatherv_int.rcvbuf = rbuf; - gatherv.cmd.xfer_gatherv_int.rtype = rtype; - gatherv.cmd.xfer_gatherv_int.rtypecounts = (int *) recvcounts; - gatherv.cmd.xfer_gatherv_int.rdispls = (int *) displs; - - gatherv.cmd.xfer_gatherv_int.sndbuf = NULL; - gatherv.cmd.xfer_gatherv_int.stype = stype; - gatherv.cmd.xfer_gatherv_int.stypecount = sendcount; - - if(rank == root) - { - if(sendbuf == MPI_IN_PLACE) - { - if(unlikely(verbose)) - fprintf(stderr,"gatherv MPI_IN_PLACE buffering\n"); - sbuf = PAMI_IN_PLACE; - gatherv.cmd.xfer_gatherv_int.stype = rtype; - gatherv.cmd.xfer_gatherv_int.stypecount = recvcounts[rank]; - } - else - { - MPIDI_Datatype_get_info(1, sendtype, contig, ssize, dt_ptr, send_true_lb); - sbuf = (char *)sbuf + send_true_lb; - } - } - gatherv.cmd.xfer_gatherv_int.sndbuf = sbuf; - - pami_algorithm_t my_gatherv; - const pami_metadata_t *my_md = (pami_metadata_t *)NULL; - int queryreq = 0; - - if(selected_type == MPID_COLL_OPTIMIZED) - { - TRACE_ERR("Optimized gatherv %s was selected\n", - mpid->opt_protocol_md[PAMI_XFER_GATHERV_INT][0].name); - my_gatherv = mpid->opt_protocol[PAMI_XFER_GATHERV_INT][0]; - my_md = &mpid->opt_protocol_md[PAMI_XFER_GATHERV_INT][0]; - queryreq = mpid->must_query[PAMI_XFER_GATHERV_INT][0]; - } - else - { - TRACE_ERR("Optimized gatherv %s was set by user\n", - mpid->user_metadata[PAMI_XFER_GATHERV_INT].name); - my_gatherv = mpid->user_selected[PAMI_XFER_GATHERV_INT]; - my_md = &mpid->user_metadata[PAMI_XFER_GATHERV_INT]; - queryreq = selected_type; - } - - gatherv.algorithm = my_gatherv; - - - if(unlikely(queryreq == MPID_COLL_ALWAYS_QUERY || - queryreq == MPID_COLL_CHECK_FN_REQUIRED)) - { - metadata_result_t result = {0}; - TRACE_ERR("querying gatherv protocol %s, type was %d\n", - my_md->name, queryreq); - if(my_md->check_fn == NULL) - { - /* process metadata bits */ - if((!my_md->check_correct.values.inplace) && (sendbuf == MPI_IN_PLACE)) - result.check.unspecified = 1; -/* Can't check ranges like this. Non-local. Comment out for now. - if(my_md->check_correct.values.rangeminmax) - { - MPI_Aint data_true_lb; - MPIDU_Datatype*data_ptr; - int data_size, data_contig; - MPIDI_Datatype_get_info(sendcount, sendtype, data_contig, data_size, data_ptr, data_true_lb); - if((my_md->range_lo <= data_size) && - (my_md->range_hi >= data_size)) - ; - else - { - result.check.range = 1; - if(unlikely(verbose)) - { - fprintf(stderr,"message size (%u) outside range (%zu<->%zu) for %s.\n", - data_size, - my_md->range_lo, - my_md->range_hi, - my_md->name); - } - } - } - */ - } - else /* calling the check fn is sufficient */ - result = my_md->check_fn(&gatherv); - TRACE_ERR("bitmask: %#X\n", result.bitmask); - result.check.nonlocal = 0; /* #warning REMOVE THIS WHEN IMPLEMENTED */ - if(result.bitmask) - { - if(unlikely(verbose)) - fprintf(stderr,"Query failed for %s. Using MPICH gatherv.\n", my_md->name); - MPIDI_Update_last_algorithm(comm_ptr, "GATHERV_MPICH"); - return MPIR_Gatherv(sendbuf, sendcount, sendtype, - recvbuf, recvcounts, displs, recvtype, - root, comm_ptr, mpierrno); - } - if(my_md->check_correct.values.asyncflowctl && !(--(comm_ptr->mpid.num_requests))) - { - comm_ptr->mpid.num_requests = MPIDI_Process.optimized.num_requests; - int tmpmpierrno; - if(unlikely(verbose)) - fprintf(stderr,"Query barrier required for %s\n", my_md->name); - MPIDO_Barrier(comm_ptr, &tmpmpierrno); - } - } - - MPIDI_Update_last_algorithm(comm_ptr, my_md->name); - - if(unlikely(verbose)) - { - unsigned long long int threadID; - MPL_thread_id_t tid; - MPL_thread_self(&tid); - threadID = (unsigned long long int)tid; - fprintf(stderr,"<%llx> Using protocol %s for gatherv on %u\n", - threadID, - my_md->name, - (unsigned) comm_ptr->context_id); - } - - MPIDI_Post_coll_t gatherv_post; - MPIDI_Context_post(MPIDI_Context[0], &gatherv_post.state, - MPIDI_Pami_post_wrapper, (void *)&gatherv); - - TRACE_ERR("Waiting on active %d\n", gatherv_active); - MPID_PROGRESS_WAIT_WHILE(gatherv_active); - - TRACE_ERR("Leaving MPIDO_Gatherv\n"); - return 0; -} - - -int MPIDO_Gatherv_simple(const void *sendbuf, - int sendcount, - MPI_Datatype sendtype, - void *recvbuf, - const int *recvcounts, - const int *displs, - MPI_Datatype recvtype, - int root, - MPIR_Comm * comm_ptr, - int *mpierrno) - -{ -#ifndef HAVE_PAMI_IN_PLACE - if (sendbuf == MPI_IN_PLACE) - { - MPID_Abort (NULL, 0, 1, "'MPI_IN_PLACE' requries support for `PAMI_IN_PLACE`"); - return -1; - } -#endif - TRACE_ERR("Entering MPIDO_Gatherv_optimized\n"); - int snd_contig = 1, rcv_contig = 1; - void *snd_noncontig_buff = NULL, *rcv_noncontig_buff = NULL; - void *sbuf = NULL, *rbuf = NULL; - int *rcounts = NULL; - int *rdispls = NULL; - int send_size = 0; - int recv_size = 0; - int rcvlen = 0; - int totalrecvcount = 0; - pami_type_t rtype = PAMI_TYPE_NULL; - MPIDU_Segment segment; - MPIDU_Datatype*data_ptr = NULL; - int send_true_lb, recv_true_lb = 0; - int i, tmp; - volatile unsigned gatherv_active = 1; - const int rank = comm_ptr->rank; - const int size = comm_ptr->local_size; -#if ASSERT_LEVEL==0 - /* We can't afford the tracing in ndebug/performance libraries */ - const unsigned verbose = 0; -#else - const unsigned verbose = (MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_ALL) && (rank == 0); -#endif - - const struct MPIDI_Comm* const mpid = &(comm_ptr->mpid); - int recvok=PAMI_SUCCESS, recvcontinuous=0; - - if(sendbuf != MPI_IN_PLACE) - { - MPIDI_Datatype_get_info(sendcount, sendtype, snd_contig, - send_size, data_ptr, send_true_lb); - if(MPIDI_Pamix_collsel_advise != NULL && mpid->collsel_fast_query != NULL) - { - advisor_algorithm_t advisor_algorithms[1]; - int num_algorithms = MPIDI_Pamix_collsel_advise(mpid->collsel_fast_query, PAMI_XFER_GATHERV_INT, 64, advisor_algorithms, 1); - if(num_algorithms) - { - if(advisor_algorithms[0].algorithm_type == COLLSEL_EXTERNAL_ALGO) - { - return MPIR_Gatherv(sendbuf, sendcount, sendtype, - recvbuf, recvcounts, displs, recvtype, - root, comm_ptr, mpierrno); - } - else if(advisor_algorithms[0].metadata && advisor_algorithms[0].metadata->check_correct.values.asyncflowctl && !(--(comm_ptr->mpid.num_requests))) - { - comm_ptr->mpid.num_requests = MPIDI_Process.optimized.num_requests; - int tmpmpierrno; - if(unlikely(verbose)) - fprintf(stderr,"Query barrier required for %s\n", advisor_algorithms[0].metadata->name); - MPIDO_Barrier(comm_ptr, &tmpmpierrno); - } - } - } - - sbuf = (char *)sendbuf + send_true_lb; - if(!snd_contig) - { - snd_noncontig_buff = MPL_malloc(send_size); - sbuf = snd_noncontig_buff; - if(snd_noncontig_buff == NULL) - { - MPID_Abort(NULL, MPI_ERR_NO_SPACE, 1, - "Fatal: Cannot allocate pack buffer"); - } - DLOOP_Offset last = send_size; - MPIDU_Segment_init(sendbuf, sendcount, sendtype, &segment, 0); - MPIDU_Segment_pack(&segment, 0, &last, snd_noncontig_buff); - } - } - else - { - MPIDI_Datatype_get_info(1, recvtype, rcv_contig, - rcvlen, data_ptr, recv_true_lb); - if(MPIDI_Pamix_collsel_advise != NULL && mpid->collsel_fast_query != NULL) - { - advisor_algorithm_t advisor_algorithms[1]; - int num_algorithms = MPIDI_Pamix_collsel_advise(mpid->collsel_fast_query, PAMI_XFER_GATHERV_INT, 64, advisor_algorithms, 1); - if(num_algorithms) - { - if(advisor_algorithms[0].algorithm_type == COLLSEL_EXTERNAL_ALGO) - { - return MPIR_Gatherv(sendbuf, sendcount, sendtype, - recvbuf, recvcounts, displs, recvtype, - root, comm_ptr, mpierrno); - } - else if(advisor_algorithms[0].metadata && advisor_algorithms[0].metadata->check_correct.values.asyncflowctl && !(--(comm_ptr->mpid.num_requests))) - { - comm_ptr->mpid.num_requests = MPIDI_Process.optimized.num_requests; - int tmpmpierrno; - if(unlikely(verbose)) - fprintf(stderr,"Query barrier required for %s\n", advisor_algorithms[0].metadata->name); - MPIDO_Barrier(comm_ptr, &tmpmpierrno); - } - } - } - } - - pami_xfer_t gatherv; - rbuf = (char *)recvbuf + recv_true_lb; - rcounts = (int*)recvcounts; - rdispls = (int*)displs; - if(rank == root) - { - if((recvok = MPIDI_Datatype_to_pami(recvtype, &rtype, -1, NULL, &tmp)) != MPI_SUCCESS) - { - MPIDI_Datatype_get_info(1, recvtype, rcv_contig, - rcvlen, data_ptr, recv_true_lb); - totalrecvcount = recvcounts[0]; - recvcontinuous = displs[0] == 0? 1 : 0 ; - rcounts = (int*)MPL_malloc(size); - rdispls = (int*)MPL_malloc(size); - rdispls[0] = 0; - rcounts[0] = rcvlen * recvcounts[0]; - for(i = 1; i < size; i++) - { - rdispls[i]= rcvlen * totalrecvcount; - totalrecvcount += recvcounts[i]; - if(displs[i] != (displs[i-1] + recvcounts[i-1])) - recvcontinuous = 0; - rcounts[i] = rcvlen * recvcounts[i]; - } - recv_size = rcvlen * totalrecvcount; - - rcv_noncontig_buff = MPL_malloc(recv_size); - rbuf = rcv_noncontig_buff; - rtype = PAMI_TYPE_BYTE; - if(rcv_noncontig_buff == NULL) - { - MPID_Abort(NULL, MPI_ERR_NO_SPACE, 1, - "Fatal: Cannot allocate pack buffer"); - } - if(sendbuf == MPI_IN_PLACE) - { - size_t extent; - MPIDU_Datatype_get_extent_macro(recvtype,extent); - MPIR_Localcopy(recvbuf + displs[rank]*extent, recvcounts[rank], recvtype, - rcv_noncontig_buff + rdispls[rank], rcounts[rank],MPI_CHAR); - } - } - if(sendbuf == MPI_IN_PLACE) - { - gatherv.cmd.xfer_gatherv_int.sndbuf = PAMI_IN_PLACE; - } - else - { - gatherv.cmd.xfer_gatherv_int.sndbuf = sbuf; - } - gatherv.cmd.xfer_gatherv_int.stype = PAMI_TYPE_BYTE;/* stype is ignored when sndbuf == PAMI_IN_PLACE */ - gatherv.cmd.xfer_gatherv_int.stypecount = send_size; - - } - else - { - gatherv.cmd.xfer_gatherv_int.sndbuf = sbuf; - gatherv.cmd.xfer_gatherv_int.stype = PAMI_TYPE_BYTE; - gatherv.cmd.xfer_gatherv_int.stypecount = send_size; - } - - - gatherv.cb_done = cb_gatherv; - gatherv.cookie = (void *)&gatherv_active; - gatherv.cmd.xfer_gatherv_int.root = MPIDI_Task_to_endpoint(MPID_VCR_GET_LPID(comm_ptr->vcr, root), 0); - gatherv.cmd.xfer_gatherv_int.rcvbuf = rbuf; - gatherv.cmd.xfer_gatherv_int.rtype = rtype; - gatherv.cmd.xfer_gatherv_int.rtypecounts = (int *) rcounts; - gatherv.cmd.xfer_gatherv_int.rdispls = (int *) rdispls; - - - const pami_metadata_t *my_gatherv_md; - - gatherv.algorithm = mpid->coll_algorithm[PAMI_XFER_GATHERV_INT][0][0]; - my_gatherv_md = &mpid->coll_metadata[PAMI_XFER_GATHERV_INT][0][0]; - - MPIDI_Update_last_algorithm(comm_ptr, my_gatherv_md->name); - - MPIDI_Post_coll_t gatherv_post; - TRACE_ERR("%s gatherv\n", MPIDI_Process.context_post.active>0?"Posting":"Invoking"); - MPIDI_Context_post(MPIDI_Context[0], &gatherv_post.state, - MPIDI_Pami_post_wrapper, (void *)&gatherv); - TRACE_ERR("Gatherv %s\n", MPIDI_Process.context_post.active>0?"posted":"invoked"); - - TRACE_ERR("Waiting on active %d\n", gatherv_active); - MPID_PROGRESS_WAIT_WHILE(gatherv_active); - - if(!rcv_contig || recvok != PAMI_SUCCESS) - { - if(recvcontinuous) - { - MPIR_Localcopy(rcv_noncontig_buff, recv_size, MPI_CHAR, - recvbuf, totalrecvcount, recvtype); - } - else - { - size_t extent; - MPIDU_Datatype_get_extent_macro(recvtype,extent); - for(i=0; i<size; ++i) - { - char* scbuf = (char*)rcv_noncontig_buff+ rdispls[i]; - char* rcbuf = (char*)recvbuf + displs[i]*extent; - MPIR_Localcopy(scbuf, rcounts[i], MPI_CHAR, - rcbuf, recvcounts[i], recvtype); - TRACE_ERR("Pack recv src extent %zu, displ[%zu]=%zu, count[%zu]=%zu buf[%zu]=%u\n", - (size_t)extent, (size_t)i,(size_t)precvdispls[i],(size_t)i,(size_t)precvcounts[i],(size_t)precvdispls[i], *(int*)scbuf); - TRACE_ERR("Pack recv dest extent %zu, displ[%zu]=%zu, count[%zu]=%zu buf[%zu]=%u\n", - (size_t)extent, (size_t)i,(size_t)displs[i],(size_t)i,(size_t)recvcounts[i],(size_t)displs[i], *(int*)rcbuf); - } - - } - MPL_free(rcv_noncontig_buff); - if(rank == root) - { - MPL_free(rcounts); - MPL_free(rdispls); - } - } - if(!snd_contig) MPL_free(snd_noncontig_buff); - - - TRACE_ERR("Leaving MPIDO_Gatherv_optimized\n"); - return MPI_SUCCESS; -} - -int -MPIDO_CSWrapper_gatherv(pami_xfer_t *gatherv, - void *comm) -{ - int mpierrno = 0; - MPIR_Comm *comm_ptr = (MPIR_Comm*)comm; - MPI_Datatype sendtype, recvtype; - void *sbuf; - MPIDI_coll_check_in_place(gatherv->cmd.xfer_gatherv_int.sndbuf, &sbuf); - int rc = MPIDI_Dtpami_to_dtmpi( gatherv->cmd.xfer_gatherv_int.stype, - &sendtype, - NULL, - NULL); - if(rc == -1) return rc; - - if(gatherv->cmd.xfer_gatherv_int.rtype == PAMI_TYPE_NULL) - recvtype = MPI_DATATYPE_NULL; - else - rc = MPIDI_Dtpami_to_dtmpi( gatherv->cmd.xfer_gatherv_int.rtype, - &recvtype, - NULL, - NULL); - if(rc == -1) return rc; - - rc = MPIR_Gatherv(sbuf, - gatherv->cmd.xfer_gatherv_int.stypecount, sendtype, - gatherv->cmd.xfer_gatherv_int.rcvbuf, - gatherv->cmd.xfer_gatherv_int.rtypecounts, - gatherv->cmd.xfer_gatherv_int.rdispls, recvtype, - gatherv->cmd.xfer_gatherv_int.root, comm_ptr, &mpierrno); - if(gatherv->cb_done && rc == 0) - gatherv->cb_done(NULL, gatherv->cookie, PAMI_SUCCESS); - return rc; - -} - diff --git a/src/mpid/pamid/src/coll/gatherv/mpido_igatherv.c b/src/mpid/pamid/src/coll/gatherv/mpido_igatherv.c deleted file mode 100644 index 3a315d6..0000000 --- a/src/mpid/pamid/src/coll/gatherv/mpido_igatherv.c +++ /dev/null @@ -1,78 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/coll/gatherv/mpido_igatherv.c - * \brief ??? - */ - -/*#define TRACE_ON*/ -#include <mpidimpl.h> - -int MPIDO_Igatherv(const void *sendbuf, - int sendcount, - MPI_Datatype sendtype, - void *recvbuf, - const int *recvcounts, - const int *displs, - MPI_Datatype recvtype, - int root, - MPIR_Comm * comm_ptr, - MPIR_Request **request) - -{ - TRACE_ERR("Entering MPIDO_Igatherv\n"); - - /*if (unlikely((data_size == 0) || (user_selected_type == MPID_COLL_USE_MPICH)))*/ - { - /* - * If the mpich mpir non-blocking collectives are enabled, return without - * first constructing the MPIR_Request. This signals to the - * MPIR_Igatherv_impl() function to invoke the mpich nbc implementation - * of MPI_Igatherv(). - */ - if (MPIDI_Process.mpir_nbc != 0) - return 0; - - /* - * MPIR_* nbc implementation is not enabled. Fake a non-blocking - * MPIR_Igatherv() with a blocking MPIR_Gatherv(). - */ - if(unlikely(MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_ALL && comm_ptr->rank == 0)) - fprintf(stderr,"Using MPICH blocking gatherv algorithm\n"); - - int mpierrno = 0; - int rc = MPIR_Gatherv_impl(sendbuf, sendcount, sendtype, - recvbuf, recvcounts, displs, recvtype, - root, comm_ptr, &mpierrno); - - /* - * The blocking gather has completed - create and complete a - * MPIR_Request object so the MPIR_Igatherv_impl() function does not - * perform an additional igatherv. - */ - MPIR_Request * mpid_request = MPID_Request_create_inline(); - mpid_request->kind = MPIR_REQUEST_KIND__COLL; - *request = mpid_request; - MPIDI_Request_complete_norelease_inline(mpid_request); - - return rc; - } - - - return 0; -} diff --git a/src/mpid/pamid/src/coll/ired_scat/Makefile.mk b/src/mpid/pamid/src/coll/ired_scat/Makefile.mk deleted file mode 100644 index b975ffd..0000000 --- a/src/mpid/pamid/src/coll/ired_scat/Makefile.mk +++ /dev/null @@ -1,28 +0,0 @@ -# begin_generated_IBM_copyright_prolog -# -# This is an automatically generated copyright prolog. -# After initializing, DO NOT MODIFY OR MOVE -# --------------------------------------------------------------- -# Licensed Materials - Property of IBM -# Blue Gene/Q 5765-PER 5765-PRP -# -# (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved -# US Government Users Restricted Rights - -# Use, duplication, or disclosure restricted -# by GSA ADP Schedule Contract with IBM Corp. -# -# --------------------------------------------------------------- -# -# end_generated_IBM_copyright_prolog -# -*- mode: makefile-gmake; -*- - -# note that the includes always happen but the effects of their contents are -# affected by "if BUILD_PAMID" -if BUILD_PAMID - - -mpi_core_sources += \ - src/mpid/pamid/src/coll/ired_scat/mpido_ired_scat.c - - -endif BUILD_PAMID diff --git a/src/mpid/pamid/src/coll/ired_scat/mpido_ired_scat.c b/src/mpid/pamid/src/coll/ired_scat/mpido_ired_scat.c deleted file mode 100644 index 80d54d8..0000000 --- a/src/mpid/pamid/src/coll/ired_scat/mpido_ired_scat.c +++ /dev/null @@ -1,72 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/coll/ired_scat/mpido_ired_scat.c - * \brief ??? - */ - -/*#define TRACE_ON*/ -#include <mpidimpl.h> - -int MPIDO_Ireduce_scatter(const void *sendbuf, - void *recvbuf, - const int *recvcounts, - MPI_Datatype datatype, - MPI_Op op, - MPIR_Comm *comm_ptr, - MPIR_Request **request) -{ - TRACE_ERR("Entering MPIDO_Ireduce_scatter\n"); - - /*if (unlikely((data_size == 0) || (user_selected_type == MPID_COLL_USE_MPICH)))*/ - { - /* - * If the mpich mpir non-blocking collectives are enabled, return without - * first constructing the MPIR_Request. This signals to the - * MPIR_Ireduce_scatter_impl() function to invoke the mpich nbc - * implementation of MPI_Ireduce_scatter(). - */ - if (MPIDI_Process.mpir_nbc != 0) - return 0; - - /* - * MPIR_* nbc implementation is not enabled. Fake a non-blocking - * MPIR_Ireduce_scatter() with a blocking MPIR_Reduce_scatter_block(). - */ - if(unlikely(MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_ALL && comm_ptr->rank == 0)) - fprintf(stderr,"Using MPICH blocking reduce_scatter algorithm\n"); - - int mpierrno = 0; - int rc = MPIR_Reduce_scatter_impl(sendbuf, recvbuf, recvcounts, datatype, - op, comm_ptr, &mpierrno); - - /* - * The blocking gather has completed - create and complete a - * MPIR_Request object so the MPIR_Ireduce_scatter_impl() function - * does not perform an additional ireduce_scatter. - */ - MPIR_Request * mpid_request = MPID_Request_create_inline(); - mpid_request->kind = MPIR_REQUEST_KIND__COLL; - *request = mpid_request; - MPIDI_Request_complete_norelease_inline(mpid_request); - - return rc; - } - - return 0; -} diff --git a/src/mpid/pamid/src/coll/ired_scat_block/Makefile.mk b/src/mpid/pamid/src/coll/ired_scat_block/Makefile.mk deleted file mode 100644 index 562c719..0000000 --- a/src/mpid/pamid/src/coll/ired_scat_block/Makefile.mk +++ /dev/null @@ -1,28 +0,0 @@ -# begin_generated_IBM_copyright_prolog -# -# This is an automatically generated copyright prolog. -# After initializing, DO NOT MODIFY OR MOVE -# --------------------------------------------------------------- -# Licensed Materials - Property of IBM -# Blue Gene/Q 5765-PER 5765-PRP -# -# (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved -# US Government Users Restricted Rights - -# Use, duplication, or disclosure restricted -# by GSA ADP Schedule Contract with IBM Corp. -# -# --------------------------------------------------------------- -# -# end_generated_IBM_copyright_prolog -# -*- mode: makefile-gmake; -*- - -# note that the includes always happen but the effects of their contents are -# affected by "if BUILD_PAMID" -if BUILD_PAMID - - -mpi_core_sources += \ - src/mpid/pamid/src/coll/ired_scat_block/mpido_ired_scat_block.c - - -endif BUILD_PAMID diff --git a/src/mpid/pamid/src/coll/ired_scat_block/mpido_ired_scat_block.c b/src/mpid/pamid/src/coll/ired_scat_block/mpido_ired_scat_block.c deleted file mode 100644 index 1a9f738..0000000 --- a/src/mpid/pamid/src/coll/ired_scat_block/mpido_ired_scat_block.c +++ /dev/null @@ -1,72 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/coll/ired_scat_block/mpido_ired_scat_block.c - * \brief ??? - */ - -/*#define TRACE_ON*/ -#include <mpidimpl.h> - -int MPIDO_Ireduce_scatter_block(const void *sendbuf, - void *recvbuf, - int recvcount, - MPI_Datatype datatype, - MPI_Op op, - MPIR_Comm *comm_ptr, - MPIR_Request **request) -{ - TRACE_ERR("Entering MPIDO_Ireduce_scatter_block\n"); - - /*if (unlikely((data_size == 0) || (user_selected_type == MPID_COLL_USE_MPICH)))*/ - { - /* - * If the mpich mpir non-blocking collectives are enabled, return without - * first constructing the MPIR_Request. This signals to the - * MPIR_Ireduce_scatter_block_impl() function to invoke the mpich nbc - * implementation of MPI_Ireduce_scatter_block(). - */ - if (MPIDI_Process.mpir_nbc != 0) - return 0; - - /* - * MPIR_* nbc implementation is not enabled. Fake a non-blocking - * MPIR_Ireduce_scatter_block() with a blocking MPIR_Reduce_scatter_block(). - */ - if(unlikely(MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_ALL && comm_ptr->rank == 0)) - fprintf(stderr,"Using MPICH blocking reduce_scatter_block algorithm\n"); - - int mpierrno = 0; - int rc = MPIR_Reduce_scatter_block_impl(sendbuf, recvbuf, recvcount, datatype, - op, comm_ptr, &mpierrno); - - /* - * The blocking gather has completed - create and complete a - * MPIR_Request object so the MPIR_Ireduce_scatter_block_impl() function - * does not perform an additional ireduce_scatter_block. - */ - MPIR_Request * mpid_request = MPID_Request_create_inline(); - mpid_request->kind = MPIR_REQUEST_KIND__COLL; - *request = mpid_request; - MPIDI_Request_complete_norelease_inline(mpid_request); - - return rc; - } - - return 0; -} diff --git a/src/mpid/pamid/src/coll/red_scat/Makefile.mk b/src/mpid/pamid/src/coll/red_scat/Makefile.mk deleted file mode 100644 index a95beaf..0000000 --- a/src/mpid/pamid/src/coll/red_scat/Makefile.mk +++ /dev/null @@ -1,29 +0,0 @@ -# begin_generated_IBM_copyright_prolog -# -# This is an automatically generated copyright prolog. -# After initializing, DO NOT MODIFY OR MOVE -# --------------------------------------------------------------- -# Licensed Materials - Property of IBM -# Blue Gene/Q 5765-PER 5765-PRP -# -# (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved -# US Government Users Restricted Rights - -# Use, duplication, or disclosure restricted -# by GSA ADP Schedule Contract with IBM Corp. -# -# --------------------------------------------------------------- -# -# end_generated_IBM_copyright_prolog -# -*- mode: makefile-gmake; -*- - -# note that the includes always happen but the effects of their contents are -# affected by "if BUILD_PAMID" -if BUILD_PAMID - - -mpi_core_sources += \ - src/mpid/pamid/src/coll/red_scat/mpido_red_scat.c - - -endif BUILD_PAMID - diff --git a/src/mpid/pamid/src/coll/red_scat/mpido_red_scat.c b/src/mpid/pamid/src/coll/red_scat/mpido_red_scat.c deleted file mode 100644 index d207f45..0000000 --- a/src/mpid/pamid/src/coll/red_scat/mpido_red_scat.c +++ /dev/null @@ -1,185 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/coll/red_scat/mpido_red_scat.c - * \brief ??? - */ - -/* #define TRACE_ON */ -#include <mpidimpl.h> - - -int MPIDO_Reduce_scatter(const void *sendbuf, - void *recvbuf, - int *recvcounts, - MPI_Datatype datatype, - MPI_Op op, - MPIR_Comm *comm_ptr, - int *mpierrno) - -{ - const int rank = comm_ptr->rank; - const int size = comm_ptr->local_size; -#if ASSERT_LEVEL==0 - /* We can't afford the tracing in ndebug/performance libraries */ - const unsigned verbose = 0; -#else - const unsigned verbose = (MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_ALL) && (rank == 0); -#endif - - if(unlikely(verbose)) - fprintf(stderr,"Using MPICH reduce_scatter algorithm\n"); -#if CUDA_AWARE_SUPPORT - if(MPIDI_Process.cuda_aware_support_on) - { - MPI_Aint dt_extent; - MPIDU_Datatype_get_extent_macro(datatype, dt_extent); - char *scbuf = NULL; - char *rcbuf = NULL; - int is_send_dev_buf = MPIDI_cuda_is_device_buf(sendbuf); - int is_recv_dev_buf = MPIDI_cuda_is_device_buf(recvbuf); - int i; - size_t total_buf = 0; - for(i = 0; i < size; i++) - { - total_buf += recvcounts[i]; - } - - if(is_send_dev_buf) - { - scbuf = MPL_malloc(dt_extent * total_buf); - cudaError_t cudaerr = CudaMemcpy(scbuf, sendbuf, dt_extent * total_buf, cudaMemcpyDeviceToHost); - if (cudaSuccess != cudaerr) - fprintf(stderr, "cudaMemcpy failed: %s\n", CudaGetErrorString(cudaerr)); - } - else - scbuf = sendbuf; - - if(is_recv_dev_buf) - { - rcbuf = MPL_malloc(total_buf * dt_extent); - if(sendbuf == MPI_IN_PLACE) - { - cudaError_t cudaerr = CudaMemcpy(rcbuf, recvbuf, dt_extent * total_buf, cudaMemcpyDeviceToHost); - if (cudaSuccess != cudaerr) - fprintf(stderr, "cudaMemcpy failed: %s\n", CudaGetErrorString(cudaerr)); - } - else - memset(rcbuf, 0, total_buf * dt_extent); - } - else - rcbuf = recvbuf; - - int cuda_res = MPIR_Reduce_scatter(scbuf, rcbuf, recvcounts, datatype, op, comm_ptr, mpierrno); - if(is_send_dev_buf)MPL_free(scbuf); - if(is_recv_dev_buf) - { - cudaError_t cudaerr = CudaMemcpy(recvbuf, rcbuf, dt_extent * total_buf, cudaMemcpyHostToDevice); - if (cudaSuccess != cudaerr) - fprintf(stderr, "cudaMemcpy failed: %s\n", CudaGetErrorString(cudaerr)); - MPL_free(rcbuf); - } - return cuda_res; - } - else -#endif - return MPIR_Reduce_scatter(sendbuf, recvbuf, recvcounts, datatype, op, comm_ptr, mpierrno); - -} - - - -int MPIDO_Reduce_scatter_block(const void *sendbuf, - void *recvbuf, - int recvcount, - MPI_Datatype datatype, - MPI_Op op, - MPIR_Comm *comm_ptr, - int *mpierrno) - -{ - const int rank = comm_ptr->rank; - const int size = comm_ptr->local_size; -#if ASSERT_LEVEL==0 - /* We can't afford the tracing in ndebug/performance libraries */ - const unsigned verbose = 0; -#else - const unsigned verbose = (MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_ALL) && (rank == 0); -#endif - if(unlikely(verbose)) - fprintf(stderr,"Using MPICH reduce_scatter algorithm\n"); -#if CUDA_AWARE_SUPPORT - if(MPIDI_Process.cuda_aware_support_on) - { - MPI_Aint dt_extent; - MPIDU_Datatype_get_extent_macro(datatype, dt_extent); - char *scbuf = NULL; - char *rcbuf = NULL; - int is_send_dev_buf = MPIDI_cuda_is_device_buf(sendbuf); - int is_recv_dev_buf = MPIDI_cuda_is_device_buf(recvbuf); - int i; - if(is_send_dev_buf) - { - scbuf = MPL_malloc(dt_extent * recvcount * size); - cudaError_t cudaerr = CudaMemcpy(scbuf, sendbuf, dt_extent * recvcount * size, cudaMemcpyDeviceToHost); - if (cudaSuccess != cudaerr) - fprintf(stderr, "cudaMemcpy failed: %s recvbuf: %p scbuf: %p is_send_dev_buf: %d is_recv_dev_buf: %p sendbuf: %p\n", CudaGetErrorString(cudaerr), recvbuf, scbuf, is_send_dev_buf,is_recv_dev_buf, sendbuf ); - } - else - scbuf = sendbuf; - - if(is_recv_dev_buf) - { - rcbuf = MPL_malloc(dt_extent * recvcount * size); - if(sendbuf == MPI_IN_PLACE) - { - cudaError_t cudaerr = CudaMemcpy(rcbuf, recvbuf, dt_extent * recvcount * size, cudaMemcpyDeviceToHost); - if (cudaSuccess != cudaerr) - fprintf(stderr, "cudaMemcpy failed: %s\n", CudaGetErrorString(cudaerr)); - } - else - memset(rcbuf, 0, recvcount * size * dt_extent); - } - else - rcbuf = recvbuf; - - int cuda_res; - if(comm_ptr->comm_kind == MPIR_COMM_KIND__INTRACOMM) - cuda_res = MPIR_Reduce_scatter_block_intra(scbuf, rcbuf, recvcount, datatype, op, comm_ptr, mpierrno); - else - cuda_res = MPIR_Reduce_scatter_block_inter(scbuf, rcbuf, recvcount, datatype, op, comm_ptr, mpierrno); - if(is_send_dev_buf)MPL_free(scbuf); - if(is_recv_dev_buf) - { - cudaError_t cudaerr = CudaMemcpy(recvbuf, rcbuf, dt_extent * recvcount * size, cudaMemcpyHostToDevice); - if (cudaSuccess != cudaerr) - fprintf(stderr, "cudaMemcpy failed: %s recvbuf: %p rcbuf: %p is_send_dev_buf: %d is_recv_dev_buf: %p sendbuf: %p\n", CudaGetErrorString(cudaerr), recvbuf, rcbuf, is_send_dev_buf,is_recv_dev_buf, sendbuf ); - MPL_free(rcbuf); - } - return cuda_res; - } - else -#endif - if(comm_ptr->comm_kind == MPIR_COMM_KIND__INTRACOMM) - return MPIR_Reduce_scatter_block_intra(sendbuf, recvbuf, recvcount, datatype, op, comm_ptr, mpierrno); - else - return MPIR_Reduce_scatter_block_inter(sendbuf, recvbuf, recvcount, datatype, op, comm_ptr, mpierrno); - -} - - diff --git a/src/mpid/pamid/src/coll/reduce/Makefile.mk b/src/mpid/pamid/src/coll/reduce/Makefile.mk deleted file mode 100644 index c962c91..0000000 --- a/src/mpid/pamid/src/coll/reduce/Makefile.mk +++ /dev/null @@ -1,30 +0,0 @@ -# begin_generated_IBM_copyright_prolog -# -# This is an automatically generated copyright prolog. -# After initializing, DO NOT MODIFY OR MOVE -# --------------------------------------------------------------- -# Licensed Materials - Property of IBM -# Blue Gene/Q 5765-PER 5765-PRP -# -# (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved -# US Government Users Restricted Rights - -# Use, duplication, or disclosure restricted -# by GSA ADP Schedule Contract with IBM Corp. -# -# --------------------------------------------------------------- -# -# end_generated_IBM_copyright_prolog -# -*- mode: makefile-gmake; -*- - -# note that the includes always happen but the effects of their contents are -# affected by "if BUILD_PAMID" -if BUILD_PAMID - - -mpi_core_sources += \ - src/mpid/pamid/src/coll/reduce/mpido_reduce.c \ - src/mpid/pamid/src/coll/reduce/mpido_ireduce.c - - -endif BUILD_PAMID - diff --git a/src/mpid/pamid/src/coll/reduce/mpido_ireduce.c b/src/mpid/pamid/src/coll/reduce/mpido_ireduce.c deleted file mode 100644 index e7f7c57..0000000 --- a/src/mpid/pamid/src/coll/reduce/mpido_ireduce.c +++ /dev/null @@ -1,72 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/coll/gather/mpido_ireduce.c - * \brief ??? - */ - -/*#define TRACE_ON*/ -#include <mpidimpl.h> - -int MPIDO_Ireduce(const void *sendbuf, - void *recvbuf, - int count, - MPI_Datatype datatype, - MPI_Op op, - int root, - MPIR_Comm *comm_ptr, - MPIR_Request **request) - -{ - /*if (unlikely((data_size == 0) || (user_selected_type == MPID_COLL_USE_MPICH)))*/ - { - /* - * If the mpich mpir non-blocking collectives are enabled, return without - * first constructing the MPIR_Request. This signals to the - * MPIR_Ireduce_impl() function to invoke the mpich nbc - * implementation of MPI_Ireduce(). - */ - if (MPIDI_Process.mpir_nbc != 0) - return 0; - - /* - * MPIR_* nbc implementation is not enabled. Fake a non-blocking - * MPIR_Ireduce() with a blocking MPIR_Reduce(). - */ - if(unlikely(MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_ALL && comm_ptr->rank == 0)) - fprintf(stderr,"Using MPICH blocking reduce_algorithm\n"); - - int mpierrno = 0; - int rc = MPIR_Reduce_impl(sendbuf, recvbuf, count, datatype, - op, root, comm_ptr, &mpierrno); - - /* - * The blocking reduce has completed - create and complete a - * MPIR_Request object so the MPIR_Ireduce_impl() function - * does not perform an additional ireduce. - */ - MPIR_Request * mpid_request = MPID_Request_create_inline(); - mpid_request->kind = MPIR_REQUEST_KIND__COLL; - *request = mpid_request; - MPIDI_Request_complete_norelease_inline(mpid_request); - - return rc; - } - - return 0; -} diff --git a/src/mpid/pamid/src/coll/reduce/mpido_reduce.c b/src/mpid/pamid/src/coll/reduce/mpido_reduce.c deleted file mode 100644 index b548a63..0000000 --- a/src/mpid/pamid/src/coll/reduce/mpido_reduce.c +++ /dev/null @@ -1,422 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/coll/gather/mpido_reduce.c - * \brief ??? - */ - -/* #define TRACE_ON */ -#include <mpidimpl.h> - -static void reduce_cb_done(void *ctxt, void *clientdata, pami_result_t err) -{ - unsigned *active = (unsigned *)clientdata; - TRACE_ERR("cb_reduce enter, active: %u\n", (*active)); - (*active)--; -} - -int MPIDO_Reduce(const void *sendbuf, - void *recvbuf, - int count, - MPI_Datatype datatype, - MPI_Op op, - int root, - MPIR_Comm *comm_ptr, - int *mpierrno) - -{ -#ifndef HAVE_PAMI_IN_PLACE - if (sendbuf == MPI_IN_PLACE) - { - MPID_Abort (NULL, 0, 1, "'MPI_IN_PLACE' requries support for `PAMI_IN_PLACE`"); - return -1; - } -#endif - MPIDU_Datatype*dt_null = NULL; - MPI_Aint true_lb = 0; - int dt_contig ATTRIBUTE((unused)), tsize; - int mu; - char *sbuf, *rbuf; - pami_data_function pop; - pami_type_t pdt; - int rc; - int alg_selected = 0; - const int rank = comm_ptr->rank; -#if ASSERT_LEVEL==0 - /* We can't afford the tracing in ndebug/performance libraries */ - const unsigned verbose = 0; -#else - const unsigned verbose = (MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_ALL) && (rank == 0); -#endif - const struct MPIDI_Comm* const mpid = &(comm_ptr->mpid); - const int selected_type = mpid->user_selected_type[PAMI_XFER_REDUCE]; - - rc = MPIDI_Datatype_to_pami(datatype, &pdt, op, &pop, &mu); - if(unlikely(verbose)) - fprintf(stderr,"reduce - rc %u, root %u, count %d, dt: %p, op: %p, mu: %u, selectedvar %u != %u (MPICH) sendbuf %p, recvbuf %p\n", - rc, root, count, pdt, pop, mu, - (unsigned)selected_type, MPID_COLL_USE_MPICH,sendbuf, recvbuf); - - pami_xfer_t reduce; - pami_algorithm_t my_reduce=0; - const pami_metadata_t *my_md = (pami_metadata_t *)NULL; - int queryreq = 0; - volatile unsigned reduce_active = 1; - - MPIDI_Datatype_get_info(count, datatype, dt_contig, tsize, dt_null, true_lb); - rbuf = (char *)recvbuf + true_lb; - sbuf = (char *)sendbuf + true_lb; - if(sendbuf == MPI_IN_PLACE) - { - if(unlikely(verbose)) - fprintf(stderr,"reduce MPI_IN_PLACE send buffering (%d,%d)\n",count,tsize); - sbuf = PAMI_IN_PLACE; - } - - reduce.cb_done = reduce_cb_done; - reduce.cookie = (void *)&reduce_active; - if(mpid->optreduce) /* GLUE_ALLREDUCE */ - { - char* tbuf = NULL; - if(unlikely(verbose)) - fprintf(stderr,"Using protocol GLUE_ALLREDUCE for reduce (%d,%d)\n",count,tsize); - MPIDI_Update_last_algorithm(comm_ptr, "REDUCE_OPT_ALLREDUCE"); - void *destbuf = recvbuf; - if(rank != root) /* temp buffer for non-root destbuf */ - { - tbuf = destbuf = MPL_malloc(tsize); - } - /* Switch to comm->coll_fns->fn() */ - MPIDO_Allreduce(sendbuf, - destbuf, - count, - datatype, - op, - comm_ptr, - mpierrno); - if(tbuf) - MPL_free(tbuf); - return 0; - } - if(selected_type == MPID_COLL_USE_MPICH || rc != MPI_SUCCESS) - { - if(unlikely(verbose)) - fprintf(stderr,"Using MPICH reduce algorithm\n"); -#if CUDA_AWARE_SUPPORT - if(MPIDI_Process.cuda_aware_support_on) - { - MPI_Aint dt_extent; - MPIDU_Datatype_get_extent_macro(datatype, dt_extent); - char *scbuf = NULL; - char *rcbuf = NULL; - int is_send_dev_buf = MPIDI_cuda_is_device_buf(sendbuf); - int is_recv_dev_buf = MPIDI_cuda_is_device_buf(recvbuf); - if(is_send_dev_buf) - { - scbuf = MPL_malloc(dt_extent * count); - cudaError_t cudaerr = CudaMemcpy(scbuf, sendbuf, dt_extent * count, cudaMemcpyDeviceToHost); - if (cudaSuccess != cudaerr) - fprintf(stderr, "cudaMemcpy failed: %s\n", CudaGetErrorString(cudaerr)); - } - else - scbuf = sendbuf; - if(is_recv_dev_buf) - { - rcbuf = MPL_malloc(dt_extent * count); - if(sendbuf == MPI_IN_PLACE) - { - cudaError_t cudaerr = CudaMemcpy(rcbuf, recvbuf, dt_extent * count, cudaMemcpyDeviceToHost); - if (cudaSuccess != cudaerr) - fprintf(stderr, "cudaMemcpy failed: %s\n", CudaGetErrorString(cudaerr)); - } - else - memset(rcbuf, 0, dt_extent * count); - } - else - rcbuf = recvbuf; - int cuda_res = MPIR_Reduce(scbuf, rcbuf, count, datatype, op, root, comm_ptr, mpierrno); - if(is_send_dev_buf)MPL_free(scbuf); - if(is_recv_dev_buf) - { - cudaError_t cudaerr = CudaMemcpy(recvbuf, rcbuf, dt_extent * count, cudaMemcpyHostToDevice); - if (cudaSuccess != cudaerr) - fprintf(stderr, "cudaMemcpy failed: %s\n", CudaGetErrorString(cudaerr)); - MPL_free(rcbuf); - } - return cuda_res; - } - else -#endif - return MPIR_Reduce(sendbuf, recvbuf, count, datatype, op, root, comm_ptr, mpierrno); - } - - if(selected_type == MPID_COLL_OPTIMIZED) - { - if((mpid->cutoff_size[PAMI_XFER_REDUCE][0] == 0) || - (mpid->cutoff_size[PAMI_XFER_REDUCE][0] >= tsize && mpid->cutoff_size[PAMI_XFER_REDUCE][0] > 0)) - { - TRACE_ERR("Optimized Reduce (%s) was pre-selected\n", - mpid->opt_protocol_md[PAMI_XFER_REDUCE][0].name); - my_reduce = mpid->opt_protocol[PAMI_XFER_REDUCE][0]; - my_md = &mpid->opt_protocol_md[PAMI_XFER_REDUCE][0]; - queryreq = mpid->must_query[PAMI_XFER_REDUCE][0]; - } - - } - else - { - TRACE_ERR("Optimized reduce (%s) was specified by user\n", - mpid->user_metadata[PAMI_XFER_REDUCE].name); - my_reduce = mpid->user_selected[PAMI_XFER_REDUCE]; - my_md = &mpid->user_metadata[PAMI_XFER_REDUCE]; - queryreq = selected_type; - } - reduce.algorithm = my_reduce; - reduce.cmd.xfer_reduce.sndbuf = sbuf; - reduce.cmd.xfer_reduce.rcvbuf = rbuf; - reduce.cmd.xfer_reduce.stype = pdt; - reduce.cmd.xfer_reduce.rtype = pdt; - reduce.cmd.xfer_reduce.stypecount = count; - reduce.cmd.xfer_reduce.rtypecount = count; - reduce.cmd.xfer_reduce.op = pop; - reduce.cmd.xfer_reduce.root = MPIDI_Task_to_endpoint(MPID_VCR_GET_LPID(comm_ptr->vcr, root), 0); - - - if(unlikely(queryreq == MPID_COLL_ALWAYS_QUERY || - queryreq == MPID_COLL_CHECK_FN_REQUIRED)) - { - metadata_result_t result = {0}; - TRACE_ERR("Querying reduce protocol %s, type was %d\n", - my_md->name, - queryreq); - if(my_md->check_fn == NULL) - { - /* process metadata bits */ - if((!my_md->check_correct.values.inplace) && (sendbuf == MPI_IN_PLACE)) - result.check.unspecified = 1; - if(my_md->check_correct.values.rangeminmax) - { - MPI_Aint data_true_lb ATTRIBUTE((unused)); - MPIDU_Datatype*data_ptr; - int data_size, data_contig ATTRIBUTE((unused)); - MPIDI_Datatype_get_info(count, datatype, data_contig, data_size, data_ptr, data_true_lb); - if((my_md->range_lo <= data_size) && - (my_md->range_hi >= data_size)) - ; /* ok, algorithm selected */ - else - { - result.check.range = 1; - if(unlikely(verbose)) - { - fprintf(stderr,"message size (%u) outside range (%zu<->%zu) for %s.\n", - data_size, - my_md->range_lo, - my_md->range_hi, - my_md->name); - } - } - } - } - else /* calling the check fn is sufficient */ - result = my_md->check_fn(&reduce); - TRACE_ERR("Bitmask: %#X\n", result.bitmask); - result.check.nonlocal = 0; /* #warning REMOVE THIS WHEN IMPLEMENTED */ - if(result.bitmask) - { - if(unlikely(verbose)) - fprintf(stderr,"Query failed for %s. Using MPICH reduce.\n", - my_md->name); - } - else - { - if(my_md->check_correct.values.asyncflowctl && !(--(comm_ptr->mpid.num_requests))) - { - comm_ptr->mpid.num_requests = MPIDI_Process.optimized.num_requests; - int tmpmpierrno; - if(unlikely(verbose)) - fprintf(stderr,"Query barrier required for %s\n", my_md->name); - MPIDO_Barrier(comm_ptr, &tmpmpierrno); - } - alg_selected = 1; - } - } - - if(alg_selected) - { - if(unlikely(verbose)) - { - unsigned long long int threadID; - MPL_thread_id_t tid; - MPL_thread_self(&tid); - threadID = (unsigned long long int)tid; - fprintf(stderr,"<%llx> Using protocol %s for reduce on %u\n", - threadID, - my_md->name, - (unsigned) comm_ptr->context_id); - } - MPIDI_Post_coll_t reduce_post; - MPIDI_Context_post(MPIDI_Context[0], &reduce_post.state, - MPIDI_Pami_post_wrapper, (void *)&reduce); - } - else - { - MPIDI_Update_last_algorithm(comm_ptr, "REDUCE_MPICH"); - if(unlikely(verbose)) - fprintf(stderr,"Using MPICH reduce algorithm\n"); - return MPIR_Reduce(sendbuf, recvbuf, count, datatype, op, root, comm_ptr, mpierrno); - } - - MPIDI_Update_last_algorithm(comm_ptr, - my_md->name); - MPID_PROGRESS_WAIT_WHILE(reduce_active); - TRACE_ERR("Reduce done\n"); - return 0; -} - - -int MPIDO_Reduce_simple(const void *sendbuf, - void *recvbuf, - int count, - MPI_Datatype datatype, - MPI_Op op, - int root, - MPIR_Comm *comm_ptr, - int *mpierrno) - -{ -#ifndef HAVE_PAMI_IN_PLACE - if (sendbuf == MPI_IN_PLACE) - { - MPID_Abort (NULL, 0, 1, "'MPI_IN_PLACE' requries support for `PAMI_IN_PLACE`"); - return -1; - } -#endif - MPIDU_Datatype*dt_null = NULL; - MPI_Aint true_lb = 0; - int dt_contig, tsize; - int mu; - char *sbuf, *rbuf; - pami_data_function pop; - pami_type_t pdt; - int rc; - const int rank = comm_ptr->rank; -#if ASSERT_LEVEL==0 - /* We can't afford the tracing in ndebug/performance libraries */ - const unsigned verbose = 0; -#else - const unsigned verbose = (MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_ALL) && (rank == 0); -#endif - - const struct MPIDI_Comm* const mpid = &(comm_ptr->mpid); - - MPIDI_Datatype_get_info(count, datatype, dt_contig, tsize, dt_null, true_lb); - if(MPIDI_Pamix_collsel_advise != NULL && mpid->collsel_fast_query != NULL) - { - advisor_algorithm_t advisor_algorithms[1]; - int num_algorithms = MPIDI_Pamix_collsel_advise(mpid->collsel_fast_query, PAMI_XFER_REDUCE, tsize, advisor_algorithms, 1); - if(num_algorithms) - { - if(advisor_algorithms[0].algorithm_type == COLLSEL_EXTERNAL_ALGO) - { - return MPIR_Reduce(sendbuf, recvbuf, count, datatype, op, root, comm_ptr, mpierrno); - } - else if(advisor_algorithms[0].metadata && advisor_algorithms[0].metadata->check_correct.values.asyncflowctl && !(--(comm_ptr->mpid.num_requests))) - { - comm_ptr->mpid.num_requests = MPIDI_Process.optimized.num_requests; - int tmpmpierrno; - if(unlikely(verbose)) - fprintf(stderr,"Query barrier required for %s\n", advisor_algorithms[0].metadata->name); - MPIDO_Barrier(comm_ptr, &tmpmpierrno); - } - } - } - - rc = MPIDI_Datatype_to_pami(datatype, &pdt, op, &pop, &mu); - - pami_xfer_t reduce; - const pami_metadata_t *my_reduce_md=NULL; - volatile unsigned reduce_active = 1; - - if(rc != MPI_SUCCESS || !dt_contig) - { - return MPIR_Reduce(sendbuf, recvbuf, count, datatype, op, root, comm_ptr, mpierrno); - } - - - rbuf = (char *)recvbuf + true_lb; - sbuf = (char *)sendbuf + true_lb; - if(sendbuf == MPI_IN_PLACE) - { - sbuf = PAMI_IN_PLACE; - } - - reduce.cb_done = reduce_cb_done; - reduce.cookie = (void *)&reduce_active; - reduce.algorithm = mpid->coll_algorithm[PAMI_XFER_REDUCE][0][0]; - reduce.cmd.xfer_reduce.sndbuf = sbuf; - reduce.cmd.xfer_reduce.rcvbuf = rbuf; - reduce.cmd.xfer_reduce.stype = pdt; - reduce.cmd.xfer_reduce.rtype = pdt; - reduce.cmd.xfer_reduce.stypecount = count; - reduce.cmd.xfer_reduce.rtypecount = count; - reduce.cmd.xfer_reduce.op = pop; - reduce.cmd.xfer_reduce.root = MPIDI_Task_to_endpoint(MPID_VCR_GET_LPID(comm_ptr->vcr, root), 0); - my_reduce_md = &mpid->coll_metadata[PAMI_XFER_REDUCE][0][0]; - - TRACE_ERR("%s reduce, context %d, algoname: %s, exflag: %d\n", MPIDI_Process.context_post.active>0?"Posting":"Invoking", 0, - my_reduce_md->name, exflag); - MPIDI_Post_coll_t reduce_post; - MPIDI_Context_post(MPIDI_Context[0], &reduce_post.state, - MPIDI_Pami_post_wrapper, (void *)&reduce); - TRACE_ERR("Reduce %s\n", MPIDI_Process.context_post.active>0?"posted":"invoked"); - - MPIDI_Update_last_algorithm(comm_ptr, - my_reduce_md->name); - MPID_PROGRESS_WAIT_WHILE(reduce_active); - TRACE_ERR("Reduce done\n"); - return MPI_SUCCESS; -} - - -int -MPIDO_CSWrapper_reduce(pami_xfer_t *reduce, - void *comm) -{ - int mpierrno = 0; - MPIR_Comm *comm_ptr = (MPIR_Comm*)comm; - MPI_Datatype type; - MPI_Op op; - void *sbuf; - MPIDI_coll_check_in_place(reduce->cmd.xfer_reduce.sndbuf, &sbuf); - int rc = MPIDI_Dtpami_to_dtmpi( reduce->cmd.xfer_reduce.stype, - &type, - reduce->cmd.xfer_reduce.op, - &op); - if(rc == -1) return rc; - - - rc = MPIR_Reduce(sbuf, - reduce->cmd.xfer_reduce.rcvbuf, - reduce->cmd.xfer_reduce.rtypecount, type, op, - reduce->cmd.xfer_reduce.root, comm_ptr, &mpierrno); - if(reduce->cb_done && rc == 0) - reduce->cb_done(NULL, reduce->cookie, PAMI_SUCCESS); - return rc; - -} - diff --git a/src/mpid/pamid/src/coll/scan/Makefile.mk b/src/mpid/pamid/src/coll/scan/Makefile.mk deleted file mode 100644 index 88a44e0..0000000 --- a/src/mpid/pamid/src/coll/scan/Makefile.mk +++ /dev/null @@ -1,29 +0,0 @@ -# begin_generated_IBM_copyright_prolog -# -# This is an automatically generated copyright prolog. -# After initializing, DO NOT MODIFY OR MOVE -# --------------------------------------------------------------- -# Licensed Materials - Property of IBM -# Blue Gene/Q 5765-PER 5765-PRP -# -# (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved -# US Government Users Restricted Rights - -# Use, duplication, or disclosure restricted -# by GSA ADP Schedule Contract with IBM Corp. -# -# --------------------------------------------------------------- -# -# end_generated_IBM_copyright_prolog -# -*- mode: makefile-gmake; -*- - -# note that the includes always happen but the effects of their contents are -# affected by "if BUILD_PAMID" -if BUILD_PAMID - - -mpi_core_sources += \ - src/mpid/pamid/src/coll/scan/mpido_scan.c - - -endif BUILD_PAMID - diff --git a/src/mpid/pamid/src/coll/scan/mpido_scan.c b/src/mpid/pamid/src/coll/scan/mpido_scan.c deleted file mode 100644 index ee4c8d2..0000000 --- a/src/mpid/pamid/src/coll/scan/mpido_scan.c +++ /dev/null @@ -1,452 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/coll/gather/mpido_scan.c - * \brief ??? - */ - -/* #define TRACE_ON */ -#include <mpidimpl.h> - -static void scan_cb_done(void *ctxt, void *clientdata, pami_result_t err) -{ - unsigned *active = (unsigned *)clientdata; - TRACE_ERR("cb_scan enter, active: %u\n", (*active)); - (*active)--; -} -int MPIDO_Doscan(const void *sendbuf, void *recvbuf, - int count, MPI_Datatype datatype, - MPI_Op op, MPIR_Comm * comm_ptr, int *mpierrno, int exflag); - - -int MPIDO_Scan(const void *sendbuf, void *recvbuf, - int count, MPI_Datatype datatype, - MPI_Op op, MPIR_Comm * comm_ptr, int *mpierrno) -{ - return MPIDO_Doscan(sendbuf, recvbuf, count, datatype, - op, comm_ptr, mpierrno, 0); -} - -int MPIDO_Iscan(const void *sendbuf, void *recvbuf, - int count, MPI_Datatype datatype, - MPI_Op op, MPIR_Comm * comm_ptr, MPIR_Request **request) -{ - /*if (unlikely((data_size == 0) || (user_selected_type == MPID_COLL_USE_MPICH)))*/ - { - /* - * If the mpich mpir non-blocking collectives are enabled, return without - * first constructing the MPIR_Request. This signals to the - * MPIR_Iscan_impl() function to invoke the mpich nbc - * implementation of MPI_Iscan(). - */ - if (MPIDI_Process.mpir_nbc != 0) - return 0; - - /* - * MPIR_* nbc implementation is not enabled. Fake a non-blocking - * MPIR_Iscan() with a blocking MPIR_Scan(). - */ - if(unlikely(MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_ALL && comm_ptr->rank == 0)) - fprintf(stderr,"Using MPICH blocking scan_algorithm\n"); - - int mpierrno = 0; - int rc = MPIR_Scan_impl(sendbuf, recvbuf, count, datatype, - op, comm_ptr, &mpierrno); - - /* - * The blocking scan has completed - create and complete a - * MPIR_Request object so the MPIR_Iscan_impl() function - * does not perform an additional iscan. - */ - MPIR_Request * mpid_request = MPID_Request_create_inline(); - mpid_request->kind = MPIR_REQUEST_KIND__COLL; - *request = mpid_request; - MPIDI_Request_complete_norelease_inline(mpid_request); - - return rc; - } - - return 0; -} - -int MPIDO_Exscan(const void *sendbuf, void *recvbuf, - int count, MPI_Datatype datatype, - MPI_Op op, MPIR_Comm * comm_ptr, int *mpierrno) -{ - return MPIDO_Doscan(sendbuf, recvbuf, count, datatype, - op, comm_ptr, mpierrno, 1); -} - -int MPIDO_Doscan(const void *sendbuf, void *recvbuf, - int count, MPI_Datatype datatype, - MPI_Op op, MPIR_Comm * comm_ptr, int *mpierrno, int exflag) -{ -#ifndef HAVE_PAMI_IN_PLACE - if (sendbuf == MPI_IN_PLACE) - { - MPID_Abort (NULL, 0, 1, "'MPI_IN_PLACE' requries support for `PAMI_IN_PLACE`"); - return -1; - } -#endif - MPIDU_Datatype*dt_null = NULL; - MPI_Aint true_lb = 0; - int dt_contig ATTRIBUTE((unused)), tsize ATTRIBUTE((unused)); - int mu; - char *sbuf, *rbuf; - pami_data_function pop; - pami_type_t pdt; - int rc; - const pami_metadata_t *my_md = (pami_metadata_t *)NULL; - int queryreq = 0; -#if ASSERT_LEVEL==0 - /* We can't afford the tracing in ndebug/performance libraries */ - const unsigned verbose = 0; -#else - const unsigned verbose = (MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_ALL) && (comm_ptr->rank == 0); -#endif - const struct MPIDI_Comm* const mpid = &(comm_ptr->mpid); - const int selected_type = mpid->user_selected_type[PAMI_XFER_SCAN]; - - rc = MPIDI_Datatype_to_pami(datatype, &pdt, op, &pop, &mu); - if(unlikely(verbose)) - fprintf(stderr,"rc %u, dt: %p, op: %p, mu: %u, selectedvar %u != %u (MPICH)\n", - rc, pdt, pop, mu, - (unsigned)selected_type, MPID_COLL_USE_MPICH); - - pami_xfer_t scan; - volatile unsigned scan_active = 1; - - if((selected_type == MPID_COLL_USE_MPICH || rc != MPI_SUCCESS)) - { - if(unlikely(verbose)) - fprintf(stderr,"Using MPICH scan algorithm (exflag %d)\n",exflag); -#if CUDA_AWARE_SUPPORT - if(MPIDI_Process.cuda_aware_support_on) - { - MPI_Aint dt_extent; - MPIDU_Datatype_get_extent_macro(datatype, dt_extent); - char *scbuf = NULL; - char *rcbuf = NULL; - int is_send_dev_buf = MPIDI_cuda_is_device_buf(sendbuf); - int is_recv_dev_buf = MPIDI_cuda_is_device_buf(recvbuf); - if(is_send_dev_buf) - { - scbuf = MPL_malloc(dt_extent * count); - cudaError_t cudaerr = CudaMemcpy(scbuf, sendbuf, dt_extent * count, cudaMemcpyDeviceToHost); - if (cudaSuccess != cudaerr) - fprintf(stderr, "cudaMemcpy failed: %s\n", CudaGetErrorString(cudaerr)); - } - else - scbuf = sendbuf; - if(is_recv_dev_buf) - { - rcbuf = MPL_malloc(dt_extent * count); - if(sendbuf == MPI_IN_PLACE) - { - cudaError_t cudaerr = CudaMemcpy(rcbuf, recvbuf, dt_extent * count, cudaMemcpyDeviceToHost); - if (cudaSuccess != cudaerr) - fprintf(stderr, "cudaMemcpy failed: %s\n", CudaGetErrorString(cudaerr)); - } - else - memset(rcbuf, 0, dt_extent * count); - } - else - rcbuf = recvbuf; - int cuda_res; - if(exflag) - cuda_res = MPIR_Exscan(scbuf, rcbuf, count, datatype, op, comm_ptr, mpierrno); - else - cuda_res = MPIR_Scan(scbuf, rcbuf, count, datatype, op, comm_ptr, mpierrno); - if(is_send_dev_buf)MPL_free(scbuf); - if(is_recv_dev_buf) - { - cudaError_t cudaerr = CudaMemcpy(recvbuf, rcbuf, dt_extent * count, cudaMemcpyHostToDevice); - if (cudaSuccess != cudaerr) - fprintf(stderr, "cudaMemcpy failed: %s\n", CudaGetErrorString(cudaerr)); - MPL_free(rcbuf); - } - return cuda_res; - } - else -#endif - if(exflag) - return MPIR_Exscan(sendbuf, recvbuf, count, datatype, op, comm_ptr, mpierrno); - else - return MPIR_Scan(sendbuf, recvbuf, count, datatype, op, comm_ptr, mpierrno); - } - - MPIDI_Datatype_get_info(count, datatype, dt_contig, tsize, dt_null, true_lb); - rbuf = (char *)recvbuf + true_lb; - if(sendbuf == MPI_IN_PLACE) - { - if(unlikely(verbose)) - fprintf(stderr,"scan MPI_IN_PLACE buffering\n"); - sbuf = PAMI_IN_PLACE; - } - else - { - sbuf = (char *)sendbuf + true_lb; - } - - scan.cb_done = scan_cb_done; - scan.cookie = (void *)&scan_active; - if(selected_type == MPID_COLL_OPTIMIZED) - { - scan.algorithm = mpid->opt_protocol[PAMI_XFER_SCAN][0]; - my_md = &mpid->opt_protocol_md[PAMI_XFER_SCAN][0]; - queryreq = mpid->must_query[PAMI_XFER_SCAN][0]; - } - else - { - scan.algorithm = mpid->user_selected[PAMI_XFER_SCAN]; - my_md = &mpid->user_metadata[PAMI_XFER_SCAN]; - queryreq = selected_type; - } - scan.cmd.xfer_scan.sndbuf = sbuf; - scan.cmd.xfer_scan.rcvbuf = rbuf; - scan.cmd.xfer_scan.stype = pdt; - scan.cmd.xfer_scan.rtype = pdt; - scan.cmd.xfer_scan.stypecount = count; - scan.cmd.xfer_scan.rtypecount = count; - scan.cmd.xfer_scan.op = pop; - scan.cmd.xfer_scan.exclusive = exflag; - - - if(unlikely(queryreq == MPID_COLL_ALWAYS_QUERY || - queryreq == MPID_COLL_CHECK_FN_REQUIRED)) - { - metadata_result_t result = {0}; - TRACE_ERR("Querying scan protocol %s, type was %d\n", - my_md->name, - selected_type); - if(my_md->check_fn == NULL) - { - /* process metadata bits */ - if((!my_md->check_correct.values.inplace) && (sendbuf == MPI_IN_PLACE)) - result.check.unspecified = 1; - if(my_md->check_correct.values.rangeminmax) - { - MPI_Aint data_true_lb ATTRIBUTE((unused)); - MPIDU_Datatype*data_ptr; - int data_size ATTRIBUTE((unused)), data_contig ATTRIBUTE((unused)); - MPIDI_Datatype_get_info(count, datatype, data_contig, data_size, data_ptr, data_true_lb); - if((my_md->range_lo <= data_size) && - (my_md->range_hi >= data_size)) - ; /* ok, algorithm selected */ - else - { - result.check.range = 1; - if(unlikely(verbose)) - { - fprintf(stderr,"message size (%u) outside range (%zu<->%zu) for %s.\n", - data_size, - my_md->range_lo, - my_md->range_hi, - my_md->name); - } - } - } - } - else /* calling the check fn is sufficient */ - result = my_md->check_fn(&scan); - TRACE_ERR("Bitmask: %#X\n", result.bitmask); - result.check.nonlocal = 0; /* #warning REMOVE THIS WHEN IMPLEMENTED */ - if(result.bitmask) - { - if(unlikely(verbose)) - fprintf(stderr,"Query failed for %s. Using MPICH scan\n", - my_md->name); - MPIDI_Update_last_algorithm(comm_ptr, "SCAN_MPICH"); - if(exflag) - return MPIR_Exscan(sendbuf, recvbuf, count, datatype, op, comm_ptr, mpierrno); - else - return MPIR_Scan(sendbuf, recvbuf, count, datatype, op, comm_ptr, mpierrno); - } - if(my_md->check_correct.values.asyncflowctl && !(--(comm_ptr->mpid.num_requests))) - { - comm_ptr->mpid.num_requests = MPIDI_Process.optimized.num_requests; - int tmpmpierrno; - if(unlikely(verbose)) - fprintf(stderr,"Query barrier required for %s\n", my_md->name); - MPIDO_Barrier(comm_ptr, &tmpmpierrno); - } - } - - if(unlikely(verbose)) - { - unsigned long long int threadID; - MPL_thread_id_t tid; - MPL_thread_self(&tid); - threadID = (unsigned long long int)tid; - fprintf(stderr,"<%llx> Using protocol %s for scan on %u (exflag %d)\n", - threadID, - my_md->name, - (unsigned) comm_ptr->context_id, - exflag); - } - MPIDI_Post_coll_t scan_post; - MPIDI_Context_post(MPIDI_Context[0], &scan_post.state, - MPIDI_Pami_post_wrapper, (void *)&scan); - MPIDI_Update_last_algorithm(comm_ptr, my_md->name); - MPID_PROGRESS_WAIT_WHILE(scan_active); - TRACE_ERR("Scan done\n"); - return rc; -} - - -int MPIDO_Doscan_simple(const void *sendbuf, void *recvbuf, - int count, MPI_Datatype datatype, - MPI_Op op, MPIR_Comm * comm_ptr, int *mpierrno, int exflag) -{ -#ifndef HAVE_PAMI_IN_PLACE - if (sendbuf == MPI_IN_PLACE) - { - MPID_Abort (NULL, 0, 1, "'MPI_IN_PLACE' requries support for `PAMI_IN_PLACE`"); - return -1; - } -#endif - MPIDU_Datatype*dt_null = NULL; - MPI_Aint true_lb = 0; - int dt_contig, tsize; - int mu; - char *sbuf, *rbuf; - pami_data_function pop; - pami_type_t pdt; - int rc; - const pami_metadata_t *my_md; - - const struct MPIDI_Comm* const mpid = &(comm_ptr->mpid); - - pami_xfer_t scan; - volatile unsigned scan_active = 1; - MPIDI_Datatype_get_info(count, datatype, dt_contig, tsize, dt_null, true_lb); - if(MPIDI_Pamix_collsel_advise != NULL && mpid->collsel_fast_query != NULL) - { - advisor_algorithm_t advisor_algorithms[1]; - int num_algorithms = MPIDI_Pamix_collsel_advise(mpid->collsel_fast_query, PAMI_XFER_SCAN, tsize, advisor_algorithms, 1); - if(num_algorithms) - { - if(advisor_algorithms[0].algorithm_type == COLLSEL_EXTERNAL_ALGO) - { - if(exflag) - return MPIR_Exscan(sendbuf, recvbuf, count, datatype, op, comm_ptr, mpierrno); - else - return MPIR_Scan(sendbuf, recvbuf, count, datatype, op, comm_ptr, mpierrno); - } - else if(advisor_algorithms[0].metadata && advisor_algorithms[0].metadata->check_correct.values.asyncflowctl && !(--(comm_ptr->mpid.num_requests))) - { - comm_ptr->mpid.num_requests = MPIDI_Process.optimized.num_requests; - int tmpmpierrno; - MPIDO_Barrier(comm_ptr, &tmpmpierrno); - } - } - } - - rc = MPIDI_Datatype_to_pami(datatype, &pdt, op, &pop, &mu); - - if(rc != MPI_SUCCESS || !dt_contig) - { - if(exflag) - return MPIR_Exscan(sendbuf, recvbuf, count, datatype, op, comm_ptr, mpierrno); - else - return MPIR_Scan(sendbuf, recvbuf, count, datatype, op, comm_ptr, mpierrno); - } - - - rbuf = (char *)recvbuf + true_lb; - if(sendbuf == MPI_IN_PLACE) - { - sbuf = PAMI_IN_PLACE; - } - else - { - sbuf = (char *)sendbuf + true_lb; - } - - scan.cb_done = scan_cb_done; - scan.cookie = (void *)&scan_active; - scan.algorithm = mpid->coll_algorithm[PAMI_XFER_SCAN][0][0]; - my_md = &mpid->coll_metadata[PAMI_XFER_SCAN][0][0]; - scan.cmd.xfer_scan.sndbuf = sbuf; - scan.cmd.xfer_scan.rcvbuf = rbuf; - scan.cmd.xfer_scan.stype = pdt; - scan.cmd.xfer_scan.rtype = pdt; - scan.cmd.xfer_scan.stypecount = count; - scan.cmd.xfer_scan.rtypecount = count; - scan.cmd.xfer_scan.op = pop; - scan.cmd.xfer_scan.exclusive = exflag; - - MPIDI_Post_coll_t scan_post; - MPIDI_Context_post(MPIDI_Context[0], &scan_post.state, - MPIDI_Pami_post_wrapper, (void *)&scan); - TRACE_ERR("Scan %s\n", MPIDI_Process.context_post.active>0?"posted":"invoked"); - MPIDI_Update_last_algorithm(comm_ptr, my_md->name); - MPID_PROGRESS_WAIT_WHILE(scan_active); - TRACE_ERR("Scan done\n"); - return rc; -} - - -int MPIDO_Exscan_simple(const void *sendbuf, void *recvbuf, - int count, MPI_Datatype datatype, - MPI_Op op, MPIR_Comm * comm_ptr, int *mpierrno) -{ - return MPIDO_Doscan_simple(sendbuf, recvbuf, count, datatype, - op, comm_ptr, mpierrno, 1); -} - -int MPIDO_Scan_simple(const void *sendbuf, void *recvbuf, - int count, MPI_Datatype datatype, - MPI_Op op, MPIR_Comm * comm_ptr, int *mpierrno) -{ - return MPIDO_Doscan_simple(sendbuf, recvbuf, count, datatype, - op, comm_ptr, mpierrno, 0); -} - -int -MPIDO_CSWrapper_scan(pami_xfer_t *scan, - void *comm) -{ - int mpierrno = 0; - MPIR_Comm *comm_ptr = (MPIR_Comm*)comm; - MPI_Datatype type; - MPI_Op op; - void *sbuf; - MPIDI_coll_check_in_place(scan->cmd.xfer_scan.sndbuf, &sbuf); - int rc = MPIDI_Dtpami_to_dtmpi( scan->cmd.xfer_scan.stype, - &type, - scan->cmd.xfer_scan.op, - &op); - if(rc == -1) return rc; - - if(scan->cmd.xfer_scan.exclusive) - rc = MPIR_Exscan(sbuf, - scan->cmd.xfer_scan.rcvbuf, - scan->cmd.xfer_scan.rtypecount, type, op, - comm_ptr, &mpierrno); - else - rc = MPIR_Scan(sbuf, - scan->cmd.xfer_scan.rcvbuf, - scan->cmd.xfer_scan.rtypecount, type, op, - comm_ptr, &mpierrno); - if(scan->cb_done && rc == 0) - scan->cb_done(NULL, scan->cookie, PAMI_SUCCESS); - return rc; - -} - diff --git a/src/mpid/pamid/src/coll/scatter/Makefile.mk b/src/mpid/pamid/src/coll/scatter/Makefile.mk deleted file mode 100644 index 9cf6067..0000000 --- a/src/mpid/pamid/src/coll/scatter/Makefile.mk +++ /dev/null @@ -1,30 +0,0 @@ -# begin_generated_IBM_copyright_prolog -# -# This is an automatically generated copyright prolog. -# After initializing, DO NOT MODIFY OR MOVE -# --------------------------------------------------------------- -# Licensed Materials - Property of IBM -# Blue Gene/Q 5765-PER 5765-PRP -# -# (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved -# US Government Users Restricted Rights - -# Use, duplication, or disclosure restricted -# by GSA ADP Schedule Contract with IBM Corp. -# -# --------------------------------------------------------------- -# -# end_generated_IBM_copyright_prolog -# -*- mode: makefile-gmake; -*- - -# note that the includes always happen but the effects of their contents are -# affected by "if BUILD_PAMID" -if BUILD_PAMID - - -mpi_core_sources += \ - src/mpid/pamid/src/coll/scatter/mpido_scatter.c \ - src/mpid/pamid/src/coll/scatter/mpido_iscatter.c - - -endif BUILD_PAMID - diff --git a/src/mpid/pamid/src/coll/scatter/mpido_iscatter.c b/src/mpid/pamid/src/coll/scatter/mpido_iscatter.c deleted file mode 100644 index 9d11d91..0000000 --- a/src/mpid/pamid/src/coll/scatter/mpido_iscatter.c +++ /dev/null @@ -1,74 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/coll/scatter/mpido_iscatter.c - * \brief ??? - */ - -/*#define TRACE_ON */ - -#include <mpidimpl.h> - -int MPIDO_Iscatter(const void *sendbuf, - int sendcount, - MPI_Datatype sendtype, - void *recvbuf, - int recvcount, - MPI_Datatype recvtype, - int root, - MPIR_Comm *comm_ptr, - MPIR_Request **request) -{ - /*if (unlikely((data_size == 0) || (user_selected_type == MPID_COLL_USE_MPICH)))*/ - { - /* - * If the mpich mpir non-blocking collectives are enabled, return without - * first constructing the MPIR_Request. This signals to the - * MPIR_Iscatter_impl() function to invoke the mpich nbc - * implementation of MPI_Iscatter(). - */ - if (MPIDI_Process.mpir_nbc != 0) - return 0; - - /* - * MPIR_* nbc implementation is not enabled. Fake a non-blocking - * MPIR_Iscatter() with a blocking MPIR_Scatter(). - */ - if(unlikely(MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_ALL && comm_ptr->rank == 0)) - fprintf(stderr,"Using MPICH blocking scatter_algorithm\n"); - - int mpierrno = 0; - int rc = MPIR_Scatter_impl(sendbuf, sendcount, sendtype, - recvbuf, recvcount, recvtype, - root, comm_ptr, &mpierrno); - - /* - * The blocking scatter has completed - create and complete a - * MPIR_Request object so the MPIR_Iscatter_impl() function - * does not perform an additional iscatter. - */ - MPIR_Request * mpid_request = MPID_Request_create_inline(); - mpid_request->kind = MPIR_REQUEST_KIND__COLL; - *request = mpid_request; - MPIDI_Request_complete_norelease_inline(mpid_request); - - return rc; - } - - return 0; -} diff --git a/src/mpid/pamid/src/coll/scatter/mpido_scatter.c b/src/mpid/pamid/src/coll/scatter/mpido_scatter.c deleted file mode 100644 index 038da9f..0000000 --- a/src/mpid/pamid/src/coll/scatter/mpido_scatter.c +++ /dev/null @@ -1,591 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/coll/scatter/mpido_scatter.c - * \brief ??? - */ - -/* #define TRACE_ON */ - -#include <mpidimpl.h> - -static void cb_scatter(void *ctxt, void *clientdata, pami_result_t err) -{ - unsigned *active = (unsigned *)clientdata; - TRACE_ERR("cb_scatter enter, active: %u\n", (*active)); - MPIDI_Progress_signal(); - (*active)--; -} - -/* works for simple data types, assumes fast bcast is available */ -int MPIDO_Scatter_bcast(void * sendbuf, - int sendcount, - MPI_Datatype sendtype, - void *recvbuf, - int recvcount, - MPI_Datatype recvtype, - int root, - MPIR_Comm *comm_ptr, - int *mpierrno) -{ - /* Pretty simple - bcast a temp buffer and copy our little chunk out */ - int nbytes, rc; - const int rank = comm_ptr->rank; - const int size = comm_ptr->local_size; - char *tempbuf = NULL; - - if(rank == root) - { - MPIDI_Datatype_get_data_size(sendcount, - sendtype, - nbytes); - tempbuf = sendbuf; - } - else - { - MPIDI_Datatype_get_data_size(recvcount, - recvtype, - nbytes); - - tempbuf = MPL_malloc(nbytes * size); - if(!tempbuf) - { - return MPIR_Err_create_code(MPI_SUCCESS, - MPIR_ERR_RECOVERABLE, - __FUNCTION__, - __LINE__, - MPI_ERR_OTHER, - "**nomem", 0); - } - } - - /* Switch to comm->coll_fns->fn() */ - rc = MPIDO_Bcast(tempbuf, nbytes*size, MPI_CHAR, root, comm_ptr, mpierrno); - - if(rank == root && recvbuf == MPI_IN_PLACE) - return rc; - else - memcpy(recvbuf, tempbuf+(rank*nbytes), nbytes); - - if (rank!=root) - MPL_free(tempbuf); - - return rc; -} - - -int MPIDO_Scatter(const void *sendbuf, - int sendcount, - MPI_Datatype sendtype, - void *recvbuf, - int recvcount, - MPI_Datatype recvtype, - int root, - MPIR_Comm *comm_ptr, - int *mpierrno) -{ -#ifndef HAVE_PAMI_IN_PLACE - if (sendbuf == MPI_IN_PLACE) - { - MPID_Abort (NULL, 0, 1, "'MPI_IN_PLACE' requries support for `PAMI_IN_PLACE`"); - return -1; - } -#endif - MPIDU_Datatype* data_ptr; - MPI_Aint true_lb ATTRIBUTE((unused)); - int contig, nbytes ATTRIBUTE((unused)); - const int rank = comm_ptr->rank; - int success = 1; - pami_type_t stype, rtype; - int tmp; -#if ASSERT_LEVEL==0 - /* We can't afford the tracing in ndebug/performance libraries */ - const unsigned verbose = 0; -#else - const unsigned verbose = (MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_ALL) && (rank == 0); -#endif - const struct MPIDI_Comm* const mpid = &(comm_ptr->mpid); - const int selected_type = mpid->user_selected_type[PAMI_XFER_SCATTER]; - char use_pami = !(selected_type == MPID_COLL_USE_MPICH); - - /* if (rank == root) - We can't decide on just the root to use MPICH. Really need a pre-allreduce. - For now check sendtype on non-roots too and hope it matches? I think that's what - scatterv does... */ - { - if(MPIDI_Datatype_to_pami(sendtype, &stype, -1, NULL, &tmp) != MPI_SUCCESS) - use_pami = 0; - } - if(recvbuf != MPI_IN_PLACE && (MPIDI_Datatype_to_pami(recvtype, &rtype, -1, NULL, &tmp) != MPI_SUCCESS)) - use_pami = 0; - - if(!use_pami) - { - if(unlikely(verbose)) - fprintf(stderr,"Using MPICH scatter algorithm\n"); - MPIDI_Update_last_algorithm(comm_ptr, "SCATTER_MPICH"); -#if CUDA_AWARE_SUPPORT - if(MPIDI_Process.cuda_aware_support_on) - { - MPI_Aint sdt_extent,rdt_extent; - MPIDU_Datatype_get_extent_macro(sendtype, sdt_extent); - MPIDU_Datatype_get_extent_macro(recvtype, rdt_extent); - char *scbuf = NULL; - char *rcbuf = NULL; - int is_send_dev_buf = (rank == root) ? MPIDI_cuda_is_device_buf(sendbuf) : 0; - int is_recv_dev_buf = MPIDI_cuda_is_device_buf(recvbuf); - if(is_send_dev_buf) - { - scbuf = MPL_malloc(sdt_extent * sendcount); - cudaError_t cudaerr = CudaMemcpy(scbuf, sendbuf, sdt_extent * sendcount, cudaMemcpyDeviceToHost); - if (cudaSuccess != cudaerr) - fprintf(stderr, "cudaMemcpy failed: %s\n", CudaGetErrorString(cudaerr)); - } - else - scbuf = sendbuf; - if(is_recv_dev_buf) - { - rcbuf = MPL_malloc(rdt_extent * recvcount); - CudaMemcpy(rcbuf, recvbuf, rdt_extent * recvcount, cudaMemcpyDeviceToHost); - } - else - rcbuf = recvbuf; - int cuda_res = MPIR_Scatter(scbuf, sendcount, sendtype, rcbuf, recvcount, recvtype, root, comm_ptr, mpierrno); - if(is_send_dev_buf)MPL_free(scbuf); - if(is_recv_dev_buf) - { - cudaError_t cudaerr = CudaMemcpy(recvbuf, rcbuf, rdt_extent * recvcount, cudaMemcpyHostToDevice); - if (cudaSuccess != cudaerr) - fprintf(stderr, "cudaMemcpy failed: %s\n", CudaGetErrorString(cudaerr)); - MPL_free(rcbuf); - } - return cuda_res; - } - else -#endif - return MPIR_Scatter(sendbuf, sendcount, sendtype, - recvbuf, recvcount, recvtype, - root, comm_ptr, mpierrno); - } - - if (rank == root) - { - if (recvtype != MPI_DATATYPE_NULL && recvcount >= 0)/* Should this be send or recv? */ - { - MPIDI_Datatype_get_info(sendcount, sendtype, contig, - nbytes, data_ptr, true_lb); - if (!contig) success = 0; - } - else - success = 0; - - if (success) - { - if (recvtype != MPI_DATATYPE_NULL && recvcount >= 0) - { - MPIDI_Datatype_get_info(recvcount, recvtype, contig, - nbytes, data_ptr, true_lb); - if (!contig) success = 0; - } - else success = 0; - } - } - - else - { - if (sendtype != MPI_DATATYPE_NULL && sendcount >= 0)/* Should this be send or recv? */ - { - MPIDI_Datatype_get_info(recvcount, recvtype, contig, - nbytes, data_ptr, true_lb); - if (!contig) success = 0; - } - else - success = 0; - } - - - pami_xfer_t scatter; - MPIDI_Post_coll_t scatter_post; - pami_algorithm_t my_scatter; - const pami_metadata_t *my_md = (pami_metadata_t *)NULL; - volatile unsigned scatter_active = 1; - int queryreq = 0; - - if(selected_type == MPID_COLL_OPTIMIZED) - { - TRACE_ERR("Optimized scatter %s was selected\n", - mpid->opt_protocol_md[PAMI_XFER_SCATTER][0].name); - my_scatter = mpid->opt_protocol[PAMI_XFER_SCATTER][0]; - my_md = &mpid->opt_protocol_md[PAMI_XFER_SCATTER][0]; - queryreq = mpid->must_query[PAMI_XFER_SCATTER][0]; - } - else - { - TRACE_ERR("Optimized scatter %s was set by user\n", - mpid->user_metadata[PAMI_XFER_SCATTER].name); - my_scatter = mpid->user_selected[PAMI_XFER_SCATTER]; - my_md = &mpid->user_metadata[PAMI_XFER_SCATTER]; - queryreq = selected_type; - } - - scatter.algorithm = my_scatter; - scatter.cb_done = cb_scatter; - scatter.cookie = (void *)&scatter_active; - scatter.cmd.xfer_scatter.root = MPIDI_Task_to_endpoint(MPID_VCR_GET_LPID(comm_ptr->vcr, root), 0); - scatter.cmd.xfer_scatter.sndbuf = (void *)sendbuf; - scatter.cmd.xfer_scatter.stype = stype; - scatter.cmd.xfer_scatter.stypecount = sendcount; - if(recvbuf == MPI_IN_PLACE) - { - if(unlikely(verbose)) - fprintf(stderr,"scatter MPI_IN_PLACE buffering\n"); - MPIDI_Datatype_get_info(sendcount, sendtype, contig, - nbytes, data_ptr, true_lb); - scatter.cmd.xfer_scatter.rcvbuf = PAMI_IN_PLACE; - scatter.cmd.xfer_scatter.rtype = stype; - scatter.cmd.xfer_scatter.rtypecount = sendcount; - } - else - { - scatter.cmd.xfer_scatter.rcvbuf = (void *)recvbuf; - scatter.cmd.xfer_scatter.rtype = rtype; - scatter.cmd.xfer_scatter.rtypecount = recvcount; - } - - if(unlikely(queryreq == MPID_COLL_ALWAYS_QUERY || - queryreq == MPID_COLL_CHECK_FN_REQUIRED)) - { - metadata_result_t result = {0}; - TRACE_ERR("querying scatter protoocl %s, type was %d\n", - my_md->name, queryreq); - if(my_md->check_fn == NULL) - { - /* process metadata bits */ - if((!my_md->check_correct.values.inplace) && (recvbuf == MPI_IN_PLACE)) - result.check.unspecified = 1; - if(my_md->check_correct.values.rangeminmax) - { - int data_size; - if(rank == root) - MPIDI_Datatype_get_data_size(sendcount, sendtype, data_size); - else - MPIDI_Datatype_get_data_size(recvcount, recvtype, data_size); - if((my_md->range_lo <= data_size) && - (my_md->range_hi >= data_size)) - ; /* ok, algorithm selected */ - else - { - result.check.range = 1; - if(unlikely(verbose)) - { - fprintf(stderr,"message size (%u) outside range (%zu<->%zu) for %s.\n", - data_size, - my_md->range_lo, - my_md->range_hi, - my_md->name); - } - } - } - } - else /* calling the check fn is sufficient */ - result = my_md->check_fn(&scatter); - TRACE_ERR("bitmask: %#X\n", result.bitmask); - result.check.nonlocal = 0; /* #warning REMOVE THIS WHEN IMPLEMENTED */ - if(result.bitmask) - { - if(unlikely(verbose)) - fprintf(stderr,"query failed for %s\n", my_md->name); - MPIDI_Update_last_algorithm(comm_ptr, "SCATTER_MPICH"); - return MPIR_Scatter(sendbuf, sendcount, sendtype, - recvbuf, recvcount, recvtype, - root, comm_ptr, mpierrno); - } - if(my_md->check_correct.values.asyncflowctl && !(--(comm_ptr->mpid.num_requests))) - { - comm_ptr->mpid.num_requests = MPIDI_Process.optimized.num_requests; - int tmpmpierrno; - if(unlikely(verbose)) - fprintf(stderr,"Query barrier required for %s\n", my_md->name); - MPIDO_Barrier(comm_ptr, &tmpmpierrno); - } - } - - if(unlikely(verbose)) - { - unsigned long long int threadID; - MPL_thread_id_t tid; - MPL_thread_self(&tid); - threadID = (unsigned long long int)tid; - fprintf(stderr,"<%llx> Using protocol %s for scatter on %u\n", - threadID, - my_md->name, - (unsigned) comm_ptr->context_id); - } - MPIDI_Context_post(MPIDI_Context[0], &scatter_post.state, - MPIDI_Pami_post_wrapper, (void *)&scatter); - TRACE_ERR("Waiting on active %d\n", scatter_active); - MPID_PROGRESS_WAIT_WHILE(scatter_active); - - - TRACE_ERR("Leaving MPIDO_Scatter\n"); - - return 0; -} - - -int MPIDO_Scatter_simple(const void *sendbuf, - int sendcount, - MPI_Datatype sendtype, - void *recvbuf, - int recvcount, - MPI_Datatype recvtype, - int root, - MPIR_Comm *comm_ptr, - int *mpierrno) -{ -#ifndef HAVE_PAMI_IN_PLACE - if (sendbuf == MPI_IN_PLACE) - { - MPID_Abort (NULL, 0, 1, "'MPI_IN_PLACE' requries support for `PAMI_IN_PLACE`"); - return -1; - } -#endif - MPIDU_Datatype* data_ptr; - const int rank = comm_ptr->rank; - int success = 1, snd_contig = 1, rcv_contig = 1; - void *snd_noncontig_buff = NULL, *rcv_noncontig_buff = NULL; - void *sbuf = NULL, *rbuf = NULL; - size_t send_size = 0, recv_size = 0, data_size = 0; - MPI_Aint snd_true_lb = 0, rcv_true_lb = 0; - MPIDU_Segment segment; - const struct MPIDI_Comm* const mpid = &(comm_ptr->mpid); - const int size = comm_ptr->local_size; - advisor_algorithm_t advisor_algorithms[1]; - - if (rank == root) - { - MPIDI_Datatype_get_info(sendcount, sendtype, snd_contig, - send_size, data_ptr, snd_true_lb); - - if (recvbuf != MPI_IN_PLACE && recvtype != MPI_DATATYPE_NULL && recvcount >= 0) - { - MPIDI_Datatype_get_info(recvcount, recvtype, rcv_contig, - recv_size, data_ptr, rcv_true_lb); - } - data_size = send_size; - } - else if (recvtype != MPI_DATATYPE_NULL && recvcount >= 0) - { - MPIDI_Datatype_get_info(recvcount, recvtype, rcv_contig, - recv_size, data_ptr, rcv_true_lb); - data_size = recv_size; - } - - advisor_algorithms[0].metadata = NULL;/* We check for NULL further down */ - if(MPIDI_Pamix_collsel_advise != NULL && mpid->collsel_fast_query != NULL) - { - int num_algorithms = MPIDI_Pamix_collsel_advise(mpid->collsel_fast_query, - PAMI_XFER_SCATTER, data_size, advisor_algorithms, 1); - if(num_algorithms) - { - if(advisor_algorithms[0].algorithm_type == COLLSEL_EXTERNAL_ALGO) - { - return MPIR_Scatter(sendbuf, sendcount, sendtype, - recvbuf, recvcount, recvtype, - root, comm_ptr, mpierrno); - } - } - } - sbuf = (char *)sendbuf + snd_true_lb; - rbuf = (char *)recvbuf + rcv_true_lb; - if (rank == root) - { - if (send_size) - { - if (!snd_contig) - { - snd_noncontig_buff = MPL_malloc(send_size * size); - sbuf = snd_noncontig_buff; - if(snd_noncontig_buff == NULL) - { - MPID_Abort(NULL, MPI_ERR_NO_SPACE, 1, - "Fatal: Cannot allocate pack buffer"); - } - DLOOP_Offset last = send_size * size; - MPIDU_Segment_init(sendbuf, sendcount * size, sendtype, &segment, 0); - MPIDU_Segment_pack(&segment, 0, &last, snd_noncontig_buff); - } - } - else - success = 0; - - if (success && recvbuf != MPI_IN_PLACE) - { - if (recv_size) - { - if (!rcv_contig) - { - rcv_noncontig_buff = MPL_malloc(recv_size); - rbuf = rcv_noncontig_buff; - if(rcv_noncontig_buff == NULL) - { - MPID_Abort(NULL, MPI_ERR_NO_SPACE, 1, - "Fatal: Cannot allocate pack buffer"); - } - } - } - else success = 0; - } - } - else - { - if (recv_size)/* Should this be send or recv? */ - { - if (!rcv_contig) - { - rcv_noncontig_buff = MPL_malloc(recv_size); - rbuf = rcv_noncontig_buff; - if(rcv_noncontig_buff == NULL) - { - MPID_Abort(NULL, MPI_ERR_NO_SPACE, 1, - "Fatal: Cannot allocate pack buffer"); - } - } - } - else - success = 0; - } - - if(!success) - { - MPIDI_Update_last_algorithm(comm_ptr, "SCATTER_MPICH"); - return MPIR_Scatter(sendbuf, sendcount, sendtype, - recvbuf, recvcount, recvtype, - root, comm_ptr, mpierrno); - } - - if(advisor_algorithms[0].metadata && - advisor_algorithms[0].metadata->check_correct.values.asyncflowctl && - !(--(comm_ptr->mpid.num_requests))) - { - comm_ptr->mpid.num_requests = MPIDI_Process.optimized.num_requests; - int tmpmpierrno; - MPIDO_Barrier(comm_ptr, &tmpmpierrno); - } - - pami_xfer_t scatter; - MPIDI_Post_coll_t scatter_post; - volatile unsigned scatter_active = 1; - - - scatter.algorithm = mpid->coll_algorithm[PAMI_XFER_SCATTER][0][0]; - - scatter.cb_done = cb_scatter; - scatter.cookie = (void *)&scatter_active; - scatter.cmd.xfer_scatter.root = MPIDI_Task_to_endpoint(MPID_VCR_GET_LPID(comm_ptr->vcr, root), 0); - scatter.cmd.xfer_scatter.sndbuf = (void *)sbuf; - scatter.cmd.xfer_scatter.stype = PAMI_TYPE_BYTE; - scatter.cmd.xfer_scatter.stypecount = send_size; - scatter.cmd.xfer_scatter.rcvbuf = (void *)rbuf; - scatter.cmd.xfer_scatter.rtype = PAMI_TYPE_BYTE;/* rtype is ignored when rcvbuf == PAMI_IN_PLACE */ - scatter.cmd.xfer_scatter.rtypecount = recv_size; - - if(recvbuf == MPI_IN_PLACE) - { - scatter.cmd.xfer_scatter.rcvbuf = PAMI_IN_PLACE; - } - - TRACE_ERR("%s scatter\n", MPIDI_Process.context_post.active>0?"Posting":"Invoking"); - MPIDI_Context_post(MPIDI_Context[0], &scatter_post.state, - MPIDI_Pami_post_wrapper, (void *)&scatter); - TRACE_ERR("Waiting on active %d\n", scatter_active); - MPID_PROGRESS_WAIT_WHILE(scatter_active); - - if(!rcv_contig && rcv_noncontig_buff) - { - MPIR_Localcopy(rcv_noncontig_buff, recv_size, MPI_CHAR, - recvbuf, recvcount, recvtype); - MPL_free(rcv_noncontig_buff); - } - if(!snd_contig) MPL_free(snd_noncontig_buff); - - - TRACE_ERR("Leaving MPIDO_Scatter_optimized\n"); - - return MPI_SUCCESS; -} - - -int -MPIDO_CSWrapper_scatter(pami_xfer_t *scatter, - void *comm) -{ - int mpierrno = 0; - MPIR_Comm *comm_ptr = (MPIR_Comm*)comm; - MPI_Datatype sendtype, recvtype; - void *rbuf; - MPIDI_coll_check_in_place(scatter->cmd.xfer_scatter.rcvbuf, &rbuf); - int rc = MPIDI_Dtpami_to_dtmpi( scatter->cmd.xfer_scatter.stype, - &sendtype, - NULL, - NULL); - if(rc == -1) return rc; - - rc = MPIDI_Dtpami_to_dtmpi( scatter->cmd.xfer_scatter.rtype, - &recvtype, - NULL, - NULL); - if(rc == -1) return rc; - - rc = MPIR_Scatter(scatter->cmd.xfer_scatter.sndbuf, - scatter->cmd.xfer_scatter.stypecount, sendtype, - rbuf, - scatter->cmd.xfer_scatter.rtypecount, recvtype, - scatter->cmd.xfer_scatter.root, comm_ptr, &mpierrno); - if(scatter->cb_done && rc == 0) - scatter->cb_done(NULL, scatter->cookie, PAMI_SUCCESS); - return rc; - -} - -#if 0 /* old glue-based scatter-via-bcast */ - - /* Assume glue protocol sucks for now.... Needs work if not or to enable */ - - - - /* see if we all agree to use bcast scatter */ - MPIDO_Allreduce(MPI_IN_PLACE, &success, 1, MPI_INT, MPI_BAND, comm_ptr, mpierrno); - - if (!success) - { - if(unlikely(verbose)) - fprintf(stderr,"Using MPICH scatter algorithm\n"); - return MPIR_Scatter(sendbuf, sendcount, sendtype, - recvbuf, recvcount, recvtype, - root, comm_ptr, mpierrno); - } - - sbuf = (char *)sendbuf+true_lb; - rbuf = (char *)recvbuf+true_lb; - - MPIDI_Update_last_algorithm(comm_ptr, "SCATTER_OPT_BCAST"); - return MPIDO_Scatter_bcast(sbuf, sendcount, sendtype, - rbuf, recvcount, recvtype, - root, comm_ptr, mpierrno); -#endif diff --git a/src/mpid/pamid/src/coll/scatterv/Makefile.mk b/src/mpid/pamid/src/coll/scatterv/Makefile.mk deleted file mode 100644 index 7e00f99..0000000 --- a/src/mpid/pamid/src/coll/scatterv/Makefile.mk +++ /dev/null @@ -1,30 +0,0 @@ -# begin_generated_IBM_copyright_prolog -# -# This is an automatically generated copyright prolog. -# After initializing, DO NOT MODIFY OR MOVE -# --------------------------------------------------------------- -# Licensed Materials - Property of IBM -# Blue Gene/Q 5765-PER 5765-PRP -# -# (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved -# US Government Users Restricted Rights - -# Use, duplication, or disclosure restricted -# by GSA ADP Schedule Contract with IBM Corp. -# -# --------------------------------------------------------------- -# -# end_generated_IBM_copyright_prolog -# -*- mode: makefile-gmake; -*- - -# note that the includes always happen but the effects of their contents are -# affected by "if BUILD_PAMID" -if BUILD_PAMID - - -mpi_core_sources += \ - src/mpid/pamid/src/coll/scatterv/mpido_scatterv.c \ - src/mpid/pamid/src/coll/scatterv/mpido_iscatterv.c - - -endif BUILD_PAMID - diff --git a/src/mpid/pamid/src/coll/scatterv/mpido_iscatterv.c b/src/mpid/pamid/src/coll/scatterv/mpido_iscatterv.c deleted file mode 100644 index 11d9173..0000000 --- a/src/mpid/pamid/src/coll/scatterv/mpido_iscatterv.c +++ /dev/null @@ -1,73 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/coll/scatterv/mpido_iscatterv.c - * \brief ??? - */ - -#include <mpidimpl.h> - -int MPIDO_Iscatterv(const void *sendbuf, - const int *sendcounts, - const int *displs, - MPI_Datatype sendtype, - void *recvbuf, - int recvcount, - MPI_Datatype recvtype, - int root, - MPIR_Comm *comm_ptr, - MPIR_Request **request) -{ - /*if (unlikely((data_size == 0) || (user_selected_type == MPID_COLL_USE_MPICH)))*/ - { - /* - * If the mpich mpir non-blocking collectives are enabled, return without - * first constructing the MPIR_Request. This signals to the - * MPIR_Iscatterv_impl() function to invoke the mpich nbc - * implementation of MPI_Iscatterv(). - */ - if (MPIDI_Process.mpir_nbc != 0) - return 0; - - /* - * MPIR_* nbc implementation is not enabled. Fake a non-blocking - * MPIR_Iscatterv() with a blocking MPIR_Scatterv(). - */ - if(unlikely(MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_ALL && comm_ptr->rank == 0)) - fprintf(stderr,"Using MPICH blocking scatterv_algorithm\n"); - - int mpierrno = 0; - int rc = MPIR_Scatterv_impl(sendbuf, sendcounts, displs, sendtype, - recvbuf, recvcount, recvtype, - root, comm_ptr, &mpierrno); - - /* - * The blocking scatterv has completed - create and complete a - * MPIR_Request object so the MPIR_Iscatterv_impl() function - * does not perform an additional iscatterv. - */ - MPIR_Request * mpid_request = MPID_Request_create_inline(); - mpid_request->kind = MPIR_REQUEST_KIND__COLL; - *request = mpid_request; - MPIDI_Request_complete_norelease_inline(mpid_request); - - return rc; - } - - return 0; -} diff --git a/src/mpid/pamid/src/coll/scatterv/mpido_scatterv.c b/src/mpid/pamid/src/coll/scatterv/mpido_scatterv.c deleted file mode 100644 index 75c5186..0000000 --- a/src/mpid/pamid/src/coll/scatterv/mpido_scatterv.c +++ /dev/null @@ -1,812 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/coll/scatterv/mpido_scatterv.c - * \brief ??? - */ - -#include <mpidimpl.h> - - -/* basically, everyone receives recvcount via bcast */ -/* requires a contiguous/continous buffer on root though */ -int MPIDO_Scatterv_bcast(void *sendbuf, - int *sendcounts, - int *displs, - MPI_Datatype sendtype, - void *recvbuf, - int recvcount, - MPI_Datatype recvtype, - int root, - MPIR_Comm *comm_ptr, - int *mpierrno) -{ - const int rank = comm_ptr->rank; - const int size = comm_ptr->local_size; - char *tempbuf; - int i, sum = 0, dtsize, rc=0; - - for (i = 0; i < size; i++) - if (sendcounts > 0) - sum += sendcounts[i]; - - MPIDI_Datatype_get_data_size(1, - recvtype, - dtsize); - - if (rank != root) - { - tempbuf = MPL_malloc(dtsize * sum); - if (!tempbuf) - return MPIR_Err_create_code(MPI_SUCCESS, - MPIR_ERR_RECOVERABLE, - __FUNCTION__, - __LINE__, - MPI_ERR_OTHER, - "**nomem", 0); - } - else - tempbuf = sendbuf; - - /* Switch to comm->coll_fns->fn() */ - rc = MPIDO_Bcast(tempbuf, sum, sendtype, root, comm_ptr, mpierrno); - - if(rank == root && recvbuf == MPI_IN_PLACE) - return rc; - - memcpy(recvbuf, tempbuf + displs[rank], sendcounts[rank] * dtsize); - - if (rank != root) - MPL_free(tempbuf); - - return rc; -} - -/* this guy requires quite a few buffers. maybe - * we should somehow "steal" the comm_ptr alltoall ones? */ -int MPIDO_Scatterv_alltoallv(void * sendbuf, - int * sendcounts, - int * displs, - MPI_Datatype sendtype, - void * recvbuf, - int recvcount, - MPI_Datatype recvtype, - int root, - MPIR_Comm * comm_ptr, - int *mpierrno) -{ - const int rank = comm_ptr->rank; - const int size = comm_ptr->local_size; - - int *sdispls, *scounts; - int *rdispls, *rcounts; - char *sbuf, *rbuf; - int rbytes; - int rc; - - MPIDI_Datatype_get_data_size(recvcount, - recvtype, - rbytes); - - rbuf = MPL_malloc(size * rbytes * sizeof(char)); - if(!rbuf) - { - return MPIR_Err_create_code(MPI_SUCCESS, - MPIR_ERR_RECOVERABLE, - __FUNCTION__, - __LINE__, - MPI_ERR_OTHER, - "**nomem", 0); - } - - if(rank == root) - { - sdispls = displs; - scounts = sendcounts; - sbuf = sendbuf; - } - else - { - sdispls = MPL_malloc(size * sizeof(int)); - scounts = MPL_malloc(size * sizeof(int)); - sbuf = MPL_malloc(rbytes * sizeof(char)); - if(!sdispls || !scounts || !sbuf) - { - if(sdispls) - MPL_free(sdispls); - if(scounts) - MPL_free(scounts); - return MPIR_Err_create_code(MPI_SUCCESS, - MPIR_ERR_RECOVERABLE, - __FUNCTION__, - __LINE__, - MPI_ERR_OTHER, - "**nomem", 0); - } - memset(sdispls, 0, size*sizeof(int)); - memset(scounts, 0, size*sizeof(int)); - } - - rdispls = MPL_malloc(size * sizeof(int)); - rcounts = MPL_malloc(size * sizeof(int)); - if(!rdispls || !rcounts) - { - if(rdispls) - MPL_free(rdispls); - return MPIR_Err_create_code(MPI_SUCCESS, - MPIR_ERR_RECOVERABLE, - __FUNCTION__, - __LINE__, - MPI_ERR_OTHER, - "**nomem", 0); - } - - memset(rdispls, 0, size*sizeof(unsigned)); - memset(rcounts, 0, size*sizeof(unsigned)); - - rcounts[root] = rbytes; - - /* Switch to comm->coll_fns->fn() */ - rc = MPIDO_Alltoallv(sbuf, - scounts, - sdispls, - sendtype, - rbuf, - rcounts, - rdispls, - MPI_CHAR, - comm_ptr, - mpierrno); - - if(rank == root && recvbuf == MPI_IN_PLACE) - { - MPL_free(rbuf); - MPL_free(rdispls); - MPL_free(rcounts); - return rc; - } - else - { - memcpy(recvbuf, rbuf, rbytes); - MPL_free(rbuf); - MPL_free(rdispls); - MPL_free(rcounts); - if(rank != root) - { - MPL_free(sbuf); - MPL_free(sdispls); - MPL_free(scounts); - } - } - - return rc; -} - -static void cb_scatterv(void *ctxt, void *clientdata, pami_result_t err) -{ - unsigned *active = (unsigned *)clientdata; - TRACE_ERR("cb_scatterv enter, active: %u\n", (*active)); - (*active)--; -} - -int MPIDO_Scatterv(const void *sendbuf, - const int *sendcounts, - const int *displs, - MPI_Datatype sendtype, - void *recvbuf, - int recvcount, - MPI_Datatype recvtype, - int root, - MPIR_Comm *comm_ptr, - int *mpierrno) -{ -#ifndef HAVE_PAMI_IN_PLACE - if (sendbuf == MPI_IN_PLACE) - { - MPID_Abort (NULL, 0, 1, "'MPI_IN_PLACE' requries support for `PAMI_IN_PLACE`"); - return -1; - } -#endif - int tmp, i, pamidt = 1; - int contig ATTRIBUTE((unused)); - int ssize ATTRIBUTE((unused)); - int rsize ATTRIBUTE((unused)); - MPIDU_Datatype*dt_ptr = NULL; - MPI_Aint send_true_lb=0, recv_true_lb; - char *sbuf, *rbuf; - pami_type_t stype, rtype; - const int rank = comm_ptr->rank; - const int size = comm_ptr->local_size; -#if ASSERT_LEVEL==0 - /* We can't afford the tracing in ndebug/performance libraries */ - const unsigned verbose = 0; -#else - const unsigned verbose = (MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_ALL) && (rank == 0); -#endif - const struct MPIDI_Comm* const mpid = &(comm_ptr->mpid); - const int selected_type = mpid->user_selected_type[PAMI_XFER_SCATTERV_INT]; - - if(selected_type == MPID_COLL_USE_MPICH) - { - if(unlikely(verbose)) - fprintf(stderr,"Using MPICH scatterv algorithm\n"); - MPIDI_Update_last_algorithm(comm_ptr, "SCATTERV_MPICH"); -#if CUDA_AWARE_SUPPORT - if(MPIDI_Process.cuda_aware_support_on) - { - MPI_Aint sdt_extent,rdt_extent; - MPIDU_Datatype_get_extent_macro(sendtype, sdt_extent); - MPIDU_Datatype_get_extent_macro(recvtype, rdt_extent); - char *scbuf = NULL; - char *rcbuf = NULL; - int is_send_dev_buf = (rank == root) ? MPIDI_cuda_is_device_buf(sendbuf) : 0; - int is_recv_dev_buf = MPIDI_cuda_is_device_buf(recvbuf); - if(is_send_dev_buf) - { - //Since displs can be non-continous, we need to calculate max buffer size - int highest_displs = displs[size - 1]; - int highest_sendcount = sendcounts[size - 1]; - size_t stotal_buf; - for(i = 0; i < size; i++) - { - if(displs[i]+sendcounts[i] > highest_displs+highest_sendcount) - { - highest_displs = displs[i]; - highest_sendcount = sendcounts[i]; - } - } - stotal_buf = (highest_displs+highest_sendcount)*sdt_extent; - scbuf = MPL_malloc(stotal_buf); - cudaError_t cudaerr = CudaMemcpy(scbuf, sendbuf, stotal_buf, cudaMemcpyDeviceToHost); - if (cudaSuccess != cudaerr) - fprintf(stderr, "cudaMemcpy failed: %s\n", CudaGetErrorString(cudaerr)); - } - else - scbuf = sendbuf; - if(is_recv_dev_buf) - { - rcbuf = MPL_malloc(recvcount * rdt_extent); - CudaMemcpy(rcbuf, recvbuf, recvcount * rdt_extent, cudaMemcpyDeviceToHost); - } - else - rcbuf = recvbuf; - int cuda_res = MPIR_Scatterv(scbuf, sendcounts, displs, sendtype, rcbuf, recvcount, recvtype, root, comm_ptr, mpierrno); - if(is_send_dev_buf)MPL_free(scbuf); - if(is_recv_dev_buf) - { - cudaError_t cudaerr = CudaMemcpy(recvbuf, rcbuf, recvcount * rdt_extent, cudaMemcpyHostToDevice); - if (cudaSuccess != cudaerr) - fprintf(stderr, "cudaMemcpy failed: %s\n", CudaGetErrorString(cudaerr)); - MPL_free(rcbuf); - } - return cuda_res; - } - else -#endif - return MPIR_Scatterv(sendbuf, sendcounts, displs, sendtype, - recvbuf, recvcount, recvtype, - root, comm_ptr, mpierrno); - } - - - pami_xfer_t scatterv; - pami_algorithm_t my_scatterv; - const pami_metadata_t *my_md = (pami_metadata_t *)NULL; - volatile unsigned scatterv_active = 1; - int queryreq = 0; - - if(selected_type == MPID_COLL_OPTIMIZED) - { - TRACE_ERR("Optimized scatterv %s was selected\n", - mpid->opt_protocol_md[PAMI_XFER_SCATTERV_INT][0].name); - my_scatterv = mpid->opt_protocol[PAMI_XFER_SCATTERV_INT][0]; - my_md = &mpid->opt_protocol_md[PAMI_XFER_SCATTERV_INT][0]; - queryreq = mpid->must_query[PAMI_XFER_SCATTERV_INT][0]; - } - else - { - TRACE_ERR("User selected %s for scatterv\n", - mpid->user_selected[PAMI_XFER_SCATTERV_INT]); - my_scatterv = mpid->user_selected[PAMI_XFER_SCATTERV_INT]; - my_md = &mpid->user_metadata[PAMI_XFER_SCATTERV_INT]; - queryreq = selected_type; - } - - if((recvbuf != MPI_IN_PLACE) && MPIDI_Datatype_to_pami(recvtype, &rtype, -1, NULL, &tmp) != MPI_SUCCESS) - pamidt = 0; - - if(MPIDI_Datatype_to_pami(sendtype, &stype, -1, NULL, &tmp) != MPI_SUCCESS) - pamidt = 0; - - if(pamidt == 0 || selected_type == MPID_COLL_USE_MPICH) - { - if(unlikely(verbose)) - fprintf(stderr,"Using MPICH scatterv algorithm\n"); - TRACE_ERR("Scatterv using MPICH\n"); - MPIDI_Update_last_algorithm(comm_ptr, "SCATTERV_MPICH"); - return MPIR_Scatterv(sendbuf, sendcounts, displs, sendtype, - recvbuf, recvcount, recvtype, - root, comm_ptr, mpierrno); - } - - MPIDI_Datatype_get_info(1, sendtype, contig, ssize, dt_ptr, send_true_lb); - sbuf = (char *)sendbuf + send_true_lb; - rbuf = recvbuf; - - if(rank == root) - { - if(recvbuf == MPI_IN_PLACE) - { - if(unlikely(verbose)) - fprintf(stderr,"scatterv MPI_IN_PLACE buffering\n"); - rbuf = PAMI_IN_PLACE; - } - else - { - MPIDI_Datatype_get_info(1, recvtype, contig, rsize, dt_ptr, recv_true_lb); - rbuf = (char *)recvbuf + recv_true_lb; - } - } - - scatterv.cb_done = cb_scatterv; - scatterv.cookie = (void *)&scatterv_active; - scatterv.cmd.xfer_scatterv_int.root = MPIDI_Task_to_endpoint(MPID_VCR_GET_LPID(comm_ptr->vcr, root), 0); - - scatterv.algorithm = my_scatterv; - - scatterv.cmd.xfer_scatterv_int.rcvbuf = rbuf; - scatterv.cmd.xfer_scatterv_int.sndbuf = sbuf; - scatterv.cmd.xfer_scatterv_int.stype = stype; - scatterv.cmd.xfer_scatterv_int.rtype = rtype; - scatterv.cmd.xfer_scatterv_int.stypecounts = (int *) sendcounts; - scatterv.cmd.xfer_scatterv_int.rtypecount = recvcount; - scatterv.cmd.xfer_scatterv_int.sdispls = (int *) displs; - - if(unlikely(queryreq == MPID_COLL_ALWAYS_QUERY || - queryreq == MPID_COLL_CHECK_FN_REQUIRED)) - { - metadata_result_t result = {0}; - TRACE_ERR("querying scatterv protocol %s, type was %d\n", - my_md->name, queryreq); - if(my_md->check_fn == NULL) - { - /* process metadata bits */ - if((!my_md->check_correct.values.inplace) && (recvbuf == MPI_IN_PLACE)) - result.check.unspecified = 1; -/* Can't check ranges like this. Non-local. Comment out for now. - if(my_md->check_correct.values.rangeminmax) - { - MPI_Aint data_true_lb; - MPIDU_Datatype*data_ptr; - int data_size, data_contig; - MPIDI_Datatype_get_info(recvcount, recvtype, data_contig, data_size, data_ptr, data_true_lb); - if((my_md->range_lo <= data_size) && - (my_md->range_hi >= data_size)) - ; - else - { - result.check.range = 1; - if(unlikely(verbose)) - { - fprintf(stderr,"message size (%u) outside range (%zu<->%zu) for %s.\n", - data_size, - my_md->range_lo, - my_md->range_hi, - my_md->name); - } - } - } - */ - } - else /* calling the check fn is sufficient */ - result = my_md->check_fn(&scatterv); - TRACE_ERR("bitmask: %#X\n", result.bitmask); - result.check.nonlocal = 0; /* #warning REMOVE THIS WHEN IMPLEMENTED */ - if(result.bitmask) - { - if(unlikely(verbose)) - fprintf(stderr,"Query failed for %s. Using MPICH scatterv.\n", my_md->name); - MPIDI_Update_last_algorithm(comm_ptr, "SCATTERV_MPICH"); - return MPIR_Scatterv(sendbuf, sendcounts, displs, sendtype, - recvbuf, recvcount, recvtype, - root, comm_ptr, mpierrno); - } - if(my_md->check_correct.values.asyncflowctl && !(--(comm_ptr->mpid.num_requests))) - { - comm_ptr->mpid.num_requests = MPIDI_Process.optimized.num_requests; - int tmpmpierrno; - if(unlikely(verbose)) - fprintf(stderr,"Query barrier required for %s\n", my_md->name); - MPIDO_Barrier(comm_ptr, &tmpmpierrno); - } - } - - MPIDI_Update_last_algorithm(comm_ptr, my_md->name); - - if(unlikely(verbose)) - { - unsigned long long int threadID; - MPL_thread_id_t tid; - MPL_thread_self(&tid); - threadID = (unsigned long long int)tid; - fprintf(stderr,"<%llx> Using protocol %s for scatterv on %u\n", - threadID, - my_md->name, - (unsigned) comm_ptr->context_id); - } - MPIDI_Post_coll_t scatterv_post; - MPIDI_Context_post(MPIDI_Context[0], &scatterv_post.state, - MPIDI_Pami_post_wrapper, (void *)&scatterv); - - TRACE_ERR("Waiting on active %d\n", scatterv_active); - MPID_PROGRESS_WAIT_WHILE(scatterv_active); - - TRACE_ERR("Leaving MPIDO_Scatterv\n"); - return 0; -} - -int MPIDO_Scatterv_simple(const void *sendbuf, - const int *sendcounts, - const int *displs, - MPI_Datatype sendtype, - void *recvbuf, - int recvcount, - MPI_Datatype recvtype, - int root, - MPIR_Comm *comm_ptr, - int *mpierrno) -{ -#ifndef HAVE_PAMI_IN_PLACE - if (sendbuf == MPI_IN_PLACE) - { - MPID_Abort (NULL, 0, 1, "'MPI_IN_PLACE' requries support for `PAMI_IN_PLACE`"); - return -1; - } -#endif - int snd_contig = 1; - int rcv_contig = 1; - int send_size = 0, recv_size = 0; - int ssize = 0; - MPIDU_Datatype*dt_ptr = NULL; - MPI_Aint send_true_lb=0, recv_true_lb=0; - void *snd_noncontig_buff = NULL, *rcv_noncontig_buff = NULL; - void *sbuf = NULL, *rbuf = NULL; - int *sdispls = NULL, *scounts = NULL; - int sndcount = 0; - MPIDU_Segment segment; - int tmp, i; - pami_type_t stype = PAMI_TYPE_NULL; - const int rank = comm_ptr->rank; - const int size = comm_ptr->local_size; - const struct MPIDI_Comm* const mpid = &(comm_ptr->mpid); - - if (rank == root && sendtype != MPI_DATATYPE_NULL && sendcounts[0] >= 0) - { - MPIDI_Datatype_get_info(1, sendtype, snd_contig, ssize, dt_ptr, send_true_lb); - if(MPIDI_Pamix_collsel_advise != NULL && mpid->collsel_fast_query != NULL) - { - advisor_algorithm_t advisor_algorithms[1]; - int num_algorithms = MPIDI_Pamix_collsel_advise(mpid->collsel_fast_query, PAMI_XFER_SCATTERV_INT, 64, advisor_algorithms, 1); - if(num_algorithms) - { - if(advisor_algorithms[0].algorithm_type == COLLSEL_EXTERNAL_ALGO) - { - return MPIR_Scatterv(sendbuf, sendcounts, displs, sendtype, - recvbuf, recvcount, recvtype, - root, comm_ptr, mpierrno); - } - else if(advisor_algorithms[0].metadata && advisor_algorithms[0].metadata->check_correct.values.asyncflowctl && !(--(comm_ptr->mpid.num_requests))) - { - comm_ptr->mpid.num_requests = MPIDI_Process.optimized.num_requests; - int tmpmpierrno; - MPIDO_Barrier(comm_ptr, &tmpmpierrno); - } - - } - } - } - - if (recvtype != MPI_DATATYPE_NULL && recvcount >= 0) - { - MPIDI_Datatype_get_info(recvcount, recvtype, rcv_contig, - recv_size, dt_ptr, recv_true_lb); - if(MPIDI_Pamix_collsel_advise != NULL && mpid->collsel_fast_query != NULL) - { - advisor_algorithm_t advisor_algorithms[1]; - int num_algorithms = MPIDI_Pamix_collsel_advise(mpid->collsel_fast_query, PAMI_XFER_SCATTERV_INT, 64, advisor_algorithms, 1); - if(num_algorithms) - { - if(advisor_algorithms[0].algorithm_type == COLLSEL_EXTERNAL_ALGO) - { - return MPIR_Scatterv(sendbuf, sendcounts, displs, sendtype, - recvbuf, recvcount, recvtype, - root, comm_ptr, mpierrno); - } - else if(advisor_algorithms[0].metadata && advisor_algorithms[0].metadata->check_correct.values.asyncflowctl && !(--(comm_ptr->mpid.num_requests))) - { - comm_ptr->mpid.num_requests = MPIDI_Process.optimized.num_requests; - int tmpmpierrno; - MPIDO_Barrier(comm_ptr, &tmpmpierrno); - } - - } - } - } - - pami_xfer_t scatterv; - const pami_metadata_t *my_scatterv_md; - volatile unsigned scatterv_active = 1; - - sbuf = (char *)sendbuf + send_true_lb; - rbuf = (char *)recvbuf + recv_true_lb; - scounts = (int*)sendcounts; - sdispls = (int*)displs; - if(rank == root) - { - if(MPIDI_Datatype_to_pami(sendtype, &stype, -1, NULL, &tmp) != MPI_SUCCESS) - { - if (!snd_contig) - { - scounts = (int*)MPL_malloc(size); - sdispls = (int*)MPL_malloc(size); - for(i = 0; i < size; i++) - { - scounts[i] = ssize * sendcounts[i]; - sdispls[i] = ssize * displs[i]; - send_size += scounts[i]; - sndcount += sendcounts[i]; - } - snd_noncontig_buff = MPL_malloc(send_size); - sbuf = snd_noncontig_buff; - stype = PAMI_TYPE_BYTE; - if(snd_noncontig_buff == NULL) - { - MPID_Abort(NULL, MPI_ERR_NO_SPACE, 1, - "Fatal: Cannot allocate pack buffer"); - } - DLOOP_Offset last = send_size; - MPIDU_Segment_init(sendbuf, sndcount, sendtype, &segment, 0); - MPIDU_Segment_pack(&segment, 0, &last, snd_noncontig_buff); - } - } - if(recvbuf == MPI_IN_PLACE) - { - rbuf = PAMI_IN_PLACE; - } - } - - if(recvbuf != MPI_IN_PLACE) - { - if (!rcv_contig) - { - rcv_noncontig_buff = MPL_malloc(recv_size); - rbuf = rcv_noncontig_buff; - if(rcv_noncontig_buff == NULL) - { - MPID_Abort(NULL, MPI_ERR_NO_SPACE, 1, - "Fatal: Cannot allocate pack buffer"); - } - } - } - - scatterv.cb_done = cb_scatterv; - scatterv.cookie = (void *)&scatterv_active; - scatterv.cmd.xfer_scatterv_int.root = MPIDI_Task_to_endpoint(MPID_VCR_GET_LPID(comm_ptr->vcr, root), 0); - - scatterv.algorithm = mpid->coll_algorithm[PAMI_XFER_SCATTERV_INT][0][0]; - my_scatterv_md = &mpid->coll_metadata[PAMI_XFER_SCATTERV_INT][0][0]; - - scatterv.cmd.xfer_scatterv_int.rcvbuf = rbuf; - scatterv.cmd.xfer_scatterv_int.sndbuf = sbuf; - scatterv.cmd.xfer_scatterv_int.stype = stype; - scatterv.cmd.xfer_scatterv_int.rtype = PAMI_TYPE_BYTE;/* rtype is ignored when rcvbuf == PAMI_IN_PLACE */ - scatterv.cmd.xfer_scatterv_int.stypecounts = (int *) scounts; - scatterv.cmd.xfer_scatterv_int.rtypecount = recv_size; - scatterv.cmd.xfer_scatterv_int.sdispls = (int *) sdispls; - - - MPIDI_Update_last_algorithm(comm_ptr, my_scatterv_md->name); - - - MPIDI_Post_coll_t scatterv_post; - TRACE_ERR("%s scatterv\n", MPIDI_Process.context_post.active>0?"Posting":"Invoking"); - MPIDI_Context_post(MPIDI_Context[0], &scatterv_post.state, - MPIDI_Pami_post_wrapper, (void *)&scatterv); - - TRACE_ERR("Waiting on active %d\n", scatterv_active); - MPID_PROGRESS_WAIT_WHILE(scatterv_active); - - if(!rcv_contig) - { - MPIR_Localcopy(rcv_noncontig_buff, recv_size, MPI_CHAR, - recvbuf, recvcount, recvtype); - MPL_free(rcv_noncontig_buff); - } - if(!snd_contig) - { - MPL_free(snd_noncontig_buff); - MPL_free(scounts); - MPL_free(sdispls); - } - - TRACE_ERR("Leaving MPIDO_Scatterv_optimized\n"); - return MPI_SUCCESS; -} - - -int -MPIDO_CSWrapper_scatterv(pami_xfer_t *scatterv, - void *comm) -{ - int mpierrno = 0, rc = 0; - MPIR_Comm *comm_ptr = (MPIR_Comm*)comm; - MPI_Datatype sendtype, recvtype; - void *rbuf; - MPIDI_coll_check_in_place(scatterv->cmd.xfer_scatterv_int.rcvbuf, &rbuf); - /* Since collective selection in PAMI fixes message size when selecting the - algorithm, this wrapper function may be called back from PAMI */ - if(PAMI_TYPE_NULL == scatterv->cmd.xfer_scatterv_int.stype) - sendtype = MPI_DATATYPE_NULL; - else - rc = MPIDI_Dtpami_to_dtmpi( scatterv->cmd.xfer_scatterv_int.stype, - &sendtype, - NULL, - NULL); - if(rc == -1) return rc; - - rc = MPIDI_Dtpami_to_dtmpi( scatterv->cmd.xfer_scatterv_int.rtype, - &recvtype, - NULL, - NULL); - if(rc == -1) return rc; - - rc = MPIR_Scatterv(scatterv->cmd.xfer_scatterv_int.sndbuf, - scatterv->cmd.xfer_scatterv_int.stypecounts, - scatterv->cmd.xfer_scatterv_int.sdispls, sendtype, - rbuf, - scatterv->cmd.xfer_scatterv_int.rtypecount, recvtype, - scatterv->cmd.xfer_scatterv_int.root, comm_ptr, &mpierrno); - if(scatterv->cb_done && rc == 0) - scatterv->cb_done(NULL, scatterv->cookie, PAMI_SUCCESS); - return rc; - -} - - -#if 0 -/* remove the glue-based optimized scattervs for now. */ - - - /* we can't call scatterv-via-bcast unless we know all nodes have - * valid sendcount arrays. so the user must explicitly ask for it. - */ - - /* optscatterv[0] == optscatterv bcast? - * optscatterv[1] == optscatterv alltoall? - * (having both allows cutoff agreement) - * optscatterv[2] == sum of sendcounts - */ - - optscatterv[0] = !mpid->scattervs[0]; - optscatterv[1] = !mpid->scattervs[1]; - optscatterv[2] = 1; - - if(rank == root) - { - if(!optscatterv[1]) - { - if(sendcounts) - { - for(i=0;i<size;i++) - sum+=sendcounts[i]; - } - optscatterv[2] = sum; - } - - MPIDI_Datatype_get_info(1, - sendtype, - contig, - nbytes, - dt_ptr, - true_lb); - if(recvtype == MPI_DATATYPE_NULL || recvcount <= 0 || !contig) - { - optscatterv[0] = 1; - optscatterv[1] = 1; - } - } - else - { - MPIDI_Datatype_get_info(1, - recvtype, - contig, - nbytes, - dt_ptr, - true_lb); - if(sendtype == MPI_DATATYPE_NULL || !contig) - { - optscatterv[0] = 1; - optscatterv[1] = 1; - } - } - - /* Make sure parameters are the same on all the nodes */ - /* specifically, noncontig on the receive */ - /* set the internal control flow to disable internal star tuning */ - if(mpid->preallreduces[MPID_SCATTERV_PREALLREDUCE]) - { - MPIDI_Post_coll_t allred_post; - rc = MPIDI_Context_post(MPIDI_Context[0], &allred_post.state, - MPIDI_Pami_post_wrapper, (void *)&allred); - - MPID_PROGRESS_WAIT_WHILE(allred_active); - } - /* reset flag */ - - if(!optscatterv[0] || (!optscatterv[1])) - { - char *newrecvbuf = recvbuf; - char *newsendbuf = sendbuf; - if(rank == root) - { - newsendbuf = (char *)sendbuf + true_lb; - } - else - { - newrecvbuf = (char *)recvbuf + true_lb; - } - if(!optscatterv[0]) - { - MPIDI_Update_last_algorithm(comm_ptr, "SCATTERV_OPT_ALLTOALL"); - return MPIDO_Scatterv_alltoallv(newsendbuf, - sendcounts, - displs, - sendtype, - newrecvbuf, - recvcount, - recvtype, - root, - comm_ptr, - mpierrno); - - } - else - { - MPIDI_Update_last_algorithm(comm_ptr, "SCATTERV_OPT_BCAST"); - return MPIDO_Scatterv_bcast(newsendbuf, - sendcounts, - displs, - sendtype, - newrecvbuf, - recvcount, - recvtype, - root, - comm_ptr, - mpierrno); - } - } /* nothing valid to try, go to mpich */ - else - { - if(unlikely(verbose)) - fprintf(stderr,"Using MPICH scatterv algorithm\n"); - MPIDI_Update_last_algorithm(comm_ptr, "SCATTERV_MPICH"); - return MPIR_Scatterv(sendbuf, sendcounts, displs, sendtype, - recvbuf, recvcount, recvtype, - root, comm_ptr, mpierrno); - } -#endif diff --git a/src/mpid/pamid/src/comm/Makefile.mk b/src/mpid/pamid/src/comm/Makefile.mk deleted file mode 100644 index de786c7..0000000 --- a/src/mpid/pamid/src/comm/Makefile.mk +++ /dev/null @@ -1,31 +0,0 @@ -# begin_generated_IBM_copyright_prolog -# -# This is an automatically generated copyright prolog. -# After initializing, DO NOT MODIFY OR MOVE -# --------------------------------------------------------------- -# Licensed Materials - Property of IBM -# Blue Gene/Q 5765-PER 5765-PRP -# -# (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved -# US Government Users Restricted Rights - -# Use, duplication, or disclosure restricted -# by GSA ADP Schedule Contract with IBM Corp. -# -# --------------------------------------------------------------- -# -# end_generated_IBM_copyright_prolog -# -*- mode: makefile-gmake; -*- - -# note that the includes always happen but the effects of their contents are -# affected by "if BUILD_PAMID" -if BUILD_PAMID - - -mpi_core_sources += \ - src/mpid/pamid/src/comm/mpid_comm.c \ - src/mpid/pamid/src/comm/mpid_selectcolls.c \ - src/mpid/pamid/src/comm/mpid_optcolls.c - - -endif BUILD_PAMID - diff --git a/src/mpid/pamid/src/comm/mpid_comm.c b/src/mpid/pamid/src/comm/mpid_comm.c deleted file mode 100644 index 34bb908..0000000 --- a/src/mpid/pamid/src/comm/mpid_comm.c +++ /dev/null @@ -1,432 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/comm/mpid_comm.c - * \brief ??? - */ - -/*#define TRACE_ON */ - -#include <mpidimpl.h> -extern void MPIDI_Comm_coll_query(MPIR_Comm *); -extern void MPIDI_Comm_coll_envvars(MPIR_Comm *); - -void geom_create_cb_done(void *ctxt, void *data, pami_result_t err) -{ - int *active = (int *)data; - (*active)--; -} - -void geom_destroy_cb_done(void *ctxt, void *data, pami_result_t err) -{ - int *active = (int *)data; - (*active)--; -} - -int MPIDI_Comm_create (MPIR_Comm *comm) -{ - MPIDI_Coll_comm_create(comm); - return MPI_SUCCESS; -} - -int MPIDI_Comm_destroy (MPIR_Comm *comm) -{ - MPIDI_Coll_comm_destroy(comm); - return MPI_SUCCESS; -} - -pami_result_t MPIDI_Comm_create_from_pami_geom(pami_geometry_range_t *task_slices, - size_t slice_count, - pami_geometry_t *geometry, - void **cookie) -{ - int mpi_errno = MPI_SUCCESS; - int num_tasks = 0; - int *ranks = NULL; - MPIR_Comm *comm_ptr = NULL, *new_comm_ptr = NULL; - MPIR_Group *group_ptr = NULL, *new_group_ptr = NULL; - int i = 0, j = 0; - - /* Get comm_ptr for MPI_COMM_WORLD and get the group_ptr for it */ - MPIR_Comm_get_ptr( MPI_COMM_WORLD, comm_ptr ); - mpi_errno = MPIR_Comm_group_impl(comm_ptr, &group_ptr); - if (mpi_errno) - { - TRACE_ERR("Error while creating group_ptr from MPI_COMM_WORLD in MPIDI_Comm_create_from_pami_geom\n"); - return PAMI_ERROR; - } - - /* Create the ranks list from the pami_geometry_range_t array */ - for(i = 0; i < slice_count; i++) - { - num_tasks += (task_slices[i].hi - task_slices[i].lo) + 1; - } - ranks = MPL_calloc0(num_tasks, int); - for(i = 0; i < slice_count; i++) - { - int slice_sz = (task_slices[i].hi - task_slices[i].lo) + 1; - int k = 0; - for(k = 0; k < slice_sz; k++) - { - ranks[j] = task_slices[i].lo + k; - j++; - } - } - - /* Now we have all we need to create the new group. Create it */ - mpi_errno = MPIR_Group_incl_impl(group_ptr, num_tasks, ranks, &new_group_ptr); - if (mpi_errno) - { - TRACE_ERR("Error while creating new_group_ptr from group_ptr in MPIDI_Comm_create_from_pami_geom\n"); - return PAMI_ERROR; - } - - /* Now create the communicator using the new_group_ptr */ - mpi_errno = MPIR_Comm_create_intra(comm_ptr, new_group_ptr, &new_comm_ptr); - if (mpi_errno) - { - TRACE_ERR("Error while creating new_comm_ptr from group_ptr in MPIDI_Comm_create_from_pami_geom\n"); - return PAMI_ERROR; - } - - if(new_comm_ptr) - { - /* Get the geometry from the communicator and set the out parameters */ - *geometry = new_comm_ptr->mpid.geometry; - *cookie = new_comm_ptr; - } - else - { - *geometry = PAMI_GEOMETRY_NULL; - *cookie = NULL; - } - - /* Cleanup */ - MPIU_TestFree(&ranks); - - return PAMI_SUCCESS; -} - -pami_result_t MPIDI_Comm_destroy_external(void *comm_ext) -{ - int mpi_errno = 0; - MPIR_Comm* comm_ptr = (MPIR_Comm*)comm_ext; - mpi_errno = MPIR_Comm_free_impl(comm_ptr); - if (mpi_errno) - { - TRACE_ERR("Error while destroying comm_ptr in MPIDI_Comm_destroy_external\n"); - return PAMI_ERROR; - } - return PAMI_SUCCESS; -} - -typedef struct MPIDI_Post_geom_create -{ - pami_work_t state; - size_t context_offset; - pami_client_t client; - pami_configuration_t *configs; - size_t num_configs; - pami_geometry_t newgeom; - pami_geometry_t parent; - unsigned id; - pami_geometry_range_t *ranges; - pami_task_t *tasks; - size_t count; /* count of ranges or tasks */ - pami_event_function fn; - void* cookie; -} MPIDI_Post_geom_create_t; - -typedef struct MPIDI_Post_geom_destroy -{ - pami_work_t state; - pami_client_t client; - pami_geometry_t *geom; - pami_event_function fn; - void *cookie; -} MPIDI_Post_geom_destroy_t; - -static pami_result_t geom_rangelist_create_wrapper(pami_context_t context, void *cookie) -{ - /* I'll need one of these per geometry creation function..... */ - MPIDI_Post_geom_create_t *geom_struct = (MPIDI_Post_geom_create_t *)cookie; - TRACE_ERR("In geom create wrapper\n"); - return PAMI_Geometry_create_taskrange( - geom_struct->client, - geom_struct->context_offset, - geom_struct->configs, - geom_struct->num_configs, - geom_struct->newgeom, - geom_struct->parent, - geom_struct->id, - geom_struct->ranges, - geom_struct->count, - context, - geom_struct->fn, - geom_struct->cookie); - TRACE_ERR("Done in geom create wrapper\n"); -} -static pami_result_t geom_tasklist_create_wrapper(pami_context_t context, void *cookie) -{ - /* I'll need one of these per geometry creation function..... */ - MPIDI_Post_geom_create_t *geom_struct = (MPIDI_Post_geom_create_t *)cookie; - TRACE_ERR("In geom create wrapper\n"); - return PAMI_Geometry_create_tasklist( - geom_struct->client, - geom_struct->context_offset, - geom_struct->configs, - geom_struct->num_configs, - geom_struct->newgeom, - geom_struct->parent, - geom_struct->id, - geom_struct->tasks, - geom_struct->count, - context, - geom_struct->fn, - geom_struct->cookie); - TRACE_ERR("Done in geom create wrapper\n"); -} - -static pami_result_t geom_destroy_wrapper(pami_context_t context, void *cookie) -{ - MPIDI_Post_geom_destroy_t *geom_destroy_struct = (MPIDI_Post_geom_destroy_t *)cookie; - TRACE_ERR("In geom destroy wrapper\n"); - return PAMI_Geometry_destroy( - geom_destroy_struct->client, - geom_destroy_struct->geom, - context, - geom_destroy_struct->fn, - geom_destroy_struct->cookie); - TRACE_ERR("Done in geom destroy wrapper\n"); -} - - - -void MPIDI_Coll_comm_create(MPIR_Comm *comm) -{ - volatile int geom_init = 1; - int i; - MPIDI_Post_geom_create_t geom_post; - - TRACE_ERR("MPIDI_Coll_comm_create enter\n"); - if (!MPIDI_Process.optimized.collectives) - return; - - if(comm->comm_kind != MPIR_COMM_KIND__INTRACOMM) return; - /* Create a geometry */ - - comm->coll_fns = MPL_calloc0(1, MPIR_Collops); - MPID_assert(comm->coll_fns != NULL); - - if(comm->mpid.geometry != MPIDI_Process.world_geometry) - { - if(unlikely(MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_0 && comm->rank == 0)) - fprintf(stderr,"world geom: %p parent geom: %p\n", MPIDI_Process.world_geometry, comm->mpid.parent); - TRACE_ERR("Creating subgeom\n"); - /* Change to this at some point */ - - comm->mpid.tasks = NULL; - for(i=1;i<comm->local_size;i++) - { - /* only if sequential tasks should we use a (single) range. - Multi or reordered ranges are inefficient */ - if(MPID_VCR_GET_LPID(comm->vcr, i) != (MPID_VCR_GET_LPID(comm->vcr, i-1) + 1)) { - /* not sequential, use tasklist */ - MPID_VCR_GET_LPIDS(comm, comm->mpid.tasks); - break; - } - } - /* Should we use a range? (no task list set) */ - if(comm->mpid.tasks == NULL) - { - /* one range, {first rank ... last rank} */ - comm->mpid.range.lo = MPID_VCR_GET_LPID(comm->vcr, 0); - comm->mpid.range.hi = MPID_VCR_GET_LPID(comm->vcr, comm->local_size-1); - } - - if(unlikely(MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_0 && comm->rank == 0)) - fprintf(stderr,"create geometry tasks %p {%u..%u}\n", comm->mpid.tasks, MPID_VCR_GET_LPID(comm->vcr, 0),MPID_VCR_GET_LPID(comm->vcr, comm->local_size-1)); - - pami_configuration_t config[3]; - size_t numconfigs = 0; -#ifdef HAVE_PAMI_GEOMETRY_NONCONTIG - config[0].name = PAMI_GEOMETRY_NONCONTIG; - if(MPIDI_Process.optimized.memory & MPIR_OPT_LVL_NONCONTIG) - config[0].value.intval = 0; // Disable non-contig, pamid doesn't use pami for non-contig data collectives - else - config[0].value.intval = 1; // Enable non-contig even though pamid doesn't use pami for non-contig data collectives, - // we still possibly want those collectives for other reasons. - ++numconfigs; -#endif - if(MPIDI_Process.optimized.subcomms) - { - config[numconfigs].name = PAMI_GEOMETRY_OPTIMIZE; - config[numconfigs].value.intval = 1; - ++numconfigs; - } -#ifdef HAVE_PAMI_GEOMETRY_MEMORY_OPTIMIZE - if(MPIDI_Process.optimized.memory) - { - config[numconfigs].name = PAMI_GEOMETRY_MEMORY_OPTIMIZE; - config[numconfigs].value.intval = MPIDI_Process.optimized.memory; /* level of optimization */ - ++numconfigs; - } -#endif - - if((MPIDI_Process.optimized.memory & MPIR_OPT_LVL_IRREG) && (comm->local_size & (comm->local_size-1))) - { - /* Don't create irregular geometries. Fallback to MPICH only collectives */ - geom_init = 0; - comm->mpid.geometry = PAMI_GEOMETRY_NULL; - } - else if(comm->mpid.tasks == NULL) - { - geom_post.client = MPIDI_Client; - geom_post.configs = config; - geom_post.context_offset = 0; /* TODO BES investigate */ - geom_post.num_configs = numconfigs; - geom_post.newgeom = &comm->mpid.geometry, - geom_post.parent = PAMI_GEOMETRY_NULL; - geom_post.id = comm->context_id; - geom_post.ranges = &comm->mpid.range; - geom_post.tasks = NULL;; - geom_post.count = (size_t)1; - geom_post.fn = geom_create_cb_done; - geom_post.cookie = (void*)&geom_init; - - TRACE_ERR("%s geom_rangelist_create\n", MPIDI_Process.context_post>0?"Posting":"Invoking"); - MPIDI_Context_post(MPIDI_Context[0], &geom_post.state, - geom_rangelist_create_wrapper, (void *)&geom_post); - } - else - { - geom_post.client = MPIDI_Client; - geom_post.configs = config; - geom_post.context_offset = 0; /* TODO BES investigate */ - geom_post.num_configs = numconfigs; - geom_post.newgeom = &comm->mpid.geometry, - geom_post.parent = PAMI_GEOMETRY_NULL; - geom_post.id = comm->context_id; - geom_post.ranges = NULL; - geom_post.tasks = comm->mpid.tasks; - geom_post.count = (size_t)comm->local_size; - geom_post.fn = geom_create_cb_done; - geom_post.cookie = (void*)&geom_init; - - TRACE_ERR("%s geom_tasklist_create\n", MPIDI_Process.context_post>0?"Posting":"Invoking"); - MPIDI_Context_post(MPIDI_Context[0], &geom_post.state, - geom_tasklist_create_wrapper, (void *)&geom_post); - } - - TRACE_ERR("Waiting for geom create to finish\n"); - MPID_PROGRESS_WAIT_WHILE(geom_init); - - if(comm->mpid.geometry == PAMI_GEOMETRY_NULL) - { - if(unlikely(MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_0 && comm->rank == 0)) - fprintf(stderr,"Created unoptimized communicator id=%u, size=%u\n", (unsigned) comm->context_id,comm->local_size); - MPIU_TestFree(&comm->coll_fns); - return; - } - } - /* Initialize the async flow control in case it will be used. */ - comm->mpid.num_requests = MPIDI_Process.optimized.num_requests; - - TRACE_ERR("Querying protocols\n"); - /* Determine what protocols are available for this comm/geom */ - /* These two functions moved to mpid_collselect.c */ - MPIDI_Comm_coll_query(comm); - MPIDI_Comm_coll_envvars(comm); - if(MPIDI_Process.optimized.select_colls) - MPIDI_Comm_coll_select(comm); - TRACE_ERR("mpir barrier\n"); - int mpierrno = FALSE; - /* Switch to comm->coll_fns->fn() */ - MPIDO_Barrier(comm, &mpierrno); - - - TRACE_ERR("MPIDI_Coll_comm_create exit\n"); -} - -void MPIDI_Coll_comm_destroy(MPIR_Comm *comm) -{ - TRACE_ERR("MPIDI_Coll_comm_destroy enter\n"); - int i; - volatile int geom_destroy = 1; - MPIDI_Post_geom_destroy_t geom_destroy_post; - if (!MPIDI_Process.optimized.collectives) - return; - - if(comm->comm_kind != MPIR_COMM_KIND__INTRACOMM) - return; - - /* It's possible (MPII_Setup_intercomm_localcomm) to have an intracomm - without a geometry even when using optimized collectives */ - if(comm->mpid.geometry == PAMI_GEOMETRY_NULL) - return; - - MPIU_TestFree(&comm->coll_fns); - for(i=0;i<PAMI_XFER_COUNT;i++) - { - TRACE_ERR("Freeing algo/meta %d\n", i); - /* When allocating comm->mpid.coll_algorithm, we skip allocations for - AM collectives. Also there is no explicit initialization of - comm->mpid.coll_algorithm to NULLs. This may cause MPIU_TestFree to - cause problems when freeing. We skip AM collectives here as we skip - allocating them in MPIDI_Comm_coll_query */ - if(i == PAMI_XFER_AMBROADCAST || i == PAMI_XFER_AMSCATTER || - i == PAMI_XFER_AMGATHER || i == PAMI_XFER_AMREDUCE) - continue; - - MPIU_TestFree(&comm->mpid.coll_algorithm[i][0]); - MPIU_TestFree(&comm->mpid.coll_algorithm[i][1]); - MPIU_TestFree(&comm->mpid.coll_metadata[i][0]); - MPIU_TestFree(&comm->mpid.coll_metadata[i][1]); - } - - - if(MPIDI_Process.optimized.auto_select_colls != MPID_AUTO_SELECT_COLLS_NONE && MPIDI_Process.optimized.auto_select_colls != MPID_AUTO_SELECT_COLLS_TUNE && comm->local_size > 1) - { - /* Destroy the fast query object. */ - pami_extension_collsel_query_destroy pamix_collsel_query_destroy = - (pami_extension_collsel_query_destroy) PAMI_Extension_symbol(MPIDI_Collsel_extension, - "Collsel_query_destroy"); - if(pamix_collsel_query_destroy != NULL) - { - pamix_collsel_query_destroy(&(comm->mpid.collsel_fast_query)); - } - } - - TRACE_ERR("Destroying geometry\n"); - - geom_destroy_post.client = MPIDI_Client; - geom_destroy_post.geom = &comm->mpid.geometry; - geom_destroy_post.fn = geom_destroy_cb_done; - geom_destroy_post.cookie = (void *)&geom_destroy; - TRACE_ERR("%s geom_destroy\n", MPIDI_Process.context_post>0?"Posting":"Invoking"); - MPIDI_Context_post(MPIDI_Context[0], &geom_destroy_post.state, - geom_destroy_wrapper, (void *)&geom_destroy_post); - - TRACE_ERR("Waiting for geom destroy to finish\n"); - MPID_PROGRESS_WAIT_WHILE(geom_destroy); - MPID_VCR_FREE_LPIDS(comm->mpid.tasks); -/* TRACE_ERR("Freeing geometry ranges\n"); - MPIU_TestFree(&comm->mpid.tasks_descriptor.ranges); -*/ - TRACE_ERR("MPIDI_Coll_comm_destroy exit\n"); -} diff --git a/src/mpid/pamid/src/comm/mpid_optcolls.c b/src/mpid/pamid/src/comm/mpid_optcolls.c deleted file mode 100644 index d64ed04..0000000 --- a/src/mpid/pamid/src/comm/mpid_optcolls.c +++ /dev/null @@ -1,1047 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/comm/mpid_optcolls.c - * \brief Code for setting up optimized collectives per geometry/communicator - * \brief currently uses strcmp and known protocol performance - */ - - - -/*#define TRACE_ON */ - -#include <mpidimpl.h> - - -static int MPIDI_Check_FCA_envvar(char *string, int *user_range_hi) -{ - char *env = getenv("MP_MPI_PAMI_FOR"); - if(env != NULL) - { - if(strcasecmp(env, "ALL") == 0) - return 1; - int len = strlen(env); - len++; - char *temp = MPL_malloc(sizeof(char) * len); - char *ptrToFree = temp; - strcpy(temp, env); - char *sepptr; - for(sepptr = temp; (sepptr = strsep(&temp, ",")) != NULL ; ) - { - char *subsepptr, *temp_sepptr; - temp_sepptr = sepptr; - subsepptr = strsep(&temp_sepptr, ":"); - if(temp_sepptr != NULL)/* SSS: There is a a colon for this collective */ - { - if(strcasecmp(subsepptr, string) == 0) - { - *user_range_hi = atoi(temp_sepptr); - MPL_free(ptrToFree); - return 1; - } - else - sepptr++; - } - else - { - if(strcasecmp(sepptr, string) == 0) - { - *user_range_hi = -1; - MPL_free(ptrToFree); - return 1; - } - else - sepptr++; - } - } - /* We didn't find it, but the end var was set, so return 0 */ - MPL_free(ptrToFree); - return 0; - } - if(MPIDI_Process.optimized.collectives == MPID_COLL_FCA) - return 1; /* To have gotten this far, opt colls are on. If the env doesn't exist, - we should use the FCA protocol for "string" */ - else - return -1; /* we don't have any FCA things */ -} - -static inline void -MPIDI_Coll_comm_check_FCA(char *coll_name, - char *protocol_name, - int pami_xfer, - int query_type, - int proto_num, - MPIR_Comm *comm_ptr) -{ - int opt_proto = -1; - int i; - int user_range_hi = -1;/* SSS: By default we assume user hasn't defined a range_hi (cutoff_size) */ -#ifdef TRACE_ON - char *envstring = getenv("MP_MPI_PAMI_FOR"); -#endif - TRACE_ERR("Checking for %s in %s\n", coll_name, envstring); - int check_var = MPIDI_Check_FCA_envvar(coll_name, &user_range_hi); - if(check_var == 1) - { - TRACE_ERR("Found %s\n",coll_name); - /* Look for protocol_name in the "always works list */ - for(i = 0; i <comm_ptr->mpid.coll_count[pami_xfer][0]; i++) - { - if(strcasecmp(comm_ptr->mpid.coll_metadata[pami_xfer][0][i].name, protocol_name) == 0) - { - opt_proto = i; - break; - } - } - if(opt_proto != -1) /* we found it, so copy it to the optimized var */ - { - TRACE_ERR("Memcpy protocol type %d, number %d (%s) to optimized protocol\n", - pami_xfer, opt_proto, - comm_ptr->mpid.coll_metadata[pami_xfer][0][opt_proto].name); - comm_ptr->mpid.opt_protocol[pami_xfer][proto_num] = - comm_ptr->mpid.coll_algorithm[pami_xfer][0][opt_proto]; - memcpy(&comm_ptr->mpid.opt_protocol_md[pami_xfer][proto_num], - &comm_ptr->mpid.coll_metadata[pami_xfer][0][opt_proto], - sizeof(pami_metadata_t)); - comm_ptr->mpid.must_query[pami_xfer][proto_num] = query_type; - if(user_range_hi != -1) - comm_ptr->mpid.cutoff_size[pami_xfer][proto_num] = user_range_hi; - else - comm_ptr->mpid.cutoff_size[pami_xfer][proto_num] = 0; - comm_ptr->mpid.user_selected_type[pami_xfer] = MPID_COLL_OPTIMIZED; - } - else /* see if it is in the must query list instead */ - { - for(i = 0; i <comm_ptr->mpid.coll_count[pami_xfer][1]; i++) - { - if(strcasecmp(comm_ptr->mpid.coll_metadata[pami_xfer][1][i].name, protocol_name) == 0) - { - opt_proto = i; - break; - } - } - if(opt_proto != -1) /* ok, it was in the must query list */ - { - TRACE_ERR("Memcpy protocol type %d, number %d (%s) to optimized protocol\n", - pami_xfer, opt_proto, - comm_ptr->mpid.coll_metadata[pami_xfer][1][opt_proto].name); - comm_ptr->mpid.opt_protocol[pami_xfer][proto_num] = - comm_ptr->mpid.coll_algorithm[pami_xfer][1][opt_proto]; - memcpy(&comm_ptr->mpid.opt_protocol_md[pami_xfer][proto_num], - &comm_ptr->mpid.coll_metadata[pami_xfer][1][opt_proto], - sizeof(pami_metadata_t)); - comm_ptr->mpid.must_query[pami_xfer][proto_num] = query_type; - if(user_range_hi != -1) - comm_ptr->mpid.cutoff_size[pami_xfer][proto_num] = user_range_hi; - else - comm_ptr->mpid.cutoff_size[pami_xfer][proto_num] = 0; - comm_ptr->mpid.user_selected_type[pami_xfer] = MPID_COLL_OPTIMIZED; - } - else /* that protocol doesn't exist */ - { - TRACE_ERR("Couldn't find %s in the list for %s, reverting to MPICH\n",protocol_name,coll_name); - comm_ptr->mpid.user_selected_type[pami_xfer] = MPID_COLL_USE_MPICH; - } - } - } - else if(check_var == 0)/* The env var was set, but wasn't set for coll_name */ - { - TRACE_ERR("Couldn't find any optimal %s protocols or user chose not to set it. Selecting MPICH\n",coll_name); - comm_ptr->mpid.user_selected_type[pami_xfer] = MPID_COLL_USE_MPICH; - } - else - return; -} - - -void MPIDI_Comm_coll_select(MPIR_Comm *comm_ptr) -{ - TRACE_ERR("Entering MPIDI_Comm_coll_select\n"); - int opt_proto = -1; - int mustquery = 0; - int i; - int use_threaded_collectives = 1; - - /* Some highly optimized protocols (limited resource) do not - support MPI_THREAD_MULTIPLE semantics so do not enable them - except on COMM_WORLD. - NOTE: we are not checking metadata because these are known, - hardcoded optimized protocols. - */ - if((MPIR_ThreadInfo.thread_provided == MPI_THREAD_MULTIPLE) && - (comm_ptr != MPIR_Process.comm_world)) use_threaded_collectives = 0; - if(unlikely(MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_0 && comm_ptr->rank == 0)) - fprintf(stderr, "thread_provided=%s, %scomm_world, use_threaded_collectives %u\n", - MPIR_ThreadInfo.thread_provided == MPI_THREAD_MULTIPLE? "MPI_THREAD_MULTIPLE": - MPIR_ThreadInfo.thread_provided == MPI_THREAD_SINGLE?"MPI_THREAD_SINGLE": - MPIR_ThreadInfo.thread_provided == MPI_THREAD_FUNNELED?"MPI_THREAD_FUNNELED": - MPIR_ThreadInfo.thread_provided == MPI_THREAD_SERIALIZED?"MPI_THREAD_SERIALIZED": - "??", - (comm_ptr != MPIR_Process.comm_world)?"!":"", - use_threaded_collectives); - - /* First, setup the (easy, allreduce is complicated) FCA collectives if there - * are any because they are always usable when they are on */ - if(comm_ptr->mpid.user_selected_type[PAMI_XFER_REDUCE] == MPID_COLL_NOSELECTION) - { - MPIDI_Coll_comm_check_FCA("REDUCE","I1:Reduce:FCA:FCA",PAMI_XFER_REDUCE,MPID_COLL_CHECK_FN_REQUIRED, 0, comm_ptr); - } - if(comm_ptr->mpid.user_selected_type[PAMI_XFER_ALLGATHER] == MPID_COLL_NOSELECTION) - { - MPIDI_Coll_comm_check_FCA("ALLGATHER","I1:Allgather:FCA:FCA",PAMI_XFER_ALLGATHER,MPID_COLL_NOQUERY, 0, comm_ptr); - } - if(comm_ptr->mpid.user_selected_type[PAMI_XFER_ALLGATHERV_INT] == MPID_COLL_NOSELECTION) - { - MPIDI_Coll_comm_check_FCA("ALLGATHERV","I1:AllgathervInt:FCA:FCA",PAMI_XFER_ALLGATHERV_INT,MPID_COLL_NOQUERY, 0, comm_ptr); - } - if(comm_ptr->mpid.user_selected_type[PAMI_XFER_BROADCAST] == MPID_COLL_NOSELECTION) - { - MPIDI_Coll_comm_check_FCA("BCAST", "I1:Broadcast:FCA:FCA", PAMI_XFER_BROADCAST, MPID_COLL_NOQUERY, 0, comm_ptr); - MPIDI_Coll_comm_check_FCA("BCAST", "I1:Broadcast:FCA:FCA", PAMI_XFER_BROADCAST, MPID_COLL_NOQUERY, 1, comm_ptr); - } - if(comm_ptr->mpid.user_selected_type[PAMI_XFER_BARRIER] == MPID_COLL_NOSELECTION) - { - MPIDI_Coll_comm_check_FCA("BARRIER","I1:Barrier:FCA:FCA",PAMI_XFER_BARRIER,MPID_COLL_NOQUERY, 0, comm_ptr); - } - /* SSS: There isn't really an FCA Gatherv protocol. We do this call to force the use of MPICH for gatherv - * when FCA is enabled so we don't have to use PAMI protocol. */ - if(comm_ptr->mpid.user_selected_type[PAMI_XFER_GATHERV_INT] == MPID_COLL_NOSELECTION) - { - MPIDI_Coll_comm_check_FCA("GATHERV","I1:GathervInt:FCA:FCA",PAMI_XFER_GATHERV_INT,MPID_COLL_NOQUERY, 0, comm_ptr); - } - - opt_proto = -1; - mustquery = 0; - /* So, several protocols are really easy. Tackle them first. */ - if(comm_ptr->mpid.user_selected_type[PAMI_XFER_ALLGATHERV_INT] == MPID_COLL_NOSELECTION) - { - TRACE_ERR("No allgatherv[int] env var, so setting optimized allgatherv[int]\n"); - /* Use I0:RectangleDput */ - for(i = 0; i < comm_ptr->mpid.coll_count[PAMI_XFER_ALLGATHERV_INT][1]; i++) - { - if(strcasecmp(comm_ptr->mpid.coll_metadata[PAMI_XFER_ALLGATHERV_INT][1][i].name, "I0:RectangleDput:SHMEM:MU") == 0) - { - opt_proto = i; - mustquery = 1; - break; - } - } - if(opt_proto != -1) - { - TRACE_ERR("Memcpy protocol type %d number %d (%s) to optimized protocol\n", - PAMI_XFER_ALLGATHERV_INT, opt_proto, - comm_ptr->mpid.coll_metadata[PAMI_XFER_ALLGATHERV_INT][mustquery][opt_proto].name); - comm_ptr->mpid.opt_protocol[PAMI_XFER_ALLGATHERV_INT][0] = - comm_ptr->mpid.coll_algorithm[PAMI_XFER_ALLGATHERV_INT][mustquery][opt_proto]; - memcpy(&comm_ptr->mpid.opt_protocol_md[PAMI_XFER_ALLGATHERV_INT][0], - &comm_ptr->mpid.coll_metadata[PAMI_XFER_ALLGATHERV_INT][mustquery][opt_proto], - sizeof(pami_metadata_t)); - comm_ptr->mpid.must_query[PAMI_XFER_ALLGATHERV_INT][0] = mustquery?MPID_COLL_ALWAYS_QUERY:MPID_COLL_NOQUERY; - comm_ptr->mpid.user_selected_type[PAMI_XFER_ALLGATHERV_INT] = MPID_COLL_OPTIMIZED; - } - else /* no optimized allgatherv? */ - { - TRACE_ERR("Couldn't find optimial allgatherv[int] protocol\n"); - comm_ptr->mpid.user_selected_type[PAMI_XFER_ALLGATHERV_INT] = MPID_COLL_USE_MPICH; - comm_ptr->mpid.opt_protocol[PAMI_XFER_ALLGATHERV_INT][0] = 0; - comm_ptr->mpid.allgathervs[0] = 1; /* Use GLUE_ALLREDUCE */ - } - TRACE_ERR("Done setting optimized allgatherv[int]\n"); - } - - if(comm_ptr->mpid.user_selected_type[PAMI_XFER_GATHER] == MPID_COLL_NOSELECTION) - { - TRACE_ERR("Default gather to MPICH\n"); - comm_ptr->mpid.user_selected_type[PAMI_XFER_GATHER] = MPID_COLL_USE_MPICH; - comm_ptr->mpid.opt_protocol[PAMI_XFER_GATHER][0] = 0; - } - - opt_proto = -1; - mustquery = 0; - /* Scatter */ - /* If the user has forced a selection, don't bother setting it here */ - if(comm_ptr->mpid.user_selected_type[PAMI_XFER_SCATTER] == MPID_COLL_NOSELECTION) - { - TRACE_ERR("No scatter env var, so setting optimized scatter\n"); - /* The best scatter seems to be I0:Binomial (at least better than I0:Flat) */ - /* Loop over the protocols until we find the one we want */ - for(i = 0; i < comm_ptr->mpid.coll_count[PAMI_XFER_SCATTER][0]; i++) - { - if(strcasecmp(comm_ptr->mpid.coll_metadata[PAMI_XFER_SCATTER][0][i].name, "I0:Binomial:-:MU") == 0) - { - opt_proto = i; - break; - } - if(strcasecmp(comm_ptr->mpid.coll_metadata[PAMI_XFER_SCATTER][0][i].name, "I0:Binomial:SHMEM:MU") == 0) - { - opt_proto = i; - break; - } - } - if(opt_proto == -1) /* check other list */ - for(i = 0; i < comm_ptr->mpid.coll_count[PAMI_XFER_SCATTER][1]; i++) - { - if(strcasecmp(comm_ptr->mpid.coll_metadata[PAMI_XFER_SCATTER][1][i].name, "I0:Binomial:-:MU") == 0) - { - opt_proto = i; - mustquery = 1; - break; - } - if(strcasecmp(comm_ptr->mpid.coll_metadata[PAMI_XFER_SCATTER][1][i].name, "I0:Binomial:SHMEM:MU") == 0) - { - opt_proto = i; - mustquery = 1; - break; - } - } - /* Actually, MPICH looks better right now, so use MPICH - if(opt_proto != -1) - { - TRACE_ERR("Memcpy protocol type %d, number %d (%s) to optimized protocol\n", - PAMI_XFER_SCATTER, opt_proto, - comm_ptr->mpid.coll_metadata[PAMI_XFER_SCATTER][mustquery][opt_proto].name); - comm_ptr->mpid.opt_protocol[PAMI_XFER_SCATTER][0] = - comm_ptr->mpid.coll_algorithm[PAMI_XFER_SCATTER][mustquery][opt_proto]; - memcpy(&comm_ptr->mpid.opt_protocol_md[PAMI_XFER_SCATTER][0], - &comm_ptr->mpid.coll_metadata[PAMI_XFER_SCATTER][mustquery][opt_proto], - sizeof(pami_metadata_t)); - comm_ptr->mpid.must_query[PAMI_XFER_SCATTER][0] = mustquery?MPID_COLL_ALWAYS_QUERY:MPID_COLL_NOQUERY; - comm_ptr->mpid.user_selected_type[PAMI_XFER_SCATTER] = MPID_COLL_OPTIMIZED; - } - else - */ - { - TRACE_ERR("Scatter reverting to MPICH\n"); - comm_ptr->mpid.user_selected_type[PAMI_XFER_SCATTER] = MPID_COLL_USE_MPICH; - comm_ptr->mpid.opt_protocol[PAMI_XFER_SCATTER][0] = 0; - } - TRACE_ERR("Done setting optimized scatter\n"); - } - opt_proto = -1; - mustquery = 0; - /* Scatterv */ - /* If the user has forced a selection, don't bother setting it here */ - if(comm_ptr->mpid.user_selected_type[PAMI_XFER_SCATTERV_INT] == MPID_COLL_NOSELECTION) - { - TRACE_ERR("No scatterv env var, so setting optimized scatterv\n"); - /* The best scatterv seems to be I0:SoftwareTree: */ - /* Loop over the protocols until we find the one we want */ - for(i = 0; i < comm_ptr->mpid.coll_count[PAMI_XFER_SCATTERV_INT][0]; i++) - { - if(strcasecmp(comm_ptr->mpid.coll_metadata[PAMI_XFER_SCATTERV_INT][0][i].name, "I0:SoftwareTree:-:MU") == 0) - { - opt_proto = i; - break; - } - if(strcasecmp(comm_ptr->mpid.coll_metadata[PAMI_XFER_SCATTERV_INT][0][i].name, "I0:SoftwareTree:SHMEM:MU") == 0) - { - opt_proto = i; - break; - } - } - if(opt_proto == -1) /* check other list */ - for(i = 0; i < comm_ptr->mpid.coll_count[PAMI_XFER_SCATTERV_INT][1]; i++) - { - if(strcasecmp(comm_ptr->mpid.coll_metadata[PAMI_XFER_SCATTERV_INT][1][i].name, "I0:SoftwareTree:-:MU") == 0) - { - opt_proto = i; - mustquery = 1; - break; - } - if(strcasecmp(comm_ptr->mpid.coll_metadata[PAMI_XFER_SCATTERV_INT][1][i].name, "I0:SoftwareTree:SHMEM:MU") == 0) - { - opt_proto = i; - mustquery = 1; - break; - } - } - if(opt_proto != -1) - { - TRACE_ERR("Memcpy protocol type %d, number %d (%s) to optimized protocol\n", - PAMI_XFER_SCATTERV_INT, opt_proto, - comm_ptr->mpid.coll_metadata[PAMI_XFER_SCATTERV_INT][mustquery][opt_proto].name); - comm_ptr->mpid.opt_protocol[PAMI_XFER_SCATTERV_INT][0] = - comm_ptr->mpid.coll_algorithm[PAMI_XFER_SCATTERV_INT][mustquery][opt_proto]; - memcpy(&comm_ptr->mpid.opt_protocol_md[PAMI_XFER_SCATTERV_INT][0], - &comm_ptr->mpid.coll_metadata[PAMI_XFER_SCATTERV_INT][mustquery][opt_proto], - sizeof(pami_metadata_t)); - comm_ptr->mpid.must_query[PAMI_XFER_SCATTERV_INT][0] = mustquery?MPID_COLL_ALWAYS_QUERY:MPID_COLL_NOQUERY; - comm_ptr->mpid.user_selected_type[PAMI_XFER_SCATTERV_INT] = MPID_COLL_OPTIMIZED; - } - else - { - TRACE_ERR("Scatterv reverting to MPICH\n"); - comm_ptr->mpid.user_selected_type[PAMI_XFER_SCATTERV_INT] = MPID_COLL_USE_MPICH; - comm_ptr->mpid.opt_protocol[PAMI_XFER_SCATTERV_INT][0] = 0; - } - TRACE_ERR("Done setting optimized scatterv\n"); - } - - - opt_proto = -1; - mustquery = 0; - /* Alltoall */ - /* If the user has forced a selection, don't bother setting it here */ - if(comm_ptr->mpid.user_selected_type[PAMI_XFER_ALLTOALL] == MPID_COLL_NOSELECTION) - { - TRACE_ERR("No alltoall env var, so setting optimized alltoall\n"); - /* The best alltoall is always I0:M2MComposite:MU:MU */ - /* Loop over the protocols until we find the one we want */ - if(use_threaded_collectives) - for(i = 0; i < comm_ptr->mpid.coll_count[PAMI_XFER_ALLTOALL][0]; i++) - { - if(strcasecmp(comm_ptr->mpid.coll_metadata[PAMI_XFER_ALLTOALL][0][i].name, "I0:M2MComposite:MU:MU") == 0) - { - opt_proto = i; - break; - } - } - if(use_threaded_collectives && (opt_proto == -1)) /* check other list */ - for(i = 0; i < comm_ptr->mpid.coll_count[PAMI_XFER_ALLTOALL][1]; i++) - { - if(strcasecmp(comm_ptr->mpid.coll_metadata[PAMI_XFER_ALLTOALL][1][i].name, "I0:M2MComposite:MU:MU") == 0) - { - opt_proto = i; - mustquery = 1; - break; - } - } - if(opt_proto != -1) - { - TRACE_ERR("Memcpy protocol type %d, number %d (%s) to optimized protocol\n", - PAMI_XFER_ALLTOALL, opt_proto, - comm_ptr->mpid.coll_metadata[PAMI_XFER_ALLTOALL][mustquery][opt_proto].name); - comm_ptr->mpid.opt_protocol[PAMI_XFER_ALLTOALL][0] = - comm_ptr->mpid.coll_algorithm[PAMI_XFER_ALLTOALL][mustquery][opt_proto]; - memcpy(&comm_ptr->mpid.opt_protocol_md[PAMI_XFER_ALLTOALL][0], - &comm_ptr->mpid.coll_metadata[PAMI_XFER_ALLTOALL][mustquery][opt_proto], - sizeof(pami_metadata_t)); - comm_ptr->mpid.must_query[PAMI_XFER_ALLTOALL][0] = mustquery?MPID_COLL_ALWAYS_QUERY:MPID_COLL_NOQUERY; - comm_ptr->mpid.user_selected_type[PAMI_XFER_ALLTOALL] = MPID_COLL_OPTIMIZED; - } - else - { - TRACE_ERR("Couldn't find I0:M2MComposite:MU:MU in the list for alltoall, reverting to MPICH\n"); - comm_ptr->mpid.user_selected_type[PAMI_XFER_ALLTOALL] = MPID_COLL_USE_MPICH; - comm_ptr->mpid.opt_protocol[PAMI_XFER_ALLTOALL][0] = 0; - } - TRACE_ERR("Done setting optimized alltoall\n"); - } - - - opt_proto = -1; - mustquery = 0; - /* Alltoallv */ - if(comm_ptr->mpid.user_selected_type[PAMI_XFER_ALLTOALLV_INT] == MPID_COLL_NOSELECTION) - { - TRACE_ERR("No alltoallv env var, so setting optimized alltoallv\n"); - /* The best alltoall is always I0:M2MComposite:MU:MU */ - /* Loop over the protocols until we find the one we want */ - if(use_threaded_collectives) - for(i = 0; i < comm_ptr->mpid.coll_count[PAMI_XFER_ALLTOALLV_INT][0]; i++) - { - if(strcasecmp(comm_ptr->mpid.coll_metadata[PAMI_XFER_ALLTOALLV_INT][0][i].name, "I0:M2MComposite:MU:MU") == 0) - { - opt_proto = i; - break; - } - } - if(use_threaded_collectives && (opt_proto == -1)) /* check other list */ - for(i = 0; i <comm_ptr->mpid.coll_count[PAMI_XFER_ALLTOALLV_INT][1]; i++) - { - if(strcasecmp(comm_ptr->mpid.coll_metadata[PAMI_XFER_ALLTOALLV_INT][1][i].name, "I0:M2MComposite:MU:MU") == 0) - { - opt_proto = i; - mustquery = 1; - break; - } - } - if(opt_proto != -1) - { - TRACE_ERR("Memcpy protocol type %d, number %d (%s) to optimized protocol\n", - PAMI_XFER_ALLTOALLV_INT, opt_proto, - comm_ptr->mpid.coll_metadata[PAMI_XFER_ALLTOALLV_INT][mustquery][opt_proto].name); - comm_ptr->mpid.opt_protocol[PAMI_XFER_ALLTOALLV_INT][0] = - comm_ptr->mpid.coll_algorithm[PAMI_XFER_ALLTOALLV_INT][mustquery][opt_proto]; - memcpy(&comm_ptr->mpid.opt_protocol_md[PAMI_XFER_ALLTOALLV_INT][0], - &comm_ptr->mpid.coll_metadata[PAMI_XFER_ALLTOALLV_INT][mustquery][opt_proto], - sizeof(pami_metadata_t)); - comm_ptr->mpid.must_query[PAMI_XFER_ALLTOALLV_INT][0] = mustquery?MPID_COLL_ALWAYS_QUERY:MPID_COLL_NOQUERY; - comm_ptr->mpid.user_selected_type[PAMI_XFER_ALLTOALLV_INT] = MPID_COLL_OPTIMIZED; - } - else - { - TRACE_ERR("Couldn't find I0:M2MComposite:MU:MU in the list for alltoallv, reverting to MPICH\n"); - comm_ptr->mpid.user_selected_type[PAMI_XFER_ALLTOALLV_INT] = MPID_COLL_USE_MPICH; - comm_ptr->mpid.opt_protocol[PAMI_XFER_ALLTOALLV_INT][0] = 0; - } - TRACE_ERR("Done setting optimized alltoallv\n"); - } - - opt_proto = -1; - mustquery = 0; - /* Barrier */ - if(comm_ptr->mpid.user_selected_type[PAMI_XFER_BARRIER] == MPID_COLL_NOSELECTION) - { - TRACE_ERR("No barrier env var, so setting optimized barrier\n"); - /* For 1ppn, I0:MultiSync:-:GI is best, followed by - * I0:RectangleMultiSync:-:MU, followed by - * I0:OptBinomial:P2P:P2P - */ - /* For 16 and 64 ppn, I0:MultiSync2Device:SHMEM:GI (which doesn't exist at 1ppn) - * is best, followed by - * I0:RectangleMultiSync2Device:SHMEM:MU for rectangles, followed by - * I0:OptBinomial:P2P:P2P */ - /* So we basically check for the protocols in reverse-optimal order */ - - - /* In order, >1ppn we use - * I0:MultiSync2Device:SHMEM:GI - * I0:RectangleMultiSync2Device:SHMEM:MU - * I0:OptBinomial:P2P:P2P - * MPICH - * - * In order 1ppn we use - * I0:MultiSync:-:GI - * I0:RectangleMultiSync:-:MU - * I0:OptBinomial:P2P:P2P - * MPICH - */ - if(use_threaded_collectives) - for(i = 0 ; i < comm_ptr->mpid.coll_count[PAMI_XFER_BARRIER][0]; i++) - { - /* These two are mutually exclusive */ - if(strcasecmp(comm_ptr->mpid.coll_metadata[PAMI_XFER_BARRIER][0][i].name, "I0:MultiSync2Device:SHMEM:GI") == 0) - { - opt_proto = i; - } - if(strcasecmp(comm_ptr->mpid.coll_metadata[PAMI_XFER_BARRIER][0][i].name, "I0:MultiSync:-:GI") == 0) - { - opt_proto = i; - } - } - /* Next best rectangular to check */ - if(opt_proto == -1) - { - if(use_threaded_collectives) - for(i = 0 ; i < comm_ptr->mpid.coll_count[PAMI_XFER_BARRIER][0]; i++) - { - if(strcasecmp(comm_ptr->mpid.coll_metadata[PAMI_XFER_BARRIER][0][i].name, "I0:RectangleMultiSync2Device:SHMEM:MU") == 0) - opt_proto = i; - if(strcasecmp(comm_ptr->mpid.coll_metadata[PAMI_XFER_BARRIER][0][i].name, "I0:RectangleMultiSync:-:MU") == 0) - opt_proto = i; - } - } - /* Finally, see if we have opt binomial */ - if(opt_proto == -1) - { - for(i = 0 ; i < comm_ptr->mpid.coll_count[PAMI_XFER_BARRIER][0]; i++) - { - if(strcasecmp(comm_ptr->mpid.coll_metadata[PAMI_XFER_BARRIER][0][i].name, "I0:OptBinomial:P2P:P2P") == 0) - opt_proto = i; - } - } - - if(opt_proto != -1) - { - TRACE_ERR("Memcpy protocol type %d, number %d (%s) to optimize protocol\n", - PAMI_XFER_BARRIER, opt_proto, - comm_ptr->mpid.coll_metadata[PAMI_XFER_BARRIER][0][opt_proto].name); - - comm_ptr->mpid.opt_protocol[PAMI_XFER_BARRIER][0] = - comm_ptr->mpid.coll_algorithm[PAMI_XFER_BARRIER][0][opt_proto]; - memcpy(&comm_ptr->mpid.opt_protocol_md[PAMI_XFER_BARRIER][0], - &comm_ptr->mpid.coll_metadata[PAMI_XFER_BARRIER][0][opt_proto], - sizeof(pami_metadata_t)); - comm_ptr->mpid.must_query[PAMI_XFER_BARRIER][0] = MPID_COLL_NOQUERY; - comm_ptr->mpid.user_selected_type[PAMI_XFER_BARRIER] = MPID_COLL_OPTIMIZED; - } - else - { - TRACE_ERR("Couldn't find any optimal barrier protocols. Selecting MPICH\n"); - comm_ptr->mpid.user_selected_type[PAMI_XFER_BARRIER] = MPID_COLL_USE_MPICH; - comm_ptr->mpid.opt_protocol[PAMI_XFER_BARRIER][0] = 0; - } - - TRACE_ERR("Done setting optimized barrier\n"); - } - - opt_proto = -1; - mustquery = 0; - - /* This becomes messy when we have to message sizes. If we were gutting the - * existing framework, it might be easier, but I think the existing framework - * is useful for future collective selection libraries/mechanisms, so I'd - * rather leave it in place and deal with it here instead.... */ - - /* Broadcast */ - /* 1ppn */ - /* small messages: I0:MulticastDput:-:MU - * if it exists, I0:RectangleDput:MU:MU for >64k */ - - /* 16ppn */ - /* small messages: I0:MultiCastDput:SHMEM:MU - * for 16ppn/1node: I0:MultiCastDput:SHMEM:- perhaps - * I0:RectangleDput:SHMEM:MU for >128k */ - /* nonrect(?): I0:MultiCast2DeviceDput:SHMEM:MU - * no hw: I0:Sync2-nary:Shmem:MUDput */ - /* 64ppn */ - /* all sizes: I0:MultiCastDput:SHMEM:MU */ - /* otherwise, I0:2-nomial:SHMEM:MU */ - - - - /* First, set up small message bcasts */ - if(comm_ptr->mpid.user_selected_type[PAMI_XFER_BROADCAST] == MPID_COLL_NOSELECTION) - { - /* Complicated exceptions: */ - /* I0:RankBased_Binomial:-:ShortMU is good on irregular for <256 bytes */ - /* I0:MultiCast:SHMEM:- is good at 1 node/16ppn, which is a SOW point */ - TRACE_ERR("No bcast env var, so setting optimized bcast\n"); - - if(use_threaded_collectives) - for(i = 0 ; i < comm_ptr->mpid.coll_count[PAMI_XFER_BROADCAST][1]; i++) - { - /* These two are mutually exclusive */ - if(strcasecmp(comm_ptr->mpid.coll_metadata[PAMI_XFER_BROADCAST][1][i].name, "I0:MultiCastDput:-:MU") == 0) - { - opt_proto = i; - mustquery = 1; - break; - } - if(strcasecmp(comm_ptr->mpid.coll_metadata[PAMI_XFER_BROADCAST][1][i].name, "I0:MultiCastDput:SHMEM:MU") == 0) - { - opt_proto = i; - mustquery = 1; - break; - } - } - /* Next best MU 2 device to check */ - if(use_threaded_collectives) - if(opt_proto == -1) - { - if(use_threaded_collectives) - for(i = 0; i < comm_ptr->mpid.coll_count[PAMI_XFER_BROADCAST][1]; i++) - { - if(strcasecmp(comm_ptr->mpid.coll_metadata[PAMI_XFER_BROADCAST][1][i].name, "I0:MultiCast2DeviceDput:SHMEM:MU") == 0) - { - opt_proto = i; - mustquery = 1; - break; - } - } - } - /* Check for rectangle */ - if(use_threaded_collectives) - if(opt_proto == -1) - { - unsigned len = strlen("I0:RectangleDput:"); - if(use_threaded_collectives) - for(i = 0; i < comm_ptr->mpid.coll_count[PAMI_XFER_BROADCAST][1]; i++) - { - if(strcasecmp (comm_ptr->mpid.coll_metadata[PAMI_XFER_BROADCAST][1][i].name, "I0:RectangleDput:SHMEM:MU") == 0) - { /* Prefer the :SHMEM:MU so break when it's found */ - opt_proto = i; - mustquery = 1; - break; - } - /* Otherwise any RectangleDput is better than nothing. */ - if(strncasecmp(comm_ptr->mpid.coll_metadata[PAMI_XFER_BROADCAST][1][i].name, "I0:RectangleDput:",len) == 0) - { - opt_proto = i; - mustquery = 1; - } - } - } - if(opt_proto == -1) - { - /* this protocol is sometimes query, sometimes always works so check both lists but prefer always works */ - for(i = 0; i < comm_ptr->mpid.coll_count[PAMI_XFER_BROADCAST][0]; i++) - { - /* This is a good choice for small messages only */ - if(strcasecmp(comm_ptr->mpid.coll_metadata[PAMI_XFER_BROADCAST][0][i].name, "I0:SequenceBased_Binomial:SHMEM:MU") == 0) - { - opt_proto = i; - comm_ptr->mpid.cutoff_size[PAMI_XFER_BROADCAST][0] = 256; - break; - } - } - if(opt_proto == -1) for(i = 0; i < comm_ptr->mpid.coll_count[PAMI_XFER_BROADCAST][1]; i++) - { - /* This is a good choice for small messages only */ - if(strcasecmp(comm_ptr->mpid.coll_metadata[PAMI_XFER_BROADCAST][1][i].name, "I0:SequenceBased_Binomial:SHMEM:MU") == 0) - { - opt_proto = i; - mustquery = 1; - comm_ptr->mpid.cutoff_size[PAMI_XFER_BROADCAST][0] = 256; - break; - } - } - } - /* Next best to check */ - if(opt_proto == -1) - { - /* this protocol is sometimes query, sometimes always works so check both lists but prefer always works */ - for(i = 0; i < comm_ptr->mpid.coll_count[PAMI_XFER_BROADCAST][0]; i++) - { - if(strcasecmp(comm_ptr->mpid.coll_metadata[PAMI_XFER_BROADCAST][0][i].name, "I0:2-nomial:SHMEM:MU") == 0) - opt_proto = i; - } - if(opt_proto == -1) for(i = 0; i < comm_ptr->mpid.coll_count[PAMI_XFER_BROADCAST][1]; i++) - { - if(strcasecmp(comm_ptr->mpid.coll_metadata[PAMI_XFER_BROADCAST][1][i].name, "I0:2-nomial:SHMEM:MU") == 0) - opt_proto = i; - mustquery = 1; - } - } - - /* These protocols are good for most message sizes, but there are some - * better choices for larger messages */ - /* Set opt_proto for bcast[0] right now */ - if(opt_proto != -1) - { - TRACE_ERR("Memcpy protocol type %d, number %d (%s) to optimize protocol 0\n", - PAMI_XFER_BROADCAST, opt_proto, - comm_ptr->mpid.coll_metadata[PAMI_XFER_BROADCAST][mustquery][opt_proto].name); - - comm_ptr->mpid.opt_protocol[PAMI_XFER_BROADCAST][0] = - comm_ptr->mpid.coll_algorithm[PAMI_XFER_BROADCAST][mustquery][opt_proto]; - memcpy(&comm_ptr->mpid.opt_protocol_md[PAMI_XFER_BROADCAST][0], - &comm_ptr->mpid.coll_metadata[PAMI_XFER_BROADCAST][mustquery][opt_proto], - sizeof(pami_metadata_t)); - comm_ptr->mpid.must_query[PAMI_XFER_BROADCAST][0] = mustquery?MPID_COLL_ALWAYS_QUERY:MPID_COLL_NOQUERY; - comm_ptr->mpid.user_selected_type[PAMI_XFER_BROADCAST] = MPID_COLL_OPTIMIZED; - } - else - { - TRACE_ERR("Couldn't find any optimal bcast protocols. Selecting MPICH\n"); - comm_ptr->mpid.user_selected_type[PAMI_XFER_BROADCAST] = MPID_COLL_USE_MPICH; - comm_ptr->mpid.opt_protocol[PAMI_XFER_BROADCAST][0] = 0; - } - - TRACE_ERR("Done setting optimized bcast 0\n"); - - /* Now, look into large message bcasts */ - opt_proto = -1; - mustquery = 0; - /* If bcast0 is I0:MultiCastDput:-:MU, and I0:RectangleDput:MU:MU is available, use - * it for 64k messages */ - if(comm_ptr->mpid.user_selected_type[PAMI_XFER_BROADCAST] != MPID_COLL_USE_MPICH) - { - if(use_threaded_collectives) - if(strcasecmp(comm_ptr->mpid.opt_protocol_md[PAMI_XFER_BROADCAST][0].name, "I0:MultiCastDput:-:MU") == 0) - { - /* See if I0:RectangleDput:MU:MU is available */ - for(i = 0; i < comm_ptr->mpid.coll_count[PAMI_XFER_BROADCAST][1]; i++) - { - if(strcasecmp(comm_ptr->mpid.coll_metadata[PAMI_XFER_BROADCAST][1][i].name, "I0:RectangleDput:MU:MU") == 0) - { - opt_proto = i; - mustquery = 1; - comm_ptr->mpid.cutoff_size[PAMI_XFER_BROADCAST][0] = 65536; - break; - } - } - } - /* small messages: I0:MultiCastDput:SHMEM:MU*/ - /* I0:RectangleDput:SHMEM:MU for >128k */ - if(use_threaded_collectives) - if(strcasecmp(comm_ptr->mpid.opt_protocol_md[PAMI_XFER_BROADCAST][0].name, "I0:MultiCastDput:SHMEM:MU") == 0) - { - /* See if I0:RectangleDput:SHMEM:MU is available */ - for(i = 0; i < comm_ptr->mpid.coll_count[PAMI_XFER_BROADCAST][1]; i++) - { - if(strcasecmp(comm_ptr->mpid.coll_metadata[PAMI_XFER_BROADCAST][1][i].name, "I0:RectangleDput:SHMEM:MU") == 0) - { - opt_proto = i; - mustquery = 1; - comm_ptr->mpid.cutoff_size[PAMI_XFER_BROADCAST][0] = 131072; - break; - } - } - } - if(strcasecmp(comm_ptr->mpid.opt_protocol_md[PAMI_XFER_BROADCAST][0].name, "I0:SequenceBased_Binomial:SHMEM:MU") == 0) - { - /* This protocol was only good for up to 256, and it was an irregular, so let's set - * 2-nomial for larger message sizes. Cutoff should have already been set to 256 too */ - /* this protocol is sometimes query, sometimes always works so check both lists but prefer always works */ - for(i = 0; i < comm_ptr->mpid.coll_count[PAMI_XFER_BROADCAST][0]; i++) - { - if(strcasecmp(comm_ptr->mpid.coll_metadata[PAMI_XFER_BROADCAST][0][i].name, "I0:2-nomial:SHMEM:MU") == 0) - opt_proto = i; - } - if(opt_proto == -1) for(i = 0; i < comm_ptr->mpid.coll_count[PAMI_XFER_BROADCAST][1]; i++) - { - if(strcasecmp(comm_ptr->mpid.coll_metadata[PAMI_XFER_BROADCAST][1][i].name, "I0:2-nomial:SHMEM:MU") == 0) - opt_proto = i; - mustquery = 1; - } - } - - if(opt_proto != -1) - { - if(MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_0 && comm_ptr->rank == 0) - { - fprintf(stderr,"Selecting %s as optimal broadcast 1 (above %d)\n", - comm_ptr->mpid.coll_metadata[PAMI_XFER_BROADCAST][mustquery][opt_proto].name, - comm_ptr->mpid.cutoff_size[PAMI_XFER_BROADCAST][0]); - } - TRACE_ERR("Memcpy protocol type %d, number %d (%s) to optimize protocol 1 (above %d)\n", - PAMI_XFER_BROADCAST, opt_proto, - comm_ptr->mpid.coll_metadata[PAMI_XFER_BROADCAST][mustquery][opt_proto].name, - comm_ptr->mpid.cutoff_size[PAMI_XFER_BROADCAST][0]); - - comm_ptr->mpid.opt_protocol[PAMI_XFER_BROADCAST][1] = - comm_ptr->mpid.coll_algorithm[PAMI_XFER_BROADCAST][mustquery][opt_proto]; - memcpy(&comm_ptr->mpid.opt_protocol_md[PAMI_XFER_BROADCAST][1], - &comm_ptr->mpid.coll_metadata[PAMI_XFER_BROADCAST][mustquery][opt_proto], - sizeof(pami_metadata_t)); - comm_ptr->mpid.must_query[PAMI_XFER_BROADCAST][1] = mustquery?MPID_COLL_ALWAYS_QUERY:MPID_COLL_NOQUERY; - /* This should already be set... */ - /* comm_ptr->mpid.user_selected_type[PAMI_XFER_BROADCAST] = MPID_COLL_OPTIMIZED; */ - } - else - { - TRACE_ERR("Secondary bcast protocols unavilable; using primary for all sizes\n"); - - TRACE_ERR("Duplicating protocol type %d, number %d (%s) to optimize protocol 1 (above %d)\n", - PAMI_XFER_BROADCAST, 0, - comm_ptr->mpid.opt_protocol_md[PAMI_XFER_BROADCAST][0].name, - comm_ptr->mpid.cutoff_size[PAMI_XFER_BROADCAST][0]); - - comm_ptr->mpid.opt_protocol[PAMI_XFER_BROADCAST][1] = - comm_ptr->mpid.opt_protocol[PAMI_XFER_BROADCAST][0]; - memcpy(&comm_ptr->mpid.opt_protocol_md[PAMI_XFER_BROADCAST][1], - &comm_ptr->mpid.opt_protocol_md[PAMI_XFER_BROADCAST][0], - sizeof(pami_metadata_t)); - comm_ptr->mpid.must_query[PAMI_XFER_BROADCAST][1] = comm_ptr->mpid.must_query[PAMI_XFER_BROADCAST][0]; - } - } - TRACE_ERR("Done with bcast protocol selection\n"); - } - - opt_proto = -1; - mustquery = 0; - /* The most fun... allreduce */ - /* 512-way data: */ - /* For starters, Amith's protocol works on doubles on sum/min/max. Because - * those are targetted types/ops, we will pre-cache it. - * That protocol works on ints, up to 8k/ppn max message size. We'll precache - * it too - */ - - if(comm_ptr->mpid.user_selected_type[PAMI_XFER_ALLREDUCE] == MPID_COLL_NOSELECTION) - { - /* the user hasn't selected a protocol, so we can NULL the protocol/metadatas */ - comm_ptr->mpid.query_cached_allreduce = MPID_COLL_USE_MPICH; - - comm_ptr->mpid.cutoff_size[PAMI_XFER_ALLREDUCE][0] = 0; - comm_ptr->mpid.must_query[PAMI_XFER_ALLREDUCE][0] = MPID_COLL_USE_MPICH; - comm_ptr->mpid.must_query[PAMI_XFER_ALLREDUCE][1] = MPID_COLL_USE_MPICH; - /* For BGQ */ - /* 1ppn: I0:MultiCombineDput:-:MU if it is available, but it has a check_fn - * since it is MU-based*/ - /* Next best is I1:ShortAllreduce:P2P:P2P for short messages, then MPICH is best*/ - /* I0:MultiCombineDput:-:MU could be used in the i/dsmm cached protocols, so we'll do that */ - /* First, look in the 'must query' list and see i I0:MultiCombine:Dput:-:MU is there */ - - char *pname="...none..."; - int user_range_hi = -1; - int fca_enabled = 0; - if((fca_enabled = MPIDI_Check_FCA_envvar("ALLREDUCE", &user_range_hi)) == 1) - pname = "I1:Allreduce:FCA:FCA"; - else if(use_threaded_collectives) - pname = "I0:MultiCombineDput:-:MU"; - /*SSS: Any "MU" protocol will not be available on non-BG systems. I just need to check for FCA in the - first if only. No need to do another check since the second if will never succeed for PE systems*/ - for(i = 0; i < comm_ptr->mpid.coll_count[PAMI_XFER_ALLREDUCE][1]; i++) - { - if(strcasecmp(comm_ptr->mpid.coll_metadata[PAMI_XFER_ALLREDUCE][1][i].name, pname) == 0) - { - /* So, this should be fine for the i/dsmm protocols. everything else needs to call the check function */ - /* This also works for all message sizes, so no need to deal with it specially for query */ - comm_ptr->mpid.cached_allreduce = - comm_ptr->mpid.coll_algorithm[PAMI_XFER_ALLREDUCE][1][i]; - memcpy(&comm_ptr->mpid.cached_allreduce_md, - &comm_ptr->mpid.coll_metadata[PAMI_XFER_ALLREDUCE][1][i], - sizeof(pami_metadata_t)); - comm_ptr->mpid.query_cached_allreduce = MPID_COLL_QUERY; - - comm_ptr->mpid.user_selected_type[PAMI_XFER_ALLREDUCE] = MPID_COLL_OPTIMIZED; - if(fca_enabled && user_range_hi != -1) - comm_ptr->mpid.cutoff_size[PAMI_XFER_ALLREDUCE][0] = user_range_hi; - opt_proto = i; - - } - if(use_threaded_collectives) - if(strcasecmp(comm_ptr->mpid.coll_metadata[PAMI_XFER_ALLREDUCE][1][i].name, "I0:MultiCombineDput:SHMEM:MU") == 0) - { - /* This works well for doubles sum/min/max but has trouble with int > 8k/ppn */ - comm_ptr->mpid.cached_allreduce = - comm_ptr->mpid.coll_algorithm[PAMI_XFER_ALLREDUCE][1][i]; - memcpy(&comm_ptr->mpid.cached_allreduce_md, - &comm_ptr->mpid.coll_metadata[PAMI_XFER_ALLREDUCE][1][i], - sizeof(pami_metadata_t)); - comm_ptr->mpid.query_cached_allreduce = MPID_COLL_CHECK_FN_REQUIRED; - - comm_ptr->mpid.user_selected_type[PAMI_XFER_ALLREDUCE] = MPID_COLL_OPTIMIZED; - opt_proto = i; - } - } - /* At this point, if opt_proto != -1, we have must-query protocols in the i/dsmm caches */ - /* We should pick a backup, non-must query */ - /* I0:ShortAllreduce:P2P:P2P < 128, then mpich*/ - - /*SSS: ShortAllreduce is available on both BG and PE. I have to pick just one to check for in this case. - However, I need to add FCA for both opt_protocol[0]and[1] to cover all data sizes*/ - if(fca_enabled == 1) - pname = "I1:Allreduce:FCA:FCA"; - else - pname = "I1:ShortAllreduce:P2P:P2P"; - - for(i = 0; i < comm_ptr->mpid.coll_count[PAMI_XFER_ALLREDUCE][1]; i++) - { - if(strcasecmp(comm_ptr->mpid.coll_metadata[PAMI_XFER_ALLREDUCE][1][i].name, pname) == 0) - { - comm_ptr->mpid.opt_protocol[PAMI_XFER_ALLREDUCE][0] = - comm_ptr->mpid.coll_algorithm[PAMI_XFER_ALLREDUCE][1][i]; - memcpy(&comm_ptr->mpid.opt_protocol_md[PAMI_XFER_ALLREDUCE][0], - &comm_ptr->mpid.coll_metadata[PAMI_XFER_ALLREDUCE][1][i], - sizeof(pami_metadata_t)); - if(fca_enabled == 1) - { - comm_ptr->mpid.must_query[PAMI_XFER_ALLREDUCE][0] = MPID_COLL_CHECK_FN_REQUIRED; - if(user_range_hi != -1) - comm_ptr->mpid.cutoff_size[PAMI_XFER_ALLREDUCE][0] = user_range_hi; - /*SSS: Otherwise another protocol may get selected in mpido_allreduce if we don't set this flag here*/ - comm_ptr->mpid.must_query[PAMI_XFER_ALLREDUCE][1] = MPID_COLL_CHECK_FN_REQUIRED; - } - else - { - /* Short is good for up to 512 bytes... but it's a query protocol */ - comm_ptr->mpid.must_query[PAMI_XFER_ALLREDUCE][0] = MPID_COLL_QUERY; - /* MPICH above that ... when short query fails */ - comm_ptr->mpid.must_query[PAMI_XFER_ALLREDUCE][1] = MPID_COLL_USE_MPICH; - } - comm_ptr->mpid.user_selected_type[PAMI_XFER_ALLREDUCE] = MPID_COLL_OPTIMIZED; - - opt_proto = i; - } - } - if(opt_proto == -1) - { - if(MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_0 && comm_ptr->rank == 0) - fprintf(stderr,"Optimized allreduce falls back to MPICH\n"); - comm_ptr->mpid.must_query[PAMI_XFER_ALLREDUCE][0] = MPID_COLL_USE_MPICH; - comm_ptr->mpid.must_query[PAMI_XFER_ALLREDUCE][1] = MPID_COLL_USE_MPICH; - } - TRACE_ERR("Done setting optimized allreduce protocols\n"); - } - - - if(MPIDI_Process.optimized.select_colls != 2) - { - for(i = 0; i < PAMI_XFER_COUNT; i++) - { - if(i == PAMI_XFER_AMBROADCAST || i == PAMI_XFER_AMSCATTER || - i == PAMI_XFER_AMGATHER || i == PAMI_XFER_AMREDUCE) - continue; - if(comm_ptr->mpid.user_selected_type[i] != MPID_COLL_OPTIMIZED) - { - if(MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_0 && comm_ptr->rank == 0) - fprintf(stderr, "Collective wasn't selected for type %d,using MPICH (comm %p)\n", i, comm_ptr); - comm_ptr->mpid.user_selected_type[i] = MPID_COLL_USE_MPICH; - } - } - } - - - if(MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_0 && comm_ptr->rank == 0) - { - if(comm_ptr->mpid.user_selected_type[PAMI_XFER_BARRIER] == MPID_COLL_OPTIMIZED) - fprintf(stderr,"Selecting %s for opt barrier comm %p\n", comm_ptr->mpid.opt_protocol_md[PAMI_XFER_BARRIER][0].name, comm_ptr); - if(comm_ptr->mpid.user_selected_type[PAMI_XFER_ALLGATHERV_INT] == MPID_COLL_OPTIMIZED) - fprintf(stderr,"Selecting %s for opt allgatherv comm %p\n", comm_ptr->mpid.opt_protocol_md[PAMI_XFER_ALLGATHERV_INT][0].name, comm_ptr); - if(comm_ptr->mpid.must_query[PAMI_XFER_ALLGATHERV_INT][0] == MPID_COLL_USE_MPICH) - fprintf(stderr,"Selecting MPICH for allgatherv below %d size comm %p\n", comm_ptr->mpid.cutoff_size[PAMI_XFER_ALLGATHERV_INT][0], comm_ptr); - if(comm_ptr->mpid.user_selected_type[PAMI_XFER_GATHER] == MPID_COLL_USE_MPICH) - fprintf(stderr,"Selecting MPICH for gather comm %p\n", comm_ptr); - if(comm_ptr->mpid.user_selected_type[PAMI_XFER_SCATTER] == MPID_COLL_USE_MPICH) - fprintf(stderr,"Selecting MPICH for scatter comm %p\n", comm_ptr); - else if(comm_ptr->mpid.user_selected_type[PAMI_XFER_SCATTER] == MPID_COLL_OPTIMIZED) - fprintf(stderr,"Selecting %s for scatter comm %p\n", - comm_ptr->mpid.opt_protocol_md[PAMI_XFER_SCATTER][0].name, - comm_ptr); - if(comm_ptr->mpid.user_selected_type[PAMI_XFER_SCATTERV_INT] == MPID_COLL_USE_MPICH) - fprintf(stderr,"Selecting MPICH for scatterv comm %p\n", comm_ptr); - else if(comm_ptr->mpid.user_selected_type[PAMI_XFER_SCATTERV_INT] == MPID_COLL_OPTIMIZED) - fprintf(stderr,"Selecting %s for scatterv comm %p\n", - comm_ptr->mpid.opt_protocol_md[PAMI_XFER_SCATTERV_INT][0].name, - comm_ptr); - if(comm_ptr->mpid.user_selected_type[PAMI_XFER_BROADCAST] == MPID_COLL_OPTIMIZED) - fprintf(stderr,"Selecting %s for opt bcast up to size %d comm %p\n", comm_ptr->mpid.opt_protocol_md[PAMI_XFER_BROADCAST][0].name, - comm_ptr->mpid.cutoff_size[PAMI_XFER_BROADCAST][0], comm_ptr); - if((comm_ptr->mpid.must_query[PAMI_XFER_BROADCAST][1] == MPID_COLL_NOQUERY) || - (comm_ptr->mpid.must_query[PAMI_XFER_BROADCAST][1] == MPID_COLL_ALWAYS_QUERY)) - fprintf(stderr,"Selecting %s (mustquery=%d) for opt bcast above size %d comm %p\n", - comm_ptr->mpid.opt_protocol_md[PAMI_XFER_BROADCAST][1].name, - comm_ptr->mpid.must_query[PAMI_XFER_BROADCAST][1], - comm_ptr->mpid.cutoff_size[PAMI_XFER_BROADCAST][0], comm_ptr); - if(comm_ptr->mpid.user_selected_type[PAMI_XFER_ALLTOALLV_INT] == MPID_COLL_OPTIMIZED) - fprintf(stderr,"Selecting %s for opt alltoallv comm %p\n", comm_ptr->mpid.opt_protocol_md[PAMI_XFER_ALLTOALLV_INT][0].name, comm_ptr); - if(comm_ptr->mpid.user_selected_type[PAMI_XFER_ALLTOALL] == MPID_COLL_OPTIMIZED) - fprintf(stderr,"Selecting %s for opt alltoall comm %p\n", comm_ptr->mpid.opt_protocol_md[PAMI_XFER_ALLTOALL][0].name, comm_ptr); - if(comm_ptr->mpid.must_query[PAMI_XFER_ALLREDUCE][0] == MPID_COLL_USE_MPICH) - fprintf(stderr,"Selecting MPICH for allreduce below %d size comm %p\n", comm_ptr->mpid.cutoff_size[PAMI_XFER_ALLREDUCE][0], comm_ptr); - else - { - if(comm_ptr->mpid.query_cached_allreduce != MPID_COLL_USE_MPICH) - { - fprintf(stderr,"Selecting %s for double sum/min/max ops allreduce, query: %d comm %p\n", - comm_ptr->mpid.cached_allreduce_md.name, comm_ptr->mpid.query_cached_allreduce, comm_ptr); - } - fprintf(stderr,"Selecting %s for other operations allreduce up to %d comm %p\n", - comm_ptr->mpid.opt_protocol_md[PAMI_XFER_ALLREDUCE][0].name, - comm_ptr->mpid.cutoff_size[PAMI_XFER_ALLREDUCE][0], comm_ptr); - } - if(comm_ptr->mpid.must_query[PAMI_XFER_ALLREDUCE][1] == MPID_COLL_USE_MPICH) - fprintf(stderr,"Selecting MPICH for allreduce above %d size comm %p\n", comm_ptr->mpid.cutoff_size[PAMI_XFER_ALLREDUCE][0], comm_ptr); - else - { - if(comm_ptr->mpid.query_cached_allreduce != MPID_COLL_USE_MPICH) - { - fprintf(stderr,"Selecting %s for double sum/min/max ops allreduce, above %d query: %d comm %p\n", - comm_ptr->mpid.cached_allreduce_md.name, - comm_ptr->mpid.cutoff_size[PAMI_XFER_ALLREDUCE][0], - comm_ptr->mpid.query_cached_allreduce, comm_ptr); - } - else - { - fprintf(stderr,"Selecting MPICH for double sum/min/max ops allreduce, above %d size comm %p\n", - comm_ptr->mpid.cutoff_size[PAMI_XFER_ALLREDUCE][0], comm_ptr); - } - fprintf(stderr,"Selecting %s for other operations allreduce over %d comm %p\n", - comm_ptr->mpid.opt_protocol_md[PAMI_XFER_ALLREDUCE][1].name, - comm_ptr->mpid.cutoff_size[PAMI_XFER_ALLREDUCE][0], comm_ptr); - } - } - - TRACE_ERR("Leaving MPIDI_Comm_coll_select\n"); - -} - diff --git a/src/mpid/pamid/src/comm/mpid_selectcolls.c b/src/mpid/pamid/src/comm/mpid_selectcolls.c deleted file mode 100644 index 8e0fa36..0000000 --- a/src/mpid/pamid/src/comm/mpid_selectcolls.c +++ /dev/null @@ -1,746 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/comm/mpid_collselect.c - * \brief Code for setting up collectives per geometry/communicator - */ - -/*#define TRACE_ON */ - -#include <mpidimpl.h> - -pami_metadata_t ext_metadata; -advisor_algorithm_t ext_algorithms[1]; -external_algorithm_t ext_algorithm; - -#define MPIDI_UPDATE_COLLSEL_EXT_ALGO(cb,nm,xfer_type) { \ - ext_algorithm.callback = cb; \ - ext_algorithm.cookie = comm; \ - ext_metadata.name = nm; \ - ext_algorithms[0].algorithm.external = ext_algorithm; \ - ext_algorithms[0].metadata = &ext_metadata; \ - ext_algorithms[0].algorithm_type = COLLSEL_EXTERNAL_ALGO; \ - pamix_collsel_register_algorithms(MPIDI_Collsel_advisor_table, \ - comm->mpid.geometry, \ - xfer_type, \ - &ext_algorithms[0], \ - 1); \ -} - - -pami_result_t MPIDI_Register_algorithms_ext(void *cookie, - pami_xfer_type_t collective, - advisor_algorithm_t **algorithms, - size_t *num_algorithms) -{ - external_algorithm_fn callback; - char *algoname; - - switch(collective) - { - case PAMI_XFER_BROADCAST: callback = MPIDO_CSWrapper_bcast; algoname = "EXT:Bcast:P2P:P2P"; break; - case PAMI_XFER_ALLREDUCE: callback = MPIDO_CSWrapper_allreduce; algoname = "EXT:Allreduce:P2P:P2P"; break; - case PAMI_XFER_REDUCE: callback = MPIDO_CSWrapper_reduce; algoname = "EXT:Reduce:P2P:P2P"; break; - case PAMI_XFER_ALLGATHER: callback = MPIDO_CSWrapper_allgather; algoname = "EXT:Allgather:P2P:P2P"; break; - case PAMI_XFER_ALLGATHERV_INT: callback = MPIDO_CSWrapper_allgatherv; algoname = "EXT:Allgatherv:P2P:P2P"; break; - case PAMI_XFER_SCATTER: callback = MPIDO_CSWrapper_scatter; algoname = "EXT:Scatter:P2P:P2P"; break; - case PAMI_XFER_SCATTERV_INT: callback = MPIDO_CSWrapper_scatterv; algoname = "EXT:Scatterv:P2P:P2P"; break; - case PAMI_XFER_GATHER: callback = MPIDO_CSWrapper_gather; algoname = "EXT:Gather:P2P:P2P"; break; - case PAMI_XFER_GATHERV_INT: callback = MPIDO_CSWrapper_gatherv; algoname = "EXT:Gatherv:P2P:P2P"; break; - case PAMI_XFER_BARRIER: callback = MPIDO_CSWrapper_barrier; algoname = "EXT:Barrier:P2P:P2P"; break; - case PAMI_XFER_ALLTOALL: callback = MPIDO_CSWrapper_alltoall; algoname = "EXT:Alltoall:P2P:P2P"; break; - case PAMI_XFER_ALLTOALLV_INT: callback = MPIDO_CSWrapper_alltoallv; algoname = "EXT:Alltoallv:P2P:P2P"; break; - case PAMI_XFER_SCAN: callback = MPIDO_CSWrapper_scan; algoname = "EXT:Scan:P2P:P2P"; break; - case PAMI_XFER_ALLGATHERV: - case PAMI_XFER_SCATTERV: - case PAMI_XFER_GATHERV: - case PAMI_XFER_ALLTOALLV: - case PAMI_XFER_REDUCE_SCATTER: - *num_algorithms = 0; - return PAMI_SUCCESS; - default: return -1; - } - *num_algorithms = 1; - ext_algorithm.callback = callback; - ext_algorithm.cookie = cookie; - ext_metadata.name = algoname; - ext_algorithms[0].algorithm.external = ext_algorithm; - ext_algorithms[0].metadata = &ext_metadata; - ext_algorithms[0].algorithm_type = COLLSEL_EXTERNAL_ALGO; - *algorithms = &ext_algorithms[0]; - return PAMI_SUCCESS; -} - -static char* MPIDI_Coll_type_name(int i) -{ - switch(i) - { - case 0: return("Broadcast"); - case 1: return("Allreduce"); - case 2: return("Reduce"); - case 3: return("Allgather"); - case 4: return("Allgatherv"); - case 5: return("Allgatherv_int"); - case 6: return("Scatter"); - case 7: return("Scatterv"); - case 8: return("Scatterv_int"); - case 9: return("Gather"); - case 10: return("Gatherv"); - case 11: return("Gatherv_int"); - case 12: return("Barrier"); - case 13: return("Alltoall"); - case 14: return("Alltoallv"); - case 15: return("Alltoallv_int"); - case 16: return("Scan"); - case 17: return("Reduce_scatter"); - default: return("AM Collective"); - } -} -static void MPIDI_Update_coll(pami_algorithm_t coll, - int type, - int index, - MPIR_Comm *comm); - -static void MPIDI_Update_coll(pami_algorithm_t coll, - int type, /* must query vs always works */ - int index, - MPIR_Comm *comm) -{ - - comm->mpid.user_selected_type[coll] = type; - TRACE_ERR("Update_coll for protocol %s, type: %d index: %d\n", - comm->mpid.coll_metadata[coll][type][index].name, type, index); - - /* Are we in the 'must query' list? If so determine how "bad" it is */ - if(type == MPID_COLL_QUERY) - { - /* First, is a check always required? */ - if(comm->mpid.coll_metadata[coll][type][index].check_correct.values.checkrequired) - { - TRACE_ERR("Protocol %s check_fn required always\n", comm->mpid.coll_metadata[coll][type][index].name); - /* We must have a check_fn */ - MPID_assert_always(comm->mpid.coll_metadata[coll][type][index].check_fn !=NULL); - comm->mpid.user_selected_type[coll] = MPID_COLL_CHECK_FN_REQUIRED; - } - else if(comm->mpid.coll_metadata[coll][type][index].check_fn != NULL) - { - /* For now, if there's a check_fn we will always call it and not cache. - We *could* be smarter about this eventually. */ - TRACE_ERR("Protocol %s setting to always query/call check_fn\n", comm->mpid.coll_metadata[coll][type][index].name); - comm->mpid.user_selected_type[coll] = MPID_COLL_CHECK_FN_REQUIRED; - } - else /* No check fn but we still need to check metadata bits (query protocol) */ - { - TRACE_ERR("Protocol %s setting to always query/no check_fn\n", comm->mpid.coll_metadata[coll][type][index].name); - comm->mpid.user_selected_type[coll] = MPID_COLL_ALWAYS_QUERY; - } - - } - - comm->mpid.user_selected[coll] = - comm->mpid.coll_algorithm[coll][type][index]; - - memcpy(&comm->mpid.user_metadata[coll], - &comm->mpid.coll_metadata[coll][type][index], - sizeof(pami_metadata_t)); -} - -static void MPIDI_Check_preallreduce(char *env, MPIR_Comm *comm, char *name, int constant) -{ - /* If a given protocol only requires a check for nonlocal conditions and preallreduce - * is turned off, we can consider it a always works protocol instead - */ - char *envopts = getenv(env); - if(envopts != NULL) - { - if(strncasecmp(env, "N", 1) == 0) - { - if(MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_0 && comm->rank == 0) - fprintf(stderr,"Bypassing breallreduce for %s\n", name); - comm->mpid.preallreduces[constant] = 0; - } - } -} -static int MPIDI_Check_protocols(char *names[], MPIR_Comm *comm, char *name, int constant) -{ - int i = 0; - char *envopts; - for(;; ++i) - { - if(names[i] == NULL) - return 0; - envopts = getenv(names[i]); - if(envopts != NULL) - break; - } - /* Now, change it if we have a match */ - if(envopts != NULL) - { - if(MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_0 && comm->rank == 0) - fprintf(stderr,"Checking %s against known %s protocols\n", envopts, name); - /* Check if MPICH was specifically requested */ - if(strcasecmp(envopts, "MPICH") == 0) - { - TRACE_ERR("Selecting MPICH for %s\n", name); - if(MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_0 && comm->rank == 0) - fprintf(stderr,"Selecting MPICH for %d (%s)\n", constant, name); - comm->mpid.user_selected_type[constant] = MPID_COLL_USE_MPICH; - comm->mpid.user_selected[constant] = 0; - return 0; - } - - for(i=0; i < comm->mpid.coll_count[constant][0];i++) - { - if(strncasecmp(envopts, comm->mpid.coll_metadata[constant][0][i].name,strlen(envopts)) == 0) - { - MPIDI_Update_coll(constant, MPID_COLL_NOQUERY, i, comm); - if(MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_0 && comm->rank == 0) - fprintf(stderr,"setting %s as default %s for comm %p\n", comm->mpid.coll_metadata[constant][0][i].name, name, comm); - return 0; - } - } - for(i=0; i < comm->mpid.coll_count[constant][1];i++) - { - if(strncasecmp(envopts, comm->mpid.coll_metadata[constant][1][i].name,strlen(envopts)) == 0) - { - TRACE_ERR("Calling updatecoll...\n"); - MPIDI_Update_coll(constant, MPID_COLL_QUERY, i, comm); - TRACE_ERR("Done calling update coll\n"); - if(MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_0 && comm->rank == 0) - fprintf(stderr,"setting (query required protocol) %s as default %s for comm %p\n", comm->mpid.coll_metadata[constant][1][i].name, name, comm); - return 0; - } - } - /* An envvar was specified, so we should probably use MPICH of we can't find - * the specified protocol */ - TRACE_ERR("Specified protocol %s was unavailable; using MPICH for %s\n", envopts, name); - comm->mpid.user_selected_type[constant] = MPID_COLL_USE_MPICH; - comm->mpid.user_selected[constant] = 0; - return 0; - } - /* Looks like we didn't get anything, set NOSELECTION so automated selection can pick something */ - comm->mpid.user_selected_type[constant] = MPID_COLL_NOSELECTION; - comm->mpid.user_selected[constant] = 0; - return 0; -} - -void MPIDI_Comm_coll_envvars(MPIR_Comm *comm) -{ - char *envopts; - int i; - MPID_assert_always(comm!=NULL); - TRACE_ERR("MPIDI_Comm_coll_envvars enter\n"); - - /* Set up always-works defaults */ - for(i = 0; i < PAMI_XFER_COUNT; i++) - { - if(i == PAMI_XFER_AMBROADCAST || i == PAMI_XFER_AMSCATTER || - i == PAMI_XFER_AMGATHER || i == PAMI_XFER_AMREDUCE) - continue; - - /* Initialize to noselection instead of noquery for PE/FCA stuff. Is this the right answer? */ - comm->mpid.user_selected_type[i] = MPID_COLL_NOSELECTION; - if(MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_0 && comm->rank == 0) - fprintf(stderr,"Setting up collective %d on comm %p\n", i, comm); - if(((comm->mpid.coll_count[i][0] == 0) && (comm->mpid.coll_count[i][1] == 0)) || MPIDI_Process.optimized.collectives == MPID_COLL_CUDA) - { - comm->mpid.user_selected_type[i] = MPID_COLL_USE_MPICH; - comm->mpid.user_selected[i] = 0; - } - else if(comm->mpid.coll_count[i][0] != 0) - { - comm->mpid.user_selected[i] = comm->mpid.coll_algorithm[i][0][0]; - memcpy(&comm->mpid.user_metadata[i], &comm->mpid.coll_metadata[i][0][0], - sizeof(pami_metadata_t)); - } - else - { - MPIDI_Update_coll(i, MPID_COLL_QUERY, 0, comm); - /* even though it's a query protocol, say NOSELECTION - so the optcoll selection will override (maybe) */ - comm->mpid.user_selected_type[i] = MPID_COLL_NOSELECTION; - } - } - - - TRACE_ERR("Checking env vars\n"); - - MPIDI_Check_preallreduce("PAMID_COLLECTIVE_ALLGATHER_PREALLREDUCE", comm, "allgather", - MPID_ALLGATHER_PREALLREDUCE); - - MPIDI_Check_preallreduce("PAMID_COLLECTIVE_ALLGATHERV_PREALLREDUCE", comm, "allgatherv", - MPID_ALLGATHERV_PREALLREDUCE); - - MPIDI_Check_preallreduce("PAMID_COLLECTIVE_ALLREDUCE_PREALLREDUCE", comm, "allreduce", - MPID_ALLREDUCE_PREALLREDUCE); - - MPIDI_Check_preallreduce("PAMID_COLLECTIVE_BCAST_PREALLREDUCE", comm, "broadcast", - MPID_BCAST_PREALLREDUCE); - - MPIDI_Check_preallreduce("PAMID_COLLECTIVE_SCATTERV_PREALLREDUCE", comm, "scatterv", - MPID_SCATTERV_PREALLREDUCE); - - { - TRACE_ERR("Checking bcast\n"); - char* names[] = {"PAMID_COLLECTIVE_BCAST", "MP_S_MPI_BCAST", NULL}; - MPIDI_Check_protocols(names, comm, "broadcast", PAMI_XFER_BROADCAST); - } - { - TRACE_ERR("Checking allreduce\n"); - char* names[] = {"PAMID_COLLECTIVE_ALLREDUCE", "MP_S_MPI_ALLREDUCE", NULL}; - MPIDI_Check_protocols(names, comm, "allreduce", PAMI_XFER_ALLREDUCE); - } - { - TRACE_ERR("Checking barrier\n"); - char* names[] = {"PAMID_COLLECTIVE_BARRIER", "MP_S_MPI_BARRIER", NULL}; - MPIDI_Check_protocols(names, comm, "barrier", PAMI_XFER_BARRIER); - } - { - TRACE_ERR("Checking alltaoll\n"); - char* names[] = {"PAMID_COLLECTIVE_ALLTOALL", "MP_S_MPI_ALLTOALL", NULL}; - MPIDI_Check_protocols(names, comm, "alltoall", PAMI_XFER_ALLTOALL); - } - comm->mpid.optreduce = 0; - envopts = getenv("PAMID_COLLECTIVE_REDUCE"); - if(envopts != NULL) - { - if(strcasecmp(envopts, "GLUE_ALLREDUCE") == 0) - { - if(MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_0 && comm->rank == 0) - fprintf(stderr,"Selecting glue allreduce for reduce\n"); - comm->mpid.optreduce = 1; - } - } - /* In addition to glue protocols, check for other PAMI protocols and check for PE now */ - { - TRACE_ERR("Checking reduce\n"); - char* names[] = {"PAMID_COLLECTIVE_REDUCE", "MP_S_MPI_REDUCE", NULL}; - MPIDI_Check_protocols(names, comm, "reduce", PAMI_XFER_REDUCE); - } - { - TRACE_ERR("Checking alltoallv\n"); - char* names[] = {"PAMID_COLLECTIVE_ALLTOALLV", "MP_S_MPI_ALLTOALLV", NULL}; - MPIDI_Check_protocols(names, comm, "alltoallv", PAMI_XFER_ALLTOALLV_INT); - } - { - TRACE_ERR("Checking gatherv\n"); - char* names[] = {"PAMID_COLLECTIVE_GATHERV", "MP_S_MPI_GATHERV", NULL}; - MPIDI_Check_protocols(names, comm, "gatherv", PAMI_XFER_GATHERV_INT); - } - { - TRACE_ERR("Checking scan\n"); - char* names[] = {"PAMID_COLLECTIVE_SCAN", "MP_S_MPI_SCAN", NULL}; - MPIDI_Check_protocols(names, comm, "scan", PAMI_XFER_SCAN); - } - - comm->mpid.scattervs[0] = comm->mpid.scattervs[1] = 0; - - TRACE_ERR("Checking scatterv\n"); - envopts = getenv("PAMID_COLLECTIVE_SCATTERV"); - if(envopts != NULL) - { - if(strcasecmp(envopts, "GLUE_BCAST") == 0) - { - if(MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_0 && comm->rank == 0) - fprintf(stderr,"Selecting glue bcast for scatterv\n"); - comm->mpid.scattervs[0] = 1; - } - else if(strcasecmp(envopts, "GLUE_ALLTOALLV") == 0) - { - if(MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_0 && comm->rank == 0) - fprintf(stderr,"Selecting glue alltoallv for scatterv\n"); - comm->mpid.scattervs[1] = 1; - } - } - { /* In addition to glue protocols, check for other PAMI protocols and check for PE now */ - char* names[] = {"PAMID_COLLECTIVE_SCATTERV", "MP_S_MPI_SCATTERV", NULL}; - MPIDI_Check_protocols(names, comm, "scatterv", PAMI_XFER_SCATTERV_INT); - } - - TRACE_ERR("Checking scatter\n"); - comm->mpid.optscatter = 0; - envopts = getenv("PAMID_COLLECTIVE_SCATTER"); - if(envopts != NULL) - { - if(strcasecmp(envopts, "GLUE_BCAST") == 0) - { - if(MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_0 && comm->rank == 0) - fprintf(stderr,"Selecting glue_bcast for scatter\n"); - comm->mpid.optscatter = 1; - } - } - { /* In addition to glue protocols, check for other PAMI protocols and check for PE now */ - char* names[] = {"PAMID_COLLECTIVE_SCATTER", "MP_S_MPI_SCATTER", NULL}; - MPIDI_Check_protocols(names, comm, "scatter", PAMI_XFER_SCATTER); - } - - TRACE_ERR("Checking allgather\n"); - comm->mpid.allgathers[0] = comm->mpid.allgathers[1] = comm->mpid.allgathers[2] = 0; - envopts = getenv("PAMID_COLLECTIVE_ALLGATHER"); - if(envopts != NULL) - { - if(strcasecmp(envopts, "GLUE_ALLREDUCE") == 0) - { - if(MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_0 && comm->rank == 0) - fprintf(stderr,"Selecting glue_allreduce for allgather\n"); - comm->mpid.allgathers[0] = 1; - } - - else if(strcasecmp(envopts, "GLUE_BCAST") == 0) - { - if(MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_0 && comm->rank == 0) - fprintf(stderr,"Selecting glue_bcast for allgather\n"); - comm->mpid.allgathers[1] = 1; - } - - else if(strcasecmp(envopts, "GLUE_ALLTOALL") == 0) - { - if(MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_0 && comm->rank == 0) - fprintf(stderr,"Selecting glue_alltoall for allgather\n"); - comm->mpid.allgathers[2] = 1; - } - } - { /* In addition to glue protocols, check for other PAMI protocols and check for PE now */ - char* names[] = {"PAMID_COLLECTIVE_ALLGATHER", "MP_S_MPI_ALLGATHER", NULL}; - MPIDI_Check_protocols(names, comm, "allgather", PAMI_XFER_ALLGATHER); - } - - TRACE_ERR("Checking allgatherv\n"); - comm->mpid.allgathervs[0] = comm->mpid.allgathervs[1] = comm->mpid.allgathervs[2] = 0; - envopts = getenv("PAMID_COLLECTIVE_ALLGATHERV"); - if(envopts != NULL) - { - if(strcasecmp(envopts, "GLUE_ALLREDUCE") == 0) - { - if(MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_0 && comm->rank == 0) - fprintf(stderr,"Selecting glue_allreduce for allgatherv\n"); - comm->mpid.allgathervs[0] = 1; - } - - else if(strcasecmp(envopts, "GLUE_BCAST") == 0) - { - if(MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_0 && comm->rank == 0) - fprintf(stderr,"Selecting glue_bcast for allgatherv\n"); - comm->mpid.allgathervs[1] = 1; - } - - else if(strcasecmp(envopts, "GLUE_ALLTOALL") == 0) - { - if(MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_0 && comm->rank == 0) - fprintf(stderr,"Selecting glue_alltoall for allgatherv\n"); - comm->mpid.allgathervs[2] = 1; - } - } - { /* In addition to glue protocols, check for other PAMI protocols and check for PE now */ - char* names[] = {"PAMID_COLLECTIVE_ALLGATHERV", "MP_S_MPI_ALLGATHERV", NULL}; - MPIDI_Check_protocols(names, comm, "allgatherv", PAMI_XFER_ALLGATHERV_INT); - } - - TRACE_ERR("CHecking gather\n"); - comm->mpid.optgather = 0; - envopts = getenv("PAMID_COLLECTIVE_GATHER"); - if(envopts != NULL) - { - if(strcasecmp(envopts, "GLUE_REDUCE") == 0) - { - if(MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_0 && comm->rank == 0) - fprintf(stderr,"Selecting glue reduce for gather\n"); - comm->mpid.optgather = 1; - } - } - { /* In addition to glue protocols, check for other PAMI protocols and check for PE now */ - char* names[] = {"PAMID_COLLECTIVE_GATHER", "MP_S_MPI_GATHER", NULL}; - MPIDI_Check_protocols(names, comm, "gather", PAMI_XFER_GATHER); - } - - /* If automatic collective selection is enabled and user didn't specifically overwrite - it, then use auto coll sel.. Otherwise, go through the manual coll sel code path. */ - comm->mpid.collsel_fast_query = NULL; /* Init to NULL.. Should only have a value if we create query */ - if(MPIDI_Process.optimized.auto_select_colls != MPID_AUTO_SELECT_COLLS_NONE && MPIDI_Process.optimized.auto_select_colls != MPID_AUTO_SELECT_COLLS_TUNE && comm->local_size > 1) - { - /* Create a fast query object, cache it on the comm/geometry and use it in each collective */ - pami_extension_collsel_query_create pamix_collsel_query_create = - (pami_extension_collsel_query_create) PAMI_Extension_symbol(MPIDI_Collsel_extension, - "Collsel_query_create"); - if(pamix_collsel_query_create != NULL) - { - pamix_collsel_query_create(MPIDI_Collsel_advisor_table, comm->mpid.geometry, &(comm->mpid.collsel_fast_query)); - } - - MPIDI_Pamix_collsel_advise = /* Get the function pointer and cache it */ - (pami_extension_collsel_advise) PAMI_Extension_symbol(MPIDI_Collsel_extension, - "Collsel_advise"); - - pami_extension_collsel_register_algorithms pamix_collsel_register_algorithms = - (pami_extension_collsel_register_algorithms) PAMI_Extension_symbol(MPIDI_Collsel_extension, - "Collsel_register_algorithms"); - if(pamix_collsel_register_algorithms != NULL) - { - - /* ************ Barrier ************ */ - if((MPIDI_Process.optimized.auto_select_colls & MPID_AUTO_SELECT_COLLS_BARRIER) && - comm->mpid.user_selected_type[PAMI_XFER_BARRIER] == MPID_COLL_NOSELECTION) - { - comm->coll_fns->Barrier = MPIDO_Barrier_simple; - MPIDI_UPDATE_COLLSEL_EXT_ALGO(MPIDO_CSWrapper_barrier,"EXT:Barrier:P2P:P2P",PAMI_XFER_BARRIER); - } - /* ************ Bcast ************ */ - if((MPIDI_Process.optimized.auto_select_colls & MPID_AUTO_SELECT_COLLS_BCAST) && - comm->mpid.user_selected_type[PAMI_XFER_BROADCAST] == MPID_COLL_NOSELECTION) - { - comm->coll_fns->Bcast = MPIDO_Bcast_simple; - MPIDI_UPDATE_COLLSEL_EXT_ALGO(MPIDO_CSWrapper_bcast,"EXT:Bcast:P2P:P2P",PAMI_XFER_BROADCAST); - } - /* ************ Allreduce ************ */ - if((MPIDI_Process.optimized.auto_select_colls & MPID_AUTO_SELECT_COLLS_ALLREDUCE) && - comm->mpid.user_selected_type[PAMI_XFER_ALLREDUCE] == MPID_COLL_NOSELECTION) - { - comm->coll_fns->Allreduce = MPIDO_Allreduce_simple; - MPIDI_UPDATE_COLLSEL_EXT_ALGO(MPIDO_CSWrapper_allreduce,"EXT:Allreduce:P2P:P2P",PAMI_XFER_ALLREDUCE); - } - /* ************ Allgather ************ */ - if((MPIDI_Process.optimized.auto_select_colls & MPID_AUTO_SELECT_COLLS_ALLGATHER) && - comm->mpid.user_selected_type[PAMI_XFER_ALLGATHER] == MPID_COLL_NOSELECTION) - { - comm->coll_fns->Allgather = MPIDO_Allgather_simple; - MPIDI_UPDATE_COLLSEL_EXT_ALGO(MPIDO_CSWrapper_allgather,"EXT:Allgather:P2P:P2P",PAMI_XFER_ALLGATHER); - } - /* ************ Allgatherv ************ */ - if((MPIDI_Process.optimized.auto_select_colls & MPID_AUTO_SELECT_COLLS_ALLGATHERV) && - comm->mpid.user_selected_type[PAMI_XFER_ALLGATHERV_INT] == MPID_COLL_NOSELECTION) - { - comm->coll_fns->Allgatherv = MPIDO_Allgatherv_simple; - MPIDI_UPDATE_COLLSEL_EXT_ALGO(MPIDO_CSWrapper_allgatherv,"EXT:Allgatherv:P2P:P2P",PAMI_XFER_ALLGATHERV_INT); - } - /* ************ Scatterv ************ */ - if((MPIDI_Process.optimized.auto_select_colls & MPID_AUTO_SELECT_COLLS_SCATTERV) && - comm->mpid.user_selected_type[PAMI_XFER_SCATTERV_INT] == MPID_COLL_NOSELECTION) - { - comm->coll_fns->Scatterv = MPIDO_Scatterv_simple; - MPIDI_UPDATE_COLLSEL_EXT_ALGO(MPIDO_CSWrapper_scatterv,"EXT:Scatterv:P2P:P2P",PAMI_XFER_SCATTERV_INT); - } - /* ************ Scatter ************ */ - if((MPIDI_Process.optimized.auto_select_colls & MPID_AUTO_SELECT_COLLS_SCATTER) && - comm->mpid.user_selected_type[PAMI_XFER_SCATTER] == MPID_COLL_NOSELECTION) - { - comm->coll_fns->Scatter = MPIDO_Scatter_simple; - MPIDI_UPDATE_COLLSEL_EXT_ALGO(MPIDO_CSWrapper_scatter,"EXT:Scatter:P2P:P2P",PAMI_XFER_SCATTER); - } - /* ************ Gather ************ */ - if((MPIDI_Process.optimized.auto_select_colls & MPID_AUTO_SELECT_COLLS_GATHER) && - comm->mpid.user_selected_type[PAMI_XFER_GATHER] == MPID_COLL_NOSELECTION) - { - comm->coll_fns->Gather = MPIDO_Gather_simple; - MPIDI_UPDATE_COLLSEL_EXT_ALGO(MPIDO_CSWrapper_gather,"EXT:Gather:P2P:P2P",PAMI_XFER_GATHER); - } - /* ************ Alltoallv ************ */ - if((MPIDI_Process.optimized.auto_select_colls & MPID_AUTO_SELECT_COLLS_ALLTOALLV) && - comm->mpid.user_selected_type[PAMI_XFER_ALLTOALLV_INT] == MPID_COLL_NOSELECTION) - { - comm->coll_fns->Alltoallv = MPIDO_Alltoallv_simple; - MPIDI_UPDATE_COLLSEL_EXT_ALGO(MPIDO_CSWrapper_alltoallv,"EXT:Alltoallv:P2P:P2P",PAMI_XFER_ALLTOALLV_INT); - } - /* ************ Alltoall ************ */ - if((MPIDI_Process.optimized.auto_select_colls & MPID_AUTO_SELECT_COLLS_ALLTOALL) && - comm->mpid.user_selected_type[PAMI_XFER_ALLTOALL] == MPID_COLL_NOSELECTION) - { - comm->coll_fns->Alltoall = MPIDO_Alltoall_simple; - MPIDI_UPDATE_COLLSEL_EXT_ALGO(MPIDO_CSWrapper_alltoall,"EXT:Alltoall:P2P:P2P",PAMI_XFER_ALLTOALL); - } - /* ************ Gatherv ************ */ - if((MPIDI_Process.optimized.auto_select_colls & MPID_AUTO_SELECT_COLLS_GATHERV) && - comm->mpid.user_selected_type[PAMI_XFER_GATHERV_INT] == MPID_COLL_NOSELECTION) - { - comm->coll_fns->Gatherv = MPIDO_Gatherv_simple; - MPIDI_UPDATE_COLLSEL_EXT_ALGO(MPIDO_CSWrapper_gatherv,"EXT:Gatherv:P2P:P2P",PAMI_XFER_GATHERV_INT); - } - /* ************ Reduce ************ */ - if((MPIDI_Process.optimized.auto_select_colls & MPID_AUTO_SELECT_COLLS_REDUCE) && - comm->mpid.user_selected_type[PAMI_XFER_REDUCE] == MPID_COLL_NOSELECTION) - { - comm->coll_fns->Reduce = MPIDO_Reduce_simple; - MPIDI_UPDATE_COLLSEL_EXT_ALGO(MPIDO_CSWrapper_reduce,"EXT:Reduce:P2P:P2P",PAMI_XFER_REDUCE); - } - /* ************ Scan ************ */ - if((MPIDI_Process.optimized.auto_select_colls & MPID_AUTO_SELECT_COLLS_SCAN) && - comm->mpid.user_selected_type[PAMI_XFER_SCAN] == MPID_COLL_NOSELECTION) - { - comm->coll_fns->Scan = MPIDO_Scan_simple; - MPIDI_UPDATE_COLLSEL_EXT_ALGO(MPIDO_CSWrapper_scan,"EXT:Scan:P2P:P2P",PAMI_XFER_SCAN); - } - } - } - TRACE_ERR("MPIDI_Comm_coll_envvars exit\n"); -} - - -/* Determine how many of each collective type this communicator supports */ -void MPIDI_Comm_coll_query(MPIR_Comm *comm) -{ - TRACE_ERR("MPIDI_Comm_coll_query enter\n"); - int rc = 0, i, j; - size_t num_algorithms[2]; - TRACE_ERR("Getting geometry from comm %p\n", comm); - pami_geometry_t *geom = comm->mpid.geometry;; - for(i = 0; i < PAMI_XFER_COUNT; i++) - { - if(i == PAMI_XFER_AMBROADCAST || i == PAMI_XFER_AMSCATTER || - i == PAMI_XFER_AMGATHER || i == PAMI_XFER_AMREDUCE) - continue; - rc = PAMI_Geometry_algorithms_num(geom, - i, - num_algorithms); - TRACE_ERR("Num algorithms of type %d: %zd %zd\n", i, num_algorithms[0], num_algorithms[1]); - if(rc != PAMI_SUCCESS) - { - fprintf(stderr,"PAMI_Geometry_algorithms_num returned %d for type %d\n", rc, i); - continue; - } - - comm->mpid.coll_count[i][0] = 0; - comm->mpid.coll_count[i][1] = 0; - if(num_algorithms[0] || num_algorithms[1]) - { - comm->mpid.coll_algorithm[i][0] = (pami_algorithm_t *) - MPL_malloc(sizeof(pami_algorithm_t) * num_algorithms[0]); - comm->mpid.coll_metadata[i][0] = (pami_metadata_t *) - MPL_malloc(sizeof(pami_metadata_t) * num_algorithms[0]); - comm->mpid.coll_algorithm[i][1] = (pami_algorithm_t *) - MPL_malloc(sizeof(pami_algorithm_t) * num_algorithms[1]); - comm->mpid.coll_metadata[i][1] = (pami_metadata_t *) - MPL_malloc(sizeof(pami_metadata_t) * num_algorithms[1]); - comm->mpid.coll_count[i][0] = num_algorithms[0]; - comm->mpid.coll_count[i][1] = num_algorithms[1]; - - /* Despite the bad name, this looks at algorithms associated with - * the geometry, NOT inherent physical properties of the geometry*/ - - /* BES TODO I am assuming all contexts have the same algorithms. Probably - * need to investigate that assumption - */ - rc = PAMI_Geometry_algorithms_query(geom, - i, - comm->mpid.coll_algorithm[i][0], - comm->mpid.coll_metadata[i][0], - num_algorithms[0], - comm->mpid.coll_algorithm[i][1], - comm->mpid.coll_metadata[i][1], - num_algorithms[1]); - if(rc != PAMI_SUCCESS) - { - fprintf(stderr,"PAMI_Geometry_algorithms_query returned %d for type %d\n", rc, i); - continue; - } - - if(MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_0 && comm->rank == 0) - { - for(j = 0; j < num_algorithms[0]; j++) - fprintf(stderr,"comm[%p] coll type %d (%s), algorithm %d 0: %s\n", comm, i, MPIDI_Coll_type_name(i), j, comm->mpid.coll_metadata[i][0][j].name); - for(j = 0; j < num_algorithms[1]; j++) - fprintf(stderr,"comm[%p] coll type %d (%s), algorithm %d 1: %s\n", comm, i, MPIDI_Coll_type_name(i), j, comm->mpid.coll_metadata[i][1][j].name); - if(i == PAMI_XFER_ALLGATHERV_INT || i == PAMI_XFER_ALLGATHER) - { - fprintf(stderr,"comm[%p] coll type %d (%s), \"glue\" algorithm: GLUE_ALLREDUCE\n", comm, i, MPIDI_Coll_type_name(i)); - fprintf(stderr,"comm[%p] coll type %d (%s), \"glue\" algorithm: GLUE_BCAST\n", comm, i, MPIDI_Coll_type_name(i)); - fprintf(stderr,"comm[%p] coll type %d (%s), \"glue\" algorithm: GLUE_ALLTOALL\n", comm, i, MPIDI_Coll_type_name(i)); - } - if(i == PAMI_XFER_SCATTERV_INT) - { - fprintf(stderr,"comm[%p] coll type %d (%s), \"glue\" algorithm: GLUE_BCAST\n", comm, i, MPIDI_Coll_type_name(i)); - fprintf(stderr,"comm[%p] coll type %d (%s), \"glue\" algorithm: GLUE_ALLTOALLV\n", comm, i, MPIDI_Coll_type_name(i)); - } - if(i == PAMI_XFER_SCATTER) - { - fprintf(stderr,"comm[%p] coll type %d (%s), \"glue\" algorithm: GLUE_BCAST\n", comm, i, MPIDI_Coll_type_name(i)); - } - if(i == PAMI_XFER_REDUCE) - { - fprintf(stderr,"comm[%p] coll type %d (%s), \"glue\" algorithm: GLUE_ALLREDUCE\n", comm, i, MPIDI_Coll_type_name(i)); - } - } - } - } - /* Determine if we have protocols for these maybe, rather than just setting them? */ - comm->coll_fns->Barrier = MPIDO_Barrier; - comm->coll_fns->Bcast = MPIDO_Bcast; - comm->coll_fns->Allreduce = MPIDO_Allreduce; - comm->coll_fns->Allgather = MPIDO_Allgather; - comm->coll_fns->Allgatherv = MPIDO_Allgatherv; - comm->coll_fns->Scatterv = MPIDO_Scatterv; - comm->coll_fns->Scatter = MPIDO_Scatter; - comm->coll_fns->Gather = MPIDO_Gather; - comm->coll_fns->Alltoallv = MPIDO_Alltoallv; - comm->coll_fns->Alltoall = MPIDO_Alltoall; - comm->coll_fns->Gatherv = MPIDO_Gatherv; - comm->coll_fns->Reduce = MPIDO_Reduce; - comm->coll_fns->Scan = MPIDO_Scan; - comm->coll_fns->Exscan = MPIDO_Exscan; - comm->coll_fns->Reduce_scatter_block = MPIDO_Reduce_scatter_block; - comm->coll_fns->Reduce_scatter = MPIDO_Reduce_scatter; - - /* MPI-3 Support, no optimized collectives hooked in yet */ - comm->coll_fns->Ibarrier_sched = MPIR_Ibarrier_intra; - comm->coll_fns->Ibcast_sched = MPIR_Ibcast_intra; - comm->coll_fns->Igather_sched = MPIR_Igather_intra; - comm->coll_fns->Igatherv_sched = MPIR_Igatherv; - comm->coll_fns->Iscatter_sched = MPIR_Iscatter_intra; - comm->coll_fns->Iscatterv_sched = MPIR_Iscatterv; - comm->coll_fns->Iallgather_sched = MPIR_Iallgather_intra; - comm->coll_fns->Iallgatherv_sched = MPIR_Iallgatherv_intra; - comm->coll_fns->Ialltoall_sched = MPIR_Ialltoall_intra; - comm->coll_fns->Ialltoallv_sched = MPIR_Ialltoallv_intra; - comm->coll_fns->Ialltoallw_sched = MPIR_Ialltoallw_intra; - comm->coll_fns->Iallreduce_sched = MPIR_Iallreduce_intra; - comm->coll_fns->Ireduce_sched = MPIR_Ireduce_intra; - comm->coll_fns->Ireduce_scatter_sched = MPIR_Ireduce_scatter_intra; - comm->coll_fns->Ireduce_scatter_block_sched = MPIR_Ireduce_scatter_block_intra; - comm->coll_fns->Iscan_sched = MPIR_Iscan_rec_dbl; - comm->coll_fns->Iexscan_sched = MPIR_Iexscan; - comm->coll_fns->Neighbor_allgather = MPIR_Neighbor_allgather_default; - comm->coll_fns->Neighbor_allgatherv = MPIR_Neighbor_allgatherv_default; - comm->coll_fns->Neighbor_alltoall = MPIR_Neighbor_alltoall_default; - comm->coll_fns->Neighbor_alltoallv = MPIR_Neighbor_alltoallv_default; - comm->coll_fns->Neighbor_alltoallw = MPIR_Neighbor_alltoallw_default; - comm->coll_fns->Ineighbor_allgather = MPIR_Ineighbor_allgather_default; - comm->coll_fns->Ineighbor_allgatherv = MPIR_Ineighbor_allgatherv_default; - comm->coll_fns->Ineighbor_alltoall = MPIR_Ineighbor_alltoall_default; - comm->coll_fns->Ineighbor_alltoallv = MPIR_Ineighbor_alltoallv_default; - comm->coll_fns->Ineighbor_alltoallw = MPIR_Ineighbor_alltoallw_default; - - /* MPI-3 Support, optimized collectives hooked in */ - comm->coll_fns->Ibarrier_req = MPIDO_Ibarrier; - comm->coll_fns->Ibcast_req = MPIDO_Ibcast; - comm->coll_fns->Iallgather_req = MPIDO_Iallgather; - comm->coll_fns->Iallgatherv_req = MPIDO_Iallgatherv; - comm->coll_fns->Iallreduce_req = MPIDO_Iallreduce; - comm->coll_fns->Ialltoall_req = MPIDO_Ialltoall; - comm->coll_fns->Ialltoallv_req = MPIDO_Ialltoallv; - comm->coll_fns->Ialltoallw_req = MPIDO_Ialltoallw; - comm->coll_fns->Iexscan_req = MPIDO_Iexscan; - comm->coll_fns->Igather_req = MPIDO_Igather; - comm->coll_fns->Igatherv_req = MPIDO_Igatherv; - comm->coll_fns->Ireduce_scatter_block_req = MPIDO_Ireduce_scatter_block; - comm->coll_fns->Ireduce_scatter_req = MPIDO_Ireduce_scatter; - comm->coll_fns->Ireduce_req = MPIDO_Ireduce; - comm->coll_fns->Iscan_req = MPIDO_Iscan; - comm->coll_fns->Iscatter_req = MPIDO_Iscatter; - comm->coll_fns->Iscatterv_req = MPIDO_Iscatterv; - - TRACE_ERR("MPIDI_Comm_coll_query exit\n"); -} - diff --git a/src/mpid/pamid/src/dyntask/Makefile.mk b/src/mpid/pamid/src/dyntask/Makefile.mk deleted file mode 100644 index 508de5d..0000000 --- a/src/mpid/pamid/src/dyntask/Makefile.mk +++ /dev/null @@ -1,32 +0,0 @@ -# begin_generated_IBM_copyright_prolog -# -# This is an automatically generated copyright prolog. -# After initializing, DO NOT MODIFY OR MOVE -# --------------------------------------------------------------- -# Licensed Materials - Property of IBM -# Blue Gene/Q 5765-PER 5765-PRP -# -# (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved -# US Government Users Restricted Rights - -# Use, duplication, or disclosure restricted -# by GSA ADP Schedule Contract with IBM Corp. -# -# --------------------------------------------------------------- -# -# end_generated_IBM_copyright_prolog -# -*- mode: makefile-gmake; -*- - -# note that the includes always happen but the effects of their contents are -# affected by "if BUILD_PAMID" -if BUILD_PAMID - - -mpi_core_sources += \ - src/mpid/pamid/src/dyntask/mpid_comm_spawn_multiple.c \ - src/mpid/pamid/src/dyntask/mpid_comm_disconnect.c \ - src/mpid/pamid/src/dyntask/mpid_port.c \ - src/mpid/pamid/src/dyntask/mpidi_pg.c \ - src/mpid/pamid/src/dyntask/mpidi_port.c - - -endif BUILD_PAMID diff --git a/src/mpid/pamid/src/dyntask/mpid_comm_disconnect.c b/src/mpid/pamid/src/dyntask/mpid_comm_disconnect.c deleted file mode 100644 index c573c82..0000000 --- a/src/mpid/pamid/src/dyntask/mpid_comm_disconnect.c +++ /dev/null @@ -1,541 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -#include "mpidimpl.h" - -#ifdef DYNAMIC_TASKING - -extern conn_info *_conn_info_list; - -#define DISCONNECT_LAPI_XFER_TIMEOUT 5*60*1000000 -#define TOTAL_AM 3 -#define FIRST_AM 0 -#define SECOND_AM 1 -#define LAST_AM 2 - -/* Returns time in micro seconds "double ticks" */ -#define CURTIME(ticks) { \ - struct timeval tp; \ - struct timezone tzp; \ - gettimeofday(&tp,&tzp); \ - ticks = (double) tp.tv_sec * 1000000 + (double) tp.tv_usec; \ -} - - -/* Used inside termination message send by smaller task to larger task in discon -nect */ -typedef struct { - long long tranid; - int whichAM; -}AM_struct2; - - -extern transactionID_struct *_transactionID_list; -void MPIDI_get_allremote_leaders(int *tid_arr, MPIR_Comm *comm_ptr); - -void MPIDI_send_AM_to_remote_leader_on_disconnect(int taskid, long long comm_cntr, int whichAM) -{ - pami_send_immediate_t xferP; - - int rc, current_val; - AM_struct2 AM_data; - pami_endpoint_t dest; - - AM_data.tranid = comm_cntr; - AM_data.whichAM = whichAM; - - memset(&xferP, 0, sizeof(pami_send_immediate_t)); - xferP.header.iov_base = (void*)&AM_data; - xferP.header.iov_len = sizeof(AM_struct2); - xferP.dispatch = (size_t)MPIDI_Protocols_Dyntask_disconnect; - - rc = PAMI_Endpoint_create(MPIDI_Client, taskid, 0, &dest); - TRACE_ERR("PAMI_Resume to taskid=%d\n", taskid); - PAMI_Resume(MPIDI_Context[0], - &dest, 1); - - if(rc != 0) - TRACE_ERR("PAMI_Endpoint_create failed\n"); - - xferP.dest = dest; - - rc = PAMI_Send_immediate(MPIDI_Context[0], &xferP); -} - -void MPIDI_Recvfrom_remote_world_disconnect(pami_context_t context, - void * cookie, - const void * _msginfo, - size_t msginfo_size, - const void * sndbuf, - size_t sndlen, - pami_endpoint_t sender, - pami_recv_t * recv) -{ - AM_struct2 *AM_data; - long long tranid; - int whichAM; - - AM_data = ((AM_struct2 *)_msginfo); - tranid = AM_data->tranid; - whichAM = AM_data->whichAM; - MPIDI_increment_AM_cntr_for_tranid(tranid, whichAM); - - TRACE_ERR("MPIDI_Recvfrom_remote_world_disconnect: invoked for tranid = %lld, whichAM = %d \n",tranid, whichAM); - - return; -} - - -/** - * Function to retreive the active message counter for a particular trasaction id. - * This function is used inside disconnect routine. - * whichAM = FIRST_AM/SECOND_AM/LAST_AM - */ -int MPIDI_get_AM_cntr_for_tranid(long long tranid, int whichAM) -{ - transactionID_struct *tridtmp; - - if(_transactionID_list == NULL) - TRACE_ERR("MPIDI_get_AM_cntr_for_tranid - _transactionID_list is NULL\n"); - - tridtmp = _transactionID_list; - - while(tridtmp != NULL) { - if(tridtmp->tranid == tranid) { - return tridtmp->cntr_for_AM[whichAM]; - } - tridtmp = tridtmp->next; - } - - return -1; -} - - -/** - * Function used to gurantee the delivery of LAPI active message at the - * destination. Called at the destination taskid and returns only whe - * the expected LAPI active message is recived. If the sequence number - * of the LAPI active message is LAST_AM, the other condition under which - * this function may exit is if DISCONNECT_LAPI_XFER_TIMEOUT happens. - */ -void MPIDI_wait_for_AM(long long tranid, int expected_AM, int whichAM) -{ - double starttime, currtime, elapsetime; - int rc, curr_AMcntr; - - rc = PAMI_Context_advance(MPIDI_Context[0], (size_t)100); - if(whichAM == LAST_AM) { - CURTIME(starttime) - do { - CURTIME(currtime) - elapsetime = currtime - starttime; - - rc = PAMI_Context_advance(MPIDI_Context[0], (size_t)100); - curr_AMcntr = MPIDI_get_AM_cntr_for_tranid(tranid, whichAM); - /*TRACE_ERR("_try_to_disconnect: Looping in timer for TranID %lld, whichAM %d expected_AM = %d, Current AM = %d\n",tranid,whichAM,expected_AM,curr_AMcntr); */ - }while(curr_AMcntr != expected_AM && elapsetime < DISCONNECT_LAPI_XFER_TIMEOUT); - } - else { - do { - rc = PAMI_Context_advance(MPIDI_Context[0], (size_t)100); - curr_AMcntr = MPIDI_get_AM_cntr_for_tranid(tranid, whichAM); - /*TRACE_ERR("_try_to_disconnect: Looping in timer for TranID %lld, whichAM %d expected_AM = %d, Current AM = %d\n",tranid,whichAM,expected_AM,curr_AMcntr);*/ - }while(curr_AMcntr != expected_AM); - } -} - -/* function to swap two integers. Used inside function _qsort_dyntask below */ -static void _swap_dyntask(int t[],int i,int j) -{ - int tmp; - - tmp = t[i]; - t[i] = t[j]; - t[j] = tmp; -} - -/* qsort sorting function which is used only in this file */ -static void _qsort_dyntask(int t[],int left,int right) -{ - int i,last; - - if(left >= right) return; - last = left; - for(i=left+1;i<=right;i++) - if(t[i] < t[left]) - _swap_dyntask(t,++last,i); - _swap_dyntask(t,left,last); - _qsort_dyntask(t,left,last-1); - _qsort_dyntask(t,last+1,right); -} - - - -/*@ - MPID_Comm_disconnect - Disconnect a communicator - - Arguments: -. comm_ptr - communicator - - Notes: - -.N Errors -.N MPI_SUCCESS -@*/ -int MPID_Comm_disconnect(MPIR_Comm *comm_ptr) -{ - int rc, i,j, k, ref_count,mpi_errno=0, probe_flag=0; - pami_task_t *local_list; - MPI_Status status; - MPIR_Errflag_t errflag = MPIR_ERR_NONE; - MPIDI_PG_t *pg; - int total_leaders=0, gsize; - pami_task_t *leader_tids; - int expected_firstAM=0, expected_secondAM=0, expected_lastAM=0; - MPIR_Comm *commworld_ptr; - MPIR_Group *group_ptr = NULL, *new_group_ptr = NULL; - MPID_VCR *glist; - MPIR_Comm *lcomm; - int *ranks; - int local_tasks=0, localtasks_in_remglist=0; - int jobIdSize=64; - char jobId[jobIdSize]; - int MY_TASKID = PAMIX_Client_query(MPIDI_Client, PAMI_CLIENT_TASK_ID ).value.intval; - - /*if( (comm_ptr->comm_kind == MPIR_COMM_KIND__INTERCOMM) && (comm_ptr->mpid.world_ids != NULL)) { */ - if(comm_ptr->mpid.world_ids != NULL) { - rc = MPID_Iprobe(comm_ptr->rank, MPI_ANY_TAG, comm_ptr, MPIR_CONTEXT_INTER_PT2PT, &probe_flag, &status); - if(rc || probe_flag) { - TRACE_ERR("PENDING_PTP"); - exit(1); - } - - /* make commSubWorld */ - { - /* MPIR_Comm_get_ptr( MPI_COMM_WORLD, commworld_ptr ); */ - commworld_ptr = MPIR_Process.comm_world; - mpi_errno = MPIR_Comm_group_impl(commworld_ptr, &group_ptr); - if (mpi_errno) - { - TRACE_ERR("Error while creating group_ptr from MPI_COMM_WORLD in MPIDI_Comm_create_from_pami_geom\n"); - return PAMI_ERROR; - } - - - glist = commworld_ptr->vcr; - gsize = commworld_ptr->local_size; - for(i=0;i<comm_ptr->local_size;i++) { - for(j=0;j<gsize;j++) { - if(comm_ptr->local_vcr[i]->taskid == glist[j]->taskid) - local_tasks++; - } - } - - /** - * Tasks belonging to the same local world may also be part of - * the GROUPREMLIST, so these tasks will have to be use in addition - * to the tasks in GROUPLIST to construct lcomm - **/ - if(comm_ptr->comm_kind == MPIR_COMM_KIND__INTERCOMM) { - for(i=0;i<comm_ptr->remote_size;i++) { - for(j=0;j<gsize;j++) { - if(comm_ptr->vcr[i]->taskid == glist[j]->taskid) { - local_tasks++; - localtasks_in_remglist++; - } - } - } - } - k=0; - /* local_list = MPL_malloc(local_tasks*sizeof(pami_task_t)); */ - ranks = MPL_malloc(local_tasks*sizeof(int)); - - for(i=0;i<comm_ptr->local_size;i++) { - for(j=0;j<gsize;j++) { - if(comm_ptr->local_vcr[i]->taskid == glist[j]->taskid) - /* local_list[k] = glist[j]->taskid; */ - ranks[k++] = j; - } - } - if((comm_ptr->comm_kind == MPIR_COMM_KIND__INTERCOMM) && localtasks_in_remglist) { - for(i=0;i<comm_ptr->remote_size;i++) { - for(j=0;j<gsize;j++) { - if(comm_ptr->vcr[i]->taskid == glist[j]->taskid) - /* local_list[k] = glist[j]->taskid; */ - ranks[k++] = j; - } - } - /* Sort the local_list when there are localtasks_in_remglist */ -/* _qsort_dyntask(local_list, 0, local_tasks-1); */ - _qsort_dyntask(ranks, 0, local_tasks-1); - } - - /* Now we have all we need to create the new group. Create it */ - /* mpi_errno = MPIR_Group_incl_impl(group_ptr, local_tasks, ranks, &new_group_ptr); */ - mpi_errno = MPIR_Group_incl_impl(group_ptr, local_tasks, ranks, &new_group_ptr); - if (mpi_errno) - { - TRACE_ERR("Error while creating new_group_ptr from group_ptr in MPIDI_Comm_create_from_pami_geom\n"); - return PAMI_ERROR; - } - - /* Now create the communicator using the new_group_ptr */ - mpi_errno = MPIR_Comm_create_group(commworld_ptr, new_group_ptr, 0, &lcomm); - /* mpi_errno = MPIR_Comm_create_intra(commworld_ptr, new_group_ptr, &lcomm); */ - if (mpi_errno) - { - TRACE_ERR("Error while creating new_comm_ptr from group_ptr in MPIDI_Comm_create_from_pami_geom\n"); - return PAMI_ERROR; - } - -#if 0 - mpi_errno = MPIR_Comm_create(&lcomm); - if (mpi_errno != MPI_SUCCESS) { - TRACE_ERR("MPIR_Comm_create returned with mpi_errno=%d\n", mpi_errno); - } - - /* fill in all the fields of lcomm. */ - if(localtasks_in_remglist==0) { - lcomm->context_id = MPIR_CONTEXT_SET_FIELD(DYNAMIC_PROC, comm_ptr->recvcontext_id, 1); - lcomm->recvcontext_id = lcomm->context_id; - } else { - lcomm->context_id = MPIR_CONTEXT_SET_FIELD(DYNAMIC_PROC, comm_ptr->recvcontext_id, 1); - lcomm->recvcontext_id = MPIR_CONTEXT_SET_FIELD(DYNAMIC_PROC, comm_ptr->context_id, 1); - } - TRACE_ERR("lcomm->context_id =%d\n", lcomm->context_id); - - /* sanity: the INVALID context ID value could potentially conflict with the - * dynamic proccess space */ - MPIR_Assert(lcomm->context_id != MPIR_INVALID_CONTEXT_ID); - MPIR_Assert(lcomm->recvcontext_id != MPIR_INVALID_CONTEXT_ID); - - /* FIXME - we probably need a unique context_id. */ - - /* Fill in new intercomm */ - lcomm->comm_kind = MPIR_COMM_KIND__INTRACOMM; - lcomm->remote_size = lcomm->local_size = local_tasks; - - /* Set up VC reference table */ - mpi_errno = MPID_VCRT_Create(lcomm->remote_size, &lcomm->vcrt); - if (mpi_errno != MPI_SUCCESS) { - TRACE_ERR("MPID_VCRT_Create returned with mpi_errno=%d", mpi_errno); - } - MPID_VCRT_Add_ref(lcomm->vcrt); - mpi_errno = MPID_VCRT_Get_ptr(lcomm->vcrt, &lcomm->vcr); - if (mpi_errno != MPI_SUCCESS) { - TRACE_ERR("MPID_VCRT_Get_ptr returned with mpi_errno=%d", mpi_errno); - } - - for(i=0; i<local_tasks; i++) { - if(MY_TASKID == local_list[i]) lcomm->rank = i; - lcomm->vcr[i]->taskid = local_list[i]; - } -#endif - - } - - TRACE_ERR("subcomm for disconnect is established local_tasks=%d calling MPIR_Barrier_intra\n", local_tasks); - mpi_errno = MPIR_Barrier_intra(lcomm, &errflag); - if (mpi_errno != MPI_SUCCESS) { - TRACE_ERR("MPIR_Barrier_intra returned with mpi_errno=%d\n", mpi_errno); - } - TRACE_ERR("after barrier in disconnect\n"); - - if(MY_TASKID != comm_ptr->mpid.local_leader) { - for(i=0; comm_ptr->mpid.world_ids[i] != -1; i++) { - ref_count = MPIDI_Decrement_ref_count(comm_ptr->mpid.world_ids[i]); - TRACE_ERR("ref_count=%d with world=%d comm_ptr=%x\n", ref_count, comm_ptr->mpid.world_ids[i], comm_ptr); - if(ref_count == -1) - TRACE_ERR("something is wrong\n"); - } - } - - if(MY_TASKID == comm_ptr->mpid.local_leader) { - PMI2_Job_GetId(jobId, jobIdSize); - for(i=0;comm_ptr->mpid.world_ids[i]!=-1;i++) { - if(atoi(jobId) != comm_ptr->mpid.world_ids[i]) - total_leaders++; - } - TRACE_ERR("total_leaders=%d\n", total_leaders); - leader_tids = MPL_malloc(total_leaders*sizeof(int)); - MPIDI_get_allremote_leaders(leader_tids, comm_ptr); - - { /* First Pair of Send / Recv -- All smaller task send to all larger tasks */ - for(i=0;i<total_leaders;i++) { - MPID_assert(leader_tids[i] != -1); - if(MY_TASKID < leader_tids[i]) { - TRACE_ERR("_try_to_disconnect: FIRST: comm_ptr->mpid.world_intercomm_cntr %lld, toTaskid %d\n",comm_ptr->mpid.world_intercomm_cntr,leader_tids[i]); - expected_firstAM++; - MPIDI_send_AM_to_remote_leader_on_disconnect(leader_tids[i], comm_ptr->mpid.world_intercomm_cntr, FIRST_AM); - } - else { - expected_secondAM++; - } - } - if(expected_secondAM) { - MPIDI_wait_for_AM(comm_ptr->mpid.world_intercomm_cntr, expected_secondAM, FIRST_AM); - } - } - - { /* Second Pair of Send / Recv -- All larger tasks send to all smaller tasks */ - for(i=0;i<total_leaders;i++) { - MPID_assert(leader_tids[i] != -1); - if(MY_TASKID > leader_tids[i]) { - TRACE_ERR("_try_to_disconnect: SECOND: comm_ptr->mpid.world_intercomm_cntr %lld, toTaskid %d\n",comm_ptr->mpid.world_intercomm_cntr,leader_tids[i]); - MPIDI_send_AM_to_remote_leader_on_disconnect(leader_tids[i], comm_ptr->mpid.world_intercomm_cntr, SECOND_AM); - } - } - if(expected_firstAM) { - MPIDI_wait_for_AM(comm_ptr->mpid.world_intercomm_cntr, expected_firstAM, SECOND_AM); - } - } - - for(i=0; comm_ptr->mpid.world_ids[i] != -1; i++) { - ref_count = MPIDI_Decrement_ref_count(comm_ptr->mpid.world_ids[i]); - TRACE_ERR("ref_count=%d with world=%d comm_ptr=%x\n", ref_count, comm_ptr->mpid.world_ids[i], comm_ptr); - if(ref_count == -1) - TRACE_ERR("something is wrong\n"); - } - - for(i=0;i<total_leaders;i++) { - MPID_assert(leader_tids[i] != -1); - if(MY_TASKID < leader_tids[i]) { - TRACE_ERR("_try_to_disconnect: LAST: comm_ptr->mpid.world_intercomm_cntr %lld, toTaskid %d\n",comm_ptr->mpid.world_intercomm_cntr,leader_tids[i]); - MPIDI_send_AM_to_remote_leader_on_disconnect(leader_tids[i], comm_ptr->mpid.world_intercomm_cntr, LAST_AM); - } - else { - expected_lastAM++; - } - } - if(expected_lastAM) { - MPIDI_wait_for_AM(comm_ptr->mpid.world_intercomm_cntr, expected_lastAM, - LAST_AM); - } - MPL_free(leader_tids); - } - - TRACE_ERR("_try_to_disconnect: Going inside final barrier for tranid %lld\n",comm_ptr->mpid.world_intercomm_cntr); - mpi_errno = MPIR_Barrier_intra(lcomm, &errflag); - if (mpi_errno != MPI_SUCCESS) { - TRACE_ERR("MPIR_Barrier_intra returned with mpi_errno=%d\n", mpi_errno); - } - mpi_errno = MPIR_Comm_release(lcomm,0); - if (mpi_errno) TRACE_ERR("MPIR_Comm_release returned with mpi_errno=%d\n", mpi_errno); - - MPIDI_free_tranid_node(comm_ptr->mpid.world_intercomm_cntr); - mpi_errno = MPIR_Comm_release(comm_ptr,1); - if (mpi_errno) TRACE_ERR("MPIR_Comm_release returned with mpi_errno=%d\n", mpi_errno); - /* MPL_free(local_list); */ - MPL_free(ranks); - } - return mpi_errno; -} - - -int MPIDI_Decrement_ref_count(int wid) { - conn_info *tmp_node; - int ref_count=-1; - - tmp_node = _conn_info_list; - while(tmp_node != NULL) { - if(tmp_node->rem_world_id == wid) { - ref_count = --tmp_node->ref_count; - TRACE_ERR("decrement_ref_count: ref_count decremented to %d for remote world %d\n",ref_count,wid); - break; - } - tmp_node = tmp_node->next; - } - return ref_count; -} - -void MPIDI_get_allremote_leaders(int *tid_arr, MPIR_Comm *comm_ptr) -{ - conn_info *tmp_node; - int i,j,k,arr_len,gsize, found=0; - int leader1=-1, leader2=-1; - MPID_VCR *glist; - - for(i=0;comm_ptr->mpid.world_ids[i] != -1;i++) - { - TRACE_ERR("i=%d comm_ptr->mpid.world_ids[i]=%d\n", i, comm_ptr->mpid.world_ids[i]); - tmp_node = _conn_info_list; - found=0; - if(tmp_node==NULL) {TRACE_ERR("_conn_info_list is NULL\n");} - while(tmp_node != NULL) { - if(tmp_node->rem_world_id == comm_ptr->mpid.world_ids[i]) { - if(comm_ptr->comm_kind == MPIR_COMM_KIND__INTRACOMM) { - glist = comm_ptr->local_vcr; - gsize = comm_ptr->local_size; - } - else { - glist = comm_ptr->vcr; - gsize = comm_ptr->remote_size; - } - for(j=0;j<gsize;j++) { - for(k=0;tmp_node->rem_taskids[k]!=-1;k++) { - TRACE_ERR("j=%d k=%d glist[j]->taskid=%d tmp_node->rem_taskids[k]=%d\n", j, k,glist[j]->taskid, tmp_node->rem_taskids[k]); - if(glist[j]->taskid == tmp_node->rem_taskids[k]) { - leader1 = glist[j]->taskid; - found = 1; - break; - } - } - if(found) break; - } - /* - * There may be the case where my local_comm's GROUPLIST contains tasks - * fro remote world-x and GROUPREMLIST contains other remaining tasks of world-x - * If the smallest task of world-x is in my GROUPLIST then the above iteration - * will give the leader as smallest task from world-x in my GROUPREMLIST. - * But this will not be the correct leader_taskid. I should find the smallest task - * of world-x in my GROUPLIST and then see which of the two leaders is the - * smallest one. The smallest one is the one in which I am interested. - **/ - if(comm_ptr->comm_kind == MPIR_COMM_KIND__INTERCOMM) { - found=0; - glist = comm_ptr->local_vcr; - gsize = comm_ptr->local_size; - for(j=0;j<gsize;j++) { - for(k=0;tmp_node->rem_taskids[k]!=-1;k++) { - TRACE_ERR("j=%d k=%d glist[j]->taskid=%d tmp_node->rem_taskids[k]=%d\n", j, k, glist[j]->taskid, tmp_node->rem_taskids[k]); - if(glist[j]->taskid == tmp_node->rem_taskids[k]) { - leader2 = glist[j]->taskid; - found = 1; - break; - } - } - if(found) break; - } - } - if(found) { - break; - } - } else {TRACE_ERR("world id is different tmp_node->rem_world_id =%d comm_ptr=%x comm_ptr->mpid.world_ids[i]=%d\n", tmp_node->rem_world_id, comm_ptr, comm_ptr->mpid.world_ids[i]);} - tmp_node = tmp_node->next; - } - - TRACE_ERR("comm_ptr=%x leader1=%d leader2=%d\n", comm_ptr, leader1, leader2); - if(leader1 == -1) - *(tid_arr+i) = leader2; - else if(leader2 == -1) - *(tid_arr+i) = leader1; - else - *(tid_arr+i) = leader1 < leader2 ? leader1 : leader2; - } -} - -#endif diff --git a/src/mpid/pamid/src/dyntask/mpid_comm_spawn_multiple.c b/src/mpid/pamid/src/dyntask/mpid_comm_spawn_multiple.c deleted file mode 100644 index 6e3b21c..0000000 --- a/src/mpid/pamid/src/dyntask/mpid_comm_spawn_multiple.c +++ /dev/null @@ -1,414 +0,0 @@ -/* -*- Mode: C; c-basic-offset:4 ; -*- */ -/* - * (C) 2001 by Argonne National Laboratory. - * See COPYRIGHT in top-level directory. - */ - -#include "mpidimpl.h" -#ifdef USE_PMI2_API -#include "pmi2.h" -#else -#include "pmi.h" -#endif - -#ifdef DYNAMIC_TASKING - -extern mpidi_dynamic_tasking; - -/* Define the name of the kvs key used to provide the port name to the - children */ -#define MPIDI_PARENT_PORT_KVSKEY "PARENT_ROOT_PORT_NAME" - -/* FIXME: We can avoid these two routines if we define PMI as using - MPI info values */ -/* Turn a SINGLE MPI_Info into an array of PMI_keyvals (return the pointer - to the array of PMI keyvals) */ -static int MPIDI_mpi_to_pmi_keyvals( MPIR_Info *info_ptr, PMI_keyval_t **kv_ptr, int *nkeys_ptr ) -{ - char key[MPI_MAX_INFO_KEY]; - PMI_keyval_t *kv = 0; - int i, nkeys = 0, vallen, flag, mpi_errno=MPI_SUCCESS; - - if (!info_ptr || info_ptr->handle == MPI_INFO_NULL) { - goto fn_exit; - } - - MPIR_Info_get_nkeys_impl( info_ptr, &nkeys ); - if (nkeys == 0) { - goto fn_exit; - } - kv = (PMI_keyval_t *)MPL_malloc( nkeys * sizeof(PMI_keyval_t) ); - - for (i=0; i<nkeys; i++) { - mpi_errno = MPIR_Info_get_nthkey_impl( info_ptr, i, key ); - if (mpi_errno) - TRACE_ERR("MPIR_Info_get_nthkey_impl returned with mpi_errno=%d\n", mpi_errno); - MPIR_Info_get_valuelen_impl( info_ptr, key, &vallen, &flag ); - - kv[i].key = MPL_strdup(key); - kv[i].val = MPL_malloc( vallen + 1 ); - MPIR_Info_get_impl( info_ptr, key, vallen+1, kv[i].val, &flag ); - TRACE_OUT("key: <%s>, value: <%s>\n", kv[i].key, kv[i].val); - } - - fn_fail: - fn_exit: - *kv_ptr = kv; - *nkeys_ptr = nkeys; - return mpi_errno; -} - - -/* Free the entire array of PMI keyvals */ -static void MPIDI_free_pmi_keyvals(PMI_keyval_t **kv, int size, int *counts) -{ - int i,j; - - for (i=0; i<size; i++) - { - for (j=0; j<counts[i]; j++) - { - if (kv[i][j].key != NULL) - MPL_free((char *)kv[i][j].key); - if (kv[i][j].val != NULL) - MPL_free(kv[i][j].val); - } - if (kv[i] != NULL) - { - MPL_free(kv[i]); - } - } -} - -/*@ - MPID_Comm_spawn_multiple - - - Input Arguments: -+ int count - count -. char *array_of_commands[] - commands -. char* *array_of_argv[] - arguments -. int array_of_maxprocs[] - maxprocs -. MPI_Info array_of_info[] - infos -. int root - root -- MPI_Comm comm - communicator - - Output Arguments: -+ MPI_Comm *intercomm - intercommunicator -- int array_of_errcodes[] - error codes - - Notes: - -.N Errors -.N MPI_SUCCESS -@*/ -#undef FUNCNAME -#define FUNCNAME MPID_Comm_spawn_multiple -#undef FCNAME -#define FCNAME MPIU_QUOTE(FUNCNAME) -int MPID_Comm_spawn_multiple(int count, char *array_of_commands[], - char ** array_of_argv[], const int array_of_maxprocs[], - MPIR_Info * array_of_info_ptrs[], int root, - MPIR_Comm * comm_ptr, MPIR_Comm ** intercomm, - int array_of_errcodes[]) -{ - int mpi_errno = MPI_SUCCESS; - - if(mpidi_dynamic_tasking == 0) { - fprintf(stderr, "Received spawn request for non-dynamic jobs\n"); - MPIU_ERR_SETANDSTMT(mpi_errno, MPI_ERR_SPAWN, - return mpi_errno, "**spawn"); - } - - /* We allow an empty implementation of this function to - simplify building MPICH on systems that have difficulty - supporing process creation */ - mpi_errno = MPIDI_Comm_spawn_multiple(count, array_of_commands, - array_of_argv, array_of_maxprocs, - array_of_info_ptrs, - root, comm_ptr, intercomm, - array_of_errcodes); - return mpi_errno; -} - - -/* - * MPIDI_Comm_spawn_multiple() - */ -int MPIDI_Comm_spawn_multiple(int count, char **commands, - char ***argvs, int *maxprocs, - MPIR_Info **info_ptrs, int root, - MPIR_Comm *comm_ptr, MPIR_Comm - **intercomm, int *errcodes) -{ - char port_name[MPI_MAX_PORT_NAME]; - char jobId[64]; - char ctxid_str[16]; - int jobIdSize = 64; - int len=0; - int *info_keyval_sizes=0, i, mpi_errno=MPI_SUCCESS; - PMI_keyval_t **info_keyval_vectors=0, preput_keyval_vector; - int *pmi_errcodes = 0, pmi_errno=0; - int total_num_processes, should_accept = 1; - MPIR_Info tmp_info_ptr; - char *tmp; - int tmp_ret = 0; - - if (comm_ptr->rank == root) { - /* create an array for the pmi error codes */ - total_num_processes = 0; - for (i=0; i<count; i++) { - total_num_processes += maxprocs[i]; - } - pmi_errcodes = (int*)MPL_malloc(sizeof(int) * total_num_processes); - - /* initialize them to 0 */ - for (i=0; i<total_num_processes; i++) - pmi_errcodes[i] = 0; - - /* Open a port for the spawned processes to connect to */ - /* FIXME: info may be needed for port name */ - mpi_errno = MPID_Open_port(NULL, port_name); - TRACE_ERR("mpi_errno from MPID_Open_port=%d\n", mpi_errno); - - /* Spawn the processes */ -#ifdef USE_PMI2_API - MPIR_Assert(count > 0); - { - int *argcs = MPL_malloc(count*sizeof(int)); - struct MPIR_Info preput; - struct MPIR_Info *preput_p[2] = { &preput, &tmp_info_ptr }; - - MPIR_Assert(argcs); - - info_keyval_sizes = MPL_malloc(count * sizeof(int)); - - /* FIXME cheating on constness */ - preput.key = (char *)MPIDI_PARENT_PORT_KVSKEY; - preput.value = port_name; - preput.next = &tmp_info_ptr; - - tmp_info_ptr.key = "COMMCTX"; - len=sprintf(ctxid_str, "%d", comm_ptr->context_id); - TRACE_ERR("COMMCTX=%d\n", comm_ptr->context_id); - ctxid_str[len]='\0'; - tmp_info_ptr.value = ctxid_str; - tmp_info_ptr.next = NULL; - - /* compute argcs array */ - for (i = 0; i < count; ++i) { - argcs[i] = 0; - if (argvs != NULL && argvs[i] != NULL) { - while (argvs[i][argcs[i]]) { - ++argcs[i]; - } - } - } - - /*MPIU_THREAD_CS_ENTER(PMI,);*/ - /* release the global CS for spawn PMI calls */ - MPIU_THREAD_CS_EXIT(ALLFUNC,); - pmi_errno = PMI2_Job_Spawn(count, (const char **)commands, - argcs, (const char ***)argvs, - maxprocs, - info_keyval_sizes, (const MPIR_Info **)info_ptrs, - 2, (const struct MPIR_Info **)preput_p, - jobId, jobIdSize, - pmi_errcodes); - TRACE_ERR("after PMI2_Job_Spawn - pmi_errno=%d jobId=%s\n", pmi_errno, jobId); - MPIU_THREAD_CS_ENTER(ALLFUNC,); - - tmp=MPL_strdup(jobId); - tmp_ret = atoi(strtok(tmp, ";")); - - if( (pmi_errno == PMI2_SUCCESS) && (tmp_ret != -1) ) { - pami_task_t leader_taskid = atoi(strtok(NULL, ";")); - pami_endpoint_t ldest; - - PAMI_Endpoint_create(MPIDI_Client, leader_taskid, 0, &ldest); - TRACE_ERR("PAMI_Resume to taskid=%d\n", leader_taskid); - PAMI_Resume(MPIDI_Context[0], &ldest, 1); - } - - MPL_free(tmp); - - MPL_free(argcs); - if (pmi_errno != PMI2_SUCCESS) { - TRACE_ERR("PMI2_Job_Spawn returned with pmi_errno=%d\n", pmi_errno); - } - } -#else - /* FIXME: This is *really* awkward. We should either - Fix on MPI-style info data structures for PMI (avoid unnecessary - duplication) or add an MPIU_Info_getall(...) that creates - the necessary arrays of key/value pairs */ - - /* convert the infos into PMI keyvals */ - info_keyval_sizes = (int *) MPL_malloc(count * sizeof(int)); - info_keyval_vectors = - (PMI_keyval_t**) MPL_malloc(count * sizeof(PMI_keyval_t*)); - - if (!info_ptrs) { - for (i=0; i<count; i++) { - info_keyval_vectors[i] = 0; - info_keyval_sizes[i] = 0; - } - } - else { - for (i=0; i<count; i++) { - mpi_errno = MPIDI_mpi_to_pmi_keyvals( info_ptrs[i], - &info_keyval_vectors[i], - &info_keyval_sizes[i] ); - if (mpi_errno) { TRACE_ERR("MPIDI_mpi_to_pmi_keyvals returned with mpi_errno=%d\n", mpi_errno); } - } - } - - preput_keyval_vector.key = MPIDI_PARENT_PORT_KVSKEY; - preput_keyval_vector.val = port_name; - - pmi_errno = PMI_Spawn_multiple(count, (const char **) - commands, - (const char ***) argvs, - maxprocs, info_keyval_sizes, - (const PMI_keyval_t **) - info_keyval_vectors, 1, - &preput_keyval_vector, - pmi_errcodes); - TRACE_ERR("pmi_errno from PMI_Spawn_multiple=%d\n", pmi_errno); -#endif - - if (errcodes != MPI_ERRCODES_IGNORE) { - for (i=0; i<total_num_processes; i++) { - /* FIXME: translate the pmi error codes here */ - errcodes[i] = pmi_errcodes[0]; - /* We want to accept if any of the spawns succeeded. - Alternatively, this is the same as we want to NOT accept if - all of them failed. should_accept = NAND(e_0, ..., e_n) - Remember, success equals false (0). */ - should_accept = should_accept && errcodes[i]; - } - should_accept = !should_accept; /* the `N' in NAND */ - } - -#ifdef USE_PMI2_API - if( (pmi_errno == PMI2_SUCCESS) && (tmp_ret == -1) ) -#else - if( (pmi_errno == PMI_SUCCESS) && (tmp_ret == -1) ) -#endif - should_accept = 0; - } - - if (errcodes != MPI_ERRCODES_IGNORE) { - MPIR_Errflag_t errflag = MPIR_ERR_NONE; - mpi_errno = MPIR_Bcast_impl(&should_accept, 1, MPI_INT, root, comm_ptr, &errflag); - if (mpi_errno) TRACE_ERR("MPIR_Bcast_impl returned with mpi_errno=%d\n", mpi_errno); - - mpi_errno = MPIR_Bcast_impl(&pmi_errno, 1, MPI_INT, root, comm_ptr, &errflag); - if (mpi_errno) TRACE_ERR("MPIR_Bcast_impl returned with mpi_errno=%d\n", mpi_errno); - - mpi_errno = MPIR_Bcast_impl(&total_num_processes, 1, MPI_INT, root, comm_ptr, &errflag); - if (mpi_errno) TRACE_ERR("MPIR_Bcast_impl returned with mpi_errno=%d\n", mpi_errno); - - mpi_errno = MPIR_Bcast_impl(errcodes, total_num_processes, MPI_INT, root, comm_ptr, &errflag); - if (mpi_errno) TRACE_ERR("MPIR_Bcast_impl returned with mpi_errno=%d\n", mpi_errno); - } - - if (should_accept) { - mpi_errno = MPID_Comm_accept(port_name, NULL, root, comm_ptr, intercomm); - TRACE_ERR("mpi_errno from MPID_Comm_accept=%d\n", mpi_errno); - } else { - if( (pmi_errno == PMI2_SUCCESS) && (errcodes[0] != 0) ) { - MPIR_Comm_create(intercomm); - } - } - - if (comm_ptr->rank == root) { - /* Close the port opened for the spawned processes to connect to */ - mpi_errno = MPID_Close_port(port_name); - /* --BEGIN ERROR HANDLING-- */ - if (mpi_errno != MPI_SUCCESS) - TRACE_ERR("MPID_Close_port returned with mpi_errno=%d\n", mpi_errno); - /* --END ERROR HANDLING-- */ - } - - if(pmi_errno) { - mpi_errno = MPIR_Err_create_code(MPI_SUCCESS, MPIR_ERR_FATAL, __FILE__, __LINE__, MPI_ERR_SPAWN, - "**mpi_comm_spawn", 0); - } - - fn_exit: - if (info_keyval_vectors) { - MPIDI_free_pmi_keyvals(info_keyval_vectors, count, info_keyval_sizes); - MPL_free(info_keyval_vectors); - } - if (info_keyval_sizes) { - MPL_free(info_keyval_sizes); - } - if (pmi_errcodes) { - MPL_free(pmi_errcodes); - } - return mpi_errno; - fn_fail: - goto fn_exit; -} - - -/* This function is used only with mpid_init to set up the parent communicator - if there is one. The routine should be in this file because the parent - port name is setup with the "preput" arguments to PMI_Spawn_multiple */ -static char *parent_port_name = 0; /* Name of parent port if this - process was spawned (and is root - of comm world) or null */ -#undef FUNCNAME -#define FUNCNAME MPIDI_GetParentPort -#undef FCNAME -#define FCNAME MPIU_QUOTE(FUNCNAME) -int MPIDI_GetParentPort(char ** parent_port) -{ - int mpi_errno = MPI_SUCCESS; - int pmi_errno; - char val[MPIDI_MAX_KVS_VALUE_LEN]; - - if (parent_port_name == NULL) - { - char *kvsname = NULL; - /* We can always use PMI_KVS_Get on our own process group */ - MPIDI_PG_GetConnKVSname( &kvsname ); -#ifdef USE_PMI2_API - { - int vallen = 0; - pmi_errno = PMI2_KVS_Get(kvsname, PMI2_ID_NULL, MPIDI_PARENT_PORT_KVSKEY, val, sizeof(val), &vallen); - TRACE_ERR("PMI2_KVS_Get - val=%s\n", val); - if (pmi_errno) - TRACE_ERR("PMI2_KVS_Get returned with pmi_errno=%d\n", pmi_errno); - } -#else - /*MPIU_THREAD_CS_ENTER(PMI,);*/ - pmi_errno = PMI_KVS_Get( kvsname, MPIDI_PARENT_PORT_KVSKEY, val, sizeof(val)); -/* MPIU_THREAD_CS_EXIT(PMI,);*/ - if (pmi_errno) { - mpi_errno = MPIR_Err_create_code(MPI_SUCCESS, MPIR_ERR_FATAL, FCNAME, __LINE__, MPI_ERR_OTHER, "**pmi_kvsget", "**pmi_kvsget %d", pmi_errno); - goto fn_exit; - } -#endif - parent_port_name = MPL_strdup(val); - } - - *parent_port = parent_port_name; - - fn_exit: - return mpi_errno; - fn_fail: - goto fn_exit; -} - - -void MPIDI_FreeParentPort(void) -{ - if (parent_port_name) { - MPL_free( parent_port_name ); - parent_port_name = 0; - } -} - - -#endif diff --git a/src/mpid/pamid/src/dyntask/mpid_port.c b/src/mpid/pamid/src/dyntask/mpid_port.c deleted file mode 100644 index 7f115d8..0000000 --- a/src/mpid/pamid/src/dyntask/mpid_port.c +++ /dev/null @@ -1,317 +0,0 @@ -/* -*- Mode: C; c-basic-offset:4 ; -*- */ -/* - * (C) 2001 by Argonne National Laboratory. - * See COPYRIGHT in top-level directory. - */ - -#include "mpidimpl.h" -#include "netdb.h" -#include <net/if.h> - - -#ifdef DYNAMIC_TASKING - -#define MAX_HOST_DESCRIPTION_LEN 128 -#define NUM_IFREQS 10 -#define MPI_MAX_TASKID_NAME 8 -#define MPIDI_TASKID_TAG_KEY "taskid" - -extern mpidi_dynamic_tasking; - -static int MPIDI_Open_port(MPIR_Info *, char *); -static int MPIDI_Close_port(const char *); - - -/* Define the functions that are used to implement the port - * operations */ -static MPIDI_PortFns portFns = { MPIDI_Open_port, - MPIDI_Close_port, - MPIDI_Comm_accept, - MPIDI_Comm_connect }; - -/*@ - MPID_Open_port - Open an MPI Port - - Input Arguments: -. MPI_Info info - info - - Output Arguments: -. char *port_name - port name - - Notes: - - -.N Errors -.N MPI_SUCCESS -.N MPI_ERR_OTHER -@*/ -int MPID_Open_port(MPIR_Info *info_ptr, char *port_name) -{ - int mpi_errno=MPI_SUCCESS; - - if (portFns.OpenPort) { - mpi_errno = portFns.OpenPort( info_ptr, port_name ); - if (mpi_errno != MPI_SUCCESS) { - TRACE_ERR("OpenPort returned with mpi_errno=%d\n", mpi_errno); - } - } - - fn_fail: - return mpi_errno; -} - - - -/*@ - MPID_Close_port - Close port - - Input Parameter: -. port_name - Name of MPI port to close - - Notes: - -.N Errors -.N MPI_SUCCESS -.N MPI_ERR_OTHER - -@*/ -int MPID_Close_port(const char *port_name) -{ - int mpi_errno=MPI_SUCCESS; - - if (portFns.ClosePort) { - mpi_errno = portFns.ClosePort( port_name ); - if (mpi_errno != MPI_SUCCESS) { - TRACE_ERR("ClosePort returned with mpi_errno=%d\n", mpi_errno); - } - } - - fn_fail: - return mpi_errno; -} - -#undef FUNCNAME -#define FUNCNAME MPID_Comm_accept -#undef FCNAME -#define FCNAME MPIU_QUOTE(FUNCNAME) -int MPID_Comm_accept(const char * port_name, MPIR_Info * info, int root, - MPIR_Comm * comm, MPIR_Comm ** newcomm_ptr) -{ - int mpi_errno = MPI_SUCCESS; - - if(mpidi_dynamic_tasking == 0) { - fprintf(stderr, "Dynamic tasking API is called on non-dynamic jobs\n"); - MPIU_ERR_SETANDSTMT(mpi_errno, MPI_ERR_SPAWN, - return mpi_errno, "**spawn"); - - } - - if (portFns.CommAccept) { - mpi_errno = portFns.CommAccept( port_name, info, root, comm, - newcomm_ptr ); - if (mpi_errno != MPI_SUCCESS) { - TRACE_ERR("CommAccept returned with mpi_errno=%d\n", mpi_errno); - } - } - - fn_fail: - return mpi_errno; -} - -#undef FUNCNAME -#define FUNCNAME MPID_Comm_connect -#undef FCNAME -#define FCNAME MPIU_QUOTE(FUNCNAME) -int MPID_Comm_connect(const char * port_name, MPIR_Info * info, int root, - MPIR_Comm * comm, MPIR_Comm ** newcomm_ptr) -{ - int mpi_errno=MPI_SUCCESS; - - if(mpidi_dynamic_tasking == 0) { - fprintf(stderr, "Dynamic tasking API is called on non-dynamic jobs\n"); - MPIU_ERR_SETANDSTMT(mpi_errno, MPI_ERR_SPAWN, - return mpi_errno, "**spawn"); - - } - - if (portFns.CommConnect) { - mpi_errno = portFns.CommConnect( port_name, info, root, comm, - newcomm_ptr ); - if (mpi_errno != MPI_SUCCESS) { - TRACE_ERR("CommConnect returned with mpi_errno=%d\n", mpi_errno); - } - } - - fn_fail: - return mpi_errno; -} - - -/* - * Here are the routines that provide some of the default implementations - * for the Port routines. - * - * MPIDI_Open_port - creates a port "name" that includes a tag value that - * is used to separate different MPI Port values. That tag value is - * extracted with MPIDI_GetTagFromPort - * MPIDI_GetTagFromPort - Routine to return the tag associated with a port. - * - * The port_name_tag is used in the connect and accept messages that - * are used in the connect/accept protocol. - */ - -#define MPIDI_PORT_NAME_TAG_KEY "tag" -#define MPIDI_TASKID_TAG_KEY "taskid" - -/* Though the port_name_tag_mask itself is an int, we can only have as - * many tags as the context_id space can support. */ -static int port_name_tag_mask[MPIR_MAX_CONTEXT_MASK] = { 0 }; - -static int MPIDI_get_port_name_tag(int * port_name_tag) -{ - int i, j; - int mpi_errno = MPI_SUCCESS; - - - for (i = 0; i < MPIR_MAX_CONTEXT_MASK; i++) - if (port_name_tag_mask[i] != ~0) - break; - - TRACE_ERR("MPIDI_get_port_name_tag - i=%d MPIR_MAX_CONTEXT_MASK=%d", i, MPIR_MAX_CONTEXT_MASK); - if (i < MPIR_MAX_CONTEXT_MASK) { - /* Found a free tag. port_name_tag_mask[i] is not fully used - * up. */ - - /* OR the mask value with powers of two. If the OR value is - * the same as the original value, then it means that the - * OR'ed bit was originally 1 (used); otherwise, it was - * originally 0 (free). */ - for (j = 0; j < (8 * sizeof(int)); j++) { - if ((port_name_tag_mask[i] | (1 << ((8 * sizeof(int)) - j - 1))) != - port_name_tag_mask[i]) { - /* Mark the appropriate bit as used and return that */ - port_name_tag_mask[i] |= (1 << ((8 * sizeof(int)) - j - 1)); - *port_name_tag = ((i * 8 * sizeof(int)) + j); - goto fn_exit; - } - } - } - else { - goto fn_fail; - } - -fn_exit: - return mpi_errno; - -fn_fail: - /* Everything is used up */ - *port_name_tag = -1; - mpi_errno = MPI_ERR_OTHER; - goto fn_exit; -} - -static void MPIDI_free_port_name_tag(int tag) -{ - int index, rem_tag; - - index = tag / (sizeof(int) * 8); - rem_tag = tag - (index * sizeof(int) * 8); - - port_name_tag_mask[index] &= ~(1 << ((8 * sizeof(int)) - 1 - rem_tag)); -} - - -/* - * MPIDI_Open_port() - */ -static int MPIDI_Open_port(MPIR_Info *info_ptr, char *port_name) -{ - int mpi_errno = MPI_SUCCESS; - int str_errno = MPL_STR_SUCCESS; - int len; - int port_name_tag = 0; /* this tag is added to the business card, - which is then returned as the port name */ - int taskid_tag; - int myRank = MPIR_Process.comm_world->rank; - - mpi_errno = MPIDI_get_port_name_tag(&port_name_tag); - TRACE_ERR("MPIDI_get_port_name_tag - port_name_tag=%d mpi_errno=%d\n", port_name_tag, mpi_errno); - - len = MPI_MAX_PORT_NAME; - str_errno = MPL_str_add_int_arg(&port_name, &len, - MPIDI_PORT_NAME_TAG_KEY, port_name_tag); - /*len = MPI_MAX_TASKID_NAME;*/ - taskid_tag = PAMIX_Client_query(MPIDI_Client, PAMI_CLIENT_TASK_ID ).value.intval; - str_errno = MPL_str_add_int_arg(&port_name, &len, - MPIDI_TASKID_TAG_KEY, taskid_tag); - TRACE_ERR("MPL_str_add_int_arg - port_name=%s str_errno=%d\n", port_name, str_errno); - - /* This works because Get_business_card uses the same MPL_str_xxx - functions as above to add the business card to the input string */ - /* FIXME: We should instead ask the mpid_pg routines to give us - a connection string. There may need to be a separate step to - restrict us to a connection information that is only valid for - connections between processes that are started separately (e.g., - may not use shared memory). We may need a channel-specific - function to create an exportable connection string. */ - -fn_exit: - return mpi_errno; -fn_fail: - goto fn_exit; -} - -/* - * MPIDI_Close_port() - */ -static int MPIDI_Close_port(const char *port_name) -{ - int mpi_errno = MPI_SUCCESS; - int port_name_tag; - - mpi_errno = MPIDI_GetTagFromPort(port_name, &port_name_tag); - - MPIDI_free_port_name_tag(port_name_tag); - -fn_exit: - return mpi_errno; -fn_fail: - goto fn_exit; -} - -/* - * The connect and accept routines use this routine to get the port tag - * from the port name. - */ -int MPIDI_GetTagFromPort( const char *port_name, int *port_name_tag ) -{ - int mpi_errno = MPI_SUCCESS; - int str_errno = MPL_STR_SUCCESS; - - str_errno = MPL_str_get_int_arg(port_name, MPIDI_PORT_NAME_TAG_KEY, - port_name_tag); - - fn_exit: - return mpi_errno; - fn_fail: - goto fn_exit; -} - -/* - * The connect and accept routines use this routine to get the port tag - * from the port name. - */ -int MPIDI_GetTaskidFromPort( const char *port_name, int *taskid_tag ) -{ - int mpi_errno = MPI_SUCCESS; - int str_errno = MPL_STR_SUCCESS; - - str_errno = MPL_str_get_int_arg(port_name, MPIDI_TASKID_TAG_KEY, - taskid_tag); - - fn_exit: - return mpi_errno; - fn_fail: - goto fn_exit; -} -#endif diff --git a/src/mpid/pamid/src/dyntask/mpidi_pg.c b/src/mpid/pamid/src/dyntask/mpidi_pg.c deleted file mode 100644 index a344b6a..0000000 --- a/src/mpid/pamid/src/dyntask/mpidi_pg.c +++ /dev/null @@ -1,1063 +0,0 @@ -/* -*- Mode: C; c-basic-offset:4 ; -*- */ -/* - * (C) 2001 by Argonne National Laboratory. - * See COPYRIGHT in top-level directory. - */ -#include <mpidimpl.h> -#ifdef USE_PMI2_API -#include "pmi2.h" -#else -#include "pmi.h" -#endif - -#ifdef DYNAMIC_TASKING - -extern int mpidi_dynamic_tasking; -int mpidi_sync_done=0; - - -#define MAX_JOBID_LEN 1024 - -/* FIXME: These routines need a description. What is their purpose? Who - calls them and why? What does each one do? -*/ -static MPIDI_PG_t * MPIDI_PG_list = NULL; -static MPIDI_PG_t * MPIDI_PG_iterator_next = NULL; -static MPIDI_PG_Compare_ids_fn_t MPIDI_PG_Compare_ids_fn; -static MPIDI_PG_Destroy_fn_t MPIDI_PG_Destroy_fn; - -/* Set verbose to 1 to record changes to the process group structure. */ -static int verbose = 0; - -/* Key track of the process group corresponding to the MPI_COMM_WORLD - of this process */ -MPIDI_PG_t *pg_world = NULL; - -#define MPIDI_MAX_KVS_KEY_LEN 256 - -extern conn_info *_conn_info_list; - -int MPIDI_PG_Init(int *argc_p, char ***argv_p, - MPIDI_PG_Compare_ids_fn_t compare_ids_fn, - MPIDI_PG_Destroy_fn_t destroy_fn) -{ - int mpi_errno = MPI_SUCCESS; - char *p; - - MPIDI_PG_Compare_ids_fn = compare_ids_fn; - MPIDI_PG_Destroy_fn = destroy_fn; - - /* Check for debugging options. We use MPICHD_DBG and -mpichd-dbg - to avoid confusion with the code in src/util/dbg/dbg_printf.c */ - p = getenv( "MPICHD_DBG_PG" ); - if (p && ( strcmp( p, "YES" ) == 0 || strcmp( p, "yes" ) == 0) ) - verbose = 1; - if (argc_p && argv_p) { - int argc = *argc_p, i; - char **argv = *argv_p; - /* applied patch from Juha Jeronen, req #3920 */ - for (i=1; i<argc && argv[i]; i++) { - if (strcmp( "-mpichd-dbg-pg", argv[i] ) == 0) { - int j; - verbose = 1; - for (j=i; j<argc-1; j++) { - argv[j] = argv[j+1]; - } - argv[argc-1] = NULL; - *argc_p = argc - 1; - break; - } - } - } - - return mpi_errno; -} - -void *mpidi_finalize_req(void *arg) { - PMI2_Finalize(); - mpidi_sync_done=1; -} - -/*@ - MPIDI_PG_Finalize - Finalize the process groups, including freeing all - process group structures - @*/ -int MPIDI_PG_Finalize(void) -{ - int mpi_errno = MPI_SUCCESS; - conn_info *conn_node; - int my_max_worldid, world_max_worldid; - int wid_bit_array_size=0, wid; - unsigned char *wid_bit_array=NULL, *root_wid_barray=NULL; - MPIDI_PG_t *pg, *pgNext; - char key[PMI2_MAX_KEYLEN]; - char value[PMI2_MAX_VALLEN]; - pthread_t finalize_req_thread; - - /* Print the state of the process groups */ - if (verbose) { - MPIU_PG_Printall( stdout ); - } - - /* FIXME - straighten out the use of PMI_Finalize - no use after - PG_Finalize */ - conn_node = _conn_info_list; - my_max_worldid = -1; - - while(NULL != conn_node) { - if(conn_node->rem_world_id>my_max_worldid && conn_node->ref_count>0) { - TRACE_ERR("conn_node->rem_world_id=%d conn_node->ref_count=%d\n", conn_node->rem_world_id, conn_node->ref_count); - my_max_worldid = conn_node->rem_world_id; - } else { - TRACE_ERR("conn_node->rem_world_id=%d conn_node->ref_count=%d\n", conn_node->rem_world_id, conn_node->ref_count); - } - conn_node = conn_node->next; - } - MPIR_Allreduce_impl( &my_max_worldid, &world_max_worldid, 1, MPI_INT, MPI_MAX, MPIR_Process.comm_world, &mpi_errno); - - /* create bit array of size = world_max_worldid + 1 - * We add 1 to world_max_worldid because suppose my world - * is only connected to world_id 0 then world_max_worldid=0 - * and if we do not add 1, then size of bite array will be 0. - * Also suppose in my world world_max_worldid is 8. Then if we - * dont add 1, then the bit array will be size 1 byte, and when - * we try to set bit in position 8, we will get segfault. - */ - TRACE_ERR("my_max_worldid=%d world_max_worldid=%d\n", my_max_worldid, world_max_worldid); - if(world_max_worldid != -1) { - world_max_worldid++; - wid_bit_array_size = (world_max_worldid + CHAR_BIT -1) / CHAR_BIT; - wid_bit_array = MPL_malloc(wid_bit_array_size*sizeof(unsigned char)); - memset(wid_bit_array, 0, wid_bit_array_size*sizeof(unsigned char)); - root_wid_barray = MPL_malloc(wid_bit_array_size*sizeof(unsigned char)); - - memset(root_wid_barray, 0, wid_bit_array_size*sizeof(unsigned char)); - conn_node = _conn_info_list; - while(NULL != conn_node) { - if(conn_node->ref_count >0) { - wid = conn_node->rem_world_id; - wid_bit_array[wid/CHAR_BIT] |= 1 << (wid%CHAR_BIT); - TRACE_ERR("wid=%d wid_bit_array[%d]=%x\n", wid, wid/CHAR_BIT, 1 << (wid%CHAR_BIT)); - } - conn_node = conn_node->next; - - } - /* Let root of my world know about this bit array */ - MPIR_Reduce_impl(wid_bit_array,root_wid_barray,wid_bit_array_size, - MPI_UNSIGNED_CHAR,MPI_BOR,0,MPIR_Process.comm_world,&mpi_errno); - - MPL_free(wid_bit_array); - } - - if(MPIR_Process.comm_world->rank == 0) { - - MPL_snprintf(key, PMI2_MAX_KEYLEN-1, "%s", "ROOTWIDARRAY"); - MPL_snprintf(value, PMI2_MAX_VALLEN-1, "%s", root_wid_barray); - TRACE_ERR("root_wid_barray=%s\n", value); - mpi_errno = PMI2_KVS_Put(key, value); - TRACE_ERR("PMI2_KVS_Put returned with mpi_errno=%d\n", mpi_errno); - - MPL_snprintf(key, PMI2_MAX_KEYLEN-1, "%s", "WIDBITARRAYSZ"); - MPL_snprintf(value, PMI2_MAX_VALLEN-1, "%x", wid_bit_array_size); - key[strlen(key)+1]='\0'; - value[strlen(value)+1]='\0'; - mpi_errno = PMI2_KVS_Put(key, value); - TRACE_ERR("PMI2_KVS_Put returned with mpi_errno=%d\n", mpi_errno); - - } - mpi_errno = PMI2_KVS_Fence(); - TRACE_ERR("PMI2_KVS_Fence returned with mpi_errno=%d\n", mpi_errno); - - MPL_free(root_wid_barray); /* root_wid_barray is now NULL for non-root */ - -#if 0 - pthread_create(&finalize_req_thread, NULL, mpidi_finalize_req, NULL); - /*MPIU_THREAD_CS_EXIT(ALLFUNC,); */ - while (mpidi_sync_done !=1) { - mpi_errno=PAMI_Context_advance(MPIDI_Context[0], 1000); - if (mpi_errno == PAMI_EAGAIN) { - usleep(1); - } - } - - if (mpi_errno = pthread_join(finalize_req_thread, NULL) ) { - TRACE_ERR("error returned from pthread_join() mpi_errno=%d\n",mpi_errno); - } -#endif - MPIU_THREAD_CS_EXIT(ALLFUNC,); - PMI2_Finalize(); - MPIU_THREAD_CS_ENTER(ALLFUNC,); - - if(_conn_info_list) { - if(_conn_info_list->rem_taskids) - MPL_free(_conn_info_list->rem_taskids); - else - MPL_free(_conn_info_list); - } - /* Free the storage associated with the process groups */ - pg = MPIDI_PG_list; - while (pg) { - pgNext = pg->next; - - /* In finalize, we free all process group information, even if - the ref count is not zero. This can happen if the user - fails to use MPI_Comm_disconnect on communicators that - were created with the dynamic process routines.*/ - /* XXX DJG FIXME-MT should we be checking this? */ - if (MPIR_Object_get_ref(pg) == 0 ) { - if (pg == MPIDI_Process.my_pg) - MPIDI_Process.my_pg = NULL; - MPIR_Object_set_ref(pg, 0); /* satisfy assertions in PG_Destroy */ - MPIDI_PG_Destroy( pg ); - } - pg = pgNext; - } - - /* If COMM_WORLD is still around (it normally should be), - try to free it here. The reason that we need to free it at this - point is that comm_world (and comm_self) still exist, and - hence the usual process to free the related VC structures will - not be invoked. */ - - /* The process group associated with MPI_COMM_WORLD will be - freed when MPI_COMM_WORLD is freed */ - - return mpi_errno; -} - - -/* This routine creates a new process group description and appends it to - the list of the known process groups. The pg_id is saved, not copied. - The PG_Destroy routine that was set with MPIDI_PG_Init is responsible for - freeing any storage associated with the pg_id. - - The new process group is returned in pg_ptr -*/ -int MPIDI_PG_Create(int vct_sz, void * pg_id, MPIDI_PG_t ** pg_ptr) -{ - MPIDI_PG_t * pg = NULL, *pgnext; - int p, i, j; - int mpi_errno = MPI_SUCCESS; - char *cp, *world_tasks, *cp1; - - pg = MPL_malloc(sizeof(MPIDI_PG_t)); - pg->vct = MPL_malloc(sizeof(struct MPID_VCR_t)*vct_sz); - - pg->handle = 0; - /* The reference count indicates the number of vc's that are or - have been in use and not disconnected. It starts at zero, - except for MPI_COMM_WORLD. */ - MPIR_Object_set_ref(pg, 0); - pg->size = vct_sz; - pg->id = MPL_strdup(pg_id); - TRACE_ERR("PG_Create - pg=%x pg->id=%s pg->vct=%x\n", pg, pg->id, pg->vct); - /* Initialize the connection information to null. Use - the appropriate MPIDI_PG_InitConnXXX routine to set up these - fields */ - - pg->connData = 0; - pg->getConnInfo = 0; - pg->connInfoToString = 0; - pg->connInfoFromString = 0; - pg->freeConnInfo = 0; - - for (p = 0; p < vct_sz; p++) - { - /* Initialize device fields in the VC object */ - MPIDI_VC_Init(&pg->vct[p], pg,p); - } - - /* The first process group is always the world group */ - if (!pg_world) { pg_world = pg; } - - /* Add pg's at the tail so that comm world is always the first pg */ - pg->next = 0; - if (!MPIDI_PG_list) - { - MPIDI_PG_list = pg; - } - else - { - pgnext = MPIDI_PG_list; - while (pgnext->next) - { - pgnext = pgnext->next; - } - pgnext->next = pg; - } - /* These are now done in MPIDI_VC_Init */ - *pg_ptr = pg; - - fn_exit: - return mpi_errno; - - fn_fail: - goto fn_exit; -} - -int MPIDI_PG_Destroy(MPIDI_PG_t * pg) -{ - MPIDI_PG_t * pg_prev; - MPIDI_PG_t * pg_cur; - int i; - int mpi_errno = MPI_SUCCESS; - - MPIR_Assert(MPIR_Object_get_ref(pg) == 0); - - pg_prev = NULL; - pg_cur = MPIDI_PG_list; - - while(pg_cur != NULL) - { - if (pg_cur == pg) - { - if (MPIDI_PG_iterator_next == pg) - { - MPIDI_PG_iterator_next = MPIDI_PG_iterator_next->next; - } - - if (pg_prev == NULL) - MPIDI_PG_list = pg->next; - else - pg_prev->next = pg->next; - - TRACE_ERR("destroying pg=%p pg->id=%s\n", pg, (char *)pg->id); - - for (i = 0; i < pg->size; ++i) { - /* FIXME it would be good if we could make this assertion. - Unfortunately, either: - 1) We're not being disciplined and some caller of this - function doesn't bother to manage all the refcounts - because he thinks he knows better. Annoying, but not - strictly a bug. - (wdg - actually, that is a bug - managing the ref - counts IS required and missing one is a bug.) - 2) There is a real bug lurking out there somewhere and we - just haven't hit it in the tests yet. */ - - /* This used to be handled in MPID_VCRT_Release, but that was - not the right place to do this. The VC should only be freed - when the PG that it belongs to is freed, not just when the - VC's refcount drops to zero. [goodell@ 2008-06-13] */ - /* In that case, the fact that the VC is in the PG should - increment the ref count - reflecting the fact that the - use in the PG constitutes a reference-count-incrementing - use. Alternately, if the PG is able to recreate a VC, - and can thus free unused (or idle) VCs, it should be allowed - to do so. [wdg 2008-08-31] */ - } - - MPIDI_PG_Destroy_fn(pg); - TRACE_ERR("destroying pg->vct=%x\n", pg->vct); - MPL_free(pg->vct); - TRACE_ERR("after destroying pg->vct=%x\n", pg->vct); - - if (pg->connData) { - if (pg->freeConnInfo) { - TRACE_ERR("calling freeConnInfo on pg\n"); - (*pg->freeConnInfo)( pg ); - } - else { - TRACE_ERR("free pg->connData\n"); - MPL_free(pg->connData); - } - } - - TRACE_ERR("final destroying pg\n"); - MPL_free(pg); - - goto fn_exit; - } - - pg_prev = pg_cur; - pg_cur = pg_cur->next; - } - - fn_exit: - return mpi_errno; - fn_fail: - goto fn_exit; -} - -int MPIDI_PG_Find(void * id, MPIDI_PG_t ** pg_ptr) -{ - MPIDI_PG_t * pg; - int mpi_errno = MPI_SUCCESS; - - pg = MPIDI_PG_list; - - while (pg != NULL) - { - if (MPIDI_PG_Compare_ids_fn(id, pg->id) != FALSE) - { - *pg_ptr = pg; - goto fn_exit; - } - - pg = pg->next; - } - - *pg_ptr = NULL; - - fn_exit: - return mpi_errno; -} - - -int MPIDI_PG_Id_compare(void * id1, void *id2) -{ - return MPIDI_PG_Compare_ids_fn(id1, id2); -} - -/* iter always points at the next element */ -int MPIDI_PG_Get_next(MPIDI_PG_iterator *iter, MPIDI_PG_t ** pg_ptr) -{ - *pg_ptr = (*iter); - if ((*iter) != NULL) { - (*iter) = (*iter)->next; - } - - return MPI_SUCCESS; -} - -int MPIDI_PG_Has_next(MPIDI_PG_iterator *iter) -{ - return (*iter != NULL); -} - -int MPIDI_PG_Get_iterator(MPIDI_PG_iterator *iter) -{ - *iter = MPIDI_PG_list; - return MPI_SUCCESS; -} - -/* FIXME: What does DEV_IMPLEMENTS_KVS mean? Why is it used? Who uses - PG_To_string and why? */ - -/* PG_To_string is used in the implementation of connect/accept (and - hence in spawn) */ -/* Note: Allocated memory that is returned in str_ptr. The user of - this routine must free that data */ -int MPIDI_PG_To_string(MPIDI_PG_t *pg_ptr, char **str_ptr, int *lenStr) -{ - int mpi_errno = MPI_SUCCESS; - - /* Replace this with the new string */ - MPIDI_connToStringKVS( str_ptr, lenStr, pg_ptr ); -#if 0 - if (pg_ptr->connInfoToString) { - (*pg_ptr->connInfoToString)( str_ptr, lenStr, pg_ptr ); - } - else { - MPIU_ERR_SETANDJUMP(mpi_errno,MPI_ERR_INTERN,"**noConnInfoToString"); - } -#endif - -fn_exit: - return mpi_errno; -fn_fail: - goto fn_exit; -} - -/* This routine takes a string description of a process group (created with - MPIDI_PG_To_string, usually on a different process) and returns a pointer to - the matching process group. If the group already exists, flag is set to - false. If the group does not exist, it is created with MPIDI_PG_Create (and - hence is added to the list of active process groups) and flag is set to - true. In addition, the connection information is set up using the - information in the input string. -*/ -int MPIDI_PG_Create_from_string(const char * str, MPIDI_PG_t ** pg_pptr, - int *flag) -{ - int mpi_errno = MPI_SUCCESS; - const char *p; - char *pg_id, *pg_id2, *cp2, *cp3,*str2, *str3; - int vct_sz, i; - MPIDI_PG_t *existing_pg, *pg_ptr=0; - - TRACE_ERR("MPIDI_PG_Create_from_string - str=%s\n", str); - /* The pg_id is at the beginning of the string, so we can just pass - it to the find routine */ - /* printf( "Looking for pg with id %s\n", str );fflush(stdout); */ - mpi_errno = MPIDI_PG_Find((void *)str, &existing_pg); - if (mpi_errno) TRACE_ERR("MPIDI_PG_Find returned with mpi_errno=%d\n", mpi_errno); - - if (existing_pg != NULL) { - /* return the existing PG */ - *pg_pptr = existing_pg; - *flag = 0; - /* Note that the memory for the pg_id is freed in the exit */ - goto fn_exit; - } - *flag = 1; - - /* Get the size from the string */ - p = str; - while (*p) p++; p++; - vct_sz = atoi(p); - - while (*p) p++;p++; - char *p_tmp = MPL_strdup(p); - TRACE_ERR("before MPIDI_PG_Create - p=%s p_tmp=%s vct_sz=%d\n", p, p_tmp, vct_sz); - mpi_errno = MPIDI_PG_Create(vct_sz, (void *)str, pg_pptr); - if (mpi_errno != MPI_SUCCESS) { - TRACE_ERR("MPIDI_PG_Create returned with mpi_errno=%d\n", mpi_errno); - } - - pg_ptr = *pg_pptr; - pg_ptr->vct[0].taskid=atoi(strtok(p_tmp,":")); - for(i=1; i<vct_sz; i++) { - pg_ptr->vct[i].taskid=atoi(strtok(NULL,":")); - } - TRACE_ERR("pg_ptr->id = %s\n",(*pg_pptr)->id); - MPL_free(p_tmp); - - if(verbose) - MPIU_PG_Printall(stderr); - -fn_exit: - return mpi_errno; -fn_fail: - goto fn_exit; -} - -#ifdef HAVE_CTYPE_H -/* Needed for isdigit */ -#include <ctype.h> -#endif - - -/* For all of these routines, the format of the process group description - that is created and used by the connTo/FromString routines is this: - (All items are strings, terminated by null) - - process group id string - sizeof process group (as string) - conninfo for rank 0 - conninfo for rank 1 - ... - - The "conninfo for rank 0" etc. for the original (MPI_COMM_WORLD) - process group are stored in the PMI_KVS space with the keys - p<rank>-businesscard . - - Fixme: Add a routine to publish the connection info to this file so that - the key for the businesscard is defined in just this one file. -*/ - - -/* The "KVS" versions are for the process group to which the calling - process belongs. These use the PMI_KVS routines to access the - process information */ -static int MPIDI_getConnInfoKVS( int rank, char *buf, int bufsize, MPIDI_PG_t *pg ) -{ -#ifdef USE_PMI2_API - char key[MPIDI_MAX_KVS_KEY_LEN]; - int mpi_errno = MPI_SUCCESS, rc; - int vallen; - - rc = MPL_snprintf(key, MPIDI_MAX_KVS_KEY_LEN, "P%d-businesscard", rank ); - - mpi_errno = PMI2_KVS_Get(pg->connData, PMI2_ID_NULL, key, buf, bufsize, &vallen); - if (mpi_errno) { - TRACE_ERR("PMI2_KVS_Get returned with mpi_errno=%d\n", mpi_errno); - } - fn_exit: - return mpi_errno; - fn_fail: - goto fn_exit; -#else - char key[MPIDI_MAX_KVS_KEY_LEN]; - int mpi_errno = MPI_SUCCESS, rc, pmi_errno; - - rc = MPL_snprintf(key, MPIDI_MAX_KVS_KEY_LEN, "P%d-businesscard", rank ); - if (rc < 0 || rc > MPIDI_MAX_KVS_KEY_LEN) { - MPIU_ERR_SETANDJUMP(mpi_errno,MPI_ERR_OTHER,"**nomem"); - } - -/* MPIU_THREAD_CS_ENTER(PMI,);*/ - pmi_errno = PMI_KVS_Get(pg->connData, key, buf, bufsize ); - if (pmi_errno) { - MPIDI_PG_CheckForSingleton(); - pmi_errno = PMI_KVS_Get(pg->connData, key, buf, bufsize ); - } -/* MPIU_THREAD_CS_EXIT(PMI,);*/ - if (pmi_errno) { - MPIU_ERR_SETANDJUMP(mpi_errno,MPI_ERR_OTHER,"**pmi_kvs_get"); - } - - fn_exit: - return mpi_errno; - fn_fail: - goto fn_exit; -#endif -} - -/* *slen is the length of the string, including the null terminator. So if the - resulting string is |foo\0bar\0|, then *slen == 8. */ -int MPIDI_connToStringKVS( char **buf_p, int *slen, MPIDI_PG_t *pg ) -{ - char *string = 0; - char *pg_idStr = (char *)pg->id; /* In the PMI/KVS space, - the pg id is a string */ - char buf[MPIDI_MAX_KVS_VALUE_LEN]; - int i, j, vallen, rc, mpi_errno = MPI_SUCCESS, len; - int curSlen, nChars; - - /* Make an initial allocation of a string with an estimate of the - needed space */ - len = 0; - curSlen = 10 + pg->size * 128; - string = (char *)MPL_malloc( curSlen ); - - /* Start with the id of the pg */ - while (*pg_idStr && len < curSlen) - string[len++] = *pg_idStr++; - string[len++] = 0; - - /* Add the size of the pg */ - MPL_snprintf( &string[len], curSlen - len, "%d", pg->size ); - while (string[len]) len++; - string[len++] = 0; - - /* add the taskids of the pg */ - for(i = 0; i < pg->size; i++) { - MPL_snprintf(buf, MPIDI_MAX_KVS_VALUE_LEN, "%d:", pg->vct[i].taskid); - vallen = strlen(buf); - if (len+vallen+1 >= curSlen) { - char *nstring = 0; - curSlen += (pg->size - i) * (vallen + 1 ); - nstring = MPL_realloc( string, curSlen); - MPID_assert(nstring != NULL); - string = nstring; - } - /* Append to string */ - nChars = MPL_snprintf(&string[len], curSlen - len, "%d:", pg->vct[i].taskid); - len+=nChars; - } - -#if 0 - for (i=0; i<pg->size; i++) { - rc = getConnInfoKVS( i, buf, MPIDI_MAX_KVS_VALUE_LEN, pg ); - if (rc) { - MPL_internal_error_printf( - "Panic: getConnInfoKVS failed for %s (rc=%d)\n", - (char *)pg->id, rc ); - } -#ifndef USE_PERSISTENT_SHARED_MEMORY - /* FIXME: This is a hack to avoid including shared-memory - queue names in the business card that may be used - by processes that were not part of the same COMM_WORLD. - To fix this, the shared memory channels should look at the - returned connection info and decide whether to use - sockets or shared memory by determining whether the - process is in the same MPI_COMM_WORLD. */ - /* FIXME: The more general problem is that the connection information - needs to include some information on the range of validity (e.g., - all processes, same comm world, particular ranks), and that - representation needs to be scalable */ -/* printf( "Adding key %s value %s\n", key, val ); */ - { - char *p = strstr( buf, "$shm_host" ); - if (p) p[1] = 0; - /* printf( "(fixed) Adding key %s value %s\n", key, val ); */ - } -#endif - /* Add the information to the output buffer */ - vallen = strlen(buf); - /* Check that this will fix in the remaining space */ - if (len + vallen + 1 >= curSlen) { - char *nstring = 0; - curSlen += (pg->size - i) * (vallen + 1 ); - nstring = MPL_realloc( string, curSlen); - if (!nstring) { - MPIU_ERR_SETANDJUMP(mpi_errno,MPI_ERR_OTHER,"**nomem"); - } - string = nstring; - } - /* Append to string */ - for (j=0; j<vallen+1; j++) { - string[len++] = buf[j]; - } - } -#endif - - MPIR_Assert(len <= curSlen); - - *buf_p = string; - *slen = len; - fn_exit: - return mpi_errno; - fn_fail: - if (string) MPL_free(string); - goto fn_exit; -} - -static int MPIDI_connFromStringKVS( const char *buf ATTRIBUTE((unused)), - MPIDI_PG_t *pg ATTRIBUTE((unused)) ) -{ - /* Fixme: this should be a failure to call this routine */ - return MPI_SUCCESS; -} -static int MPIDI_connFreeKVS( MPIDI_PG_t *pg ) -{ - if (pg->connData) { - MPL_free( pg->connData ); - } - return MPI_SUCCESS; -} - - -int MPIDI_PG_InitConnKVS( MPIDI_PG_t *pg ) -{ -#ifdef USE_PMI2_API - int mpi_errno = MPI_SUCCESS; - - pg->connData = (char *)MPL_malloc(MAX_JOBID_LEN); - if (pg->connData == NULL) { - TRACE_ERR("MPIDI_PG_InitConnKVS - MPL_malloc failure\n"); - } - - mpi_errno = PMI2_Job_GetId(pg->connData, MAX_JOBID_LEN); - if (mpi_errno) TRACE_ERR("PMI2_Job_GetId returned with mpi_errno=%d\n", mpi_errno); -#else - int pmi_errno, kvs_name_sz; - int mpi_errno = MPI_SUCCESS; - - pmi_errno = PMI_KVS_Get_name_length_max( &kvs_name_sz ); - if (pmi_errno != PMI_SUCCESS) { - MPIU_ERR_SETANDJUMP1(mpi_errno,MPI_ERR_OTHER, - "**pmi_kvs_get_name_length_max", - "**pmi_kvs_get_name_length_max %d", pmi_errno); - } - - pg->connData = (char *)MPL_malloc(kvs_name_sz + 1); - if (pg->connData == NULL) { - MPIU_ERR_SETANDJUMP(mpi_errno,MPI_ERR_OTHER, "**nomem"); - } - - pmi_errno = PMI_KVS_Get_my_name(pg->connData, kvs_name_sz); - if (pmi_errno != PMI_SUCCESS) { - MPIU_ERR_SETANDJUMP1(mpi_errno,MPI_ERR_OTHER, - "**pmi_kvs_get_my_name", - "**pmi_kvs_get_my_name %d", pmi_errno); - } -#endif - pg->getConnInfo = MPIDI_getConnInfoKVS; - pg->connInfoToString = MPIDI_connToStringKVS; - pg->connInfoFromString = MPIDI_connFromStringKVS; - pg->freeConnInfo = MPIDI_connFreeKVS; - - fn_exit: - return mpi_errno; - fn_fail: - if (pg->connData) { MPL_free(pg->connData); } - goto fn_exit; -} - -/* Return the kvsname associated with the MPI_COMM_WORLD of this process. */ -int MPIDI_PG_GetConnKVSname( char ** kvsname ) -{ - *kvsname = pg_world->connData; - return MPI_SUCCESS; -} - -/* For process groups that are not our MPI_COMM_WORLD, store the connection - information in an array of strings. These routines and structure - implement the access to this information. */ -typedef struct { - int toStringLen; /* Length needed to encode this connection info */ - char ** connStrings; /* pointer to an array, indexed by rank, containing - connection information */ -} MPIDI_ConnInfo; - - -static int MPIDI_getConnInfo( int rank, char *buf, int bufsize, MPIDI_PG_t *pg ) -{ - MPIDI_ConnInfo *connInfo = (MPIDI_ConnInfo *)pg->connData; - - /* printf( "Entering getConnInfo\n" ); fflush(stdout); */ - if (!connInfo || !connInfo->connStrings || !connInfo->connStrings[rank]) { - /* FIXME: Turn this into a valid error code create/return */ - /*printf( "Fatal error in getConnInfo (rank = %d)\n", rank ); - printf( "connInfo = %p\n", connInfo );fflush(stdout); */ - if (connInfo) { -/* printf( "connInfo->connStrings = %p\n", connInfo->connStrings ); */ - } - /* Fatal error. Connection information missing */ - fflush(stdout); - } - - /* printf( "Copying %s to buf\n", connInfo->connStrings[rank] ); fflush(stdout); */ - - MPL_strncpy( buf, connInfo->connStrings[rank], bufsize ); - return MPI_SUCCESS; -} - - -static int MPIDI_connToString( char **buf_p, int *slen, MPIDI_PG_t *pg ) -{ - int mpi_errno = MPI_SUCCESS; - char *str = NULL, *pg_id; - int i, len=0; - MPIDI_ConnInfo *connInfo = (MPIDI_ConnInfo *)pg->connData; - - /* Create this from the string array */ - str = (char *)MPL_malloc(connInfo->toStringLen); - -#if defined(MPICH_DEBUG_MEMINIT) - memset(str, 0, connInfo->toStringLen); -#endif - - pg_id = pg->id; - /* FIXME: This is a hack, and it doesn't even work */ - /* MPIDI_PrintConnStrToFile( stdout, __FILE__, __LINE__, - "connToString: pg id is", (char *)pg_id );*/ - /* This is intended to cause a process to transition from a singleton - to a non-singleton. */ - /* XXX DJG TODO figure out what this little bit is all about. */ - if (strstr( pg_id, "singinit_kvs" ) == pg_id) { -#ifdef USE_PMI2_API - MPIR_Assertp(0); /* don't know what to do here for pmi2 yet. DARIUS */ -#else - PMI_KVS_Get_my_name( pg->id, 256 ); -#endif - } - - while (*pg_id) str[len++] = *pg_id++; - str[len++] = 0; - - MPL_snprintf( &str[len], 20, "%d", pg->size); - /* Skip over the length */ - while (str[len++]); - - /* Copy each connection string */ - for (i=0; i<pg->size; i++) { - char *p = connInfo->connStrings[i]; - while (*p) { str[len++] = *p++; } - str[len++] = 0; - } - - if (len > connInfo->toStringLen) { - *buf_p = 0; - *slen = 0; - TRACE_ERR("len > connInfo->toStringLen"); - } - - *buf_p = str; - *slen = len; - -fn_exit: - return mpi_errno; -fn_fail: - goto fn_exit; - -} - - -static int MPIDI_connFromString( const char *buf, MPIDI_PG_t *pg ) -{ - MPIDI_ConnInfo *conninfo = 0; - int i, mpi_errno = MPI_SUCCESS; - const char *buf0 = buf; /* save the start of buf */ - - /* printf( "Starting with buf = %s\n", buf );fflush(stdout); */ - - /* Skip the pg id */ - while (*buf) buf++; buf++; - - /* Determine the size of the pg */ - pg->size = atoi( buf ); - while (*buf) buf++; buf++; - - conninfo = (MPIDI_ConnInfo *)MPL_malloc( sizeof(MPIDI_ConnInfo) ); - conninfo->connStrings = (char **)MPL_malloc( pg->size * sizeof(char *)); - - /* For now, make a copy of each item */ - for (i=0; i<pg->size; i++) { - /* printf( "Adding conn[%d] = %s\n", i, buf );fflush(stdout); */ - conninfo->connStrings[i] = MPL_strdup( buf ); - while (*buf) buf++; - buf++; - } - pg->connData = conninfo; - - /* Save the length of the string needed to encode the connection - information */ - conninfo->toStringLen = (int)(buf - buf0) + 1; - - return mpi_errno; -} - - -static int MPIDI_connFree( MPIDI_PG_t *pg ) -{ - MPIDI_ConnInfo *conninfo = (MPIDI_ConnInfo *)pg->connData; - int i; - - for (i=0; i<pg->size; i++) { - MPL_free( conninfo->connStrings[i] ); - } - MPL_free( conninfo->connStrings ); - MPL_free( conninfo ); - - return MPI_SUCCESS; -} - - -/*@ - MPIDI_PG_Dup_vcr - Duplicate a virtual connection from a process group - - Notes: - This routine provides a dup of a virtual connection given a process group - and a rank in that group. This routine is used only in initializing - the MPI-1 communicators 'MPI_COMM_WORLD' and 'MPI_COMM_SELF', and in creating - the initial intercommunicator after an 'MPI_Comm_spawn', - 'MPI_Comm_spawn_multiple', or 'MPI_Comm_connect/MPI_Comm_accept'. - - In addition to returning a dup of the virtual connection, it manages the - reference count of the process group, which is always the number of inuse - virtual connections. - @*/ -int MPIDI_PG_Dup_vcr( MPIDI_PG_t *pg, int rank, pami_task_t taskid, MPID_VCR *vcr_p ) -{ - int inuse; - MPID_VCR vcr; - - TRACE_ERR("ENTER MPIDI_PG_Dup_vcr - pg->id=%s rank=%d taskid=%d\n", pg->id, rank, taskid); - pg->vct[rank].taskid = taskid; - - vcr = MPL_malloc(sizeof(struct MPID_VCR_t)); - TRACE_ERR("MPIDI_PG_Dup_vcr- pg->vct[%d].pg=%x pg=%x vcr=%x vcr->pg=%x\n", rank, pg->vct[rank].pg, pg, vcr, vcr->pg); - vcr->pg = pg; - vcr->pg_rank = rank; - vcr->taskid = taskid; - /* Increase the reference count of the vc. If the reference count - increases from 0 to 1, increase the reference count of the - process group *and* the reference count of the vc (this - allows us to distinquish between Comm_free and Comm_disconnect) */ - /* FIXME-MT: This should be a fetch and increment for thread-safety */ - /*if (MPIR_Object_get_ref(vcr_p) == 0) { */ - TRACE_ERR("MPIDI_PG_add_ref on pg=%s pg=%x\n", pg->id, pg); - MPIDI_PG_add_ref(pg); - inuse=MPIR_Object_get_ref(pg); - TRACE_ERR("after MPIDI_PG_add_ref on pg=%s inuse=%d\n", pg->id, inuse); -/* MPIDI_VC_add_ref(vcr_p); - } - MPIDI_VC_add_ref(vcr_p);*/ - *vcr_p = vcr; - - return MPI_SUCCESS; -} - - -/* - * This routine may be called to print the contents (including states and - * reference counts) for process groups. - */ -int MPIU_PG_Printall( FILE *fp ) -{ - MPIDI_PG_t *pg; - int i; - - pg = MPIDI_PG_list; - - fprintf( fp, "Process groups:\n" ); - while (pg) { - /* XXX DJG FIXME-MT should we be checking this? */ - fprintf( fp, "size = %d, refcount = %d, id = %s\n", - pg->size, MPIR_Object_get_ref(pg), (char *)pg->id ); - for (i=0; i<pg->size; i++) { - fprintf( fp, "\tVCT rank = %d, refcount = %d, taskid = %d\n", - pg->vct[i].pg_rank, MPIR_Object_get_ref(pg), - pg->vct[i].taskid ); - } - fflush(fp); - pg = pg->next; - } - - return 0; -} - -#ifdef HAVE_CTYPE_H -/* Needed for isdigit */ -#include <ctype.h> -#endif -/* Convert a process group id into a number. This is a hash-based approach, - * which has the potential for some collisions. This is an alternative to the - * previous approach that caused req#3930, which was to sum up the values of the - * characters. The summing approach worked OK when the id's were all similar - * but with an incrementing prefix or suffix, but terrible for a 32 hex-character - * UUID type of id. - * - * FIXME It would really be best if the PM could give us this value. - */ -/* FIXME: This is a temporary hack for devices that do not define - MPIDI_DEV_IMPLEMENTS_KVS - FIXME: MPIDI_DEV_IMPLEMENTS_KVS should be removed - */ -void MPIDI_PG_IdToNum( MPIDI_PG_t *pg, int *id ) -{ - *id = atoi((char *)pg->id); -} - - -int MPID_PG_ForwardPGInfo( MPIR_Comm *peer_ptr, MPIR_Comm *comm_ptr, - int nPGids, const int gpids[], - int root ) -{ - int mpi_errno = MPI_SUCCESS; - int i, allfound = 1, pgid, pgidWorld; - MPIDI_PG_t *pg = 0; - MPIDI_PG_iterator iter; - MPIR_Errflag_t errflag = MPIR_ERR_NONE; - - if(mpidi_dynamic_tasking) { - /* Get the pgid for CommWorld (always attached to the first process - group) */ - MPIDI_PG_Get_iterator(&iter); - MPIDI_PG_Get_next( &iter, &pg ); - MPIDI_PG_IdToNum( pg, &pgidWorld ); - - /* Extract the unique process groups */ - for (i=0; i<nPGids && allfound; i++) { - if (gpids[0] != pgidWorld) { - /* Add this gpid to the list of values to check */ - /* FIXME: For testing, we just test in place */ - MPIDI_PG_Get_iterator(&iter); - do { - MPIDI_PG_Get_next( &iter, &pg ); - if (!pg) { - /* We don't know this pgid */ - allfound = 0; - break; - } - MPIDI_PG_IdToNum( pg, &pgid ); - } while (pgid != gpids[0]); - } - gpids += 2; - } - - /* See if everyone is happy */ - mpi_errno = MPIR_Allreduce_impl( MPI_IN_PLACE, &allfound, 1, MPI_INT, MPI_LAND, comm_ptr, &errflag ); - - if (allfound) return MPI_SUCCESS; - - /* FIXME: We need a cleaner way to handle this case than using an ifdef. - We could have an empty version of MPID_PG_BCast in ch3u_port.c, but - that's a rather crude way of addressing this problem. Better is to - make the handling of local and remote PIDS for the dynamic process - case part of the dynamic process "module"; devices that don't support - dynamic processes (and hence have only COMM_WORLD) could optimize for - that case */ - /* We need to share the process groups. We use routines - from ch3u_port.c */ - MPID_PG_BCast( peer_ptr, comm_ptr, root ); - } - fn_exit: - return MPI_SUCCESS; - fn_fail: - goto fn_exit; -} -#endif diff --git a/src/mpid/pamid/src/dyntask/mpidi_port.c b/src/mpid/pamid/src/dyntask/mpidi_port.c deleted file mode 100644 index d539323..0000000 --- a/src/mpid/pamid/src/dyntask/mpidi_port.c +++ /dev/null @@ -1,1766 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ - -#include <mpidimpl.h> - -#ifdef DYNAMIC_TASKING -#define MAX_HOST_DESCRIPTION_LEN 256 -#define WORLDINTCOMMCNTR _global_world_intercomm_cntr -#ifdef USE_PMI2_API -#define MPID_MAX_JOBID_LEN 256 -#define TOTAL_AM 3 -#endif - -transactionID_struct *_transactionID_list = NULL; - -typedef struct { - MPID_VCR vcr; - int port_name_tag; -}AM_struct; - -conn_info *_conn_info_list = NULL; -extern int mpidi_dynamic_tasking; -long long _global_world_intercomm_cntr = 0; - -typedef struct MPIDI_Acceptq -{ - int port_name_tag; - MPID_VCR vcr; - struct MPIDI_Acceptq *next; -} -MPIDI_Acceptq_t; - -static MPIDI_Acceptq_t * acceptq_head=0; -static int maxAcceptQueueSize = 0; -static int AcceptQueueSize = 0; - -pthread_mutex_t rem_connlist_mutex = PTHREAD_MUTEX_INITIALIZER; - -/* FIXME: If dynamic processes are not supported, this file will contain - no code and some compilers may warn about an "empty translation unit" */ - -/* FIXME: pg_translation is used for ? */ -typedef struct pg_translation { - int pg_index; /* index of a process group (index in pg_node) */ - int pg_rank; /* rank in that process group */ - pami_task_t pg_taskid; /* rank in that process group */ -} pg_translation; - - -typedef struct pg_node { - int index; /* Internal index of process group - (see pg_translation) */ - char *pg_id; - char *str; /* String describing connection info for pg */ - int lenStr; /* Length of this string (including the null terminator(s)) */ - struct pg_node *next; -} pg_node; - - -void MPIDI_Recvfrom_remote_world(pami_context_t context, - void * cookie, - const void * _msginfo, - size_t msginfo_size, - const void * sndbuf, - size_t sndlen, - pami_endpoint_t sender, - pami_recv_t * recv) -{ - AM_struct *AM_data; - MPID_VCR *new_vcr; - int port_name_tag; - MPIDI_Acceptq_t *q_item; - pami_endpoint_t dest; - - - q_item = MPL_malloc(sizeof(MPIDI_Acceptq_t)); - q_item->vcr = MPL_malloc(sizeof(struct MPID_VCR_t)); - q_item->vcr->pg = MPL_malloc(sizeof(MPIDI_PG_t)); - MPIR_Object_set_ref(q_item->vcr->pg, 0); - TRACE_ERR("ENTER MPIDI_Acceptq_enqueue-1 q_item=%llx _msginfo=%llx (AM_struct *)_msginfo=%llx ((AM_struct *)_msginfo)->vcr=%llx\n", q_item, _msginfo, (AM_struct *)_msginfo, ((AM_struct *)_msginfo)->vcr); - q_item->port_name_tag = ((AM_struct *)_msginfo)->port_name_tag; - q_item->vcr->taskid = PAMIX_Endpoint_query(sender); - TRACE_ERR("MPIDI_Recvfrom_remote_world INVOKED with new_vcr->taskid=%d\n",sender); - - /* Keep some statistics on the accept queue */ - AcceptQueueSize++; - if (AcceptQueueSize > maxAcceptQueueSize) - maxAcceptQueueSize = AcceptQueueSize; - - q_item->next = acceptq_head; - acceptq_head = q_item; - return; -} - - -/* These functions help implement the connect/accept algorithm */ -static int MPIDI_ExtractLocalPGInfo( struct MPIR_Comm *, pg_translation [], - pg_node **, int * ); -static int MPIDI_ReceivePGAndDistribute( struct MPIR_Comm *, struct MPIR_Comm *, int, int *, - int, MPIDI_PG_t *[] ); -static int MPIDI_SendPGtoPeerAndFree( struct MPIR_Comm *, int *, pg_node * ); -static int MPIDI_SetupNewIntercomm( struct MPIR_Comm *comm_ptr, int remote_comm_size, - pg_translation remote_translation[], - int n_remote_pgs, MPIDI_PG_t **remote_pg, - struct MPIR_Comm *intercomm ); -static int MPIDI_Initialize_tmp_comm(struct MPIR_Comm **comm_pptr, - struct MPID_VCR_t *vcr_ptr, int is_low_group, int context_id_offset); - - -/* ------------------------------------------------------------------------- */ -/* - * Structure of this file and the connect/accept algorithm: - * - * Here are the steps involved in implementating MPI_Comm_connect and - * MPI_Comm_accept. These same steps are used withing MPI_Comm_spawn - * and MPI_Comm_spawn_multiple. - * - * First, the connecting process establishes a connection (not a virtual - * connection!) to the designated accepting process. - * This makes use of the usual (channel-specific) connection code. - * Once this connection is established, the connecting process sends a packet - * to the accepting process. - * This packet contains a "port_tag_name", which is a value that - * is used to separate different MPI port names (values from MPI_Open_port) - * on the same process (this is a way to multiplex many MPI port names on - * a single communication connection port). - * - * On the accepting side, the process waits until the progress engine - * inserts the connect request into the accept queue (this is done with the - * routine MPIDI_Acceptq_dequeue). This routine returns the matched - * virtual connection (VC). - * - * Once both sides have established there VC, they both invoke - * MPIDI_Initialize_tmp_comm to create a temporary intercommunicator. - * A temporary intercommunicator is constructed so that we can use - * MPI routines to send the other information that we need to complete - * the connect/accept operation (described below). - * - * The above is implemented with the routines - * MPIDI_Create_inter_root_communicator_connect - * MPIDI_Create_inter_root_communicator_accept - * MPIDI_Initialize_tmp_comm - * - * At this point, the two "root" processes of the communicators that are - * connecting can use MPI communication. They must then exchange the - * following information: - * - * The size of the "remote" communicator - * Description of all process groups; that is, all of the MPI_COMM_WORLDs - * that they know. - * The shared context id that will be used - * - * - */ -/* ------------------------------------------------------------------------- */ - -int MPIDU_send_AM_to_leader(MPID_VCR new_vcr, int port_name_tag, pami_task_t taskid) -{ - pami_send_t xferP; - pami_endpoint_t dest; - int rc, current_val; - - AM_struct AM_data; - - AM_data.vcr = new_vcr; - TRACE_ERR("MPIDU_send_AM_to_leader - new_vcr->taskid=%d\n", new_vcr->taskid); - AM_data.port_name_tag = port_name_tag; - TRACE_ERR("send - %p %d %p %d\n", AM_data.vcr, AM_data.port_name_tag, AM_data.vcr, AM_data.vcr->taskid); - - - memset(&xferP, 0, sizeof(pami_send_t)); - xferP.send.header.iov_base = (void*)&AM_data; - xferP.send.header.iov_len = sizeof(AM_struct); - xferP.send.dispatch = MPIDI_Protocols_Dyntask; - /*xferP.hints.use_rdma = mpci_enviro.use_shmem; - xferP.hints.use_shmem = mpci_enviro.use_shmem;*/ - rc = PAMI_Endpoint_create(MPIDI_Client, taskid, 0, &dest); - TRACE_ERR("PAMI_Resume to taskid=%d\n", taskid); - PAMI_Resume(MPIDI_Context[0], - &dest, 1); - - if(rc != 0) - TRACE_ERR("PAMI_Endpoint_create failed\n"); - - xferP.send.dest = dest; - - rc = PAMI_Send(MPIDI_Context[0], &xferP); - -} - - -/* - * These next two routines are used to create a virtual connection - * (VC) and a temporary intercommunicator that can be used to - * communicate between the two "root" processes for the - * connect and accept. - */ - -/* FIXME: Describe the algorithm for the connection logic */ -int MPIDI_Connect_to_root(const char * port_name, - MPID_VCR * new_vc) -{ - int mpi_errno = MPI_SUCCESS; - MPID_VCR vc; - char host_description[MAX_HOST_DESCRIPTION_LEN]; - int port, port_name_tag; pami_task_t taskid_tag; - int hasIfaddr = 0; - AM_struct *conn; - - /* First, create a new vc (we may use this to pass to a generic - connection routine) */ - vc = MPL_malloc(sizeof(struct MPID_VCR_t)); - vc->pg = MPL_malloc(sizeof(MPIDI_PG_t)); - MPIR_Object_set_ref(vc->pg, 0); - TRACE_ERR("vc from MPIDI_Connect_to_root=%llx vc->pg=%llx\n", vc, vc->pg); - /* FIXME - where does this vc get freed? */ - - *new_vc = vc; - - /* FIXME: There may need to be an additional routine here, to ensure that the - channel is initialized for this pair of process groups (this process - and the remote process to which the vc will connect). */ -/* MPIDI_VC_Init(vc->vc, NULL, 0); */ - vc->taskid = PAMIX_Client_query(MPIDI_Client, PAMI_CLIENT_TASK_ID ).value.intval; - TRACE_ERR("MPIDI_Connect_to_root - vc->taskid=%d\n", vc->taskid); - - mpi_errno = MPIDI_GetTagFromPort(port_name, &port_name_tag); - if (mpi_errno != MPL_STR_SUCCESS) { - TRACE_ERR("MPIDI_GetTagFromPort returned with mpi_errno=%d", mpi_errno); - } - mpi_errno = MPIDI_GetTaskidFromPort(port_name, &taskid_tag); - if (mpi_errno != MPL_STR_SUCCESS) { - TRACE_ERR("MPIDI_GetTaskidFromPort returned with mpi_errno=%d", mpi_errno); - } - - TRACE_ERR("posting connect to host %s, port %d task %d vc %p\n", - host_description, port, taskid_tag, vc ); - mpi_errno = MPIDU_send_AM_to_leader(vc, port_name_tag, taskid_tag); - - fn_exit: - return mpi_errno; - fn_fail: - goto fn_exit; -} - - -/* ------------------------------------------------------------------------- */ -/* Business card management. These routines insert or extract connection - information when using sockets from the business card */ -/* ------------------------------------------------------------------------- */ - -/* FIXME: These are small routines; we may want to bring them together - into a more specific post-connection-for-sock */ - -/* The host_description should be of length MAX_HOST_DESCRIPTION_LEN */ - - -static int MPIDI_Create_inter_root_communicator_connect(const char *port_name, - struct MPIR_Comm **comm_pptr, - MPID_VCR *vc_pptr) -{ - int mpi_errno = MPI_SUCCESS; - struct MPIR_Comm *tmp_comm; - struct MPID_VCR_t *connect_vc= NULL; - int port_name_tag, taskid_tag; - /* Connect to the root on the other side. Create a - temporary intercommunicator between the two roots so that - we can use MPI functions to communicate data between them. */ - - MPIDI_Connect_to_root(port_name, &(connect_vc)); - if (mpi_errno != MPI_SUCCESS) { - TRACE_ERR("MPIDI_Connect_to_root returned with mpi_errno=%d\n", mpi_errno); - } - - /* extract the tag from the port_name */ - mpi_errno = MPIDI_GetTagFromPort( port_name, &port_name_tag); - if (mpi_errno != MPL_STR_SUCCESS) { - TRACE_ERR("MPIDI_GetTagFromPort returned with mpi_errno=%d\n", mpi_errno); - } - - mpi_errno = MPIDI_GetTaskidFromPort(port_name, &taskid_tag); - if (mpi_errno != MPL_STR_SUCCESS) { - TRACE_ERR("MPIDI_GetTaskidFromPort returned with mpi_errno=%d\n", mpi_errno); - } - connect_vc->taskid=taskid_tag; - mpi_errno = MPIDI_Initialize_tmp_comm(&tmp_comm, connect_vc, 1, port_name_tag); - if (mpi_errno != MPI_SUCCESS) { - TRACE_ERR("MPIDI_Initialize_tmp_comm returned with mpi_errno=%d\n", mpi_errno); - } - - *comm_pptr = tmp_comm; - *vc_pptr = connect_vc; - - fn_exit: - return mpi_errno; - fn_fail: - goto fn_exit; -} - -/* Creates a communicator for the purpose of communicating with one other - process (the root of the other group). It also returns the virtual - connection */ -static int MPIDI_Create_inter_root_communicator_accept(const char *port_name, - struct MPIR_Comm **comm_pptr, - MPID_VCR *vc_pptr) -{ - int mpi_errno = MPI_SUCCESS; - struct MPIR_Comm *tmp_comm; - MPID_VCR new_vc; - - MPID_Progress_state progress_state; - int port_name_tag; - - /* extract the tag from the port_name */ - mpi_errno = MPIDI_GetTagFromPort( port_name, &port_name_tag); - if (mpi_errno != MPL_STR_SUCCESS) { - TRACE_ERR("MPIDI_GetTagFromPort returned with mpi_errnp=%d\n", mpi_errno); - } - - /* FIXME: Describe the algorithm used here, and what routine - is user on the other side of this connection */ - /* dequeue the accept queue to see if a connection with the - root on the connect side has been formed in the progress - engine (the connection is returned in the form of a vc). If - not, poke the progress engine. */ - - for(;;) - { - MPIDI_Acceptq_dequeue(&new_vc, port_name_tag); - if (new_vc != NULL) - { - break; - } - - mpi_errno = MPID_Progress_wait(100); - /* --BEGIN ERROR HANDLING-- */ - if (mpi_errno) - { - TRACE_ERR("MPID_Progress_wait returned with mpi_errno=%d\n", mpi_errno); - } - /* --END ERROR HANDLING-- */ - } - - mpi_errno = MPIDI_Initialize_tmp_comm(&tmp_comm, new_vc, 0, port_name_tag); - if (mpi_errno != MPI_SUCCESS) { - TRACE_ERR("MPIDI_Initialize_tmp_comm returned with mpi_errno=%d\n", mpi_errno); - } - - *comm_pptr = tmp_comm; - *vc_pptr = new_vc; - -fn_exit: - return mpi_errno; -fn_fail: - goto fn_exit; -} - -/* This is a utility routine used to initialize temporary communicators - used in connect/accept operations, and is only used in the above two - routines */ -static int MPIDI_Initialize_tmp_comm(struct MPIR_Comm **comm_pptr, - struct MPID_VCR_t *vc_ptr, int is_low_group, int context_id_offset) -{ - int mpi_errno = MPI_SUCCESS; - struct MPIR_Comm *tmp_comm, *commself_ptr; - - MPIR_Comm_get_ptr( MPI_COMM_SELF, commself_ptr ); - - /* WDG-old code allocated a context id that was then discarded */ - mpi_errno = MPIR_Comm_create(&tmp_comm); - if (mpi_errno != MPI_SUCCESS) { - TRACE_ERR("MPIR_Comm_create returned with mpi_errno=%d\n", mpi_errno); - } - /* fill in all the fields of tmp_comm. */ - - /* We use the second half of the context ID bits for dynamic - * processes. This assumes that the context ID mask array is made - * up of uint32_t's. */ - /* FIXME: This code is still broken for the following case: - * If the same process opens connections to the multiple - * processes, this context ID might get out of sync. - */ - tmp_comm->context_id = MPIR_CONTEXT_SET_FIELD(DYNAMIC_PROC, context_id_offset, 1); - tmp_comm->recvcontext_id = tmp_comm->context_id; - - /* sanity: the INVALID context ID value could potentially conflict with the - * dynamic proccess space */ - MPIR_Assert(tmp_comm->context_id != MPIR_INVALID_CONTEXT_ID); - MPIR_Assert(tmp_comm->recvcontext_id != MPIR_INVALID_CONTEXT_ID); - - /* FIXME - we probably need a unique context_id. */ - tmp_comm->remote_size = 1; - - /* Fill in new intercomm */ - tmp_comm->local_size = 1; - tmp_comm->rank = 0; - tmp_comm->comm_kind = MPIR_COMM_KIND__INTERCOMM; - tmp_comm->local_comm = NULL; - tmp_comm->is_low_group = is_low_group; - - /* No pg structure needed since vc has already been set up - (connection has been established). */ - - /* Point local vcr, vcrt at those of commself_ptr */ - /* FIXME: Explain why */ - tmp_comm->local_vcrt = commself_ptr->vcrt; - MPID_VCRT_Add_ref(commself_ptr->vcrt); - tmp_comm->local_vcr = commself_ptr->vcr; - - /* No pg needed since connection has already been formed. - FIXME - ensure that the comm_release code does not try to - free an unallocated pg */ - - /* Set up VC reference table */ - mpi_errno = MPID_VCRT_Create(tmp_comm->remote_size, &tmp_comm->vcrt); - if (mpi_errno != MPI_SUCCESS) { - TRACE_ERR("MPID_VCRT_Create returned with mpi_errno=%d", mpi_errno); - } - mpi_errno = MPID_VCRT_Get_ptr(tmp_comm->vcrt, &tmp_comm->vcr); - if (mpi_errno != MPI_SUCCESS) { - TRACE_ERR("MPID_VCRT_Get_ptr returned with mpi_errno=%d", mpi_errno); - } - - /* FIXME: Why do we do a dup here? */ - MPID_VCR_Dup(vc_ptr, tmp_comm->vcr); - - *comm_pptr = tmp_comm; - -fn_exit: - return mpi_errno; -fn_fail: - goto fn_exit; -} - - -/** - * * Function to add a new trasaction id in the transaction id list. This function - * * gets called only when a new connection is made with remote tasks. - * */ -void MPIDI_add_new_tranid(long long tranid) -{ - int i; - transactionID_struct *tridtmp=NULL; - - MPIU_THREAD_CS_ENTER(MSGQUEUE,0); - if(_transactionID_list == NULL) { - _transactionID_list = (transactionID_struct*) MPL_malloc(sizeof(transactionID_struct)); - _transactionID_list->cntr_for_AM = MPL_malloc(TOTAL_AM*sizeof(int)); - _transactionID_list->tranid = tranid; - for(i=0;i<TOTAL_AM;i++) - _transactionID_list->cntr_for_AM[i] = 0; - _transactionID_list->next = NULL; - MPIU_THREAD_CS_EXIT(MSGQUEUE,0); - return; - } - - tridtmp = _transactionID_list; - while(tridtmp->next != NULL) - tridtmp = tridtmp->next; - - tridtmp->next = (transactionID_struct*) MPL_malloc(sizeof(transactionID_struct)); - tridtmp = tridtmp->next; - tridtmp->tranid = tranid; - tridtmp->cntr_for_AM = MPL_malloc(TOTAL_AM*sizeof(int)); - for(i=0;i<TOTAL_AM;i++) - tridtmp->cntr_for_AM[i] = 0; - tridtmp->next = NULL; - MPIU_THREAD_CS_EXIT(MSGQUEUE,0); -} - - -/* ------------------------------------------------------------------------- */ -/* - MPIDI_Comm_connect() - - Algorithm: First create a connection (vc) between this root and the - root on the accept side. Using this vc, create a temporary - intercomm between the two roots. Use MPI functions to communicate - the other information needed to create the real intercommunicator - between the processes on the two sides. Then free the - intercommunicator between the roots. Most of the complexity is - because there can be multiple process groups on each side. -*/ -int MPIDI_Comm_connect(const char *port_name, MPIR_Info *info, int root, - struct MPIR_Comm *comm_ptr, struct MPIR_Comm **newcomm) -{ - int mpi_errno=MPI_SUCCESS; - int j, i, rank, recv_ints[3], send_ints[3], context_id; - int remote_comm_size=0; - struct MPIR_Comm *tmp_comm = NULL; - MPID_VCR new_vc= NULL; - int sendtag=100, recvtag=100, n_remote_pgs; - int n_local_pgs=1, local_comm_size; - pg_translation *local_translation = NULL, *remote_translation = NULL; - pg_node *pg_list = NULL; - MPIDI_PG_t **remote_pg = NULL; - MPIR_Context_id_t recvcontext_id = MPIR_INVALID_CONTEXT_ID; - MPIR_Errflag_t errflag = MPIR_ERR_NONE; - long long comm_cntr, lcomm_cntr; - - /* Get the context ID here because we need to send it to the remote side */ - mpi_errno = MPIR_Get_contextid( comm_ptr, &recvcontext_id ); - TRACE_ERR("MPIDI_Comm_connect calling MPIR_Get_contextid = %d\n", recvcontext_id); - if (mpi_errno) TRACE_ERR("MPIR_Get_contextid returned with mpi_errno=%d\n", mpi_errno); - - rank = comm_ptr->rank; - local_comm_size = comm_ptr->local_size; - TRACE_ERR("In MPIDI_Comm_connect - port_name=%s rank=%d root=%d\n", port_name, rank, root); - - WORLDINTCOMMCNTR += 1; - comm_cntr = WORLDINTCOMMCNTR; - lcomm_cntr = WORLDINTCOMMCNTR; - - if (rank == root) - { - /* Establish a communicator to communicate with the root on the - other side. */ - mpi_errno = MPIDI_Create_inter_root_communicator_connect( - port_name, &tmp_comm, &new_vc); - if (mpi_errno != MPI_SUCCESS) { - TRACE_ERR("MPIDI_Create_inter_root_communicator_connect returned mpi_errno=%d\n", mpi_errno); - } - TRACE_ERR("after MPIDI_Create_inter_root_communicator_connect - tmp_comm=%p new_vc=%p mpi_errno=%d\n", tmp_comm, new_vc, mpi_errno); - - /* Make an array to translate local ranks to process group index - and rank */ - local_translation = MPL_malloc(local_comm_size*sizeof(pg_translation)); -/* MPIR_CHKLMEM_MALLOC(local_translation,pg_translation*, - local_comm_size*sizeof(pg_translation), - mpi_errno,"local_translation"); */ - - /* Make a list of the local communicator's process groups and encode - them in strings to be sent to the other side. - The encoded string for each process group contains the process - group id, size and all its KVS values */ - mpi_errno = MPIDI_ExtractLocalPGInfo( comm_ptr, local_translation, - &pg_list, &n_local_pgs ); - - /* Send the remote root: n_local_pgs, local_comm_size, - Recv from the remote root: n_remote_pgs, remote_comm_size, - recvcontext_id for newcomm */ - - send_ints[0] = n_local_pgs; - send_ints[1] = local_comm_size; - send_ints[2] = recvcontext_id; - - TRACE_ERR("connect:sending 3 ints, %d, %d, %d, and receiving 2 ints with sendtag=%d recvtag=%d\n", send_ints[0], send_ints[1], send_ints[2], sendtag, recvtag); - mpi_errno = MPIC_Sendrecv(send_ints, 3, MPI_INT, 0, - sendtag++, recv_ints, 3, MPI_INT, - 0, recvtag++, tmp_comm, - MPI_STATUS_IGNORE, &errflag); - if (mpi_errno != MPI_SUCCESS) { - /* this is a no_port error because we may fail to connect - on the send if the port name is invalid */ - TRACE_ERR("MPIC_Sendrecv returned with mpi_errno=%d\n", mpi_errno); - } - - mpi_errno = MPIC_Sendrecv_replace(&comm_cntr, 1, MPI_LONG_LONG_INT, 0, - sendtag++, 0, recvtag++, tmp_comm, - MPI_STATUS_IGNORE, &errflag); - if (mpi_errno != MPI_SUCCESS) { - /* this is a no_port error because we may fail to connect - on the send if the port name is invalid */ - TRACE_ERR("MPIC_Sendrecv returned with mpi_errno=%d\n", mpi_errno); - } - } - - /* broadcast the received info to local processes */ - TRACE_ERR("accept:broadcasting 2 ints - %d and %d\n", recv_ints[0], recv_ints[1]); - mpi_errno = MPIR_Bcast_intra(recv_ints, 3, MPI_INT, root, comm_ptr, &errflag); - if (mpi_errno) TRACE_ERR("MPIR_Bcast_intra returned with mpi_errno=%d\n", mpi_errno); - - mpi_errno = MPIR_Bcast_intra(&comm_cntr, 1, MPI_LONG_LONG_INT, root, comm_ptr, &errflag); - if (mpi_errno) TRACE_ERR("MPIR_Bcast_intra returned with mpi_errno=%d\n", mpi_errno); - - if(lcomm_cntr > comm_cntr) comm_cntr = lcomm_cntr; - - /* check if root was unable to connect to the port */ - - n_remote_pgs = recv_ints[0]; - remote_comm_size = recv_ints[1]; - context_id = recv_ints[2]; - - TRACE_ERR("MPIDI_Comm_connect - n_remote_pgs=%d remote_comm_size=%d context_id=%d\n", n_remote_pgs, - remote_comm_size, context_id); - remote_pg = MPL_malloc(n_remote_pgs * sizeof(MPIDI_PG_t*)); - remote_translation = MPL_malloc(remote_comm_size * sizeof(pg_translation)); - /* Exchange the process groups and their corresponding KVSes */ - if (rank == root) - { - mpi_errno = MPIDI_SendPGtoPeerAndFree( tmp_comm, &sendtag, pg_list ); - mpi_errno = MPIDI_ReceivePGAndDistribute( tmp_comm, comm_ptr, root, &recvtag, - n_remote_pgs, remote_pg ); - /* Receive the translations from remote process rank to process group - index */ - mpi_errno = MPIC_Sendrecv(local_translation, local_comm_size * 3, - MPI_INT, 0, sendtag++, - remote_translation, remote_comm_size * 3, - MPI_INT, 0, recvtag++, tmp_comm, - MPI_STATUS_IGNORE, &errflag); - if (mpi_errno) { - TRACE_ERR("MPIC_Sendrecv returned with mpi_errno=%d\n", mpi_errno); - } - - for (i=0; i<remote_comm_size; i++) - { - TRACE_ERR(" remote_translation[%d].pg_index = %d\n remote_translation[%d].pg_rank = %d\n", - i, remote_translation[i].pg_index, i, remote_translation[i].pg_rank); - } - } - else - { - mpi_errno = MPIDI_ReceivePGAndDistribute( tmp_comm, comm_ptr, root, &recvtag, - n_remote_pgs, remote_pg ); - } - - /* Broadcast out the remote rank translation array */ - mpi_errno = MPIR_Bcast_intra(remote_translation, remote_comm_size * 3, MPI_INT, - root, comm_ptr, &errflag); - if (mpi_errno) TRACE_ERR("MPIR_Bcast_intra returned with mpi_errno=%d\n", mpi_errno); - - char *pginfo = MPL_malloc(256*sizeof(char)); - memset(pginfo, 0, 256); - char cp[20]; - for (i=0; i<remote_comm_size; i++) - { - TRACE_ERR(" remote_translation[%d].pg_index = %d remote_translation[%d].pg_rank = %d remote_translation[%d].pg_taskid=%d\n", - i, remote_translation[i].pg_index, i, remote_translation[i].pg_rank, i, remote_translation[i].pg_taskid); - TRACE_ERR("remote_pg[remote_translation[%d].pg_index]->id=%s\n",i, (char *)(remote_pg[remote_translation[i].pg_index]->id)); - strcat(pginfo, (char *)(remote_pg[remote_translation[i].pg_index]->id)); - sprintf(cp, ":%d ", remote_translation[i].pg_taskid); - strcat(pginfo, cp); - - - } - pginfo[strlen(pginfo)]='\0'; - TRACE_ERR("connection info %s\n", pginfo); - /*MPIDI_Parse_connection_info(n_remote_pgs, remote_pg);*/ - MPL_free(pginfo); - - mpi_errno = MPIR_Comm_create(newcomm); - if (mpi_errno) TRACE_ERR("MPIR_Comm_create returned with mpi_errno=%d\n", mpi_errno); - - (*newcomm)->context_id = context_id; - (*newcomm)->recvcontext_id = recvcontext_id; - (*newcomm)->is_low_group = 1; - - mpi_errno = MPIDI_SetupNewIntercomm( comm_ptr, remote_comm_size, - remote_translation, n_remote_pgs, remote_pg, *newcomm ); - (*newcomm)->mpid.world_intercomm_cntr = comm_cntr; - WORLDINTCOMMCNTR = comm_cntr; - MPIDI_add_new_tranid(comm_cntr); - -/* MPIDI_Parse_connection_info(n_remote_pgs, remote_pg); */ - if (mpi_errno != MPI_SUCCESS) { - TRACE_ERR("MPIDI_SetupNewIntercomm returned with mpi_errno=%d\n", mpi_errno); - } - - /* synchronize with remote root */ - if (rank == root) - { - mpi_errno = MPIC_Sendrecv(&i, 0, MPI_INT, 0, - sendtag++, &j, 0, MPI_INT, - 0, recvtag++, tmp_comm, - MPI_STATUS_IGNORE, &errflag); - if (mpi_errno != MPI_SUCCESS) { - TRACE_ERR("MPIC_Sendrecv returned with mpi_errno=%d\n", mpi_errno); - } - - /* All communication with remote root done. Release the communicator. */ - MPIR_Comm_release(tmp_comm,0); - } - - TRACE_ERR("connect:barrier\n"); - mpi_errno = MPIR_Barrier_intra(comm_ptr, &errflag); - if (mpi_errno != MPI_SUCCESS) { - TRACE_ERR("MPIR_Barrier_intra returned with mpi_errno=%d\n", mpi_errno); - } - - TRACE_ERR("connect:free new vc\n"); - -fn_exit: - if(local_translation) MPL_free(local_translation); - return mpi_errno; - -fn_fail: - goto fn_exit; -} - -/* - * Extract all of the process groups from the given communicator and - * form a list (returned in pg_list) of those process groups. - * Also returned is an array (local_translation) that contains tuples mapping - * rank in process group to rank in that communicator (local translation - * must be allocated before this routine is called). The number of - * distinct process groups is returned in n_local_pgs_p . - * - * This allows an intercomm_create to exchange the full description of - * all of the process groups that have made up the communicator that - * will define the "remote group". - */ -static int MPIDI_ExtractLocalPGInfo( struct MPIR_Comm *comm_p, - pg_translation local_translation[], - pg_node **pg_list_p, - int *n_local_pgs_p ) -{ - pg_node *pg_list = 0, *pg_iter, *pg_trailer; - int i, cur_index = 0, local_comm_size, mpi_errno = 0; - char *pg_id; - - local_comm_size = comm_p->local_size; - - /* Make a list of the local communicator's process groups and encode - them in strings to be sent to the other side. - The encoded string for each process group contains the process - group id, size and all its KVS values */ - - cur_index = 0; - pg_list = MPL_malloc(sizeof(pg_node)); - - pg_list->pg_id = MPL_strdup(comm_p->vcr[0]->pg->id); - pg_list->index = cur_index++; - pg_list->next = NULL; - /* XXX DJG FIXME-MT should we be checking this? the add/release macros already check this */ - TRACE_ERR("MPIR_Object_get_ref(comm_p->vcr[0]->pg) comm_p=%x vsr=%x pg=%x %d\n", comm_p, comm_p->vcr[0], comm_p->vcr[0]->pg, MPIR_Object_get_ref(comm_p->vcr[0]->pg)); - MPIR_Assert( MPIR_Object_get_ref(comm_p->vcr[0]->pg)); - mpi_errno = MPIDI_PG_To_string(comm_p->vcr[0]->pg, &pg_list->str, - &pg_list->lenStr ); - TRACE_ERR("pg_list->str=%s pg_list->lenStr=%d\n", pg_list->str, pg_list->lenStr); - if (mpi_errno != MPI_SUCCESS) { - TRACE_ERR("MPIDI_PG_To_string returned with mpi_errno=%d\n", mpi_errno); - } - TRACE_ERR("PG as string is %s\n", pg_list->str ); - local_translation[0].pg_index = 0; - local_translation[0].pg_rank = comm_p->vcr[0]->pg_rank; - local_translation[0].pg_taskid = comm_p->vcr[0]->taskid; - TRACE_ERR("local_translation[0].pg_index=%d local_translation[0].pg_rank=%d\n", local_translation[0].pg_index, local_translation[0].pg_rank); - pg_iter = pg_list; - for (i=1; i<local_comm_size; i++) { - pg_iter = pg_list; - pg_trailer = pg_list; - while (pg_iter != NULL) { - /* Check to ensure pg is (probably) valid */ - /* XXX DJG FIXME-MT should we be checking this? the add/release macros already check this */ - MPIR_Assert(MPIR_Object_get_ref(comm_p->vcr[i]->pg) != 0); - if (MPIDI_PG_Id_compare(comm_p->vcr[i]->pg->id, pg_iter->pg_id)) { - local_translation[i].pg_index = pg_iter->index; - local_translation[i].pg_rank = comm_p->vcr[i]->pg_rank; - local_translation[i].pg_taskid = comm_p->vcr[i]->taskid; - TRACE_ERR("local_translation[%d].pg_index=%d local_translation[%d].pg_rank=%d\n", i, local_translation[i].pg_index, i,local_translation[i].pg_rank); - break; - } - if (pg_trailer != pg_iter) - pg_trailer = pg_trailer->next; - pg_iter = pg_iter->next; - } - if (pg_iter == NULL) { - /* We use MPL_malloc directly because we do not know in - advance how many nodes we may allocate */ - pg_iter = (pg_node*)MPL_malloc(sizeof(pg_node)); - pg_iter->pg_id = MPL_strdup(comm_p->vcr[i]->pg->id); - pg_iter->index = cur_index++; - pg_iter->next = NULL; - mpi_errno = MPIDI_PG_To_string(comm_p->vcr[i]->pg, &pg_iter->str, - &pg_iter->lenStr ); - - TRACE_ERR("cur_index=%d pg_iter->str=%s pg_iter->lenStr=%d\n", cur_index, pg_iter->str, pg_iter->lenStr); - if (mpi_errno != MPI_SUCCESS) { - TRACE_ERR("MPIDI_PG_To_string returned with mpi_errno=%d\n", mpi_errno); - } - local_translation[i].pg_index = pg_iter->index; - local_translation[i].pg_rank = comm_p->vcr[i]->pg_rank; - local_translation[i].pg_taskid = comm_p->vcr[i]->taskid; - pg_trailer->next = pg_iter; - } - } - - *n_local_pgs_p = cur_index; - *pg_list_p = pg_list; - - fn_exit: - return mpi_errno; - fn_fail: - goto fn_exit; -} - - - -/* The root process in comm_ptr receives strings describing the - process groups and then distributes them to the other processes - in comm_ptr. - See SendPGToPeer for the routine that sends the descriptions */ -static int MPIDI_ReceivePGAndDistribute( struct MPIR_Comm *tmp_comm, struct MPIR_Comm *comm_ptr, - int root, int *recvtag_p, - int n_remote_pgs, MPIDI_PG_t *remote_pg[] ) -{ - char *pg_str = 0; - char *pginfo = 0; - int i, j, flag; - int rank = comm_ptr->rank; - int mpi_errno = 0; - int recvtag = *recvtag_p; - MPIR_Errflag_t errflag = MPIR_ERR_NONE; - - TRACE_ERR("MPIDI_ReceivePGAndDistribute - n_remote_pgs=%d\n", n_remote_pgs); - for (i=0; i<n_remote_pgs; i++) { - - if (rank == root) { - /* First, receive the pg description from the partner */ - mpi_errno = MPIC_Recv(&j, 1, MPI_INT, 0, recvtag++, - tmp_comm, MPI_STATUS_IGNORE, &errflag); - *recvtag_p = recvtag; - if (mpi_errno != MPI_SUCCESS) { - TRACE_ERR("MPIC_Recv returned with mpi_errno=%d\n", mpi_errno); - } - pg_str = (char*)MPL_malloc(j); - mpi_errno = MPIC_Recv(pg_str, j, MPI_CHAR, 0, recvtag++, - tmp_comm, MPI_STATUS_IGNORE, &errflag); - *recvtag_p = recvtag; - if (mpi_errno != MPI_SUCCESS) { - TRACE_ERR("MPIC_Recv returned with mpi_errno=%d\n", mpi_errno); - } - } - - /* Broadcast the size and data to the local communicator */ - TRACE_ERR("accept:broadcasting 1 int\n"); - mpi_errno = MPIR_Bcast_intra(&j, 1, MPI_INT, root, comm_ptr, &errflag); - if (mpi_errno != MPI_SUCCESS) TRACE_ERR("MPIR_Bcast_intra returned with mpi_errno=%d\n", mpi_errno); - - if (rank != root) { - /* The root has already allocated this string */ - pg_str = (char*)MPL_malloc(j); - } - TRACE_ERR("accept:broadcasting string of length %d\n", j); - pg_str[j-1]='\0'; - mpi_errno = MPIR_Bcast_intra(pg_str, j, MPI_CHAR, root, comm_ptr, &errflag); - if (mpi_errno != MPI_SUCCESS) - TRACE_ERR("MPIR_Bcast_intra returned with mpi_errno=%d\n", mpi_errno); - /* Then reconstruct the received process group. This step - also initializes the created process group */ - - TRACE_ERR("Adding connection information - pg_str=%s\n", pg_str); - TRACE_ERR("Creating pg from string %s flag=%d\n", pg_str, flag); - mpi_errno = MPIDI_PG_Create_from_string(pg_str, &remote_pg[i], &flag); - TRACE_ERR("remote_pg[%d]->id=%s\n", i, (char*)(remote_pg[i]->id)); - if (mpi_errno != MPI_SUCCESS) { - TRACE_ERR("MPIDI_PG_Create_from_string returned with mpi_errno=%d\n", mpi_errno); - } - - MPL_free(pg_str); - } - /*MPIDI_Parse_connection_info(pg_str); */ - fn_exit: - return mpi_errno; - fn_fail: - goto fn_exit; -} - - -void MPIDI_Add_connection_info(int wid, int wsize, pami_task_t *taskids) { - int jobIdSize=64; - char jobId[jobIdSize]; - int ref_count, i; - conn_info *tmp_node1=NULL, *tmp_node2=NULL; - - TRACE_ERR("MPIDI_Add_connection_info ENTER wid=%d wsize=%d\n", wid, wsize); - PMI2_Job_GetId(jobId, jobIdSize); - if(atoi(jobId) == wid) - return; - - /* FIXME: check the lock */ - MPIU_THREAD_CS_ENTER(MSGQUEUE,0); - if(_conn_info_list == NULL) { /* Connection list is not yet created */ - _conn_info_list = (conn_info*) MPL_malloc(sizeof(conn_info)); - _conn_info_list->rem_world_id = wid; - _conn_info_list->ref_count = 1; - - ref_count = _conn_info_list->ref_count; - if(taskids != NULL) { - _conn_info_list->rem_taskids = MPL_malloc((wsize+1)*sizeof(int)); - for(i=0;i<wsize;i++) { - _conn_info_list->rem_taskids[i] = taskids[i]; - } - _conn_info_list->rem_taskids[i] = -1; - } - else - _conn_info_list->rem_taskids = NULL; - _conn_info_list->next = NULL; - } - else { - tmp_node1 = _conn_info_list; - while(tmp_node1) { - tmp_node2 = tmp_node1; - if(tmp_node1->rem_world_id == wid) - break; - tmp_node1 = tmp_node1->next; - } - if(tmp_node1) { /* Connection already exists. Increment reference count */ - if(tmp_node1->ref_count == 0) { - if(taskids != NULL) { - tmp_node1->rem_taskids = MPL_malloc((wsize+1)*sizeof(int)); - for(i=0;i<wsize;i++) { - tmp_node1->rem_taskids[i] = taskids[i]; - } - tmp_node1->rem_taskids[i] = -1; - } - tmp_node1->rem_world_id = wid; - } - tmp_node1->ref_count++; - ref_count = tmp_node1->ref_count; - } - else { /* Connection do not exists. Create a new connection */ - tmp_node2->next = (conn_info*) MPL_malloc(sizeof(conn_info)); - tmp_node2 = tmp_node2->next; - tmp_node2->rem_world_id = wid; - tmp_node2->ref_count = 1; - - ref_count = tmp_node2->ref_count; - if(taskids != NULL) { - tmp_node2->rem_taskids = MPL_malloc((wsize+1)*sizeof(int)); - for(i=0;i<wsize;i++) { - tmp_node2->rem_taskids[i] = taskids[i]; - } - tmp_node2->rem_taskids[i] = -1; - } - else - tmp_node2->rem_taskids = NULL; - tmp_node2->next = NULL; - } - } - MPIU_THREAD_CS_EXIT(MSGQUEUE,0); - - tmp_node1 = _conn_info_list; - while(tmp_node1) { - TRACE_ERR("REM WORLD=%d ref_count=%d", tmp_node1->rem_world_id,tmp_node1->ref_count); - - tmp_node1 = tmp_node1->next; - } -} - - -/** - * This routine adds the remote world (wid) to local known world linked list - * if there is no record of it before, or increment the reference count - * associated with world (wid) if it is known before - */ -void MPIDI_Parse_connection_info(int n_remote_pgs, MPIDI_PG_t **remote_pg) { - int i, p, ref_count=0; - int jobIdSize=8; - char jobId[jobIdSize]; - char *pginfo_sav, *pgid_taskid_sav, *pgid, *pgid_taskid[20], *pginfo_tmp, *cp3, *cp2; - pami_task_t *taskids; - int n_rem_wids=0; - int mpi_errno = MPI_SUCCESS; - MPIDI_PG_t *existing_pg; - - for(p=0; p<n_remote_pgs; p++) { - TRACE_ERR("call MPIDI_PG_Find to find %s\n", (char*)(remote_pg[p]->id)); - mpi_errno = MPIDI_PG_Find(remote_pg[p]->id, &existing_pg); - if (mpi_errno) TRACE_ERR("MPIDI_PG_Find failed\n"); - - if (existing_pg != NULL) { - taskids = MPL_malloc((existing_pg->size)*sizeof(pami_task_t)); - for(i=0; i<existing_pg->size; i++) { - taskids[i]=existing_pg->vct[i].taskid; - TRACE_ERR("id=%s taskids[%d]=%d\n", (char*)(remote_pg[p]->id), i, taskids[i]); - } - MPIDI_Add_connection_info(atoi((char*)(remote_pg[p]->id)), existing_pg->size, taskids); - MPL_free(taskids); - } - } -} - - - -/** - * Function to increment the active message counter for a particular trasaction id. - * This function is used inside disconnect routine - * whichAM = FIRST_AM/SECOND_AM/LAST_AM - */ -void MPIDI_increment_AM_cntr_for_tranid(long long tranid, int whichAM) -{ - transactionID_struct *tridtmp; - - /* No error thrown here if tranid not found. This is for the case where timout - * happened in MPI_Comm_disconnect and tasks have freed the tranid list node - * and now after this the Active message is received. - */ - - tridtmp = _transactionID_list; - - MPIU_THREAD_CS_ENTER(MSGQUEUE,0); - while(tridtmp != NULL) { - if(tridtmp->tranid == tranid) { - tridtmp->cntr_for_AM[whichAM]++; - break; - } - tridtmp = tridtmp->next; - } - MPIU_THREAD_CS_EXIT(MSGQUEUE,0); - - TRACE_ERR("MPIDI_increment_AM_cntr_for_tranid - tridtmp->cntr_for_AM[%d]=%d\n", - whichAM, tridtmp->cntr_for_AM[whichAM]); -} - -/** - * Function to free a partucular trasaction id node from the trasaction id list. - * This function is called inside disconnect routine once the remote connection is - * terminated - */ -void MPIDI_free_tranid_node(long long tranid) -{ - transactionID_struct *tridtmp, *tridtmp2; - - MPID_assert(_transactionID_list != NULL); - - tridtmp = tridtmp2 = _transactionID_list; - - MPIU_THREAD_CS_ENTER(MSGQUEUE,0); - while(tridtmp != NULL) { - if(tridtmp->tranid == tranid) { - /* If there is only one node */ - if(_transactionID_list->next == NULL) { - MPL_free(_transactionID_list->cntr_for_AM); - MPL_free(_transactionID_list); - _transactionID_list = NULL; - MPIU_THREAD_CS_EXIT(MSGQUEUE,0); - return; - } - /* If more than one node and if this is the first node of the list */ - if(tridtmp == _transactionID_list && tridtmp->next != NULL) { - _transactionID_list = _transactionID_list->next; - MPL_free(tridtmp->cntr_for_AM); - MPL_free(tridtmp); - MPIU_THREAD_CS_EXIT(MSGQUEUE,0); - return; - } - /* For rest all other nodes position of the list */ - tridtmp2->next = tridtmp->next; - MPL_free(tridtmp->cntr_for_AM); - MPL_free(tridtmp); - MPIU_THREAD_CS_EXIT(MSGQUEUE,0); - return; - } - tridtmp2 = tridtmp; - tridtmp = tridtmp->next; - } - MPIU_THREAD_CS_EXIT(MSGQUEUE,0); -} - -/** This routine is used inside finalize to free all the nodes - * if the disconnect call has not been called - */ -void MPIDI_free_all_tranid_node() -{ - transactionID_struct *tridtmp; - - MPIU_THREAD_CS_ENTER(MSGQUEUE,0); - while(_transactionID_list != NULL) { - tridtmp = _transactionID_list; - _transactionID_list = _transactionID_list->next; - MPL_free(tridtmp->cntr_for_AM); - MPL_free(tridtmp); - } - MPIU_THREAD_CS_EXIT(MSGQUEUE,0); -} - -/* Sends the process group information to the peer and frees the - pg_list */ -static int MPIDI_SendPGtoPeerAndFree( struct MPIR_Comm *tmp_comm, int *sendtag_p, - pg_node *pg_list ) -{ - int mpi_errno = 0; - int sendtag = *sendtag_p, i; - pg_node *pg_iter; - MPIR_Errflag_t errflag = MPIR_ERR_NONE; - - while (pg_list != NULL) { - pg_iter = pg_list; - i = pg_iter->lenStr; - TRACE_ERR("connect:sending 1 int: %d\n", i); - mpi_errno = MPIC_Send(&i, 1, MPI_INT, 0, sendtag++, tmp_comm, &errflag); - *sendtag_p = sendtag; - if (mpi_errno != MPI_SUCCESS) { - TRACE_ERR("MPIC_Send returned with mpi_errno=%d\n", mpi_errno); - } - - TRACE_ERR("connect:sending string length %d\n", i); - mpi_errno = MPIC_Send(pg_iter->str, i, MPI_CHAR, 0, sendtag++, - tmp_comm, &errflag); - *sendtag_p = sendtag; - if (mpi_errno != MPI_SUCCESS) { - TRACE_ERR("MPIC_Send returned with mpi_errno=%d\n", mpi_errno); - } - - pg_list = pg_list->next; - MPL_free(pg_iter->str); - MPL_free(pg_iter->pg_id); - MPL_free(pg_iter); - } - - fn_exit: - return mpi_errno; - fn_fail: - goto fn_exit; -} - -/* ---------------------------------------------------------------------- */ -/* - * MPIDI_Comm_accept() - - Algorithm: First dequeue the vc from the accept queue (it was - enqueued by the progress engine in response to a connect request - from the root process that is attempting the connection on - the connect side). Use this vc to create an - intercommunicator between this root and the root on the connect - side. Use this intercomm. to communicate the other information - needed to create the real intercommunicator between the processes - on the two sides. Then free the intercommunicator between the - roots. Most of the complexity is because there can be multiple - process groups on each side. - - */ -int MPIDI_Comm_accept(const char *port_name, MPIR_Info *info, int root, - struct MPIR_Comm *comm_ptr, struct MPIR_Comm **newcomm) -{ - int mpi_errno=MPI_SUCCESS; - int i, j, rank, recv_ints[3], send_ints[3], context_id; - int remote_comm_size=0; - struct MPIR_Comm *tmp_comm = NULL, *intercomm; - MPID_VCR new_vc = NULL; - int sendtag=100, recvtag=100, local_comm_size; - int n_local_pgs=1, n_remote_pgs; - pg_translation *local_translation = NULL, *remote_translation = NULL; - pg_node *pg_list = NULL; - MPIDI_PG_t **remote_pg = NULL; - MPIR_Errflag_t errflag = MPIR_ERR_NONE; - char send_char[16], recv_char[16], remote_taskids[16]; - long long comm_cntr, lcomm_cntr; - int leader_taskid; - - /* Create the new intercommunicator here. We need to send the - context id to the other side. */ - mpi_errno = MPIR_Comm_create(newcomm); - if (mpi_errno != MPI_SUCCESS) { - TRACE_ERR("MPIR_Comm_create returned with mpi_errno=%d\n", mpi_errno); - } - mpi_errno = MPIR_Get_contextid( comm_ptr, &(*newcomm)->recvcontext_id ); - TRACE_ERR("In MPIDI_Comm_accept - MPIR_Get_contextid=%d\n", (*newcomm)->recvcontext_id); - if (mpi_errno) TRACE_ERR("MPIR_Get_contextid returned with mpi_errno=%d\n", mpi_errno); - /* FIXME why is this commented out? */ - /* (*newcomm)->context_id = (*newcomm)->recvcontext_id; */ - - rank = comm_ptr->rank; - local_comm_size = comm_ptr->local_size; - - WORLDINTCOMMCNTR += 1; - comm_cntr = WORLDINTCOMMCNTR; - lcomm_cntr = WORLDINTCOMMCNTR; - - if (rank == root) - { - /* Establish a communicator to communicate with the root on the - other side. */ - mpi_errno = MPIDI_Create_inter_root_communicator_accept(port_name, - &tmp_comm, &new_vc); - TRACE_ERR("done MPIDI_Create_inter_root_communicator_accept mpi_errno=%d tmp_comm=%p new_vc=%p \n", mpi_errno, tmp_comm, new_vc); - if (mpi_errno != MPI_SUCCESS) { - TRACE_ERR("MPIDI_Create_inter_root_communicator_accept returned with mpi_errno=%d\n", mpi_errno); - } - - /* Make an array to translate local ranks to process group index and - rank */ - local_translation = MPL_malloc(local_comm_size*sizeof(pg_translation)); -/* MPIR_CHKLMEM_MALLOC(local_translation,pg_translation*, - local_comm_size*sizeof(pg_translation), - mpi_errno,"local_translation"); */ - - /* Make a list of the local communicator's process groups and encode - them in strings to be sent to the other side. - The encoded string for each process group contains the process - group id, size and all its KVS values */ - mpi_errno = MPIDI_ExtractLocalPGInfo( comm_ptr, local_translation, - &pg_list, &n_local_pgs ); - /* Send the remote root: n_local_pgs, local_comm_size, context_id for - newcomm. - Recv from the remote root: n_remote_pgs, remote_comm_size */ - - send_ints[0] = n_local_pgs; - send_ints[1] = local_comm_size; - send_ints[2] = (*newcomm)->recvcontext_id; - - TRACE_ERR("accept:sending 3 ints, %d, %d, %d, and receiving 2 ints with sendtag=%d recvtag=%d\n", send_ints[0], send_ints[1], send_ints[2], sendtag, recvtag); - mpi_errno = MPIC_Sendrecv(send_ints, 3, MPI_INT, 0, - sendtag++, recv_ints, 3, MPI_INT, - 0, recvtag++, tmp_comm, - MPI_STATUS_IGNORE, &errflag); - if (mpi_errno != MPI_SUCCESS) { - TRACE_ERR("MPIC_Sendrecv returned with mpi_errno=%d\n", mpi_errno); - } -#if 0 - send_char = pg_list->str; - TRACE_ERR("accept:sending 1 string and receiving 1 string\n", send_char, recv_char); - mpi_errno = MPIC_Sendrecv(send_char, 1, MPI_CHAR, 0, - sendtag++, recv_char, 3, MPI_CHAR, - 0, recvtag++, tmp_comm, - MPI_STATUS_IGNORE, &errflag); - if (mpi_errno != MPI_SUCCESS) { - TRACE_ERR("MPIC_Sendrecv returned with mpi_errno=%d\n", mpi_errno); - } -#endif - mpi_errno = MPIC_Sendrecv_replace(&comm_cntr, 1, MPI_LONG_LONG_INT, 0, - sendtag++, 0, recvtag++, tmp_comm, - MPI_STATUS_IGNORE, &errflag); - if (mpi_errno != MPI_SUCCESS) { - /* this is a no_port error because we may fail to connect - on the send if the port name is invalid */ - TRACE_ERR("MPIC_Sendrecv returned with mpi_errno=%d\n", mpi_errno); - } - - } - - /* broadcast the received info to local processes */ - TRACE_ERR("accept:broadcasting 2 ints - %d and %d\n", recv_ints[0], recv_ints[1]); - mpi_errno = MPIR_Bcast_intra(recv_ints, 3, MPI_INT, root, comm_ptr, &errflag); - if (mpi_errno) TRACE_ERR("MPIR_Bcast_intra returned with mpi_errno=%d\n", mpi_errno); - - mpi_errno = MPIR_Bcast_intra(&comm_cntr, 1, MPI_LONG_LONG_INT, root, comm_ptr, &errflag); - if (mpi_errno) TRACE_ERR("MPIR_Bcast_intra returned with mpi_errno=%d\n", mpi_errno); - - if(lcomm_cntr > comm_cntr) comm_cntr = lcomm_cntr; - n_remote_pgs = recv_ints[0]; - remote_comm_size = recv_ints[1]; - context_id = recv_ints[2]; - remote_pg = MPL_malloc(n_remote_pgs * sizeof(MPIDI_PG_t*)); - remote_translation = MPL_malloc(remote_comm_size * sizeof(pg_translation)); - TRACE_ERR("[%d]accept:remote process groups: %d\nremote comm size: %d\nrecv_char: %s\n", rank, n_remote_pgs, remote_comm_size, remote_taskids); - - /* Exchange the process groups and their corresponding KVSes */ - if (rank == root) - { - /* The root receives the PG from the peer (in tmp_comm) and - distributes them to the processes in comm_ptr */ - mpi_errno = MPIDI_ReceivePGAndDistribute( tmp_comm, comm_ptr, root, &recvtag, - n_remote_pgs, remote_pg ); - - mpi_errno = MPIDI_SendPGtoPeerAndFree( tmp_comm, &sendtag, pg_list ); - - /* Receive the translations from remote process rank to process group index */ - TRACE_ERR("accept:sending %d ints and receiving %d ints\n", local_comm_size * 2, remote_comm_size * 2); - mpi_errno = MPIC_Sendrecv(local_translation, local_comm_size * 3, - MPI_INT, 0, sendtag++, - remote_translation, remote_comm_size * 3, - MPI_INT, 0, recvtag++, tmp_comm, - MPI_STATUS_IGNORE, &errflag); - for (i=0; i<remote_comm_size; i++) - { - TRACE_ERR(" remote_translation[%d].pg_index = %d\n remote_translation[%d].pg_rank = %d\n", - i, remote_translation[i].pg_index, i, remote_translation[i].pg_rank); - } - } - else - { - mpi_errno = MPIDI_ReceivePGAndDistribute( tmp_comm, comm_ptr, root, &recvtag, - n_remote_pgs, remote_pg ); - } - for(i=0; i<n_remote_pgs; i++) - { - TRACE_ERR("after calling MPIDI_ReceivePGAndDistribute - remote_pg[%d]->id=%s\n",i, (char *)(remote_pg[i]->id)); - } - - - /* Broadcast out the remote rank translation array */ - TRACE_ERR("Broadcast remote_translation"); - mpi_errno = MPIR_Bcast_intra(remote_translation, remote_comm_size * 3, MPI_INT, - root, comm_ptr, &errflag); - if (mpi_errno) TRACE_ERR("MPIR_Bcast_intra returned with mpi_errno=%d\n", mpi_errno); - TRACE_ERR("[%d]accept:Received remote_translation after broadcast:\n", rank); - char *pginfo = MPL_malloc(256*sizeof(char)); - memset(pginfo, 0, 256); - char cp[20]; - for (i=0; i<remote_comm_size; i++) - { - TRACE_ERR(" remote_translation[%d].pg_index = %d remote_translation[%d].pg_rank = %d remote_translation[%d].pg_taskid=%d\n", - i, remote_translation[i].pg_index, i, remote_translation[i].pg_rank, i, remote_translation[i].pg_taskid); - TRACE_ERR("remote_pg[remote_translation[%d].pg_index]->id=%s\n",i, (char *)(remote_pg[remote_translation[i].pg_index]->id)); - strcat(pginfo, (char *)(remote_pg[remote_translation[i].pg_index]->id)); - sprintf(cp, ":%d ", remote_translation[i].pg_taskid); - strcat(pginfo, cp); - - - } - pginfo[strlen(pginfo)]='\0'; - TRACE_ERR("connection info %s\n", pginfo); -/* MPIDI_Parse_connection_info(n_remote_pgs, remote_pg); */ - MPL_free(pginfo); - - - /* Now fill in newcomm */ - intercomm = *newcomm; - intercomm->context_id = context_id; - intercomm->is_low_group = 0; - - mpi_errno = MPIDI_SetupNewIntercomm( comm_ptr, remote_comm_size, - remote_translation, n_remote_pgs, remote_pg, intercomm ); - intercomm->mpid.world_intercomm_cntr = comm_cntr; - WORLDINTCOMMCNTR = comm_cntr; - MPIDI_add_new_tranid(comm_cntr); - - if (mpi_errno != MPI_SUCCESS) { - TRACE_ERR("MPIDI_SetupNewIntercomm returned with mpi_errno=%d\n", mpi_errno); - } - - /* synchronize with remote root */ - if (rank == root) - { - mpi_errno = MPIC_Sendrecv(&i, 0, MPI_INT, 0, - sendtag++, &j, 0, MPI_INT, - 0, recvtag++, tmp_comm, - MPI_STATUS_IGNORE, &errflag); - if (mpi_errno != MPI_SUCCESS) { - TRACE_ERR("MPIC_Sendrecv returned with mpi_errno=%d\n", mpi_errno); - } - - /* All communication with remote root done. Release the communicator. */ - MPIR_Comm_release(tmp_comm,0); - } - - mpi_errno = MPIR_Barrier_intra(comm_ptr, &errflag); - if (mpi_errno != MPI_SUCCESS) { - TRACE_ERR("MPIR_Barrier_intra returned with mpi_errno=%d\n", mpi_errno); - } - -fn_exit: - if(local_translation) MPL_free(local_translation); - return mpi_errno; - -fn_fail: - goto fn_exit; -} - -/* ------------------------------------------------------------------------- */ - -/* This routine initializes the new intercomm, setting up the - VCRT and other common structures. The is_low_group and context_id - fields are NOT set because they differ in the use of this - routine in Comm_accept and Comm_connect. The virtual connections - are initialized from a collection of process groups. - - Input parameters: -+ comm_ptr - communicator that gives the group for the "local" group on the - new intercommnicator -. remote_comm_size - size of remote group -. remote_translation - array that specifies the process group and rank in - that group for each of the processes to include in the remote group of the - new intercommunicator -- remote_pg - array of remote process groups - - Input/Output Parameter: -. intercomm - New intercommunicator. The intercommunicator must already - have been allocated; this routine initializes many of the fields - - Note: - This routine performance a barrier over 'comm_ptr'. Why? -*/ -static int MPIDI_SetupNewIntercomm( struct MPIR_Comm *comm_ptr, int remote_comm_size, - pg_translation remote_translation[], - int n_remote_pgs, MPIDI_PG_t **remote_pg, - struct MPIR_Comm *intercomm ) -{ - int mpi_errno = MPI_SUCCESS, i, j, index=0; - MPIR_Errflag_t errflag = MPIR_ERR_NONE; - int total_rem_world_cnts, p=0; - char *world_tasks, *cp1; - conn_info *tmp_node; - int conn_world_ids[64]; - MPID_VCR *worldlist; - int worldsize; - pami_endpoint_t dest; - MPIR_Comm *comm; - pami_task_t leader1=-1, leader2=-1, leader_taskid=-1; - long long comm_cntr=0, lcomm_cntr=-1; - int jobIdSize=64; - char jobId[jobIdSize]; - - TRACE_ERR("MPIDI_SetupNewIntercomm - remote_comm_size=%d\n", remote_comm_size); - /* FIXME: How much of this could/should be common with the - upper level (src/mpi/comm/ *.c) code? For best robustness, - this should use the same routine (not copy/paste code) as - in the upper level code. */ - intercomm->attributes = NULL; - intercomm->remote_size = remote_comm_size; - intercomm->local_size = comm_ptr->local_size; - intercomm->rank = comm_ptr->rank; - intercomm->local_group = NULL; - intercomm->remote_group = NULL; - intercomm->comm_kind = MPIR_COMM_KIND__INTERCOMM; - intercomm->local_comm = NULL; - intercomm->coll_fns = NULL; - intercomm->mpid.world_ids = NULL; /*FIXME*/ - - /* Point local vcr, vcrt at those of incoming intracommunicator */ - intercomm->local_vcrt = comm_ptr->vcrt; - MPID_VCRT_Add_ref(comm_ptr->vcrt); - intercomm->local_vcr = comm_ptr->vcr; - for(i=0; i<comm_ptr->local_size; i++) - TRACE_ERR("intercomm->local_vcr[%d]->pg_rank=%d comm_ptr->vcr[%d].pg_rank=%d intercomm->local_vcr[%d]->taskid=%d comm_ptr->vcr[%d]->taskid=%d\n", i, intercomm->local_vcr[i]->pg_rank, i, comm_ptr->vcr[i]->pg_rank, i, intercomm->local_vcr[i]->taskid, i, comm_ptr->vcr[i]->taskid); - - /* Set up VC reference table */ - mpi_errno = MPID_VCRT_Create(intercomm->remote_size, &intercomm->vcrt); - if (mpi_errno != MPI_SUCCESS) { - TRACE_ERR("MPID_VCRT_Create returned with mpi_errno=%d\n", mpi_errno); - } - mpi_errno = MPID_VCRT_Get_ptr(intercomm->vcrt, &intercomm->vcr); - if (mpi_errno != MPI_SUCCESS) { - TRACE_ERR("MPID_VCRT_Get_ptr returned with mpi_errno=%d\n", mpi_errno); - } - - for (i=0; i < intercomm->remote_size; i++) { - MPIDI_PG_Dup_vcr(remote_pg[remote_translation[i].pg_index], - remote_translation[i].pg_rank, remote_translation[i].pg_taskid,&intercomm->vcr[i]); - TRACE_ERR("MPIDI_SetupNewIntercomm - pg_id=%s pg_rank=%d pg_taskid=%d intercomm->vcr[%d]->taskid=%d intercomm->vcr[%d]->pg=%x\n ", remote_pg[remote_translation[i].pg_index]->id, remote_translation[i].pg_rank, remote_translation[i].pg_taskid, i, intercomm->vcr[i]->taskid, i, intercomm->vcr[i]->pg); - PAMI_Endpoint_create(MPIDI_Client, remote_translation[i].pg_taskid, 0, &dest); - PAMI_Resume(MPIDI_Context[0], - &dest, 1); - } - - MPIDI_Parse_connection_info(n_remote_pgs, remote_pg); - - /* anchor connection information in mpid */ - total_rem_world_cnts = 0; - tmp_node = _conn_info_list; - p=0; - while(tmp_node != NULL) { - total_rem_world_cnts++; - conn_world_ids[p++]=tmp_node->rem_world_id; - tmp_node = tmp_node->next; - } - if(intercomm->mpid.world_ids) { /* need to look at other places that may populate world id list for this communicator */ - for(i=0;intercomm->mpid.world_ids[i]!=-1;i++) - { - for(j=0;j<total_rem_world_cnts;j++) { - if(intercomm->mpid.world_ids[i] == conn_world_ids[j]) { - conn_world_ids[j] = -1; - } - } - } - /* Now Total world_ids inside intercomm->world_ids = i, excluding last entry of ' -1' */ - index = 0; - for(j=0;j<total_rem_world_cnts;j++) { - if(conn_world_ids[j] != -1) - index++; - } - if(index) { - intercomm->mpid.world_ids = MPL_malloc((index+i+1)*sizeof(int)); - /* Current index i inside intercomm->mpid.world_ids is - * the place where next world_id can be added - */ - for(j=0;j<total_rem_world_cnts;j++) { - if(conn_world_ids[j] != -1) { - intercomm->mpid.world_ids[i++] = conn_world_ids[j]; - } - } - intercomm->mpid.world_ids[i] = -1; - } - } - else { - index=0; - intercomm->mpid.world_ids = MPL_malloc((n_remote_pgs+1)*sizeof(int)); - PMI2_Job_GetId(jobId, jobIdSize); - for(i=0;i<n_remote_pgs;i++) { - if(atoi(jobId) != atoi((char *)remote_pg[i]->id) ) - intercomm->mpid.world_ids[index++] = atoi((char *)remote_pg[i]->id); - } - intercomm->mpid.world_ids[index++] = -1; - } - for(i=0; intercomm->mpid.world_ids[i] != -1; i++) - TRACE_ERR("intercomm=%x intercomm->mpid.world_ids[%d]=%d\n", intercomm, i, intercomm->mpid.world_ids[i]); - - leader_taskid = comm_ptr->vcr[0]->taskid; - - MPIR_Comm *comm_world_ptr = MPIR_Process.comm_world; - worldlist = comm_world_ptr->vcr; - worldsize = comm_world_ptr->local_size; - comm = intercomm; - for(i=0;i<intercomm->local_size;i++) - { - for(j=0;j<comm_world_ptr->local_size;j++) - { - if(intercomm->local_vcr[i]->taskid == comm_world_ptr->vcr[j]->taskid) { - leader1 = comm_world_ptr->vcr[j]->taskid; - break; - } - } - if(leader1 != -1) - break; - } - for(i=0;i<intercomm->remote_size;i++) - { - for(j=0;j<comm_world_ptr->local_size;j++) - { - if(intercomm->vcr[i]->taskid == comm_world_ptr->vcr[j]->taskid) { - leader2 = comm_world_ptr->vcr[j]->taskid; - break; - } - } - if(leader2 != -1) - break; - } - - if(leader1 == -1) - leader_taskid = leader2; - else if(leader2 == -1) - leader_taskid = leader1; - else - leader_taskid = leader1 < leader2 ? leader1 : leader2; - intercomm->mpid.local_leader = leader_taskid; - - mpi_errno = MPIR_Comm_commit(intercomm); - if (mpi_errno) TRACE_ERR("MPIR_Comm_commit returned with mpi_errno=%d\n", mpi_errno); - - mpi_errno = MPIR_Barrier_intra(comm_ptr, &errflag); - if (mpi_errno != MPI_SUCCESS) { - TRACE_ERR("MPIR_Barrier_intra returned with mpi_errno=%d\n", mpi_errno); - } - - fn_exit: - if(remote_pg) MPL_free(remote_pg); - if(remote_translation) MPL_free(remote_translation); - return mpi_errno; - - fn_fail: - goto fn_exit; -} - - -/* Attempt to dequeue a vc from the accept queue. If the queue is - empty or the port_name_tag doesn't match, return a NULL vc. */ -int MPIDI_Acceptq_dequeue(MPID_VCR * vcr, int port_name_tag) -{ - int mpi_errno=MPI_SUCCESS; - MPIDI_Acceptq_t *q_item, *prev; - *vcr = NULL; - q_item = acceptq_head; - prev = q_item; - - while (q_item != NULL) - { - if (q_item->port_name_tag == port_name_tag) - { - *vcr = q_item->vcr; - - if ( q_item == acceptq_head ) - acceptq_head = q_item->next; - else - prev->next = q_item->next; - - MPL_free(q_item); - AcceptQueueSize--; - break;; - } - else - { - prev = q_item; - q_item = q_item->next; - } - } - - return mpi_errno; -} - - -/** - * This routine return the list of taskids associated with world (wid) - */ -int* MPIDI_get_taskids_in_world_id(int wid) { - conn_info *tmp_node; - - MPIU_THREAD_CS_ENTER(MSGQUEUE,0); - tmp_node = _conn_info_list; - while(tmp_node != NULL) { - if(tmp_node->rem_world_id == wid) { - break; - } - tmp_node = tmp_node->next; - } - MPIU_THREAD_CS_EXIT(MSGQUEUE,0); - if(tmp_node == NULL) - return NULL; - else - return (tmp_node->rem_taskids); -} - -void MPIDI_IpState_reset(int dest) -{ - MPIDI_In_cntr_t *in_cntr; - in_cntr=&MPIDI_In_cntr[dest]; - - in_cntr->n_OutOfOrderMsgs = 0; - in_cntr->nMsgs = 0; - in_cntr->OutOfOrderList = NULL; -} - - -void MPIDI_OpState_reset(int dest) -{ - MPIDI_Out_cntr_t *out_cntr; - out_cntr=&MPIDI_Out_cntr[dest]; - - out_cntr->nMsgs = 0; - out_cntr->unmatched = 0; -} - - -/** - * This routine return the connection reference count associated with the - * remote world identified by wid - */ -int MPIDI_get_refcnt_of_world(int wid) { - conn_info *tmp_node; - - MPIU_THREAD_CS_ENTER(MSGQUEUE,0); - tmp_node = _conn_info_list; - while(tmp_node != NULL) { - if(tmp_node->rem_world_id == wid) { - break; - } - tmp_node = tmp_node->next; - } - MPIU_THREAD_CS_EXIT(MSGQUEUE,0); - if(tmp_node == NULL) - return 0; - else - return (tmp_node->ref_count); -} - -/** - * This routine delete world (wid) from linked list of known world descriptors - */ -void MPIDI_delete_conn_record(int wid) { - conn_info *tmp_node1, *tmp_node2; - - MPIU_THREAD_CS_ENTER(MSGQUEUE,0); - tmp_node1 = tmp_node2 = _conn_info_list; - while(tmp_node1) { - if(tmp_node1->rem_world_id == wid) { - if(tmp_node1 == tmp_node2) { - _conn_info_list = tmp_node1->next; - } - else { - tmp_node2->next = tmp_node1->next; - } - if(tmp_node1->rem_taskids != NULL) - MPL_free(tmp_node1->rem_taskids); - MPL_free(tmp_node1); - break; - } - tmp_node2 = tmp_node1; - tmp_node1 = tmp_node1->next; - } - MPIU_THREAD_CS_EXIT(MSGQUEUE,0); -} - - -int MPID_PG_BCast( MPIR_Comm *peercomm_p, MPIR_Comm *comm_p, int root ) -{ - int n_local_pgs=0, mpi_errno = MPI_SUCCESS; - pg_translation *local_translation = 0; - pg_node *pg_list, *pg_next, *pg_head = 0; - int rank, i, peer_comm_size; - MPIR_Errflag_t errflag = MPIR_ERR_NONE; - MPIR_CHKLMEM_DECL(1); - - peer_comm_size = comm_p->local_size; - rank = comm_p->rank; - - local_translation = (pg_translation*)MPL_malloc(peer_comm_size*sizeof(pg_translation)); - if (rank == root) { - /* Get the process groups known to the *peercomm* */ - MPIDI_ExtractLocalPGInfo( peercomm_p, local_translation, &pg_head, - &n_local_pgs ); - } - - /* Now, broadcast the number of local pgs */ - mpi_errno = MPIR_Bcast_impl( &n_local_pgs, 1, MPI_INT, root, comm_p, &errflag); - - pg_list = pg_head; - for (i=0; i<n_local_pgs; i++) { - int len, flag; - char *pg_str=0; - MPIDI_PG_t *pgptr; - - if (rank == root) { - if (!pg_list) { - /* FIXME: Error, the pg_list is broken */ - printf( "Unexpected end of pg_list\n" ); fflush(stdout); - break; - } - pg_str = pg_list->str; - len = pg_list->lenStr; - pg_list = pg_list->next; - } - mpi_errno = MPIR_Bcast_impl( &len, 1, MPI_INT, root, comm_p, &errflag); - if (rank != root) { - pg_str = (char *)MPL_malloc(len); - if (!pg_str) { - goto fn_exit; - } - } - mpi_errno = MPIR_Bcast_impl( pg_str, len, MPI_CHAR, root, comm_p, &errflag); - if (mpi_errno) { - if (rank != root) - MPL_free( pg_str ); - } - - if (rank != root) { - /* flag is true if the pg was created, false if it - already existed. This step - also initializes the created process group */ - MPIDI_PG_Create_from_string( pg_str, &pgptr, &flag ); - if (flag) { - /*printf( "[%d]Added pg named %s to list\n", rank, - (char *)pgptr->id ); - fflush(stdout); */ - } - MPL_free( pg_str ); - } - } - - /* Free pg_list */ - pg_list = pg_head; - - /* FIXME: We should use the PG destroy function for this, and ensure that - the PG fields are valid for that function */ - while (pg_list) { - pg_next = pg_list->next; - MPL_free( pg_list->str ); - if (pg_list->pg_id ) { - MPL_free( pg_list->pg_id ); - } - MPL_free( pg_list ); - pg_list = pg_next; - } - - fn_exit: - MPL_free(local_translation); - return mpi_errno; - fn_fail: - goto fn_exit; -} -#endif diff --git a/src/mpid/pamid/src/misc/Makefile.mk b/src/mpid/pamid/src/misc/Makefile.mk deleted file mode 100644 index 10f6587..0000000 --- a/src/mpid/pamid/src/misc/Makefile.mk +++ /dev/null @@ -1,33 +0,0 @@ -# begin_generated_IBM_copyright_prolog -# -# This is an automatically generated copyright prolog. -# After initializing, DO NOT MODIFY OR MOVE -# --------------------------------------------------------------- -# Licensed Materials - Property of IBM -# Blue Gene/Q 5765-PER 5765-PRP -# -# (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved -# US Government Users Restricted Rights - -# Use, duplication, or disclosure restricted -# by GSA ADP Schedule Contract with IBM Corp. -# -# --------------------------------------------------------------- -# -# end_generated_IBM_copyright_prolog -# -*- mode: makefile-gmake; -*- - -# note that the includes always happen but the effects of their contents are -# affected by "if BUILD_PAMID" -if BUILD_PAMID - - -mpi_core_sources += \ - src/mpid/pamid/src/misc/mpid_abort.c \ - src/mpid/pamid/src/misc/mpid_get_universe_size.c \ - src/mpid/pamid/src/misc/mpid_getpname.c \ - src/mpid/pamid/src/misc/mpid_mem.c \ - src/mpid/pamid/src/misc/mpid_unimpl.c - - -endif BUILD_PAMID - diff --git a/src/mpid/pamid/src/misc/mpid_abort.c b/src/mpid/pamid/src/misc/mpid_abort.c deleted file mode 100644 index 8cc669b..0000000 --- a/src/mpid/pamid/src/misc/mpid_abort.c +++ /dev/null @@ -1,103 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/misc/mpid_abort.c - * \brief Handle general operations assosicated with erroneous job termination - */ -#include <mpidimpl.h> -#include <mpix.h> - -/** - * \brief The central parts of the MPID_Abort() call - * - * \param[in] comm The communicator associated with the failure (can be null). - * \param[in] mpi_errno The MPI error associated with the failure (can be zero). - * \param[in] exit_code The requested exit code. - * \param[in] error_msg The message to display (may be NULL) - * - * This is the majority of the call to MPID_Abort(). The only - * difference is that it does not call exit. That allows it to be - * used as a test function to ensure that the output is what you would - * expect. - * - * MPIDI_Abort_core() simply uses the same params from MPID_Abort(). - */ -void MPIDI_Abort_core(MPIR_Comm * comm, int mpi_errno, int exit_code, const char *user_str) -{ - char sys_str[MPI_MAX_ERROR_STRING+5] = ""; - char comm_str[MPI_MAX_ERROR_STRING] = ""; - char world_str[MPI_MAX_ERROR_STRING] = ""; - char error_str[2*MPI_MAX_ERROR_STRING + 128]; - - if (MPIR_Process.comm_world) - { - int rank = MPIR_Process.comm_world->rank; - snprintf(world_str, sizeof(world_str), " on node %d", rank); - } - if (comm) - { - int rank = comm->rank; - int handle = comm->handle; - snprintf(comm_str, sizeof(comm_str), " (rank %d in comm %d)", rank, handle); - } - if (!user_str) - user_str = "Internal error"; - if (mpi_errno != MPI_SUCCESS) - { - char msg[MPI_MAX_ERROR_STRING] = ""; - MPIR_Err_get_string(mpi_errno, msg, MPI_MAX_ERROR_STRING, NULL); - snprintf(sys_str, sizeof(msg), " (%s)", msg); - } - - snprintf(error_str, sizeof(error_str), "Abort(%d)%s%s: %s%s\n", exit_code, world_str, comm_str, user_str, sys_str); - MPL_error_printf("%s", error_str); - - fflush(stderr); fflush(stdout); -} - -/** - * \brief The central parts of the MPID_Abort call - * \param[in] comm The communicator associated with the failure (can be null). - * \param[in] mpi_errno The MPI error associated with the failure (can be zero). - * \param[in] exit_code The requested exit code. - * \param[in] error_msg The message to display (may be NULL) - * \return MPI_ERR_INTERN - * - * This function MUST NEVER return. - */ -int MPID_Abort(MPIR_Comm * comm, int mpi_errno, int exit_code, const char *error_msg) -{ - MPIDI_Abort_core(comm, mpi_errno, exit_code, error_msg); - -#ifdef DYNAMIC_TASKING - extern int mpidi_dynamic_tasking; - if (mpidi_dynamic_tasking) - return PMI2_Abort(1,error_msg); -#endif - - /* The POE and BGQ control systems both catch the exit value for additional - * processing. If a process exits with '1' then all processes in the job - * are terminated. The requested error code is lost in this process however - * this is acceptable, but not desirable, behavior according to the MPI - * standard. - * - * On BGQ, the user may force the process (rank) that exited with '1' to core - * dump by setting the environment variable 'BG_COREDUMPONERROR=1'. - */ - exit(1); -} diff --git a/src/mpid/pamid/src/misc/mpid_get_universe_size.c b/src/mpid/pamid/src/misc/mpid_get_universe_size.c deleted file mode 100644 index 87049bb..0000000 --- a/src/mpid/pamid/src/misc/mpid_get_universe_size.c +++ /dev/null @@ -1,96 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/misc/mpid_get_universe_size.c - * \brief ??? - */ -/* - * (C) 2001 by Argonne National Laboratory. - * See COPYRIGHT in top-level directory. - */ - -#include <mpidimpl.h> - -#ifdef DYNAMIC_TASKING -#ifdef USE_PMI2_API -#include "pmi2.h" -#else -#include "pmi.h" -#endif /* USE_PMI2_API */ -#endif /* DYNAMIC_TASKING */ - -extern int mpidi_dynamic_tasking; - -/* - * MPID_Get_universe_size - Get the universe size from the process manager - * - * Notes: This requires that the PMI routines are used to - * communicate with the process manager. - */ -int MPID_Get_universe_size(int * universe_size) -{ - int mpi_errno = MPI_SUCCESS; -#ifdef DYNAMIC_TASKING -#ifdef USE_PMI2_API - if(mpidi_dynamic_tasking) { - char val[PMI2_MAX_VALLEN]; - int found = 0; - char *endptr; - - mpi_errno = PMI2_Info_GetJobAttr("universeSize", val, sizeof(val), &found); - TRACE_ERR("mpi_errno from PMI2_Info_GetJobAttr=%d\n", mpi_errno); - - if (!found) { - TRACE_ERR("PMI2_Info_GetJobAttr not found\n"); - *universe_size = MPIR_UNIVERSE_SIZE_NOT_AVAILABLE; - } - else { - *universe_size = strtol(val, &endptr, 0); - TRACE_ERR("PMI2_Info_GetJobAttr found universe_size=%d\n", *universe_size); - } -#else - int pmi_errno = PMI_SUCCESS; - - pmi_errno = PMI_Get_universe_size(universe_size); - if (pmi_errno != PMI_SUCCESS) { - MPIU_ERR_SETANDJUMP1(mpi_errno, MPI_ERR_OTHER, - "**pmi_get_universe_size", - "**pmi_get_universe_size %d", pmi_errno); - } - if (*universe_size < 0) - { - *universe_size = MPIR_UNIVERSE_SIZE_NOT_AVAILABLE; - } -#endif /*USE_PMI2_API*/ - } else { - *universe_size = MPIR_UNIVERSE_SIZE_NOT_AVAILABLE; - } -#else - *universe_size = MPIR_UNIVERSE_SIZE_NOT_AVAILABLE; -#endif /*DYNAMIC_TASKING*/ - - -fn_exit: - return mpi_errno; - - /* --BEGIN ERROR HANDLING-- */ -fn_fail: - *universe_size = MPIR_UNIVERSE_SIZE_NOT_AVAILABLE; - goto fn_exit; - /* --END ERROR HANDLING-- */ -} diff --git a/src/mpid/pamid/src/misc/mpid_getpname.c b/src/mpid/pamid/src/misc/mpid_getpname.c deleted file mode 100644 index c0fbf26..0000000 --- a/src/mpid/pamid/src/misc/mpid_getpname.c +++ /dev/null @@ -1,46 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/misc/mpid_getpname.c - * \brief Device interface to MPI_Get_processor_name() - */ -#include <mpidimpl.h> - -/** - * \brief Device interface to MPI_Get_processor_name() - * \param[out] name Storage for the name as a string - * \param[in] namelen The maximum allowed length - * \param[out] resultlen The actual length written - * \returns MPI_SUCCESS - * - * All this does is convert the rank to a string and return the data - */ -int MPID_Get_processor_name(char * name, int namelen, int * resultlen) -{ - /* Get the name from PAMI */ - const char* pami_name = PAMIX_Client_query(MPIDI_Client, PAMI_CLIENT_PROCESSOR_NAME).value.chararray; - /* Copy to the destination */ - strncpy(name, pami_name, namelen); - /* Ensure that there is a trailing NULL */ - if (namelen > 0) - name[namelen - 1]= '\0'; - /* Get the size of the name */ - *resultlen = strlen(name); - - return MPI_SUCCESS; -} diff --git a/src/mpid/pamid/src/misc/mpid_mem.c b/src/mpid/pamid/src/misc/mpid_mem.c deleted file mode 100644 index 2cd3f55..0000000 --- a/src/mpid/pamid/src/misc/mpid_mem.c +++ /dev/null @@ -1,42 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/misc/mpid_mem.c - * \brief ??? - */ -/* - * - * (C) 2001 by Argonne National Laboratory. - * See COPYRIGHT in top-level directory. - */ - -#include <mpidimpl.h> - -void *MPID_Alloc_mem( size_t size, MPIR_Info *info_ptr ) -{ - void *ap; - ap = MPL_malloc(size); - return ap; -} - -int MPID_Free_mem( void *ptr ) -{ - int mpi_errno = MPI_SUCCESS; - MPL_free(ptr); - return mpi_errno; -} diff --git a/src/mpid/pamid/src/misc/mpid_unimpl.c b/src/mpid/pamid/src/misc/mpid_unimpl.c deleted file mode 100644 index 6b33086..0000000 --- a/src/mpid/pamid/src/misc/mpid_unimpl.c +++ /dev/null @@ -1,112 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/misc/mpid_unimpl.c - * \brief These are functions that are not supported - */ -#include <mpidimpl.h> - -#ifndef DYNAMIC_TASKING -int MPID_Close_port(const char *port_name) -{ - MPID_abort(); - return 0; -} -int MPID_Open_port(MPIR_Info *info_ptr, - char *port_name) -{ - MPID_abort(); - return 0; -} - -int MPID_Comm_accept(const char *port_name, - MPIR_Info *info_ptr, - int root, - MPIR_Comm *comm_ptr, - MPIR_Comm **newcomm) -{ - MPID_abort(); - return 0; -} -int MPID_Comm_connect(const char *port_name, - MPIR_Info *info_ptr, - int root, - MPIR_Comm *comm_ptr, - MPIR_Comm **newcomm) -{ - MPID_abort(); - return 0; -} -int MPID_Comm_disconnect(MPIR_Comm *comm_ptr) -{ - MPID_abort(); - return 0; -} -int MPID_Comm_spawn_multiple(int count, - char *array_of_commands[], - char* *array_of_argv[], - const int array_of_maxprocs[], - MPIR_Info *array_of_info[], - int root, - MPIR_Comm *comm_ptr, - MPIR_Comm **intercomm, - int array_of_errcodes[]) -{ - MPID_abort(); - return 0; -} -#endif - -int MPID_Comm_failure_ack(MPIR_Comm *comm_ptr) -{ - MPID_abort(); - return 0; -} - -int MPID_Comm_failure_get_acked(MPIR_Comm *comm_ptr, MPIR_Group **failed_group_ptr) -{ - MPID_abort(); - return 0; -} - -int MPID_Comm_get_all_failed_procs(MPIR_Comm *comm_ptr, MPIR_Group **failed_group, int tag) -{ - MPID_abort(); - return 0; -} - -int MPID_Comm_revoke(MPIR_Comm *comm_ptr, int is_remote) -{ - MPID_abort(); - return 0; -} - -int MPID_Comm_AS_enabled(MPIR_Comm *comm_ptr) -{ - /* This function must return 1 in the default case and should not be ignored - * by the implementation. */ - return 1; -} - -int MPID_Request_is_anysource(MPIR_Request *request_ptr) -{ - /* This function must not abort in the default case since it is used in many - * MPI functions. As long as the device does not implement FT, it doesn't - * matter what this function returns. */ - return 0; -} diff --git a/src/mpid/pamid/src/mpid_aint.c b/src/mpid/pamid/src/mpid_aint.c deleted file mode 100644 index e41459d..0000000 --- a/src/mpid/pamid/src/mpid_aint.c +++ /dev/null @@ -1,61 +0,0 @@ -/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ -/* - * (C) 2014 by Argonne National Laboratory. - * See COPYRIGHT in top-level directory. - */ - -#include "mpidimpl.h" - -/* - Input Parameters: -+ base - base address (integer) -- disp - displacement (integer) - - Return value: - A new MPI_Aint value that is equivalent to the sum of the base and disp - arguments, where base represents a base address returned by a call - to MPI_GET_ADDRESS and disp represents a signed integer displacement. -*/ -#undef FUNCNAME -#define FUNCNAME MPID_Aint_add -#undef FCNAME -#define FCNAME MPIU_QUOTE(FUNCNAME) -MPI_Aint MPID_Aint_add(MPI_Aint base, MPI_Aint disp) -{ - MPI_Aint result; - - MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPID_AINT_ADD); - MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPID_AINT_ADD); - - result = MPIR_VOID_PTR_CAST_TO_MPI_AINT ((char*)MPIR_AINT_CAST_TO_VOID_PTR(base) + disp); - - MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPID_AINT_ADD); - return result; -} - -/* - Input Parameters: -+ addr1 - minuend address (integer) -- addr2 - subtrahend address (integer) - - Return value: - A new MPI_Aint value that is equivalent to the difference between addr1 and - addr2 arguments, where addr1 and addr2 represent addresses returned by calls - to MPI_GET_ADDRESS. -*/ -#undef FUNCNAME -#define FUNCNAME MPID_Aint_diff -#undef FCNAME -#define FCNAME MPIU_QUOTE(FUNCNAME) -MPI_Aint MPID_Aint_diff(MPI_Aint addr1, MPI_Aint addr2) -{ - MPI_Aint result; - - MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPID_AINT_DIFF); - MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPID_AINT_DIFF); - - result = MPIR_PTR_DISP_CAST_TO_MPI_AINT ((char*)MPIR_AINT_CAST_TO_VOID_PTR(addr1) - (char*)MPIR_AINT_CAST_TO_VOID_PTR(addr2)); - - MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPID_AINT_DIFF); - return result; -} diff --git a/src/mpid/pamid/src/mpid_buffer.c b/src/mpid/pamid/src/mpid_buffer.c deleted file mode 100644 index 5129d37..0000000 --- a/src/mpid/pamid/src/mpid_buffer.c +++ /dev/null @@ -1,250 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/mpid_buffer.c - * \brief MPID buffer copy - */ -/* - * (C) 2001 by Argonne National Laboratory. - * See COPYRIGHT in top-level directory. - */ - -#include <mpidimpl.h> - -#ifndef MPIDI_COPY_BUFFER_SZ -#define MPIDI_COPY_BUFFER_SZ 16384 -#endif - -/** - * \brief MPID buffer copy - * - * Implements non-contiguous buffers correctly. - * - * \param[in] sbuf The address of the input buffer - * \param[in] scount The number of elements in that buffer - * \param[in] sdt The datatype of those elements - * \param[out] smpi_errno Returns errors - * \param[in] rbuf The address of the output buffer - * \param[out] rcount The number of elements in that buffer - * \param[in] rdt The datatype of those elements - * \param[out] rsz The size of the ouput data - * \param[out] rmpi_errno Returns errors - */ -void MPIDI_Buffer_copy( - const void * const sbuf, MPI_Aint scount, MPI_Datatype sdt, int * smpi_errno, - void * const rbuf, MPI_Aint rcount, MPI_Datatype rdt, intptr_t * rsz, int * rmpi_errno) -{ - int sdt_contig; - int rdt_contig; - MPI_Aint sdt_true_lb, rdt_true_lb; - intptr_t sdata_sz; - intptr_t rdata_sz; - MPIDU_Datatype* sdt_ptr; - MPIDU_Datatype* rdt_ptr; - - MPI_Aint sdt_extent; - MPI_Aint rdt_extent; - - *smpi_errno = MPI_SUCCESS; - *rmpi_errno = MPI_SUCCESS; - - /* printf("bufcopy: src count=%d dt=%d\n", scount, sdt); */ - /* printf("bufcopy: dst count=%d dt=%d\n", rcount, rdt); */ - - MPIDI_Datatype_get_info(scount, sdt, sdt_contig, sdata_sz, sdt_ptr, sdt_true_lb); - MPIDI_Datatype_get_info(rcount, rdt, rdt_contig, rdata_sz, rdt_ptr, rdt_true_lb); - - /* --BEGIN ERROR HANDLING-- */ - if (sdata_sz > rdata_sz) - { - *rmpi_errno = MPIR_Err_create_code(MPI_SUCCESS, MPIR_ERR_RECOVERABLE, __FUNCTION__, __LINE__, MPI_ERR_TRUNCATE, "**truncate", "**truncate %d %d", sdata_sz, rdata_sz ); - sdata_sz = rdata_sz; - } - /* --END ERROR HANDLING-- */ - - if (sdata_sz == 0) - { - *rsz = 0; - goto fn_exit; - } - - if (sdt_contig && rdt_contig) - { -#if CUDA_AWARE_SUPPORT - if(MPIDI_Process.cuda_aware_support_on && MPIDI_cuda_is_device_buf(rbuf)) - { - cudaError_t cudaerr = CudaMemcpy(rbuf + rdt_true_lb, sbuf + sdt_true_lb, sdata_sz, cudaMemcpyHostToDevice); - } - else -#endif - memcpy((char*)rbuf + rdt_true_lb, (const char *)sbuf + sdt_true_lb, sdata_sz); - *rsz = sdata_sz; - } - else if (sdt_contig) - { -#if CUDA_AWARE_SUPPORT - // This will need to be done in two steps: - // 1 - Allocate a temp buffer which is the same size as user buffer and unpack in it. - // 2 - Copy unpacked data into user buffer from temp buffer. - if(MPIDI_Process.cuda_aware_support_on && MPIDI_cuda_is_device_buf(rbuf)) - { - MPIDU_Datatype_get_extent_macro(rdt, rdt_extent); - char *buf = MPL_malloc(rdt_extent * rcount); - memset(buf, 0, rdt_extent * rcount); - MPIDU_Segment seg; - DLOOP_Offset last; - - MPIDU_Segment_init(buf, rcount, rdt, &seg, 0); - last = sdata_sz; - MPIDU_Segment_unpack(&seg, 0, &last, (char*)sbuf + sdt_true_lb); - /* --BEGIN ERROR HANDLING-- */ - if (last != sdata_sz) - { - *rmpi_errno = MPIR_Err_create_code(MPI_SUCCESS, MPIR_ERR_RECOVERABLE, __FUNCTION__, __LINE__, MPI_ERR_TYPE, "**dtypemismatch", 0); - } - /* --END ERROR HANDLING-- */ - - *rsz = last; - - - cudaError_t cudaerr = CudaMemcpy(rbuf + rdt_true_lb, buf, rdt_extent * rcount, cudaMemcpyHostToDevice); - - MPL_free(buf); - - goto fn_exit; - - } -#endif - - MPIDU_Segment seg; - DLOOP_Offset last; - - MPIDU_Segment_init(rbuf, rcount, rdt, &seg, 0); - last = sdata_sz; - MPIDU_Segment_unpack(&seg, 0, &last, (char*)sbuf + sdt_true_lb); - /* --BEGIN ERROR HANDLING-- */ - if (last != sdata_sz) - { - *rmpi_errno = MPIR_Err_create_code(MPI_SUCCESS, MPIR_ERR_RECOVERABLE, __FUNCTION__, __LINE__, MPI_ERR_TYPE, "**dtypemismatch", 0); - } - /* --END ERROR HANDLING-- */ - - *rsz = last; - } - else if (rdt_contig) - { - MPIDU_Segment seg; - DLOOP_Offset last; - - MPIDU_Segment_init(sbuf, scount, sdt, &seg, 0); - last = sdata_sz; - MPIDU_Segment_pack(&seg, 0, &last, (char*)rbuf + rdt_true_lb); - /* --BEGIN ERROR HANDLING-- */ - if (last != sdata_sz) - { - *rmpi_errno = MPIR_Err_create_code(MPI_SUCCESS, MPIR_ERR_RECOVERABLE, __FUNCTION__, __LINE__, MPI_ERR_TYPE, "**dtypemismatch", 0); - } - /* --END ERROR HANDLING-- */ - - *rsz = last; - } - else - { - char * buf; - intptr_t buf_off; - MPIDU_Segment sseg; - intptr_t sfirst; - MPIDU_Segment rseg; - intptr_t rfirst; - - buf = MPL_malloc(MPIDI_COPY_BUFFER_SZ); - /* --BEGIN ERROR HANDLING-- */ - if (buf == NULL) - { - *smpi_errno = MPIR_Err_create_code(MPI_SUCCESS, MPIR_ERR_FATAL, __FUNCTION__, __LINE__, MPI_ERR_OTHER, "**nomem", 0); - *rmpi_errno = *smpi_errno; - *rsz = 0; - goto fn_exit; - } - /* --END ERROR HANDLING-- */ - - MPIDU_Segment_init(sbuf, scount, sdt, &sseg, 0); - MPIDU_Segment_init(rbuf, rcount, rdt, &rseg, 0); - - sfirst = 0; - rfirst = 0; - buf_off = 0; - - for(;;) - { - DLOOP_Offset last; - char * buf_end; - - if (sdata_sz - sfirst > MPIDI_COPY_BUFFER_SZ - buf_off) - { - last = sfirst + (MPIDI_COPY_BUFFER_SZ - buf_off); - } - else - { - last = sdata_sz; - } - - MPIDU_Segment_pack(&sseg, sfirst, &last, buf + buf_off); - /* --BEGIN ERROR HANDLING-- */ - MPID_assert(last > sfirst); - /* --END ERROR HANDLING-- */ - - buf_end = buf + buf_off + (last - sfirst); - sfirst = last; - - MPIDU_Segment_unpack(&rseg, rfirst, &last, buf); - /* --BEGIN ERROR HANDLING-- */ - MPID_assert(last > rfirst); - /* --END ERROR HANDLING-- */ - - rfirst = last; - - if (rfirst == sdata_sz) - { - /* successful completion */ - break; - } - - /* --BEGIN ERROR HANDLING-- */ - if (sfirst == sdata_sz) - { - /* datatype mismatch -- remaining bytes could not be unpacked */ - *rmpi_errno = MPIR_Err_create_code(MPI_SUCCESS, MPIR_ERR_RECOVERABLE, __FUNCTION__, __LINE__, MPI_ERR_TYPE, "**dtypemismatch", 0); - break; - } - /* --END ERROR HANDLING-- */ - - buf_off = sfirst - rfirst; - if (buf_off > 0) - { - memmove(buf, buf_end - buf_off, buf_off); - } - } - - *rsz = rfirst; - MPL_free(buf); - } - - fn_exit: - return; -} diff --git a/src/mpid/pamid/src/mpid_finalize.c b/src/mpid/pamid/src/mpid_finalize.c deleted file mode 100644 index 2790369..0000000 --- a/src/mpid/pamid/src/mpid_finalize.c +++ /dev/null @@ -1,149 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/mpid_finalize.c - * \brief Normal job termination code - */ -#include <mpidimpl.h> - - -#if TOKEN_FLOW_CONTROL -extern void MPIDI_close_mm(); -#endif - -#ifdef MPIDI_STATISTICS -extern pami_extension_t pe_extension; - -extern int mpidi_dynamic_tasking; -int mpidi_finalized = 0; -#ifdef DYNAMIC_TASKING -extern conn_info *_conn_info_list; -#endif - - -void MPIDI_close_pe_extension() { - extern MPIDI_printenv_t *mpich_env; - extern MPIX_stats_t *mpid_statp; - int rc; - /* PAMI_Extension_open in pami_init */ - rc = PAMI_Extension_close (pe_extension); - if (rc != PAMI_SUCCESS) { - TRACE_ERR("ERROR close PAMI_Extension failed rc %d", rc); - } - if (mpich_env) - MPL_free(mpich_env); - if (mpid_statp) - MPL_free(mpid_statp); - -} -#endif - -/** - * \brief Shut down the system - * - * At this time, no attempt is made to free memory being used for MPI structures. - * \return MPI_SUCCESS -*/ -int MPID_Finalize() -{ - pami_result_t rc; - int mpierrno = MPI_SUCCESS; - MPIR_Errflag_t errflag=MPIR_ERR_NONE; - MPIR_Barrier_impl(MPIR_Process.comm_world, &errflag); - -#ifdef MPIDI_STATISTICS - if (MPIDI_Process.mp_statistics) { - MPIDI_print_statistics(); - } - MPIDI_close_pe_extension(); -#endif - -#ifdef DYNAMIC_TASKING - mpidi_finalized = 1; - if(mpidi_dynamic_tasking) { - /* Tell the process group code that we're done with the process groups. - This will notify PMI (with PMI_Finalize) if necessary. It - also frees all PG structures, including the PG for COMM_WORLD, whose - pointer is also saved in MPIDI_Process.my_pg */ - mpierrno = MPIDI_PG_Finalize(); - if (mpierrno) { - TRACE_ERR("MPIDI_PG_Finalize returned with mpierrno=%d\n", mpierrno); - } - - MPIDI_FreeParentPort(); - } - if(_conn_info_list) - MPL_free(_conn_info_list); - MPIDI_free_all_tranid_node(); -#endif - - - /* ------------------------- */ - /* shutdown request queues */ - /* ------------------------- */ - MPIDI_Recvq_finalize(); - - PAMIX_Finalize(MPIDI_Client); - -#ifdef MPID_NEEDS_ICOMM_WORLD - MPIR_Comm_release_always(MPIR_Process.icomm_world, 0); -#endif - - MPIR_Comm_release_always(MPIR_Process.comm_self,0); - MPIR_Comm_release_always(MPIR_Process.comm_world,0); - - rc = PAMI_Context_destroyv(MPIDI_Context, MPIDI_Process.avail_contexts); - MPID_assert_always(rc == PAMI_SUCCESS); - - rc = PAMI_Client_destroy(&MPIDI_Client); - MPID_assert_always(rc == PAMI_SUCCESS); - -#ifdef MPIDI_TRACE - { int i; - for (i=0; i< MPIDI_Process.numTasks; i++) { - if (MPIDI_Trace_buf[i].R) - MPL_free(MPIDI_Trace_buf[i].R); - if (MPIDI_Trace_buf[i].PR) - MPL_free(MPIDI_Trace_buf[i].PR); - if (MPIDI_Trace_buf[i].S) - MPL_free(MPIDI_Trace_buf[i].S); - } - } - MPL_free(MPIDI_Trace_buf); -#endif - -#ifdef OUT_OF_ORDER_HANDLING - MPL_free(MPIDI_In_cntr); - MPL_free(MPIDI_Out_cntr); -#endif - - if (TOKEN_FLOW_CONTROL_ON) - { - #if TOKEN_FLOW_CONTROL - extern char *EagerLimit; - - if (EagerLimit) MPL_free(EagerLimit); - MPL_free(MPIDI_Token_cntr); - MPIDI_close_mm(); - #else - MPID_assert_always(0); - #endif - } - - return MPI_SUCCESS; -} diff --git a/src/mpid/pamid/src/mpid_improbe.c b/src/mpid/pamid/src/mpid_improbe.c deleted file mode 100644 index 97e8d7a..0000000 --- a/src/mpid/pamid/src/mpid_improbe.c +++ /dev/null @@ -1,49 +0,0 @@ -/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ -/* - * (C) 2012 by Argonne National Laboratory. - * See COPYRIGHT in top-level directory. - */ - -#include "mpidimpl.h" - -int MPID_Improbe(int source, int tag, MPIR_Comm *comm, int context_offset, - int *flag, MPIR_Request **message, MPI_Status *status) -{ - const int context = comm->recvcontext_id + context_offset; - int foundp = FALSE; - MPIR_Request * rreq = NULL; - - if (source == MPI_PROC_NULL) - { - MPIR_Status_set_procnull(status); - /* We set the flag to true because an MPI_Recv with this rank will - * return immediately */ - *flag = TRUE; - *message = NULL; - return MPI_SUCCESS; - } - -#ifndef OUT_OF_ORDER_HANDLING - rreq = MPIDI_Recvq_FDU(source, tag, context, &foundp); -#else - int pami_source; - if(source != MPI_ANY_SOURCE) { - pami_source = MPID_VCR_GET_LPID(comm->vcr, source); - } else { - pami_source = MPI_ANY_SOURCE; - } - rreq = MPIDI_Recvq_FDU(source, pami_source, tag, context, &foundp); -#endif - if (rreq==NULL) { - MPID_Progress_poke(); - } - else { - rreq->kind = MPIR_REQUEST_KIND__MPROBE; - MPIR_Request_extract_status(rreq, status); - } - - *message = rreq; - *flag = foundp; - return MPI_SUCCESS; -} - diff --git a/src/mpid/pamid/src/mpid_imrecv.c b/src/mpid/pamid/src/mpid_imrecv.c deleted file mode 100644 index 93ad50b..0000000 --- a/src/mpid/pamid/src/mpid_imrecv.c +++ /dev/null @@ -1,91 +0,0 @@ -/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ -/* - * (C) 2012 by Argonne National Laboratory. - * See COPYRIGHT in top-level directory. - */ - -#include "mpidimpl.h" - -int MPID_Imrecv(void *buf, int count, MPI_Datatype datatype, - MPIR_Request *message, MPIR_Request **rreqp) -{ - int mpi_errno = MPI_SUCCESS; - - MPIR_Request * rreq; - - /* ---------------------------------------- */ - /* NULL rank means empty request */ - /* ---------------------------------------- */ - if (unlikely(message == NULL)) - { - rreq = MPIDI_Request_create2(); - MPIR_Status_set_procnull(&rreq->status); - rreq->kind = MPIR_REQUEST_KIND__RECV; - MPIDI_Request_complete(rreq); - *rreqp = rreq; - return MPI_SUCCESS; - - } - - MPIR_Assert(message != NULL); - MPIR_Assert(message->kind == MPIR_REQUEST_KIND__MPROBE); - - /* promote the request object to be a "real" recv request */ - message->kind = MPIR_REQUEST_KIND__RECV; - - *rreqp = rreq = message; - -#ifdef OUT_OF_ORDER_HANDLING - if ((MPIDI_In_cntr[rreq->mpid.peer_pami].n_OutOfOrderMsgs>0)) - MPIDI_Recvq_process_out_of_order_msgs(rreq->mpid.peer_pami, MPIDI_Context[0]); -#endif - -#if (MPIDI_STATISTICS) - MPID_NSTAT(mpid_statp->recvs); -#endif - -#ifdef MPIDI_TRACE -{ - size_t ll; - ll = count * MPIDU_Datatype_get_basic_size(datatype); - /*MPIDI_SET_PR_REC(rreq,buf,count,ll,datatype,pami_source,rank,tag,comm,is_blocking); */ -} -#endif - - /* ----------------------------------------------------------------- */ - /* populate request with our data */ - /* We can do this because this is not a multithreaded implementation */ - /* ----------------------------------------------------------------- */ - - rreq->mpid.userbuf = buf; - rreq->mpid.userbufcount = count; - rreq->mpid.datatype = datatype; - MPIDI_RecvMsg_Unexp(rreq, buf, count, datatype); - mpi_errno = rreq->status.MPI_ERROR; - if (TOKEN_FLOW_CONTROL_ON) { - #if TOKEN_FLOW_CONTROL - if ((rreq->mpid.uebuflen) && (!(rreq->mpid.envelope.msginfo.isRzv))) { - MPIDI_Token_cntr[(rreq->mpid.peer_pami)].unmatched--; - MPIDI_Update_rettoks(rreq->mpid.peer_pami); - } - MPIDI_Must_return_tokens(MPIDI_Context[0],(rreq->mpid.peer_pami)); - #else - MPID_assert_always(0); - #endif - } - -#ifdef OUT_OF_ORDER_HANDLING - if ((MPIDI_In_cntr[rreq->mpid.peer_pami].n_OutOfOrderMsgs>0)) - MPIDI_Recvq_process_out_of_order_msgs(rreq->mpid.peer_pami, MPIDI_Context[0]); -#endif - -#ifdef MPIDI_STATISTICS - if (!(MPIR_cc_is_complete(&rreq->cc))) - { - MPID_NSTAT(mpid_statp->recvWaitsComplete); - } -#endif - - return mpi_errno; -} - diff --git a/src/mpid/pamid/src/mpid_init.c b/src/mpid/pamid/src/mpid_init.c deleted file mode 100644 index 20f3236..0000000 --- a/src/mpid/pamid/src/mpid_init.c +++ /dev/null @@ -1,1703 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/mpid_init.c - * \brief Normal job startup code - */ - -#include <stdlib.h> -#include <string.h> - -#include <mpidimpl.h> -#include "mpidi_platform.h" -#include "onesided/mpidi_onesided.h" - -#include "mpidi_util.h" - -#ifdef DYNAMIC_TASKING -#define PAMIX_CLIENT_DYNAMIC_TASKING 1032 -#define PAMIX_CLIENT_WORLD_TASKS 1033 -#define MAX_JOBID_LEN 1024 -int world_rank; -int world_size; -extern int (*mp_world_exiting_handler)(int); -extern int _mpi_world_exiting_handler(int); -#endif -int mpidi_dynamic_tasking = 0; - -#if TOKEN_FLOW_CONTROL - extern int MPIDI_mm_init(int,uint *,unsigned long *); - extern int MPIDI_tfctrl_enabled; -#endif - -#if (MPIDI_STATISTICS || MPIDI_PRINTENV) - pami_extension_t pe_extension; -#endif - -pami_client_t MPIDI_Client; -pami_context_t MPIDI_Context[MPIDI_MAX_CONTEXTS]; - -MPIDI_Process_t MPIDI_Process = { - .verbose = 0, - .statistics = 0, - -#if (MPICH_THREAD_GRANULARITY == MPICH_THREAD_GRANULARITY__POBJ) - .avail_contexts = MPIDI_MAX_CONTEXTS, - .async_progress = { - .active = 0, - .mode = ASYNC_PROGRESS_MODE_DEFAULT, - }, - .perobj = { - .context_post = { - .requested = (ASYNC_PROGRESS_MODE_DEFAULT == ASYNC_PROGRESS_MODE_LOCKED), - .active = 0, - }, - }, -#else - .avail_contexts = 1, - .async_progress = { - .active = 0, - .mode = ASYNC_PROGRESS_MODE_DISABLED, - }, - .perobj = { - .context_post = { - .requested = 0, - .active = 0, - }, - }, -#endif - .pt2pt = { - .limits = { - .application = { - .eager = { - .remote = MPIDI_EAGER_LIMIT, - .local = MPIDI_EAGER_LIMIT_LOCAL, - }, - .immediate = { - .remote = MPIDI_SHORT_LIMIT, - .local = MPIDI_SHORT_LIMIT, - }, - }, - .internal = { - .eager = { - .remote = MPIDI_EAGER_LIMIT, - .local = MPIDI_EAGER_LIMIT_LOCAL, - }, - .immediate = { - .remote = MPIDI_SHORT_LIMIT, - .local = MPIDI_SHORT_LIMIT, - }, - }, - }, - }, - .disable_internal_eager_scale = MPIDI_DISABLE_INTERNAL_EAGER_SCALE, -#if TOKEN_FLOW_CONTROL - .mp_buf_mem = BUFFER_MEM_DEFAULT, - .mp_buf_mem_max = BUFFER_MEM_DEFAULT, - .is_token_flow_control_on = 0, -#endif -#if (MPIDI_STATISTICS || MPIDI_PRINTENV) - .mp_infolevel = 0, - .mp_statistics = 0, - .mp_printenv = 0, -#endif -#ifdef QUEUE_BINARY_SEARCH_SUPPORT - .queue_binary_search_support_on = 0, -#endif -#if CUDA_AWARE_SUPPORT - .cuda_aware_support_on = 0, -#endif - .rma_pending = 1000, - .shmem_pt2pt = 1, - .smp_detect = MPIDI_SMP_DETECT_DEFAULT, - .optimized = { - .collectives = MPIDI_OPTIMIZED_COLLECTIVE_DEFAULT, - .subcomms = 1, - .select_colls = 2, - .memory = 0, - .num_requests = 16, - }, - - .mpir_nbc = 1, - .numTasks = 0, - .typed_onesided = 0, -}; - - -struct protocol_t -{ - pami_dispatch_p2p_function func; - size_t dispatch; - size_t immediate_min; - pami_dispatch_hint_t options; -}; -static struct -{ - struct protocol_t Short; - struct protocol_t ShortSync; - struct protocol_t Eager; - struct protocol_t RVZ; - struct protocol_t Cancel; - struct protocol_t Control; - struct protocol_t WinCtrl; - struct protocol_t WinAccum; - struct protocol_t RVZ_zerobyte; - struct protocol_t WinGetAccum; - struct protocol_t WinGetAccumAck; - struct protocol_t WinAtomic; - struct protocol_t WinAtomicAck; -#ifdef DYNAMIC_TASKING - struct protocol_t Dyntask; - struct protocol_t Dyntask_disconnect; -#endif -} proto_list = { - .Short = { - .func = MPIDI_RecvShortAsyncCB, - .dispatch = MPIDI_Protocols_Short, - .options = { - .consistency = USE_PAMI_CONSISTENCY, - .long_header = PAMI_HINT_DISABLE, - .recv_immediate = PAMI_HINT_ENABLE, - .use_rdma = PAMI_HINT_DISABLE, - }, - .immediate_min = sizeof(MPIDI_MsgInfo), - }, - .ShortSync = { - .func = MPIDI_RecvShortSyncCB, - .dispatch = MPIDI_Protocols_ShortSync, - .options = { - .consistency = USE_PAMI_CONSISTENCY, - .long_header = PAMI_HINT_DISABLE, - .recv_immediate = PAMI_HINT_ENABLE, - .use_rdma = PAMI_HINT_DISABLE, - }, - .immediate_min = sizeof(MPIDI_MsgInfo), - }, - .Eager = { - .func = MPIDI_RecvCB, - .dispatch = MPIDI_Protocols_Eager, - .options = { - .consistency = USE_PAMI_CONSISTENCY, - .long_header = PAMI_HINT_DISABLE, - .recv_contiguous = PAMI_HINT_ENABLE, - .recv_copy = PAMI_HINT_ENABLE, - }, - .immediate_min = sizeof(MPIDI_MsgInfo), - }, - .RVZ = { - .func = MPIDI_RecvRzvCB, - .dispatch = MPIDI_Protocols_RVZ, - .options = { - .consistency = USE_PAMI_CONSISTENCY, - .long_header = PAMI_HINT_DISABLE, - .recv_immediate = PAMI_HINT_ENABLE, - .use_rdma = PAMI_HINT_DISABLE, - }, - .immediate_min = sizeof(MPIDI_MsgEnvelope), - }, - .Cancel = { - .func = MPIDI_ControlCB, - .dispatch = MPIDI_Protocols_Cancel, - .options = { - .consistency = PAMI_HINT_ENABLE, - .long_header = PAMI_HINT_DISABLE, - .recv_immediate = PAMI_HINT_ENABLE, - .use_rdma = PAMI_HINT_DISABLE, - }, - .immediate_min = sizeof(MPIDI_MsgInfo), - }, - .Control = { - .func = MPIDI_ControlCB, - .dispatch = MPIDI_Protocols_Control, - .options = { - .long_header = PAMI_HINT_DISABLE, - .recv_immediate = PAMI_HINT_ENABLE, - .use_rdma = PAMI_HINT_DISABLE, - }, - .immediate_min = sizeof(MPIDI_MsgInfo), - }, - .WinCtrl = { - .func = MPIDI_WinControlCB, - .dispatch = MPIDI_Protocols_WinCtrl, - .options = { - .long_header = PAMI_HINT_DISABLE, - .recv_immediate = PAMI_HINT_ENABLE, - .use_rdma = PAMI_HINT_DISABLE, - }, - .immediate_min = sizeof(MPIDI_Win_control_t), - }, - .WinAccum = { - .func = MPIDI_WinAccumCB, - .dispatch = MPIDI_Protocols_WinAccum, - .options = { - .consistency = PAMI_HINT_ENABLE, - .long_header = PAMI_HINT_DISABLE, - .recv_immediate = PAMI_HINT_DISABLE, - }, - .immediate_min = sizeof(MPIDI_MsgInfo), - }, - .RVZ_zerobyte = { - .func = MPIDI_RecvRzvCB_zerobyte, - .dispatch = MPIDI_Protocols_RVZ_zerobyte, - .options = { - .consistency = USE_PAMI_CONSISTENCY, - .long_header = PAMI_HINT_DISABLE, - .recv_immediate = PAMI_HINT_ENABLE, - .use_rdma = PAMI_HINT_DISABLE, - }, - .immediate_min = sizeof(MPIDI_MsgEnvelope), - }, - .WinGetAccum = { - .func = MPIDI_WinGetAccumCB, - .dispatch = MPIDI_Protocols_WinGetAccum, - .options = { - .consistency = PAMI_HINT_ENABLE, - .long_header = PAMI_HINT_DISABLE, - .recv_immediate = PAMI_HINT_DISABLE, - }, - .immediate_min = sizeof(MPIDI_Win_GetAccMsgInfo), - }, - .WinGetAccumAck = { - .func = MPIDI_WinGetAccumAckCB, - .dispatch = MPIDI_Protocols_WinGetAccumAck, - .options = { - .consistency = PAMI_HINT_ENABLE, - .long_header = PAMI_HINT_DISABLE, - .recv_immediate = PAMI_HINT_DISABLE, - }, - .immediate_min = sizeof(MPIDI_Win_GetAccMsgInfo), - }, - .WinAtomic = { - .func = MPIDI_WinAtomicCB, - .dispatch = MPIDI_Protocols_WinAtomic, - .options = { - .consistency = USE_PAMI_CONSISTENCY, - .long_header = PAMI_HINT_DISABLE, - .recv_immediate = PAMI_HINT_ENABLE, - .use_rdma = PAMI_HINT_DISABLE, - }, - .immediate_min = sizeof(MPIDI_AtomicHeader_t), - }, - .WinAtomicAck = { - .func = MPIDI_WinAtomicAckCB, - .dispatch = MPIDI_Protocols_WinAtomicAck, - .options = { - .consistency = USE_PAMI_CONSISTENCY, - .long_header = PAMI_HINT_DISABLE, - .recv_immediate = PAMI_HINT_ENABLE, - .use_rdma = PAMI_HINT_DISABLE, - }, - .immediate_min = sizeof(MPIDI_AtomicHeader_t), - }, -#ifdef DYNAMIC_TASKING - .Dyntask = { - .func = MPIDI_Recvfrom_remote_world, - .dispatch = MPIDI_Protocols_Dyntask, - .options = { - .consistency = USE_PAMI_CONSISTENCY, - .long_header = PAMI_HINT_DISABLE, - .recv_immediate = PAMI_HINT_ENABLE, - .use_rdma = PAMI_HINT_DISABLE, - }, - .immediate_min = sizeof(MPIDI_MsgInfo), - }, - .Dyntask_disconnect = { - .func = MPIDI_Recvfrom_remote_world_disconnect, - .dispatch = MPIDI_Protocols_Dyntask_disconnect, - .options = { - .consistency = USE_PAMI_CONSISTENCY, - .long_header = PAMI_HINT_DISABLE, - .recv_immediate = PAMI_HINT_ENABLE, - .use_rdma = PAMI_HINT_DISABLE, - }, - .immediate_min = sizeof(MPIDI_MsgInfo), - }, -#endif -}; - - -#undef FUNCNAME -#define FUNCNAME split_type -#undef FCNAME -#define FCNAME MPIU_QUOTE(FUNCNAME) -static int split_type(MPIR_Comm * comm_ptr, int stype, int key, - MPIR_Info *info_ptr, MPIR_Comm ** newcomm_ptr) -{ - MPID_Node_id_t id; - int nid; - int mpi_errno = MPI_SUCCESS; - - mpi_errno = MPID_Get_node_id(comm_ptr, comm_ptr->rank, &id); - if (mpi_errno) MPIU_ERR_POP(mpi_errno); - - nid = (stype == MPI_COMM_TYPE_SHARED) ? id : MPI_UNDEFINED; - mpi_errno = MPIR_Comm_split_impl(comm_ptr, nid, key, newcomm_ptr); - if (mpi_errno) MPIU_ERR_POP(mpi_errno); - - fn_exit: - return mpi_errno; - - /* --BEGIN ERROR HANDLING-- */ - fn_fail: - goto fn_exit; - /* --END ERROR HANDLING-- */ -} - -static MPIR_Commops comm_fns = { - split_type -}; - - -/* ------------------------------ */ -/* Collective selection extension */ -/* ------------------------------ */ -pami_extension_t MPIDI_Collsel_extension; -advisor_t MPIDI_Collsel_advisor; -advisor_table_t MPIDI_Collsel_advisor_table; -advisor_params_t MPIDI_Collsel_advisor_params; -char *MPIDI_Collsel_output_file; -pami_extension_collsel_advise MPIDI_Pamix_collsel_advise; -static void -MPIDI_PAMI_client_init(int* rank, int* size, int* mpidi_dynamic_tasking, char **world_tasks) -{ - /* ------------------------------------ */ - /* Initialize the MPICH->PAMI Client */ - /* ------------------------------------ */ - pami_result_t rc = PAMI_ERROR; - - pami_configuration_t config[2]; - size_t numconfigs = 0; - - /* Set the status for memory optimized collectives */ - { - char* env = getenv("PAMID_COLLECTIVES_MEMORY_OPTIMIZED"); - if (env != NULL) - MPIDI_atoi(env,&MPIDI_Process.optimized.memory); - } - -#ifdef HAVE_PAMI_CLIENT_NONCONTIG - config[0].name = PAMI_CLIENT_NONCONTIG; - if(MPIDI_Process.optimized.memory & MPIR_OPT_LVL_NONCONTIG) - config[0].value.intval = 0; // Disable non-contig, pamid doesn't use pami for non-contig data collectives so save memory - else - config[0].value.intval = 1; // Enable non-contig even though pamid doesn't use pami for non-contig data collectives, - // we still possibly want those collectives for other reasons. - ++numconfigs; -#endif -#ifdef HAVE_PAMI_CLIENT_MEMORY_OPTIMIZE - if(MPIDI_Process.optimized.memory) - { - config[numconfigs].name = PAMI_CLIENT_MEMORY_OPTIMIZE; - config[numconfigs].value.intval = MPIDI_Process.optimized.memory; - ++numconfigs; - } -#endif - - rc = PAMI_Client_create("MPI", &MPIDI_Client, config, numconfigs); - MPID_assert_always(rc == PAMI_SUCCESS); - PAMIX_Initialize(MPIDI_Client); - - - *mpidi_dynamic_tasking=0; -#ifdef DYNAMIC_TASKING - *world_tasks = NULL; - pami_result_t status = PAMI_ERROR; - - typedef pami_result_t (*dyn_task_query_fn) ( - pami_client_t client, - pami_configuration_t config[], - size_t num_configs); - dyn_task_query_fn dyn_task_query = NULL; - - pami_extension_t extension; - status = PAMI_Extension_open (MPIDI_Client, "PE_dyn_task", &extension); - if(status != PAMI_SUCCESS) - { - TRACE_ERR("Error. The PE_dyn_task extension is not implemented. result = %d\n", status); - } - - dyn_task_query = (dyn_task_query_fn) PAMI_Extension_symbol(extension, "query"); - if (dyn_task_query == (void*)NULL) { - TRACE_ERR("Err: the Dynamic Tasking extension function dyn_task_query is not implememted.\n"); - - } else { - pami_configuration_t config2[] = - { - {PAMI_CLIENT_TASK_ID, -1}, - {PAMI_CLIENT_NUM_TASKS, -1}, - {(pami_attribute_name_t)PAMIX_CLIENT_DYNAMIC_TASKING}, - {(pami_attribute_name_t)PAMIX_CLIENT_WORLD_TASKS}, - }; - - dyn_task_query(MPIDI_Client, config2, 4); - TRACE_ERR("dyn_task_query: task_id %d num_tasks %d dynamic_tasking %d world_tasks %s\n", - config2[0].value.intval, - config2[1].value.intval, - config2[2].value.intval, - config2[3].value.chararray); - *rank = world_rank = config2[0].value.intval; - *size = world_size = config2[1].value.intval; - *mpidi_dynamic_tasking = config2[2].value.intval; - *world_tasks = config2[3].value.chararray; - } - - status = PAMI_Extension_close (extension); - if(status != PAMI_SUCCESS) - { - TRACE_ERR("Error. The PE_dyn_task extension could not be closed. result = %d\n", status); - } -#endif - - if(*mpidi_dynamic_tasking == 0) { - /* ---------------------------------- */ - /* Get my rank and the process size */ - /* ---------------------------------- */ - *rank = PAMIX_Client_query(MPIDI_Client, PAMI_CLIENT_TASK_ID ).value.intval; - MPIR_Process.comm_world->rank = *rank; /* Set the rank early to make tracing better */ - *size = PAMIX_Client_query(MPIDI_Client, PAMI_CLIENT_NUM_TASKS).value.intval; - } - - /* --------------------------------------------------------------- */ - /* Determine if the eager point-to-point protocol for internal mpi */ - /* operations should be disabled. */ - /* --------------------------------------------------------------- */ - { - char * env = getenv("PAMID_DISABLE_INTERNAL_EAGER_TASK_LIMIT"); - if (env != NULL) - { - size_t n = strlen(env); - char * tmp = (char *) MPL_malloc(n+1); - strncpy(tmp,env,n); - if (n>0) tmp[n]=0; - - MPIDI_atoi(tmp, &MPIDI_Process.disable_internal_eager_scale); - - MPL_free(tmp); - } - - if (MPIDI_Process.disable_internal_eager_scale <= *size) - { - MPIDI_Process.pt2pt.limits.internal.eager.remote = 0; - MPIDI_Process.pt2pt.limits.internal.eager.local = 0; - MPIDI_Process.pt2pt.limits.internal.immediate.remote = 0; - MPIDI_Process.pt2pt.limits.internal.immediate.local = 0; - } - } -} - -void MPIDI_Init_collsel_extension() -{ - pami_result_t status = PAMI_ERROR; - status = PAMI_Extension_open (MPIDI_Client, "EXT_collsel", &MPIDI_Collsel_extension); - if(status == PAMI_SUCCESS) - { - if(MPIDI_Process.optimized.auto_select_colls == MPID_AUTO_SELECT_COLLS_TUNE) - { - advisor_configuration_t configuration[1]; - pami_extension_collsel_init pamix_collsel_init = - (pami_extension_collsel_init) PAMI_Extension_symbol (MPIDI_Collsel_extension, "Collsel_init"); - status = pamix_collsel_init (MPIDI_Client, configuration, 1, &MPIDI_Context[0], 1, &MPIDI_Collsel_advisor); - if(status != PAMI_SUCCESS) - { - fprintf (stderr, "Error. The collsel_init failed. result = %d\n", status); - MPIDI_Process.optimized.auto_select_colls = MPID_AUTO_SELECT_COLLS_NONE; - } - - } - else if(MPIDI_Process.optimized.auto_select_colls == MPID_AUTO_SELECT_COLLS_ALL) - { - pami_extension_collsel_initialized pamix_collsel_initialized = - (pami_extension_collsel_initialized) PAMI_Extension_symbol(MPIDI_Collsel_extension, - "Collsel_initialized"); - if(pamix_collsel_initialized(MPIDI_Client, &MPIDI_Collsel_advisor) == 1) - { - char *collselfile; - collselfile = getenv("MP_COLLECTIVE_SELECTION_FILE"); - pami_extension_collsel_table_load pamix_collsel_table_load = - (pami_extension_collsel_table_load) PAMI_Extension_symbol(MPIDI_Collsel_extension, - "Collsel_table_load"); - if(collselfile != NULL) - status = pamix_collsel_table_load(MPIDI_Collsel_advisor, collselfile, &MPIDI_Collsel_advisor_table); - else - status = pamix_collsel_table_load(MPIDI_Collsel_advisor, "pami_tune_results.xml", &MPIDI_Collsel_advisor_table); - if (status == PAMI_SUCCESS) - { - pami_xfer_type_t *collsel_collectives = NULL; - unsigned num_collectives; - pami_extension_collsel_get_collectives pamix_collsel_get_collectives = - (pami_extension_collsel_get_collectives) PAMI_Extension_symbol(MPIDI_Collsel_extension, - "Collsel_get_collectives"); - status = pamix_collsel_get_collectives(MPIDI_Collsel_advisor_table, &collsel_collectives, &num_collectives); - MPIDI_Process.optimized.auto_select_colls = MPID_AUTO_SELECT_COLLS_NONE; - if(collsel_collectives != NULL) - { - unsigned i = 0; - for(i = 0; i < num_collectives; i++) - { - switch(collsel_collectives[i]) - { - case PAMI_XFER_BROADCAST: - MPIDI_Process.optimized.auto_select_colls |= MPID_AUTO_SELECT_COLLS_BCAST; - break; - case PAMI_XFER_ALLREDUCE: - MPIDI_Process.optimized.auto_select_colls |= MPID_AUTO_SELECT_COLLS_ALLREDUCE; - break; - case PAMI_XFER_REDUCE: - MPIDI_Process.optimized.auto_select_colls |= MPID_AUTO_SELECT_COLLS_REDUCE; - break; - case PAMI_XFER_ALLGATHER: - MPIDI_Process.optimized.auto_select_colls |= MPID_AUTO_SELECT_COLLS_ALLGATHER; - break; - case PAMI_XFER_ALLGATHERV: -#if (MPIDI_STATISTICS || MPIDI_PRINTENV) - if(MPIDI_Process.mp_infolevel >= 1) - fprintf(stderr,"WARNING: MPICH (collective selection) doesn't support ALLGATHERV, only ALLGATHERV_INT is supported\n"); -#endif - break; - case PAMI_XFER_ALLGATHERV_INT: - MPIDI_Process.optimized.auto_select_colls |= MPID_AUTO_SELECT_COLLS_ALLGATHERV; - break; - case PAMI_XFER_SCATTER: - MPIDI_Process.optimized.auto_select_colls |= MPID_AUTO_SELECT_COLLS_SCATTER; - break; - case PAMI_XFER_SCATTERV: -#if (MPIDI_STATISTICS || MPIDI_PRINTENV) - if(MPIDI_Process.mp_infolevel >= 1) - fprintf(stderr,"WARNING: MPICH (collective selection) doesn't support SCATTERV, only SCATTERV_INT is supported\n"); -#endif - break; - case PAMI_XFER_SCATTERV_INT: - MPIDI_Process.optimized.auto_select_colls |= MPID_AUTO_SELECT_COLLS_SCATTERV; - break; - case PAMI_XFER_GATHER: - MPIDI_Process.optimized.auto_select_colls |= MPID_AUTO_SELECT_COLLS_GATHER; - break; - case PAMI_XFER_GATHERV: -#if (MPIDI_STATISTICS || MPIDI_PRINTENV) - if(MPIDI_Process.mp_infolevel >= 1) - fprintf(stderr,"WARNING: MPICH (collective selection) doesn't support GATHERV, only GATHERV_INT is supported\n"); -#endif - break; - case PAMI_XFER_GATHERV_INT: - MPIDI_Process.optimized.auto_select_colls |= MPID_AUTO_SELECT_COLLS_GATHERV; - break; - case PAMI_XFER_BARRIER: - MPIDI_Process.optimized.auto_select_colls |= MPID_AUTO_SELECT_COLLS_BARRIER; - break; - case PAMI_XFER_ALLTOALL: - MPIDI_Process.optimized.auto_select_colls |= MPID_AUTO_SELECT_COLLS_ALLTOALL; - break; - case PAMI_XFER_ALLTOALLV: -#if (MPIDI_STATISTICS || MPIDI_PRINTENV) - if(MPIDI_Process.mp_infolevel >= 1) - fprintf(stderr,"WARNING: MPICH (collective selection) doesn't support ALLTOALLV, only ALLTOALLV_INT is supported\n"); -#endif - break; - case PAMI_XFER_ALLTOALLV_INT: - MPIDI_Process.optimized.auto_select_colls |= MPID_AUTO_SELECT_COLLS_ALLTOALLV; - break; - case PAMI_XFER_SCAN: - MPIDI_Process.optimized.auto_select_colls |= MPID_AUTO_SELECT_COLLS_SCAN; - MPIDI_Process.optimized.auto_select_colls |= MPID_AUTO_SELECT_COLLS_EXSCAN; - break; - case PAMI_XFER_REDUCE_SCATTER: - MPIDI_Process.optimized.auto_select_colls |= MPID_AUTO_SELECT_COLLS_REDUCE_SCATTER; - break; - default: - MPIDI_Process.optimized.auto_select_colls |= MPID_AUTO_SELECT_COLLS_NONE; - } - } - MPL_free(collsel_collectives); - } - } - else - { - fprintf (stderr, "Error. Collsel_table_load failed. result = %d\n", status); - MPIDI_Process.optimized.auto_select_colls = MPID_AUTO_SELECT_COLLS_NONE; - } - } - else - MPIDI_Process.optimized.auto_select_colls = MPID_AUTO_SELECT_COLLS_NONE; - } - else - PAMI_Extension_close(MPIDI_Collsel_extension); - } - else - MPIDI_Process.optimized.auto_select_colls = MPID_AUTO_SELECT_COLLS_NONE; - -#ifndef __BGQ__ - //If collective selection will be disabled, check on fca and CUDA if both not required, disable pami alltogether - if(MPIDI_Process.optimized.auto_select_colls == MPID_AUTO_SELECT_COLLS_NONE && MPIDI_Process.optimized.collectives != MPID_COLL_FCA && MPIDI_Process.optimized.collectives != MPID_COLL_CUDA) - MPIDI_Process.optimized.collectives = MPID_COLL_OFF; -#endif -} - -void MPIDI_Collsel_table_generate() -{ - external_geometry_ops_t external_ops; - external_ops.geometry_create = MPIDI_Comm_create_from_pami_geom; - external_ops.geometry_destroy = MPIDI_Comm_destroy_external; - external_ops.register_algorithms = MPIDI_Register_algorithms_ext; - pami_result_t status = PAMI_SUCCESS; - pami_extension_collsel_table_generate pamix_collsel_table_generate = - (pami_extension_collsel_table_generate) PAMI_Extension_symbol (MPIDI_Collsel_extension, "Collsel_table_generate"); - - status = pamix_collsel_table_generate (MPIDI_Collsel_advisor, MPIDI_Collsel_output_file, &MPIDI_Collsel_advisor_params, &external_ops, 1); - if(status != PAMI_SUCCESS) - { - fprintf (stderr, "Error. The collsel_table_generate failed. result = %d\n", status); - } - -} - - -static void -MPIDI_PAMI_context_init(int* threading, int *size) -{ -#ifdef TRACE_ON - int requested_thread_level; - requested_thread_level = *threading; -#endif - int numTasks; - -#if (MPICH_THREAD_GRANULARITY == MPICH_THREAD_GRANULARITY__POBJ) - /* - * ASYNC_PROGRESS_MODE_LOCKED requires context post because the async thread - * will hold the context lock indefinitely; the only option for an application - * thread to interact with the context is to use context post. See discussion - * in src/mpid/pamid/src/mpid_progress.h for more information. - * - * There are three possible resolutions for the situation when context post is - * disabled and async progess mode is 'locked': - * 1. abort - * 2. silently enable context post - * 3. silently demote async progress mode to ASYNC_PROGRESS_MODE_TRIGGER - * - * For now this configuration is considered erroneous and mpi will abort. - */ - if (MPIDI_Process.async_progress.mode == ASYNC_PROGRESS_MODE_LOCKED && - MPIDI_Process.perobj.context_post.requested == 0) - MPID_Abort (NULL, 0, 1, "'locking' async progress requires context post"); - -#else /* MPICH_THREAD_GRANULARITY != MPICH_THREAD_GRANULARITY__POBJ */ - /* - * ASYNC_PROGRESS_MODE_LOCKED is not applicable in the "global lock" thread - * mode. See discussion in src/mpid/pamid/src/mpid_progress.h for more - * information. - * - * This configuration is considered erroneous and mpi will abort. - */ - if (MPIDI_Process.async_progress.mode == ASYNC_PROGRESS_MODE_LOCKED) - MPID_Abort (NULL, 0, 1, "'locking' async progress not applicable"); -#endif - - /* ---------------------------------- - * Figure out the context situation - * ---------------------------------- */ -#if (MPICH_THREAD_GRANULARITY == MPICH_THREAD_GRANULARITY__POBJ) - - /* Limit the number of requested contexts by the maximum number of contexts - * allowed. The default number of requested contexts depends on the mpich - * lock mode, 'global' or 'perobj', and may be changed before this point - * by an environment variables. - */ - if (MPIDI_Process.avail_contexts > MPIDI_MAX_CONTEXTS) - MPIDI_Process.avail_contexts = MPIDI_MAX_CONTEXTS; - - unsigned same = PAMIX_Client_query(MPIDI_Client, PAMI_CLIENT_CONST_CONTEXTS).value.intval; - if (same) - { - /* Determine the maximum number of contexts supported; limit the number of - * requested contexts by this value. - */ - unsigned possible_contexts = PAMIX_Client_query(MPIDI_Client, PAMI_CLIENT_NUM_CONTEXTS).value.intval; - TRACE_ERR("PAMI allows up to %u contexts; MPICH allows up to %u\n", - possible_contexts, MPIDI_Process.avail_contexts); - if (MPIDI_Process.avail_contexts > possible_contexts) - MPIDI_Process.avail_contexts = possible_contexts; - } - else - { - /* If PAMI didn't give all nodes the same number of contexts, all bets - * are off for now */ - MPIDI_Process.avail_contexts = 1; - } - - /* The number of contexts must be a power-of-two, as required by the - * MPIDI_Context_hash() function. Decrement until we hit a power-of-two */ - while(MPIDI_Process.avail_contexts & (MPIDI_Process.avail_contexts-1)) - --MPIDI_Process.avail_contexts; - MPID_assert_always(MPIDI_Process.avail_contexts); - -#else /* (MPICH_THREAD_GRANULARITY != MPICH_THREAD_GRANULARITY__POBJ) */ - - /* Only a single context is supported in the 'global' mpich lock mode. - * - * A multi-context application will always perform better with the - * 'per object' mpich lock mode - regardless of whether async progress is - * enabled or not. This is because all threads, application and async - * progress, must acquire the single global lock which effectively serializes - * the threads and negates any benefit of multiple contexts. - * - * This single context limitation removes code and greatly simplifies logic. - */ - MPIDI_Process.avail_contexts = 1; - -#endif - - TRACE_ERR ("Thread-level=%d, requested=%d\n", *threading, requested_thread_level); - - MPIDI_Process.numTasks= numTasks = PAMIX_Client_query(MPIDI_Client, PAMI_CLIENT_NUM_TASKS).value.intval; -#ifdef OUT_OF_ORDER_HANDLING - MPIDI_In_cntr = MPL_calloc0(numTasks, MPIDI_In_cntr_t); - if(MPIDI_In_cntr == NULL) - MPID_abort(); - MPIDI_Out_cntr = MPL_calloc0(numTasks, MPIDI_Out_cntr_t); - if(MPIDI_Out_cntr == NULL) - MPID_abort(); - memset((void *) MPIDI_In_cntr,0, sizeof(MPIDI_In_cntr_t)); - memset((void *) MPIDI_Out_cntr,0, sizeof(MPIDI_Out_cntr_t)); -#endif - - -#ifdef MPIDI_TRACE - int i; - MPIDI_Trace_buf = MPL_calloc0(numTasks, MPIDI_Trace_buf_t); - if(MPIDI_Trace_buf == NULL) MPID_abort(); - memset((void *) MPIDI_Trace_buf,0, sizeof(MPIDI_Trace_buf_t)); - for (i=0; i < numTasks; i++) { - MPIDI_Trace_buf[i].R=MPL_calloc0(N_MSGS, recv_status); - if (MPIDI_Trace_buf[i].R==NULL) MPID_abort(); - MPIDI_Trace_buf[i].PR=MPL_calloc0(N_MSGS, posted_recv); - if (MPIDI_Trace_buf[i].PR ==NULL) MPID_abort(); - MPIDI_Trace_buf[i].S=MPL_calloc0(N_MSGS, send_status); - if (MPIDI_Trace_buf[i].S ==NULL) MPID_abort(); - } -#endif - - /* ----------------------------------- */ - /* Create the communication contexts */ - /* ----------------------------------- */ - TRACE_ERR("Creating %d contexts\n", MPIDI_Process.avail_contexts); - pami_result_t rc = PAMI_ERROR; - pami_configuration_t config[3]; - int cfgval=0; - config[cfgval].name = PAMI_CLIENT_CONST_CONTEXTS, - config[cfgval].value.intval = 1; - cfgval++; -#ifndef HAVE_ERROR_CHECKING -#ifdef OUT_OF_ORDER_HANDLING - /* disable parameter checking in PAMI - fast library only */ - config[cfgval].name = PAMI_CONTEXT_CHECK_PARAM; - config[cfgval].value.intval = 0; - cfgval++; -#endif -#endif - rc = PAMI_Context_createv(MPIDI_Client, config, cfgval, MPIDI_Context, MPIDI_Process.avail_contexts); - - MPID_assert_always(rc == PAMI_SUCCESS); - - /* --------------------------------------------- */ - /* Get collective selection advisor and cache it */ - /* --------------------------------------------- */ - /* Context is created, i.e. collective selection extension is initialized in PAMI. Now I can get the - advisor if I am not in TUNE mode. If in TUNE mode, I can init collsel and generate the table. - This is not supported on BGQ. - */ -#ifndef __BGQ_ - MPIDI_Init_collsel_extension(); -#endif - -#if (MPIDI_STATISTICS || MPIDI_PRINTENV) - MPIDI_open_pe_extension(); -#endif -} - - -static void -MPIDI_PAMI_dispath_set(size_t dispatch, - struct protocol_t * proto, - unsigned * immediate_max) -{ - size_t im_max = 0; - pami_dispatch_callback_function Recv = {.p2p = proto->func}; - MPID_assert_always(dispatch == proto->dispatch); - - if (MPIDI_Process.shmem_pt2pt == 0) - proto->options.use_shmem = PAMI_HINT_DISABLE; - - PAMIX_Dispatch_set(MPIDI_Context, - MPIDI_Process.avail_contexts, - proto->dispatch, - Recv, - proto->options, - &im_max); - TRACE_ERR("Immediate-max query: dispatch=%zu got=%zu required=%zu\n", - dispatch, im_max, proto->immediate_min); - MPID_assert_always(proto->immediate_min <= im_max); - if ((immediate_max != NULL) && (im_max < *immediate_max)) - *immediate_max = im_max; -} - - -static void -MPIDI_PAMI_dispath_init() -{ -#ifdef OUT_OF_ORDER_HANDLING - { - pami_configuration_t config; - pami_result_t rc = PAMI_ERROR; - - memset(&config, 0, sizeof(config)); - config.name = PAMI_DISPATCH_SEND_IMMEDIATE_MAX; - rc = PAMI_Dispatch_query(MPIDI_Context[0], (size_t)0, &config, 1); - if ( rc == PAMI_SUCCESS ) - { - TRACE_ERR("PAMI_DISPATCH_SEND_IMMEDIATE_MAX=%d.\n", config.value.intval, rc); - MPIDI_Process.pt2pt.limits_array[2] = config.value.intval; - } - else - { - TRACE_ERR(" Attention: PAMI_Client_query(DISPATCH_SEND_IMMEDIATE_MAX=%d) rc=%d\n", config.name, rc); - MPIDI_Process.pt2pt.limits_array[2] = 256; - } - - MPIDI_Process.pt2pt.limits_array[3] = MPIDI_Process.pt2pt.limits_array[2]; - MPIDI_Process.pt2pt.limits_array[6] = MPIDI_Process.pt2pt.limits_array[2]; - MPIDI_Process.pt2pt.limits_array[7] = MPIDI_Process.pt2pt.limits_array[2]; - } -#endif - /* ------------------------------------ */ - /* Set up the communication protocols */ - /* ------------------------------------ */ - unsigned send_immediate_max_bytes = (unsigned) -1; - MPIDI_PAMI_dispath_set(MPIDI_Protocols_Short, &proto_list.Short, &send_immediate_max_bytes); - MPIDI_PAMI_dispath_set(MPIDI_Protocols_ShortSync, &proto_list.ShortSync, &send_immediate_max_bytes); - MPIDI_PAMI_dispath_set(MPIDI_Protocols_Eager, &proto_list.Eager, NULL); - MPIDI_PAMI_dispath_set(MPIDI_Protocols_RVZ, &proto_list.RVZ, NULL); - MPIDI_PAMI_dispath_set(MPIDI_Protocols_Cancel, &proto_list.Cancel, NULL); - MPIDI_PAMI_dispath_set(MPIDI_Protocols_Control, &proto_list.Control, NULL); - MPIDI_PAMI_dispath_set(MPIDI_Protocols_WinCtrl, &proto_list.WinCtrl, NULL); - MPIDI_PAMI_dispath_set(MPIDI_Protocols_WinAccum, &proto_list.WinAccum, NULL); - MPIDI_PAMI_dispath_set(MPIDI_Protocols_RVZ_zerobyte, &proto_list.RVZ_zerobyte, NULL); - MPIDI_PAMI_dispath_set(MPIDI_Protocols_WinGetAccum, &proto_list.WinGetAccum, NULL); - MPIDI_PAMI_dispath_set(MPIDI_Protocols_WinGetAccumAck, &proto_list.WinGetAccumAck, NULL); - MPIDI_PAMI_dispath_set(MPIDI_Protocols_WinAtomic, &proto_list.WinAtomic, NULL); - MPIDI_PAMI_dispath_set(MPIDI_Protocols_WinAtomicAck, &proto_list.WinAtomicAck, NULL); - -#ifdef DYNAMIC_TASKING - MPIDI_PAMI_dispath_set(MPIDI_Protocols_Dyntask, &proto_list.Dyntask, NULL); - MPIDI_PAMI_dispath_set(MPIDI_Protocols_Dyntask_disconnect, &proto_list.Dyntask_disconnect, NULL); -#endif - - /* - * The first two protocols are our short protocols: they use - * PAMI_Send_immediate() exclusively. We get the short limit twice - * because they could be different. - * - * - The returned value is the max amount of header+data. We have - * to remove the header size. - * - * - We need to add one back, since we don't use "=" in the - * comparison. We use "if (size < short_limit) ...". - * - * - We use the min of the results just to be safe. - */ - send_immediate_max_bytes -= (sizeof(MPIDI_MsgInfo) - 1); - - if (MPIDI_Process.pt2pt.limits.application.immediate.remote > send_immediate_max_bytes) - MPIDI_Process.pt2pt.limits.application.immediate.remote = send_immediate_max_bytes; - - if (MPIDI_Process.pt2pt.limits.application.immediate.local > send_immediate_max_bytes) - MPIDI_Process.pt2pt.limits.application.immediate.local = send_immediate_max_bytes; - - if (MPIDI_Process.pt2pt.limits.internal.immediate.remote > send_immediate_max_bytes) - MPIDI_Process.pt2pt.limits.internal.immediate.remote = send_immediate_max_bytes; - - if (MPIDI_Process.pt2pt.limits.internal.immediate.local > send_immediate_max_bytes) - MPIDI_Process.pt2pt.limits.internal.immediate.local = send_immediate_max_bytes; - - if (TOKEN_FLOW_CONTROL_ON) - { - #if TOKEN_FLOW_CONTROL - int i; - MPIDI_mm_init(MPIDI_Process.numTasks,&MPIDI_Process.pt2pt.limits.application.eager.remote,&MPIDI_Process.mp_buf_mem); - MPIDI_Token_cntr = MPL_calloc0(MPIDI_Process.numTasks, MPIDI_Token_cntr_t); - memset((void *) MPIDI_Token_cntr,0, (sizeof(MPIDI_Token_cntr_t) * MPIDI_Process.numTasks)); - for (i=0; i < MPIDI_Process.numTasks; i++) - { - MPIDI_Token_cntr[i].tokens=MPIDI_tfctrl_enabled; - } - #else - MPID_assert_always(0); - #endif - } -} - - - -extern char **environ; -static void -printEnvVars(char *type) -{ - printf("The following %s* environment variables were specified:\n", type); - char **env; - for(env = environ; *env != 0 ; env++) - { - if(!strncasecmp(*env, type, strlen(type))) - printf(" %s\n", *env); - } -} - - -static void -MPIDI_PAMI_init(int* rank, int* size, int* threading) -{ - MPIDI_PAMI_context_init(threading, size); - - - MPIDI_PAMI_dispath_init(); - - - if ( (*rank == 0) && (MPIDI_Process.verbose >= MPIDI_VERBOSE_SUMMARY_0) ) - { - printf("MPIDI_Process.*\n" - " verbose : %u\n" - " statistics : %u\n" - " contexts : %u\n" - " async_progress : %u\n" - " context_post : %u\n" - " pt2pt.limits\n" - " application\n" - " eager\n" - " remote, local : %u, %u\n" - " short\n" - " remote, local : %u, %u\n" - " internal\n" - " eager\n" - " remote, local : %u, %u\n" - " short\n" - " remote, local : %u, %u\n" - " rma_pending : %u\n" - " shmem_pt2pt : %u\n" - " disable_internal_eager_scale : %u\n" -#if TOKEN_FLOW_CONTROL - " mp_buf_mem : %u\n" - " mp_buf_mem_max : %u\n" - " is_token_flow_control_on : %u\n" -#endif -#if (MPIDI_STATISTICS || MPIDI_PRINTENV) - " mp_infolevel : %u\n" - " mp_statistics: %u\n" - " mp_printenv : %u\n" - " mp_interrupts: %u\n" -#endif -#ifdef QUEUE_BINARY_SEARCH_SUPPORT - " queue_binary_search_support_on : %u\n" -#endif - " optimized.collectives : %u\n" - " optimized.select_colls: %u\n" - " optimized.subcomms : %u\n" - " optimized.memory : %u\n" - " optimized.num_requests: %u\n" - " mpir_nbc : %u\n" - " numTasks : %u\n", - " typed_onesided : %u\n", - MPIDI_Process.verbose, - MPIDI_Process.statistics, - MPIDI_Process.avail_contexts, - MPIDI_Process.async_progress.mode, - MPIDI_Process.perobj.context_post.requested, - MPIDI_Process.pt2pt.limits_array[0], - MPIDI_Process.pt2pt.limits_array[1], - MPIDI_Process.pt2pt.limits_array[2], - MPIDI_Process.pt2pt.limits_array[3], - MPIDI_Process.pt2pt.limits_array[4], - MPIDI_Process.pt2pt.limits_array[5], - MPIDI_Process.pt2pt.limits_array[6], - MPIDI_Process.pt2pt.limits_array[7], - MPIDI_Process.rma_pending, - MPIDI_Process.shmem_pt2pt, - MPIDI_Process.disable_internal_eager_scale, -#if TOKEN_FLOW_CONTROL - MPIDI_Process.mp_buf_mem, - MPIDI_Process.mp_buf_mem_max, - MPIDI_Process.is_token_flow_control_on, -#endif -#if (MPIDI_STATISTICS || MPIDI_PRINTENV) - MPIDI_Process.mp_infolevel, - MPIDI_Process.mp_statistics, - MPIDI_Process.mp_printenv, - (MPIDI_Process.async_progress.mode != ASYNC_PROGRESS_MODE_DISABLED), -#endif -#ifdef QUEUE_BINARY_SEARCH_SUPPORT - MPIDI_Process.queue_binary_search_support_on, -#endif - MPIDI_Process.optimized.collectives, - MPIDI_Process.optimized.select_colls, - MPIDI_Process.optimized.subcomms, - MPIDI_Process.optimized.memory, - MPIDI_Process.optimized.num_requests, - MPIDI_Process.mpir_nbc, - MPIDI_Process.numTasks, - MPIDI_Process.typed_onesided); - switch (*threading) - { - case MPI_THREAD_MULTIPLE: - printf("mpi thread level : 'MPI_THREAD_MULTIPLE'\n"); - break; - case MPI_THREAD_SERIALIZED: - printf("mpi thread level : 'MPI_THREAD_SERIALIZED'\n"); - break; - case MPI_THREAD_FUNNELED: - printf("mpi thread level : 'MPI_THREAD_FUNNELED'\n"); - break; - case MPI_THREAD_SINGLE: - printf("mpi thread level : 'MPI_THREAD_SINGLE'\n"); - break; - } - printf("MPICH_THREAD_GRANULARITY : '%s'\n", - (MPICH_THREAD_GRANULARITY==MPICH_THREAD_GRANULARITY__POBJ)?"per object":"global"); -#ifdef ASSERT_LEVEL - printf("ASSERT_LEVEL : %d\n", ASSERT_LEVEL); -#else - printf("ASSERT_LEVEL : not defined\n"); -#endif -#ifdef MPICH_LIBDIR - printf("MPICH_LIBDIR : %s\n", MPICH_LIBDIR); -#else - printf("MPICH_LIBDIR : not defined\n"); -#endif - printEnvVars("MPICH_"); - printEnvVars("PAMID_"); - printEnvVars("PAMI_"); - printEnvVars("COMMAGENT_"); - printEnvVars("MUSPI_"); - printEnvVars("BG_"); - } -#ifdef MPIDI_BANNER - if ((*rank == 0) && (MPIDI_Process.mp_infolevel >=1)) { - char* buf = (char *) MPL_malloc(160); - int rc = MPIDI_Banner(buf); - if ( rc == 0 ) - fprintf(stderr, "%s\n", buf); - else - TRACE_ERR("mpid_banner() return code=%d task %d",rc,*rank); - MPL_free(buf); - } -#endif -} - -#ifndef DYNAMIC_TASKING -static void -MPIDI_VCRT_init(int rank, int size) -#else -static void -MPIDI_VCRT_init(int rank, int size, char *world_tasks, MPIDI_PG_t *pg) -#endif -{ - int i, rc; - MPIR_Comm * comm; -#ifdef DYNAMIC_TASKING - int p, mpi_errno=0; - char *world_tasks_save,*cp; - char *pg_id; -#endif - - /* ------------------------------- */ - /* Initialize MPI_COMM_SELF object */ - /* ------------------------------- */ - comm = MPIR_Process.comm_self; - comm->rank = 0; - comm->remote_size = comm->local_size = 1; - rc = MPID_VCRT_Create(comm->remote_size, &comm->vcrt); - MPID_assert_always(rc == MPI_SUCCESS); - rc = MPID_VCRT_Get_ptr(comm->vcrt, &comm->vcr); - MPID_assert_always(rc == MPI_SUCCESS); - comm->vcr[0]->taskid= PAMIX_Client_query(MPIDI_Client, PAMI_CLIENT_TASK_ID ).value.intval; - -#ifdef DYNAMIC_TASKING - if(mpidi_dynamic_tasking) { - comm->vcr[0]->pg=pg->vct[rank].pg; - comm->vcr[0]->pg_rank=pg->vct[rank].pg_rank; - pg->vct[rank].taskid = comm->vcr[0]->taskid; - if(comm->vcr[0]->pg) { - TRACE_ERR("Adding ref for comm=%x vcr=%x pg=%x\n", comm, comm->vcr[0], comm->vcr[0]->pg); - MPIDI_PG_add_ref(comm->vcr[0]->pg); - } - comm->local_vcr = comm->vcr; - } - -#endif - - /* -------------------------------- */ - /* Initialize MPI_COMM_WORLD object */ - /* -------------------------------- */ - comm = MPIR_Process.comm_world; - comm->rank = rank; - comm->remote_size = comm->local_size = size; - rc = MPID_VCRT_Create(comm->remote_size, &comm->vcrt); - MPID_assert_always(rc == MPI_SUCCESS); - rc = MPID_VCRT_Get_ptr(comm->vcrt, &comm->vcr); - MPID_assert_always(rc == MPI_SUCCESS); - -#ifdef DYNAMIC_TASKING - if(mpidi_dynamic_tasking) { - i=0; - world_tasks_save = MPL_strdup(world_tasks); - if(world_tasks != NULL) { - comm->vcr[0]->taskid = atoi(strtok(world_tasks, ":")); - while( (cp=strtok(NULL, ":")) != NULL) { - comm->vcr[++i]->taskid= atoi(cp); - } - } - MPL_free(world_tasks_save); - - /* This memory will be freed by the PG_Destroy if there is an error */ - pg_id = MPL_malloc(MAX_JOBID_LEN); - - mpi_errno = PMI2_Job_GetId(pg_id, MAX_JOBID_LEN); - TRACE_ERR("PMI2_Job_GetId - pg_id=%s\n", pg_id); - - /* Initialize the connection table on COMM_WORLD from the process group's - connection table */ - for (p = 0; p < comm->local_size; p++) - { - comm->vcr[p]->pg=pg->vct[p].pg; - comm->vcr[p]->pg_rank=pg->vct[p].pg_rank; - pg->vct[p].taskid = comm->vcr[p]->taskid; - if(comm->vcr[p]->pg) { - TRACE_ERR("Adding ref for comm=%x vcr=%x pg=%x\n", comm, comm->vcr[p], comm->vcr[p]->pg); - MPIDI_PG_add_ref(comm->vcr[p]->pg); - } - /* MPID_VCR_Dup(&pg->vct[p], &(comm->vcr[p]));*/ - TRACE_ERR("comm->vcr[%d]->pg->id=%s comm->vcr[%d]->pg_rank=%d\n", p, comm->vcr[p]->pg->id, p, comm->vcr[p]->pg_rank); - TRACE_ERR("TASKID -- comm->vcr[%d]=%d\n", p, comm->vcr[p]->taskid); - } - - comm->local_vcr = comm->vcr; - }else { - for (i=0; i<size; i++) { - comm->vcr[i]->taskid = i; - TRACE_ERR("comm->vcr[%d]=%d\n", i, comm->vcr[i]->taskid); - } - TRACE_ERR("MP_I_WORLD_TASKS not SET\n"); - } -#else - for (i=0; i<size; i++) { - comm->vcr[i]->taskid = i; - TRACE_ERR("comm->vcr[%d]=%d\n", i, comm->vcr[i]->taskid); - } -#endif -} - - -/** - * \brief Initialize MPICH at ADI level. - * \param[in,out] argc Unused - * \param[in,out] argv Unused - * \param[in] requested The thread model requested by the user. - * \param[out] provided The thread model provided to user. It is the same as requested, except in VNM. - * \param[out] has_args Set to TRUE - * \param[out] has_env Set to TRUE - * \return MPI_SUCCESS - */ -int MPID_Init(int * argc, - char *** argv, - int requested, - int * provided, - int * has_args, - int * has_env) -{ - int rank, size; -#ifdef DYNAMIC_TASKING - int has_parent=0; - MPIDI_PG_t * pg=NULL; - int pg_rank=-1; - int pg_size; - int appnum,mpi_errno; - MPIR_Comm * comm; - int i,j; - pami_configuration_t config; - int world_size; -#endif - char *world_tasks; - pami_result_t rc; - - /* Override split_type */ - MPIR_Comm_fns = &comm_fns; - - /* ------------------------------------------------------------------------------- */ - /* Initialize the pami client to get the process rank; needed for env var output. */ - /* ------------------------------------------------------------------------------- */ - MPIDI_PAMI_client_init(&rank, &size, &mpidi_dynamic_tasking, &world_tasks); - TRACE_OUT("after MPIDI_PAMI_client_init rank=%d size=%d mpidi_dynamic_tasking=%d\n", rank, size, mpidi_dynamic_tasking); - - /* ------------------------------------ */ - /* Get new defaults from the Env Vars */ - /* ------------------------------------ */ - MPIDI_Env_setup(rank, requested); - - /* ----------------------------- */ - /* Initialize messager */ - /* ----------------------------- */ - if ( (MPIDI_Process.async_progress.mode == ASYNC_PROGRESS_MODE_TRIGGER) || mpidi_dynamic_tasking) - { - /* The 'trigger' async progress mode requires MPI_THREAD_MULTIPLE. - * Silently promote the thread level. - * - * See discussion in src/mpid/pamid/src/mpid_progress.h for more - * information. - */ - *provided = MPI_THREAD_MULTIPLE; - } - else - { - *provided = requested; - } -#if (MPIDI_STATISTICS || MPIDI_PRINTENV) - if (requested != MPI_THREAD_MULTIPLE) - mpich_env->single_thread=1; -#endif - MPIDI_PAMI_init(&rank, &size, provided); - -#ifdef DYNAMIC_TASKING - if (mpidi_dynamic_tasking) { - - /* - * Perform PMI initialization - */ - mpi_errno = MPIDI_InitPG( argc, argv, - has_args, has_env, &has_parent, &pg_rank, &pg ); - if (mpi_errno) { - TRACE_ERR("MPIDI_InitPG returned with mpi_errno=%d\n", mpi_errno); - } - - /* FIXME: Why are pg_size and pg_rank handled differently? */ - pg_size = MPIDI_PG_Get_size(pg); - - TRACE_ERR("MPID_Init - pg_size=%d\n", pg_size); - MPIDI_Process.my_pg = pg; /* brad : this is rework for shared memories - * because they need this set earlier - * for getting the business card - */ - MPIDI_Process.my_pg_rank = pg_rank; - - } -#endif - - /* ------------------------- */ - /* initialize request queues */ - /* ------------------------- */ - MPIDI_Recvq_init(); - - /* -------------------------------------- */ - /* Fill in some hardware structure fields */ - /* -------------------------------------- */ - extern void MPIX_Init(); - MPIX_Init(); - - /* ------------------------------- */ - /* Set process attributes */ - /* ------------------------------- */ - MPIR_Process.attrs.tag_ub = INT_MAX; - MPIR_Process.attrs.wtime_is_global = 1; - MPIR_Process.attrs.io = MPI_ANY_SOURCE; - - - /* ------------------------------- */ - /* Initialize communicator objects */ - /* ------------------------------- */ -#ifndef DYNAMIC_TASKING - MPIDI_VCRT_init(rank, size); -#else - MPIDI_VCRT_init(rank, size, world_tasks, pg); -#endif - - /* ------------------------------- */ - /* Setup optimized communicators */ - /* ------------------------------- */ - TRACE_ERR("creating world geometry\n"); - rc = PAMI_Geometry_world(MPIDI_Client, &MPIDI_Process.world_geometry); - MPID_assert_always(rc == PAMI_SUCCESS); - TRACE_ERR("calling comm_create on comm world %p\n", MPIR_Process.comm_world); - MPIR_Process.comm_world->mpid.geometry = MPIDI_Process.world_geometry; - MPIR_Process.comm_world->mpid.parent = PAMI_GEOMETRY_NULL; - MPIR_Comm_commit(MPIR_Process.comm_world); - -#ifdef DYNAMIC_TASKING - if (has_parent) { - char * parent_port; - - /* FIXME: To allow just the "root" process to - request the port and then use MPIR_Bcast_intra to - distribute it to the rest of the processes, - we need to perform the Bcast after MPI is - otherwise initialized. We could do this - by adding another MPID call that the MPI_Init(_thread) - routine would make after the rest of MPI is - initialized, but before MPI_Init returns. - In fact, such a routine could be used to - perform various checks, including parameter - consistency value (e.g., all processes have the - same environment variable values). Alternately, - we could allow a few routines to operate with - predefined parameter choices (e.g., bcast, allreduce) - for the purposes of initialization. */ - mpi_errno = MPIDI_GetParentPort(&parent_port); - if (mpi_errno != MPI_SUCCESS) { - TRACE_ERR("MPIDI_GetParentPort returned with mpi_errno=%d\n", mpi_errno); - } - - mpi_errno = MPID_Comm_connect(parent_port, NULL, 0, - MPIR_Process.comm_world, &comm); - if (mpi_errno != MPI_SUCCESS) { - TRACE_ERR("mpi_errno from Comm_connect=%d\n", mpi_errno); - } - - MPIR_Process.comm_parent = comm; - MPIR_Assert(MPIR_Process.comm_parent != NULL); - MPL_strncpy(comm->name, "MPI_COMM_PARENT", MPI_MAX_OBJECT_NAME); - - /* FIXME: Check that this intercommunicator gets freed in MPI_Finalize - if not already freed. */ - } - mp_world_exiting_handler = &(_mpi_world_exiting_handler); -#endif - /* ------------------------------- */ - /* Initialize timer data */ - /* ------------------------------- */ - MPIDI_PAMID_Timer_is_ready = 1; - - /* ------------------------------- */ - /* ??? */ - /* ------------------------------- */ - *has_args = TRUE; - *has_env = TRUE; -#ifdef MPIDI_PRINTENV - if (MPIDI_Process.mp_printenv) { - MPIDI_Print_mpenv(rank,size); - } -#endif - /* ----------------------------------------------- */ - /* parse params for pami_tune if in benchmark mode */ - /* ----------------------------------------------- */ - if(MPIDI_Process.optimized.auto_select_colls == MPID_AUTO_SELECT_COLLS_TUNE) - { - if(argc != NULL && argv != NULL) - { - if(MPIDI_collsel_pami_tune_parse_params(*argc, *argv) != PAMI_SUCCESS) - { - MPIDI_Process.optimized.auto_select_colls = MPID_AUTO_SELECT_COLLS_NONE; - } - } - else - { - if(MPIDI_collsel_pami_tune_parse_params(0, NULL) != PAMI_SUCCESS) - { - MPIDI_Process.optimized.auto_select_colls = MPID_AUTO_SELECT_COLLS_NONE; - } - } - } - return MPI_SUCCESS; -} - - -/* - * \brief This is called by MPI to let us know that MPI_Init is done. - */ -int MPID_InitCompleted() -{ - MPIDI_NBC_init(); - MPIDI_Progress_init(); - /* ----------------------------------------------- */ - /* Now all is ready.. call table generate */ - /* ----------------------------------------------- */ - if(MPIDI_Process.optimized.auto_select_colls == MPID_AUTO_SELECT_COLLS_TUNE) - { - MPIDI_Collsel_table_generate(); - MPIDI_collsel_pami_tune_cleanup(); - } - return MPI_SUCCESS; -} - -#if (MPIDI_PRINTENV || MPIDI_STATISTICS || MPIDI_BANNER) -void MPIDI_open_pe_extension() { - int rc; - /* open PE extension */ - memset(&pe_extension,0, sizeof(pami_extension_t)); - rc = PAMI_Extension_open (MPIDI_Client, "EXT_pe_extension", &pe_extension); - TRACE_ERR("PAMI_Extension_open: rc %d\n", rc); - if (rc != PAMI_SUCCESS) { - TRACE_ERR("ERROR open PAMI_Extension_open failed rc %d", rc); - MPID_assert_always(rc == PAMI_SUCCESS); - } -} - - -int MPIDI_Banner(char * bufPtr) { - char *cp, *level=NULL; - char buf[30]; - char *ASC_time; - time_t ltime; - char msgBuf[60]; - char type[64], ver_buf[64]; - struct tm *tmx,*tm1; - - /* Note: The _ibm_release_version will be expanded to a full string */ - /* ONLY IF this file is extracted from CMVC. */ - /* If this file is cloned from GIT the the string will be just */ - /* "%W%. */ - if ( strncmp(_ibm_release_version_, "%W", 2) ) { - /* IBMPE's expanded version string has release name like ppe_rbarlx */ - /* and that '_' in the name is what we are looking for. */ - /* BGQ's version string does not have a '_' in it. */ - level = strrchr(_ibm_release_version_, '_'); - if ( level ) { - level -=3; - - /* The version string generated by CMVC during weekly build has a */ - /* date which is the time when the mpidi_platform.h file is last */ - /* updated. This date can be quite old and is better removed. */ - memset(ver_buf, 0, sizeof(ver_buf)); - strncpy(ver_buf, level, sizeof(ver_buf)-1); - if ( cp = strchr(ver_buf, ',') ) *cp = '\0'; - } - } - - if(sizeof(void*) == 8) - strcpy(type, "64bit (MPI over PAMI)"); - else if(sizeof(int) == 4) - strcpy(type, "32bit (MPI over PAMI)"); - else - strcpy(type, "UNKNOWN-bit (MPI over PAMI)"); - - sprintf(msgBuf,"MPICH library was compiled on"); - - tmx=MPL_malloc(sizeof(struct tm)); - sprintf(buf,__DATE__" "__TIME__); - - if ((void *) NULL == strptime(buf, "%B %d %Y %T", tmx)) - return(1); - - /* update isdst in tmx structure */ - ltime=0; - time(&ltime); - tm1 = localtime(&ltime); - tmx->tm_isdst=tm1->tm_isdst; - - /* localtime updates tm_wday in tmx structure */ - ltime=mktime(tmx); - tm1 = localtime(&ltime); - tmx->tm_wday = tm1->tm_wday; - ASC_time = asctime(tmx); - - if (level) { - sprintf(bufPtr, "%s %s %s %s ", type, ver_buf, msgBuf, ASC_time); - } else { - sprintf(bufPtr, "%s %s %s ", type, msgBuf, ASC_time); - - } - - MPL_free(tmx); - return MPI_SUCCESS; -} -#endif - - -static inline void -static_assertions() -{ - MPID_assert_static(sizeof(void*) == sizeof(size_t)); - MPID_assert_static(sizeof(uintptr_t) == sizeof(size_t)); -#ifdef __BGQ__ - /* MPID_VCR_GET_LPIDS relies on the VCR being a simple task list */ - MPID_VCR vcr=NULL; - MPID_assert_static(sizeof(*vcr) == sizeof(pami_task_t));/* VCR is a simple task list */ - MPID_assert_static(sizeof(vcr->taskid) == sizeof(*vcr));/* VCR is a simple task list */ - - MPID_assert_static(sizeof(MPIDI_MsgInfo) == 16); - MPID_assert_static(sizeof(uint64_t) == sizeof(size_t)); -#endif -} - -#ifdef DYNAMIC_TASKING -/* FIXME: The PG code should supply these, since it knows how the - pg_ids and other data are represented */ -int MPIDI_PG_Compare_ids(void * id1, void * id2) -{ - return (strcmp((char *) id1, (char *) id2) == 0) ? TRUE : FALSE; -} - -int MPIDI_PG_Destroy_id(MPIDI_PG_t * pg) -{ - if (pg->id != NULL) - { - TRACE_ERR("free pg id =%p pg=%p\n", pg->id, pg); - MPL_free(pg->id); - TRACE_ERR("done free pg id \n"); - } - - return MPI_SUCCESS; -} - - -int MPIDI_InitPG( int *argc, char ***argv, - int *has_args, int *has_env, int *has_parent, - int *pg_rank_p, MPIDI_PG_t **pg_p ) -{ - int pmi_errno; - int mpi_errno = MPI_SUCCESS; - int pg_rank, pg_size, appnum, pg_id_sz; - int usePMI=1; - char *pg_id; - MPIDI_PG_t *pg = 0; - - /* If we use PMI here, make the PMI calls to get the - basic values. Note that systems that return setvals == true - do not make use of PMI for the KVS routines either (it is - assumed that the discover connection information through some - other mechanism */ - /* FIXME: We may want to allow the channel to ifdef out the use - of PMI calls, or ask the channel to provide stubs that - return errors if the routines are in fact used */ - if (usePMI) { - /* - * Initialize the process manangement interface (PMI), - * and get rank and size information about our process group - */ - -#ifdef USE_PMI2_API - TRACE_ERR("Calling PMI2_Init\n"); - mpi_errno = PMI2_Init(has_parent, &pg_size, &pg_rank, &appnum); - TRACE_ERR("PMI2_Init - pg_size=%d pg_rank=%d\n", pg_size, pg_rank); - /*if (mpi_errno) MPIU_ERR_POP(mpi_errno);*/ -#else - TRACE_ERR("Calling PMI_Init\n"); - pmi_errno = PMI_Init(has_parent); - if (pmi_errno != PMI_SUCCESS) { - /* MPIU_ERR_SETANDJUMP1(mpi_errno,MPI_ERR_OTHER, "**pmi_init", - "**pmi_init %d", pmi_errno); */ - } - - pmi_errno = PMI_Get_rank(&pg_rank); - if (pmi_errno != PMI_SUCCESS) { - /*MPIU_ERR_SETANDJUMP1(mpi_errno,MPI_ERR_OTHER, "**pmi_get_rank", - "**pmi_get_rank %d", pmi_errno); */ - } - - pmi_errno = PMI_Get_size(&pg_size); - if (pmi_errno != 0) { - /*MPIU_ERR_SETANDJUMP1(mpi_errno,MPI_ERR_OTHER, "**pmi_get_size", - "**pmi_get_size %d", pmi_errno);*/ - } - - pmi_errno = PMI_Get_appnum(&appnum); - if (pmi_errno != PMI_SUCCESS) { -/* MPIU_ERR_SETANDJUMP1(mpi_errno,MPI_ERR_OTHER, "**pmi_get_appnum", - "**pmi_get_appnum %d", pmi_errno); */ - } -#endif - /* Note that if pmi is not availble, the value of MPI_APPNUM is - not set */ - if (appnum != -1) { - MPIR_Process.attrs.appnum = appnum; - } - -#ifdef USE_PMI2_API - - /* This memory will be freed by the PG_Destroy if there is an error */ - pg_id = MPL_malloc(MAX_JOBID_LEN); - - mpi_errno = PMI2_Job_GetId(pg_id, MAX_JOBID_LEN); - TRACE_ERR("PMI2_Job_GetId - pg_id=%s\n", pg_id); -#else - /* Now, initialize the process group information with PMI calls */ - /* - * Get the process group id - */ - pmi_errno = PMI_KVS_Get_name_length_max(&pg_id_sz); - if (pmi_errno != PMI_SUCCESS) { - TRACE_ERR("PMI_KVS_Get_name_length_max returned with pmi_errno=%d\n", pmi_errno); - } - - /* This memory will be freed by the PG_Destroy if there is an error */ - pg_id = MPL_malloc(pg_id_sz + 1); - - /* Note in the singleton init case, the pg_id is a dummy. - We'll want to replace this value if we join an - Process manager */ - pmi_errno = PMI_KVS_Get_my_name(pg_id, pg_id_sz); - if (pmi_errno != PMI_SUCCESS) { - TRACE_ERR("PMI_KVS_Get_my_name returned with pmi_errno=%d\n", pmi_errno); - } -#endif - } - else { - /* Create a default pg id */ - pg_id = MPL_malloc(2); - MPL_strncpy( pg_id, "0", 2 ); - } - - TRACE_ERR("pg_size=%d pg_id=%s\n", pg_size, pg_id); - /* - * Initialize the process group tracking subsystem - */ - mpi_errno = MPIDI_PG_Init(argc, argv, - MPIDI_PG_Compare_ids, MPIDI_PG_Destroy_id); - if (mpi_errno != MPI_SUCCESS) { - TRACE_ERR("MPIDI_PG_Init returned with mpi_errno=%d\n", mpi_errno); - } - - /* - * Create a new structure to track the process group for our MPI_COMM_WORLD - */ - TRACE_ERR("pg_size=%d pg_id=%p pg_id=%s\n", pg_size, pg_id, pg_id); - mpi_errno = MPIDI_PG_Create(pg_size, pg_id, &pg); - MPL_free(pg_id); - if (mpi_errno != MPI_SUCCESS) { - TRACE_ERR("MPIDI_PG_Create returned with mpi_errno=%d\n", mpi_errno); - } - - /* FIXME: We can allow the channels to tell the PG how to get - connection information by passing the pg to the channel init routine */ - if (usePMI) { - /* Tell the process group how to get connection information */ - mpi_errno = MPIDI_PG_InitConnKVS( pg ); - if (mpi_errno) - TRACE_ERR("MPIDI_PG_InitConnKVS returned with mpi_errno=%d\n", mpi_errno); - } - - /* FIXME: has_args and has_env need to come from PMI eventually... */ - *has_args = TRUE; - *has_env = TRUE; - - *pg_p = pg; - *pg_rank_p = pg_rank; - - fn_exit: - return mpi_errno; - fn_fail: - /* --BEGIN ERROR HANDLING-- */ - if (pg) { - MPIDI_PG_Destroy( pg ); - } - goto fn_exit; - /* --END ERROR HANDLING-- */ -} -#endif diff --git a/src/mpid/pamid/src/mpid_iprobe.c b/src/mpid/pamid/src/mpid_iprobe.c deleted file mode 100644 index 97e9a94..0000000 --- a/src/mpid/pamid/src/mpid_iprobe.c +++ /dev/null @@ -1,47 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/mpid_iprobe.c - * \brief ??? - */ -#include <mpidimpl.h> - - -int -MPID_Iprobe(int source, - int tag, - MPIR_Comm * comm, - int context_offset, - int *flag, - MPI_Status * status) -{ - const int context = comm->recvcontext_id + context_offset; - - if (source == MPI_PROC_NULL) - { - MPIR_Status_set_procnull(status); - /* We set the flag to true because an MPI_Recv with this rank will - * return immediately */ - *flag = TRUE; - return MPI_SUCCESS; - } - *flag = MPIDI_Recvq_FU_r(source, tag, context, status); - if (!(*flag)) - MPID_Progress_poke(); - return MPI_SUCCESS; -} diff --git a/src/mpid/pamid/src/mpid_mprobe.c b/src/mpid/pamid/src/mpid_mprobe.c deleted file mode 100644 index 29a27b5..0000000 --- a/src/mpid/pamid/src/mpid_mprobe.c +++ /dev/null @@ -1,41 +0,0 @@ -/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ -/* - * (C) 2012 by Argonne National Laboratory. - * See COPYRIGHT in top-level directory. - */ - -#include "mpidimpl.h" - -int MPID_Mprobe(int source, int tag, MPIR_Comm *comm, int context_offset, - MPIR_Request **message, MPI_Status *status) -{ - const int context = comm->recvcontext_id + context_offset; - int foundp; - MPIR_Request * rreq = NULL; - - if (source == MPI_PROC_NULL) - { - MPIR_Status_set_procnull(status); - return MPI_SUCCESS; - } - -#ifndef OUT_OF_ORDER_HANDLING - MPID_PROGRESS_WAIT_WHILE((rreq=MPIDI_Recvq_FDU(source, tag, context, &foundp) ) == NULL ); -#else - int pami_source; - if(source != MPI_ANY_SOURCE) { - pami_source = MPID_VCR_GET_LPID(comm->vcr, source); - } else { - pami_source = MPI_ANY_SOURCE; - } - MPID_PROGRESS_WAIT_WHILE((rreq=MPIDI_Recvq_FDU(source, pami_source, tag, context, &foundp) ) == NULL ); -#endif - - if (rreq) { - rreq->kind = MPIR_REQUEST_KIND__MPROBE; - MPIR_Request_extract_status(rreq, status); - } - *message = rreq; - return MPI_SUCCESS; -} - diff --git a/src/mpid/pamid/src/mpid_mrecv.c b/src/mpid/pamid/src/mpid_mrecv.c deleted file mode 100644 index a6cde3f..0000000 --- a/src/mpid/pamid/src/mpid_mrecv.c +++ /dev/null @@ -1,34 +0,0 @@ -/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ -/* - * (C) 2012 by Argonne National Laboratory. - * See COPYRIGHT in top-level directory. - */ - -#include "mpidimpl.h" - -int MPID_Mrecv(void *buf, int count, MPI_Datatype datatype, - MPIR_Request *message, MPI_Status *status) -{ - int mpi_errno = MPI_SUCCESS; - MPI_Request req_handle; /* dummy for MPIR_Request_complete */ - int active_flag; /* dummy for MPIR_Request_complete */ - MPIR_Request *rreq = NULL; - - if (message == NULL) { - /* treat as though MPI_MESSAGE_NO_PROC was passed */ - MPIR_Status_set_procnull(status); - return mpi_errno; - } - - /* There is no optimized MPID_Mrecv at this time because there is no real - * optimization potential in that case. MPID_Recv exists to prevent - * creating a request unnecessarily for messages that are already present - * and eligible for immediate completion. */ - mpi_errno = MPID_Imrecv(buf, count, datatype, message, &rreq); - - MPID_PROGRESS_WAIT_WHILE(!MPIR_Request_is_complete(rreq)); - - mpi_errno = MPIR_Request_complete(&req_handle, rreq, status, &active_flag); - return mpi_errno; -} - diff --git a/src/mpid/pamid/src/mpid_probe.c b/src/mpid/pamid/src/mpid_probe.c deleted file mode 100644 index 9287fec..0000000 --- a/src/mpid/pamid/src/mpid_probe.c +++ /dev/null @@ -1,41 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/mpid_probe.c - * \brief ??? - */ -#include <mpidimpl.h> - - -int -MPID_Probe(int source, - int tag, - MPIR_Comm * comm, - int context_offset, - MPI_Status * status) -{ - const int context = comm->recvcontext_id + context_offset; - - if (source == MPI_PROC_NULL) - { - MPIR_Status_set_procnull(status); - return MPI_SUCCESS; - } - MPID_PROGRESS_WAIT_WHILE(!MPIDI_Recvq_FU_r(source, tag, context, status)); - return MPI_SUCCESS; -} diff --git a/src/mpid/pamid/src/mpid_progress.c b/src/mpid/pamid/src/mpid_progress.c deleted file mode 100644 index fd3d321..0000000 --- a/src/mpid/pamid/src/mpid_progress.c +++ /dev/null @@ -1,355 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/mpid_progress.c - * \brief Maintain the state and rules of the MPI progress semantics - */ -#include <mpidimpl.h> - -#define MAX_PROGRESS_HOOKS 4 -typedef int (*progress_func_ptr_t) (int* made_progress); - -typedef struct progress_hook_slot { - progress_func_ptr_t func_ptr; - int active; -} progress_hook_slot_t; - -static progress_hook_slot_t progress_hooks[MAX_PROGRESS_HOOKS]; - -#undef FUNCNAME -#define FUNCNAME MPIDI_Progress_register_hook -#undef FCNAME -#define FCNAME MPIU_QUOTE(FUNCNAME) -int MPIDI_Progress_register_hook(int (*progress_fn)(int*), int *id) -{ - int mpi_errno = MPI_SUCCESS; - int i; - MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDI_PROGRESS_REGISTER_HOOK); - - MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDI_PROGRESS_REGISTER_HOOK); - MPIU_THREAD_CS_ENTER(ASYNC,); - - for (i = 0; i < MAX_PROGRESS_HOOKS; i++) { - if (progress_hooks[i].func_ptr == NULL) { - progress_hooks[i].func_ptr = progress_fn; - progress_hooks[i].active = FALSE; - break; - } - } - - if (i >= MAX_PROGRESS_HOOKS) { - return MPIR_Err_create_code( MPI_SUCCESS, MPIR_ERR_RECOVERABLE, - "MPIDI_Progress_register_hook", __LINE__, - MPI_ERR_INTERN, "**progresshookstoomany", 0 ); - } - - (*id) = i; - - fn_exit: - MPIU_THREAD_CS_EXIT(ASYNC,); - MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDI_PROGRESS_REGISTER_HOOK); - return mpi_errno; - - fn_fail: - goto fn_exit; -} - -#undef FUNCNAME -#define FUNCNAME MPIDI_Progress_deregister_hook -#undef FCNAME -#define FCNAME MPIU_QUOTE(FUNCNAME) -int MPIDI_Progress_deregister_hook(int id) -{ - int mpi_errno = MPI_SUCCESS; - MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDI_PROGRESS_DEREGISTER_HOOK); - - MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDI_PROGRESS_DEREGISTER_HOOK); - MPIU_THREAD_CS_ENTER(ASYNC,); - - MPIR_Assert(id >= 0 && id < MAX_PROGRESS_HOOKS && progress_hooks[id].func_ptr != NULL); - - progress_hooks[id].func_ptr = NULL; - progress_hooks[id].active = FALSE; - - fn_exit: - MPIU_THREAD_CS_EXIT(ASYNC,); - MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDI_PROGRESS_DEREGISTER_HOOK); - return mpi_errno; - - fn_fail: - goto fn_exit; -} - -#undef FUNCNAME -#define FUNCNAME MPIDI_CH3I_Progress_activate_hook -#undef FCNAME -#define FCNAME MPIU_QUOTE(FUNCNAME) -int MPIDI_CH3I_Progress_activate_hook(int id) -{ - int mpi_errno = MPI_SUCCESS; - MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDI_CH3I_PROGRESS_ACTIVATE_HOOK); - - MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDI_CH3I_PROGRESS_ACTIVATE_HOOK); - MPIU_THREAD_CS_ENTER(MPIDCOMM,); - - MPIR_Assert(id >= 0 && id < MAX_PROGRESS_HOOKS && - progress_hooks[id].active == FALSE && progress_hooks[id].func_ptr != NULL); - progress_hooks[id].active = TRUE; - - fn_exit: - MPIU_THREAD_CS_EXIT(MPIDCOMM,); - MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDI_CH3I_PROGRESS_ACTIVATE_HOOK); - return mpi_errno; - - fn_fail: - goto fn_exit; -} - - -#undef FUNCNAME -#define FUNCNAME MPIDI_CH3I_Progress_deactivate_hook -#undef FCNAME -#define FCNAME MPIU_QUOTE(FUNCNAME) -int MPIDI_CH3I_Progress_deactivate_hook(int id) -{ - int mpi_errno = MPI_SUCCESS; - MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDI_CH3I_PROGRESS_DEACTIVATE_HOOK); - - MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDI_CH3I_PROGRESS_DEACTIVATE_HOOK); - MPIU_THREAD_CS_ENTER(MPIDCOMM,); - - MPIR_Assert(id >= 0 && id < MAX_PROGRESS_HOOKS && - progress_hooks[id].active == TRUE && progress_hooks[id].func_ptr != NULL); - progress_hooks[id].active = FALSE; - - fn_exit: - MPIU_THREAD_CS_EXIT(MPIDCOMM,); - MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDI_CH3I_PROGRESS_DEACTIVATE_HOOK); - return mpi_errno; - - fn_fail: - goto fn_exit; -} - -void -MPIDI_Progress_init() -{ - MPIDI_Process.async_progress.active = 0; - pamix_progress_t async_progress_type=0; - - /* In the "global" mpich lock mode the only possible progress function - * is the "global mpich lock" trigger progress function. This progress - * function is ignored if async progress is disabled. - */ - pamix_progress_function progress_fn = MPIDI_Progress_async_poll; - uintptr_t i; - -#if (MPICH_THREAD_GRANULARITY == MPICH_THREAD_GRANULARITY__POBJ) - /* In the "per object" mpich lock mode the only possible progress functions - * are the "context lock" trigger progress function and the 'NULL' progress - * function. - */ - if (MPIDI_Process.async_progress.mode == ASYNC_PROGRESS_MODE_LOCKED) - progress_fn = NULL; - else - progress_fn = MPIDI_Progress_async_poll_perobj; - - MPIDI_Process.perobj.context_post.active = - MPIDI_Process.perobj.context_post.requested; -#endif - - async_progress_type = PAMIX_PROGRESS_ALL; - -#ifdef __PE__ - if (MPIDI_Process.mp_interrupts) - { - /* PAMIX_PROGRESS_TIMER, PAMIX_PROGRESS_RECV_INTERRUPT or */ - /* ASYNC_PROGRESS_ALL */ - /* mp_interrupts=0 indicates the interrupts is disabled */ - /* which is the same definition as that of PAMIX_PROGRESS_ALL. */ - /* ASYNC_PROGRESS_ALL is an internal variable which is defined */ - /* as 0x1111 and needs to be converted to corresponding pami */ - /* type PAMIX_PROGRESS_ALL */ - async_progress_type = MPIDI_Process.mp_interrupts; - if (MPIDI_Process.mp_interrupts == ASYNC_PROGRESS_ALL) - async_progress_type = PAMIX_PROGRESS_ALL; - } -#endif - - - if (MPIDI_Process.async_progress.mode != ASYNC_PROGRESS_MODE_DISABLED) - { - TRACE_ERR("Async advance beginning...\n"); - - /* Enable async progress on all contexts.*/ - for (i=0; i<MPIDI_Process.avail_contexts; ++i) - { - PAMIX_Progress_register(MPIDI_Context[i], - progress_fn, - MPIDI_Progress_async_end, - MPIDI_Progress_async_start, - (void *) i); - PAMIX_Progress_enable(MPIDI_Context[i], async_progress_type); - } - TRACE_ERR("Async advance enabled\n"); - } - - /* Initialize progress hook slots */ - for (i = 0; i < MAX_PROGRESS_HOOKS; i++) - { - progress_hooks[i].func_ptr = NULL; - progress_hooks[i].active = FALSE; - } -} - -void -MPIDI_Progress_async_start(pami_context_t context, void *cookie) -{ - /* This special async progress critical section is required for both 'global' - * and 'perobj' mpich lock modes because, regardless of the lock mode, if - * more than one context is enabled for async progress then multiple threads - * could attempt to update the shared variables at the same time. - */ - MPIU_THREAD_CS_ENTER(ASYNC,); - -#if (MPICH_THREAD_GRANULARITY == MPICH_THREAD_GRANULARITY__POBJ) - if (MPIDI_Process.async_progress.active == 0) - { - /* Asynchronous progress was completely disabled and now async progress - * is enabled for one context. Because one context is in async progress - * mode the configuration state must be updated as if all contexts were in - * async rogress mode. - */ - MPIDI_Process.perobj.context_post.active = - MPIDI_Process.perobj.context_post.requested; - } -#endif - - MPIDI_Process.async_progress.active += 1; - - MPIDI_Mutex_sync(); - - /* Required for both 'global' and 'perobj' mpich lock modes. */ - MPIU_THREAD_CS_EXIT(ASYNC,); -} - -void -MPIDI_Progress_async_end (pami_context_t context, void *cookie) -{ - /* This special async progress critical section is required for both 'global' - * and 'perobj' mpich lock modes because, regardless of the lock mode, if - * more than one context is enabled for async progress then multiple threads - * could attempt to update the shared variables at the same time. - */ - MPIU_THREAD_CS_ENTER(ASYNC,); - - MPIDI_Process.async_progress.active -= 1; - -#if (MPICH_THREAD_GRANULARITY == MPICH_THREAD_GRANULARITY__POBJ) - if (MPIDI_Process.async_progress.active == 0) - { - /* Asynchronous progress is now completely disabled on all contexts. */ - if (MPIR_ThreadInfo.thread_provided != MPI_THREAD_MULTIPLE) - { - /* The runtime is, currently, completely single threaded - there are - * no async progress threads and only one application thread. In this - * environment the context post is simply overhead and can be safely - * avoided. - */ - MPIDI_Process.perobj.context_post.active = 0; - } - } -#endif - MPIDI_Mutex_sync(); - - /* Required for both 'global' and 'perobj' mpich lock modes. */ - MPIU_THREAD_CS_EXIT(ASYNC,); -} - -void -MPIDI_Progress_async_poll (pami_context_t context, void *cookie) -{ - pami_result_t rc; - int loop_count=100; - int i, made_progress; - - /* In the "global" mpich lock mode all application threads must acquire the - * ALLFUNC global lock upon entry to the API. The async progress thread - * invoking this progress function also must first acquire the ALLFUNC - * global lock as it enters the API and before invoking advance as mpich - * callbacks invoked during advance will not take any additional locks. - */ - if (MPIU_THREAD_CS_TRY(ALLFUNC,)) /* (0==try_acquire(0)) */ - { - for (i = 0; i < MAX_PROGRESS_HOOKS; i++) { - if (progress_hooks[i].active == TRUE) { - MPIR_Assert(progress_hooks[i].func_ptr != NULL); - progress_hooks[i].func_ptr(&made_progress); - } - } - - /* There is a simplifying assertion when in the 'global' mpich lock mode - * that only a single context is supported. See the discussion in - * mpich/src/mpid/pamid/src/mpid_init.c for more information. - */ - rc = PAMI_Context_advancev (MPIDI_Context, 1, loop_count); - MPID_assert( (rc == PAMI_SUCCESS) || (rc == PAMI_EAGAIN) ); - - MPIDI_Mutex_sync(); /* is this needed ? */ - MPIU_THREAD_CS_EXIT(ALLFUNC,); /* release(0) */ -#ifdef __PE__ - if (rc == PAMI_EAGAIN) - sched_yield(); -#endif - } - - return; -} - -void -MPIDI_Progress_async_poll_perobj (pami_context_t context, void *cookie) -{ - pami_result_t rc; - int loop_count=100; - int i, made_progress; - - for (i = 0; i < MAX_PROGRESS_HOOKS; i++) { - if (progress_hooks[i].active == TRUE) { - MPIR_Assert(progress_hooks[i].func_ptr != NULL); - progress_hooks[i].func_ptr(&made_progress); - } - } - - /* In the "per object" mpich lock mode multiple application threads could be - * active within the API interacting with contexts and multiple async progress - * threads could be polling contexts in this function. The only way to force a - * serialized access to the contexts is to use the pami context locks. - */ - rc = PAMI_Context_trylock_advancev(MPIDI_Context, MPIDI_Process.avail_contexts, loop_count); - MPID_assert( (rc == PAMI_SUCCESS) || (rc == PAMI_EAGAIN) ); -#ifdef __PE__ - if (rc == PAMI_EAGAIN) - { - MPIDI_Mutex_sync(); /* is this needed ? */ - sched_yield(); - } -#endif - - return; -} - diff --git a/src/mpid/pamid/src/mpid_progress.h b/src/mpid/pamid/src/mpid_progress.h deleted file mode 100644 index be2b7fc..0000000 --- a/src/mpid/pamid/src/mpid_progress.h +++ /dev/null @@ -1,297 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/mpid_progress.h - * \brief ??? - */ - -#ifndef __src_mpid_progress_h__ -#define __src_mpid_progress_h__ - - -/** - * \defgroup MPID_PROGRESS MPID Progress engine - * - * Maintain the state and rules of the MPI progress semantics. - * - * The behavior of asynchronous progress depends on many configurable variables, - * and the requirements and/or effects of several combinations may not be - * obvious. There are certain truths and simplifying assertions that, once - * identified, can eliminate many configurations from consideration which - * greatly reduces code complexity. - * - * 1. Asynchronous progress with a NULL progress function requires context post. - * - * When a NULL progress function is specified for async progress, in some - * implementations, the context lock will be internally acquired and held - * until async progress is disabled or suspended. As such, any application - * thread that waits to acquire a context lock will block indefinitely. This - * is considered an invalid configuration. - * - * 2. Asynchronous progress with a NULL progress function requires "per object" - * mpich locks. - * - * When compiled using the "global lock" mpich thread mode the individual - * "per object" lock macros are defined as noops, yet all mpich callbacks - * that could be invoked from within a context advance only use the - * "per object" lock macros and *not* the "global" lock macros. This is - * because the async progress thread never actually leaves the API, but - * remains within the library. As designed, the mpich "global" locks are only - * acquired and released at the very top API entry and exit points. - * - * If noop "per object" lock macros are ever used the async progress threads - * will cause data corruption of mpich data structures. - * - * 3. Asynchronous progress with a non-NULL progress function using the "global" - * mpich locks requires 'MPI_THREAD_MULTIPLE'. - * - * The "global" mpich lock macros, as defined, include a check of the thread - * mode and only performs the mutex operations if the thread mode is - * 'MPI_THREAD_MULTIPLE'. When using a non-NULL progress function for async - * progress the thread that invokes the progress function is essentially - * 'outside' of the mpich library. Consequently, similar to the other API - * entry points, the "global" mpich lock must be acquired in the progress - * function before any mpich data structures may be accessed. - * - * If acquiring the "global" lock is attempted in any thread mode other - * than 'MPI_THREAD_MULTIPLE' the async progress threads will cause data - * corruption of mpich data structures. To avoid this data corruption - * problem the mpi thread mode is promoted to 'MPI_THREAD_MULTIPLE' when - * async progress with a non-NULL progress function is enabled in the - * "global" mpich lock mode. - * - * 4. Only a single context is supported in the "global" mpich lock mode. - * - * A multi-context application will always perform better with the - * "per object" mpich lock mode - regardless of whether async progress is - * enabled or not. This is because in the "global" mpich lock mode all - * threads, application and async progress, must acquire the single global - * lock which effectively serializes the threads and negates any benefit - * of multiple contexts. - * - * Asserting this single context limitation removes code, improves - * performance, and greatly simplifies logic. - * - * 5. The "global" mpich lock mode does not support context post. - * - * As the "global" mpich lock mode only supports a single context, and all - * threads, application and async progress, must first acquire the global - * lock before accessing the mpich data structures or the single context, - * it is detrimental to performance to invoke context post instead of - * directly operating on the context. - * - * Asserting this context post limitation further removes code, improves - * performance, and simplifies logic. - * - * 6. The "global" mpich lock mode does not require context lock. - * - * Access to all internal mpich data structures, including the single - * context, for all application and async progress threads, is protected by - * the "global" mpich lock upon entry to the API. This makes the context - * lock redundant. - * - * 7. The "per object" mpich lock mode assumes active asynchronous progress and - * mpi thread mode 'MPI_THREAD_MULTIPLE'. - * - * Strictly defined, the context lock is NOT required in the "per object" - * mpich lock mode if asyncronous progress is not active and the mpi thread - * level is not 'MPI_THREAD_MULTIPLE'. This means that the run environment - * is completely single threaded. - * - * An application running completely single threaded using the "per object" - * mpich lock mode will always perform worse than the same completely single - * threaded application using the "global" mpich lock mode. This is because - * a "per object" single thread will take more locks than a "global" single - * thread. - * - * Therefore, a simplifying assertion is made that, as the "per object" - * completely single threaded configuration is always worse than the "global" - * completely single threaded configuration, the "per object" mpich lock mode - * will assume the application is running a multi-threaded configuration and - * optimize the code logic accordingly. - * - * \addtogroup MPID_PROGRESS - * \{ - */ - -typedef enum -{ - ASYNC_PROGRESS_MODE_DISABLED = 0, /**< async progress is disabled */ - ASYNC_PROGRESS_MODE_LOCKED, /**< async progress uses a \c NULL progress_fn */ - ASYNC_PROGRESS_MODE_TRIGGER, /**< async progress uses a 'trigger' progress_fn */ - ASYNC_PROGRESS_MODE_COUNT /**< number of sync progress modes */ -} async_progress_mode_t; - - -/** \brief The same as MPID_Progress_wait(), since it does not block */ -#define MPID_Progress_test() MPID_Progress_wait_inline(1) -/** \brief The same as MPID_Progress_wait(), since it does not block */ -#define MPID_Progress_poke() MPID_Progress_wait_inline(1) - - -/** - * \brief A macro to easily implement advancing until a specific - * condition becomes false. - * - * \param[in] COND This is not a true parameter. It is *specifically* - * designed to be evaluated several times, allowing for the result to - * change. The condition would generally look something like - * "(cb.client == 0)". This would be used as the condition on a while - * loop. - * - * \returns MPI_SUCCESS - * - * This correctly checks the condition before attempting to loop, - * since the call to MPID_Progress_wait() may not return if the event - * is already complete. Any system *not* using this macro *must* use - * a similar check before waiting. - */ -#define MPID_PROGRESS_WAIT_WHILE(COND) \ -({ \ - while (COND) \ - MPID_Progress_wait(&__state); \ - MPI_SUCCESS; \ -}) - - -/** - * \brief A macro to easily implement advancing until a specific - * condition becomes false. - * - * \param[in] COND This is not a true parameter. It is *specifically* - * designed to be evaluated several times, allowing for the result to - * change. The condition would generally look something like - * "(cb.client == 0)". This would be used as the condition on a while - * loop. - * - * \returns MPI_SUCCESS - * - * This macro makes one pami advance regardless of the state of the COND. - */ -#define MPID_PROGRESS_WAIT_DO_WHILE(COND) \ -({ \ - do { \ - MPID_Progress_wait(&__state); \ - } while(COND); \ - MPI_SUCCESS; \ -}) - - -/** - * \brief Unused, provided since MPI calls it. - * \param[in] state The previously seen state of advance - */ -#define MPID_Progress_start(state) - -/** - * \brief Unused, provided since MPI calls it. - * \param[in] state The previously seen state of advance - */ -#define MPID_Progress_end(state) - -/** - * \brief Signal MPID_Progress_wait() that something is done/changed - * - * It is therefore important that the ADI layer include a call to - * MPIDI_Progress_signal() whenever something occurs that a node might - * be waiting on. - */ -#define MPIDI_Progress_signal() - - -#define MPID_Progress_wait(state) MPID_Progress_wait_inline(100) - - -void MPIDI_Progress_init(); -void MPIDI_Progress_async_start(pami_context_t context, void *cookie); -void MPIDI_Progress_async_end (pami_context_t context, void *cookie); -void MPIDI_Progress_async_poll (); -void MPIDI_Progress_async_poll_perobj (); - -/** - * \brief This function blocks until a request completes - * \param[in] state The previously seen state of advance - * - * It does not check what has completed, only that the counter - * changes. That happens whenever there is a call to - * MPIDI_Progress_signal(). It is therefore important that the ADI - * layer include a call to MPIDI_Progress_signal() whenever something - * occurs that a node might be waiting on. - * - */ -static inline int -MPID_Progress_wait_inline(unsigned loop_count) -{ - pami_result_t rc = 0; - -#if (MPICH_THREAD_GRANULARITY == MPICH_THREAD_GRANULARITY__POBJ) - /* - * In the "per object" thread mode the "global" lock is defined as a noop and - * therefore no locks are held at this point. - */ - if (unlikely(MPIDI_Process.async_progress.active == 0)) - { - /* - * Asynchronous progress is currently inactive; therefore this application - * thread must drive progress. - * - * It is not actually neccesary to lock the context when async progress is - * disabled and only a single context is being used and the mpi thread - * mode is not mpi thread multiple. This is unlikely to be true in the - * mpich "per obj" thread granularity compile. As an optimization, do not - * check for this condition and simply always perform the context lock. - */ - rc = PAMI_Context_trylock_advancev(MPIDI_Context, MPIDI_Process.avail_contexts, 1); - MPID_assert( (rc == PAMI_SUCCESS) || (rc == PAMI_EAGAIN) ); - } -#else - /* - * In the "global lock" thread mode the single mpich lock has already been - * acquired at this point. Any other application thread or asynchronous - * progress execution resource must also acquire this global lock and will - * block until the global lock is cycled or released. - * - * Because only one thread will enter this code at any time, including any - * async progress threads if async progress is enabled, it is unneccesary - * to acquire any context locks before the context advance operation. - * - * NOTE: The 'NULL' progress function configuation for async progress is not - * valid in the 'global' mpich lock mode. See discussion above for more - * information. - * - * NOTE: There is a simplifying assertion for the "global" mpich lock mode - * that only a single context is supported. See discussion in above for - * more information. - */ - rc = PAMI_Context_advance(MPIDI_Context[0], 1); - MPID_assert( (rc == PAMI_SUCCESS) || (rc == PAMI_EAGAIN) ); -#ifdef __PE__ - if (rc == PAMI_EAGAIN) { - MPIU_THREAD_CS_SCHED_YIELD(ALLFUNC,); /* sync, release(0), yield, acquire(0) */ - } else -#endif - MPIU_THREAD_CS_YIELD(ALLFUNC,); /* sync, release(0), acquire(0) */ -#endif - - return MPI_SUCCESS; -} - -/** \} */ - - -#endif diff --git a/src/mpid/pamid/src/mpid_recvq.c b/src/mpid/pamid/src/mpid_recvq.c deleted file mode 100644 index f0e00cb..0000000 --- a/src/mpid/pamid/src/mpid_recvq.c +++ /dev/null @@ -1,945 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/mpid_recvq.c - * \brief Functions to manage the Receive Queues - */ -/* - * (C) 2001 by Argonne National Laboratory. - * See COPYRIGHT in top-level directory. - */ - -#include <mpidimpl.h> -#include "mpid_recvq.h" - -/** - * \defgroup MPID_RECVQ MPID Receive Queue management - * - * Functions to manage the Receive Queues - */ - - -/** \brief Structure to group the common recvq pointers */ -struct MPIDI_Recvq_t MPIDI_Recvq; - -#ifdef HAVE_DEBUGGER_SUPPORT -struct MPIR_Request ** const MPID_Recvq_posted_head_ptr = - &MPIDI_Recvq.posted_head; -struct MPIR_Request ** const MPID_Recvq_unexpected_head_ptr = - &MPIDI_Recvq.unexpected_head; -#endif /* HAVE_DEBUGGER_SUPPORT */ - - -/** - * \brief Set up the request queues - */ -void -MPIDI_Recvq_init() -{ - MPIDI_Recvq.posted_head = NULL; - MPIDI_Recvq.posted_tail = NULL; - MPIDI_Recvq.unexpected_head = NULL; - MPIDI_Recvq.unexpected_tail = NULL; -#ifdef QUEUE_BINARY_SEARCH_SUPPORT - if(MPIDI_Process.queue_binary_search_support_on) - MPIDI_Recvq_init_queues(); -#endif -} - - -/** - * \brief Tear down the request queues - */ -void -MPIDI_Recvq_finalize() -{ - if(MPIDI_Process.statistics) MPIDI_Recvq_DumpQueues(MPIDI_Process.verbose); -} - - -/** - * \brief Find a request in the unexpected queue - * \param[in] source Find by Sender - * \param[in] tag Find by Tag - * \param[in] context_id Find by Context ID (communicator) - * \return 1/0 if the matching UE request was found or not - */ -int -MPIDI_Recvq_FU(int source, int tag, int context_id, MPI_Status * status) -{ - MPIR_Request * rreq; - int found = FALSE; - void * it; -#ifdef USE_STATISTICS - unsigned search_length = 0; -#endif -#ifdef OUT_OF_ORDER_HANDLING - MPIDI_In_cntr_t *in_cntr; - uint nMsgs=0; - pami_task_t pami_source; -#endif - - if (tag != MPI_ANY_TAG && source != MPI_ANY_SOURCE) - { -#ifdef QUEUE_BINARY_SEARCH_SUPPORT - if(MPIDI_Process.queue_binary_search_support_on) - { -#ifndef OUT_OF_ORDER_HANDLING - MPIDI_Recvq_find_in_uexp(source, tag, context_id, &rreq, &it); -#else - MPIDI_Recvq_find_in_uexp(source, tag, context_id, -1, &rreq, &it); -#endif - if (rreq != NULL) { - #ifdef OUT_OF_ORDER_HANDLING - pami_source= MPIDI_Request_getPeerRank_pami(rreq); - in_cntr=&MPIDI_In_cntr[pami_source]; - nMsgs = in_cntr->nMsgs + 1; - if( ((int)(nMsgs-MPIDI_Request_getMatchSeq(rreq))) >= 0 ) - { - if (rreq->mpid.nextR != NULL) { /* recv is in the out of order list */ - if (MPIDI_Request_getMatchSeq(rreq) == nMsgs) { - in_cntr->nMsgs=nMsgs; - MPIDI_Recvq_remove_req_from_ool(rreq,in_cntr); - } - } - #endif - found = TRUE; - if(status != MPI_STATUS_IGNORE) - *status = (rreq->status); -#ifdef OUT_OF_ORDER_HANDLING - } - else - { - MPIDI_Recvq_find_in_uexp(source, tag, context_id, nMsgs, &rreq, &it); - if (rreq != NULL) { - if (rreq->mpid.nextR != NULL) { /* recv is in the out of order list */ - in_cntr->nMsgs=nMsgs; - MPIDI_Recvq_remove_req_from_ool(rreq,in_cntr); - } - found = TRUE; - if(status != MPI_STATUS_IGNORE) - *status = (rreq->status); - } - } -#endif /* OUT_OF_ORDER_HANDLING */ - } - } - else - { -#endif /* QUEUE_BINARY_SEARCH_SUPPORT */ - rreq = MPIDI_Recvq.unexpected_head; - while (rreq != NULL) { -#ifdef USE_STATISTICS - ++search_length; -#endif - if ( (MPIDI_Request_getMatchCtxt(rreq) == context_id) && - (MPIDI_Request_getMatchRank(rreq) == source ) && - (MPIDI_Request_getMatchTag(rreq) == tag ) - ) - { - #ifdef OUT_OF_ORDER_HANDLING - pami_source= MPIDI_Request_getPeerRank_pami(rreq); - in_cntr=&MPIDI_In_cntr[pami_source]; - nMsgs = in_cntr->nMsgs + 1; - if( ((int)(nMsgs-MPIDI_Request_getMatchSeq(rreq))) >= 0 ) - { - if (rreq->mpid.nextR != NULL) { /* recv is in the out of order list */ - if (MPIDI_Request_getMatchSeq(rreq) == nMsgs) { - in_cntr->nMsgs=nMsgs; - MPIDI_Recvq_remove_req_from_ool(rreq,in_cntr); - } - } - #endif - found = TRUE; - if(status != MPI_STATUS_IGNORE) - *status = (rreq->status); - break; -#ifdef OUT_OF_ORDER_HANDLING - } -#endif - } - rreq = rreq->mpid.next; - } -#ifdef QUEUE_BINARY_SEARCH_SUPPORT - } -#endif /* QUEUE_BINARY_SEARCH_SUPPORT */ - } - else - { - MPIDI_Message_match match; - MPIDI_Message_match mask; - - match.context_id = context_id; - mask.context_id = ~0; - if (tag == MPI_ANY_TAG) - { - match.tag = 0; - mask.tag = 0; - } - else - { - match.tag = tag; - mask.tag = ~0; - } - if (source == MPI_ANY_SOURCE) - { - match.rank = 0; - mask.rank = 0; - } - else - { - match.rank = source; - mask.rank = ~0; - } -#ifdef QUEUE_BINARY_SEARCH_SUPPORT - if(MPIDI_Process.queue_binary_search_support_on) - { -#ifndef OUT_OF_ORDER_HANDLING - MPIDI_Recvq_find_in_uexp(source, tag, match.context_id, &rreq, &it); -#else - MPIDI_Recvq_find_in_uexp(source, tag, match.context_id, -1, &rreq, &it); -#endif - if (rreq != NULL) { -#ifdef OUT_OF_ORDER_HANDLING - pami_source= MPIDI_Request_getPeerRank_pami(rreq); - in_cntr=&MPIDI_In_cntr[pami_source]; - nMsgs = in_cntr->nMsgs + 1; - if(( ( (int)(nMsgs-MPIDI_Request_getMatchSeq(rreq))) >= 0) || (source == MPI_ANY_SOURCE)) { - if(source == MPI_ANY_SOURCE) { - if((int) (nMsgs-MPIDI_Request_getMatchSeq(rreq)) < 0 ) - goto NEXT_MSG2; - } - if (rreq->mpid.nextR != NULL) { /* recv is in the out of order list */ - if (MPIDI_Request_getMatchSeq(rreq) == nMsgs) - in_cntr->nMsgs=nMsgs; - MPIDI_Recvq_remove_req_from_ool(rreq,in_cntr); - } -#endif - found = TRUE; - if(status != MPI_STATUS_IGNORE) - *status = (rreq->status); -#ifdef OUT_OF_ORDER_HANDLING - } - else - { - NEXT_MSG2: MPIDI_Recvq_find_in_uexp(source, tag, match.context_id, nMsgs, &rreq, &it); - if (rreq != NULL) { - if (rreq->mpid.nextR != NULL) { /* recv is in the out of order list */ - in_cntr->nMsgs=nMsgs; - MPIDI_Recvq_remove_req_from_ool(rreq,in_cntr); - } - found = TRUE; - if(status != MPI_STATUS_IGNORE) - *status = (rreq->status); - } - } -#endif /* OUT_OF_ORDER_HANDLING */ - } - } - else - { -#endif /* QUEUE_BINARY_SEARCH_SUPPORT */ - rreq = MPIDI_Recvq.unexpected_head; - while (rreq != NULL) { -#ifdef USE_STATISTICS - ++search_length; -#endif - if ( ( MPIDI_Request_getMatchCtxt(rreq) == match.context_id) && - ( (MPIDI_Request_getMatchRank(rreq) & mask.rank) == match.rank ) && - ( (MPIDI_Request_getMatchTag(rreq) & mask.tag ) == match.tag ) - ) - { -#ifdef OUT_OF_ORDER_HANDLING - pami_source= MPIDI_Request_getPeerRank_pami(rreq); - in_cntr=&MPIDI_In_cntr[pami_source]; - nMsgs = in_cntr->nMsgs + 1; - if(( ( (int)(nMsgs-MPIDI_Request_getMatchSeq(rreq))) >= 0) || (source == MPI_ANY_SOURCE)) { - if(source == MPI_ANY_SOURCE) { - if((int) (nMsgs-MPIDI_Request_getMatchSeq(rreq)) < 0 ) - goto NEXT_MSG; - } - if (rreq->mpid.nextR != NULL) { /* recv is in the out of order list */ - if (MPIDI_Request_getMatchSeq(rreq) == nMsgs) - in_cntr->nMsgs=nMsgs; - MPIDI_Recvq_remove_req_from_ool(rreq,in_cntr); - } -#endif - found = TRUE; - if(status != MPI_STATUS_IGNORE) - *status = (rreq->status); - break; -#ifdef OUT_OF_ORDER_HANDLING - } -#endif - } - NEXT_MSG: - rreq = rreq->mpid.next; - } -#ifdef QUEUE_BINARY_SEARCH_SUPPORT - } -#endif - } - -#ifdef USE_STATISTICS - MPIDI_Statistics_time(MPIDI_Statistics.recvq.unexpected_search, search_length); -#endif - - return found; -} - - -/** - * \brief Find a request in the unexpected queue and dequeue it - * \param[in] req Find by address of request object on sender - * \param[in] source Find by Sender - * \param[in] tag Find by Tag - * \param[in] context_id Find by Context ID (communicator) - * \return The matching UE request or NULL - */ -MPIR_Request * -MPIDI_Recvq_FDUR(MPI_Request req, int source, int tag, int context_id) -{ - MPIR_Request * prev_rreq = NULL; /* previous request in queue */ - MPIR_Request * cur_rreq = NULL; /* current request in queue */ - MPIR_Request * matching_cur_rreq = NULL; /* matching request in queue */ - MPIR_Request * matching_prev_rreq = NULL; /* previous in queue to match */ - void * it = NULL; -#ifdef USE_STATISTICS - unsigned search_length = 0; -#endif - - /* ----------------------- */ - /* first we do the finding */ - /* ----------------------- */ -#ifdef QUEUE_BINARY_SEARCH_SUPPORT - if(MPIDI_Process.queue_binary_search_support_on) - { -#ifndef OUT_OF_ORDER_HANDLING - MPIDI_Recvq_find_in_uexp(source, tag, context_id, &cur_rreq, &it); -#else - /* Passing -1 for seqno because I don't care about it here */ - MPIDI_Recvq_find_in_uexp(source, tag, context_id, -1, &cur_rreq, &it); -#endif - if (cur_rreq != NULL) { - if (MPIDI_Request_getPeerRequestH(cur_rreq) == req) - { - matching_prev_rreq = prev_rreq; - matching_cur_rreq = cur_rreq; - } - } - } - else - { -#endif - cur_rreq = MPIDI_Recvq.unexpected_head; - while (cur_rreq != NULL) { -#ifdef USE_STATISTICS - ++search_length; -#endif - if (MPIDI_Request_getPeerRequestH(cur_rreq) == req && - MPIDI_Request_getMatchCtxt(cur_rreq) == context_id && - MPIDI_Request_getMatchRank(cur_rreq) == source && - MPIDI_Request_getMatchTag(cur_rreq) == tag) - { - matching_prev_rreq = prev_rreq; - matching_cur_rreq = cur_rreq; - break; - } - prev_rreq = cur_rreq; - cur_rreq = cur_rreq->mpid.next; - } -#ifdef QUEUE_BINARY_SEARCH_SUPPORT - } -#endif - - /* ----------------------- */ - /* found nothing; return */ - /* ----------------------- */ - if (matching_cur_rreq == NULL) - goto fn_exit; -#ifdef QUEUE_BINARY_SEARCH_SUPPORT - if(MPIDI_Process.queue_binary_search_support_on) - { -#ifndef OUT_OF_ORDER_HANDLING - MPIDI_Recvq_remove_uexp(source, tag, context_id, it); -#else - MPIDI_Recvq_remove_uexp(source, tag, context_id, MPIDI_Request_getMatchSeq(cur_rreq),it); -#endif - } - else -#endif - MPIDI_Recvq_remove(MPIDI_Recvq.unexpected, matching_cur_rreq, matching_prev_rreq); - - fn_exit: -#ifdef USE_STATISTICS - MPIDI_Statistics_time(MPIDI_Statistics.recvq.unexpected_search, search_length); -#endif - - return matching_cur_rreq; -} - - -/** - * \brief Out of band part of find a request in the unexpected queue and dequeue it, or allocate a new request and enqueue it in the posted queue - * \param[in] source Find by Sender - * \param[in] tag Find by Tag - * \param[in] context_id Find by Context ID (communicator) - * \param[out] foundp TRUE iff the request was found - * \return The matching UE request or the new posted request - */ -#ifndef OUT_OF_ORDER_HANDLING -MPIR_Request * -MPIDI_Recvq_FDU(int source, int tag, int context_id, int * foundp) -#else -MPIR_Request * -MPIDI_Recvq_FDU(int source, pami_task_t pami_source, int tag, int context_id, int * foundp) -#endif -{ - int found = FALSE; - MPIR_Request * rreq = NULL; - MPIR_Request * prev_rreq; - void * it; -#ifdef USE_STATISTICS - unsigned search_length = 0; -#endif -#ifdef OUT_OF_ORDER_HANDLING - MPIDI_In_cntr_t *in_cntr; - uint nMsgs=0; - - if(pami_source != MPI_ANY_SOURCE) { - in_cntr=&MPIDI_In_cntr[pami_source]; - nMsgs = in_cntr->nMsgs + 1; - } -#endif - - //This function is typically called when there are unexp recvs - if (tag != MPI_ANY_TAG && source != MPI_ANY_SOURCE) - { -#ifdef QUEUE_BINARY_SEARCH_SUPPORT - if(MPIDI_Process.queue_binary_search_support_on) - { -#ifdef OUT_OF_ORDER_HANDLING - MPIDI_Recvq_find_in_uexp(source, tag, context_id, nMsgs, &rreq, &it); -#else - MPIDI_Recvq_find_in_uexp(source, tag, context_id, &rreq, &it); -#endif - if (rreq != NULL) { -#ifdef OUT_OF_ORDER_HANDLING - if(rreq->mpid.nextR != NULL) { /* recv is in the out of order list */ - if (MPIDI_Request_getMatchSeq(rreq) == nMsgs) { - in_cntr->nMsgs=nMsgs; - } - MPIDI_Recvq_remove_req_from_ool(rreq,in_cntr); - } - MPIDI_Recvq_remove_uexp(source, tag, context_id, MPIDI_Request_getMatchSeq(rreq),it); -#else - MPIDI_Recvq_remove_uexp(source, tag, context_id, it); -#endif - found = TRUE; -#ifdef MPIDI_TRACE - MPIDI_Trace_buf[(rreq->mpid.partner_id)].R[(rreq->mpid.idx)].matchedInUQ2=1; -#endif - } - } - else - { -#endif - prev_rreq = NULL; - rreq = MPIDI_Recvq.unexpected_head; - while (rreq != NULL) { -#ifdef USE_STATISTICS - ++search_length; -#endif -#ifdef OUT_OF_ORDER_HANDLING - if( ((int)(nMsgs-MPIDI_Request_getMatchSeq(rreq))) >= 0 ) { -#endif - if ( (MPIDI_Request_getMatchCtxt(rreq) == context_id) && - (MPIDI_Request_getMatchRank(rreq) == source ) && - (MPIDI_Request_getMatchTag(rreq) == tag ) - ) - { -#ifdef OUT_OF_ORDER_HANDLING - if(rreq->mpid.nextR != NULL) { /* recv is in the out of order list */ - if (MPIDI_Request_getMatchSeq(rreq) == nMsgs) { - in_cntr->nMsgs=nMsgs; - } - MPIDI_Recvq_remove_req_from_ool(rreq,in_cntr); - } -#endif - MPIDI_Recvq_remove(MPIDI_Recvq.unexpected, rreq, prev_rreq); - found = TRUE; - TRACE_SET_R_BIT((rreq->mpid.partner_id),(rreq->mpid.idx),fl.f.matchedInUQ2); - goto fn_exit; - } -#ifdef OUT_OF_ORDER_HANDLING - } -#endif - - prev_rreq = rreq; - rreq = rreq->mpid.next; - } -#ifdef QUEUE_BINARY_SEARCH_SUPPORT - } -#endif - } - else - { - MPIDI_Message_match match; - MPIDI_Message_match mask; - - match.context_id = context_id; - mask.context_id = ~0; - if (tag == MPI_ANY_TAG) - { - match.tag = 0; - mask.tag = 0; - } - else - { - match.tag = tag; - mask.tag = ~0; - } - if (source == MPI_ANY_SOURCE) - { - match.rank = 0; - mask.rank = 0; - } - else - { - match.rank = source; - mask.rank = ~0; - } - -#ifdef QUEUE_BINARY_SEARCH_SUPPORT - if(MPIDI_Process.queue_binary_search_support_on) - { -#ifndef OUT_OF_ORDER_HANDLING - MPIDI_Recvq_find_in_uexp(source, tag, match.context_id, &rreq, &it); -#else - MPIDI_Recvq_find_in_uexp(source, tag, match.context_id, -1, &rreq, &it); -#endif - if (rreq != NULL) { -#ifdef OUT_OF_ORDER_HANDLING - if(( ( (int)(nMsgs-MPIDI_Request_getMatchSeq(rreq))) >= 0) || (source == MPI_ANY_SOURCE)) { - if(source == MPI_ANY_SOURCE) { - in_cntr = &MPIDI_In_cntr[MPIDI_Request_getPeerRank_pami(rreq)]; - nMsgs = in_cntr->nMsgs+1; - if((int) (nMsgs-MPIDI_Request_getMatchSeq(rreq)) < 0 ) - goto NEXT_MSG2; - } - if (rreq->mpid.nextR != NULL) { /* recv is in the out of order list */ - if (MPIDI_Request_getMatchSeq(rreq) == nMsgs) - in_cntr->nMsgs=nMsgs; - MPIDI_Recvq_remove_req_from_ool(rreq,in_cntr); - } - MPIDI_Recvq_remove_uexp(MPIDI_Request_getMatchRank(rreq), MPIDI_Request_getMatchTag(rreq), match.context_id, MPIDI_Request_getMatchSeq(rreq),it); -#else /* OUT_OF_ORDER_HANDLING */ - MPIDI_Recvq_remove_uexp(MPIDI_Request_getMatchRank(rreq), MPIDI_Request_getMatchTag(rreq), match.context_id, it); -#endif/* OUT_OF_ORDER_HANDLING */ - found = TRUE; - goto fn_exit; -#ifdef OUT_OF_ORDER_HANDLING - } - NEXT_MSG2: - MPIDI_Recvq_find_in_uexp(source, tag, match.context_id, nMsgs, &rreq, &it); - if (rreq != NULL) { - if (rreq->mpid.nextR != NULL) { /* recv is in the out of order list */ - if (MPIDI_Request_getMatchSeq(rreq) == nMsgs) - in_cntr->nMsgs=nMsgs; - MPIDI_Recvq_remove_req_from_ool(rreq,in_cntr); - } - MPIDI_Recvq_remove_uexp(MPIDI_Request_getMatchRank(rreq), MPIDI_Request_getMatchTag(rreq), match.context_id, MPIDI_Request_getMatchSeq(rreq),it); - } -#endif /* OUT_OF_ORDER_HANDLING */ - - } - } - else - { -#endif - prev_rreq = NULL; - rreq = MPIDI_Recvq.unexpected_head; - while (rreq != NULL) { -#ifdef USE_STATISTICS - ++search_length; -#endif -#ifdef OUT_OF_ORDER_HANDLING - if(( ( (int)(nMsgs-MPIDI_Request_getMatchSeq(rreq))) >= 0) || (source == MPI_ANY_SOURCE)) { -#endif - if ( ( MPIDI_Request_getMatchCtxt(rreq) == match.context_id) && - ( (MPIDI_Request_getMatchRank(rreq) & mask.rank) == match.rank ) && - ( (MPIDI_Request_getMatchTag(rreq) & mask.tag ) == match.tag ) - ) - { -#ifdef OUT_OF_ORDER_HANDLING - if(source == MPI_ANY_SOURCE) { - in_cntr = &MPIDI_In_cntr[MPIDI_Request_getPeerRank_pami(rreq)]; - nMsgs = in_cntr->nMsgs+1; - if((int) (nMsgs-MPIDI_Request_getMatchSeq(rreq)) < 0 ) - goto NEXT_MSG; - } - if (rreq->mpid.nextR != NULL) { /* recv is in the out of order list */ - if (MPIDI_Request_getMatchSeq(rreq) == nMsgs) - in_cntr->nMsgs=nMsgs; - MPIDI_Recvq_remove_req_from_ool(rreq,in_cntr); - } -#endif - MPIDI_Recvq_remove(MPIDI_Recvq.unexpected, rreq, prev_rreq); - found = TRUE; - goto fn_exit; - } -#ifdef OUT_OF_ORDER_HANDLING - } - NEXT_MSG: -#endif - prev_rreq = rreq; - rreq = rreq->mpid.next; - } -#ifdef QUEUE_BINARY_SEARCH_SUPPORT - } -#endif - } - - fn_exit: -#ifdef USE_STATISTICS - MPIDI_Statistics_time(MPIDI_Statistics.recvq.unexpected_search, search_length); -#endif - - *foundp = found; - return rreq; -} - - -/** - * \brief Find a request in the posted queue and dequeue it - * \param[in] req Find by address of request object on sender - * \return The matching posted request or NULL - */ -int -MPIDI_Recvq_FDPR(MPIR_Request * req) -{ - MPIR_Request * cur_rreq = NULL; - MPIR_Request * prev_rreq = NULL; - int found = FALSE; -#ifdef USE_STATISTICS - unsigned search_length = 0; -#endif - -#ifdef QUEUE_BINARY_SEARCH_SUPPORT - void * it; - if(MPIDI_Process.queue_binary_search_support_on) - { - MPIDI_Recvq_find_in_post(MPIDI_Request_getMatchRank(req), MPIDI_Request_getMatchTag(req), MPIDI_Request_getMatchCtxt(req), &cur_rreq, &it); - if (cur_rreq != NULL) { - if (cur_rreq == req) - { - MPIDI_Recvq_remove_post(MPIDI_Request_getMatchRank(req), MPIDI_Request_getMatchTag(req), MPIDI_Request_getMatchCtxt(req),it); - found = TRUE; - } - } - } - else - { -#endif /* QUEUE_BINARY_SEARCH_SUPPORT */ - cur_rreq = MPIDI_Recvq.posted_head; - - while (cur_rreq != NULL) { -#ifdef USE_STATISTICS - ++search_length; -#endif - if (cur_rreq == req) - { - MPIDI_Recvq_remove(MPIDI_Recvq.posted, cur_rreq, prev_rreq); - found = TRUE; - break; - } - - prev_rreq = cur_rreq; - cur_rreq = cur_rreq->mpid.next; - } -#ifdef QUEUE_BINARY_SEARCH_SUPPORT - } -#endif -#ifdef USE_STATISTICS - MPIDI_Statistics_time(MPIDI_Statistics.recvq.posted_search, search_length); -#endif - - return found; -} - - -/** - * \brief Find a request in the posted queue and dequeue it, or allocate a new request and enqueue it in the unexpected queue - * \param[in] source Find by Sender - * \param[in] tag Find by Tag - * \param[in] context_id Find by Context ID (communicator) - * \param[out] foundp TRUE iff the request was found - * \return The matching posted request or the new UE request - */ -#ifndef OUT_OF_ORDER_HANDLING -MPIR_Request * -MPIDI_Recvq_FDP_or_AEU(MPIR_Request *newreq, int source, int tag, int context_id, int * foundp) -#else -MPIR_Request * -MPIDI_Recvq_FDP_or_AEU(MPIR_Request *newreq, int source, pami_task_t pami_source, int tag, int context_id, int msg_seqno, int * foundp) -#endif -{ - MPIR_Request * rreq; - int found = FALSE; - -#ifndef OUT_OF_ORDER_HANDLING - rreq = MPIDI_Recvq_FDP(source, tag, context_id); -#else - rreq = MPIDI_Recvq_FDP(source, pami_source, tag, context_id, msg_seqno); -#endif - - if (rreq != NULL) { - found = TRUE; - } else { -#ifndef OUT_OF_ORDER_HANDLING - rreq = MPIDI_Recvq_AEU(newreq, source, tag, context_id); -#else - rreq = MPIDI_Recvq_AEU(newreq, source, pami_source, tag, context_id, msg_seqno); -#endif - } - *foundp = found; - return rreq; -} - - -/** - * \brief Allocate a new request and enqueue it in the unexpected queue - * \param[in] source Find by Sender - * \param[in] tag Find by Tag - * \param[in] context_id Find by Context ID (communicator) - * \return The matching posted request or the new UE request - */ -#ifndef OUT_OF_ORDER_HANDLING -MPIR_Request * -MPIDI_Recvq_AEU(MPIR_Request *newreq, int source, int tag, int context_id) -#else -MPIR_Request * -MPIDI_Recvq_AEU(MPIR_Request *newreq, int source, pami_task_t pami_source, int tag, int context_id, int msg_seqno) -#endif -{ - /* A matching request was not found in the posted queue, so we - need to allocate a new request and add it to the unexpected - queue */ - MPIR_Request *rreq; - rreq = newreq; - rreq->kind = MPIR_REQUEST_KIND__RECV; - TRACE_MEMSET_R(pami_source,msg_seqno,recv_status); - TRACE_SET_REQ_VAL(rreq->mpid.envelope.msginfo.MPIseqno,-1); - TRACE_SET_REQ_VAL(rreq->mpid.envelope.length,-1); - TRACE_SET_REQ_VAL(rreq->mpid.envelope.data,(void *) 0); -#ifndef OUT_OF_ORDER_HANDLING - MPIDI_Request_setMatch(rreq, tag, source, context_id); -#ifdef QUEUE_BINARY_SEARCH_SUPPORT - if(MPIDI_Process.queue_binary_search_support_on) - MPIDI_Recvq_insert_uexp((void*)rreq, source, tag, context_id); - else -#endif - MPIDI_Recvq_append(MPIDI_Recvq.unexpected, rreq); -#else /* OUT_OF_ORDER_HANDLING */ - MPIR_Request *q; - MPIDI_In_cntr_t *in_cntr; - int insert, i; - - in_cntr = &MPIDI_In_cntr[pami_source]; - MPIDI_Request_setMatch(rreq, tag, source, context_id); /* mpi rank needed */ - MPIDI_Request_setPeerRank_pami(rreq, pami_source); - MPIDI_Request_setPeerRank_comm(rreq, source); - MPIDI_Request_setMatchSeq(rreq, msg_seqno); -#ifdef QUEUE_BINARY_SEARCH_SUPPORT - if(MPIDI_Process.queue_binary_search_support_on) - { - MPIDI_Recvq_insert_uexp((void*)rreq, source, tag, context_id, msg_seqno); - } - else - { -#endif - if (!in_cntr->n_OutOfOrderMsgs) { - MPIDI_Recvq_append(MPIDI_Recvq.unexpected, rreq); - } else { - q=in_cntr->OutOfOrderList; - insert=0; - for (i=1; i<=in_cntr->n_OutOfOrderMsgs; i++) { - if ( context_id == MPIDI_Request_getMatchCtxt(q)) { - if (((int)(msg_seqno - MPIDI_Request_getMatchSeq(q))) < 0) { - MPIDI_Recvq_insert(MPIDI_Recvq.unexpected, q, rreq); - insert=1; - break; - } - } - q=q->mpid.nextR; - } - if (!insert) { - MPIDI_Recvq_append(MPIDI_Recvq.unexpected, rreq); - } - } - TRACE_SET_R_VAL(pami_source,(msg_seqno & SEQMASK),req,rreq); - TRACE_SET_R_VAL(pami_source,(msg_seqno & SEQMASK),msgid,msg_seqno); - TRACE_SET_R_BIT(pami_source,(msg_seqno & SEQMASK),fl.f.ool); - TRACE_SET_R_VAL(pami_source,(msg_seqno & SEQMASK),rtag,tag); - TRACE_SET_R_VAL(pami_source,(msg_seqno & SEQMASK),rctx,context_id); - TRACE_SET_REQ_VAL(rreq->mpid.idx,(msg_seqno & SEQMASK)); - TRACE_SET_R_VAL(pami_source,(msg_seqno & SEQMASK),rsource,pami_source); - TRACE_SET_REQ_VAL(rreq->mpid.partner_id,pami_source); -#ifdef QUEUE_BINARY_SEARCH_SUPPORT - } -#endif - - if (((int)(in_cntr->nMsgs - msg_seqno)) < 0) { /* seqno > nMsgs, out of order */ - MPIDI_Recvq_enqueue_ool(pami_source,rreq); - } -#endif/* OUT_OF_ORDER_HANDLING */ - - return rreq; -} - - -/** - * \brief Dump the queues - */ -void -MPIDI_Recvq_DumpQueues(int verbose) -{ - if(verbose < MPIDI_VERBOSE_SUMMARY_ALL) - return; - - MPIR_Request * rreq = MPIDI_Recvq.posted_head; - unsigned i=0, numposted=0, numue=0; - unsigned postedbytes=0, uebytes=0; - - if(verbose >= MPIDI_VERBOSE_DETAILS_ALL) - fprintf(stderr,"Posted Queue:\n-------------\n"); - while (rreq != NULL) { - if(verbose >= MPIDI_VERBOSE_DETAILS_ALL) - fprintf(stderr, "P %d: MPItag=%d MPIrank=%d ctxt=%d count=%d\n", - i++, - MPIDI_Request_getMatchTag(rreq), - MPIDI_Request_getMatchRank(rreq), - MPIDI_Request_getMatchCtxt(rreq), - rreq->mpid.userbufcount - ); - numposted++; - postedbytes+=rreq->mpid.userbufcount; - rreq = rreq->mpid.next; - } - fprintf(stderr, "Posted Requests %d, Total Mem: %d bytes\n", - numposted, postedbytes); - - - i=0; - rreq = MPIDI_Recvq.unexpected_head; - if(verbose >= MPIDI_VERBOSE_DETAILS_ALL) - fprintf(stderr, "Unexpected Queue:\n-----------------\n"); - while (rreq != NULL) { - if(verbose >= MPIDI_VERBOSE_DETAILS_ALL) -#ifndef OUT_OF_ORDER_HANDLING - fprintf(stderr, "UE %d: MPItag=%d MPIrank=%d ctxt=%d uebuf=%p uebuflen=%u\n", - i++, - MPIDI_Request_getMatchTag(rreq), - MPIDI_Request_getMatchRank(rreq), - MPIDI_Request_getMatchCtxt(rreq), - rreq->mpid.uebuf, - rreq->mpid.uebuflen); -#else - fprintf(stderr, "UE %d: MPItag=%d MPIrank=%d pami_task_id=%d MPIseq=%d ctxt=%d uebuf=%p uebuflen=%u\n", - i++, - MPIDI_Request_getMatchTag(rreq), - MPIDI_Request_getMatchRank(rreq), - MPIDI_Request_getPeerRank_pami(rreq), - MPIDI_Request_getMatchSeq(rreq), - MPIDI_Request_getMatchCtxt(rreq), - rreq->mpid.uebuf, - rreq->mpid.uebuflen); -#endif - numue++; - uebytes+=rreq->mpid.uebuflen; - rreq = rreq->mpid.next; - } - fprintf(stderr, "Unexpected Requests %d, Total Mem: %d bytes\n", - numue, uebytes); -} - - -#ifdef OUT_OF_ORDER_HANDLING -/** - * Insert a request in the OutOfOrderList, make sure this list is - * arranged in the ascending order. - */ -void MPIDI_Recvq_enqueue_ool(pami_task_t src, MPIR_Request *req) -{ - MPIR_Request *q; - void *head; - int insert,i; - MPIDI_In_cntr_t *in_cntr; - - in_cntr=&MPIDI_In_cntr[src]; - if (in_cntr->n_OutOfOrderMsgs != 0) { - head=in_cntr->OutOfOrderList; - q=in_cntr->OutOfOrderList; - insert=0; - MPID_assert(q->mpid.nextR != NULL); - while(q->mpid.nextR != head) { - if (((int)(MPIDI_Request_getMatchSeq(q) - MPIDI_Request_getMatchSeq(req))) > 0) { - insert=1; - break; - } - q=q->mpid.nextR; - } - if (insert) { - MPIDI_Recvq_insert_ool(q,req); - if (q == head) { /* 1st element in the list */ - in_cntr->OutOfOrderList=req; - } - } else { - if (((int)(MPIDI_Request_getMatchSeq(q) - MPIDI_Request_getMatchSeq(req))) > 0) { - MPIDI_Recvq_insert_ool(q,req); - if (q == head) { /* 1st element in the list */ - in_cntr->OutOfOrderList=req; - } - } else { - MPIDI_Recvq_insert_ool((MPIR_Request *)q->mpid.nextR,req); - } - } - } else { /* empty list */ - in_cntr->OutOfOrderList=req; - req->mpid.prevR=req; - req->mpid.nextR=req; - } - in_cntr->n_OutOfOrderMsgs++; -#if (MPIDI_STATISTICS) - MPID_NSTAT(mpid_statp->unorderedMsgs); -#endif -} /* void MPIDI_Recvq_insert_ool(pami_task_t src, MPIR_Request *N) */ - - -/** - * MPIDI_Recvq_insert_ool: place e between q and q->prevR - * - */ -void MPIDI_Recvq_insert_ool(MPIR_Request *q,MPIR_Request *e) -{ - (e)->mpid.prevR = (q)->mpid.prevR; - ((MPIR_Request *)((q)->mpid.prevR))->mpid.nextR = (e); - (e)->mpid.nextR = (q); - (q)->mpid.prevR = (e); -} -#endif diff --git a/src/mpid/pamid/src/mpid_recvq.h b/src/mpid/pamid/src/mpid_recvq.h deleted file mode 100644 index 12b8733..0000000 --- a/src/mpid/pamid/src/mpid_recvq.h +++ /dev/null @@ -1,425 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/mpid_recvq.h - * \brief Accessors and actors for MPID Requests - */ -#ifndef __src_pt2pt_mpidi_recvq_h__ -#define __src_pt2pt_mpidi_recvq_h__ - - -struct MPIDI_Recvq_t -{ - struct MPIR_Request * posted_head; /**< \brief The Head of the Posted queue */ - struct MPIR_Request * posted_tail; /**< \brief The Tail of the Posted queue */ - struct MPIR_Request * unexpected_head; /**< \brief The Head of the Unexpected queue */ - struct MPIR_Request * unexpected_tail; /**< \brief The Tail of the Unexpected queue */ -}; -extern struct MPIDI_Recvq_t MPIDI_Recvq; - - -#ifndef OUT_OF_ORDER_HANDLING -#define MPIDI_Recvq_append(__Q, __req) \ -({ \ - /* ---------------------------------------------- */ \ - /* The tail request should point to the new one */ \ - /* ---------------------------------------------- */ \ - if (__Q ## _tail != NULL) \ - __Q ## _tail->mpid.next = __req; \ - else \ - __Q ## _head = __req; \ - /* ------------------------------------------ */ \ - /* The tail should point to the new request */ \ - /* ------------------------------------------ */ \ - __Q ## _tail = __req; \ -}) -#else -#define MPIDI_Recvq_append(__Q, __req) \ -({ \ - /* ---------------------------------------------- */ \ - /* The tail request should point to the new one */ \ - /* ---------------------------------------------- */ \ - if (__Q ## _tail != NULL) { \ - __Q ## _tail->mpid.next = __req; \ - __req->mpid.prev = __Q ## _tail; \ - } \ - else { \ - __Q ## _head = __req; \ - __req->mpid.prev = NULL; \ - } \ - /* ------------------------------------------ */ \ - /* The tail should point to the new request */ \ - /* ------------------------------------------ */ \ - __Q ## _tail = __req; \ -}) -#endif - - -#ifndef OUT_OF_ORDER_HANDLING -#define MPIDI_Recvq_remove(__Q, __req, __prev) \ -({ \ - /* --------------------------------------------- */ \ - /* Patch the next pointers to skip the request */ \ - /* --------------------------------------------- */ \ - if (__prev != NULL) \ - __prev->mpid.next = __req->mpid.next; \ - else \ - __Q ## _head = __req->mpid.next; \ - /* ------------------------------------------- */ \ - /* Set tail pointer if removing the last one */ \ - /* ------------------------------------------- */ \ - if (__req->mpid.next == NULL) \ - __Q ## _tail = __prev; \ -}) -#else -#define MPIDI_Recvq_remove(__Q, __req, __prev) \ -({ \ - /* --------------------------------------------- */ \ - /* Patch the next pointers to skip the request */ \ - /* --------------------------------------------- */ \ - if (__prev != NULL) { \ - __prev->mpid.next = __req->mpid.next; \ - } \ - else \ - __Q ## _head = __req->mpid.next; \ - /* ------------------------------------------- */ \ - /* Set tail pointer if removing the last one */ \ - /* ------------------------------------------- */ \ - if (__req->mpid.next == NULL) \ - __Q ## _tail = __prev; \ - else \ - (__req->mpid.next)->mpid.prev = __prev; \ -}) -#endif - - -/** - * \brief A thread-safe version of MPIDI_Recvq_FU - * \param[in] source Find by Sender - * \param[in] tag Find by Tag - * \param[in] context_id Find by Context ID (communicator) - * \return 1/0 if the matching UE request was found or not - */ -static inline int -MPIDI_Recvq_FU_r(int source, int tag, int context, MPI_Status * status) -{ - int rc = FALSE; -#ifdef QUEUE_BINARY_SEARCH_SUPPORT - if(MPIDI_Process.queue_binary_search_support_on) - { - if (likely(!MPIDI_Recvq_empty_uexp())) - { - MPIU_THREAD_CS_ENTER(MSGQUEUE,0); - rc = MPIDI_Recvq_FU(source, tag, context, status); - MPIU_THREAD_CS_EXIT(MSGQUEUE, 0); - } - } - else - { -#endif - if (likely(MPIDI_Recvq.unexpected_head != NULL)) - { - MPIU_THREAD_CS_ENTER(MSGQUEUE,0); - rc = MPIDI_Recvq_FU(source, tag, context, status); - MPIU_THREAD_CS_EXIT(MSGQUEUE, 0); - } -#ifdef QUEUE_BINARY_SEARCH_SUPPORT - } -#endif - return rc; -} - - -/** - * \brief Find a request in the unexpected queue and dequeue it, or allocate a new request and enqueue it in the posted queue - * \param[in] source Find by Sender - * \param[in] tag Find by Tag - * \param[in] context_id Find by Context ID (communicator) - * \param[out] foundp TRUE iff the request was found - * \return The matching UE request or the new posted request - */ -#ifndef OUT_OF_ORDER_HANDLING -static inline MPIR_Request * -MPIDI_Recvq_FDU_or_AEP(MPIR_Request *newreq, int source, int tag, int context_id, int * foundp) -#else -static inline MPIR_Request * -MPIDI_Recvq_FDU_or_AEP(MPIR_Request *newreq, int source, pami_task_t pami_source, int tag, int context_id, int * foundp) -#endif -{ - MPIR_Request * rreq = NULL; - /* We have unexpected messages, so search unexpected queue */ -#ifdef QUEUE_BINARY_SEARCH_SUPPORT - if(MPIDI_Process.queue_binary_search_support_on) - { - if (unlikely(!MPIDI_Recvq_empty_uexp())) - { -#ifndef OUT_OF_ORDER_HANDLING - rreq = MPIDI_Recvq_FDU(source, tag, context_id, foundp); -#else - rreq = MPIDI_Recvq_FDU(source, pami_source, tag, context_id, foundp); -#endif - if (*foundp == TRUE) - return rreq; -#if (MPIDI_STATISTICS) - else { - MPID_NSTAT(mpid_statp->lateArrivals); - } -#endif - } - } - else - { -#endif - if (unlikely(MPIDI_Recvq.unexpected_head != NULL)) { -#ifndef OUT_OF_ORDER_HANDLING - rreq = MPIDI_Recvq_FDU(source, tag, context_id, foundp); -#else - rreq = MPIDI_Recvq_FDU(source, pami_source, tag, context_id, foundp); -#endif - if (*foundp == TRUE) - return rreq; -#if (MPIDI_STATISTICS) - else { - MPID_NSTAT(mpid_statp->lateArrivals); - } -#endif - } -#ifdef QUEUE_BINARY_SEARCH_SUPPORT - } -#endif - /* A matching request was not found in the unexpected queue, - so we need to allocate a new request and add it to the - posted queue */ - rreq = newreq; - TRACE_SET_REQ_VAL(rreq->mpid.envelope.msginfo.MPIseqno,-1); - TRACE_SET_REQ_VAL(rreq->mpid.envelope.length,-1); - TRACE_SET_REQ_VAL(rreq->mpid.envelope.data,(void *) 0); - rreq->kind = MPIR_REQUEST_KIND__RECV; - MPIDI_Request_setMatch(rreq, tag, source, context_id); -#ifdef QUEUE_BINARY_SEARCH_SUPPORT - if(MPIDI_Process.queue_binary_search_support_on) - MPIDI_Recvq_insert_post(rreq, source, tag, context_id); - else -#endif - MPIDI_Recvq_append(MPIDI_Recvq.posted, rreq); - *foundp = FALSE; - - return rreq; -} - -#if TOKEN_FLOW_CONTROL -typedef struct MPIDI_Token_cntr { - uint16_t unmatched; /* no. of unmatched EA messages */ - uint16_t rettoks; /* no. of tokens to be returned */ - int tokens; /* no. of tokens available-pairwise */ - int n_tokenStarved; /* no. of times token starvation occured */ -} MPIDI_Token_cntr_t; - -MPIDI_Token_cntr_t *MPIDI_Token_cntr; -#endif - -#ifdef OUT_OF_ORDER_HANDLING - -/** - * data structures that tracks pair-wise in-coming communication. - */ -typedef struct MPIDI_In_cntr { - uint n_OutOfOrderMsgs:16; /* the number of out-of-order messages received */ - uint nMsgs; /* the number of received messages */ - MPIR_Request *OutOfOrderList; /* link list of out-of-order messages */ -} MPIDI_In_cntr_t; - -/** - * data structures that tracks pair-wise Out-going communication. - */ -typedef struct MPIDI_Out_cntr { - uint unmatched:16; /* the number of un-matched messages */ - uint nMsgs; /* the number of out-going messages */ -} MPIDI_Out_cntr_t; - -/* global data to keep track of pair-wise communication, storage malloced -during initialization time */ -MPIDI_In_cntr_t *MPIDI_In_cntr; -MPIDI_Out_cntr_t *MPIDI_Out_cntr; - -#endif - - -/** - * \brief Find a request in the posted queue and dequeue it - * \param[in] source Find by Sender - * \param[in] tag Find by Tag - * \param[in] context_id Find by Context ID (communicator) - * \return The matching posted request or the new UE request - */ -#ifndef OUT_OF_ORDER_HANDLING -static inline MPIR_Request * -MPIDI_Recvq_FDP(size_t source, size_t tag, size_t context_id) -#else -static inline MPIR_Request * -MPIDI_Recvq_FDP(size_t source, pami_task_t pami_source, int tag, int context_id, int msg_seqno) -#endif -{ - MPIR_Request * rreq; - MPIR_Request * prev_rreq = NULL; -#ifdef USE_STATISTICS - unsigned search_length = 0; -#endif - TRACE_MEMSET_R(pami_source,msg_seqno,recv_status); - -#ifdef QUEUE_BINARY_SEARCH_SUPPORT - void * it; - if(MPIDI_Process.queue_binary_search_support_on) - { - MPIDI_Recvq_find_in_post(source, tag, context_id, &rreq, &it); - if(rreq == NULL) - { - MPIDI_Recvq_find_in_post(source, MPI_ANY_TAG, context_id, &rreq, &it); - if(rreq == NULL) - { - MPIDI_Recvq_find_in_post(MPI_ANY_SOURCE, tag, context_id, &rreq, &it); - if(rreq == NULL) - MPIDI_Recvq_find_in_post(MPI_ANY_SOURCE, MPI_ANY_TAG, context_id, &rreq, &it); - } - } - } - else -#endif - rreq = MPIDI_Recvq.posted_head; - -#ifdef OUT_OF_ORDER_HANDLING - MPIDI_In_cntr_t *in_cntr = &MPIDI_In_cntr[pami_source]; - int nMsgs=(in_cntr->nMsgs+1); - - if(msg_seqno == nMsgs) { - in_cntr->nMsgs = msg_seqno; - } - - if( ((int)(in_cntr->nMsgs - msg_seqno)) >= 0) { -#endif - while (rreq != NULL) { -#ifdef USE_STATISTICS - ++search_length; -#endif - - int match_src = MPIDI_Request_getMatchRank(rreq); - int match_tag = MPIDI_Request_getMatchTag(rreq); - int match_ctx = MPIDI_Request_getMatchCtxt(rreq); - - int flag0 = (source == match_src); - flag0 |= (match_src == MPI_ANY_SOURCE); - int flag1 = (context_id == match_ctx); - int flag2 = (tag == match_tag); - flag2 |= (match_tag == MPI_ANY_TAG); - int flag = flag0 & flag1 & flag2; - -#if 0 - if ((MPIDI_Request_getMatchRank(rreq)==source || MPIDI_Request_getMatchRank(rreq)==MPI_ANY_SOURCE) && - (MPIDI_Request_getMatchCtxt(rreq)==context_id) && - (MPIDI_Request_getMatchTag(rreq) == tag || MPIDI_Request_getMatchTag(rreq) == MPI_ANY_TAG) - ) -#else - if (flag) -#endif - { - TRACE_SET_R_VAL(pami_source,(msg_seqno & SEQMASK),req,rreq); - TRACE_SET_R_VAL(pami_source,(msg_seqno & SEQMASK),msgid,msg_seqno); - TRACE_SET_R_BIT(pami_source,(msg_seqno & SEQMASK),fl.f.posted); - TRACE_SET_R_VAL(pami_source,(msg_seqno & SEQMASK),bufadd,rreq->mpid.userbuf); - TRACE_SET_R_VAL(pami_source,(msg_seqno & SEQMASK),len,rreq->mpid.envelope.length); - TRACE_SET_R_VAL(pami_source,(msg_seqno & SEQMASK),rtag,tag); - TRACE_SET_R_VAL(pami_source,(msg_seqno & SEQMASK),rctx,context_id); - TRACE_SET_R_VAL(pami_source,(msg_seqno & SEQMASK),rsource,pami_source); - TRACE_SET_R_BIT(pami_source,(msg_seqno & SEQMASK),fl.f.matchedInHH); - TRACE_SET_REQ_VAL(rreq->mpid.idx,(msg_seqno & SEQMASK)); - TRACE_SET_REQ_VAL(rreq->mpid.partner_id,pami_source); -#ifdef OUT_OF_ORDER_HANDLING - MPIDI_Request_setPeerRank_pami(rreq, pami_source); -#endif -#ifdef QUEUE_BINARY_SEARCH_SUPPORT - if(MPIDI_Process.queue_binary_search_support_on) - MPIDI_Recvq_remove_post(match_src, match_tag, match_ctx, it); - else -#endif - MPIDI_Recvq_remove(MPIDI_Recvq.posted, rreq, prev_rreq); -#ifdef USE_STATISTICS - MPIDI_Statistics_time(MPIDI_Statistics.recvq.unexpected_search, search_length); -#endif - return rreq; - } -#ifdef QUEUE_BINARY_SEARCH_SUPPORT - if(MPIDI_Process.queue_binary_search_support_on) - break; - else - { -#endif - prev_rreq = rreq; - rreq = rreq->mpid.next; -#ifdef QUEUE_BINARY_SEARCH_SUPPORT - } -#endif - } -#ifdef OUT_OF_ORDER_HANDLING - } -#endif - -#ifdef USE_STATISTICS - MPIDI_Statistics_time(MPIDI_Statistics.recvq.unexpected_search, search_length); -#endif - return NULL; -} - - -#ifdef OUT_OF_ORDER_HANDLING -/** - * insert a request _req2 before _req1 - */ -#define MPIDI_Recvq_insert(__Q, __req1, __req2) \ -({ \ - (__req2)->mpid.next = __req1; \ - if((__req1)->mpid.prev != NULL) \ - ((__req1)->mpid.prev)->mpid.next = (__req2); \ - else \ - __Q ## _head = __req2; \ - (__req2)->mpid.prev = (__req1)->mpid.prev; \ - (__req1)->mpid.prev = (__req2); \ -}) - -/** - * remove a request from out of order list - */ -#define MPIDI_Recvq_remove_req_from_ool(req,in_cntr) \ -({ \ - in_cntr->n_OutOfOrderMsgs--; \ - if (in_cntr->n_OutOfOrderMsgs == 0) { \ - in_cntr->OutOfOrderList=NULL; \ - req->mpid.nextR=NULL; \ - req->mpid.prevR=NULL; \ - } else if (in_cntr->n_OutOfOrderMsgs > 0) { \ - in_cntr->OutOfOrderList=req->mpid.nextR; \ - /* remove req from out of order list */ \ - ((MPIR_Request *)(req)->mpid.prevR)->mpid.nextR = (req)->mpid.nextR; \ - ((MPIR_Request *)(req)->mpid.nextR)->mpid.prevR = (req)->mpid.prevR; \ - (req)->mpid.nextR=NULL; \ - (req)->mpid.prevR=NULL; \ - } \ -}) -#endif - - -#endif diff --git a/src/mpid/pamid/src/mpid_recvq_mmap.cpp b/src/mpid/pamid/src/mpid_recvq_mmap.cpp deleted file mode 100644 index c565fed..0000000 --- a/src/mpid/pamid/src/mpid_recvq_mmap.cpp +++ /dev/null @@ -1,419 +0,0 @@ -#include <map> -#include <stdio.h> -#include <cstdlib> -#include "mpidi_platform.h" - - -extern "C" { - -struct MPID_Queue_map_key_t -{ - int context_id; - int source; - int tag; - inline bool operator< (const MPID_Queue_map_key_t& qmk) const - { - if(context_id < qmk.context_id) - return true; - else if(context_id == qmk.context_id) - { - if(source < qmk.source) - return true; - else if(source == qmk.source) - { - if(tag < qmk.tag) - return true; - else - return false; - } - else - return false; - } - else - return false; - } -}; - -struct MPID_Queue_map_value_t -{ - void* rreq; -#ifdef OUT_OF_ORDER_HANDLING - int seqno; -#endif -}; - -typedef std::multimap<MPID_Queue_map_key_t,MPID_Queue_map_value_t> MPID_Req_queue_map_t; -typedef std::multimap<MPID_Queue_map_key_t,MPID_Queue_map_value_t>::iterator MPID_Req_queue_map_iterator_t; - -MPID_Req_queue_map_t MPID_Unexp_queue; -MPID_Req_queue_map_t MPID_Posted_queue; - -MPID_Req_queue_map_iterator_t itp; -MPID_Req_queue_map_iterator_t itu; - -void MPIDI_Recvq_init_queues(); -int MPIDI_Recvq_empty_uexp(); -int MPIDI_Recvq_empty_post(); - -#ifndef OUT_OF_ORDER_HANDLING -void MPIDI_Recvq_insert_uexp(void * rreq, int source, int tag, int context_id); -void MPIDI_Recvq_insert_post(void * rreq, int source, int tag, int context_id); -void MPIDI_Recvq_insrt(MPID_Req_queue_map_t* queue, void * rreq, int source, int tag, int context_id); - -void MPIDI_Recvq_remove_uexp(int source, int tag, int context_id, void* it_req); -void MPIDI_Recvq_remove_post(int source, int tag, int context_id, void* it_req); -void MPIDI_Recvq_rm(MPID_Req_queue_map_t* queue, int source, int tag, int context_id, void* it_req); - -void MPIDI_Recvq_find_in_uexp(int source, int tag, int context_id, void** req, void** it_req); -void MPIDI_Recvq_find_in_post(int source, int tag, int context_id, void** req, void** it_req); -void MPIDI_Recvq_find(MPID_Req_queue_map_t* queue, int source, int tag, int context_id, void** req, void** it_req); - -#else -void MPIDI_Recvq_insert_uexp(void * rreq, int source, int tag, int context_id, int seqno); -void MPIDI_Recvq_insert_post(void * rreq, int source, int tag, int context_id); -void MPIDI_Recvq_insrt(MPID_Req_queue_map_t* queue, void * rreq, int source, int tag, int context_id, int seqno); - -void MPIDI_Recvq_remove_uexp(int source, int tag, int context_id, int seqno, void* it_req); -void MPIDI_Recvq_remove_uexp_noit(int source, int tag, int context_id, int seqno); -void MPIDI_Recvq_remove_post(int source, int tag, int context_id, void* it_req); -void MPIDI_Recvq_rm(MPID_Req_queue_map_t* queue, int source, int tag, int context_id, int seqno, void* it_req); - -void MPIDI_Recvq_find_in_uexp(int source, int tag, int context_id, int seqno, void** req, void** it_req); -void MPIDI_Recvq_find_in_post(int source, int tag, int context_id, void** req, void** it_req); -void MPIDI_Recvq_find(MPID_Req_queue_map_t* queue, int source, int tag, int context_id, int seqno, void** req, void** it_req); - -#endif - - - - -void MPIDI_Recvq_init_queues() -{ - MPID_Unexp_queue.clear(); - MPID_Posted_queue.clear(); -} - -int MPIDI_Recvq_empty_uexp() -{ - return MPID_Unexp_queue.empty(); -} - -int MPIDI_Recvq_empty_post() -{ - return MPID_Posted_queue.empty(); -} - -#ifndef OUT_OF_ORDER_HANDLING -void MPIDI_Recvq_insert_uexp(void * rreq, int source, int tag, int context_id) -{ - MPIDI_Recvq_insrt(&MPID_Unexp_queue, rreq, source, tag, context_id); -} - - -void MPIDI_Recvq_insert_post(void * rreq, int source, int tag, int context_id) -{ - MPIDI_Recvq_insrt(&MPID_Posted_queue, rreq, source, tag, context_id); -} - - -void MPIDI_Recvq_insrt(MPID_Req_queue_map_t* queue, void * rreq, int source, int tag, int context_id) -{ - MPID_Queue_map_key_t key; - MPID_Queue_map_value_t value; - key.context_id = context_id; - key.source = source; - key.tag = tag; - value.rreq = rreq; - queue->insert(std::make_pair(key,value)); -} - - -void MPIDI_Recvq_remove_uexp(int source, int tag, int context_id, void* it_req) -{ - MPIDI_Recvq_rm(&MPID_Unexp_queue, source, tag, context_id, it_req); -} - - -void MPIDI_Recvq_remove_post(int source, int tag, int context_id, void* it_req) -{ - MPIDI_Recvq_rm(&MPID_Posted_queue, source, tag, context_id, it_req); -} - - -void MPIDI_Recvq_rm(MPID_Req_queue_map_t* queue, int source, int tag, int context_id, void* it_req) -{ - MPID_Req_queue_map_iterator_t it = *((MPID_Req_queue_map_iterator_t*)it_req); - queue->erase(it); -} - - -void MPIDI_Recvq_find_in_uexp(int source, int tag, int context_id, void** req, void** it_req) -{ - return MPIDI_Recvq_find(&MPID_Unexp_queue, source, tag, context_id, req, it_req); -} - -void MPIDI_Recvq_find_in_post(int source, int tag, int context_id, void** req, void** it_req) -{ - MPID_Queue_map_key_t key; - key.context_id = context_id; - key.source = source; - key.tag = tag; - *it_req = NULL; - *req = NULL; - itp = MPID_Posted_queue.find(key); - if(itp != MPID_Posted_queue.end()) - { - *it_req = &itp; - *req = ((MPID_Queue_map_value_t)(itp->second)).rreq; - } -} - -void MPIDI_Recvq_find(MPID_Req_queue_map_t* queue, int source, int tag, int context_id, void** req, void** it_req) -{ - MPID_Queue_map_key_t key; - key.context_id = context_id; - key.source = source; - key.tag = tag; - *req = NULL; - *it_req = NULL; - itu = queue->find(key); - if(itu != queue->end()) - { - *req = ((MPID_Queue_map_value_t)(itu->second)).rreq; - *it_req = &itu; - return; - } - - if(source < 0 && tag >= 0) - { - for(itu = queue->begin(); itu != queue->end(); itu++) - { - if(((MPID_Queue_map_key_t)itu->first).tag == tag && ((MPID_Queue_map_key_t)itu->first).context_id == context_id) - { - *it_req = &itu; - *req = ((MPID_Queue_map_value_t)itu->second).rreq; - return; - } - } - } - else if(source >= 0 && tag < 0) - { - for(itu = queue->begin(); itu != queue->end(); itu++) - { - if(((MPID_Queue_map_key_t)itu->first).source == source && ((MPID_Queue_map_key_t)itu->first).context_id == context_id) - { - *it_req = &itu; - *req = ((MPID_Queue_map_value_t)itu->second).rreq; - return; - } - } - } - else if(source < 0 && tag < 0) - { - for(itu = queue->begin(); itu != queue->end(); itu++) - { - if(((MPID_Queue_map_key_t)itu->first).context_id == context_id) - { - *it_req = &itu; - *req = ((MPID_Queue_map_value_t)itu->second).rreq; - return; - } - } - } - -} - -#else -void MPIDI_Recvq_insert_uexp(void * rreq, int source, int tag, int context_id, int seqno) -{ - MPIDI_Recvq_insrt(&MPID_Unexp_queue, rreq, source, tag, context_id, seqno); - -} - - -void MPIDI_Recvq_insert_post(void * rreq, int source, int tag, int context_id) -{ - MPIDI_Recvq_insrt(&MPID_Posted_queue, rreq, source, tag, context_id, -1); -} - - -void MPIDI_Recvq_insrt(MPID_Req_queue_map_t* queue, void * rreq, int source, int tag, int context_id, int seqno) -{ - MPID_Queue_map_key_t key; - MPID_Queue_map_value_t value; - key.context_id = context_id; - key.source = source; - key.tag = tag; - value.seqno = seqno; - value.rreq = rreq; - queue->insert(std::make_pair(key,value)); -} - - -void MPIDI_Recvq_remove_uexp(int source, int tag, int context_id, int seqno, void* it_req) -{ - MPIDI_Recvq_rm(&MPID_Unexp_queue, source, tag, context_id, seqno, it_req); -} - -void MPIDI_Recvq_remove_uexp_noit(int source, int tag, int context_id, int seqno) -{ - MPID_Queue_map_key_t key; - key.context_id = context_id; - key.source = source; - key.tag = tag; - MPID_Req_queue_map_iterator_t it; - std::pair <MPID_Req_queue_map_iterator_t, MPID_Req_queue_map_iterator_t > itpair; - itpair = MPID_Unexp_queue.equal_range(key); - for(it = itpair.first; itu != itpair.second; ++it) - if(((MPID_Queue_map_value_t)it->second).seqno == seqno) - { - MPID_Unexp_queue.erase(it); - break; - } -} - -void MPIDI_Recvq_remove_post(int source, int tag, int context_id, void* it_req) -{ - MPIDI_Recvq_rm(&MPID_Posted_queue, source, tag, context_id, -1, it_req); -} - - -void MPIDI_Recvq_rm(MPID_Req_queue_map_t* queue, int source, int tag, int context_id, int seqno, void* it_req) -{ - MPID_Req_queue_map_iterator_t it = *((MPID_Req_queue_map_iterator_t*)it_req); - queue->erase(it); -} - - -void MPIDI_Recvq_find_in_uexp(int source, int tag, int context_id, int seqno, void** req, void** it_req) -{ - return MPIDI_Recvq_find(&MPID_Unexp_queue, source, tag, context_id, seqno, req, it_req); -} - -void MPIDI_Recvq_find_in_post(int source, int tag, int context_id, void** req, void** it_req) -{ - MPID_Queue_map_key_t key; - key.context_id = context_id; - key.source = source; - key.tag = tag; - *req = NULL; - *it_req = NULL; - itp = MPID_Posted_queue.find(key); - if(itp!=MPID_Posted_queue.end()) - { - *it_req = (void*)&itp; - *req = ((MPID_Queue_map_value_t)(itp->second)).rreq; - } -} - - -void MPIDI_Recvq_find(MPID_Req_queue_map_t* queue, int source, int tag, int context_id, int seqno, void** req, void** it_req) -{ - MPID_Queue_map_key_t key; - key.context_id = context_id; - key.source = source; - key.tag = tag; - - *req = NULL; - *it_req = NULL; - if(seqno == -1) - { - itu = queue->find(key); - if(itu != queue->end()) - { - *it_req = (void*)&itu; - *req = ((MPID_Queue_map_value_t)itu->second).rreq; - } - } - else - { - std::pair <MPID_Req_queue_map_iterator_t, MPID_Req_queue_map_iterator_t > itpair; - itpair = queue->equal_range(key); - for(itu = itpair.first; itu != itpair.second; ++itu) - if(((MPID_Queue_map_value_t)itu->second).seqno <= seqno) - { - *it_req = (void*)&itu; - *req = ((MPID_Queue_map_value_t)itu->second).rreq; - break; - } - } - if(*req != NULL) - return; - - if(source < 0 && tag >= 0) - { - for(itu = queue->begin(); itu != queue->end(); itu++) - { - if(((MPID_Queue_map_key_t)itu->first).tag == tag && ((MPID_Queue_map_key_t)itu->first).context_id == context_id) - { - if(seqno == -1) - { - *it_req = (void*)&itu; - *req = ((MPID_Queue_map_value_t)itu->second).rreq; - return; - } - else - { - if(((MPID_Queue_map_value_t)itu->second).seqno <= seqno) - { - *it_req = (void*)&itu; - *req = ((MPID_Queue_map_value_t)itu->second).rreq; - return; - } - } - } - } - } - else if(source >= 0 && tag < 0) - { - for(itu = queue->begin(); itu != queue->end(); itu++) - { - if(((MPID_Queue_map_key_t)itu->first).source == source && ((MPID_Queue_map_key_t)itu->first).context_id == context_id) - { - if(seqno == -1) - { - *it_req = (void*)&itu; - *req = ((MPID_Queue_map_value_t)itu->second).rreq; - return; - } - else - { - if(((MPID_Queue_map_value_t)itu->second).seqno <= seqno) - { - *it_req = (void*)&itu; - *req = ((MPID_Queue_map_value_t)itu->second).rreq; - return; - } - } - } - } - } - else if(source < 0 && tag < 0) - { - for(itu = queue->begin(); itu != queue->end(); itu++) - { - if(((MPID_Queue_map_key_t)itu->first).context_id == context_id) - { - if(seqno == -1) - { - *it_req = (void*)&itu; - *req = ((MPID_Queue_map_value_t)itu->second).rreq; - return; - } - else - { - if(((MPID_Queue_map_value_t)itu->second).seqno <= seqno) - { - *it_req = (void*)&itu; - *req = ((MPID_Queue_map_value_t)itu->second).rreq; - return; - } - } - } - } - } -} - -#endif - -} diff --git a/src/mpid/pamid/src/mpid_request.c b/src/mpid/pamid/src/mpid_request.c deleted file mode 100644 index 0a1607d..0000000 --- a/src/mpid/pamid/src/mpid_request.c +++ /dev/null @@ -1,88 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/mpid_request.c - * \brief Accessors and actors for MPID Requests - */ -#include <mpidimpl.h> - -#ifndef MPIR_REQUEST_PREALLOC -#if (MPICH_THREAD_GRANULARITY == MPICH_THREAD_GRANULARITY__GLOBAL) -#define MPIR_REQUEST_PREALLOC 16 -#elif (MPIU_HANDLE_ALLOCATION_METHOD == MPIU_HANDLE_ALLOCATION_THREAD_LOCAL) -#define MPIR_REQUEST_PREALLOC 512 //Have direct more reqyests for all threads -#else -#define MPIR_REQUEST_PREALLOC 8 -#endif -#endif - -/** - * \defgroup MPIR_REQUEST MPID Request object management - * - * Accessors and actors for MPID Requests - */ - - -/* these are referenced by src/mpi/pt2pt/wait.c in PMPI_Wait! */ -MPIR_Request MPIR_Request_direct[MPIR_REQUEST_PREALLOC] __attribute__((__aligned__(64))); -MPIR_Object_alloc_t MPIR_Request_mem = - { - 0, 0, 0, 0, MPIR_REQUEST, sizeof(MPIR_Request), - MPIR_Request_direct, - MPIR_REQUEST_PREALLOC - }; - - -#if (MPIU_HANDLE_ALLOCATION_METHOD == MPIU_HANDLE_ALLOCATION_THREAD_LOCAL) && defined(__BGQ__) -void MPIDI_Request_allocate_pool() -{ - int i; - MPIR_Request *prev, *cur; - /* batch allocate a linked list of requests */ - MPIU_THREAD_CS_ENTER(HANDLEALLOC,); - prev = MPIR_Handle_obj_alloc_unsafe(&MPIR_Request_mem); - MPID_assert(prev != NULL); - prev->mpid.next = NULL; - for (i = 1; i < MPIR_REQUEST_TLS_MAX; ++i) { - cur = MPIR_Handle_obj_alloc_unsafe(&MPIR_Request_mem); - MPID_assert(cur != NULL); - cur->mpid.next = prev; - prev = cur; - } - MPIU_THREAD_CS_EXIT(HANDLEALLOC,); - MPIDI_Process.request_handles[MPIDI_THREAD_ID()].head = cur; - MPIDI_Process.request_handles[MPIDI_THREAD_ID()].count += MPIR_REQUEST_TLS_MAX; -} -#endif - - -void -MPIDI_Request_uncomplete(MPIR_Request *req) -{ - int count; - MPIR_Object_add_ref(req); - MPIR_cc_incr(req->cc_ptr, &count); -} - - -void -MPID_Request_set_completed(MPIR_Request *req) -{ - MPIR_cc_set(&req->cc, 0); - MPIDI_Progress_signal(); -} diff --git a/src/mpid/pamid/src/mpid_request.h b/src/mpid/pamid/src/mpid_request.h deleted file mode 100644 index ad283cf..0000000 --- a/src/mpid/pamid/src/mpid_request.h +++ /dev/null @@ -1,354 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/mpid_request.h - * \brief ??? - */ - -#ifndef __src_mpid_request_h__ -#define __src_mpid_request_h__ - -#include "mpidu_datatype.h" - -/** - * \addtogroup MPIR_REQUEST - * \{ - */ - -#define MPIR_Request_create MPID_Request_create_inline -#define MPIR_Request_free MPID_Request_free_inline -#define MPIDI_Request_complete MPIDI_Request_complete_inline -#define MPIDI_Request_complete_norelease MPIDI_Request_complete_norelease_inline -#define MPID_Request_discard MPID_Request_discard_inline - - -extern MPIR_Object_alloc_t MPIR_Request_mem; -#if TOKEN_FLOW_CONTROL -extern void MPIDI_mm_free(void *,size_t); -#endif -typedef enum {mpiuMalloc=1,mpidiBufMM} MPIDI_mallocType; - -void MPIDI_Request_uncomplete(MPIR_Request *req); -#if (MPIU_HANDLE_ALLOCATION_METHOD == MPIU_HANDLE_ALLOCATION_THREAD_LOCAL) && defined(__BGQ__) -void MPIDI_Request_allocate_pool(); -#endif - -#define MPIDI_Request_getCA(_req) ({ (_req)->mpid.ca; }) -#define MPIDI_Request_getPeerRank_pami(_req) ({ (_req)->mpid.peer_pami; }) -#define MPIDI_Request_getPeerRank_comm(_req) ({ (_req)->mpid.peer_comm; }) -#define MPIDI_Request_getPType(_req) ({ (_req)->mpid.ptype; }) -#define MPIDI_Request_getControl(_req) ({ (_req)->mpid.envelope.msginfo.control; }) -#define MPIDI_Request_isSync(_req) ({ (_req)->mpid.envelope.msginfo.isSync; }) -#define MPIDI_Request_isRzv(_req) ({ (_req)->mpid.envelope.msginfo.isRzv; }) -#define MPIDI_Request_getMatchTag(_req) ({ (_req)->mpid.envelope.msginfo.MPItag; }) -#define MPIDI_Request_getMatchRank(_req) ({ (_req)->mpid.envelope.msginfo.MPIrank; }) -#define MPIDI_Request_getMatchCtxt(_req) ({ (_req)->mpid.envelope.msginfo.MPIctxt; }) - -#define MPIDI_Request_setCA(_req, _ca) ({ (_req)->mpid.ca = (_ca); }) -#define MPIDI_Request_setPeerRank_pami(_req,_r) ({ (_req)->mpid.peer_pami = (_r); }) -#define MPIDI_Request_setPeerRank_comm(_req,_r) ({ (_req)->mpid.peer_comm = (_r); }) -#define MPIDI_Request_setPType(_req,_t) ({ (_req)->mpid.ptype = (_t); }) -#define MPIDI_Request_setControl(_req,_t) ({ (_req)->mpid.envelope.msginfo.control = (_t); }) -#define MPIDI_Request_setSync(_req,_t) ({ (_req)->mpid.envelope.msginfo.isSync = (_t); }) -#define MPIDI_Request_setRzv(_req,_t) ({ (_req)->mpid.envelope.msginfo.isRzv = (_t); }) -#ifdef OUT_OF_ORDER_HANDLING -#define MPIDI_Request_getMatchSeq(_req) ({ (_req)->mpid.envelope.msginfo.MPIseqno; }) -#define MPIDI_Request_setMatchSeq(_req,_sq) ({ (_req)->mpid.envelope.msginfo.MPIseqno = (_sq); }) -#endif -#define MPIDI_Request_setMatch(_req,_tag,_rank,_ctxtid) \ -({ \ - (_req)->mpid.envelope.msginfo.MPItag=(_tag); \ - (_req)->mpid.envelope.msginfo.MPIrank=(_rank); \ - (_req)->mpid.envelope.msginfo.MPIctxt=(_ctxtid); \ -}) - -#define MPIDI_Msginfo_getPeerRequest(_msg) ({ MPIR_Request *req=NULL; MPIR_Request_get_ptr((_msg)->req, req); MPID_assert(req != NULL); req; }) -#define MPIDI_Msginfo_getPeerRequestH(_msg) ({ (_msg)->req; }) -#define MPIDI_Msginfo_cpyPeerRequestH(_dst,_src) ({ (_dst)->req = (_src)->req; MPI_SUCCESS; }) -#define MPIDI_Request_getPeerRequest(_req) MPIDI_Msginfo_getPeerRequest(&(_req)->mpid.envelope.msginfo) -#define MPIDI_Request_getPeerRequestH(_req) ({ (_req)->mpid.envelope.msginfo.req; }) -#define MPIDI_Request_setPeerRequestH(_req) ({ (_req)->mpid.envelope.msginfo.req = (_req)->handle; MPI_SUCCESS; }) -#define MPIDI_Request_cpyPeerRequestH(_dst,_src) MPIDI_Msginfo_cpyPeerRequestH(&(_dst)->mpid.envelope.msginfo,_src) - - -#define MPIU_HANDLE_ALLOCATION_MUTEX 0 -#define MPIU_HANDLE_ALLOCATION_THREAD_LOCAL 1 - -/* XXX DJG for TLS hack */ -#define MPIR_REQUEST_TLS_MAX 128 - -#if (MPIU_HANDLE_ALLOCATION_METHOD == MPIU_HANDLE_ALLOCATION_THREAD_LOCAL) && defined(__BGQ__) - -# define MPIDI_Request_tls_alloc(req) \ -({ \ - size_t tid = MPIDI_THREAD_ID(); \ - MPIDI_RequestHandle_t *rh = &MPIDI_Process.request_handles[tid]; \ - if (unlikely(rh->head == NULL)) \ - MPIDI_Request_allocate_pool(); \ - (req) = rh->head; \ - rh->head = req->mpid.next; \ - rh->count --; \ -}) - -# define MPIDI_Request_tls_free(req) \ -({ \ - size_t tid = MPIDI_THREAD_ID(); \ - MPIDI_RequestHandle_t *rh = &MPIDI_Process.request_handles[tid]; \ - if (likely(rh->count < MPIR_REQUEST_TLS_MAX)) \ - { \ - /* push request onto the top of the stack */ \ - req->mpid.next = rh->head; \ - rh->head = req; \ - rh->count ++; \ - } \ - else \ - { \ - MPIR_Handle_obj_free(&MPIR_Request_mem, req); \ - } \ -}) - -#else - -# define MPIDI_Request_tls_alloc(req) \ -({ \ - (req) = MPIR_Handle_obj_alloc(&MPIR_Request_mem); \ - if (req == NULL) \ - MPID_Abort(NULL, MPI_ERR_NO_SPACE, -1, "Cannot allocate Request"); \ -}) - -# define MPIDI_Request_tls_free(req) MPIR_Handle_obj_free(&MPIR_Request_mem, (req)) - -#endif - -#ifdef HAVE_DEBUGGER_SUPPORT -#define MPIDI_Request_clear_dbg(req_) ((req_)->u.send.dbg_next = NULL) -#else -#define MPIDI_Request_clear_dbg(req_) -#endif - -/** - * \brief Create a very generic request - * \note This should only ever be called by more specific allocators - */ -static inline MPIR_Request * -MPIDI_Request_create_basic() -{ - MPIR_Request * req = NULL; - - MPIDI_Request_tls_alloc(req); - MPID_assert(req != NULL); - MPID_assert(HANDLE_GET_MPI_KIND(req->handle) == MPIR_REQUEST); - MPIR_cc_set(&req->cc, 1); - req->cc_ptr = &req->cc; - -#if 0 - /* This will destroy the MPID part of the request. Use this to - check for fields that are not being correctly initialized. */ - memset(&req->mpid, 0xFFFFFFFF, sizeof(struct MPIDI_Request)); -#endif - req->mpid.next = NULL; - MPIDI_Request_clear_dbg(req); - - return req; -} - - -/** - * \brief Create new request without initalizing - */ -static inline MPIR_Request * -MPIDI_Request_create2_fast() -{ - MPIR_Request * req; - req = MPIDI_Request_create_basic(); - MPIR_Object_set_ref(req, 2); - - return req; -} - - -/** - * \brief Create and initialize a new request - */ -static inline void -MPIDI_Request_initialize(MPIR_Request * req) -{ - req->greq_fns = NULL; - - MPIR_STATUS_SET_COUNT(req->status, 0); - MPIR_STATUS_SET_CANCEL_BIT(req->status, FALSE); - req->status.MPI_SOURCE = MPI_UNDEFINED; - req->status.MPI_TAG = MPI_UNDEFINED; - req->status.MPI_ERROR = MPI_SUCCESS; - - struct MPIDI_Request* mpid = &req->mpid; - mpid->envelope.msginfo.flags = 0; - mpid->cancel_pending = FALSE; - mpid->datatype_ptr = NULL; - mpid->uebuf = NULL; - mpid->uebuflen = 0; - mpid->uebuf_malloc = 0; -#ifdef OUT_OF_ORDER_HANDLING - mpid->prev = NULL; - mpid->prevR = NULL; - mpid->nextR = NULL; - mpid->oo_peer = NULL; -#endif - mpid->win_req = NULL; - MPIDI_Request_setCA(req, MPIDI_CA_COMPLETE); -} - - -/** - * \brief Create and initialize a new request - */ -static inline MPIR_Request * -MPID_Request_create_inline() -{ - MPIR_Request * req; - req = MPIDI_Request_create_basic(); - MPIR_Object_set_ref(req, 1); - - MPIDI_Request_initialize(req); - req->comm=NULL; - - return req; -} - - -/** - * \brief Create and initialize a new request - */ -static inline MPIR_Request * -MPIDI_Request_create2() -{ - MPIR_Request * req; - req = MPIR_Request_create(); - MPIR_Object_set_ref(req, 2); - - return req; -} - -static inline MPIR_Request * -MPIDI_Request_create1() -{ - MPIR_Request * req; - req = MPIR_Request_create(); - MPIR_Object_set_ref(req, 1); - - return req; -} - -/** - * \brief Mark a request as cancel-pending - * \param[in] _req The request to cancel - * \return The previous state - */ -#define MPIDI_Request_cancel_pending(_req) \ -({ \ - int _flag = (_req)->mpid.cancel_pending; \ - (_req)->mpid.cancel_pending = TRUE; \ - _flag; \ -}) - - -static inline void -MPID_Request_free_inline(MPIR_Request *req) -{ - int count; - MPID_assert(HANDLE_GET_MPI_KIND(req->handle) == MPIR_REQUEST); - MPIR_Object_release_ref(req, &count); - MPID_assert(count >= 0); - - - if (count == 0) - { - MPID_assert(MPIR_cc_is_complete(&req->cc)); - - if (req->comm) MPIR_Comm_release(req->comm, 0); - if (req->greq_fns) MPL_free(req->greq_fns); - if (req->mpid.datatype_ptr) MPIDU_Datatype_release(req->mpid.datatype_ptr); - if (req->mpid.uebuf_malloc== mpiuMalloc) { - MPL_free(req->mpid.uebuf); - } - if(req->mpid.win_req) MPL_free(req->mpid.win_req); -#if TOKEN_FLOW_CONTROL - else if (req->mpid.uebuf_malloc == mpidiBufMM) { - MPIU_THREAD_CS_ENTER(MSGQUEUE,0); - MPIDI_mm_free(req->mpid.uebuf,req->mpid.uebuflen); - MPIU_THREAD_CS_EXIT(MSGQUEUE,0); - } -#endif - MPIDI_Request_tls_free(req); - } -} - - -/* This request was never used, at most had uebuf allocated. */ -static inline void -MPID_Request_discard_inline(MPIR_Request *req) -{ - if (req->mpid.uebuf_malloc == mpiuMalloc) { - MPL_free(req->mpid.uebuf); - } -#if TOKEN_FLOW_CONTROL - else if (req->mpid.uebuf_malloc == mpidiBufMM) { - MPIU_THREAD_CS_ENTER(MSGQUEUE,0); - MPIDI_mm_free(req->mpid.uebuf,req->mpid.uebuflen); - MPIU_THREAD_CS_EXIT(MSGQUEUE,0); - } -#endif - MPIDI_Request_tls_free(req); -} - -#define MPIR_REQUEST_SET_COMPLETED(req_) \ - MPIDI_Request_complete_norelease_inline(req_) - -static inline void -MPIDI_Request_complete_inline(MPIR_Request *req) -{ - int count; - MPIR_cc_decr(req->cc_ptr, &count); - MPID_assert(count >= 0); - - MPIR_Request_free(req); - if (count == 0) /* decrement completion count; if 0, signal progress engine */ - { - MPIDI_Progress_signal(); - } -} - - -static inline void -MPIDI_Request_complete_norelease_inline(MPIR_Request *req) -{ - int count; - MPIR_cc_decr(req->cc_ptr, &count); - MPID_assert(count >= 0); - - if (count == 0) /* decrement completion count; if 0, signal progress engine */ - { - MPIDI_Progress_signal(); - } -} - - -/** \} */ - - -#endif diff --git a/src/mpid/pamid/src/mpid_time.c b/src/mpid/pamid/src/mpid_time.c deleted file mode 100644 index 95f3856..0000000 --- a/src/mpid/pamid/src/mpid_time.c +++ /dev/null @@ -1,112 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/mpid_time.c - * \brief Devince interface between MPI_Wtime() and PAMI_Wtime() - */ -#include <mpidimpl.h> - -int MPIDI_PAMID_Timer_is_ready = 0; - -static int wtime(MPL_time_t *tval) -{ - if (MPIDI_PAMID_Timer_is_ready) { - *((MPID_Time_t *) tval) = PAMI_Wtime(MPIDI_Client); - return MPID_TIMER_SUCCESS; - } - else - return MPID_TIMER_ERR_NOT_INITIALIZED; -} - -int MPID_Wtime(MPID_Time_t *tval) -{ - return wtime((MPL_time_t *) tval); -} - -static int wtick(double *wtick) -{ - if (MPIDI_PAMID_Timer_is_ready) { - *((double *) wtick) = PAMIX_Client_query(MPIDI_Client, PAMI_CLIENT_WTICK).value.doubleval; - return MPID_TIMER_SUCCESS; - } - else - return MPID_TIMER_ERR_NOT_INITIALIZED; -} - -int MPID_Wtick(double *tick) -{ - return wtick((MPL_time_t *) tick); -} - -static int wtime_diff(MPL_time_t *t1, MPL_time_t *t2, double *diff) -{ - if (MPIDI_PAMID_Timer_is_ready) { - *diff = *((MPID_Time_t *) t2) - *((MPID_Time_t *) t1); - return MPID_TIMER_SUCCESS; - } - else - return MPID_TIMER_ERR_NOT_INITIALIZED; -} - -int MPID_Wtime_diff(MPID_Time_t *t1, MPID_Time_t *t2, double *diff) -{ - return wtime_diff((MPL_time_t *) t1, (MPL_time_t *) t2, diff); -} - -static int wtime_todouble(MPL_time_t *t, double *val) -{ - if (MPIDI_PAMID_Timer_is_ready) { - *val = *((MPID_Time_t *) t); - return MPID_TIMER_SUCCESS; - } - else - return MPID_TIMER_ERR_NOT_INITIALIZED; -} - -int MPID_Wtime_todouble(MPID_Time_t *t, double *val) -{ - return wtime_todouble((MPL_time_t *) t, val); -} - -static int wtime_acc(MPL_time_t *t1, MPL_time_t *t2, MPL_time_t *t3) -{ - if (MPIDI_PAMID_Timer_is_ready) { - *((MPID_Time_t *) t3) += *((MPID_Time_t *) t1) - *((MPID_Time_t *) t2); - return MPID_TIMER_SUCCESS; - } - else - return MPID_TIMER_ERR_NOT_INITIALIZED; -} - -int MPID_Wtime_acc(MPID_Time_t *t1, MPID_Time_t *t2, MPID_Time_t *t3) -{ - return wtime_acc((MPL_time_t *) t1, (MPL_time_t *) t2, (MPL_time_t *) t3); -} - -int MPID_Wtime_init( void ) -{ - MPL_wtime_fn = wtime; - MPL_wtick_fn = wtick; - MPL_wtime_diff_fn = wtime_diff; - MPL_wtime_todouble_fn = wtime_todouble; - MPL_wtime_acc_fn = wtime_acc; - - return MPID_TIMER_SUCCESS; -} - -#endif diff --git a/src/mpid/pamid/src/mpid_vc.c b/src/mpid/pamid/src/mpid_vc.c deleted file mode 100644 index fed17b7..0000000 --- a/src/mpid/pamid/src/mpid_vc.c +++ /dev/null @@ -1,360 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/mpid_vc.c - * \brief Maintain the virtual connection reference table - */ -/* - * (C) 2001 by Argonne National Laboratory. - * See COPYRIGHT in top-level directory. - */ - -#include <mpidimpl.h> - -extern int mpidi_dynamic_tasking; -/** - * \brief Virtual connection reference table - */ -struct MPIDI_VCRT -{ - MPIR_OBJECT_HEADER; - unsigned size; /**< Number of entries in the table */ - MPID_VCR *vcr_table; /**< Array of virtual connection references */ -}; - - -int MPID_VCR_Dup(MPID_VCR orig_vcr, MPID_VCR * new_vcr) -{ -#ifdef DYNAMIC_TASKING - if(mpidi_dynamic_tasking) { - if (orig_vcr->pg) { - MPIDI_PG_add_ref( orig_vcr->pg ); - } - } -#endif - - (*new_vcr)->taskid = orig_vcr->taskid; -#ifdef DYNAMIC_TASKING - (*new_vcr)->pg_rank = orig_vcr->pg_rank; - (*new_vcr)->pg = orig_vcr->pg; -#endif - return MPI_SUCCESS; -} - -int MPID_VCR_Get_lpid(MPID_VCR vcr, int * lpid_ptr) -{ - *lpid_ptr = (int)(vcr->taskid); - return MPI_SUCCESS; -} - -int MPID_VCRT_Create(int size, MPID_VCRT *vcrt_ptr) -{ - struct MPIDI_VCRT * vcrt; - int i,result; - - vcrt = MPL_malloc(sizeof(struct MPIDI_VCRT)); - vcrt->vcr_table = MPL_malloc(size*sizeof(MPID_VCR)); - - for(i = 0; i < size; i++) { - vcrt->vcr_table[i] = MPL_malloc(sizeof(struct MPID_VCR_t)); - } - if (vcrt != NULL) - { - MPIR_Object_set_ref(vcrt, 1); - vcrt->size = size; - *vcrt_ptr = vcrt; - result = MPI_SUCCESS; - } - else - { - result = MPIR_ERR_MEMALLOCFAILED; - } - return result; -} - -int MPID_VCRT_Add_ref(MPID_VCRT vcrt) -{ - MPIR_Object_add_ref(vcrt); - return MPI_SUCCESS; -} - -int MPID_VCRT_Release(MPID_VCRT vcrt, int isDisconnect) -{ - int count, i; - - MPIR_Object_release_ref(vcrt, &count); - - if (count == 0) { -#ifdef DYNAMIC_TASKING - int inuse; - if(mpidi_dynamic_tasking) { - for (i = 0; i < vcrt->size; i++) - { - MPID_VCR const vcr = vcrt->vcr_table[i]; - - if (vcr->pg == MPIDI_Process.my_pg && - vcr->pg_rank == MPIDI_Process.my_pg_rank) - { - TRACE_ERR("before MPIDI_PG_release_ref on vcr=%x pg=%x pg=%s inuse=%d\n", vcr, vcr->pg, (vcr->pg)->id, inuse); - inuse=MPIR_Object_get_ref(vcr->pg); - MPIDI_PG_release_ref(vcr->pg, &inuse); - if (inuse == 0) - { - MPIDI_PG_Destroy(vcr->pg); - MPL_free(vcr); - } - continue; - } - inuse=MPIR_Object_get_ref(vcr->pg); - - MPIDI_PG_release_ref(vcr->pg, &inuse); - if (inuse == 0) - MPIDI_PG_Destroy(vcr->pg); - if(vcr) MPL_free(vcr); - } - MPL_free(vcrt->vcr_table); - } /** CHECK */ - else { - for (i = 0; i < vcrt->size; i++) - MPL_free(vcrt->vcr_table[i]); - MPL_free(vcrt->vcr_table);vcrt->vcr_table=NULL; - } -#else - for (i = 0; i < vcrt->size; i++) - MPL_free(vcrt->vcr_table[i]); - MPL_free(vcrt->vcr_table);vcrt->vcr_table=NULL; -#endif - MPL_free(vcrt);vcrt=NULL; - } - return MPI_SUCCESS; -} - -int MPID_VCRT_Get_ptr(MPID_VCRT vcrt, MPID_VCR **vc_pptr) -{ - *vc_pptr = vcrt->vcr_table; - return MPI_SUCCESS; -} - -#ifdef DYNAMIC_TASKING -int MPID_VCR_CommFromLpids( MPIR_Comm *newcomm_ptr, - int size, const int lpids[] ) -{ - int mpi_errno = MPI_SUCCESS; - MPIR_Comm *commworld_ptr; - int i; - MPIDI_PG_iterator iter; - - commworld_ptr = MPIR_Process.comm_world; - /* Setup the communicator's vc table: remote group */ - MPID_VCRT_Create( size, &newcomm_ptr->vcrt ); - MPID_VCRT_Get_ptr( newcomm_ptr->vcrt, &newcomm_ptr->vcr ); - if(mpidi_dynamic_tasking) { - for (i=0; i<size; i++) { - MPID_VCR vc = 0; - - /* For rank i in the new communicator, find the corresponding - virtual connection. For lpids less than the size of comm_world, - we can just take the corresponding entry from comm_world. - Otherwise, we need to search through the process groups. - */ - /* printf( "[%d] Remote rank %d has lpid %d\n", - MPIR_Process.comm_world->rank, i, lpids[i] ); */ -#if 0 - if (lpids[i] < commworld_ptr->remote_size) { - vc = commworld_ptr->vcr[lpids[i]]; - } - else { -#endif - /* We must find the corresponding vcr for a given lpid */ - /* For now, this means iterating through the process groups */ - MPIDI_PG_t *pg = 0; - int j; - - MPIDI_PG_Get_iterator(&iter); - /* Skip comm_world */ - /*MPIDI_PG_Get_next( &iter, &pg ); */ - do { - MPIDI_PG_Get_next( &iter, &pg ); - /*MPIU_ERR_CHKINTERNAL(!pg, mpi_errno, "no pg"); */ - /* FIXME: a quick check on the min/max values of the lpid - for this process group could help speed this search */ - for (j=0; j<pg->size; j++) { - /*printf( "Checking lpid %d against %d in pg %s\n", - lpids[i], pg->vct[j].taskid, (char *)pg->id ); - fflush(stdout); */ - if (pg->vct[j].taskid == lpids[i]) { - vc = &pg->vct[j]; - /*printf( "found vc %x for lpid = %d in another pg\n", - (int)vc, lpids[i] );*/ - break; - } - } - } while (!vc); -#if 0 - } -#endif - - /* printf( "about to dup vc %x for lpid = %d in another pg\n", - (int)vc, lpids[i] ); */ - /* Note that his will increment the ref count for the associate - PG if necessary. */ - MPID_VCR_Dup( vc, &newcomm_ptr->vcr[i] ); - } - } else { - for (i=0; i<size; i++) { - /* For rank i in the new communicator, find the corresponding - rank in the comm world (FIXME FOR MPI2) */ - /* printf( "[%d] Remote rank %d has lpid %d\n", - MPIR_Process.comm_world->rank, i, lpids[i] ); */ - if (lpids[i] < commworld_ptr->remote_size) { - MPID_VCR_Dup( commworld_ptr->vcr[lpids[i]], - &newcomm_ptr->vcr[i] ); - } - else { - /* We must find the corresponding vcr for a given lpid */ - /* FIXME: Error */ - return 1; - /* MPID_VCR_Dup( ???, &newcomm_ptr->vcr[i] ); */ - } - } - - } -fn_exit: - return mpi_errno; -fn_fail: - goto fn_exit; -} - -/* - * The following is a very simple code for looping through - * the GPIDs. Note that this code requires that all processes - * have information on the process groups. - */ -int MPID_GPID_ToLpidArray( int size, int gpid[], int lpid[] ) -{ - int i, mpi_errno = MPI_SUCCESS; - int pgid; - MPIDI_PG_t *pg = 0; - MPIDI_PG_iterator iter; - - if(mpidi_dynamic_tasking) { - for (i=0; i<size; i++) { - MPIDI_PG_Get_iterator(&iter); - do { - MPIDI_PG_Get_next( &iter, &pg ); - if (!pg) { - /* Internal error. This gpid is unknown on this process */ - TRACE_ERR("No matching pg foung for id = %d\n", pgid ); - lpid[i] = -1; - /*MPIU_ERR_SET2(mpi_errno,MPI_ERR_INTERN, "**unknowngpid", - "**unknowngpid %d %d", gpid[0], gpid[1] ); */ - return mpi_errno; - } - MPIDI_PG_IdToNum( pg, &pgid ); - - if (pgid == gpid[0]) { - /* found the process group. gpid[1] is the rank in - this process group */ - /* Sanity check on size */ - TRACE_ERR("found the progress group for id = %d\n", pgid ); - TRACE_ERR("pg->size = %d gpid[1]=%d\n", pg->size, gpid[1] ); - if (pg->size > gpid[1]) { - TRACE_ERR("pg->vct[gpid[1]].taskid = %d\n", pg->vct[gpid[1]].taskid ); - lpid[i] = pg->vct[gpid[1]].taskid; - } - else { - lpid[i] = -1; - /*MPIU_ERR_SET2(mpi_errno,MPI_ERR_INTERN, "**unknowngpid", - "**unknowngpid %d %d", gpid[0], gpid[1] ); */ - return mpi_errno; - } - /* printf( "lpid[%d] = %d for gpid = (%d)%d\n", i, lpid[i], - gpid[0], gpid[1] ); */ - break; - } - } while (1); - gpid += 2; - } - } else { - for (i=0; i<size; i++) { - lpid[i] = *++gpid; gpid++; - } - return 0; - - } - - return mpi_errno; -} -/* - * The following routines convert to/from the global pids, which are - * represented as pairs of ints (process group id, rank in that process group) - */ - -/* FIXME: These routines belong in a different place */ -int MPID_GPID_GetAllInComm( MPIR_Comm *comm_ptr, int local_size, - int local_gpids[], int *singlePG ) -{ - int mpi_errno = MPI_SUCCESS; - int i; - int *gpid = local_gpids; - int lastPGID = -1, pgid; - MPID_VCR vc; - - MPIR_Assert(comm_ptr->local_size == local_size); - - if(mpidi_dynamic_tasking) { - *singlePG = 1; - for (i=0; i<comm_ptr->local_size; i++) { - vc = comm_ptr->vcr[i]; - - /* Get the process group id as an int */ - MPIDI_PG_IdToNum( vc->pg, &pgid ); - - *gpid++ = pgid; - if (lastPGID != pgid) { - if (lastPGID != -1) - *singlePG = 0; - lastPGID = pgid; - } - *gpid++ = vc->pg_rank; - - MPL_DBG_MSG_FMT(MPIR_DBG_COMM,VERBOSE, (MPL_DBG_FDEST, - "pgid=%d vc->pg_rank=%d", - pgid, vc->pg_rank)); - } - } else { - for (i=0; i<comm_ptr->local_size; i++) { - *gpid++ = 0; - (void)MPID_VCR_Get_lpid( comm_ptr->vcr[i], gpid ); - gpid++; - } - *singlePG = 1; - } - - return mpi_errno; -} - - -int MPIDI_VC_Init( MPID_VCR vcr, MPIDI_PG_t *pg, int rank ) -{ - vcr->pg = pg; - vcr->pg_rank = rank; -} - - -#endif diff --git a/src/mpid/pamid/src/mpidi_bufmm.c b/src/mpid/pamid/src/mpidi_bufmm.c deleted file mode 100644 index a466fe1..0000000 --- a/src/mpid/pamid/src/mpidi_bufmm.c +++ /dev/null @@ -1,716 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/mpidi_bufmm.c - * \brief Memory management for early arrivals - */ - - /*******************************************************************/ - /* DESCRIPTION: */ - /* Dynamic memory manager which allows allocation and */ - /* deallocation for earl arrivals sent via eager protocol. */ - /* */ - /* The basic method is for buffers of size between MIN_SIZE */ - /* and MAX_SIZE. The allocation scheme is a modified version */ - /* of Knuth's buddy algorithm. Regardless of what size buffer */ - /* is requested the size is rounded up to the nearest power of 2.*/ - /* Note that there is a buddy_header overhead per buffer (8 bytes*/ - /* in 32 bit and 16 bytes in 64 bit). So, for example, if a */ - /* 256 byte buffer is requested that would require 512 bytes */ - /* of memory. A 248-byte buffer needs 256 bytes of space. */ - /* Only for the maxsize buffers, it is guaranteed that the */ - /* allocation is a power of two, since typically applications */ - /* have such requirements. */ - /* */ - /* To speed up the buddy algorithm there are some preallocated */ - /* buffers. There are FLEX_NUM number of buffers from the */ - /* FLEX_COUNT number of smallest buffers. So, for example, if */ - /* MIN_SIZE is 16, FLEX_COUNT is 4, and FLEX_NUM is 256 then */ - /* there are buffers of size 16, 32, 64, and 128 preallocted */ - /* (256 buffers each). These buffers are arranged into stacks. */ - /* */ - /* If the system runs out of preallocated buffers or the size */ - /* is bigger than the biggest preallocated one then the buddy */ - /* algorithm is applied. Originally there is a list of MAX_SIZE */ - /* buffers. (The size is MAX_SIZE + the 8 or 16 byte overhead.) */ - /* These are not merged as the traditional buddy system would */ - /* require, since we never need bigger buffers than these. */ - /* Originally, the lists of smaller size buffers are empty. When */ - /* there is an allocation request the program searches for the */ - /* smallest free buffer available in the lists. If it is bigger */ - /* than the requested one then it is repeatedly split into half. */ - /* The other halves are inserted into the appropriate list of */ - /* free buffers. At deallocation the program attempts to merge */ - /* the buffer with it's buddy repeatedly to get the largest */ - /* buffer possible. */ - /*******************************************************************/ - -#include <mpidimpl.h> - -#define NO 0 -#define YES 1 -int application_set_eager_limit=0; - -#if TOKEN_FLOW_CONTROL -#define BUFFER_MEM_MAX (1<<26) /* 64MB */ -#define MAX_BUF_BKT_SIZE (1<<18) /* Max eager_limit is 256K */ -#define MIN_BUF_BKT_SIZE (64) -#define MIN_LOG2SIZE 6 /* minimum buffer size log-2 */ -#define MIN_SIZE 64 /* minimum buffer size */ -#define MAX_BUCKET 13 /* log of maximum buffer size/MIN_SIZE */ -#define MAX_SIZE (MIN_SIZE<<(MAX_BUCKET-1)) /* maximum buffer size */ -#define FLEX_COUNT 5 /* num of buf types to preallocate */ -#define FLEX_NUM 32 /* num of preallocated buffers */ - /* overhead for each buffer */ -#define MAX_BUDDIES 50 /* absolutely maximum number of buddies */ -#define BUDDY 1 -#define FLEX 0 -#define MAX_MALLOCS 10 - -#define OVERHEAD (sizeof(buddy_header) - 2 * sizeof (void *)) -#define TAB_SIZE (MAX_BUCKET+1) -#define ALIGN8(x) ((((unsigned long)(x)) + 0x7) & ~0x7L) -#ifndef max -#define max(a,b) ((a) > (b) ? (a) : (b)) -#endif -#ifndef min -#define min(a,b) (a>b ? b : a) -#endif - -/* normalize the size to number of MIN_SIZE blocks required to hold */ -/* required size. This includes the OVERHEAD */ -#define NORMSIZE(sz) (((sz)+MIN_SIZE+OVERHEAD-1) >> MIN_LOG2SIZE) - -typedef struct bhdr_struct{ - char buddy; /* buddy or flex alloc */ - char free; /* available or not */ - char bucket; /* log of buffer size */ - char *base_buddy; /* addr of max size buddy */ - struct bhdr_struct *next; /* list ptr, used if free */ - struct bhdr_struct *prev; /* list bptr, used if free */ -} buddy_header; - -typedef struct fhdr_struct{ - char buddy; /* buddy or flex alloc */ - char ind; /* which stack */ - /* bucket - min_bucket */ -} flex_header; - -typedef struct{ - void *ptr; /* malloc ptr need to be freed */ - int size; /* space allocted */ - int type; /* FLEX or BUDDY */ -} malloc_list_t; -malloc_list_t *malloc_list; - -static int nMallocs, maxMallocs; /* no. of malloc() issued */ -static int maxMallocs; /* max. no. of malloc() allowed */ -static int max_bucket; /* max. number of "buddy" bucket*/ -static int flex_count; /* pre-allocated bucket */ -static size_t max_size; /* max. size for each msg */ -static size_t flex_size; /* size for flex slot */ -static char *buddy_heap_ptr; /* ptr points to beg. of buddy */ -static char *end_heap_ptr; /* ptr points to end of buddy */ -static char *heap; /* begin. address of flex stack */ -static long mem_inuse; /* memory in use */ -long mem_hwmark; /* highest memory usage */ - - -static int sizetable[TAB_SIZE + 1]; /* from bucket to size */ -static int sizetrans[(1<<(MAX_BUCKET-1))+2];/* from size to power of 2 size*/ - -static char** flex_stack[TAB_SIZE]; /* for flex size alloc */ -static int flex_sp [TAB_SIZE]; /* flex stack pointer */ - -static buddy_header *free_buddy[TAB_SIZE];/* for buddy alloc */ - -int MPIDI_tfctrl_enabled=0; /* token flow control enabled */ -int MPIDI_tfctrl_hwmark=0; /* high water mark for tfc */ -int application_set_buf_mem=0; /* MP_BUFFER_MEM set by the user? */ -char *EagerLimit=NULL; /* export MP_EAGER_LIMIT if the */ - /* number is adjusted */ - -/***************************************************************************/ -/* calculate number of tokens available for each pair-wise communication. */ -/***************************************************************************/ - -void MPIDI_calc_tokens(int nTasks,uint *eager_limit_in, unsigned long *buf_mem_in ) -{ - char *cp; - unsigned long new_buf_mem_max,buf_mem_max; - int val; - int rc; - int i; - - /* Round up passed eager limit to power of 2 */ - if (MPIDI_Process.mp_buf_mem_max > *buf_mem_in) - buf_mem_max = MPIDI_Process.mp_buf_mem_max; - else - buf_mem_max= *buf_mem_in; - - if (*eager_limit_in != 0) { - for (val=1 ; val < *eager_limit_in ; val *= 2); - if (val > MAX_BUF_BKT_SIZE) { /* Maximum eager_limit is 256K */ - val = MAX_BUF_BKT_SIZE; - } - if (val < MIN_BUF_BKT_SIZE) { /* Minimum eager_limit is 64 */ - val = MIN_BUF_BKT_SIZE; - } - MPIDI_tfctrl_enabled = buf_mem_max / ((long)nTasks * val); - - /* We need to have a minimum of 2 tokens. If number of tokens is - * less than 2, re-calculate by reducing the eager-limit. - * If the number of tokens is still less than 2 then suggest a - * new minimum buf_mem. - */ - if (MPIDI_tfctrl_enabled < 2) { - for (val ; val>=MIN_BUF_BKT_SIZE; val/=2) { - MPIDI_tfctrl_enabled = (buf_mem_max) / ((long)nTasks * val); - if (MPIDI_tfctrl_enabled >= 2) { - break; - } - } - /* If the number of flow control tokens is still less than two */ - /* even with the eager_limit being reduced to 64, calculate a */ - /* new buf_mem value for 2 tokens and eager_limit = 64. */ - /* This will only happen if tasks>4K and original buf_mem=1M. */ - if (MPIDI_tfctrl_enabled < 2) { - /* Sometimes we are off by 1 - due to integer arithmetic. */ - new_buf_mem_max = (2 * nTasks * MIN_BUF_BKT_SIZE); - if ( (new_buf_mem_max <= BUFFER_MEM_MAX) && (!application_set_buf_mem) ) { - MPIDI_tfctrl_enabled = 2; - /* Reset val to mini (64) because the for loop above */ - /* would have changed it to 32. */ - val = MIN_BUF_BKT_SIZE; - buf_mem_max = new_buf_mem_max; - if ( application_set_buf_mem ) { - TRACE_ERR("informational messge \n"); - } - } - else { - /* Still not enough ...... Turn off eager send protocol */ - MPIDI_tfctrl_enabled = 0; - val = 0; - if (((MPIDI_Process.verbose >= MPIDI_VERBOSE_SUMMARY_0)|| (MPIDI_Process.mp_infolevel > 0)) - && (MPIR_Process.comm_world->rank == 0)) { - if ( application_set_buf_mem ) { - printf("ATTENTION: MP_BUFFER_MEM=%d was set too low, eager send protocol is disabled\n", - *buf_mem_in); - } - } - } - } - } - MPIDI_tfctrl_hwmark = (MPIDI_tfctrl_enabled+1) / 2; /* high water mark */ - /* Eager_limit may have been changed -- either rounded up or reduced */ - if ( *eager_limit_in != val ) { - if ( application_set_eager_limit && (*eager_limit_in > val)) { - /* Only give warning on reduce. */ - printf("ATTENTION: eager limit is reduced from %d to %d \n",*eager_limit_in,val); fflush(stdout); - } - *eager_limit_in = val; - - /* putenv MP_EAGER_LIMIT if it is changed */ - /* MP_EAGER_LIMIT always has a value. */ - /* need to check whether MP_EAGER_LIMIT has been export */ - EagerLimit = (char*)MPL_malloc(32 * sizeof(char)); - sprintf(EagerLimit, "MP_EAGER_LIMIT=%d",val); - rc = putenv(EagerLimit); - if (rc !=0) { - TRACE_ERR("PUTENV with Eager Limit failed \n"); - } - } - } - else { - /* Eager_limit = 0, all messages will be sent using rendezvous protocol */ - MPIDI_tfctrl_enabled = 0; - MPIDI_tfctrl_hwmark = 0; - } - /* user may want to set MP_EAGER_LIMIT to 0 or less than 256 */ - if (*eager_limit_in < MPIDI_Process.pt2pt.limits.application.immediate.remote) - MPIDI_Process.pt2pt.limits.application.immediate.remote= *eager_limit_in; - if (*eager_limit_in < MPIDI_Process.pt2pt.limits.application.eager.remote) - MPIDI_Process.pt2pt.limits.application.eager.remote= *eager_limit_in; -# ifdef DUMP_MM - printf("MPIDI_tfctrl_enabled=%d eager_limit=%d buf_mem=%d buf_mem_max=%d\n", - MPIDI_tfctrl_enabled,*eager_limit_in,*buf_mem_in,buf_mem_max); fflush(stdout); -# endif - -} - - - -/***************************************************************************/ -/* Initialize flex stack and stack pointers for each of FLEX_NUM slots */ -/***************************************************************************/ - -static char *MPIDI_init_flex(char *hp) -{ - int i,j,fcount; - char** area; - char *temp; - int size; - int kk; - - fcount = flex_count; - if ((fcount = flex_count) == 0) { - flex_size = 0; - return hp; - } -# ifdef DUMP_MM - printf("fcount=%d sizetable[fcount+1]=%d sizetable[1]=%d FLEX_NUM=%d overhead=%d\n", - fcount,sizetable[fcount+1],sizetable[1],FLEX_NUM,OVERHEAD); fflush(stdout); -# endif - flex_size = (sizetable[fcount+1] - sizetable[1]) *FLEX_NUM + - OVERHEAD * fcount * FLEX_NUM; - kk=fcount *FLEX_NUM *sizeof(char *); - size = ALIGN8(kk); - area = (char **) MPL_malloc(size); - malloc_list[nMallocs].ptr=(void *) area; - malloc_list[nMallocs].size=size; - malloc_list[nMallocs].type=FLEX; - nMallocs++; - - - flex_stack[0] = NULL; - for(i =1; i <=fcount; hp +=FLEX_NUM *(OVERHEAD +sizetable[i]), i++) { -# ifdef DEBUG - flex_heap[i] =hp; -# endif - flex_stack[i] =area; - area += FLEX_NUM; - flex_sp[i] =0; -# ifdef DUMP_MM - printf("MPIDI_init_flex() i=%d FLEX_NUM=%d fcount=%d i=%d flex_stack[i]=%p OVERHEAD=%d\n", - i,FLEX_NUM,fcount,i,flex_stack[i],OVERHEAD); fflush(stdout); -# endif - for(j =0; j <FLEX_NUM; j++){ - flex_stack[i][j] =hp +j *(OVERHEAD +sizetable[i]); -# ifdef DUMP_MM - printf("j=%d hp=%p advance =%d final=%x\n", j,(int *)hp,j *(OVERHEAD +sizetable[i]), - (int *) (hp +j *(OVERHEAD +sizetable[i]))); - fflush(stdout); -# endif - ((flex_header *)flex_stack[i][j])->buddy =FLEX; - ((flex_header *)flex_stack[i][j])->ind =i; -# ifdef DUMP_MM - printf("j=%d flex_stack[%02d][%02d] = %p advance=%d flag buddy=0x%x ind=0x%x\n", - j,i,j,((int *)(flex_stack[i][j])),(sizetable[i]+OVERHEAD), - ((flex_header *)flex_stack[i][j])->buddy, - ((flex_header *)flex_stack[i][j])->ind); fflush(stdout); -#endif - } - } - return hp; -} - -/***************************************************************************/ -/* Initialize buddy heap for each of MAX_BUDDIES slots */ -/***************************************************************************/ - - -static void MPIDI_alloc_buddies(int nums, int *space) -{ - int i; - uint size; - char *buddy,*prev; - - size = nums * (max_size +OVERHEAD); - buddy = buddy_heap_ptr; - if ((buddy_heap_ptr + size) > end_heap_ptr) { - /* preallocated space is exhausted, the caller needs to make */ - /* a malloc() call for storing the message */ - *space=NO; - return; - } - buddy_heap_ptr += size; - free_buddy[max_bucket] = (buddy_header *)buddy; - for(i =0, prev =NULL; i <nums; i++){ - ((buddy_header *)buddy)->buddy =BUDDY; - ((buddy_header *)buddy)->free =1; - ((buddy_header *)buddy)->bucket =max_bucket; - ((buddy_header *)buddy)->base_buddy =buddy; - ((buddy_header *)buddy)->prev =(buddy_header *)prev; - prev =buddy; -# ifdef DUMP_MM - printf("ALLOC_BUDDY i=%2d buddy=%d free=%d bucket=%d base_buddy=%p prev=%p next=%p max_size=%d \n", - i,(int)((buddy_header *)buddy)->buddy,(int)((buddy_header *)buddy)->free, - (int)((buddy_header *)buddy)->bucket,(int *) ((buddy_header *)buddy)->base_buddy, - (int *)((buddy_header *)buddy)->prev,(int *)(buddy + max_size +OVERHEAD),max_size); - fflush(stdout); -# endif - buddy +=max_size +OVERHEAD; - ((buddy_header *)prev)->next =(buddy_header *)buddy; - } - ((buddy_header *)prev)->next =NULL; -} - -/***************************************************************************/ -/* Initialize each of buddy slot */ -/***************************************************************************/ - -static void MPIDI_init_buddy(unsigned long buf_mem) -{ - int i; - int buddy_num; - size_t size; - int space=YES; - -# ifdef DEBUG - buddy_heap =buddy_heap_ptr; -# endif - for(i =0; i <= max_bucket; i++) - free_buddy[i] =NULL; - - /* figure out how many buddies we wanna preallocate - * size = BUFFER_MEM_SIZE >> 2; - * size -= flex_size; - */ - size = buf_mem; - size = size / (max_size + OVERHEAD); - size = (size == 0) ? 1 : (size > MAX_BUDDIES) ? MAX_BUDDIES : size; - MPIDI_alloc_buddies(size,&space); - if ( space == NO ) { - TRACE_ERR("out of memory %s(%d)\n",__FILE__,__LINE__); - MPID_abort(); - } -} - - -/***************************************************************************/ -/* initializ memory buffer for eager messages */ -/***************************************************************************/ - -int MPIDI_mm_init(int nTasks,uint *eager_limit_in,unsigned long *buf_mem_in) -{ - int i, bucket; - size_t size; - unsigned int eager_limit; - unsigned long buf_mem; - unsigned long buf_mem_max; - int need_allocation = 1; - - MPIDI_calc_tokens(nTasks,eager_limit_in, buf_mem_in); - buf_mem = *buf_mem_in; - eager_limit = *eager_limit_in; -# ifdef DEBUG - printf("Eager Limit=%d buf_mem=%ld tokens=%d hwmark=%d\n", - eager_limit,buf_mem, MPIDI_tfctrl_enabled,MPIDI_tfctrl_hwmark); - fflush(stdout); -# endif - if (eager_limit_in == 0) return 0; /* no EA buffer is needed */ - maxMallocs=MAX_MALLOCS; - malloc_list=(malloc_list_t *) MPL_malloc(maxMallocs * sizeof(malloc_list_t)); - if (malloc_list == NULL) return errno; - nMallocs=0; - mem_inuse=0; - mem_hwmark=0; - - - for (max_bucket=0,size=1 ; size < eager_limit ; max_bucket++,size *= 2); - max_size = 1 << max_bucket; - max_bucket -= (MIN_LOG2SIZE - 1); - flex_count = min(FLEX_COUNT,max_bucket); - for(i =1, size =MIN_SIZE; i <=MAX_BUCKET+1; i++, size =size << 1) - sizetable[i] =size; - sizetable[0] = 0; - - for (bucket=1, size = 1, i=1 ; bucket <= max_bucket; ) { - sizetrans[i++] = bucket; - if (i > size) { - size *= 2; - bucket++; - } - } - sizetrans[i] = sizetrans[i-1]; - /* 65536 is for flex stack which is not part of buf_mem_size */ - heap = MPL_malloc(buf_mem + 65536); - if (heap == NULL) return errno; - malloc_list[nMallocs].ptr=(void *) heap; - malloc_list[nMallocs].size=buf_mem + 65536; - malloc_list[nMallocs].type=BUDDY; - buddy_heap_ptr = heap; - end_heap_ptr = heap + buf_mem + 65536; -# ifdef DUMP_MM - printf("OVERHEAD=%d MAX_BUCKET=%d TAB_SIZE=%d buddy_header size=%d mem_size=%ld\n", - OVERHEAD,MAX_BUCKET,TAB_SIZE,sizeof(buddy_header),buf_mem); fflush(stdout); -# endif -# ifdef DEBUG - printf("nMallocs=%d ptr=%p size=%d type=%d bPtr=%p ePtr=%p\n", - nMallocs,(void *)malloc_list[nMallocs].ptr,malloc_list[nMallocs].size, - malloc_list[nMallocs].type,buddy_heap_ptr,end_heap_ptr); - fflush(stdout); -# endif - nMallocs++; - - buddy_heap_ptr = MPIDI_init_flex(heap); - MPIDI_init_buddy(buf_mem); -# ifdef MPIMM_DEBUG - if (mpimm_std_debug) { - printf("DevMemMgr uses\n"); - printf("\tmem-size=%d, maxbufsize=%ld\n",mem_size,max_size); - printf("\tfcount=%d fsize=%ld, max_bucket=%d\n", - flex_count,flex_size,max_bucket); - fflush(stdout); - } -# endif - return 0; -} - - -void MPIDI_close_mm() -{ - int i; - - if (nMallocs != 0) { - for (i=0; i< nMallocs; i++) { - MPL_free((void *) (malloc_list[i].ptr)); - } - MPL_free(malloc_list); - } -} - -/****************************************************************************/ -/* macros for MPIDI_mm_alloc() and MPIDI_mm_free() */ -/****************************************************************************/ - -#define MPIDI_flex_alloc(bucket) ((flex_sp[bucket] >=FLEX_NUM) ? \ - NULL : (char *)(flex_stack[bucket] \ - [flex_sp[bucket]++]) +OVERHEAD) - -#define MPIDI_flex_free(ptr) \ - int n; \ - ptr =(char *)ptr -OVERHEAD; \ - n =((flex_header *)ptr)->ind; \ - flex_stack[n][--flex_sp[n]] =(char *)ptr; - - -#define MPIDI_remove_head(ind) \ - { \ - if((free_buddy[ind] = \ - free_buddy[ind]->next) !=NULL) \ - free_buddy[ind]->prev =NULL; \ - } - -#define MPIDI_remove(bud) \ - { \ - if(bud->prev !=NULL) \ - bud->prev->next =bud->next; \ - else \ - free_buddy[bud->bucket] =bud->next; \ - if(bud->next !=NULL) \ - bud->next->prev =bud->prev; \ - } - -#define MPIDI_add_head(bud,ind) \ - { \ - if((bud->next =free_buddy[ind]) !=NULL)\ - free_buddy[ind]->prev =bud; \ - free_buddy[ind] =bud; \ - bud->prev =NULL; \ - } - -#define MPIDI_fill_header(bud,ind,base) \ - { \ - bud->buddy =BUDDY; \ - bud->free =1; \ - bud->bucket =ind; \ - bud->base_buddy =base; \ - } - -#define MPIDI_pair(bud,size) (((char*)(bud) - (bud)->base_buddy) & (size) ? \ - (buddy_header *)((char*)(bud) - (size)) : \ - (buddy_header *)((char*)(bud) + (size))) - -static buddy_header *MPIDI_split_buddy(int big,int bucket) -{ - buddy_header *bud,*buddy; - char *base; - int i; - - bud =free_buddy[big]; - MPIDI_remove_head(big); - base =bud->base_buddy; - for(i =big -1; i >=bucket; i--){ - buddy =(buddy_header *)((char*)bud +sizetable[i]); - MPIDI_fill_header(buddy,i,base); - MPIDI_add_head(buddy,i); - } - bud->bucket =bucket; - bud->free =0; - return bud; -} - -static void *MPIDI_buddy_alloc(int bucket) -{ - int i; - buddy_header *bud; - int space=YES; - -# ifdef TRACE - printf("(buddy) "); -# endif - if((bud =free_buddy[i =bucket]) ==NULL){ - i++; - do { - for(; i <=max_bucket; i++) - if(free_buddy[i] !=NULL){ - bud =MPIDI_split_buddy(i,bucket); - return (char *)bud +OVERHEAD; - } - MPIDI_alloc_buddies(1,&space); - if (space == NO) - return NULL; - i = max_bucket; - } while (1); - } - else{ - MPIDI_remove_head(bucket); - bud->free =0; - return (char *)bud +OVERHEAD; - } -} - -static buddy_header *MPIDI_merge_buddy(buddy_header *bud) -{ - buddy_header *buddy; - int size; - - while( (bud->bucket <max_bucket) && - (size = sizetable[bud->bucket]) && - ((buddy =MPIDI_pair(bud,size))->free) && - (buddy->bucket ==bud->bucket) ) - { - MPIDI_remove(buddy); - bud =min(bud,buddy); - bud->bucket++; - } - return bud; -} - -static void MPIDI_buddy_free(void *ptr) -{ - buddy_header *bud; - -# ifdef TRACE - printf("(buddy) "); -# endif - bud =(buddy_header *)((char *)ptr -OVERHEAD); - if(bud->bucket <max_bucket) - bud =MPIDI_merge_buddy(bud); - bud->free =1; - MPIDI_add_head(bud,bud->bucket); -} -# ifdef TRACE - int nAllocs =0; /* number of times MPIDI_mm_alloc() is called */ - int nFree =0; /* number of times MPIDI_mm_free() is called */ - int nM=0; /* number of times MPL_malloc() is called */ - int nF=0; /* number of times MPL_free() is called */ -# endif -void *MPIDI_mm_alloc(size_t size) -{ - void *pt; - int bucket,tmp; - - MPID_assert(size <= max_size); - tmp = NORMSIZE(size); - tmp =bucket =sizetrans[tmp]; - if(bucket >flex_count || (pt =MPIDI_flex_alloc(tmp)) ==NULL) { - pt =MPIDI_buddy_alloc(bucket); - if (MPIDI_Process.mp_statistics) { - mem_inuse = mem_inuse + sizetable[tmp]; - if (mem_inuse > mem_hwmark) { - mem_hwmark = mem_inuse; - } - } - } - if (pt == NULL) { - MPIU_THREAD_CS_ENTER(MSGQUEUE,0); - pt=MPL_malloc(size); - MPIU_THREAD_CS_EXIT(MSGQUEUE,0); - if (MPIDI_Process.mp_statistics) { - mem_inuse = mem_inuse + sizetable[tmp]; - if (mem_inuse > mem_hwmark) { - mem_hwmark = mem_inuse; - } - } -# ifdef TRACE - nM++; - if (pt == NULL) { - printf("ERROR line=%d\n",__LINE__); fflush(stdout); - } else { - printf("malloc nM=%d size=%d pt=0x%p \n",nM,size,pt); fflush(stdout); - } -# endif - } -# ifdef TRACE - printf("MPIDI_mm_alloc(%4d): %p\n",size,pt); - nAllocs++; -# endif - return pt; -} - -void MPIDI_mm_free(void *ptr, size_t size) -{ - int tmp,bucket; - - if (size > MAX_SIZE) { - TRACE_ERR("Out of memory in %s(%d)\n",__FILE__,__LINE__); - MPID_abort(); - } - if ((ptr >= (void *) heap) && (ptr < (void *)end_heap_ptr)) { - if(*((char *)ptr -OVERHEAD) ==FLEX){ - MPIDI_flex_free(ptr); - } - else - MPIDI_buddy_free(ptr); - if (MPIDI_Process.mp_statistics) { - tmp = NORMSIZE(size); - bucket =sizetrans[tmp]; - mem_inuse = mem_inuse - sizetable[bucket]; - if (mem_inuse > mem_hwmark) { - mem_hwmark = mem_inuse; - } - } - } else { - if (!ptr) { - TRACE_ERR("NULL ptr passed MPIDI_mm_free() in %s(%d)\n",__FILE__,__LINE__); - MPID_abort(); - } - if (MPIDI_Process.mp_statistics) { - tmp = NORMSIZE(size); - bucket =sizetrans[tmp]; - mem_inuse = mem_inuse - sizetable[bucket]; - if (mem_inuse > mem_hwmark) { - mem_hwmark = mem_inuse; - } - } - MPIU_THREAD_CS_ENTER(MSGQUEUE,0); -# ifdef TRACE - nF++; - printf("free nF=%d size=%d ptr=0x%p \n",nF,sizetable[bucket],ptr); fflush(stdout); -# endif - MPL_free(ptr); - MPIU_THREAD_CS_EXIT(MSGQUEUE,0); - } -# ifdef TRACE - nFrees++; - printf("MPIDI_mm_free: %p \n",ptr); -# endif -} -#endif /* #if TOKEN_FLOW_CONTROL */ diff --git a/src/mpid/pamid/src/mpidi_env.c b/src/mpid/pamid/src/mpidi_env.c deleted file mode 100644 index 740bcee..0000000 --- a/src/mpid/pamid/src/mpidi_env.c +++ /dev/null @@ -1,1314 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/mpidi_env.c - * \brief Read Env Vars - */ - -#include <mpidimpl.h> - -#include "mpidi_util.h" - -/** \page env_vars Environment Variables - *************************************************************************** - * Pt2Pt Options * - *************************************************************************** - * - * - PAMID_EAGER - - * - PAMID_RZV - Sets the cutoff for the switch to the rendezvous protocol. - * The two options are identical. This takes an argument, in bytes, - * to switch from the eager protocol to the rendezvous protocol for point-to-point - * messaging. Increasing the limit might help for larger partitions - * and if most of the communication is nearest neighbor. - * - Default is 2049 bytes. - * - * - PAMID_EAGER_LOCAL - - * - PAMID_RZV_LOCAL - Sets the cutoff for the switch to the rendezvous protocol - * when the destination rank is local. The two options are identical. This - * takes an argument, in bytes, to switch from the eager protocol to the - * rendezvous protocol for point-to-point messaging. The default value - * effectively disables the eager protocol for local transfers. - * - Default is 0 bytes. - * - * - PAMID_RMA_PENDING - Maximum outstanding RMA requests. - * Limits number of PAMI_Request objects allocated by MPI Onesided operations. - * - Default is 1000. - * - * - PAMID_SHMEM_PT2PT - Determines if intranode point-to-point communication will - * use the optimized shared memory protocols. - * - Default is 1. - *************************************************************************** - * High-level Options * - *************************************************************************** - * - * - PAMID_THREAD_MULTIPLE - Specifies the messaging execution environment. - * It specifically selects whether there can be multiple independent - * communications occurring in parallel, driven by internal communications - * threads. - * - 0: The application threads drive the communications. No additional - * internal communications threads are used. - * - 1: There can be multiple independent communications occurring in - * parallel, driven by internal communications threads. - * Defaults to 0 when the application is linked with one of the "legacy" - * MPICH libraries (gcc.legacy, xl.legacy, xl.legacy.ndebug), or when - * MPI_Init_thread() is called without MPI_THREAD_MULTIPLE, and defaults - * to 1 when the application is linked with one of the "non-legacy" MPICH - * libraries (gcc, xl, xl.ndebug) and MPI_Init_thread() is called with - * MPI_THREAD_MULTIPLE. - * - NOTE: This environment variable has the same effect as setting - * MPIR_CVAR_DEFAULT_THREAD_LEVEL=multiple - * - * - PAMID_CONTEXT_MAX - This variable sets the maximum allowable number - * of contexts. Contexts are a method of dividing hardware resources - * among a parallel active messaging interface (PAMI) client (for example, - * MPI) to set how many parallel operations can occur at one time. Contexts - * are similar to channels in a communications system. The practical - * maximum is usually 64 processes per node. The default depends on the - * processes per node. - * - * - PAMID_CONTEXT_POST - This variable is required to take advantage of - * the parallelism of multiple contexts. This variable can increase - * latency, but it is the only way to allow parallelism among contexts. - * - 0: Only one parallel communications context can be used. Each - * operation runs in the application thread. - * - 1: Multiple parallel communications contexts can be used. An - * operation is posted to one of the contexts, and communications - * for that context are driven by communications threads. - * The default value is 1 when using the gcc, xl, and xl.ndebug libraries - * and MPI_Init_thread(... MPI_THREAD_MULTIPLE ...), and 0 otherwise. - * - * - PAMID_ASYNC_PROGRESS - This variable enables or disables the async - * progress extension. - * - * - PAMID_COLLECTIVES - Controls whether optimized collectives are used. - * Possible values: - * - 0 - Optimized collectives are not used. - * - 1 - Optimized collectives are used. - * If this is set to 0, only MPICH point-to-point based collectives will be used. - * - * - PAMID_COLLECTIVES_SELECTION - Turns on optimized collective selection. If this - * is not on, only the generic PGAS "always works" algorithms will be selected. This - * can still be better than PAMID_COLLECTIVES=0. Additionally, setting this off - * still allows environment variable selection of individual collectives protocols. - * - 0 - Optimized collective selection is not used. - * - 1 - Optimized collective selection is used. (default) - * - * - PAMID_COLLECTIVES_MEMORY_OPTIMIZED - Controls whether collectives are - * optimized to reduce memory usage. This may disable some PAMI collectives. - * Possible values: - * - 0 - Collectives are not memory optimized. - * - n - Collectives are memory optimized. Levels are bitwise values : - * MPIR_OPT_LVL_IRREG = 1, Do not optimize irregular communicators - * MPIR_OPT_LVL_NONCONTIG = 2, Disable some non-contig collectives - * - * PAMID_OPTIMIZED_SUBCOMMS - Use PAMI 'optimized' collectives. Defaullt is 1. - * - 0 - Some optimized protocols may be disabled. - * - 1 - All performance optimized protocols will be enabled when available - * - * - PAMID_VERBOSE - Increases the amount of information dumped during an - * MPI_Abort() call and during varoius MPI function calls. Possible values: - * - 0 - No additional information is dumped. - * - 1 - Useful information is printed from MPI_Init(). This option does - * NOT impact performance (other than a tiny bit during MPI_Init()) and - * is highly recommended for all applications. It will display which PAMID_ - * and BG_ env vars were used. However, it does NOT guarantee you typed the - * env vars in correctly :) - * - 2 - This turns on a lot of verbose output for collective selection, - * including a list of which protocols are viable for each communicator - * creation. This can be a lot of output, but typically only at - * communicator creation. Additionally, if PAMID_STATISTICS are on, - * queue depths for point-to-point operations will be printed for each node - * during MPI_Finalize(); - * - 3 - This turns on a lot of per-node information (plus everything - * at the verbose=2 level). This can be a lot of information and is - * rarely recommended. - * - Default is 0. However, setting to 1 is recommended. - * - * - PAMID_STATISTICS - Turns on statistics printing for the message layer - * such as the maximum receive queue depth. Possible values: - * - 0 - No statistics are printedcmf_bcas. - * - 1 - Statistics are printed. - * - Default is 0. - * - *************************************************************************** - *************************************************************************** - ** Collective Options ** - *************************************************************************** - *************************************************************************** - * - *************************************************************************** - * General Collectives Options * - *************************************************************************** - * - * - PAMID_NUMREQUESTS - Sets the number of outstanding asynchronous - * collectives to have before a barrier is called. This is used when - * the PAMI collective metadata indicates 'asyncflowctl' may be needed - * to avoid 'flooding' other participants with unexpected data. - * Higher numbers can help on larger partitions and larger message sizes. - * - * After every {PAMID_NUMREQUESTS} async collectives, the "glue" will call - * a barrier. - * - Default is 1 (guaranteed functionality) - * - N>1may used to tune performance - * - *************************************************************************** - * "Safety" Options * - *************************************************************************** - * - * - PAMID_SAFEALLGATHER - Some optimized allgather protocols require - * contiguous datatypes and similar datatypes on all nodes. To verify - * this is true, we must do an allreduce at the beginning of the - * allgather call. If the application uses "well behaved" datatypes, you can - * set this option to skip over the allreduce. This is most useful in - * irregular subcommunicators where the allreduce can be expensive and in - * applications calling MPI_Allgather() with simple/regular datatypes. - * Datatypes must also be 16-byte aligned to use the optimized - * broadcast-based allgather for larger allgather sendcounts. See - * PAMID_PREALLREDUCE for more options - * Possible values: - * - N - Perform the preallreduce. - * - Y - Skip the preallreduce. Setting this with "unsafe" datatypes will - * yield unpredictable results, usually hangs. - * - Default is N. - * - * - PAMID_SAFEALLGATHERV - The optimized allgatherv protocols require - * contiguous datatypes and similar datatypes on all nodes. Allgatherv - * also requires continuous displacements. To verify - * this is true, we must do an allreduce at the beginning of the - * allgatherv call. If the application uses "well behaved" datatypes and - * displacements, you can set this option to skip over the allreduce. - * This is most useful in irregular subcommunicators where the allreduce - * can be expensive and in applications calling MPI_Allgatherv() with - * simple/regular datatypes. - * Datatypes must also be 16-byte aligned to use the optimized - * broadcast-based allgather for larger allgather sendcounts. See - * PAMID_PREALLREDUCE for more options - * Possible values: - * - N - Perform the allreduce. - * - Y - Skip the allreduce. Setting this with "unsafe" datatypes will - * yield unpredictable results, usually hangs. - * - Default is N. - * - * - PAMID_SAFESCATTERV - The optimized scatterv protocol requires - * contiguous datatypes and similar datatypes on all nodes. It - * also requires continuous displacements. To verify - * this is true, we must do an allreduce at the beginning of the - * scatterv call. If the application uses "well behaved" datatypes and - * displacements, you can set this option to skip over the allreduce. - * This is most useful in irregular subcommunicators where the allreduce - * can be expensive. We have seen more applications with strange datatypes - * passed to scatterv than allgather/allgatherv/bcast/allreduce, so it is - * more likely you need to leave this at the default setting. However, we - * encourage you to try turning this off and seeing if your application - * completes successfully. - * Possible values: - * - N - Perform the allreduce. - * - Y - Skip the allreduce. Setting this with "unsafe" datatypes will - * yield unpredictable results, usually hangs. - * - Default is N. - * - * - PAMID_PREALLREDUCE - Controls the protocol used for the pre-allreducing - * employed by allgather(v), and scatterv. This option - * is independant from PAMID_ALLREDUCE. The available protocols can be - * determined with PAMID_VERBOSE=2. If collective selection is on, we will - * attempt to use the fastest protocol for the given communicator. - * - *************************************************************************** - * Specific Collectives Switches * - *************************************************************************** - * - * - PAMID_ALLGATHER - Controls the protocol used for allgather. - * Possible values: - * - MPICH - Turn off all optimizations for allgather and use the MPICH - * point-to-point protocol. This can be very bad on larger partitions. - * - Other options can be determined by running with PAMID_VERBOSE=2. - * TODO: Implement the NO options - * NOTE: Individual options may be turned off by prefixing them with "NO". - * For example, PAMID_ALLGATHER=NO{specific protocol name} will prevent the - * use of the specified protocol. - * - * - PAMID_ALLGATHERV - Controls the protocol used for allgatherv. - * Possible values: - * - MPICH - Turn off all optimizations for allgather and use the MPICH - * point-to-point protocol. This can be very bad on larger partitions. - * - Other options can be determined by running with PAMID_VERBOSE=2. - * TODO: Implement the NO options - * NOTE: Individual options may be turned off by prefixing them with "NO". - * For example, PAMID_ALLGATHERV=NO{specific protocol name} will prevent the - * use of the specified protocol. - * - * - PAMID_ALLREDUCE - Controls the protocol used for allreduce. - * Possible values: - * - MPICH - Turn off all optimizations for allreduce and use the MPICH - * point-to-point protocol. - * - Other options can be determined by running with PAMID_VERBOSE=2. - * Note: Many allreduce algorithms are in the "must query" category and - * might or might not be available for a specific callsite. - * TODO: Implement the NO options - * NOTE: Individual options may be turned off by prefixing them with "NO". - * For example, PAMID_ALLREDUCE=NO{specific protocol name} will prevent the - * use of the specified protocol. - * - * - PAMID_ALLTOALL - - * - PAMID_ALLTOALLV - Controls the protocol used for - * alltoall/alltoallv Possible values: - * - MPICH - Turn off all optimizations and use the MPICH - * point-to-point protocol. - * - Other options can be determined by running with PAMID_VERBOSE=2. - * TODO: Implement the NO options - * NOTE: Individual options may be turned off by prefixing them with "NO". - * For example, PAMID_ALLTOALL=NO{specific protocol name} will prevent the - * use of the specified protocol. - * - * - * - PAMID_BARRIER - Controls the protocol used for barriers. - * Possible values: - * - MPICH - Turn off optimized barriers and use the MPICH - * point-to-point protocol. - * - Other options can be determined by running with PAMID_VERBOSE=2. - * TODO: Implement the NO options - * NOTE: Individual options may be turned off by prefixing them with "NO". - * For example, PAMID_BARRIER=NO{specific protocol name} will prevent the - * use of the specified protocol. - * - * - PAMID_BCAST - Controls the protocol used for broadcast. Possible values: - * - MPICH - Turn off all optimizations for broadcast and use the MPICH - * point-to-point protocol. - * - Other options can be determined by running with PAMID_VERBOSE=2. - * TODO: Implement the NO options - * NOTE: Individual options may be turned off by prefixing them with "NO". - * For example, PAMID_BCAST=NO{specific protocol name} will prevent the - * use of the specified protocol. - * - * - PAMID_GATHER - Controls the protocol used for gather. - * Possible values: - * - MPICH - Use the MPICH point-to-point protocol. - * - Other options can be determined by running with PAMID_VERBOSE=2. - * TODO: Implement the NO options - * NOTE: Individual options may be turned off by prefixing them with "NO". - * For example, PAMID_GATHER=NO{specific protocol name} will prevent the - * use of the specified protocol. - * - * - PAMID_REDUCE - Controls the protocol used for reduce. - * Possible values: - * - MPICH - Turn off all optimizations and use the MPICH point-to-point - * protocol. - * - Other options can be determined by running with PAMID_VERBOSE=2. - * TODO: Implement the NO options - * NOTE: Individual options may be turned off by prefixing them with "NO". - * For example, PAMID_REDUCE=NO{specific protocol name} will prevent the - * use of the specified protocol. - * - * - PAMID_REDUCE_SCATTER - Controls the protocol used for reduce_scatter - * operations. The options for PAMID_SCATTERV and PAMID_REDUCE can change the - * behavior of reduce_scatter. Possible values: - * - MPICH - Use the MPICH point-to-point protocol. - * - Other options can be determined by running with PAMID_VERBOSE=2. - * TODO: Implement the NO options - * NOTE: Individual options may be turned off by prefixing them with "NO". - * For example, PAMID_REDUCESCATTER=NO{specific protocol name} will prevent the - * use of the specified protocol. - * - * - PAMID_SCATTER - Controls the protocol used for scatter. - * Possible values: - * - MPICH - Use the MPICH point-to-point protocol. - * - Other options can be determined by running with PAMID_VERBOSE=2. - * TODO: Implement the NO options - * NOTE: Individual options may be turned off by prefixing them with "NO". - * For example, PAMID_SCATTER=NO{specific protocol name} will prevent the - * use of the specified protocol. - * - * - PAMID_SCATTERV - Controls the protocol used for scatterv. - * Possible values: - * - MPICH - Use the MPICH point-to-point protocol. - * - Other options can be determined by running with PAMID_VERBOSE=2. - * TODO: Implement the NO options - * NOTE: Individual options may be turned off by prefixing them with "NO". - * For example, PAMID_SCATTERV=NO{specific protocol name} will prevent the - * use of the specified protocol. - * - * - */ -#if (MPIDI_STATISTICS || MPIDI_PRINTENV) -int prtStat=0; -int prtEnv=0; - -MPIX_stats_t *mpid_statp=NULL; -extern MPIDI_printenv_t *mpich_env; -#endif - -#define ENV_Deprecated(a, b, c, d, e) ENV_Deprecated__(a, b, c, d, e) - -#ifdef TOKEN_FLOW_CONTORL - extern int MPIDI_get_buf_mem(unsigned long *); - extern int MPIDI_atoi(char* , int* ); -#endif - extern int application_set_eager_limit; - -static inline void -ENV_Deprecated__(char* name[], unsigned num_supported, unsigned* deprecated, int rank, int NA) -{ - if (name == NULL) return; - - unsigned i; - char * env; - - for (i=0; i<num_supported; ++i) - { - if (name[i] == NULL) return; - - if (NA) - { - env = getenv(name[i]); - if (env != NULL) - { - if (rank == 0) - { - if (*deprecated == 0) - fprintf (stderr, "\n"); - - fprintf (stderr, "The environment variable \"%s\" is not applicable.\n", name[i]); - } - *deprecated = 1; - } - } - } - - for (i=num_supported; name[i] != NULL; ++i) - { - env = getenv(name[i]); - if (env != NULL) - { - if (rank == 0) - { - if (*deprecated == 0) - fprintf (stderr, "\n"); - - if (NA) - fprintf (stderr, "The environment variable \"%s\" is deprecated.\n", name[i]); - else - { - char supported[10240]; - int n, index = 0; - index += snprintf(&supported[index], 10240-index-1, "\"%s\"", name[0]); - for (n=1; n<num_supported; ++n) - index += snprintf(&supported[index], 10240-index-1, " or \"%s\"", name[n]); - - fprintf (stderr, "The environment variable \"%s\" is deprecated. Consider using %s instead.\n", name[i], supported); - } - } - *deprecated = 1; - } - } -} - -#define ENV_Unsigned(a, b, c, d, e) ENV_Unsigned__(a, b, #b, c, d, e, 0) - -#define ENV_Unsigned_NA(a, b, c, d, e) ENV_Unsigned__(a, b, #b, c, d, e, 1) - -static inline void -ENV_Unsigned__(char* name[], unsigned* val, char* string, unsigned num_supported, unsigned* deprecated, int rank, int NA) -{ - /* Check for deprecated environment variables. */ - if (deprecated != NULL) - ENV_Deprecated(name, num_supported, deprecated, rank, NA); - - char * env; - int rc; - - unsigned i=0; - for (;; ++i) { - if (name[i] == NULL) - return; - env = getenv(name[i]); - if (env != NULL) - break; - } - - unsigned oldval = *val; - rc=MPIDI_atoi(env,val); - if(rc != 0) - { - /* Something went wrong with the processing this integer - * Print a warning, and restore the original value */ - *val = oldval; - fprintf(stderr, "Warning: Environment variable: %s should be an integer value: defaulting to %d", string, *val); - return; - } - - if (MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_ALL && rank == 0) - fprintf(stderr, "%s = %u\n", string, *val); -} - - -#define ENV_Char(a, b) ENV_Char__(a, b, #b) -static inline void -ENV_Char__(char* name[], unsigned* val, char* string) -{ - char * env; - unsigned i=0; - for (;; ++i) { - if (name[i] == NULL) - return; - env = getenv(name[i]); - if (env != NULL) - break; - } - - if ( (env[0]=='y')|| (env[0]=='Y') - || (env[0]=='p')|| (env[0]=='P') - || (env[0]=='t')|| (env[0]=='T') - || (env[0]=='r')|| (env[0]=='R') - || (env[0]=='F')|| (env[0]=='f')) - *val = 1; - /*This may seem redundant; however, - in some cases we need to force val=0 if value = no/none*/ - if ((env[0]=='n')|| (env[0]=='N')) - *val = 0; - if (MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_ALL) - fprintf(stderr, "%s = %u\n", string, *val); -} - -#define ENV_Char2(a, b) ENV_Char2__(a, b, #b) -static inline void -ENV_Char2__(char* name[], char** val, char* string) -{ - unsigned i=0; - for (;; ++i) { - if (name[i] == NULL) - return; - *val = getenv(name[i]); - if (*val != NULL) - break; - } - - if (MPIDI_Process.verbose >= MPIDI_VERBOSE_DETAILS_ALL) - fprintf(stderr, "%s = %s\n", string, *val); -} - -/** \brief Checks the Environment variables at initialization and stores the results. - * - * \param [in] rank The process rank; used to limit output - * \param [in] requested The thread model requested by the user - */ -void -MPIDI_Env_setup(int rank, int requested) -{ - unsigned found_deprecated_env_var = 0; - - /* Set defaults for various environment variables */ - - /* Set the verbosity level. When set, this will print information at finalize. */ - { - char* names[] = {"PAMID_VERBOSE", "PAMI_VERBOSE", NULL}; - ENV_Unsigned(names, &MPIDI_Process.verbose, 1, &found_deprecated_env_var, rank); - } - - /* Enable statistics collection. */ - { - char* names[] = {"PAMID_STATISTICS", "PAMI_STATISTICS", NULL}; - ENV_Unsigned(names, &MPIDI_Process.statistics, 1, &found_deprecated_env_var, rank); - } - - /* Set async flow control - number of collectives between barriers */ - { - char* names[] = {"PAMID_NUMREQUESTS", NULL}; - ENV_Unsigned(names, &MPIDI_Process.optimized.num_requests, 1, &found_deprecated_env_var, rank); - TRACE_ERR("MPIDI_Process.optimized.num_requests=%u\n", MPIDI_Process.optimized.num_requests); - } - - /* "Globally" set the optimization flag for low-level collectives in geometry creation. - * This is probably temporary. metadata should set this flag likely. - */ - { - /* THIS ENVIRONMENT VARIABLE NEEDS TO BE DOCUMENTED ABOVE */ - char *names[] = {"PAMID_OPTIMIZED_SUBCOMMS", "PAMI_OPTIMIZED_SUBCOMMS", NULL}; - ENV_Unsigned(names, &MPIDI_Process.optimized.subcomms, 1, &found_deprecated_env_var, rank); - TRACE_ERR("MPIDI_Process.optimized.subcomms=%u\n", MPIDI_Process.optimized.subcomms); - } - - /* Set the threads values from a single override. */ - { - unsigned value = (unsigned)-1; - char* names[] = {"PAMID_THREAD_MULTIPLE", NULL}; - ENV_Unsigned(names, &value, 1, &found_deprecated_env_var, rank); -#if (MPICH_THREAD_GRANULARITY == MPICH_THREAD_GRANULARITY__POBJ) - /* Any mpich work function posted to a context that eventually initiates - * other communcation transfers will hang on a lock attempt if the - * 'context post' feature is not enabled. Until this code flow is fixed - * the context post must not be disabled. - * - * See discussion in src/mpid/pamid/include/mpidi_macros.h - * -> MPIDI_Context_post() - */ - MPIDI_Process.perobj.context_post.requested = 1; - if (value == 1) /* force on */ - { - MPIDI_Process.async_progress.mode = ASYNC_PROGRESS_MODE_DEFAULT; - MPIDI_Process.avail_contexts = MPIDI_MAX_CONTEXTS; - } - else if (value == 0) /* force off */ - { - MPIDI_Process.async_progress.mode = ASYNC_PROGRESS_MODE_DISABLED; - MPIDI_Process.avail_contexts = 1; - } - else if (requested != MPI_THREAD_MULTIPLE) - { -#ifdef BGQ_SUPPORTS_TRIGGER_ASYNC_PROGRESS - MPIDI_Process.async_progress.mode = ASYNC_PROGRESS_MODE_TRIGGER; - MPIDI_Process.avail_contexts = MPIDI_MAX_CONTEXTS; -#else - /* BGQ does not support the 'trigger' style of async progress. Until - * this is implemented, set the async progress configuration to the - * 'single threaded' defaults. - */ - MPIDI_Process.async_progress.mode = ASYNC_PROGRESS_MODE_DISABLED; - MPIDI_Process.avail_contexts = 1; -#endif - } -#else - /* The only valid async progress mode when using the 'global' mpich lock - * mode is the 'trigger' async progress mode. Also, the 'global' mpich lock - * mode only supports a single context. - * - * See discussions in mpich/src/mpid/pamid/src/mpid_init.c and - * src/mpid/pamid/src/mpid_progress.h for more information. - */ - if (value == 1) /* force on */ - { - MPIDI_Process.async_progress.mode = ASYNC_PROGRESS_MODE_TRIGGER; - MPIDI_Process.avail_contexts = 1; - } - else if (value == 0) /* force off */ - { - MPIDI_Process.async_progress.mode = ASYNC_PROGRESS_MODE_DISABLED; - MPIDI_Process.avail_contexts = 1; - } -#endif - } - - /* Set the upper-limit of number of PAMI Contexts. */ - { - unsigned value = (unsigned)-1; - char *names[] = {"PAMID_CONTEXT_MAX", "PAMI_MAXCONTEXT", "PAMI_MAXCONTEXTS", "PAMI_MAX_CONTEXT", "PAMI_MAX_CONTEXTS", NULL}; - ENV_Unsigned(names, &value, 1, &found_deprecated_env_var, rank); - - if (value != -1) - { -#if (MPICH_THREAD_GRANULARITY != MPICH_THREAD_GRANULARITY__POBJ) - /* The 'global' mpich lock mode only supports a single context. - * See discussion in mpich/src/mpid/pamid/src/mpid_init.c for more - * information. - */ - if (value > 1) - { - found_deprecated_env_var++; - if (MPIDI_Process.verbose >= MPIDI_VERBOSE_SUMMARY_0 && rank == 0) - fprintf(stderr, "The environment variable \"PAMID_CONTEXT_MAX\" is invalid as this mpich library was configured.\n"); - } -#endif - if (value == 0) - { - if (MPIDI_Process.verbose >= MPIDI_VERBOSE_SUMMARY_0 && rank == 0) - fprintf(stderr, "The environment variable \"PAMID_CONTEXT_MAX\" must specify a value > 0. The number of contexts will be set to 1\n"); - - value=1; - } - - MPIDI_Process.avail_contexts=value; - } - - TRACE_ERR("MPIDI_Process.avail_contexts=%u\n", MPIDI_Process.avail_contexts); - } - - /* Enable context post. */ - { - unsigned value = (unsigned)-1; - char *names[] = {"PAMID_CONTEXT_POST", "PAMI_CONTEXT_POST", NULL}; - ENV_Unsigned(names, &value, 1, &found_deprecated_env_var, rank); - if (value != -1) - { -#if (MPICH_THREAD_GRANULARITY == MPICH_THREAD_GRANULARITY__POBJ) - MPIDI_Process.perobj.context_post.requested = (value > 0); -#else - found_deprecated_env_var++; - if (MPIDI_Process.verbose >= MPIDI_VERBOSE_SUMMARY_0 && rank == 0) - fprintf(stderr, "The environment variable \"PAMID_CONTEXT_POST\" is invalid as this mpich library was configured.\n"); -#endif - } - TRACE_ERR("MPIDI_Process.perobj.context_post.requested=%u\n", MPIDI_Process.perobj.context_post.requested); - } - - /* Enable/Disable asynchronous progress. */ - { - /* THIS ENVIRONMENT VARIABLE NEEDS TO BE DOCUMENTED ABOVE */ - unsigned value = (unsigned)-1; - char *names[] = {"PAMID_ASYNC_PROGRESS", "PAMID_COMMTHREADS", "PAMI_COMMTHREAD", "PAMI_COMMTHREADS", "PAMI_COMM_THREAD", "PAMI_COMM_THREADS", NULL}; - ENV_Unsigned(names, &value, 1, &found_deprecated_env_var, rank); - - if (value != (unsigned)-1) - { - if (value != ASYNC_PROGRESS_MODE_DISABLED) - { -#if (MPICH_THREAD_GRANULARITY == MPICH_THREAD_GRANULARITY__POBJ) - if (value == ASYNC_PROGRESS_MODE_LOCKED && - MPIDI_Process.perobj.context_post.requested == 0) - { - /* The 'locked' async progress mode requires context post. - * - * See discussion in src/mpid/pamid/src/mpid_progress.h for more - * information. - */ - found_deprecated_env_var++; - if (MPIDI_Process.verbose >= MPIDI_VERBOSE_SUMMARY_0 && rank == 0) - fprintf(stderr, "The environment variable \"PAMID_ASYNC_PROGRESS=1\" requires \"PAMID_CONTEXT_POST=1\".\n"); - } - -#else /* (MPICH_THREAD_GRANULARITY != MPICH_THREAD_GRANULARITY__POBJ) */ - if (value == ASYNC_PROGRESS_MODE_LOCKED) - { - /* The only valid async progress mode when using the 'global' mpich - * lock mode is the 'trigger' async progress mode. - * - * See discussion in src/mpid/pamid/src/mpid_progress.h for more - * information. - */ - found_deprecated_env_var++; - if (MPIDI_Process.verbose >= MPIDI_VERBOSE_SUMMARY_0 && rank == 0) - fprintf(stderr, "The environment variable \"PAMID_ASYNC_PROGRESS=1\" is invalid as this mpich library was configured.\n"); - - } -#endif - } - - MPIDI_Process.async_progress.mode = value; - } - TRACE_ERR("MPIDI_Process.async_progress.mode=%u\n", MPIDI_Process.async_progress.mode); - } - - /* - * Determine 'short' limit - * - sets both the 'local' and 'remote' short limit, and - * - sets both the 'application' and 'internal' short limit - * - * Identical to setting the PAMID_PT2PT_LIMITS environment variable as: - * - * PAMID_PT2PT_LIMITS="::x:x:::x:x" - */ - { - /* THIS ENVIRONMENT VARIABLE NEEDS TO BE DOCUMENTED ABOVE */ - char* names[] = {"PAMID_SHORT", "MP_S_SHORT_LIMIT", "PAMI_SHORT", NULL}; - ENV_Unsigned(names, &MPIDI_Process.pt2pt.limits.application.immediate.remote, 2, &found_deprecated_env_var, rank); - ENV_Unsigned(names, &MPIDI_Process.pt2pt.limits.application.immediate.local, 2, NULL, rank); - ENV_Unsigned(names, &MPIDI_Process.pt2pt.limits.internal.immediate.remote, 2, NULL, rank); - ENV_Unsigned(names, &MPIDI_Process.pt2pt.limits.internal.immediate.local, 2, NULL, rank); - } - - /* - * Determine 'remote' eager limit - * - sets both the 'application' and 'internal' remote eager limit - * - * Identical to setting the PAMID_PT2PT_LIMITS environment variable as: - * - * PAMID_PT2PT_LIMITS="x::::x:::" - * -- or -- - * PAMID_PT2PT_LIMITS="x::::x" - */ - { - char* names[] = {"PAMID_EAGER", "PAMID_RZV", "MP_EAGER_LIMIT", "PAMI_RVZ", "PAMI_RZV", "PAMI_EAGER", NULL}; - ENV_Unsigned(names, &MPIDI_Process.pt2pt.limits.application.eager.remote, 3, &found_deprecated_env_var, rank); - ENV_Unsigned(names, &MPIDI_Process.pt2pt.limits.internal.eager.remote, 3, NULL, rank); - } -#if TOKEN_FLOW_CONTROL - /* Determine if users set eager limit */ - { - MPIDI_set_eager_limit(&MPIDI_Process.pt2pt.limits.application.eager.remote); - } - /* Determine buffer memory for early arrivals */ - { - int rc; - rc=MPIDI_get_buf_mem(&MPIDI_Process.mp_buf_mem,&MPIDI_Process.mp_buf_mem_max); - MPID_assert_always(rc == MPI_SUCCESS); - } -#endif - - /* - * Determine 'local' eager limit - * - sets both the 'application' and 'internal' local eager limit - * - * Identical to setting the PAMID_PT2PT_LIMITS environment variable as: - * - * PAMID_PT2PT_LIMITS=":x::::x::" - * -- or -- - * PAMID_PT2PT_LIMITS=":x::::x" - */ - { - char* names[] = {"PAMID_RZV_LOCAL", "PAMID_EAGER_LOCAL", "MP_EAGER_LIMIT_LOCAL", "PAMI_RVZ_LOCAL", "PAMI_RZV_LOCAL", "PAMI_EAGER_LOCAL", NULL}; - ENV_Unsigned(names, &MPIDI_Process.pt2pt.limits.application.eager.local, 3, &found_deprecated_env_var, rank); - ENV_Unsigned(names, &MPIDI_Process.pt2pt.limits.internal.eager.local, 3, NULL, rank); - } - - /* - * Determine *all* point-to-point limit overrides. - * - * The entire point-to-point limit set is determined by three boolean - * configuration values: - * - 'is non-local limit' vs 'is local limit' - * - 'is eager limit' vs 'is immediate limit' - * - 'is application limit' vs 'is internal limit' - * - * The point-to-point configuration limit values are specified in order and - * are delimited by ':' characters. If a value is not specified for a given - * configuration then the limit is not changed. All eight configuration - * values are not required to be specified, although in order to set the - * last (eighth) configuration value the previous seven configurations must - * be listed. For example: - * - * PAMID_PT2PT_LIMITS=":::::::10240" - * - * The configuration entries can be described as: - * 0 - remote eager application limit - * 1 - local eager application limit - * 2 - remote immediate application limit - * 3 - local immediate application limit - * 4 - remote eager internal limit - * 5 - local eager internal limit - * 6 - remote immediate internal limit - * 7 - local immediate internal limit - * - * Examples: - * - * "10240" - * - sets the application internode eager (the "normal" eager limit) - * - * "10240::64" - * - sets the application internode eager and immediate limits - * - * "::::0:0:0:0" - * - disables 'eager' and 'immediate' for all internal point-to-point - */ - { - char * env = getenv("PAMID_PT2PT_LIMITS"); - if (env != NULL) - { - size_t i, n = strlen(env); - char * tmp = (char *) MPL_malloc(n+1); - strncpy(tmp,env,n); - if (n>0) tmp[n]=0; - - char * tail = tmp; - char * token = tail; - for (i = 0; token == tail; i++) - { - while (*tail != 0 && *tail != ':') tail++; - if (*tail == ':') - { - *tail = 0; - if (token != tail) - MPIDI_atoi(token, &MPIDI_Process.pt2pt.limits_array[i]); - tail++; - token = tail; - } - else - { - if (token != tail) - MPIDI_atoi(token, &MPIDI_Process.pt2pt.limits_array[i]); - } - } - - MPL_free (tmp); - } - } - - /* Set the maximum number of outstanding RDMA requests */ - { - char* names[] = {"PAMID_RMA_PENDING", "MP_RMA_PENDING", "PAMI_RMA_PENDING", NULL}; - ENV_Unsigned(names, &MPIDI_Process.rma_pending, 2, &found_deprecated_env_var, rank); - } - - /* Set the status of the optimized collectives */ - { - char* names[] = {"PAMID_COLLECTIVES", "PAMI_COLLECTIVE", "PAMI_COLLECTIVES", NULL}; - ENV_Unsigned(names, &MPIDI_Process.optimized.collectives, 1, &found_deprecated_env_var, rank); - TRACE_ERR("MPIDI_Process.optimized.collectives=%u\n", MPIDI_Process.optimized.collectives); - } - - /* First, if MP_COLLECTIVE_OFFLOAD is "on", then we want PE (FCA) collectives */ - { - unsigned temp; - temp = 0; - char* names[] = {"MP_COLLECTIVE_OFFLOAD", NULL}; - ENV_Char(names, &temp); - if(temp) - MPIDI_Process.optimized.collectives = MPID_COLL_FCA; - } - - /* However, MP_MP_PAMI_FOR can be set to "none" in which case we don't want PE (FCA) collectives */ - { - char *env = getenv("MP_MPI_PAMI_FOR"); - if(env != NULL) - { - if(strncasecmp(env, "N", 1) == 1) - MPIDI_Process.optimized.collectives = MPID_COLL_OFF; - } - } - - /* Set the status for optimized selection of collectives */ - { - char* names[] = {"PAMID_COLLECTIVES_SELECTION", NULL}; - ENV_Unsigned(names, &MPIDI_Process.optimized.select_colls, 1, &found_deprecated_env_var, rank); - TRACE_ERR("MPIDI_Process.optimized.select_colls=%u\n", MPIDI_Process.optimized.select_colls); - } - - /* Finally, if MP_COLLECTIVE_SELECTION is "on", then we want to overwrite any other setting */ - { - char *env = getenv("MP_COLLECTIVE_SELECTION"); - if(env != NULL) - { - pami_extension_t extension; - pami_result_t status = PAMI_ERROR; - status = PAMI_Extension_open (MPIDI_Client, "EXT_collsel", &extension); - if(status == PAMI_SUCCESS) - { - char *env = getenv("MP_COLLECTIVE_SELECTION"); - if(env != NULL) - { - if(strncasecmp(env, "TUN", 3) == 0) - { - MPIDI_Process.optimized.auto_select_colls = MPID_AUTO_SELECT_COLLS_TUNE; - if(MPIDI_Process.optimized.collectives != MPID_COLL_FCA) - MPIDI_Process.optimized.collectives = MPID_COLL_ON; - } - else if(strncasecmp(env, "YES", 3) == 0) - { - MPIDI_Process.optimized.auto_select_colls = MPID_AUTO_SELECT_COLLS_ALL; /* All collectives will be using auto coll sel. - We will check later on each individual coll. */ - if(MPIDI_Process.optimized.collectives != MPID_COLL_FCA) - MPIDI_Process.optimized.collectives = MPID_COLL_ON; - } - else - MPIDI_Process.optimized.auto_select_colls = MPID_AUTO_SELECT_COLLS_NONE; - - } - } - else - MPIDI_Process.optimized.auto_select_colls = MPID_AUTO_SELECT_COLLS_NONE;/* Auto coll sel is disabled for all */ - } - else - MPIDI_Process.optimized.auto_select_colls = MPID_AUTO_SELECT_COLLS_NONE;/* Auto coll sel is disabled for all */ - } - - /* Set the status for memory optimized collectives */ - { - char* names[] = {"PAMID_COLLECTIVES_MEMORY_OPTIMIZED", NULL}; - ENV_Unsigned(names, &MPIDI_Process.optimized.memory, 1, &found_deprecated_env_var, rank); - TRACE_ERR("MPIDI_Process.optimized.memory=%u\n", MPIDI_Process.optimized.memory); - } - - /* Set the status of the optimized shared memory point-to-point functions */ - { - char* names[] = {"PAMID_SHMEM_PT2PT", "PAMI_SHMEM_PT2PT", NULL}; - ENV_Unsigned(names, &MPIDI_Process.shmem_pt2pt, 2, &found_deprecated_env_var, rank); - } - - /* MP_SHMEM_PT2PT = yes or no */ - { - char* names[] = {"MP_SHMEM_PT2PT", NULL}; - ENV_Char(names, &MPIDI_Process.shmem_pt2pt); - } - - /* Enable MPIR_* implementations of non-blocking collectives */ - { - char* names[] = {"PAMID_MPIR_NBC", NULL}; - ENV_Unsigned(names, &MPIDI_Process.mpir_nbc, 1, &found_deprecated_env_var, rank); - } - - /* Enable typed PAMI calls for derived types within MPID_Put and MPID_Get. */ - { - char* names[] = {"PAMID_TYPED_ONESIDED", NULL}; - ENV_Unsigned(names, &MPIDI_Process.typed_onesided, 1, &found_deprecated_env_var, rank); - } - /* Check for deprecated collectives environment variables. These variables are - * used in src/mpid/pamid/src/comm/mpid_selectcolls.c */ - { - { - char* names[] = {"PAMID_COLLECTIVE_ALLGATHER", "PAMI_ALLGATHER", NULL}; - ENV_Deprecated(names, 1, &found_deprecated_env_var, rank, 0); - } - { - char* names[] = {"PAMID_COLLECTIVE_ALLGATHER_PREALLREDUCE", "PAMI_ALLGATHER_PREALLREDUCE", NULL}; - ENV_Deprecated(names, 1, &found_deprecated_env_var, rank, 0); - } - { - char* names[] = {"PAMID_COLLECTIVE_ALLGATHERV", "PAMI_ALLGATHERV", NULL}; - ENV_Deprecated(names, 1, &found_deprecated_env_var, rank, 0); - } - { - char* names[] = {"PAMID_COLLECTIVE_ALLGATHERV_PREALLREDUCE", "PAMI_ALLGATHERV_PREALLREDUCE", NULL}; - ENV_Deprecated(names, 1, &found_deprecated_env_var, rank, 0); - } - { - char* names[] = {"PAMID_COLLECTIVE_ALLREDUCE", "PAMI_ALLREDUCE", NULL}; - ENV_Deprecated(names, 1, &found_deprecated_env_var, rank, 0); - } - { - char* names[] = {"PAMID_COLLECTIVE_ALLREDUCE_PREALLREDUCE", "PAMI_ALLREDUCE_PREALLREDUCE", NULL}; - ENV_Deprecated(names, 1, &found_deprecated_env_var, rank, 0); - } - { - char* names[] = {"PAMID_COLLECTIVE_ALLTOALL", "PAMI_ALLTOALL", NULL}; - ENV_Deprecated(names, 1, &found_deprecated_env_var, rank, 0); - } - { - char* names[] = {"PAMID_COLLECTIVE_ALLTOALLV", "PAMI_ALLTOALLV", NULL}; - ENV_Deprecated(names, 1, &found_deprecated_env_var, rank, 0); - } - { - char* names[] = {"PAMID_COLLECTIVE_ALLTOALLV_INT", "PAMI_ALLTOALLV_INT", NULL}; - ENV_Deprecated(names, 1, &found_deprecated_env_var, rank, 0); - } - { - char* names[] = {"PAMID_COLLECTIVE_BARRIER", "PAMI_BARRIER", NULL}; - ENV_Deprecated(names, 1, &found_deprecated_env_var, rank, 0); - } - { - char* names[] = {"PAMID_COLLECTIVE_BCAST", "PAMI_BCAST", NULL}; - ENV_Deprecated(names, 1, &found_deprecated_env_var, rank, 0); - } - { - char* names[] = {"PAMID_COLLECTIVE_BCAST_PREALLREDUCE", "PAMI_BCAST_PREALLREDUCE", NULL}; - ENV_Deprecated(names, 1, &found_deprecated_env_var, rank, 0); - } - { - char* names[] = {"PAMID_COLLECTIVE_GATHER", "PAMI_GATHER", NULL}; - ENV_Deprecated(names, 1, &found_deprecated_env_var, rank, 0); - } - { - char* names[] = {"PAMID_COLLECTIVE_GATHERV", "PAMI_GATHERV", NULL}; - ENV_Deprecated(names, 1, &found_deprecated_env_var, rank, 0); - } - { - char* names[] = {"PAMID_COLLECTIVE_REDUCE", "PAMI_REDUCE", NULL}; - ENV_Deprecated(names, 1, &found_deprecated_env_var, rank, 0); - } - { - char* names[] = {"PAMID_COLLECTIVE_SCAN", "PAMI_SCAN", NULL}; - ENV_Deprecated(names, 1, &found_deprecated_env_var, rank, 0); - } - { - char* names[] = {"PAMID_COLLECTIVE_SCATTER", "PAMI_SCATTER", NULL}; - ENV_Deprecated(names, 1, &found_deprecated_env_var, rank, 0); - } - { - char* names[] = {"PAMID_COLLECTIVE_SCATTERV", "PAMI_SCATTERV", NULL}; - ENV_Deprecated(names, 1, &found_deprecated_env_var, rank, 0); - } - { - char* names[] = {"PAMID_COLLECTIVE_SCATTERV_PREALLREDUCE", "PAMI_SCATTERV_PREALLREDUCE", NULL}; - ENV_Deprecated(names, 1, &found_deprecated_env_var, rank, 0); - } - { - char* names[] = {"PAMID_CORE_ON_ABORT", "PAMI_COREONABORT", "PAMI_COREONMPIABORT", "PAMI_COREONMPIDABORT", NULL}; - ENV_Deprecated(names, 1, &found_deprecated_env_var, rank, 0); - } -#if (MPIDI_STATISTICS || MPIDI_PRINTENV) - mpich_env=(MPIDI_printenv_t *) MPL_malloc(sizeof(MPIDI_printenv_t)+1); - mpid_statp=(MPIX_stats_t *) MPL_malloc(sizeof(MPIX_stats_t)+1); - memset((void *)mpich_env,0,sizeof(MPIDI_printenv_t)); - memset((void *)mpid_statp,0,sizeof(MPIX_stats_t)); - /* If MP_STATISTICS is set, each task prints statistics data at the end of an MPI jobs */ - { - char* names[] = {"MP_STATISTICS", NULL}; - ENV_Char(names, &MPIDI_Process.mp_statistics); - if (MPIDI_Process.mp_statistics) { - prtStat=1; - } - } - /* if MP_INFOLEVEL is greater than or equal to 1, the library will display the banner */ - { - char* names[] = {"MP_INFOLEVEL", NULL}; - ENV_Unsigned(names, &MPIDI_Process.mp_infolevel, 1, &found_deprecated_env_var, rank); - } - /* If MP_PRINTENV is set, task 0 prints out the env. info after MPI_Init */ - { - char* names[] = {"MP_PRINTENV", NULL}; - ENV_Char(names, &MPIDI_Process.mp_printenv); - if (MPIDI_Process.mp_printenv) { - prtEnv=1; - } - } - /* MP_CSS_INTERRUPT */ - { - char *cp=NULL, *cp1=NULL; - int user_interrupts=0; - char* names[] = {"MP_CSS_INTERRUPT", NULL}; - ENV_Char(names, &mpich_env->interrupts); - if (mpich_env->interrupts == 1) /* force on */ - { - cp = getenv("MP_CSS_INTERRUPT"); - if (*cp=='Y' || *cp=='y') - { - user_interrupts = ASYNC_PROGRESS_ALL; - } - else - { - char delimiter='+'; - cp1 = strchr(cp,delimiter); - if (!cp1) /* timer or receive */ - { - if ( (*cp == 't') || (*cp == 'T') ) - user_interrupts = PAMIX_PROGRESS_TIMER; - else if ( (*cp == 'r') || (*cp == 'R') ) - user_interrupts = PAMIX_PROGRESS_RECV_INTERRUPT; - } - else /* timer + receive */ - { - if ((( *cp == 't' || *cp == 'T') && ( *(cp1+1) == 'r' || *(cp1+1) == 'R')) || - (( *cp == 'r' || *cp == 'R') && ( *(cp1+1) == 't' || *(cp1+1) == 'T'))) - user_interrupts = ASYNC_PROGRESS_ALL; - else - { - TRACE_ERR("ERROR in MP_CSS_INTERRUPT %s(%d)\n",__FILE__,__LINE__); - exit(1); - } - } - } - MPIDI_Process.mp_interrupts=user_interrupts; - MPIDI_Process.perobj.context_post.requested = 0; - MPIDI_Process.async_progress.mode = ASYNC_PROGRESS_MODE_TRIGGER; -#if (MPICH_THREAD_GRANULARITY == MPICH_THREAD_GRANULARITY__POBJ) - MPIDI_Process.avail_contexts = MPIDI_MAX_CONTEXTS; -#else - MPIDI_Process.avail_contexts = 1; -#endif - } - else if (mpich_env->interrupts == 0) /* force off */ - { - MPIDI_Process.perobj.context_post.requested = 0; - MPIDI_Process.async_progress.mode = ASYNC_PROGRESS_MODE_DISABLED; - MPIDI_Process.avail_contexts = 1; - } - } - /* MP_POLLING_INTERVAL */ - { - char* names[] = {"MP_POLLING_INTERVAL", NULL}; - ENV_Unsigned(names,(unsigned int *) &mpich_env->polling_interval,1, &found_deprecated_env_var, rank); - } - - /* MP_RETRANSMIT_INTERVAL */ - { - char* names[] = {"MP_RETRANSMIT_INTERVAL", NULL}; - ENV_Unsigned(names, &mpich_env->retransmit_interval,1, &found_deprecated_env_var, rank); - } - - /* MP_S_USE_PAMI_GET */ - { - char* names[] = {"MP_S_USE_PAMI_GET", NULL}; - ENV_Char(names, &MPIDI_Process.mp_s_use_pami_get); - } -#endif - /* MP_S_SMP_DETECT */ - { - char* names[] = {"MP_S_SMP_DETECT", "PAMID_SMP_DIRECT", NULL}; - ENV_Char(names, &MPIDI_Process.smp_detect); - if(!MPIDI_Process.smp_detect) - PAMIX_Extensions.is_local_task.node_info=NULL; - } - - } - { -#if TOKEN_FLOW_CONTROL - char* names[] = {"MP_USE_TOKEN_FLOW_CONTROL", NULL}; - ENV_Char(names, &MPIDI_Process.is_token_flow_control_on); - if (!MPIDI_Process.is_token_flow_control_on) - MPIDI_Process.mp_buf_mem=0; -#endif - } - -#ifdef QUEUE_BINARY_SEARCH_SUPPORT - char* names[] = {"MP_S_USE_QUEUE_BINARY_SEARCH_SUPPORT", NULL}; - ENV_Char(names, &MPIDI_Process.queue_binary_search_support_on); -#endif - -#if CUDA_AWARE_SUPPORT - char* names[] = {"MP_CUDA_AWARE", NULL}; - ENV_Char(names, &MPIDI_Process.cuda_aware_support_on); - if(MPIDI_Process.cuda_aware_support_on && MPIDI_enable_cuda() == false) - { - MPIDI_Process.cuda_aware_support_on = false; - if(rank == 0) - { - fprintf(stderr, "Error loading libcudart\n");fflush(stderr);sleep(1);exit(1); - } - } - else if(MPIDI_Process.cuda_aware_support_on) - { - if(MPIDI_Process.optimized.collectives == MPID_COLL_FCA) - if(rank == 0) - { - fprintf(stderr, "Warning: FCA is not supported with CUDA Aware support\n");fflush(stderr); - } - - MPIDI_Process.optimized.collectives = MPID_COLL_CUDA; - MPIDI_Process.optimized.select_colls = 0; - } -#endif - - /* Exit if any deprecated environment variables were specified. */ - if (found_deprecated_env_var) - { - if (rank == 0) - { - // Only rank 0 prints and exits. sleep to make sure message is sent before exiting. - // Other ranks will proceed, but will be waiting in a barrier in context create - // when the exit occurs. - fprintf (stderr, "\n"); fflush(stderr); sleep(1); - exit(1); - } - } -} - -#if TOKEN_FLOW_CONTROL -int MPIDI_set_eager_limit(unsigned int *eager_limit) -{ - char *cp; - int val; - int numTasks=MPIDI_Process.numTasks; - cp = getenv("MP_EAGER_LIMIT"); - if (cp) - { - application_set_eager_limit=1; - if ( MPIDI_atoi(cp, &val) == 0 ) - *eager_limit=val; - } - else - { - /* set default - * Number of tasks MP_EAGER_LIMIT - * ----------------- -------------- - * 1 - 256 32768 - * 257 - 512 16384 - * 513 - 1024 8192 - * 1025 - 2048 4096 - * 2049 - 4096 2048 - * 4097 & above 1024 - */ - if (numTasks < 257) *eager_limit = 32768; - else if (numTasks < 513) *eager_limit = 16384; - else if (numTasks < 1025) *eager_limit = 8192; - else if (numTasks < 2049) *eager_limit = 4096; - else if (numTasks < 4097) *eager_limit = 2048; - else *eager_limit = 1024; - - } - return 0; -} - - /******************************************************************/ - /* */ - /* Check for MP_BUFFER_MEM, if the value is not set by the user, */ - /* then set the value with the default of 64 MB. */ - /* MP_BUFFER_MEM supports the following format: */ - /* MP_BUFFER_MEM=xxM */ - /* MP_BUFFER_MEM=xxM,yyyM */ - /* MP_BUFFER_MEM=xxM,yyyG */ - /* MP_BUFFER_MEM=,yyyM */ - /* xx: pre allocated size the max. allowable value is 256 MB */ - /* the space is allocated during the initialization. */ - /* the default is 64 MB */ - /* yyy: maximum size - the maximum size to which the early arrival*/ - /* buffer can temporarily grow when the preallocated portion */ - /* of the EA buffer has been filled. */ - /* */ - /******************************************************************/ -int MPIDI_get_buf_mem(unsigned long *buf_mem,unsigned long *buf_mem_max) - { - char *cp; - int i; - int args_in_error=0; - char pre_alloc_buf[25], buf_max[25]; - char *buf_max_cp; - int pre_alloc_val=0; - unsigned long buf_max_val; - int has_error = 0; - extern int application_set_buf_mem; - - if (cp = getenv("MP_BUFFER_MEM")) { - pre_alloc_buf[24] = '\0'; - buf_max[24] = '\0'; - application_set_buf_mem=1; - if ( (buf_max_cp = strchr(cp, ',')) ) { - if ( *(++buf_max_cp) == '\0' ) { - /* Error: missing buffer_mem_max */ - has_error = 1; - } - else if ( cp[0] == ',' ) { - /* Pre_alloc value is default -- use default */ - pre_alloc_val = -1; - strncpy(buf_max, buf_max_cp, 24); - if ( MPIDI_atoll(buf_max, &buf_max_val) != 0 ) - has_error = 1; - } - else { - /* both values are present */ - for (i=0; ; i++ ) { - if ( (cp[i] != ',') && (i<24) ) - pre_alloc_buf[i] = cp[i]; - else { - pre_alloc_buf[i] = '\0'; - break; - } - } - strncpy(buf_max, buf_max_cp, 24); - if ( MPIDI_atoi(pre_alloc_buf, &pre_alloc_val) == 0 ) { - if ( MPIDI_atoll(buf_max, &buf_max_val) != 0 ) - has_error = 1; - } - else - has_error = 1; - } - } - else - { - /* Old single value format */ - if ( MPIDI_atoi(cp, &pre_alloc_val) == 0 ) - buf_max_val = (unsigned long)pre_alloc_val; - else - has_error = 1; - } - if ( has_error == 0) { - if ((int) pre_alloc_val != -1) /* MP_BUFFER_MEM=,128MB */ - *buf_mem = (int) pre_alloc_val; - if (buf_max_val > ONE_SHARED_SEGMENT) - *buf_mem = ONE_SHARED_SEGMENT; - if (buf_max_val != *buf_mem_max) - *buf_mem_max = buf_max_val; - } else { - args_in_error += 1; - TRACE_ERR("ERROR in MP_BUFFER_MEM %s(%d)\n",__FILE__,__LINE__); - return 1; - } - return 0; - } else { - /* MP_BUFFER_MEM is not specified by the user*/ - *buf_mem = BUFFER_MEM_DEFAULT; - TRACE_ERR("buffer_mem=%d buffer_mem_max=%d\n",*buf_mem,*buf_mem_max); - return 0; - } -} -#endif diff --git a/src/mpid/pamid/src/mpidi_mutex.c b/src/mpid/pamid/src/mpidi_mutex.c deleted file mode 100644 index 8411848..0000000 --- a/src/mpid/pamid/src/mpidi_mutex.c +++ /dev/null @@ -1,88 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/mpidi_mutex.c - * \brief ??? - */ -#include <mpidimpl.h> - - -#if MPIDI_MUTEX_L2_ATOMIC - -static inline void* -MPIDI_Mutex_initialize_l2atomics(size_t size) -{ - typedef pami_result_t (*pamix_proc_memalign_fn) (void**, size_t, size_t, const char*); - - pami_result_t rc; - pami_extension_t l2; - pamix_proc_memalign_fn PAMIX_L2_proc_memalign; - void* l2atomics = NULL; - - rc = PAMI_Extension_open(NULL, "EXT_bgq_l2atomic", &l2); - MPID_assert_always(rc == PAMI_SUCCESS); - PAMIX_L2_proc_memalign = (pamix_proc_memalign_fn)PAMI_Extension_symbol(l2, "proc_memalign"); - MPID_assert_always(PAMIX_L2_proc_memalign != NULL); - rc = PAMIX_L2_proc_memalign(&l2atomics, 64, size, NULL); - MPID_assert_always(rc == PAMI_SUCCESS); - MPID_assert_always(l2atomics != NULL); - /* printf("MPID L2 space: virt=%p HW=%p L2BaseAddress=%"PRIu64"\n", l2atomics, __l2_op_ptr(l2atomics, 0), Kernel_L2AtomicsBaseAddress()); */ - - return l2atomics; -} - -MPIDI_Mutex_t * MPIDI_Mutex_vector; -uint32_t MPIDI_Mutex_counter[MPIDI_MAX_THREADS][MPIDI_MAX_MUTEXES]; - -/** - * \brief Initialize a mutex. - * - * In this API, mutexes are acessed via indices from - * 0..MPIDI_MAX_MUTEXES. The mutexes are recursive - */ -int -MPIDI_Mutex_initialize() -{ - size_t i, j; - - MPIDI_Mutex_vector = (MPIDI_Mutex_t*)MPIDI_Mutex_initialize_l2atomics(sizeof (MPIDI_Mutex_t) * MPIDI_MAX_MUTEXES); - - for (i=0; i<MPIDI_MAX_MUTEXES; ++i) { - L2_AtomicStore(&(MPIDI_Mutex_vector[i].counter), 0); - L2_AtomicStore(&(MPIDI_Mutex_vector[i].bound), 1); - } - - for (i=0; i<MPIDI_MAX_MUTEXES; ++i) { - for (j=0; j<MPIDI_MAX_THREADS; ++j) { - MPIDI_Mutex_counter[j][i] = 0; - } - } - - return 0; -} - -#elif MPIDI_MUTEX_LLSC - -MPIDI_Mutex_t MPIDI_Mutex_vector [MPIDI_MAX_MUTEXES]; -uint32_t MPIDI_Mutex_counter[MPIDI_MAX_THREADS][MPIDI_MAX_MUTEXES]; - -#else - -pthread_mutex_t MPIDI_Mutex_lock; - -#endif diff --git a/src/mpid/pamid/src/mpidi_nbc_sched.c b/src/mpid/pamid/src/mpidi_nbc_sched.c deleted file mode 100644 index 998d03e..0000000 --- a/src/mpid/pamid/src/mpidi_nbc_sched.c +++ /dev/null @@ -1,65 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/mpidi_nbc_sched.c - * \brief Non-blocking collectives hooks - */ - -#include <pami.h> -#include <mpidimpl.h> - -/** - * work object for persistent advance of nbc schedules - */ -pami_work_t mpidi_nbc_work_object; - -/** - * \brief Persistent work function for nbc schedule progress - */ -pami_result_t mpidi_nbc_work_function (pami_context_t context, void *cookie) -{ - int made_progress = 0; - MPIDU_Sched_progress (&made_progress); - - return PAMI_EAGAIN; -} - -/** - * \brief Initialize support for MPIR_* nbc implementation. - * - * The MPIR_* non-blocking collectives only work if the schedule is advanced. - * This is done by posting a work function to context 0 that invokes the - * schedule progress function. - * - * Because this is a persistent work function and will negatively impact the - * performance of all other MPI operations - even when mpir non-blocking - * collectives are not used - the work function is only posted if explicitly - * requested. - */ -void MPIDI_NBC_init () -{ - if (MPIDI_Process.mpir_nbc != 0) - { - PAMI_Context_post(MPIDI_Context[0], - &mpidi_nbc_work_object, - mpidi_nbc_work_function, - NULL); - } - - return; -} diff --git a/src/mpid/pamid/src/mpidi_pami_datatype.c b/src/mpid/pamid/src/mpidi_pami_datatype.c deleted file mode 100644 index 296ef4f..0000000 --- a/src/mpid/pamid/src/mpidi_pami_datatype.c +++ /dev/null @@ -1,155 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/mpidi_pami_datatype.c - * \brief pami_type_t datatype hooks - */ - -#include <pami.h> -#include <mpidimpl.h> - -/** - * \brief Create PAMI datatype representation of MPI Datatype during commit. - * - * Signifcant performance improvements can be realized for one-sided communication - * utilizing the PAMI_Rput_typed and PAMI_Rget_typed interface which requires a - * PAMI representation of the MPI Datatype. - */ -void MPIDI_PAMI_datatype_commit_hook (MPI_Datatype *ptr) -{ - - /* If the PAMID optimization to utilize the PAMI_Rput_typed / PAMI_Rget_typed call for - * one-sided comm for derived types is enabled then we need to create the PAMI datatype. - */ - if (MPIDI_Process.typed_onesided == 1) { - - MPIDU_Datatype*datatype_ptr; - MPIDU_Datatype_get_ptr(*ptr, datatype_ptr); - - pami_result_t pami_dtop_result; - datatype_ptr->device_datatype = (pami_type_t *) MPL_malloc(sizeof(pami_type_t)); - pami_dtop_result = PAMI_Type_create ((pami_type_t *)datatype_ptr->device_datatype); - MPIR_Assert(pami_dtop_result == PAMI_SUCCESS); - - /* Flatten the non-contiguous data type into arrays describing the contiguous chunks. - */ - MPI_Aint *dt_offset_array = (MPI_Aint *) MPL_malloc(datatype_ptr->max_contig_blocks * sizeof(MPI_Aint)); - MPI_Aint *dt_size_array = (MPI_Aint *) MPL_malloc(datatype_ptr->max_contig_blocks * sizeof(MPI_Aint)); - MPI_Aint dt_array_len = datatype_ptr->max_contig_blocks; - int rc = MPIR_Type_flatten(*ptr, dt_offset_array, dt_size_array, &dt_array_len); - - /* Build the PAMI datatype adding one contiguous chunk at a time with the PAMI_Type_add_simple - * interface. - */ - int i; - - for (i=0;i<dt_array_len;i++) { - size_t num_bytes_this_entry = dt_size_array[i]; - size_t cursor_offset; - if (i == 0) - cursor_offset = (size_t) dt_offset_array[i]; - else - cursor_offset = (size_t) dt_offset_array[i] - (size_t)dt_offset_array[i-1]; - pami_dtop_result = PAMI_Type_add_simple (*(pami_type_t*)(datatype_ptr->device_datatype), num_bytes_this_entry, cursor_offset, 1, 0); - MPIR_Assert(pami_dtop_result == PAMI_SUCCESS); - } - - /* Complete the PAMI datatype and free arrays. - */ - pami_dtop_result = PAMI_Type_complete (*(pami_type_t*)(datatype_ptr->device_datatype),1); - MPIR_Assert(pami_dtop_result == PAMI_SUCCESS); - MPL_free(dt_offset_array); - MPL_free(dt_size_array); - } - return; -} - -/** - * \brief Destroy PAMI datatype representation of MPI Datatype. - * - */ -void MPIDI_PAMI_datatype_destroy_hook (MPIDU_Datatype*ptr) -{ - /* If a PAMI datatype was created, destroy it if this is the - * last reference to the MPIDU_Datatypeptr. - */ - if ((MPIDI_Process.typed_onesided == 1) && (ptr->is_committed)) { - if (ptr->device_datatype) { - pami_result_t pami_dtop_result; - pami_dtop_result = PAMI_Type_destroy ((pami_type_t *)ptr->device_datatype); - MPIR_Assert(pami_dtop_result == PAMI_SUCCESS); - MPL_free(ptr->device_datatype); - } - } -} - -/** - * \brief Create PAMI datatype representation of MPI Datatype during dup. - * - * Signifcant performance improvements can be realized for one-sided communication - * utilizing the PAMI_Rput_typed and PAMI_Rget_typed interface which requires a - * PAMI representation of the MPI Datatype. - */ -void MPIDI_PAMI_datatype_dup_hook (MPI_Datatype *ptr) -{ - - /* If the PAMID optimization to utilize the PAMI_Rput_typed / PAMI_Rget_typed call for - * one-sided comm for derived types is enabled then we need to create the PAMI datatype. - */ - if (MPIDI_Process.typed_onesided == 1) { - - MPIDU_Datatype*datatype_ptr; - MPIDU_Datatype_get_ptr(*ptr, datatype_ptr); - - pami_result_t pami_dtop_result; - datatype_ptr->device_datatype = (pami_type_t *) MPL_malloc(sizeof(pami_type_t)); - pami_dtop_result = PAMI_Type_create ((pami_type_t *)datatype_ptr->device_datatype); - MPIR_Assert(pami_dtop_result == PAMI_SUCCESS); - - /* Flatten the non-contiguous data type into arrays describing the contiguous chunks. - */ - MPI_Aint *dt_offset_array = (MPI_Aint *) MPL_malloc(datatype_ptr->max_contig_blocks * sizeof(MPI_Aint)); - MPI_Aint *dt_size_array = (MPI_Aint *) MPL_malloc(datatype_ptr->max_contig_blocks * sizeof(MPI_Aint)); - MPI_Aint dt_array_len = datatype_ptr->max_contig_blocks; - int rc = MPIR_Type_flatten(*ptr, dt_offset_array, dt_size_array, &dt_array_len); - - /* Build the PAMI datatype adding one contiguous chunk at a time with the PAMI_Type_add_simple - * interface. - */ - int i; - - for (i=0;i<dt_array_len;i++) { - size_t num_bytes_this_entry = dt_size_array[i]; - size_t cursor_offset; - if (i == 0) - cursor_offset = (size_t) dt_offset_array[i]; - else - cursor_offset = (size_t) dt_offset_array[i] - (size_t)dt_offset_array[i-1]; - pami_dtop_result = PAMI_Type_add_simple (*(pami_type_t*)(datatype_ptr->device_datatype), num_bytes_this_entry, cursor_offset, 1, 0); - MPIR_Assert(pami_dtop_result == PAMI_SUCCESS); - } - - /* Complete the PAMI datatype and free arrays. - */ - pami_dtop_result = PAMI_Type_complete (*(pami_type_t*)(datatype_ptr->device_datatype),1); - MPIR_Assert(pami_dtop_result == PAMI_SUCCESS); - MPL_free(dt_offset_array); - MPL_free(dt_size_array); - } - return; -} diff --git a/src/mpid/pamid/src/mpidi_util.c b/src/mpid/pamid/src/mpidi_util.c deleted file mode 100644 index ec88346..0000000 --- a/src/mpid/pamid/src/mpidi_util.c +++ /dev/null @@ -1,2057 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/mpid_util.c - * \brief Normal job startup code - */ - - -/* - * \brief - */ -#include <dlfcn.h> -#include <sys/time.h> -#include <sys/types.h> -#include <sys/stat.h> -#include <errno.h> -#include <limits.h> -#include <nl_types.h> -#include <mpidimpl.h> -#include "mpidi_util.h" - -#define PAMI_TUNE_MAX_ITER 2000 -/* Short hand for sizes */ -#define ONE (1) -#define ONEK (1<<10) -#define ONEM (1<<20) -#define ONEG (1<<30) - -#define PAMI_ASYNC_EXT_ATTR 2000 - -#if CUDA_AWARE_SUPPORT -void * pamidCudaPtr = NULL; -#endif -#if (MPIDI_PRINTENV || MPIDI_STATISTICS || MPIDI_BANNER) -MPIDI_printenv_t *mpich_env=NULL; -extern char* mp_euilib; -char mp_euidevice[20]; -extern pami_extension_t pe_extension; -extern pamix_extension_info_t PAMIX_Extensions; - typedef enum { - /* Attribute usage : type : default : description */ - PAMI_ASYNC_ATTR - = PAMI_ASYNC_EXT_ATTR, - PAMI_CONTEXT_TIMER_INTERVAL, /**< U: size_t : N/A : current timer interval in PAMI context */ - } pamix_attribute_name_async_t; - - -void MPIDI_Set_mpich_env(int rank, int size) { - static char polling_buf[32]=""; - int rc; - pami_configuration_t config; - - mpich_env->this_task = rank; - mpich_env->nprocs = size; - mpich_env->eager_limit=MPIDI_Process.pt2pt.limits.application.eager.remote; - mpich_env->use_token_flow_control=MPIDI_Process.is_token_flow_control_on; - mpich_env->mp_statistics=MPIDI_Process.mp_statistics; - if (mpich_env->polling_interval == 0) { - memset(&config, 0, sizeof(config)); - config.name = (pami_attribute_name_t)PAMI_CONTEXT_TIMER_INTERVAL; - rc= PAMI_Context_query(MPIDI_Context[0], &config, 1); - mpich_env->polling_interval = config.value.intval;; - sprintf(polling_buf, "MP_POLLING_INTERVAL=%d", - mpich_env->polling_interval); /* microseconds */ - rc = putenv(polling_buf); - } - if (mpich_env->retransmit_interval == 0) { - memset(&config, 0, sizeof(config)); - config.name = (pami_attribute_name_ext_t)PAMI_CONTEXT_RETRANSMIT_INTERVAL; - rc= PAMI_Context_query(MPIDI_Context[0], &config, 1); - mpich_env->retransmit_interval = config.value.intval; - sprintf(polling_buf, "MP_RETRANSMIT_INTERVAL=%d", - mpich_env->retransmit_interval); /* microseconds */ - rc = putenv(polling_buf); - } - mpich_env->buffer_mem=MPIDI_Process.mp_buf_mem; - mpich_env->buffer_mem_max=MPIDI_Process.mp_buf_mem_max; -} - - - -void MPIDI_Setup_networkenv() -{ - char *network; - char delimiter; - int rc; - pami_configuration_t config; - - - delimiter=':'; - mpich_env->instances = 1; - mpich_env->strip_type = NO_STRIPING; - mpich_env->transport_type = IS_IP; - mpich_env->network_string = NULL; - -/* - * Always try the MPI environment variables first - * then the PAMI env variable - * then query the PAMI network environment string - */ - if ((network = getenv("MP_CHILD_INET_ADDR")) != NULL ) { - mpich_env->transport_type = IS_IP; - } else if ((network = getenv("MP_MPI_NETWORK")) != NULL) { - mpich_env->transport_type = IS_US; - } else { - network=NULL; - memset(&config,0, sizeof(config)); - config.name = (pami_attribute_name_t)PAMI_CONTEXT_NETWORK; - rc= PAMI_Context_query(MPIDI_Context[0], &config, 1); - network=(char *) config.value.chararray; - if ((rc != PAMI_SUCCESS) || (network== NULL)) { - TRACE_ERR("PAMI network query returns %d", rc); - network=NULL; - return; - } else { - if ( strchr(network, ':') || - strchr(network, '.')) { - mpich_env->transport_type = IS_IP; - } else { - mpich_env->transport_type = IS_US; - } - } - } - - if ( NULL != network ) { - mpich_env->network_string = (char *) MPL_malloc(strlen(network)+1); - if( NULL != mpich_env->network_string ) { - memcpy(mpich_env->network_string,network,strlen(network)+1); - } - mpich_env->instances = atoi(network+1); - if (mpich_env->instances > 1 ) { - mpich_env->strip_type = IS_PACKET_STRIPING; - } - } - /* - * These should match the PAMI defaults. They are provided so that the - * mpi_printenv variable can print them out. - */ - if ( mpich_env->retransmit_interval == 0 ) { - memset(&config, 0, sizeof(config)); - config.name = (pami_attribute_name_t)PAMI_CONTEXT_RETRANSMIT_INTERVAL; - rc= PAMI_Context_query(MPIDI_Context[0], &config, 1); - mpich_env->retransmit_interval = config.value.intval; - } - -} - - -int MPIDI_Update_mpenv() -{ - char *buf; - int rc=0; - int i, val; /* for loop index */ - long bdfrsize; - int SendBufSize; - int CopySendBufSize; - char *cp,*p; - int infolevel, max_pkt_size; - uint tmp_val; - size_t bulkXfer=0; - pami_configuration_t *pami_config,config; - - MPIDI_Setup_networkenv(); - memset(&config,0, sizeof(config)); - config.name = PAMI_DISPATCH_SEND_IMMEDIATE_MAX; - rc = PAMI_Dispatch_query(MPIDI_Context[0], (size_t)0, &config, 1); - if ( rc == PAMI_SUCCESS ) { - TRACE_ERR("PAMI_DISPATCH_SEND_IMMEDIATE_MAX=%d.\n", config.value.intval, rc); - SendBufSize = config.value.intval; - } else { - TRACE_ERR((" Attention: PAMI_Client_query(DISPATCH_SEND_IMMEDIATE_MAX=%d) rc=%d\n", - config.name, rc)); - SendBufSize = 256; /* This is PAMI SendBufSize. */ - } - CopySendBufSize = SendBufSize - sizeof(MPIDI_MsgInfo); - mpich_env->copySendBufSize = CopySendBufSize; - - pami_config = (pami_configuration_t *) MPL_malloc(10 * sizeof(pami_configuration_t)); - memset((void *) pami_config,0, sizeof(pami_config)); - pami_config[0].name = (pami_attribute_name_t)PAMI_CONTEXT_MAX_PKT_SZ; - pami_config[1].name = (pami_attribute_name_t)PAMI_CONTEXT_RFIFO_SZ; - pami_config[2].name = (pami_attribute_name_t)PAMI_CONTEXT_SHM_ENABLED; - pami_config[3].name = (pami_attribute_name_t)PAMI_CONTEXT_BULK_XFER; - pami_config[4].name = (pami_attribute_name_t)PAMI_CONTEXT_BULK_MIN_MSG_SZ; - rc= PAMI_Context_query(MPIDI_Context[0], pami_config, 5); - if (rc != PAMI_SUCCESS) { - printf("ERROR PAMI_Context_query() failed rc=%d\n",rc); fflush(stdout); - return rc; - } - max_pkt_size = pami_config[0].value.intval; - mpich_env->recv_fifo_sz = pami_config[1].value.intval; - mpich_env->max_pkt_size = max_pkt_size; - mpich_env->max_pkts_out = 31; - mpich_env->use_shmem = pami_config[2].value.intval; - bulkXfer=pami_config[3].value.intval; - if (bulkXfer) { - mpich_env->use_rdma=PAMI_HINT_ENABLE; - } else { - /* No bulk xfer in PAMI */ - if (mpich_env->this_task == 0) { - TRACE_ERR("MPI-PAMI: Attention: No bulk xfer for the job\n"); - } - mpich_env->use_rdma=PAMI_HINT_DISABLE; /* in sync with PAMI */ - } - /* obtain minimum message size for bulk xfer (rdma) */ - mpich_env->rdma_min_msg_size = (long) pami_config[4].value.intval; - MPL_free(pami_config); -} - - -/* Print Environment */ -/* _mpidi_printenv collects the relevant MPI and internal variables to task 0 */ -/* and then prints out the differences */ -/* Returns: 0 - OK; possibly some ATTENTION messages issued; */ -/* 1 - ERROR; user must call exit(1), or risk the application hanging */ - - - - -#define MATCHI(x,y) { \ - first = 0; last = task_count-1; cflag = ' '; \ - for (i=1;i<=task_count;i++) { \ - if ((i==task_count) || (gatherer[i].x != gatherer[first].x)) { \ - last = i-1; \ - if (last != task_count-1) cflag = '*'; \ - printf("Task %1c%4d-%4d:%s %d\n", cflag,first, last, y, \ - gatherer[first].x); \ - first = i; \ - } \ - } \ - } -#define MATCHLL(x,y) { \ - first = 0; last = task_count-1; cflag = ' '; \ - for (i=1;i<=task_count;i++) { \ - if ((i==task_count) || (gatherer[i].x != gatherer[first].x)) { \ - last = i-1; \ - if (last != task_count-1) cflag = '*'; \ - printf("Task %1c%4d-%4d:%s %lld\n", cflag,first, last, \ - y, gatherer[first].x); \ - first = i; \ - } \ - } \ - } -#define MATCHD(x,y) { \ - first = 0; last = task_count-1; cflag = ' '; \ - for (i=1;i<=task_count;i++) { \ - if ((i==task_count) || (gatherer[i].x != gatherer[first].x)) { \ - last = i-1; \ - if (last != task_count-1) cflag = '*'; \ - printf("Task %1c%4d-%4d:%s %lf\n", cflag, first, last, \ - y, gatherer[first].x); \ - first = i; \ - } \ - } \ - } - -#define MATCHC(x,y,m) { \ - first = 0; last = task_count-1; cflag = ' '; \ - gatherer[first].x[m-1]='\0'; \ - for (i=1;i<=task_count;i++) { \ - if (i<task_count) gatherer[i].x[m-1]='\0'; \ - if ((i==task_count) || strncmp(gatherer[i].x, gatherer[first].x, m)) {\ - last = i-1; \ - if (last != task_count-1) cflag = '*'; \ - printf("Task %1c%4d-%4d:%s %s\n", cflag, first, last, y, \ - gatherer[first].x); \ - first = i; \ - } \ - } \ - } -#define MATCHB(x,y) { \ - first = 0; last = task_count-1; cflag = ' '; \ - for (i=1;i<=task_count;i++) { \ - if ((i==task_count) || (gatherer[i].x != gatherer[first].x)) { \ - last = i-1; \ - if (last != task_count-1) cflag = '*'; \ - if (gatherer[first].x > 0) { \ - printf("Task %1c%4d-%4d:%s %s\n", cflag, first, last, \ - y, "YES"); \ - } else if (gatherer[first].x == 0) { \ - printf("Task %1c%4d-%4d:%s %s\n", cflag, first, \ - last, y, "NO"); \ - } else { \ - printf("Task %1c%4d-%4d:%s %d\n", cflag, first, last, \ - y, gatherer[first].x); \ - } \ - first = i; \ - } \ - } \ - } -/* A modification on MATCHB, it is used to check the environment variables - * that take one of YES, NO or CONFIRM, This is used in particular for checking the value - * passed to MP_SINGLE_THREAD */ -#define MATCHT(x,y) { \ - first = 0; last = task_count-1; cflag = ' '; \ - for (i=1;i<=task_count;i++) { \ - if ((i==task_count) || (gatherer[i].x != gatherer[first].x)) { \ - last = i-1; \ - if (last != task_count-1) cflag = '*'; \ - if (gatherer[first].x == 0) { \ - printf("Task %1c%4d-%4d:%s %s\n", cflag, first, last, y, \ - "NO"); \ - } else if (gatherer[first].x == 1) { \ - printf("Task %1c%4d-%4d:%s %s\n", cflag, first, last, \ - y, "YES"); \ - } else { \ - printf("Task %1c%4d-%4d:%s %d\n", cflag, first, last, y, \ - gatherer[first].x); \ - } \ - first = i; \ - } \ - } \ - } - -#define NOTAPPLICABLE(y) { \ - printf("Task %4d-%4d:%s NA\n", 0, task_count-1, y); \ - } - - - -/* - * Parse Network string value - * Format = @instance_number:netAddr,adapter_type:..... - * or - * Format = @instance_number;netAddr,adapter_type;..... - * netAddr is for either Inet address (IP) or window ID (US) - * adapter is for adapter type - * - * Newwork_ID is exported by POE - * - * This function will fill in the given buffers as much as possible - * will start to drop the (whole) info when the buffers are getting - * full. - */ -void MPIDI_Get_net_param(char *cp, int MPI_only, int my_taskid, int total_task, - char *netAddr, long addrSize, - char *adapter, long adptSize, - char *networkId, long networkIdSize) -{ - /* parse Network string, works with either user space or IP */ - /* removing the window numbers but keeping the device names */ - char *cp1, *cp2, *cp3, *dummy1, *dummy2, *dummy3; - long nc, addrSpace=addrSize, addrLen=0, adptSpace=adptSize, adptLen=0; - int taskid, inst, i, n_comma; - char delimiter = ':'; - char msgBuf[80]; - - *netAddr = '\0'; /* Make sure strcat starts */ - *adapter = '\0'; /* from the beginning. */ - - if ( strchr(cp, ';') ) { - delimiter = ';'; - } - - - if ( *cp == '@' ) { - /* PE 4.1: MP_MPI_NETWORK = @inst:addr,adapter:...,... */ - /* Skip the instance number and get to the first colon */ - cp1 = strchr(cp,delimiter); - - while (cp1) { - cp1++; /* Next address */ - cp2 = strchr(cp1,',') + 1; /* Address ends at a comma */ - nc = cp2 - cp1; /* Add 1 to pick up the comma */ - if ( nc <= addrSpace ) { /* Yes test nc <= addrSpace */ - strncat(netAddr, cp1, nc); /* We will take care of the */ - addrLen += nc; /* last 'comma' later. */ - addrSpace -= nc; - } - else - break; - if ( (cp1 = strchr(cp2,delimiter)) != NULL ) { /* Adapter ends at a colon */ - nc = cp1 - cp2 + 1; /* Add 1 to pick up colon */ - if ( nc < adptSpace ) { - strncat(adapter, cp2, nc); - adptLen += nc; - adptSpace -= nc; - } - else { - strncat(adapter, cp2, adptSpace-1); - adptLen += (adptSpace-1); - break; - } - } - else { - /* Last one */ - nc = strlen(cp2); - if ( nc < adptSpace ) { - strcat(adapter, cp2); - adptLen += nc; - } - else { - strncat(adapter, cp2, adptSpace-1); - adptLen += (adptSpace-1); - } - } - } - /* Terminate netAddr and adapter with NULL */ - if ( addrLen > 1 ) netAddr[addrLen-1] = '\0'; /* get rid of the last comma */ - if ( adptLen > 0 ) { - if ( adapter[adptLen-1] == delimiter ) adapter[adptLen-1] = '\0'; - else adapter[adptLen] = '\0'; - } - } else { - /* MP_MPI_NETWORK = window_number:device_number */ - cp1 = strchr(cp,':'); - nc = cp1 - cp; /* Don't include ':' */ - strncpy(netAddr, cp, nc); - netAddr[nc] = '\0'; - strcpy(adapter, cp1+1); - } -} - - -int MPIDI_Print_mpenv(int rank,int size) -{ - MPI_Comm comm = MPI_COMM_WORLD; - MPIR_Comm *comm_ptr; - MPIDI_printenv_t sender; - MPIDI_printenv_t *gatherer = NULL; - int mytask; - int task_count; - MPIDI_TransportType transport_type; - MPIDI_StripingType striping_type; - int instances; - int is_mpi=1; - char *cp; - char *cp1; - int i, j, k; - int first, last; - int len, tmp_val; - char *userbuf = NULL; - int *recvcount = NULL, *recvdispl = NULL; - char **recvwork = NULL; - int max_output_size = 0; - FILE *fd; - char *ts; - char *ck; - int ssi = 0; - int current_size; - int increment; - int m, m1; - char cflag; - int rc; - pami_configuration_t *pami_config, config; - char *p; - char popenstr[]={ "/bin/ksh -c " }; - char *popenptr; - char tempstr[128]; - int mpi_errno; - int errflag=0; - - MPIDI_Set_mpich_env(rank,size); - memset(&sender,0,sizeof(MPIDI_printenv_t)); - MPIDI_Update_mpenv(); - /* decoding strings */ - char *clock_string[] = { -#ifndef __LINUX__ - "AIX", -#else - "OS", -#endif - "Switch" }; - char *develop_string[] = { - "Minimum", - "Normal", - "Debug", - "Develop" }; - char *transport_type_string[] = { - "ip", - "us" }; - char *striping_type_set[] = { - "Striping off", - "Packet striping on" }; - - /* Get MPI Library Version */ - MPIDI_Banner((char*)&sender.version); - /* Remove the trailing newline character */ - for (i=strlen(sender.version);i>0;i--) { - if (sender.version[i-1] == '\n') sender.version[i-1] = '\0'; } - - gethostname(sender.hostname, MAXHOSTNAMELEN+1); - /* Get task geometry information */ - cp = getenv("MP_PARTITION"); /* MP_PARTITION is always there */ - sender.partition = atol(cp); /* Checking is not needed. */ - sender.pid = getpid(); - mytask = mpich_env->this_task; - sender.nprocs = mpich_env->nprocs; - sender.single_thread = mpich_env->single_thread; - sender.max_pkts_out = mpich_env->max_pkts_out; - sender.recv_fifo_sz = mpich_env->recv_fifo_sz; - task_count = sender.nprocs; - strcpy(sender.wait_mode,"NOT SET"); /* will be updated in the future */ - strcpy(sender.clock_source,"OS"); - cp = getenv("MP_NODES"); - if (cp) strncpy(sender.nnodes, cp, 8); - else strcpy(sender.nnodes, "NOT SET"); - cp = getenv("MP_TASKS_PER_NODE"); - if (cp) strncpy(sender.tasks_node, cp, 8); - else strcpy(sender.tasks_node, "NOT SET"); - - if(cp=getenv("MP_EUIDEVELOP")) { - strcpy((void *) &sender.develop,(void *)cp); - } else - memcpy(&sender.develop,"NOT SET ",8); - - if(cp=getenv("MP_EUIDEVICE")) { - strcpy((void *) &mp_euidevice,(void *)cp); - } else - memcpy(&mp_euidevice,"NOT SET ",8); - strncpy(sender.device, mp_euidevice, 8); - - /* Get protocol info */ - transport_type = mpich_env->transport_type; - switch (transport_type) - { - case IS_IP: - strcpy(sender.euilib, transport_type_string[transport_type]); - cp = NULL; - cp = (char *) mpich_env->network_string; - if (cp) { - strcpy(sender.window_id, "NA"); - MPIDI_Get_net_param(cp, is_mpi, mytask, task_count, - tempstr, sizeof(tempstr), - sender.adapter_type, sizeof(sender.adapter_type), - sender.network_id, sizeof(sender.network_id)); - /* Override network_id when network device is en0 or ml0 */ - if ( !strcasecmp(sender.device, "en0") || - !strcasecmp(sender.device, "ml0") ) { - strcpy(sender.network_id, "NA"); - } - } else { - strcpy(sender.window_id, "NOT_SET"); - strcpy(sender.adapter_type, "NOT SET"); - strcpy(sender.network_id, "NOT SET"); - } - break; - case IS_US: - strcpy(sender.euilib, transport_type_string[transport_type]); - cp = NULL; - cp = (char *) mpich_env->network_string; - if (cp) { - MPIDI_Get_net_param(cp, is_mpi, mytask, task_count, - sender.window_id, sizeof(sender.window_id), - sender.adapter_type, sizeof(sender.adapter_type), - sender.network_id, sizeof(sender.network_id)); - } else { - strcpy(sender.window_id, "NOT SET"); - strcpy(sender.adapter_type, "NOT SET"); - strcpy(sender.network_id, "NOT SET"); - } - break; - default: - sprintf(tempstr, "_mpi_printenv: Get unknown transport type [%d]!\n", transport_type); - TRACE_ERR("%s \n",tempstr); - strcpy(sender.euilib, "UNKNOWN"); - strcpy(sender.window_id, "UNKNOWN"); - strcpy(sender.adapter_type, "UNKNOWN"); - strcpy(sender.network_id, "UNKNOWN"); - break; - } - /* Get run time setup */ - cp = getenv("MP_MSG_API"); - if (cp) strncpy(sender.protocol, cp, 16); - else strcpy(sender.protocol, "MPI"); - cp = getenv("LIBPATH"); - if (cp) strncpy(sender.libpath, cp, MAXPATHLEN-1); - else strcpy(sender.libpath, "NOT SET"); - getcwd(sender.directory, FILENAME_MAX+1); - - if(sizeof(void*) == 4) - sender.mode_64 = 0; - else - sender.mode_64 = 1; - - sender.threaded = 1; /* Always 1 */ - cp = getenv("AIXTHREAD_SCOPE"); - if (cp) strncpy(sender.thread_scope, cp, 8); - else strcpy(sender.thread_scope,"NOT SET"); - - cp = getenv("MP_CPU_USE"); - if (cp) strncpy(sender.cpu_use, cp, 10); - else strcpy(sender.cpu_use, "NOT SET"); - cp = getenv("MP_ADAPTER_USE"); - if (cp) strncpy(sender.adapter_use, cp, 10); - else strcpy(sender.adapter_use, "NOT SET"); -#ifndef LINUX - cp = getenv("MP_PRIORITY"); - if (cp) strncpy(sender.priority, cp, 24); - else strcpy(sender.priority, "NOT SET"); -#else - strcpy(sender.priority, "NA"); -#endif - cp = getenv("MP_TIMEOUT"); - if (cp) { - sender.timeout = atoi(cp); - if(sender.timeout <= 0) { - sender.timeout = 150; /* default MP_TIMEOUT is 150s */ - } - } else { - sender.timeout = 150; - } - - sender.interrupts = mpich_env->interrupts; - sender.mp_statistics = mpich_env->mp_statistics; - sender.polling_interval = mpich_env->polling_interval; - sender.eager_limit = mpich_env->eager_limit; - sender.use_token_flow_control=MPIDI_Process.is_token_flow_control_on; - sender.retransmit_interval = mpich_env->retransmit_interval; - sender.buffer_mem = mpich_env->buffer_mem; - sender.buffer_mem_max = mpich_env->buffer_mem_max; - - /* Get shared memory */ - sender.shmem_pt2pt = MPIDI_Process.shmem_pt2pt; - sender.use_shmem = mpich_env->use_shmem; -#ifndef LINUX - cp = getenv("MEMORY_AFFINITY"); - if ( cp ) strncpy(sender.mem_affinity, cp, 8); - else strcpy(sender.mem_affinity, "NOT SET"); -#else - strcpy(sender.mem_affinity, "NA"); -#endif - sender.instances = mpich_env->instances; - strcpy(sender.striping_type, striping_type_set[(mpich_env->strip_type)]); - - - pami_config = (pami_configuration_t *) MPL_malloc(10 * sizeof(pami_configuration_t)); - memset((void *) pami_config,0, sizeof(pami_config)); - pami_config[0].name = (pami_attribute_name_t)PAMI_CONTEXT_ACK_THRESH; - pami_config[1].name = (pami_attribute_name_t)PAMI_CONTEXT_REXMIT_BUF_CNT; - pami_config[2].name = (pami_attribute_name_t)PAMI_CONTEXT_REXMIT_BUF_SZ; - pami_config[3].name = (pami_attribute_name_t)PAMI_CONTEXT_RC_MAX_QP; - pami_config[4].name = (pami_attribute_name_t)PAMI_CONTEXT_RC_USE_LMC; - rc= PAMI_Context_query(MPIDI_Context[0], pami_config, 5); - if (rc != PAMI_SUCCESS) { - printf("mp_printenv ERROR PAMI_Context_query() failed rc=%d\n",rc); fflush(stdout); - return rc; - } - sender.ack_thresh = pami_config[0].value.intval; - sender.rexmit_buf_cnt = pami_config[1].value.intval; - sender.rexmit_buf_size = pami_config[2].value.intval; - sender.rc_max_qp = pami_config[3].value.intval; - if (pami_config[4].value.intval) - strcpy(sender.rc_qp_use_lmc, "YES"); - else - strcpy(sender.rc_qp_use_lmc, "NO"); - MPL_free(pami_config); - - /* - Begin Bulk transfer/RDMA & IB settings */ - if ( mpich_env->use_rdma == PAMI_HINT_DISABLE ) - strcpy(sender.use_bulk_xfer, "NO"); - else - strcpy(sender.use_bulk_xfer, "YES"); - - sprintf(sender.bulk_min_msg_size, "%d", mpich_env->bulk_min_msg_size); - - - /* M28,M32 -- End Bulk transfer/RDMA & IB settings */ - - if ( cp = getenv("MP_DEBUG_NOTIMEOUT") ) - strcpy(sender.debug_notimeout, cp); - else - strcpy(sender.debug_notimeout, "NOT_SET"); - - sender.statistics = mpich_env->statistics; - cp = getenv("MP_STDINMODE"); - if ( cp ) strncpy(sender.stdinmode, cp, 10); - else strcpy(sender.stdinmode, "NOT SET"); - cp = getenv("MP_STDOUTMODE"); - if ( cp ) strncpy(sender.stdoutmode, cp, 10); - else strcpy(sender.stdoutmode, "NOT SET"); - - - /* Check if there are any environment variables of the form MP_S_, and count them */ - { - extern char **environ; - char **envp = environ; - int env_count = 0; - while (*envp) { - if (!strncmp(*envp, "MP_S_",5)) env_count++; - if (!strncmp(*envp, "MP_CCL_",7)) env_count++; /* MP_CCL_TIMING is also service */ - envp++; - } - sender.service_variables = env_count; - } - - if (mytask == 0) { /* allocate a receive buffer for the gather of the base structure */ - gatherer = (MPIDI_printenv_t*) MPL_malloc(task_count * sizeof(MPIDI_printenv_t)); - if (!gatherer) { - TRACE_ERR("_mpi_printenv gatherer MPL_malloc failed rc=%d\n",rc); - return 1; - } - memset(gatherer,0,task_count*sizeof(MPIDI_printenv_t)); - } - - - mpi_errno = MPI_SUCCESS; - MPIR_Comm_get_ptr( comm, comm_ptr ); - mpi_errno = MPIR_Gather_impl(&sender, sizeof(MPIDI_printenv_t), MPI_BYTE, gatherer, - sizeof(MPIDI_printenv_t),MPI_BYTE, 0,comm_ptr, - (int *) &errflag); - if (mpi_errno) { - /* for communication errors, just record the error but continue */ - errflag = TRUE; - } - - /* work through results and compare */ - - if (mytask == 0) { - MATCHC(version,"Library:",128); - MATCHC(hostname,"Hostname:",MAXHOSTNAMELEN+1); - MATCHI(partition,"Job ID (MP_PARTITION):"); - MATCHI(nprocs,"Number of Tasks (MP_PROCS):"); - MATCHC(nnodes,"Number of Nodes (MP_NODES):",8); - MATCHC(tasks_node,"Number of Tasks per Node (MP_TASKS_PER_NODE):",8); - MATCHC(euilib,"Library Specifier (MP_EUILIB):",8); - MATCHC(adapter_type,"Adapter Name:",32); - MATCHC(window_id,"Window ID:",128); - if ( getenv("MP_S_STRIPING_DBG") ) MATCHC(network_id,"Network ID:",32); - MATCHC(device,"Device Name (MP_EUIDEVICE):",8); -#ifndef LINUX /* M27 */ - MATCHI(instances,"Window Instances (MP_INSTANCES * # of networks):"); - MATCHC(striping_type,"Striping Setup:",40); -#endif - MATCHC(protocol,"Protocols in Use (MP_MSG_API):",16); - MATCHC(libpath,"Effective Libpath (LIBPATH):",MAXPATHLEN); - MATCHC(directory,"Current Directory:",FILENAME_MAX+1); - MATCHB(mode_64,"64 Bit Mode:"); -#ifndef LINUX /* M27 */ - MATCHB(threaded,"Threaded Library:"); - MATCHC(thread_scope,"Requested Thread Scope (AIXTHREAD_SCOPE):",8); -#endif - MATCHC(cpu_use,"CPU Use (MP_CPU_USE):",10); - MATCHC(adapter_use,"Adapter Use (MP_ADAPTER_USE):",10); - MATCHC(clock_source,"Clock Source (MP_CLOCK_SOURCE):",8); - MATCHC(priority,"Priority Class (MP_PRIORITY):",24); - MATCHI(timeout,"Connection Timeout (MP_TIMEOUT/sec):"); - MATCHB(interrupts,"Adapter Interrupts Enabled (MP_CSS_INTERRUPT):"); - MATCHI(polling_interval,"Polling Interval (MP_POLLING_INTERVAL/usec):"); - MATCHI(buffer_mem,"Buffer Memory (MP_BUFFER_MEM/Bytes):"); - MATCHLL(buffer_mem_max,"Max. Buffer Memory (MP_BUFFER_MEM_MAX/Bytes):"); - MATCHI(eager_limit,"Message Eager Limit (MP_EAGER_LIMIT/Bytes):"); - MATCHI(use_token_flow_control,"Use token flow control:"); - MATCHC(wait_mode,"Message Wait Mode(MP_WAIT_MODE):",8); - MATCHI(retransmit_interval,"Retransmit Interval (MP_RETRANSMIT_INTERVAL/count):"); - MATCHB(use_shmem,"Shared Memory Enabled (MP_SHARED_MEMORY):"); - MATCHB(shmem_pt2pt,"Intranode pt2pt Shared Memory Enabled (MP_SHMEM_PT2PT):"); - MATCHB(mp_statistics,"Statistics Collection Enabled (MP_STATISTICS):"); - MATCHC(mem_affinity,"MEMORY_AFFINITY:",8); - MATCHT(single_thread,"Single Thread Usage(MP_SINGLE_THREAD):"); - MATCHI(recv_fifo_sz,"DMA Receive FIFO Size (Bytes):"); - MATCHI(max_pkts_out,"Max outstanding packets:"); - MATCHC(develop,"Develop Mode (MP_EUIDEVELOP):",16); - MATCHC(stdinmode,"Standard Input Mode (MP_STDINMODE):",12); - MATCHC(stdoutmode,"Standard Output Mode (MP_STDOUTMODE):",12); - MATCHI(service_variables,"Number of Service Variables set (MP_S_*):"); - printf("--------------------End of MPI Environment Report-------------------------\n"); - fflush(stdout); - } /* task 0 specific stuff */ - - - if (mytask == 0) { - MPL_free(gatherer); - } - - return 0; -} - - /* ------------------------- */ - /* if (MP_STATISTICS==yes) */ - /* print statistics data */ - /* ------------------------- */ -void MPIDI_print_statistics() { - if ((MPIDI_Process.mp_statistics) || - (MPIDI_Process.mp_printenv)) { - if (MPIDI_Process.mp_statistics) { - MPIDI_Statistics_write(stdout); - if (mpid_statp) { - MPL_free(mpid_statp); - mpid_statp=NULL; - } - } - if (MPIDI_Process.mp_printenv) { - if (mpich_env) { - MPL_free(mpich_env); - mpich_env=NULL; - } - } - } -} - -#endif /* MPIDI_PRINTENV || MPIDI_STATISTICS */ - -/** - * \brief validate whether a lpid is in a given group - * - * Searches the group lpid list for a match. - * - * \param[in] lpid World rank of the node in question - * \param[in] grp Group to validate against - * \return TRUE is lpid is in group - */ - -int MPIDI_valid_group_rank(int lpid, MPIR_Group *grp) { - int size = grp->size; - int z; - - for (z = 0; z < size && - lpid != grp->lrank_to_lpid[z].lpid; ++z); - return (z < size); -} - -/****************************************************************/ -/* function MPIDI_uppers converts a passed string to upper case */ -/****************************************************************/ -void MPIDI_toupper(char *s) -{ - int i; - if (s != NULL) { - for(i=0;i<strlen(s);i++) s[i] = toupper(s[i]); - } -} - -/* - ----------------------------------------------------------------- - Name: MPID_scan_str() - - Function: Scan a flag string for 2 out of 3 possible - characters (K, M, G). Return a 1 if neither - character is found otherwise return the character - along with a buffer containing the string without - the character. - value are valid. If they are valid, the - multiplication of the number and the units - will be returned as an unsigned int. If the - number and units are invalid, a 1 will be returned. - - Description: search string for character or end of string - if string contains either entered character - check which char it is, set multiplier - if no chars found, return error - - Parameters: A0 = MPIDI_scan_str(A1, A2, A3, A4, A5) - - A1 string to scan char * - A2 first char to scan for char - A3 second char to scan for char - A4 multiplier char * - A5 returned string char * - - A0 Return Code int - - - Return Codes: 0 OK - 1 input chars not found - ------------------------------------------------------------ -*/ -int MPIDI_scan_str(char *my_str, char fir_c, char sec_c, char *multiplier, char *tempbuf) -{ - int str_ptr; /*index counter into string*/ - int found; /*indicates whether one of input chars found*/ - int len_my_str; /*length of string with size and units*/ - - str_ptr = 0; /*start at beginning of string*/ - found = 0; /*no chars found yet*/ - - len_my_str = strlen(my_str); - - /* first check if all 'characters' of *my_str are digits, */ - /* str_ptr points to the first occurrence of a character */ - for (str_ptr=0; str_ptr<len_my_str; str_ptr++) { - if (str_ptr == 0) { /* there can be a '+' or a '-' in the first position */ - /* but I do not allow a negative value because there's */ - /* no negative amount of memory... */ - if (my_str[0] == '+') { - tempbuf[0] = my_str[0]; /* copy sign */ - /* this is ok but a digit MUST follow */ - str_ptr++; - /* If only a '+' was entered the next character is '\0'. */ - /* This is not a digit so the error message shows up */ - } - } - if (!isdigit(my_str[str_ptr])) { - break; - } - tempbuf[str_ptr] = my_str[str_ptr]; /* copy to return string */ - } /* endfor */ - - tempbuf[str_ptr] = 0x00; /* terminate return string, this was NOT done before this modification! */ - - if((my_str[str_ptr] == fir_c) || (my_str[str_ptr] == sec_c)) { - /*check which char it is, then set multiplier and indicate char found*/ - switch(my_str[str_ptr]) { - case 'K': - *multiplier = 'K'; - found++; - break; - case 'M': - *multiplier = 'M'; - found++; - break; - case 'G': - *multiplier = 'G'; - found++; - break; - } - /* my_str[str_ptr] = 0; */ /*change char in string to end of string char*/ - } - if (found == 0) { /*if input chars not found, indicate error*/ - return(1); } - else { - /* K, M or G should be the last character, something like 64M55 is invalid */ - if (str_ptr == len_my_str-1) { - return(0); /*if input chars found, return good status*/ - } else { - /* I only allow a 'B' to follow. This is not documented but reflects the */ - /* behaviour of earlier poe parsing. 64MB is valid, but after 'B' the */ - /* string must end */ - if (my_str[str_ptr+1] == 'B' && (str_ptr+1) == (len_my_str-1)) { - return(0); /*if input chars found, return good status*/ - } else { - return(1); - } /* endif */ - } /* endif */ - } -} -/* - ----------------------------------------------------------------- - Name: MPIDI_scan_str3() - - Function: Scan a flag string for 3 out of 3 possible - characters (K, M, G). Return a 1 if neither - character is found otherwise return the character - along with a buffer containing the string without - the character. - value are valid. If they are valid, the - multiplication of the number and the units - will be returned as an unsigned int. If the - number and units are invalid, a 1 will be returned. - - Description: search string for character or end of string - if string contains either entered character - check which char it is, set multiplier - if no chars found, return error - - Parameters: A0 = MPIDI_scan_str(A1, A2, A3, A4, A5, A6) - - A1 string to scan char * - A2 first char to scan for char - A3 second char to scan for char - A4 third char to scan for char - A5 multiplier char * - A6 returned string char * - - A0 Return Code int - - Return Codes: 0 OK - 1 input chars not found - ------------------------------------------------------------ -*/ -int MPIDI_scan_str3(char *my_str, char fir_c, char sec_c, char thr_c, char *multiplier, char *tempbuf) -{ - - int str_ptr; /*index counter into string*/ - int found; /*indicates whether one of input chars found*/ - int len_my_str; /*length of string with size and units*/ - - str_ptr = 0; /*start at beginning of string*/ - found = 0; /*no chars found yet*/ - - len_my_str = strlen(my_str); - - /* first check if all 'characters' of *my_str are digits, */ - /* str_ptr points to the first occurrence of a character */ - for (str_ptr=0; str_ptr<len_my_str; str_ptr++) { - if (str_ptr == 0) { /* there can be a '+' or a '-' in the first position */ - /* but I do not allow a negative value because there's */ - /* no negative amount of memory... */ - if (my_str[0] == '+') { - tempbuf[0] = my_str[0]; /* copy sign */ - /* this is ok but a digit MUST follow */ - str_ptr++; - /* If only a '+' was entered the next character is '\0'. */ - /* This is not a digit so the error message shows up */ - } - } - if (!isdigit(my_str[str_ptr])) { - break; - } - tempbuf[str_ptr] = my_str[str_ptr]; /* copy to return string */ - } /* endfor */ - - tempbuf[str_ptr] = 0x00; /* terminate return string, this was NOT done before this modification! */ - - if((my_str[str_ptr] == fir_c) || (my_str[str_ptr] == sec_c) || (my_str[str_ptr] == thr_c)) { - /*check which char it is, then set multiplier and indicate char found*/ - switch(my_str[str_ptr]) { - case 'K': - *multiplier = 'K'; - found++; - break; - case 'M': - *multiplier = 'M'; - found++; - break; - case 'G': - *multiplier = 'G'; - found++; - break; - } - /* my_str[str_ptr] = 0; */ /*change char in string to end of string char*/ - } - if (found == 0) { /*if input chars not found, indicate error*/ - return(1); } - else { - /* K, M or G should be the last character, something like 64M55 is invalid */ - if (str_ptr == len_my_str-1) { - return(0); /*if input chars found, return good status*/ - } else { - /* I only allow a 'B' to follow. This is not documented but reflects the */ - /* behaviour of earlier poe parsing. 64MB is valid, but after 'B' the */ - /* string must end */ - if (my_str[str_ptr+1] == 'B' && (str_ptr+1) == (len_my_str-1)) { - return(0); /*if input chars found, return good status*/ - } else { - return(1); - } /* endif */ - } /* endif */ - } -} - -/* - ----------------------------------------------------------------- - Name: MPIDI_checkit() - - Function: Determine whether a given number and units - value are valid. If they are valid, the - multiplication of the number and the units - will be returned as an unsigned int. If the - number and units are invalid, a 1 will be returned. - - Description: if units is G - if value is > 4 return error - else multiplier is 1G - else if units is M - if value is > 4K return error - else multiplier is 1M - else if units is K - if value is > 4M return error - else multiplier is 1K - if value < 1 return error - else - multiply value by multiplier - return result - - Parameters: A0 = MPIDI_checkit(A1, A2, A3) - - A1 given value int - A2 given units char * - A3 result unsigned int * - - A0 Return Code int - - Return Codes: 0 OK - 1 bad value - ------------------------------------------------------------ -*/ -int MPIDI_checkit(int myval, char myunits, unsigned int *mygoodval) -{ - int multiplier = ONE; /*units multiplier for entered value*/ - - if (myunits == 'G') { /*if units is G*/ - if (myval>4) return 1; /*entered value can't be greater than 4*/ - else multiplier = ONEG; /*if OK, mult value by units*/ - } - else if (myunits == 'M') { /*if units is M*/ - if (myval > (4*ONEK)) return 1; /*value can't be > 4096*/ - else multiplier = ONEM; /*if OK, mult value by units*/ - } - else if (myunits == 'K') { /*if units is K*/ - if (myval > (4*ONEM)) return 1; /*value can't be > 4M*/ - else multiplier = ONEK; /*if OK, mult value by units*/ - } - if (myval < 1) return 1; /*value can't be less than 1*/ - - *mygoodval = myval * multiplier; /*do multiplication*/ - return 0; /*good return*/ - -} - - - - /*************************************************************************** - Function Name: MPIDI_atoi - - Description: Convert a string into an interger. The string can be all - digits or includes symbols 'K', 'M'. - - Parameters: char * -- string to be converted - unsigned int * -- result val (caller to cast to int* or long*) - - Return: int 0 if AOK - number of errors. - ***************************************************************************/ -int MPIDI_atoi(char* str_in, unsigned int* val) -{ - char tempbuf[256]; - char size_mult; /* multiplier for size strings */ - int i, tempval; - int letter=0, retval=0; - - /***********************************/ - /* Check for letter */ - /***********************************/ - for (i=0; i<strlen(str_in); i++) { - if (!isdigit(str_in[i])) { - letter = 1; - break; - } - } - if (!letter) { /* only digits */ - errno = 0; /* should set errno to 0 before atoi() call */ - *val = atoi(str_in); - if (errno) { /* no check for negative integer, there's no '-' in the string */ - retval = errno; - } - } - else { - /***********************************/ - /* Check for K or M. */ - /***********************************/ - MPIDI_toupper(str_in); - retval = MPIDI_scan_str(str_in, 'M', 'K', &size_mult, tempbuf); - - if ( retval == 0) { - tempval = atoi(tempbuf); - - /***********************************/ - /* If 0 K or 0 M entered, set to 0 */ - /* otherwise, do conversion. */ - /***********************************/ - if (tempval != 0) - retval = MPIDI_checkit(tempval, size_mult, (unsigned int*)val); - else - *val = 0; - } - - if (retval == 0) { - tempval = atoi(tempbuf); - retval = MPIDI_checkit(tempval, size_mult, (unsigned int*)val); - } - else - *val = 0; - } - - return retval; -} - - /*************************************************************************** - Name: MPIDI_checkll() - - Function: Determine whether a given number and units - value are valid. If they are valid, the - multiplication of the number and the units - will be returned as an unsigned int. If the - number and units are invalid, a 1 will be returned. - - Description: if units is G - multiplier is 1G - else if units is M - multiplier is 1M - else if units is K - multiplier is 1K - else - return error - - multiply value by multiplier - return result - Parameters: A0 = MPIDI_checkll(A1, A2, A3) - - A1 given value int - A2 given units char * - A3 result long long * - - A0 Return Code int - - Return Codes: 0 OK - 1 bad value - ***************************************************************************/ - -int MPIDI_checkll(int myval, char myunits, long long *mygoodval) -{ - int multiplier; /* units multiplier for entered value */ - - if (myunits == 'G') { /* if units is G */ - multiplier = ONEG; - } - else if (myunits == 'M') { /* if units is M */ - multiplier = ONEM; - } - else if (myunits == 'K') { /* if units is K */ - multiplier = ONEK; - } - else - return 1; /* Unkonwn unit */ - - *mygoodval = (long long) myval * multiplier; /* do multiplication */ - return 0; /* good return */ -} - - -int MPIDI_atoll(char* str_in, long long* val) -{ - char tempbuf[256]; - char size_mult; /* multiplier for size strings */ - int i, tempval; - int letter=0, retval=0; - - /*****************************************/ - /* Check for letter, if none, MPIDI_atoi */ - /*****************************************/ - for (i=0; i<strlen(str_in); i++) { - if (!isdigit(str_in[i])) { - letter = 1; - break; - } - } - if (!letter) { /* only digits */ - errno = 0; - *val = atoll(str_in); - if (errno) { - retval = errno; - } - } - else { - /***********************************/ - /* Check for K or M. */ - /***********************************/ - MPIDI_toupper(str_in); - retval= MPIDI_scan_str3(str_in, 'G', 'M', 'K', &size_mult, tempbuf); - - if ( retval == 0) { - tempval = atoi(tempbuf); - - /***********************************/ - /* If 0 K or 0 M entered, set to 0 */ - /* otherwise, do conversion. */ - /***********************************/ - if (tempval != 0) - retval = MPIDI_checkll(tempval, size_mult, val); - else - *val = 0; - } - } - - return retval; -} - - - -/****************************************************/ -/* Collective Selection Benchmarking utils */ -/****************************************************/ - -const char * xfer_array_str[PAMI_XFER_COUNT]; -int task_id; -int num_tasks; - -static int MPIDI_collsel_print_usage() -{ - if(!task_id) - fputs("Usage: pami_tune (MPICH) [options]\n\ -Options:\n\ - -c Comma separated list of collectives to benchmark\n\ - Valid options are: \n\ - allgather, allgatherv_int, allreduce, alltoall,\n\ - alltoallv_int, ambroadcast, amgather, amreduce,\n\ - amscatter, barrier, broadcast, gather, gatherv_int,\n\ - reduce, reduce_scatter, scan, scatter, scatterv_int\n\ - (Default: all collectives)\n\n\ - -m Comma separated list of message sizes to benchmark\n\ - (Default: 1 to 2^k, where k <= 20)\n\n\ - -g Comma separated list of geometry sizes to benchmark\n\ - (Default: Powers of 2 (plus and minus one as well))\n\n\ - -i Number of benchmark iterations per algorithm\n\ - (Default: 1000. Maximum is 2000)\n\n\ - -f <file> Input INI file containing benchmark parameters\n\ - You can override a parameter with a command line argument\n\n\ - -o <file> Output XML file containing benchmark results\n\ - (Default: pami_tune_results.xml)\n\n\ - -d Diagnostics mode. Verify correctness of collective algorithms\n\ - (Default: Disabled)\n\n\ - -p Print the Collective Algorithm performance data to standard output\n\ - (Default: Disabled)\n\n\ - -h Print this help message\n\n", stdout); - - return 0; -} - - -static void MPIDI_collsel_init_advisor_params(advisor_params_t *params) -{ - params->num_collectives = 0; - params->num_procs_per_node = 0; - params->num_geometry_sizes = 0; - params->num_message_sizes = 0; - params->collectives = NULL; - params->procs_per_node = NULL; - params->geometry_sizes = NULL; - params->message_sizes = NULL; - /* Set the following to -1, so that we can - check if the user has set them or not */ - params->iter = -1; - params->verify = -1; - params->verbose = -1; - params->checkpoint = -1; -} - - -static void MPIDI_collsel_free_advisor_params(advisor_params_t *params) -{ - if(params->collectives) MPL_free(params->collectives); - if(params->procs_per_node) MPL_free(params->procs_per_node); - if(params->geometry_sizes) MPL_free(params->geometry_sizes); - if(params->message_sizes) MPL_free(params->message_sizes); -} - - -static void MPIDI_collsel_init_xfer_tables() -{ - xfer_array_str[PAMI_XFER_BROADCAST] ="broadcast"; - xfer_array_str[PAMI_XFER_ALLREDUCE] ="allreduce"; - xfer_array_str[PAMI_XFER_REDUCE] ="reduce"; - xfer_array_str[PAMI_XFER_ALLGATHER] ="allgather"; - xfer_array_str[PAMI_XFER_ALLGATHERV] ="allgatherv"; - xfer_array_str[PAMI_XFER_ALLGATHERV_INT]="allgatherv_int"; - xfer_array_str[PAMI_XFER_SCATTER] ="scatter"; - xfer_array_str[PAMI_XFER_SCATTERV] ="scatterv"; - xfer_array_str[PAMI_XFER_SCATTERV_INT] ="scatterv_int"; - xfer_array_str[PAMI_XFER_GATHER] ="gather"; - xfer_array_str[PAMI_XFER_GATHERV] ="gatherv"; - xfer_array_str[PAMI_XFER_GATHERV_INT] ="gatherv_int"; - xfer_array_str[PAMI_XFER_BARRIER] ="barrier"; - xfer_array_str[PAMI_XFER_ALLTOALL] ="alltoall"; - xfer_array_str[PAMI_XFER_ALLTOALLV] ="alltoallv"; - xfer_array_str[PAMI_XFER_ALLTOALLV_INT] ="alltoallv_int"; - xfer_array_str[PAMI_XFER_SCAN] ="scan"; - xfer_array_str[PAMI_XFER_REDUCE_SCATTER]="reduce_scatter"; - xfer_array_str[PAMI_XFER_AMBROADCAST] ="ambroadcast"; - xfer_array_str[PAMI_XFER_AMSCATTER] ="amscatter"; - xfer_array_str[PAMI_XFER_AMGATHER] ="amgather"; - xfer_array_str[PAMI_XFER_AMREDUCE] ="amreduce"; -} - -static int MPIDI_collsel_process_collectives(char *coll_arg, advisor_params_t *params) -{ - int i, ret = 0, arg_len = strlen(coll_arg); - char *collectives = (char *) MPL_malloc(arg_len + 1); - char *coll; - int infolevel = 0; -#if (MPIDI_STATISTICS || MPIDI_PRINTENV) - if(MPIDI_Process.mp_infolevel >= 1) - infolevel = 1; -#endif - /* if already set via config file, free it */ - if(params->collectives) - { - MPL_free(params->collectives); - params->num_collectives = 0; - } - /* Allocating some extra space should be fine */ - params->collectives = (pami_xfer_type_t *)MPL_malloc(sizeof(pami_xfer_type_t)*PAMI_XFER_COUNT); - - strcpy(collectives, coll_arg); - coll = strtok(collectives,","); - while (coll != NULL) - { - int invalid_collective = 0; - for(i=0; i<PAMI_XFER_COUNT; i++) - { - if(strcmp(coll, xfer_array_str[i]) == 0) - { - if(i == 4) - { - if(infolevel >= 1) - fprintf(stderr,"WARNING: MPICH (pami_tune) doesn't support tuning for ALLGATHERV. ALLGATHERV tuning will be skipped.\nTune for ALLGATHERV_INT instead\n"); - invalid_collective = 1; - break; - } - else if(i == 7) - { - if(infolevel >= 1) - fprintf(stderr,"WARNING: MPICH (pami_tune) doesn't support tuning for SCATTERV. SCATTERV tuning will be skipped.\nTune for SCATTERV_INT instead\n"); - invalid_collective = 1; - break; - } - else if(i == 10) - { - if(infolevel >= 1) - fprintf(stderr,"WARNING: MPICH (pami_tune) doesn't support tuning for GATHERV. GATHERV tuning will be skipped.\nTune for GATHERV_INT instead\n"); - invalid_collective = 1; - break; - } - else if(i == 14) - { - if(infolevel >= 1) - fprintf(stderr,"WARNING: MPICH (pami_tune) doesn't support tuning for ALLTOALLV. ALLTOALLV tuning will be skipped.\nTune for ALLTOALLV_INT instead\n"); - invalid_collective = 1; - break; - } - else - { - params->collectives[params->num_collectives++] = i; - break; - } - } - } - /* arg did not match any collective */ - if(i == PAMI_XFER_COUNT || invalid_collective) - { - if(!task_id) - { - fprintf(stderr, "Invalid collective: %s\n", coll); - } - break; - } - coll = strtok(NULL,","); - } - if(params->num_collectives == 0) - { - MPL_free(params->collectives); - params->collectives = NULL; - ret = 1; - } - - MPL_free(collectives); - return ret; -} - - -static int MPIDI_collsel_process_msg_sizes(char *msg_sizes_arg, advisor_params_t *params) -{ - int ret = 0, arg_len = strlen(msg_sizes_arg); - char *msg_sizes = (char *) MPL_malloc(arg_len + 1); - char *msg_sz; - size_t tmp; - size_t array_size = 50; - /* if already set via config file, free it */ - if(params->message_sizes) - { - MPL_free(params->message_sizes); - params->num_message_sizes = 0; - } - /* Allocating some extra space should be fine */ - params->message_sizes = (size_t *)MPL_malloc(sizeof(size_t) * array_size); - - strcpy(msg_sizes, msg_sizes_arg); - msg_sz = strtok(msg_sizes,","); - while (msg_sz != NULL) - { - tmp = strtol(msg_sz, NULL, 10); - if(tmp == 0) - { - MPL_free(params->message_sizes); - params->message_sizes = NULL; - if(!task_id) - { - fprintf(stderr, "Invalid message size: %s\n", msg_sz); - } - ret = 1; - break; - } - - if(params->num_message_sizes >= array_size) - { - array_size *= 2; - params->message_sizes = (size_t *) MPL_realloc(params->message_sizes, - sizeof(size_t) * array_size); - } - params->message_sizes[params->num_message_sizes++] = tmp; - msg_sz = strtok(NULL,","); - } - MPL_free(msg_sizes); - return ret; -} - - -static int MPIDI_collsel_process_geo_sizes(char *geo_sizes_arg, advisor_params_t *params) -{ - int ret = 0, arg_len = strlen(geo_sizes_arg); - char *geo_sizes = (char *) MPL_malloc(arg_len + 1); - char *geo_sz; - size_t tmp; - size_t array_size = 50; - /* if already set via config file, free it */ - if(params->geometry_sizes) - { - MPL_free(params->geometry_sizes); - params->num_geometry_sizes = 0; - } - /* Allocating some extra space should be fine */ - params->geometry_sizes = (size_t *)MPL_malloc(sizeof(size_t) * array_size); - - strcpy(geo_sizes, geo_sizes_arg); - geo_sz = strtok(geo_sizes,","); - while (geo_sz != NULL) - { - tmp = strtol(geo_sz, NULL, 10); - if(tmp < 2 || tmp > num_tasks) - { - MPL_free(params->geometry_sizes); - params->geometry_sizes = NULL; - if(!task_id) - { - fprintf(stderr, "Invalid geometry size: %s\n", geo_sz); - } - ret = 1; - break; - } - - if(params->num_geometry_sizes >= array_size) - { - array_size *= 2; - params->geometry_sizes = (size_t *) MPL_realloc(params->geometry_sizes, - sizeof(size_t) * array_size); - } - params->geometry_sizes[params->num_geometry_sizes++] = tmp; - geo_sz = strtok(NULL,","); - } - MPL_free(geo_sizes); - return ret; -} - - -static int MPIDI_collsel_process_output_file(char *filename, char **out_file) -{ - char *newname; - int i, filename_len, ret = 0; - struct stat ost; - - filename_len = strlen(filename); - - /* Check if file already exists */ - if(stat(filename, &ost) == 0) - { - if(!S_ISREG(ost.st_mode)) - { - fprintf(stderr, "Error: %s exists and is not a regular file\n", filename); - return 1; - } - fprintf(stderr, "File %s already exists, renaming existing file\n", filename); - - newname = (char *) MPL_malloc(filename_len + 5); - for (i = 0; i < 500; ++i) - { - sprintf(newname,"%s.%d", filename, i); - if(!(access(newname, F_OK) == 0)) - { - ret = rename(filename, newname); - break; - } - } - MPL_free(newname); - if(i == 500 || ret != 0) - { - fprintf(stderr, "Error renaming file\n"); - return 1; - } - } - /* if file name is already set via config file, free it */ - if(*out_file) MPL_free(*out_file); - *out_file = (char *)MPL_malloc(filename_len + 1); - strcpy(*out_file, filename); - - return ret; -} - - -static char* MPIDI_collsel_ltrim(char *line) -{ - while(*line && isspace(*line)) - ++line; - - return line; -} - -static char* MPIDI_collsel_rtrim(char *line) -{ - char *end = line + strlen(line); - while(end > line && isspace(*--end)) - *end = '\0'; - - return line; -} - -static int MPIDI_collsel_checkvalue(char *name, char *value, const char *filename, int *ival) -{ - int ret = 0; - char *tmp; - *ival = (int)strtol(value, &tmp, 10); - if(*ival != 1) - { - if((*ival == 0 && value == tmp)|| *ival != 0) - { - if(!task_id) - fprintf(stderr, "Invalid value for %s parameter: %s in file: %s\n", name, value, filename); - ret = 1; - } - } - return ret; -} - - -static int MPIDI_collsel_process_ini_file(const char *filename, advisor_params_t *params, char **out_file) -{ - char *line, *start, *name, *value; - int ret = 0; - struct stat ist; - - if(stat(filename, &ist) == 0) - { - if(!S_ISREG(ist.st_mode)) - { - fprintf(stderr, "Error: %s is not a regular file\n", filename); - return 1; - } - } - - FILE *file = fopen(filename, "r"); - if(!file) - { - fprintf(stderr, "Error: Cannot open file %s: %s\n", - filename, strerror(errno)); - return 1; - } - line = (char *) MPL_malloc(2000); - - while (fgets(line, 2000, file) != NULL) - { - start = MPIDI_collsel_ltrim(MPIDI_collsel_rtrim(line)); - /* Skip comments and sections */ - if(*start == ';' || *start == '[' || *start == '#') - continue; - name = strtok(line, "="); - if(name == NULL) continue; - value = strtok(NULL, "="); - if(value == NULL) continue; - name = MPIDI_collsel_rtrim(name); - value = MPIDI_collsel_ltrim(value); - /* Do not override command line values if they are set */ - if(strcmp(name, "collectives") == 0) - { - if(!params->collectives) - ret = MPIDI_collsel_process_collectives(value, params); - } - else if(strcmp(name, "message_sizes") == 0) - { - if(!params->message_sizes) - ret = MPIDI_collsel_process_msg_sizes(value, params); - } - else if(strcmp(name, "geometry_sizes") == 0) - { - if(!params->geometry_sizes) - ret = MPIDI_collsel_process_geo_sizes(value, params); - } - else if(strcmp(name, "output_file") == 0) - { - if(!*out_file && !task_id) /* Only task 0 creates o/p file */ - ret = MPIDI_collsel_process_output_file(value, out_file); - } - else if(strcmp(name, "iterations") == 0) - { - if(params->iter == -1) - { - params->iter = atoi(value); - if(params->iter <= 0 || params->iter > PAMI_TUNE_MAX_ITER) - { - if(!task_id) - fprintf(stderr, "Invalid iteration count: %s in file: %s\n", value, filename); - ret = 1; - } - } - } - else if(strcmp(name, "verbose") == 0) - { - if(params->verbose == -1) - ret = MPIDI_collsel_checkvalue(name, value, filename, &params->verbose); - } - else if(strcmp(name, "diagnostics") == 0) - { - if(params->verify == -1) - ret = MPIDI_collsel_checkvalue(name, value, filename, &params->verify); - } - /*else if(strcmp(name, "checkpoint") == 0) - { - if(params->checkpoint == -1) - ret = MPIDI_collsel_checkvalue(name, value, filename, &params->checkpoint); - }*/ - else - { - fprintf(stderr, "Invalid parameter: %s in file: %s\n", name, filename); - ret = 1; - } - if(ret) break; - } - MPL_free(line); - fclose(file); - - return ret; -} - - -static int MPIDI_collsel_process_arg(int argc, char *argv[], advisor_params_t *params, char ** out_file) -{ - int i,c,ret = 0; - char *in_file = NULL; - - MPIDI_collsel_init_xfer_tables(); - params->verify = 0; - - opterr = 0; - while ((c = getopt (argc, argv, "c:m:g:f:o:i:p::d::x::h::")) != -1) - { - switch (c) - { - case 'c': - ret = MPIDI_collsel_process_collectives(optarg, params); - break; - case 'm': - ret = MPIDI_collsel_process_msg_sizes(optarg, params); - break; - case 'g': - ret = MPIDI_collsel_process_geo_sizes(optarg, params); - break; - case 'f': - in_file = (char *) optarg; - break; - case 'o': - if(!task_id) /* Only task 0 creates o/p file */ - ret = MPIDI_collsel_process_output_file(optarg, out_file); - break; - case 'i': - params->iter = atoi(optarg); - if(params->iter <= 0 || params->iter > PAMI_TUNE_MAX_ITER) - { - if(!task_id) - fprintf(stderr, "Invalid iteration count %s\n", optarg); - ret = 1; - } - break; - case 'd': - params->verify = 1; - break; - case 'p': - params->verbose = 1; - break; - case 'x': - /*params->checkpoint = 1;*/ - break; - case 'h': - ret = 2; - break; - case '?': - if(!task_id) - { - if (optopt == 'c' || optopt == 'm' || optopt == 'g' || - optopt == 'f' || optopt == 'o' || optopt == 'i') - { - fprintf (stderr, "Option -%c requires an argument.\n", optopt); - } - else if (isprint (optopt)) - { - fprintf (stderr, "Unknown option `-%c'.\n", optopt); - } - else - { - fprintf (stderr, "Unknown option character `\\x%x'.\n", optopt); - } - } - ret = 1; - break; - default: - abort(); - break; - } - if(ret) return ret; - } - /* INI file is processed last as we don't wan't to override - * command line parameters. Also any invalid value in the INI - * file if overriden by a command line flag should not generate - * an error - */ - if(in_file != NULL) - { - ret = MPIDI_collsel_process_ini_file(in_file, params, out_file); - if(ret) return ret; - } - - if(!task_id) - { - if (optind < argc) - { - printf ("Non-option arguments: "); - while (optind < argc) - printf ("%s ", argv[optind++]); - printf ("\n"); - } - } - - /* If user did not specify any collectives, benchmark all */ - if(params->num_collectives == 0) - { - params->collectives = (pami_xfer_type_t *)MPL_malloc(sizeof(pami_xfer_type_t)*PAMI_XFER_COUNT); - for(i = 0; i < PAMI_XFER_COUNT; i++) - { - if(i == 4 || i == 7 || i == 10 || i == 14) - i++; - params->collectives[params->num_collectives++] = i; - } - } - /* If user did not set any of the following parameters, set default value */ - if(params->iter == -1) params->iter = 1000; - if(params->verbose == -1) params->verbose = 0; - if(params->verify == -1) params->verify = 0; - if(params->checkpoint == -1) params->checkpoint = 0; - return 0; -} - - -static void MPIDI_collsel_print_params(advisor_params_t *params, char *output_file) -{ - size_t i; - printf(" Benchmark Parameters\n"); - printf(" --------------------\n"); - - printf(" Collectives:\n "); - for(i=0; i<params->num_collectives; i++){ - printf(" %s", xfer_array_str[params->collectives[i]]); - } - printf("\n Geometry sizes:\n "); - for(i=0; i<params->num_geometry_sizes; i++){ - printf(" %zu", params->geometry_sizes[i]); - } - printf("\n Message sizes:\n "); - for(i=0; i<params->num_message_sizes; i++){ - printf(" %zu", params->message_sizes[i]); - } - printf("\n Iterations : %d\n", params->iter); - printf(" Output file : %s\n", output_file); -/* printf(" Checkpoint mode : %d\n", params->checkpoint); */ - printf(" Diagnostics mode : %d\n", params->verify); - printf(" Verbose mode : %d\n", params->verbose); -} - - -int MPIDI_collsel_pami_tune_parse_params(int argc, char ** argv) -{ - MPIDI_Collsel_output_file = NULL; - num_tasks = PAMIX_Client_query(MPIDI_Client, PAMI_CLIENT_NUM_TASKS).value.intval; - task_id = PAMIX_Client_query(MPIDI_Client, PAMI_CLIENT_TASK_ID ).value.intval; - MPIDI_collsel_init_advisor_params(&MPIDI_Collsel_advisor_params); - if(num_tasks < 2) - { - fprintf (stderr, "Error: pami_tune must be run with more than one task\n"); - fflush(stderr); - MPIDI_collsel_print_usage(); - return PAMI_ERROR; - } - MPIDI_Collsel_advisor_params.procs_per_node = (size_t*)MPL_malloc(1 * sizeof(size_t)); - MPIDI_Collsel_advisor_params.num_procs_per_node = 1; - MPIDI_Collsel_advisor_params.procs_per_node[0] = PAMIX_Client_query(MPIDI_Client, PAMI_CLIENT_NUM_LOCAL_TASKS ).value.intval; - - int result = MPIDI_collsel_process_arg(argc, argv, &MPIDI_Collsel_advisor_params, &MPIDI_Collsel_output_file); - if(result) - { - MPIDI_collsel_print_usage(); - return PAMI_ERROR; - } - - if(MPIDI_Collsel_output_file == NULL && !task_id) - { - if(MPIDI_collsel_process_output_file("pami_tune_results.xml", &MPIDI_Collsel_output_file)) - { - return PAMI_ERROR; - } - } - if(MPIDI_Collsel_advisor_params.verbose && !task_id) - MPIDI_collsel_print_params(&MPIDI_Collsel_advisor_params, MPIDI_Collsel_output_file); - - return PAMI_SUCCESS; - -} - -void MPIDI_collsel_pami_tune_cleanup() -{ - MPIDI_collsel_free_advisor_params(&MPIDI_Collsel_advisor_params); -} - - - -/**********************************************************/ -/* CUDA Utilities */ -/**********************************************************/ -#if CUDA_AWARE_SUPPORT -int CudaMemcpy(void* dst, const void* src, size_t count, int kind) -{ - return (*pamidCudaMemcpy)(dst, src, count, kind); -} -int CudaPointerGetAttributes(struct cudaPointerAttributes* attributes, const void* ptr) -{ - return (*pamidCudaPointerGetAttributes)(attributes, ptr); -} -const char* CudaGetErrorString( int error) -{ - return (*pamidCudaGetErrorString)(error); -} -#endif - -inline bool MPIDI_enable_cuda() -{ - bool result = false; -#if CUDA_AWARE_SUPPORT - pamidCudaPtr = dlopen("libcudart.so", RTLD_NOW|RTLD_GLOBAL); - if(pamidCudaPtr == NULL) - { - TRACE_ERR("failed to open libcudart.so error=%s\n", dlerror()); - return result; - } - else - { - pamidCudaMemcpy = (int (*)())dlsym(pamidCudaPtr, "cudaMemcpy"); - if(pamidCudaMemcpy == NULL) - { - dlclose(pamidCudaPtr); - return result; - } - pamidCudaPointerGetAttributes = (int (*)())dlsym(pamidCudaPtr, "cudaPointerGetAttributes"); - if(pamidCudaPointerGetAttributes == NULL) - { - dlclose(pamidCudaPtr); - return result; - } - pamidCudaGetErrorString = (const char* (*)())dlsym(pamidCudaPtr, "cudaGetErrorString"); - if(pamidCudaGetErrorString == NULL) - { - dlclose(pamidCudaPtr); - return result; - } - result = true; - } -#endif - return result; -} - -inline bool MPIDI_cuda_is_device_buf(const void* ptr) -{ - bool result = false; -#if CUDA_AWARE_SUPPORT - if(MPIDI_Process.cuda_aware_support_on) - { - if(ptr != MPI_IN_PLACE) - { - struct cudaPointerAttributes cuda_attr; - cudaError_t e= CudaPointerGetAttributes ( & cuda_attr, ptr); - - if (e != cudaSuccess) - result = false; - else if (cuda_attr.memoryType == cudaMemoryTypeDevice) - result = true; - else - result = false; - } - } -#endif - return result; -} - - -/**********************************************************/ -/* End CUDA Utilities */ -/**********************************************************/ - -#if defined(MPID_USE_NODE_IDS) -#undef FUNCNAME -#define FUNCNAME MPID_Get_node_id -#undef FCNAME -#define FCNAME MPIU_QUOTE(FUNCNAME) -static int _g_max_node_id = -1; -int MPID_Get_node_id(MPIR_Comm *comm, int rank, MPID_Node_id_t *id_p) -{ - int mpi_errno = MPI_SUCCESS; - uint32_t node_id; - uint32_t offset; - uint32_t max_nodes; - - if(!PAMIX_Extensions.is_local_task.node_info) - { - *id_p = rank; - } - else - { - pami_result_t rc = PAMIX_Extensions.is_local_task.node_info(comm->vcr[rank]->taskid, - &node_id,&offset,&max_nodes); - if(rc != PAMI_SUCCESS) - *id_p = rank; - else - *id_p = node_id; - } - - fn_fail: - return mpi_errno; -} - -#undef FUNCNAME -#define FUNCNAME MPID_Get_max_node_id -#undef FCNAME -#define FCNAME MPIU_QUOTE(FUNCNAME) -int MPID_Get_max_node_id(MPIR_Comm *comm, MPID_Node_id_t *max_id_p) -{ - int mpi_errno = MPI_SUCCESS; - uint32_t node_id; - uint32_t offset; - uint32_t max_nodes; - if(_g_max_node_id == -1) - { - if(!PAMIX_Extensions.is_local_task.node_info) - MPIU_ERR_SETANDJUMP(mpi_errno, MPI_ERR_OTHER, "**notimpl"); - - pami_result_t rc = PAMIX_Extensions.is_local_task.node_info(comm->vcr[0]->taskid, - &node_id,&offset,&max_nodes); - if(rc != PAMI_SUCCESS) MPIU_ERR_SETANDJUMP(mpi_errno, MPI_ERR_OTHER, "**notimpl"); - *max_id_p = max_nodes; - _g_max_node_id = max_nodes; - } - else - *max_id_p = _g_max_node_id; - - fn_fail: - return mpi_errno; -} -#endif -#undef FUNCNAME diff --git a/src/mpid/pamid/src/mpix/Makefile.mk b/src/mpid/pamid/src/mpix/Makefile.mk deleted file mode 100644 index 8f4b990..0000000 --- a/src/mpid/pamid/src/mpix/Makefile.mk +++ /dev/null @@ -1,29 +0,0 @@ -# begin_generated_IBM_copyright_prolog -# -# This is an automatically generated copyright prolog. -# After initializing, DO NOT MODIFY OR MOVE -# --------------------------------------------------------------- -# Licensed Materials - Property of IBM -# Blue Gene/Q 5765-PER 5765-PRP -# -# (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved -# US Government Users Restricted Rights - -# Use, duplication, or disclosure restricted -# by GSA ADP Schedule Contract with IBM Corp. -# -# --------------------------------------------------------------- -# -# end_generated_IBM_copyright_prolog -# -*- mode: makefile-gmake; -*- - -# note that the includes always happen but the effects of their contents are -# affected by "if BUILD_PAMID" -if BUILD_PAMID - - -mpi_core_sources += \ - src/mpid/pamid/src/mpix/mpix.c - - -endif BUILD_PAMID - diff --git a/src/mpid/pamid/src/mpix/mpix.c b/src/mpid/pamid/src/mpix/mpix.c deleted file mode 100644 index 22955fd..0000000 --- a/src/mpid/pamid/src/mpix/mpix.c +++ /dev/null @@ -1,1210 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/mpix/mpix.c - * \brief This file is for MPI extensions that MPI apps are likely to use. - */ - -#include <mpidimpl.h> - -#ifdef __BGQ__ -#include <stdlib.h> -#include <spi/include/kernel/location.h> -#endif /* __BGQ__ */ - -MPIX_Hardware_t MPIDI_HW; - -/* Determine the number of torus dimensions. Implemented to keep this code - * architecture-neutral. do we already have this cached somewhere? - * this call is non-destructive (no asserts), however future calls are - * destructive so the user needs to verify numdimensions != -1 - */ - - -static void -MPIX_Init_hw(MPIX_Hardware_t *hw) -{ - memset(hw, 0, sizeof(MPIX_Hardware_t)); - - hw->clockMHz = PAMIX_Client_query(MPIDI_Client, PAMI_CLIENT_CLOCK_MHZ).value.intval; - hw->memSize = PAMIX_Client_query(MPIDI_Client, PAMI_CLIENT_MEM_SIZE).value.intval; - -#ifdef __BGQ__ - int i=0; - const pamix_torus_info_t *info = PAMIX_Torus_info(); - /* The extension returns a "T" dimension */ - hw->torus_dimension = info->dims-1; - - hw->psize = info->size[info->dims-1]; - for(i=0;i<info->dims-1;i++) - { - hw->Size[i] = info->size[i]; - hw->Coords[i] = info->coord[i]; - hw->isTorus[i] = info->torus[i]; - hw->psize = hw->psize * info->size[i]; - } - - hw->prank = info->coord[0]; - for(i=1;i<info->dims;i++) - { - hw->prank = (hw->prank * info->size[i] + info->coord[i]); - } - - /* The torus extension returns "T" as the last element */ - hw->coreID = info->coord[info->dims-1]; - hw->ppn = info->size[info->dims-1]; -#endif -} - - -void -MPIX_Init() -{ - MPIX_Init_hw(&MPIDI_HW); -} - - -#if !defined(__AIX__) -extern int backtrace(void **buffer, int size); /**< GlibC backtrace support */ -extern char **backtrace_symbols(void *const *buffer, int size); /**< GlibC backtrace support */ -void MPIX_Dump_stacks() -{ - const size_t SIZE=32; - void *array[SIZE]; - size_t i; - size_t size = backtrace(array, SIZE); - char **bt_strings = backtrace_symbols(array, size); - fprintf(stderr, "Dumping %zd frames:\n", size - 1); - for (i = 1; i < size; i++) - { - if (bt_strings != NULL) - fprintf(stderr, "\tFrame %zd: %p: %s\n", i, array[i], bt_strings[i]); - else - fprintf(stderr, "\tFrame %zd: %p\n", i, array[i]); - } - /* This is #ifdef'd out: In debug libraries, it causes a compile error. */ -#if 0 - free(bt_strings); /* Since this is not allocated by MPL_malloc, do not use MPL_free */ -#endif -} -#else -void MPIX_Dump_stacks(){} -#endif - - -void -MPIX_Progress_poke() -{ - MPID_Progress_poke(); -} - - -int -MPIX_Progress_quiesce(double timeout) -{ - int rc; - timeout *= 1.0e6; /* convert to uSec */ - unsigned long long cycles = timeout; /* convert to long long */ - /* default to 10mS */ - cycles = (cycles ? cycles : 10000) * MPIDI_HW.clockMHz; - unsigned long long t0; - t0 = PAMI_Wtimebase(MPIDI_Client); - while (PAMI_Wtimebase(MPIDI_Client) - t0 < cycles) { - rc = MPID_Progress_wait_inline(1); - if (rc != MPI_SUCCESS) return rc; - } - return MPI_SUCCESS; -} - - -int -MPIX_Comm_rank2global(MPI_Comm comm, int crank, int *grank) -{ - if (grank == NULL) - return MPI_ERR_ARG; - - MPIR_Comm *comm_ptr = NULL; - MPIR_Comm_get_ptr(comm, comm_ptr); - if (comm_ptr == NULL) - return MPI_ERR_COMM; - - if (crank >= comm_ptr->local_size) - return MPI_ERR_RANK; - - *grank = MPID_VCR_GET_LPID(comm_ptr->vcr, crank); - return MPI_SUCCESS; -} - - -int -MPIX_Hardware(MPIX_Hardware_t *hw) -{ - if (hw == NULL) - return MPI_ERR_ARG; - - /* - * We've already initialized the hw structure in MPID_Init, - * so just copy it to the users buffer - */ - memcpy(hw, &MPIDI_HW, sizeof(MPIX_Hardware_t)); - return MPI_SUCCESS; -} - -#if (MPIDI_PRINTENV || MPIDI_STATISTICS || MPIDI_BANNER) -void mpc_statistics_write() __attribute__ ((alias("MPIX_statistics_write"))); -void mp_statistics_write() __attribute__ ((alias("MPIXF_statistics_write"))); -void mp_statistics_write_() __attribute__ ((alias("MPIXF_statistics_write"))); -void mp_statistics_write__() __attribute__ ((alias("MPIXF_statistics_write"))); -void mpc_statistics_zero() __attribute__ ((alias("MPIX_statistics_zero"))); -void mp_statistics_zero() __attribute__ ((alias("MPIXF_statistics_zero"))); -void mp_statistics_zero_() __attribute__ ((alias("MPIXF_statistics_zero"))); -void mp_statistics_zero__() __attribute__ ((alias("MPIXF_statistics_zero"))); - - /* ------------------------------------------- */ - /* - MPIDI_Statistics_zero and -------- */ - /* - MPIDI_Statistics_write can be -------- */ - /* - called during init and finalize -------- */ - /* - PE utiliti routines -------- */ - /* -------------------------------------------- */ - -int -MPIDI_Statistics_zero(void) -{ - int rc=0; - - mpid_statp->sends = 0; - mpid_statp->sendsComplete = 0; - mpid_statp->sendWaitsComplete = 0; - mpid_statp->recvs = 0; - mpid_statp->recvWaitsComplete = 0; - mpid_statp->earlyArrivals = 0; - mpid_statp->earlyArrivalsMatched = 0; - mpid_statp->lateArrivals = 0; - mpid_statp->unorderedMsgs = 0; - - return (rc); /* to map with current PE support */ -} - /*************************************************************************** - Function Name: _MPIX_statistics_zero - - Description: Call the corresponding MPIDI_statistics_zero function to initialize/clear - statistics counter. - - Parameters: - Name Type I/O - void - int >0 Success - <0 statistics not enable - ***************************************************************************/ - -int _MPIX_statistics_zero (void) -{ - int rc = MPIDI_Statistics_zero(); - if (rc < 0) { - MPID_assert(rc == PAMI_SUCCESS); - } - return(rc); -} - -int MPIX_statistics_zero(void) -{ - return(_MPIX_statistics_zero()); -} - -void MPIXF_statistics_zero(int *rc) -{ - *rc = _MPIX_statistics_zero(); -} - - -int -MPIDI_Statistics_write(FILE *statfile) { - - int rc=-1; - int i; - char time_buf[201]; - extern pami_extension_t extension; - pami_configuration_t query_stat; - pami_statistics_t *pami_stat; - pami_counter_t *pami_counters; - long long Tot_dup_pkt_cnt=0; - long long Tot_retrans_pkt_cnt=0; - long long Tot_gho_pkt_cnt=0; - long long Tot_pkt_sent_cnt=0; - long long Tot_pkt_recv_cnt=0; - long long Tot_data_sent=0; - long long Tot_data_recv=0; - extern long mem_hwmark; - - memset(&time_buf,0, 201); - sprintf(time_buf, __DATE__" "__TIME__); - mpid_statp->buffer_mem_hwmark = mem_hwmark; - mpid_statp->sendWaitsComplete = mpid_statp->sends - mpid_statp->sendsComplete; - fprintf(statfile,"Start of task (pid=%d) statistics at %s \n", getpid(), time_buf); - fprintf(statfile, "MPICH: sends = %ld\n", mpid_statp->sends); - fprintf(statfile, "MPICH: sendsComplete = %ld\n", mpid_statp->sendsComplete); - fprintf(statfile, "MPICH: sendWaitsComplete = %ld\n", mpid_statp->sendWaitsComplete); - fprintf(statfile, "MPICH: recvs = %ld\n", mpid_statp->recvs); - fprintf(statfile, "MPICH: recvWaitsComplete = %ld\n", mpid_statp->recvWaitsComplete); - fprintf(statfile, "MPICH: earlyArrivals = %ld\n", mpid_statp->earlyArrivals); - fprintf(statfile, "MPICH: earlyArrivalsMatched = %ld\n", mpid_statp->earlyArrivalsMatched); - fprintf(statfile, "MPICH: lateArrivals = %ld\n", mpid_statp->lateArrivals); - fprintf(statfile, "MPICH: unorderedMsgs = %ld\n", mpid_statp->unorderedMsgs); - fprintf(statfile, "MPICH: buffer_mem_hwmark = %ld\n", mpid_statp->buffer_mem_hwmark); - fflush(statfile); - memset(&query_stat,0, sizeof(query_stat)); - query_stat.name = (pami_attribute_name_t)PAMI_CONTEXT_STATISTICS ; - rc = PAMI_Context_query(MPIDI_Context[0], &query_stat, 1); - pami_stat = (pami_statistics_t*)query_stat.value.chararray; - pami_counters = pami_stat->counters; - if (!rc) { - for (i = 0; i < pami_stat->count; i ++) { - printf("+++%s:%llu\n", pami_counters[i].name, pami_counters[i].value); - if (!strncasecmp("Duplicate Pkt Count",pami_counters[i].name,19)) { - Tot_dup_pkt_cnt=pami_counters[i].value; - } else if (!strncasecmp("Retransmit Pkt Count",pami_counters[i].name,20)) { - Tot_retrans_pkt_cnt=pami_counters[i].value; - } else if (!strncasecmp("Ghost Pkt Count",pami_counters[i].name,15)) { - Tot_gho_pkt_cnt=pami_counters[i].value; - } else if (!strncasecmp("Packets Sent",pami_counters[i].name,12) && - (!(strchr(pami_counters[i].name, 'v')))) { - /* Packets Sent, not Packets Sent via SHM */ - Tot_pkt_sent_cnt=pami_counters[i].value; - } else if (!strncasecmp("Packets Received",pami_counters[i].name,16) && - (!(strchr(pami_counters[i].name, 'S')))) { - /* Packets Received, not Packets Received via SHM */ - Tot_pkt_recv_cnt=pami_counters[i].value; - } else if (!strncasecmp("Data Sent",pami_counters[i].name,9) && - (!(strchr(pami_counters[i].name, 'v')))) { - /* Data Sent, not Data Sent via SHM */ - Tot_data_sent=pami_counters[i].value; - } else if (!strncasecmp("Data Received",pami_counters[i].name,13) && - (!(strchr(pami_counters[i].name, 'S')))) { - /* Data Received, not Data Received via SHM */ - Tot_data_recv=pami_counters[i].value; - } - } - fprintf(statfile, "PAMI: Tot_dup_pkt_cnt=%lld\n", Tot_dup_pkt_cnt); - fprintf(statfile, "PAMI: Tot_retrans_pkt_cnt=%lld\n", Tot_retrans_pkt_cnt); - fprintf(statfile, "PAMI: Tot_gho_pkt_cnt=%lld\n", Tot_gho_pkt_cnt); - fprintf(statfile, "PAMI: Tot_pkt_sent_cnt=%lld\n", Tot_pkt_sent_cnt); - fprintf(statfile, "PAMI: Tot_pkt_recv_cnt=%lld\n", Tot_pkt_recv_cnt); - fprintf(statfile, "PAMI: Tot_data_sent=%lld\n", Tot_data_sent); - fprintf(statfile, "PAMI: Tot_data_recv=%lld\n", Tot_data_recv); - fflush(statfile); - } else { - TRACE_ERR("PAMID: PAMI_Context_query() with PAMI_CONTEXT_STATISTICS failed rc =%d\ -n",rc); - } - return (rc); -} - /*************************************************************************** - Function Name: _MPIX_statistics_write - Description: Call MPIDI_Statistics_write to write statistical - information to specified file descriptor. - Parameters: - Name Type I/O - fptr FILE* I File pointer, can be stdout or stderr. - If it is to a file, user has to open - the file. - rc (Fortran only) int 0 Return sum from MPIDI_Statistics_write calls - <returns> (C only) 0 Both MPICH and PAMI statistics - ***************************************************************************/ -int _MPIX_statistics_write(FILE* fptr) -{ - int rc = MPIDI_Statistics_write(fptr); - if (rc < 0) { - MPID_assert(rc == PAMI_SUCCESS); - } - return(rc); -} - -int MPIX_statistics_write(FILE* fptr) -{ - return(_MPIX_statistics_write(fptr)); -} - -/* Fortran: fdes is pointer to a file descriptor. - * rc is pointer to buffer for storing return code. - * - * Note: Fortran app. will convert a Fortran I/O unit to a file - * descriptor by calling Fortran utilities, flush_ and getfd. - * When fdes=1, output is to STDOUT. When fdes=2, output is to STDERR. - */ - -void MPIXF_statistics_write(int *fdes, int *rc) -{ - FILE *fp; - int dup_fd; - int closefp=0; - - /* Convert the DUP file descriptor to a FILE pointer */ - dup_fd = dup(*fdes); - if ( (fp = fdopen(dup_fd, "a")) != NULL ) - closefp = 1; - else - fp = stdout; /* If fdopen failed then default to stdout */ - - *rc = _MPIX_statistics_write(fp); - - /* The check is because I don't want to close stdout. */ - if ( closefp ) fclose(fp); -} - -void MPIXF_statistics_write_(int *fdes, int *rc) -{ - FILE *fp; - int dup_fd; - int closefp=0; - - /* Convert the DUP file descriptor to a FILE pointer */ - dup_fd = dup(*fdes); - if ( (fp = fdopen(dup_fd, "a")) != NULL ) - closefp = 1; - else - fp = stdout; /* If fdopen failed then default to stdout */ - - *rc = _MPIX_statistics_write(fp); - - /* The check is because I don't want to close stdout. */ - if ( closefp ) fclose(fp); -} -#endif - - -#ifdef __BGQ__ - -int -MPIX_Torus_ndims(int *numdimensions) -{ - const pamix_torus_info_t *info = PAMIX_Torus_info(); - *numdimensions = info->dims - 1; - return MPI_SUCCESS; -} - - -int -MPIX_Rank2torus(int rank, int *coords) -{ - size_t coord_array[MPIDI_HW.torus_dimension+1]; - - PAMIX_Task2torus(rank, coord_array); - - unsigned i; - for(i=0; i<MPIDI_HW.torus_dimension+1; ++i) - coords[i] = coord_array[i]; - - return MPI_SUCCESS; -} - - -int -MPIX_Torus2rank(int *coords, int *rank) -{ - size_t coord_array[MPIDI_HW.torus_dimension+1]; - pami_task_t task; - - unsigned i; - for(i=0; i<MPIDI_HW.torus_dimension+1; ++i) - coord_array[i] = coords[i]; - - PAMIX_Torus2task(coord_array, &task); - *rank = task; - - return MPI_SUCCESS; -} - - -typedef struct -{ - pami_geometry_t geometry; - pami_work_t state; - pami_configuration_t config; - size_t num_configs; - pami_event_function fn; - void* cookie; -} MPIX_Comm_update_data_t; - -static void -MPIX_Comm_update_done(void *ctxt, void *data, pami_result_t err) -{ - int *active = (int *)data; - (*active)--; -} - -static pami_result_t -MPIX_Comm_update_post(pami_context_t context, void *cookie) -{ - MPIX_Comm_update_data_t *data = (MPIX_Comm_update_data_t *)cookie; - - return PAMI_Geometry_update(data->geometry, - &data->config, - data->num_configs, - context, - data->fn, - data->cookie); -} - -int -MPIX_Comm_update(MPI_Comm comm, int optimize) -{ - MPIR_Comm * comm_ptr; - volatile int geom_update = 1; - MPIX_Comm_update_data_t data; - pami_configuration_t config; - - MPIR_Comm_get_ptr(comm, comm_ptr); - if (!comm_ptr || comm == MPI_COMM_NULL) - return MPI_ERR_COMM; - - /* First, check if there is a geometry. When optimized collectives - * are disabled, no geometry is created */ - if(comm_ptr->mpid.geometry == NULL) - return MPI_ERR_COMM; - - config.name = PAMI_GEOMETRY_OPTIMIZE; - config.value.intval = !!optimize; - - TRACE_ERR("About to %s geometry update function\n", MPIDI_Process.context_post.active>0?"post":"invoke"); - data.num_configs = 1; - data.config.name = config.name; - data.config.value.intval = config.value.intval; - data.fn = MPIX_Comm_update_done; - data.cookie = (void *)&geom_update; - data.geometry = comm_ptr->mpid.geometry; - MPIDI_Context_post(MPIDI_Context[0], - &data.state, - MPIX_Comm_update_post, - &data); - TRACE_ERR("Geometry update function %s\n", MPIDI_Process.context_post.active>0?"posted":"invoked"); - - TRACE_ERR("Waiting for geometry update to finish\n"); - - MPID_PROGRESS_WAIT_WHILE(geom_update); - - MPIDI_Comm_coll_query(comm_ptr); - MPIDI_Comm_coll_envvars(comm_ptr); - if(MPIDI_Process.optimized.select_colls) - MPIDI_Comm_coll_select(comm_ptr); - - return MPI_SUCCESS; -} - -int -MPIX_Get_last_algorithm_name(MPI_Comm comm, char *protocol, int length) -{ - MPIR_Comm *comm_ptr; - MPIR_Comm_get_ptr(comm, comm_ptr); - - if(!comm_ptr || comm == MPI_COMM_NULL) - return MPI_ERR_COMM; - if(!protocol || length <= 0) - return MPI_ERR_ARG; - strncpy(protocol, comm_ptr->mpid.last_algorithm, length); - return MPI_SUCCESS; -} - -#undef FUNCNAME -#define FUNCNAME MPIX_IO_node_id -#undef FCNAME -#define FCNAME MPIU_QUOTE(FUNCNAME) -int MPIX_IO_node_id () -{ - static unsigned long IO_node_id = ULONG_MAX; - - if (IO_node_id != ULONG_MAX) - return (int)(IO_node_id>>32); - - int rc; - int fd; - char* uci_str; - char buffer[4096]; - - fd = open("/dev/bgpers", O_RDONLY, 0); - assert(fd>=0); - rc = read(fd, buffer, sizeof(buffer)); - assert(rc>0); - close(fd); - - uci_str = strstr(buffer, "BG_UCI="); - assert(uci_str); - uci_str += sizeof("BG_UCI=")-1; - - IO_node_id = strtoul(uci_str, NULL, 16); - return (int)(IO_node_id>>32); -} - -#undef FUNCNAME -#define FUNCNAME MPIX_IO_link_id -#undef FCNAME -#define FCNAME MPIU_QUOTE(FUNCNAME) -int MPIX_IO_link_id () -{ - int nB, nC, nD, nE; /* Size of each torus dimension */ - int brA, brB, brC, brD, brE; /* The bridge node's coordinates */ - - Personality_t personality; - - Kernel_GetPersonality(&personality, sizeof(personality)); - - nB = personality.Network_Config.Bnodes; - nC = personality.Network_Config.Cnodes; - nD = personality.Network_Config.Dnodes; - nE = personality.Network_Config.Enodes; - - brA = personality.Network_Config.cnBridge_A; - brB = personality.Network_Config.cnBridge_B; - brC = personality.Network_Config.cnBridge_C; - brD = personality.Network_Config.cnBridge_D; - brE = personality.Network_Config.cnBridge_E; - - /* - * This is the bridge node, numbered in ABCDE order, E increments first. - * It is considered the unique "io node route identifer" because each - * bridge node only has one torus link to one io node. - */ - return brE + brD*nE + brC*nD*nE + brB*nC*nD*nE + brA*nB*nC*nD*nE; -}; - -#undef FUNCNAME -#define FUNCNAME MPIX_IO_distance -#undef FCNAME -#define FCNAME MPIU_QUOTE(FUNCNAME) -int MPIX_IO_distance () -{ - int iA, iB, iC, iD, iE; /* The local node's coordinates */ - int nA, nB, nC, nD, nE; /* Size of each torus dimension */ - int brA, brB, brC, brD, brE; /* The bridge node's coordinates */ - int Nflags; - int d1, d2; - int dA, dB, dC, dD, dE; /* distance from local node to bridge node */ - - Personality_t personality; - - Kernel_GetPersonality(&personality, sizeof(personality)); - - iA = personality.Network_Config.Acoord; - iB = personality.Network_Config.Bcoord; - iC = personality.Network_Config.Ccoord; - iD = personality.Network_Config.Dcoord; - iE = personality.Network_Config.Ecoord; - - nA = personality.Network_Config.Anodes; - nB = personality.Network_Config.Bnodes; - nC = personality.Network_Config.Cnodes; - nD = personality.Network_Config.Dnodes; - nE = personality.Network_Config.Enodes; - - brA = personality.Network_Config.cnBridge_A; - brB = personality.Network_Config.cnBridge_B; - brC = personality.Network_Config.cnBridge_C; - brD = personality.Network_Config.cnBridge_D; - brE = personality.Network_Config.cnBridge_E; - - Nflags = personality.Network_Config.NetFlags; - - dA = abs(iA - brA); - if (Nflags & ND_ENABLE_TORUS_DIM_A) - { - d1 = dA; - d2 = nA - d1; - dA = (d1 < d2) ? d1 : d2; - } - - dB = abs(iB - brB); - if (Nflags & ND_ENABLE_TORUS_DIM_B) - { - d1 = dB; - d2 = nB - d1; - dB = (d1 < d2) ? d1 : d2; - } - - dC = abs(iC - brC); - if (Nflags & ND_ENABLE_TORUS_DIM_C) - { - d1 = dC; - d2 = nC - d1; - dC = (d1 < d2) ? d1 : d2; - } - - dD = abs(iD - brD); - if (Nflags & ND_ENABLE_TORUS_DIM_D) - { - d1 = dD; - d2 = nD - d1; - dD = (d1 < d2) ? d1 : d2; - } - - dE = abs(iE - brE); - if (Nflags & ND_ENABLE_TORUS_DIM_E) - { - d1 = dE; - d2 = nE - d1; - dE = (d1 < d2) ? d1 : d2; - } - - /* This is the number of hops to the io node */ - return dA + dB + dC + dD + dE + 1; -}; - -#undef FUNCNAME -#define FUNCNAME MPIX_Pset_io_node -#undef FCNAME -#define FCNAME MPIU_QUOTE(FUNCNAME) -void -MPIX_Pset_io_node (int *io_node_route_id, int *distance_to_io_node) -{ - *io_node_route_id = MPIX_IO_link_id (); - *distance_to_io_node = MPIX_IO_distance (); - - return; -}; - -/** - * \brief Create a communicator of ranks that have a common bridge node. - * - * \note This function is private to this source file. - * - * \param [in] parent_comm_ptr Pointer to the parent communicator - * \param [out] pset_comm_ptr Pointer to the new 'MPID' communicator - * - * \return MPI status - */ -int _MPIX_Pset_same_comm_create (MPIR_Comm *parent_comm_ptr, MPIR_Comm **pset_comm_ptr) -{ - int color, key; - int mpi_errno; - - color = MPIX_IO_link_id (); - key = MPIX_IO_distance (); - - /* - * Use MPIR_Comm_split_impl to make a communicator of all ranks in the parent - * communicator that share the same bridge node; i.e. the 'color' is the - * 'io node route identifer', which is unique to each BGQ bridge node. - * - * Setting the 'key' to the 'distance to io node' ensures that rank 0 in - * the new communicator is on the bridge node, or as close to the bridge node - * as possible. - */ - - *pset_comm_ptr = NULL; - mpi_errno = MPI_SUCCESS; - mpi_errno = MPIR_Comm_split_impl(parent_comm_ptr, color, key, pset_comm_ptr); - - return mpi_errno; -} - -#undef FUNCNAME -#define FUNCNAME MPIX_Pset_same_comm_create_from_parent -#undef FCNAME -#define FCNAME MPIU_QUOTE(FUNCNAME) -int -MPIX_Pset_same_comm_create_from_parent (MPI_Comm parent_comm, MPI_Comm *pset_comm) -{ - int mpi_errno; - MPIR_Comm *parent_comm_ptr, *pset_comm_ptr; - - *pset_comm = MPI_COMM_NULL; - - /* - * Convert the parent communicator object handle to an object pointer; - * needed by the error handling code. - */ - parent_comm_ptr = NULL; - MPIR_Comm_get_ptr(parent_comm, parent_comm_ptr); - - mpi_errno = MPI_SUCCESS; - mpi_errno = _MPIX_Pset_same_comm_create (parent_comm_ptr, &pset_comm_ptr); - if (mpi_errno) MPIU_ERR_POP(mpi_errno); - if (pset_comm_ptr) - MPIR_OBJ_PUBLISH_HANDLE(*pset_comm, pset_comm_ptr->handle); - else - goto fn_fail; - -fn_exit: - return mpi_errno; -fn_fail: - mpi_errno = MPIR_Err_return_comm( parent_comm_ptr, FCNAME, mpi_errno ); - goto fn_exit; -}; - -#undef FUNCNAME -#define FUNCNAME MPIX_Pset_same_comm_create -#undef FCNAME -#define FCNAME MPIU_QUOTE(FUNCNAME) -int -MPIX_Pset_same_comm_create (MPI_Comm *pset_comm) -{ - return MPIX_Pset_same_comm_create_from_parent (MPI_COMM_WORLD, pset_comm); -}; - - -#undef FUNCNAME -#define FUNCNAME MPIX_Pset_diff_comm_create_from_parent -#undef FCNAME -#define FCNAME MPIU_QUOTE(FUNCNAME) -int -MPIX_Pset_diff_comm_create_from_parent (MPI_Comm parent_comm, MPI_Comm *pset_comm) -{ - MPIR_Comm *parent_comm_ptr, *pset_same_comm_ptr, *pset_diff_comm_ptr; - int color, key; - int mpi_errno; - - *pset_comm = MPI_COMM_NULL; - - /* - * Convert the parent communicator object handle to an object pointer; - * needed by the error handling code. - */ - parent_comm_ptr = NULL; - MPIR_Comm_get_ptr(parent_comm, parent_comm_ptr); - - /* - * Determine the 'color' of this rank to create the new communicator - which - * is the rank in a (transient) communicator where all ranks share a common - * bridge node. - */ - mpi_errno = MPI_SUCCESS; - mpi_errno = _MPIX_Pset_same_comm_create (parent_comm_ptr, &pset_same_comm_ptr); - if (mpi_errno) MPIU_ERR_POP(mpi_errno); - if (pset_same_comm_ptr == NULL) - goto fn_fail; - - color = MPIR_Comm_rank(pset_same_comm_ptr) * MPIDI_HW.ppn + MPIDI_HW.coreID; - - /* Discard the 'pset_same_comm_ptr' .. it is no longer needed. */ - mpi_errno = MPIR_Comm_free_impl(pset_same_comm_ptr); - if (mpi_errno) MPIU_ERR_POP(mpi_errno); - - /* Set the 'key' for this rank to order the ranks in the new communicator. */ - key = MPIR_Comm_rank(parent_comm_ptr); - - pset_diff_comm_ptr = NULL; - mpi_errno = MPI_SUCCESS; - mpi_errno = MPIR_Comm_split_impl(parent_comm_ptr, color, key, &pset_diff_comm_ptr); - if (mpi_errno) MPIU_ERR_POP(mpi_errno); - if (pset_diff_comm_ptr) - MPIR_OBJ_PUBLISH_HANDLE(*pset_comm, pset_diff_comm_ptr->handle); - else - goto fn_fail; - -fn_exit: - return mpi_errno; -fn_fail: - mpi_errno = MPIR_Err_return_comm( parent_comm_ptr, FCNAME, mpi_errno ); - goto fn_exit; -}; - -#undef FUNCNAME -#define FUNCNAME MPIX_Pset_diff_comm_create -#undef FCNAME -#define FCNAME MPIU_QUOTE(FUNCNAME) -int -MPIX_Pset_diff_comm_create (MPI_Comm *pset_comm) -{ - return MPIX_Pset_diff_comm_create_from_parent (MPI_COMM_WORLD, pset_comm); -}; - - - -/** - * \brief Compare each elemt of two six-element arrays - * \param [in] A The first array - * \param [in] B The first array - * \return MPI_SUCCESS (does not return on failure) - */ -#define CMP_6(A,B) \ -({ \ - assert(A[0] == B[0]); \ - assert(A[1] == B[1]); \ - assert(A[2] == B[2]); \ - assert(A[3] == B[3]); \ - assert(A[4] == B[4]); \ - assert(A[5] == B[5]); \ - MPI_SUCCESS; \ -}) - -#undef FUNCNAME -#define FUNCNAME MPIX_Cart_comm_create -#undef FCNAME -#define FCNAME MPIU_QUOTE(FUNCNAME) -int -MPIX_Cart_comm_create (MPI_Comm *cart_comm) -{ - int result; - int rank, numprocs, - dims[6], - wrap[6], - coords[6]; - int new_rank1, new_rank2; - MPI_Comm new_comm = MPI_COMM_NULL; - int cart_rank, - cart_dims[6], - cart_wrap[6], - cart_coords[6]; - int Nflags; - - *cart_comm = MPI_COMM_NULL; - PMPI_Comm_rank(MPI_COMM_WORLD, &rank); - PMPI_Comm_size(MPI_COMM_WORLD, &numprocs); - - Personality_t personality; - - Kernel_GetPersonality(&personality, sizeof(personality)); - - dims[0] = personality.Network_Config.Anodes; - dims[1] = personality.Network_Config.Bnodes; - dims[2] = personality.Network_Config.Cnodes; - dims[3] = personality.Network_Config.Dnodes; - dims[4] = personality.Network_Config.Enodes; - dims[5] = Kernel_ProcessCount(); - - /* This only works if MPI_COMM_WORLD is the full partition */ - if (dims[5] * dims[4] * dims[3] * dims[2] * dims[1] * dims[0] != numprocs) - return MPI_ERR_TOPOLOGY; - - Nflags = personality.Network_Config.NetFlags; - wrap[0] = ((Nflags & ND_ENABLE_TORUS_DIM_A) != 0); - wrap[1] = ((Nflags & ND_ENABLE_TORUS_DIM_B) != 0); - wrap[2] = ((Nflags & ND_ENABLE_TORUS_DIM_C) != 0); - wrap[3] = ((Nflags & ND_ENABLE_TORUS_DIM_D) != 0); - wrap[4] = ((Nflags & ND_ENABLE_TORUS_DIM_E) != 0); - wrap[5] = 1; - - coords[0] = personality.Network_Config.Acoord; - coords[1] = personality.Network_Config.Bcoord; - coords[2] = personality.Network_Config.Ccoord; - coords[3] = personality.Network_Config.Dcoord; - coords[4] = personality.Network_Config.Ecoord; - coords[5] = Kernel_MyTcoord(); - - new_rank1 = coords[5] + - dims[5] * coords[4] + - dims[5] * dims[4] * coords[3] + - dims[5] * dims[4] * dims[3] * coords[2] + - dims[5] * dims[4] * dims[3] * dims[2] * coords[1] + - dims[5] * dims[4] * dims[3] * dims[2] * dims[1] * coords[0]; - - result = PMPI_Comm_split(MPI_COMM_WORLD, 0, new_rank1, &new_comm); - if (result != MPI_SUCCESS) - { - PMPI_Comm_free(&new_comm); - return result; - } - PMPI_Comm_rank(new_comm, &new_rank2); - assert(new_rank1 == new_rank2); - - result = PMPI_Cart_create(new_comm, - 6, - dims, - wrap, - 0, - cart_comm); - if (result != MPI_SUCCESS) - return result; - - PMPI_Comm_rank(*cart_comm, &cart_rank); - PMPI_Cart_get (*cart_comm, 6, cart_dims, cart_wrap, cart_coords); - - CMP_6(dims, cart_dims); - CMP_6(wrap, cart_wrap); - CMP_6(coords, cart_coords); - - PMPI_Comm_free(&new_comm); - return MPI_SUCCESS; -}; - -/** - * \brief FORTRAN interface to MPIX_Comm_rank2global - * - * \param [in] comm Communicator - * \param [in] crank Pointer to the rank in the communicator variable - * \param [out] grank Pointer tot he global rank variable - * - * \return status - */ -int mpix_comm_rank2global (MPI_Comm *comm, int *crank, int *grank) -{ - return MPIX_Comm_rank2global (*comm, *crank, grank); -} - -/** - * \brief FORTRAN interface to MPIX_Pset_same_comm_create - * - * \param [out] pset_comm Communicator - * - * \return status - */ -int mpix_pset_same_comm_create (MPI_Comm *pset_comm) -{ - return MPIX_Pset_same_comm_create (pset_comm); -} - -/** - * \brief FORTRAN interface to MPIX_Pset_diff_comm_create - * - * \param [out] pset_comm Communicator - * - * \return status - */ -int mpix_pset_diff_comm_create (MPI_Comm *pset_comm) -{ - return MPIX_Pset_diff_comm_create (pset_comm); -} - -/** - * \brief FORTRAN interface to MPIX_Pset_same_comm_create_from_parent - * - * \param [in] parent_comm Parent communicator - * \param [out] pset_comm New pset communicator - * - * \return status - */ -int mpix_pset_same_comm_create_from_parent (MPI_Comm *parent_comm, MPI_Comm *pset_comm) -{ - return MPIX_Pset_same_comm_create_from_parent (*parent_comm, pset_comm); -} - -/** - * \brief FORTRAN interface to MPIX_Pset_diff_comm_create_from_parent - * - * \param [in] parent_comm Parent communicator - * \param [out] pset_comm New pset communicator - * - * \return status - */ -int mpix_pset_diff_comm_create_from_parent (MPI_Comm *parent_comm, MPI_Comm *pset_comm) -{ - return MPIX_Pset_diff_comm_create_from_parent (*parent_comm, pset_comm); -} - -/** - * \brief FORTRAN interface to MPIX_IO_node_id - * - * \param [out] io_node_id This rank's io node id - */ -void mpix_io_node_id (int *io_node_id) { *io_node_id = MPIX_IO_node_id(); } - -/** - * \brief FORTRAN interface to MPIX_IO_link_id - * - * \param [out] io_link_id This rank's io link id - */ -void mpix_io_link_id (int *io_link_id) { *io_link_id = MPIX_IO_link_id(); } - -/** - * \brief FORTRAN interface to MPIX_IO_distance - * - * \param [out] io_distance This rank's distance to the io node - */ -void mpix_io_distance (int *io_distance) { *io_distance = MPIX_IO_distance(); } - -/** - * \brief FORTRAN interface to MPIX_Cart_comm_create - * - * \param [out] cart_comm Communicator to create - * - * \return status - */ -int mpix_cart_comm_create (MPI_Comm *cart_comm) -{ - return MPIX_Cart_comm_create (cart_comm); -} - - - -#endif - -#ifdef __PE__ -void mpc_disableintr() __attribute__ ((alias("MPIX_disableintr"))); -void mp_disableintr() __attribute__ ((alias("MPIXF_disableintr"))); -void mp_disableintr_() __attribute__ ((alias("MPIXF_disableintr"))); -void mp_disableintr__() __attribute__ ((alias("MPIXF_disableintr"))); -void mpc_enableintr() __attribute__ ((alias("MPIX_enableintr"))); -void mp_enableintr() __attribute__ ((alias("MPIXF_enableintr"))); -void mp_enableintr_() __attribute__ ((alias("MPIXF_enableintr"))); -void mp_enableintr__() __attribute__ ((alias("MPIXF_enableintr"))); -void mpc_queryintr() __attribute__ ((weak,alias("MPIX_queryintr"))); -void mp_queryintr() __attribute__ ((alias("MPIXF_queryintr"))); -void mp_queryintr_() __attribute__ ((alias("MPIXF_queryintr"))); -void mp_queryintr__() __attribute__ ((alias("MPIXF_queryintr"))); - - /*************************************************************************** - Function Name: MPIX_disableintr - - Description: Call the pamid layer to disable interrupts. - (Similar to setting MP_CSS_INTERRUPT to "no") - - Parameters: The Fortran versions have an int* parameter used to pass the - return code to the calling program. - - Returns: 0 Success - <0 Failure - ***************************************************************************/ - -int -_MPIDI_disableintr() -{ - return(MPIDI_disableintr()); -} - -int -MPIX_disableintr() -{ - return(_MPIDI_disableintr()); -} - -void -MPIXF_disableintr(int *rc) -{ - *rc = _MPIDI_disableintr(); -} - -void -MPIXF_disableintr_(int *rc) -{ - *rc = _MPIDI_disableintr(); -} - -/* - ** Called by: _mp_disableintr - ** Purpose : Disables interrupts - */ -int -MPIDI_disableintr() -{ - pami_result_t rc=0; - int i; - - MPIR_ERRTEST_INITIALIZED_ORDIE(); - if (MPIDI_Process.mp_interrupts!= 0) - { - TRACE_ERR("Async advance beginning...\n"); - /* Enable async progress on all contexts.*/ - for (i=0; i<MPIDI_Process.avail_contexts; ++i) - { - PAMIX_Progress_disable(MPIDI_Context[i], PAMIX_PROGRESS_ALL); - } - TRACE_ERR("Async advance disabled\n"); - MPIDI_Process.mp_interrupts=0; - } - return(rc); -} - /*************************************************************************** - Function Name: MPIX_enableintr - - Description: Call the pamid-layer function to enable interrupts. - (Similar to setting MP_CSS_INTERRUPT to "yes") - - Parameters: The Fortran versions have an int* parameter used to pass the - return code to the calling program. - - Returns: 0 Success - <0 Failure - ***************************************************************************/ -int -_MPIDI_enableintr() -{ - return(MPIDI_enableintr()); -} - -/* C callable version */ -int -MPIX_enableintr() -{ - return(_MPIDI_enableintr()); -} - -/* Fortran callable version */ -void -MPIXF_enableintr(int *rc) -{ - *rc = _MPIDI_enableintr(); -} - -/* Fortran callable version for -qEXTNAME support */ -void -MPIXF_enableintr_(int *rc) -{ - *rc = _MPIDI_enableintr(); -} - -int -MPIDI_enableintr() -{ - pami_result_t rc=0; - int i; - - MPIR_ERRTEST_INITIALIZED_ORDIE(); - if (MPIDI_Process.mp_interrupts == 0) - { - /* Enable async progress on all contexts.*/ - for (i=0; i<MPIDI_Process.avail_contexts; ++i) - { - PAMIX_Progress_enable(MPIDI_Context[i], PAMIX_PROGRESS_ALL); - } - TRACE_ERR("Async advance enabled\n"); - MPIDI_Process.mp_interrupts=1; - } - MPID_assert(rc == PAMI_SUCCESS); - return(rc); -} - - /*************************************************************************** - Function Name: MPIX_queryintr - - Description: Call the pamid-layer function to determine if - interrupts are currently on or off. - - Parameters: The Fortran versions have an int* parameter used to pass the - current interrupt setting to the calling program. - Returns: 0 Indicates interrupts are currently off - 1 Indicates interrupts are currently on - <0 Failure - ***************************************************************************/ -int -MPIDI_queryintr() -{ - return(MPIDI_Process.mp_interrupts); -} - -int -_MPIDI_queryintr() -{ - return(MPIDI_queryintr()); -} - -int -MPIX_queryintr() -{ - return(_MPIDI_queryintr()); -} - -void -MPIXF_queryintr(int *rc) -{ - *rc = _MPIDI_queryintr(); -} - -void -MPIXF_queryintr_(int *rc) -{ - *rc = _MPIDI_queryintr(); -} -#endif diff --git a/src/mpid/pamid/src/onesided/Makefile.mk b/src/mpid/pamid/src/onesided/Makefile.mk deleted file mode 100644 index c58ca32..0000000 --- a/src/mpid/pamid/src/onesided/Makefile.mk +++ /dev/null @@ -1,57 +0,0 @@ -# begin_generated_IBM_copyright_prolog -# -# This is an automatically generated copyright prolog. -# After initializing, DO NOT MODIFY OR MOVE -# --------------------------------------------------------------- -# Licensed Materials - Property of IBM -# Blue Gene/Q 5765-PER 5765-PRP -# -# (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved -# US Government Users Restricted Rights - -# Use, duplication, or disclosure restricted -# by GSA ADP Schedule Contract with IBM Corp. -# -# --------------------------------------------------------------- -# -# end_generated_IBM_copyright_prolog -# -*- mode: makefile-gmake; -*- - -# note that the includes always happen but the effects of their contents are -# affected by "if BUILD_PAMID" -if BUILD_PAMID - - -noinst_HEADERS += \ - src/mpid/pamid/src/onesided/mpidi_onesided.h - - -mpi_core_sources += \ - src/mpid/pamid/src/onesided/mpid_1s.c \ - src/mpid/pamid/src/onesided/mpid_win_accumulate.c \ - src/mpid/pamid/src/onesided/mpid_win_create.c \ - src/mpid/pamid/src/onesided/mpid_win_fence.c \ - src/mpid/pamid/src/onesided/mpid_win_free.c \ - src/mpid/pamid/src/onesided/mpid_win_get.c \ - src/mpid/pamid/src/onesided/mpid_win_lock.c \ - src/mpid/pamid/src/onesided/mpid_win_lock_all.c \ - src/mpid/pamid/src/onesided/mpid_win_pscw.c \ - src/mpid/pamid/src/onesided/mpid_win_put.c \ - src/mpid/pamid/src/onesided/mpid_win_shared_query.c \ - src/mpid/pamid/src/onesided/mpid_win_create_dynamic.c \ - src/mpid/pamid/src/onesided/mpid_win_allocate_shared.c \ - src/mpid/pamid/src/onesided/mpid_win_flush.c \ - src/mpid/pamid/src/onesided/mpid_win_allocate.c \ - src/mpid/pamid/src/onesided/mpid_win_sync.c \ - src/mpid/pamid/src/onesided/mpid_win_attach.c \ - src/mpid/pamid/src/onesided/mpid_win_detach.c \ - src/mpid/pamid/src/onesided/mpid_win_get_info.c \ - src/mpid/pamid/src/onesided/mpid_win_set_info.c \ - src/mpid/pamid/src/onesided/mpid_win_get_accumulate.c \ - src/mpid/pamid/src/onesided/mpid_win_reqops.c \ - src/mpid/pamid/src/onesided/mpidi_win_control.c \ - src/mpid/pamid/src/onesided/mpid_win_compare_and_swap.c \ - src/mpid/pamid/src/onesided/mpid_win_fetch_and_op.c - - -endif BUILD_PAMID - diff --git a/src/mpid/pamid/src/onesided/mpid_1s.c b/src/mpid/pamid/src/onesided/mpid_1s.c deleted file mode 100644 index e9a49b0..0000000 --- a/src/mpid/pamid/src/onesided/mpid_1s.c +++ /dev/null @@ -1,151 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/onesided/mpid_1s.c - * \brief ??? - */ -#include "mpidi_onesided.h" - - -void -MPIDI_Win_DoneCB(pami_context_t context, - void * cookie, - pami_result_t result) -{ - MPIDI_Win_request *req = (MPIDI_Win_request*)cookie; - ++req->win->mpid.sync.complete; - ++req->origin.completed; - - if ((req->buffer_free) && (req->type == MPIDI_WIN_REQUEST_GET)) - { - if (req->origin.completed == req->target.dt.num_contig) - { - int mpi_errno; - mpi_errno = MPIR_Localcopy(req->buffer, - req->origin.dt.size, - MPI_CHAR, - req->origin.addr, - req->origin.count, - req->origin.datatype); - MPID_assert(mpi_errno == MPI_SUCCESS); - MPIDU_Datatype_release(req->origin.dt.pointer); - MPL_free(req->buffer); - MPL_free(req->user_buffer); - req->buffer_free = 0; - } - } - - - if ((req->origin.completed == req->target.dt.num_contig) || - ((req->type >= MPIDI_WIN_REQUEST_COMPARE_AND_SWAP) && - (req->origin.completed == req->origin.dt.num_contig))) - { - MPIR_Request * req_handle = req->req_handle; - - if (req->buffer_free) { - MPL_free(req->buffer); - MPL_free(req->user_buffer); - req->buffer_free = 0; - } - MPIDI_Win_datatype_unmap(&req->target.dt); - if (req->accum_headers) - MPL_free(req->accum_headers); - - if (!((req->type > MPIDI_WIN_REQUEST_GET_ACCUMULATE) && (req->type <=MPIDI_WIN_REQUEST_RGET_ACCUMULATE))) - MPL_free(req); - - if(req_handle) { - - /* The instant this completion counter is set to zero another thread - * may notice the change and begin freeing request resources. The - * thread executing the code in this function must not touch any - * portion of the request structure after decrementing the completion - * counter. - * - * See MPID_Request_free_inline() - */ - MPIR_cc_set(req_handle->cc_ptr, 0); - } - } - if ((MPIDI_Process.typed_onesided == 1) && (!req->target.dt.contig || !req->origin.dt.contig)) { - /* We used the PAMI_Rput/Rget_typed call and added a ref so any MPI_Type_free before the context - * executes the put/get would not free the MPIDU_Datatype, which would also free the associated PAMI datatype - * which was still needed for communication -- that communication has completed, so now release the ref - * in the callback to allow the MPIDU_Datatypeto be freed. - */ - MPIDU_Datatype_release(req->origin.dt.pointer); - MPIDU_Datatype_release(req->target.dt.pointer); - } - MPIDI_Progress_signal(); -} - - -void -MPIDI_Win_datatype_basic(int count, - MPI_Datatype datatype, - MPIDI_Datatype * dt) -{ - MPIDI_Datatype_get_info(dt->count = count, - dt->type = datatype, - dt->contig, - dt->size, - dt->pointer, - dt->true_lb); - TRACE_ERR("DT=%08x DTp=%p count=%d contig=%d size=%zu true_lb=%zu\n", - dt->type, dt->pointer, dt->count, dt->contig, (size_t)dt->size, (size_t)dt->true_lb); -} - - -void -MPIDI_Win_datatype_map(MPIDI_Datatype * dt) -{ - if (dt->contig) - { - dt->num_contig = 1; - dt->map = &dt->__map; - dt->map[0].DLOOP_VECTOR_BUF = (void*)(size_t)dt->true_lb; - dt->map[0].DLOOP_VECTOR_LEN = dt->size; - } - else - { - unsigned map_size = dt->pointer->max_contig_blocks*dt->count + 1; - dt->num_contig = map_size; - dt->map = (DLOOP_VECTOR*)MPL_malloc(map_size * sizeof(DLOOP_VECTOR)); - MPID_assert(dt->map != NULL); - - DLOOP_Offset last = dt->pointer->size*dt->count; - MPIDU_Segment seg; - MPIDU_Segment_init(NULL, dt->count, dt->type, &seg, 0); - MPIDU_Segment_pack_vector(&seg, 0, &last, dt->map, &dt->num_contig); - MPID_assert((unsigned)dt->num_contig <= map_size); -#ifdef TRACE_ON - TRACE_ERR("dt->pointer->size=%d num_contig: orig=%u new=%d\n", dt->pointer->size, map_size, dt->num_contig); - int i; - for(i=0; i<dt->num_contig; ++i) - TRACE_ERR(" %d: BUF=%zu LEN=%zu\n", i, (size_t)dt->map[i].DLOOP_VECTOR_BUF, (size_t)dt->map[i].DLOOP_VECTOR_LEN); -#endif - } -} - - -void -MPIDI_Win_datatype_unmap(MPIDI_Datatype * dt) -{ - if (dt->map != &dt->__map) - MPL_free(dt->map);; -} diff --git a/src/mpid/pamid/src/onesided/mpid_win_accumulate.c b/src/mpid/pamid/src/onesided/mpid_win_accumulate.c deleted file mode 100644 index c833e60..0000000 --- a/src/mpid/pamid/src/onesided/mpid_win_accumulate.c +++ /dev/null @@ -1,364 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/onesided/mpid_win_accumulate.c - * \brief ??? - */ -#include "mpidi_onesided.h" -#include "mpidi_util.h" - - -void -MPIDI_WinAccumCB(pami_context_t context, - void * cookie, - const void * _msginfo, - size_t msginfo_size, - const void * sndbuf, - size_t sndlen, - pami_endpoint_t sender, - pami_recv_t * recv) -{ - MPID_assert(recv != NULL); - MPID_assert(sndbuf == NULL); - MPID_assert(msginfo_size == sizeof(MPIDI_Win_MsgInfo)); - MPID_assert(_msginfo != NULL); - const MPIDI_Win_MsgInfo * msginfo = (const MPIDI_Win_MsgInfo*)_msginfo; - - int null=0; - pami_type_t pami_type; - pami_data_function pami_op; - MPIDI_Datatype_to_pami(msginfo->type, &pami_type, msginfo->op, &pami_op, &null); - -#ifdef TRACE_ON - void * buf = msginfo->addr; - unsigned* ibuf = (unsigned*)buf; - double * dbuf = (double *)buf; - TRACE_ERR("New accum msg: len=%zu type=%x op=%x l-buf=%p *(int*)buf=0x%08x *(double*)buf=%g\n", sndlen, msginfo->type, msginfo->op, buf, *ibuf, *dbuf); - TRACE_ERR(" PAMI: type=%p op=%p\n", pami_type, pami_op); -#endif - - MPID_assert(recv != NULL); - *recv = zero_recv_parms; - recv->cookie = NULL; - recv->local_fn = NULL; - recv->addr = msginfo->addr; - recv->type = pami_type; - recv->offset = 0; - recv->data_fn = pami_op; - recv->data_cookie = NULL; -} - - -static pami_result_t -MPIDI_Accumulate(pami_context_t context, - void * _req) -{ - MPIDI_Win_request *req = (MPIDI_Win_request*)_req; - pami_result_t rc; - void *map; - pami_send_t params; - - params = zero_send_parms; - params.send.header.iov_len = sizeof(MPIDI_Win_MsgInfo); - params.send.dispatch = MPIDI_Protocols_WinAccum; - params.send.dest = req->dest; - params.events.cookie = req; - params.events.remote_fn = MPIDI_Win_DoneCB; - - struct MPIDI_Win_sync* sync = &req->win->mpid.sync; - TRACE_ERR("Start index=%u/%d l-addr=%p r-base=%p r-offset=%zu (sync->started=%u sync->complete=%u)\n", - req->state.index, req->target.dt.num_contig, req->buffer, req->win->mpid.info[req->target.rank].base_addr, req->offset, sync->started, sync->complete); - while (req->state.index < req->target.dt.num_contig) { - if (sync->started > sync->complete + MPIDI_Process.rma_pending) - { - TRACE_ERR("Bailing out; index=%u/%d sync->started=%u sync->complete=%u\n", - req->state.index, req->target.dt.num_contig, sync->started, sync->complete); - return PAMI_EAGAIN; - } - ++sync->started; - - - params.send.header.iov_base = &(((MPIDI_Win_MsgInfo *)req->accum_headers)[req->state.index]); - params.send.data.iov_len = req->target.dt.map[req->state.index].DLOOP_VECTOR_LEN; - params.send.data.iov_base = req->buffer + req->state.local_offset; - -#ifdef TRACE_ON - void * buf = params.send.data.iov_base; - unsigned* ibuf = (unsigned*)buf; - double * dbuf = (double *)buf; - TRACE_ERR(" Sub index=%u bytes=%zu l-offset=%zu r-addr=%p l-buf=%p *(int*)buf=0x%08x *(double*)buf=%g\n", - req->state.index, params.send.data.iov_len, req->state.local_offset, req->accum_headers[req->state.index].addr, buf, *ibuf, *dbuf); -#endif - /** sync->total will be updated with every RMA and the complete - will not change till that RMA has completed. In the meanwhile - the rest of the RMAs will have memory leaks */ - if (req->target.dt.num_contig - req->state.index == 1) { - rc = PAMI_Send(context, &params); - MPID_assert(rc == PAMI_SUCCESS); - return PAMI_SUCCESS; - } else { - rc = PAMI_Send(context, &params); - MPID_assert(rc == PAMI_SUCCESS); - req->state.local_offset += params.send.data.iov_len; - ++req->state.index; - } - } - - - return PAMI_SUCCESS; -} - - -/** - * \brief MPI-PAMI glue for MPI_ACCUMULATE function - * - * According to the MPI Specification: - * - * Each datatype argument must be a predefined datatype or - * a derived datatype, where all basic components are of the - * same predefined datatype. Both datatype arguments must be - * constructed from the same predefined datatype. - * - * \param[in] origin_addr Source buffer - * \param[in] origin_count Number of datatype elements - * \param[in] origin_datatype Source datatype - * \param[in] target_rank Destination rank (target) - * \param[in] target_disp Displacement factor in target buffer - * \param[in] target_count Number of target datatype elements - * \param[in] target_datatype Destination datatype - * \param[in] op Operand to perform - * \param[in] win Window - * \return MPI_SUCCESS - */ -#undef FUNCNAME -#define FUNCNAME MPID_Accumulate -#undef FCNAME -#define FCNAME MPIU_QUOTE(FUNCNAME) -int -MPID_Accumulate(const void *origin_addr, - int origin_count, - MPI_Datatype origin_datatype, - int target_rank, - MPI_Aint target_disp, - int target_count, - MPI_Datatype target_datatype, - MPI_Op op, - MPIR_Win *win) -{ - int mpi_errno = MPI_SUCCESS; - int shm_locked = 0; - MPIDI_Win_request *req = MPL_calloc0(1, MPIDI_Win_request); - req->win = win; - if(win->mpid.request_based != 1) - req->type = MPIDI_WIN_REQUEST_ACCUMULATE; - else { - req->type = MPIDI_WIN_REQUEST_RACCUMULATE; - req->req_handle = win->mpid.rreq; - req->req_handle->mpid.win_req = req; - } - - if(win->mpid.sync.origin_epoch_type == win->mpid.sync.target_epoch_type && - win->mpid.sync.origin_epoch_type == MPID_EPOTYPE_REFENCE){ - win->mpid.sync.origin_epoch_type = MPID_EPOTYPE_FENCE; - win->mpid.sync.target_epoch_type = MPID_EPOTYPE_FENCE; - } - - if(win->mpid.sync.origin_epoch_type == MPID_EPOTYPE_NONE || - win->mpid.sync.origin_epoch_type == MPID_EPOTYPE_POST){ - MPIU_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, - return mpi_errno, "**rmasync"); - } - - req->offset = target_disp * win->mpid.info[target_rank].disp_unit; -#ifdef __BGQ__ - /* PAMI limitation as it doesnt permit VA of 0 to be passed into - * memregion create, so we must pass base_va of heap computed from - * an SPI call instead. So the target offset must be adjusted */ - if (req->win->create_flavor == MPI_WIN_FLAVOR_DYNAMIC) - req->offset -= (size_t)req->win->mpid.info[target_rank].base_addr; -#endif - - if (origin_datatype == MPI_DOUBLE_INT) - { - MPIDI_Win_datatype_basic(origin_count*2, - MPI_DOUBLE, - &req->origin.dt); - MPIDI_Win_datatype_basic(target_count*2, - MPI_DOUBLE, - &req->target.dt); - } - else if (origin_datatype == MPI_LONG_DOUBLE_INT) - { - MPIDI_Win_datatype_basic(origin_count*2, - MPI_LONG_DOUBLE, - &req->origin.dt); - MPIDI_Win_datatype_basic(target_count*2, - MPI_LONG_DOUBLE, - &req->target.dt); - } - else if (origin_datatype == MPI_LONG_INT) - { - MPIDI_Win_datatype_basic(origin_count*2, - MPI_LONG, - &req->origin.dt); - MPIDI_Win_datatype_basic(target_count*2, - MPI_LONG, - &req->target.dt); - } - else if (origin_datatype == MPI_SHORT_INT) - { - MPIDI_Win_datatype_basic(origin_count*2, - MPI_INT, - &req->origin.dt); - MPIDI_Win_datatype_basic(target_count*2, - MPI_INT, - &req->target.dt); - } - else - { - MPIDI_Win_datatype_basic(origin_count, - origin_datatype, - &req->origin.dt); - MPIDI_Win_datatype_basic(target_count, - target_datatype, - &req->target.dt); - } - - MPID_assert(req->origin.dt.size == req->target.dt.size); - - if ( (req->origin.dt.size == 0) || - (target_rank == MPI_PROC_NULL)) - { - if(req->req_handle) - MPIR_cc_set(req->req_handle->cc_ptr, 0); - else - MPL_free(req); - return MPI_SUCCESS; - } - - req->target.rank = target_rank; - - - if (req->origin.dt.contig) - { - req->buffer_free = 0; - req->buffer = (void *) ((uintptr_t) origin_addr + req->origin.dt.true_lb); - } - else - { - req->buffer_free = 1; - req->buffer = MPL_malloc(req->origin.dt.size); - MPID_assert(req->buffer != NULL); - int mpi_errno = 0; - mpi_errno = MPIR_Localcopy(origin_addr, - origin_count, - origin_datatype, - req->buffer, - req->origin.dt.size, - MPI_CHAR); - MPID_assert(mpi_errno == MPI_SUCCESS); - } - - - pami_result_t rc; - pami_task_t task = MPID_VCR_GET_LPID(win->comm_ptr->vcr, target_rank); - if (win->mpid.sync.origin_epoch_type == MPID_EPOTYPE_START && - !MPIDI_valid_group_rank(task, win->mpid.sync.sc.group)) - { - MPIU_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, - return mpi_errno, "**rmasync"); - } - - rc = PAMI_Endpoint_create(MPIDI_Client, task, 0, &req->dest); - MPID_assert(rc == PAMI_SUCCESS); - - - MPIDI_Win_datatype_map(&req->target.dt); - - if (win->create_flavor == MPI_WIN_FLAVOR_SHARED) - { - MPI_User_function *uop; - void *base, *dest_addr; - int disp_unit; - int len, one; - - ++win->mpid.sync.total; - base = win->mpid.info[target_rank].base_addr; - disp_unit = win->mpid.info[target_rank].disp_unit; - dest_addr = (char *) base + disp_unit * target_disp; - - MPIDU_Datatype_get_size_macro(origin_datatype, len); - - uop = MPIR_OP_HDL_TO_FN(op); - one = 1; - - (*uop)((void *) origin_addr, dest_addr, &one, &origin_datatype); - - - MPL_free(req); - ++win->mpid.sync.complete; - - } else { /* non-shared */ - { - win->mpid.sync.total += req->target.dt.num_contig; - MPI_Datatype basic_type = MPI_DATATYPE_NULL; - MPIDU_Datatype_get_basic_type(origin_datatype, basic_type); - /* MPIDU_Datatype_get_basic_type() doesn't handle the struct types */ - if ((origin_datatype == MPI_FLOAT_INT) || - (origin_datatype == MPI_DOUBLE_INT) || - (origin_datatype == MPI_LONG_INT) || - (origin_datatype == MPI_SHORT_INT) || - (origin_datatype == MPI_LONG_DOUBLE_INT)) - { - MPID_assert(basic_type == MPI_DATATYPE_NULL); - basic_type = origin_datatype; - } - MPID_assert(basic_type != MPI_DATATYPE_NULL); - - unsigned index; - MPIDI_Win_MsgInfo * headers = MPL_calloc0(req->target.dt.num_contig, MPIDI_Win_MsgInfo); - req->accum_headers = headers; - for (index=0; index < req->target.dt.num_contig; ++index) { - headers[index].addr = win->mpid.info[target_rank].base_addr + req->offset + - (size_t)req->target.dt.map[index].DLOOP_VECTOR_BUF; - headers[index].req = req; - headers[index].win = win; - headers[index].type = basic_type; - headers[index].op = op; - } - - } - - /* The pamid one-sided design requires context post in order to handle the - * case where the number of pending rma operation exceeds the - * 'PAMID_RMA_PENDING' threshold. When there are too many pending requests the - * work function remains on the context post queue (by returning PAMI_EAGAIN) - * so that the next time the context is advanced the work function will be - * invoked again. - * - * TODO - When context post is not required it would be better to attempt a - * direct context operation and then fail over to using context post if - * the rma pending threshold has been reached. This would result in - * better latency for one-sided operations. - */ - PAMI_Context_post(MPIDI_Context[0], &req->post_request, MPIDI_Accumulate, req); - - } -fn_fail: - return mpi_errno; -} diff --git a/src/mpid/pamid/src/onesided/mpid_win_allocate.c b/src/mpid/pamid/src/onesided/mpid_win_allocate.c deleted file mode 100644 index f42e3a3..0000000 --- a/src/mpid/pamid/src/onesided/mpid_win_allocate.c +++ /dev/null @@ -1,99 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/onesided/mpid_win_create.c - * \brief ??? - */ -#include "mpidi_onesided.h" - -/** - * \brief MPI-PAMI glue for MPI_Win_allocate function - * - * Create a window object. Allocates a MPIR_Win object and initializes it, - * then allocates the collective info array, initalizes our entry, and - * performs an Allgather to distribute/collect the rest of the array entries. - * On each process, it allocates memory of at least size bytes, returns a - * pointer to it, and returns a window object that can be used by all processes - * in comm to * perform RMA operations. The returned memory consists of size - * bytes local to each process, starting at address base_ptr and is associated - * with the window as if the user called 'MPI_Win_create' on existing memory. - * The size argument may be different at each process and size = 0 is valid; - * however, a library might allocate and expose more memory in order to create - * a fast, globally symmetric allocation. - * Input Parameters: - * \param[in] size size of window in bytes (nonnegative integer) - * \param[in] disp_unit local unit size for displacements, in bytes (positive integer) - * \param[in] info info argument (handle)) - * \param[in] comm_ptr Communicator (handle) - * \param[out] base_ptr - base address of the window in local memory - * \param[out] win_ptr window object returned by the call (handle) - * \return MPI_SUCCESS, MPI_ERR_ARG, MPI_ERR_COMM, MPI_ERR_INFO. MPI_ERR_OTHER, - * MPI_ERR_SIZE - */ -int -MPID_Win_allocate(MPI_Aint size, - int disp_unit, - MPIR_Info * info, - MPIR_Comm * comm_ptr, - void *base_ptr, - MPIR_Win ** win_ptr) -{ - int mpi_errno = MPI_SUCCESS; - int rc = MPI_SUCCESS; - MPIR_Errflag_t errflag = MPIR_ERR_NONE; - void *baseP; - static char FCNAME[] = "MPID_Win_allocate"; - MPIDI_Win_info *winfo; - MPIR_Win *win; - int rank; - - rc=MPIDI_Win_init(size,disp_unit,win_ptr, info, comm_ptr, MPI_WIN_FLAVOR_ALLOCATE, MPI_WIN_UNIFIED); - win = *win_ptr; - - if (size > 0) { - baseP = MPL_malloc(size); - #ifndef MPIDI_NO_ASSERT - MPID_assert(baseP != NULL); - #else - MPIU_ERR_CHKANDJUMP((baseP == NULL), mpi_errno, MPI_ERR_BUFFER, "**bufnull"); - #endif - - } else if (size == 0) { - baseP = NULL; - } else { - MPIU_ERR_CHKANDSTMT(size >=0 , mpi_errno, MPI_ERR_SIZE, - return mpi_errno, "**rmasize"); - } - - win->base = baseP; - rank = comm_ptr->rank; - winfo = &win->mpid.info[rank]; - winfo->base_addr = baseP; - winfo->win = win; - winfo->disp_unit = disp_unit; - - rc= MPIDI_Win_allgather(size,win_ptr); - if (rc != MPI_SUCCESS) - return rc; - *(void**) base_ptr = (void *) win->base; - mpi_errno = MPIR_Barrier_impl(comm_ptr, &errflag); - - fn_fail: - return mpi_errno; -} - diff --git a/src/mpid/pamid/src/onesided/mpid_win_allocate_shared.c b/src/mpid/pamid/src/onesided/mpid_win_allocate_shared.c deleted file mode 100644 index dcafece..0000000 --- a/src/mpid/pamid/src/onesided/mpid_win_allocate_shared.c +++ /dev/null @@ -1,561 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/onesided/mpid_win_allocate_shared.c - * \brief - */ -#include "mpidi_onesided.h" -#include <sys/shm.h> -#include <sys/ipc.h> -#include <sys/stat.h> -#include <fcntl.h> -#include <sys/mman.h> - -#undef FUNCNAME -#define FUNCNAME MPID_Win_allocate_shared -#undef FCNAME -#define FCNAME MPIU_QUOTE(FUNCNAME) - -#define SHM_KEY_TAIL 0xfdbc /* If this value is changed, change */ - /* in PMD (mp_pmd.c) as well. */ - -extern int mpidi_dynamic_tasking; -#define MPIDI_PAGESIZE_MASK(y) (~(((MPI_Aint)y)-1)) -#define MPIDI_ROUND_UP_PAGESIZE(x,y) ((((MPI_Aint)x)+(~MPIDI_PAGESIZE_MASK(y))) & MPIDI_PAGESIZE_MASK(y)) -#define ALIGN_BOUNDARY 128 /* Align data structures to cache line */ -#define PAD_SIZE(s) (ALIGN_BOUNDARY - (sizeof(s) & (ALIGN_BOUNDARY-1))) - -int CheckRankOnNode(MPIR_Comm * comm_ptr,int *onNode ) { - int comm_size, i; - int mpi_errno = PAMI_SUCCESS; - - comm_size = comm_ptr->local_size; - - *onNode = 1; - - for (i = 0; i < comm_size; i++) { - if (comm_ptr->intranode_table[i] == -1) { - *onNode = 0; - break; - } - } - - if (*onNode== 0) { - MPIU_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_CONFLICT, - return mpi_errno, "**rmaconflict"); - } - - return mpi_errno; -} - -int CheckSpaceType(MPIR_Win **win_ptr, MPIR_Info *info,int *noncontig) { - int mpi_errno=MPI_SUCCESS; - /* Check if we are allowed to allocate space non-contiguously */ - if (info != NULL) { - int alloc_shared_nctg_flag = 0; - char alloc_shared_nctg_value[MPI_MAX_INFO_VAL+1]; - MPIR_Info_get_impl(info, "alloc_shared_noncontig", MPI_MAX_INFO_VAL, - alloc_shared_nctg_value, &alloc_shared_nctg_flag); - if (alloc_shared_nctg_flag == 1) { - if (!strncmp(alloc_shared_nctg_value, "true", strlen("true"))) - (*win_ptr)->mpid.info_args.alloc_shared_noncontig = 1; - if (!strncmp(alloc_shared_nctg_value, "false", strlen("false"))) - (*win_ptr)->mpid.info_args.alloc_shared_noncontig = 0; - } - } - *noncontig = (*win_ptr)->mpid.info_args.alloc_shared_noncontig; - return mpi_errno; -} - -int GetPageSize(void *addr, ulong *pageSize) -{ - pid_t pid; - FILE *fp; - char Line[201],A1[50],A2[50]; - char fileName[100]; - char A3[50],A4[50]; - int i=0; - char *t1,*t2; - #ifndef REDHAT - char a='-'; - char *search = &a; - #endif - void *beg, *end; - int found=0; - int ps; - int j,k; - - *pageSize = 0; - pid = getpid(); - sprintf(fileName,"/proc/%d/smaps",pid); - /* linux-2.6.29 or greater, KernelPageSize in /proc/pid/smaps includes 4K, 16 MB */ - TRACE_ERR("fileName = %s addr=%p\n",fileName,addr); - fp = fopen(fileName,"r"); - if (fp == NULL) { - TRACE_ERR("fileName = %s open failed errno=%d\n",fileName,errno); - return errno; - } - while(fgets(Line,200,fp)) { - i++; - sscanf(Line,"%s %s %s %s \n",A1,A2,A3,A4); - if ((found == 1) && (memcmp(A1,"KernelPageSize",14)==0)) { - j=atoi(A2); - if ((A3[0]=='k') || (A3[0]=='K')) - k=1024; - else if ((A3[0]=='m') || (A3[0]=='M')) - k=1048576; - else if ((A3[0]=='g') || (A3[0]=='G')) - k=0x40000000; /* 1 GB */ - else { - TRACE_ERR("ERROR unrecognized unit A3=%s\n",A3); - break; - } - *pageSize = (ulong)(j * k); - TRACE_ERR(" addr=%p pageSize=%ld %s(%d)\n", addr,*pageSize,__FILE__,__LINE__); - break; - } - if ((strlen(A2) == 4) && ((A2[0]=='r') || (A2[3]=='p'))) { - #ifndef REDHAT - t1=strtok(A1,search); - #else - t1=strtok(A1,"-"); - #endif - t2 = A1+strlen(t1)+1; - sscanf(t1,"%p \n",&beg); - sscanf(t2,"%p \n",&end); - if (((ulong) addr >= (ulong)beg) && ((ulong)addr <= (ulong)end)) { - found=1; - TRACE_ERR("found addr=%p i=%d between beg=%p and end=%p in %s\n", - addr,i,beg,end,fileName); - } - } - } - fclose(fp); - if (*pageSize == 0) { - ps = getpagesize(); - *pageSize = (ulong) ps; - TRACE_ERR("LinuxPageSize %p not in %s getpagesize=%ld\n", addr,fileName,*pageSize); - } - return 0; -} - -int -MPID_getSharedSegment_mmap(MPIR_Win * win) -{ - int rank, rc, fd; - int mpi_errno = MPI_SUCCESS; - MPIR_Errflag_t errflag = MPIR_ERR_NONE; - int first = 0; - - snprintf (win->mpid.shm->shm_key, 63, "/mpich.comm-%d.win_shared", win->comm_ptr->context_id); - rc = shm_open (win->mpid.shm->shm_key, O_RDWR | O_CREAT | O_EXCL, 0600); - if (0 == rc) - { - first = 1; - } else { - rc = shm_open (win->mpid.shm->shm_key, O_RDWR, 0); - MPIU_ERR_CHKANDJUMP((rc == -1), mpi_errno, MPI_ERR_RMA_SHARED, "**rmashared"); - } - - fd = rc; - rc = ftruncate (fd, win->mpid.shm->segment_len); - MPIU_ERR_CHKANDJUMP((rc == -1), mpi_errno, MPI_ERR_RMA_SHARED, "**rmashared"); - - win->mpid.shm->base_addr = mmap (NULL, win->mpid.shm->segment_len, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0); - MPIU_ERR_CHKANDJUMP((win->mpid.shm->base_addr == MAP_FAILED), mpi_errno, MPI_ERR_RMA_SHARED, "**rmashared"); - - close (fd); /* no longer needed */ - - /* set mutex_lock address and initialize it */ - win->mpid.shm->ctrl = (MPIDI_Win_shm_ctrl_t *) win->mpid.shm->base_addr; - if (1 == first) { - MPIDI_SHM_MUTEX_INIT(win); - } - - mpi_errno = MPIR_Barrier_impl(win->comm_ptr, &errflag); - MPIU_ERR_CHKANDJUMP(errflag, mpi_errno, MPI_ERR_OTHER, "**coll_fail"); - - win->mpid.shm->allocated = 1; - -fn_exit: - return mpi_errno; - /* --BEGIN ERROR HANDLING-- */ -fn_fail: - shm_unlink (win->mpid.shm->shm_key); - goto fn_exit; - /* --END ERROR HANDLING-- */ -} - -int -MPID_getSharedSegment_sysv(MPIR_Win * win) -{ - int mpi_errno = MPI_SUCCESS; - MPIR_Errflag_t errflag = MPIR_ERR_NONE; - uint32_t shm_key; - int rank; - char *cp; - int shm_flag = IPC_CREAT | IPC_EXCL | S_IRUSR | S_IWUSR; - - shm_key = (uint32_t) -1; - - rank = win->comm_ptr->rank; - - if (rank == 0) { -#ifdef DYNAMIC_TASKING - /* generate an appropriate key */ - if (!mpidi_dynamic_tasking) { - cp = getenv("MP_I_PMD_PID"); - if (cp) { - shm_key = atoi(cp); - shm_key = shm_key & 0x07ffffff; - shm_key = shm_key | 0x80000000; - } else { - cp = getenv("MP_PARTITION"); - if (cp ) { - shm_key = atol(cp); - shm_key = (shm_key << 16) + SHM_KEY_TAIL; - } else { - TRACE_ERR("ERROR MP_PARTITION not set \n"); - } - } - } else { - cp = getenv("MP_I_KEY_RANGE"); - if (cp) { - sscanf(cp, "0x%x", &shm_key); - shm_key = shm_key | 0x80; - } else { - TRACE_ERR("ERROR MP_I_KEY_RANGE not set \n"); - } - } -#else - cp = getenv("MP_I_PMD_PID"); - if (cp) { - shm_key = atoi(cp); - shm_key = shm_key & 0x07ffffff; - shm_key = shm_key | 0x80000000; - } else { - cp = getenv("MP_PARTITION"); - if (cp ) { - shm_key = atol(cp); - shm_key = (shm_key << 16); -#ifdef SHMCC_KEY_TAIL - shm_key += SHMCC_KEY_TAIL; -#endif - } else { - TRACE_ERR("ERROR MP_PARTITION not set \n"); - } - } -#endif - - MPID_assert(shm_key != -1); - - win->mpid.shm->shm_id = shmget(shm_key, win->mpid.shm->segment_len, shm_flag); - MPIU_ERR_CHKANDJUMP((win->mpid.shm->shm_id == -1), mpi_errno, MPI_ERR_RMA_SHARED, "**rmashared"); - - win->mpid.shm->base_addr = (void *) shmat(win->mpid.shm->shm_id,0,0); - MPIU_ERR_CHKANDJUMP((win->mpid.shm->base_addr == (void*) -1), mpi_errno,MPI_ERR_BUFFER, "**bufnull"); - - /* set mutex_lock address and initialize it */ - win->mpid.shm->ctrl = (MPIDI_Win_shm_ctrl_t *) win->mpid.shm->base_addr; - MPIDI_SHM_MUTEX_INIT(win); - - /* successfully created shm segment - shared the key with other tasks */ - mpi_errno = MPIR_Bcast_impl((void *) &shm_key, sizeof(int), MPI_CHAR, 0, win->comm_ptr, &errflag); - - } else { /* task other than task 0 */ - mpi_errno = MPIR_Bcast_impl((void *) &shm_key, sizeof(int), MPI_CHAR, 0, win->comm_ptr, &errflag); - MPIU_ERR_CHKANDJUMP(errflag, mpi_errno, MPI_ERR_OTHER, "**coll_fail"); - - win->mpid.shm->shm_id = shmget(shm_key, 0, 0); - if (win->mpid.shm->shm_id != -1) { /* shm segment is available */ - win->mpid.shm->base_addr = (void *) shmat(win->mpid.shm->shm_id,0,0); - } - win->mpid.shm->ctrl = (MPIDI_Win_shm_ctrl_t *) win->mpid.shm->base_addr; - } - - win->mpid.shm->allocated = 1; - -fn_exit: - return mpi_errno; - /* --BEGIN ERROR HANDLING-- */ -fn_fail: - goto fn_exit; - /* --END ERROR HANDLING-- */ -} - -int -MPID_getSharedSegment(MPI_Aint size, - int disp_unit, - MPIR_Comm * comm_ptr, - MPIR_Win **win_ptr, - MPI_Aint * pSize, - int * noncontig) -{ - int mpi_errno = MPI_SUCCESS; - int i, comm_size, rank; - MPIR_Errflag_t errflag = MPIR_ERR_NONE; - MPI_Aint pageSize,pageSize2, len,new_size; - MPIR_Win *win; - int padSize; - void *base_pp; - - win = *win_ptr; - comm_size = win->comm_ptr->local_size; - rank = win->comm_ptr->rank; - - if (mpi_errno) MPIU_ERR_POP(mpi_errno); - - GetPageSize((void *) win_ptr, (ulong *) &pageSize); - *pSize = pageSize; - win->mpid.shm->segment_len = 0; - - if (comm_size == 1) { - /* Do not use shared memory when there is only one rank on the node */ - - /* 'size' must not be < 0 */ - MPIU_ERR_CHKANDSTMT(size < 0 , mpi_errno, MPI_ERR_SIZE,return mpi_errno, "**rmasize"); - - /* The beginning of the heap allocation contains a control block - * before the data begins. - */ - new_size = MPIDI_ROUND_UP_PAGESIZE((sizeof(MPIDI_Win_shm_ctrl_t)+ ((comm_size+1) * sizeof(void *))),pageSize); - - if (size > 0) { - if (*noncontig) - new_size += MPIDI_ROUND_UP_PAGESIZE(size,pageSize); - else - new_size += size; - } - - base_pp = MPL_malloc(new_size); - MPID_assert(base_pp !=NULL); - if (mpi_errno) MPIU_ERR_POP(mpi_errno); - - win->mpid.shm->segment_len = new_size; - win->mpid.shm->base_addr = base_pp; - if (size !=0) { - win->mpid.info[rank].base_addr = (void *)((MPI_Aint) base_pp + MPIDI_ROUND_UP_PAGESIZE((sizeof(MPIDI_Win_shm_ctrl_t) + ((comm_size+1) * sizeof(void *))),pageSize)); - } else { - win->mpid.info[rank].base_addr = NULL; - } - win->base = win->mpid.info[rank].base_addr; - win->mpid.info[rank].base_size = size; - - /* set mutex_lock address and initialize it */ - win->mpid.shm->ctrl = (MPIDI_Win_shm_ctrl_t *) win->mpid.shm->base_addr; - MPIDI_SHM_MUTEX_INIT(win); - OPA_fetch_and_add_int((OPA_int_t *) &win->mpid.shm->ctrl->shm_count,(int) 1); - - } else { - /* allocate a temporary buffer to gather the 'size' of each buffer on - * the node to determine the amount of shared memory to allocate - */ - MPI_Aint * size_array; - size_array = MPL_malloc (2*comm_size*sizeof(MPI_Aint)); - size_array[rank] = (MPI_Aint) size; - mpi_errno = MPIR_Allgather_impl(MPI_IN_PLACE, 0, MPI_DATATYPE_NULL, - size_array, 1 * sizeof(MPI_Aint), MPI_BYTE, - (*win_ptr)->comm_ptr, &errflag); - if (mpi_errno) { - MPL_free(size_array); - MPIU_ERR_POP(mpi_errno); - } - - /* calculate total number of bytes needed */ - MPI_Aint actual_size; - win->mpid.info[0].base_addr = NULL; - for (i = 0; i < comm_size; ++i) { - win->mpid.info[i].base_size = size_array[i]; - - actual_size = (*noncontig)?MPIDI_ROUND_UP_PAGESIZE(size_array[i],pageSize):size_array[i]; - - win->mpid.shm->segment_len += actual_size; - - /* Save the OFFSET to each rank's private shared memory area. This - * will be added to the BASE ADDRESS of the entire shared memory - * allocation to determine the virtual address. - */ - if (i < comm_size-1) - win->mpid.info[i+1].base_addr = - (void *) ((uintptr_t)win->mpid.info[i].base_addr + actual_size); - } - - /* The beginning of the shared memory allocation contains a control - * block before the data begins. - */ - win->mpid.shm->segment_len += MPIDI_ROUND_UP_PAGESIZE((sizeof(MPIDI_Win_shm_ctrl_t) + ((comm_size+1) * sizeof(void *))),pageSize); - - /* Get the shared segment which includes the control block header and - * data buffer - possibly padded if non-contiguous. - */ -#ifdef USE_SYSV_SHM - mpi_errno = MPID_getSharedSegment_sysv(win); -#elif USE_MMAP_SHM - mpi_errno = MPID_getSharedSegment_mmap(win); -#else - MPID_Abort(NULL, MPI_ERR_RMA_SHARED, -1, "RMA shared segment error"); -#endif - if (mpi_errno) MPIU_ERR_POP(mpi_errno); - - /* assign base address here */ - /* compute the base addresses of each process within the shared memory segment */ - /* will be done after MPI_Allgather in MID_Win_allocate_shared */ - win->base = (void *) ((long) win->mpid.shm->base_addr + (long ) MPIDI_ROUND_UP_PAGESIZE((sizeof(MPIDI_Win_shm_ctrl_t) + ((comm_size+1) * sizeof(void *))),pageSize)); - - - MPL_free(size_array); - - /* increment the shared counter */ - OPA_fetch_and_add_int((OPA_int_t *) &win->mpid.shm->ctrl->shm_count,(int) 1); - - /* wait for all ranks complete */ - while((int) win->mpid.shm->ctrl->shm_count != comm_size) MPIDI_QUICKSLEEP; - } - -fn_exit: - return mpi_errno; - /* --BEGIN ERROR HANDLING-- */ -fn_fail: - goto fn_exit; - /* --END ERROR HANDLING-- */ - -} - -/** - * \brief MPI-PAMI glue for MPI_Win_allocate_shared function - * - * Create a window object. Allocates a MPIR_Win object and initializes it, - * then allocates the collective info array, initalizes our entry, and - * performs an Allgather to distribute/collect the rest of the array entries. - * On each process, it allocates memory of at least size bytes that is shared - * among all processes in comm, and returns a pointer to the locally allocated - * segment in base_ptr that can be used for load/store accesses on the calling - * process. The locally allocated memory can be the target of load/store accessses - * by remote processes; the base pointers for other processes can be queried using - * the function 'MPI_Win_shared_query'. - * - * The call also returns a window object that can be used by all processes in comm - * to perform RMA operations. The size argument may be different at each process - * and size = 0 is valid. It is the user''s responsibility to ensure that the - * communicator comm represents a group of processes that can create a shared - * memory segment that can be accessed by all processes in the group. The - * allocated memory is contiguous across process ranks unless the info key - * alloc_shared_noncontig is specified. Contiguous across process ranks means that - * the first address in the memory segment of process i is consecutive with the - * last address in the memory segment of process i - 1. This may enable the user - * to calculate remote address offsets with local information only. - * - * Input Parameters: - * \param[in] size size of window in bytes (nonnegative integer) - * \param[in] disp_unit local unit size for displacements, in bytes (positive integer) - * \param[in] info info argument (handle)) - * \param[in] comm_ptr intra-Communicator (handle) - * \param[out] base_ptr address of local allocated window segment - * \param[out] win_ptr window object returned by the call (handle) - * \return MPI_SUCCESS, MPI_ERR_ARG, MPI_ERR_COMM, MPI_ERR_INFO. MPI_ERR_OTHER, - * MPI_ERR_SIZE - * - * win->mpid.shm->base_addr \* return address from shmat *\ - * win->base \* address for data starts here == win->mpid.shm->base_addr *\ - * \* + space for mutex_lock and shm_count *\ - * - */ -int -MPID_Win_allocate_shared(MPI_Aint size, - int disp_unit, - MPIR_Info * info, - MPIR_Comm * comm_ptr, - void *base_ptr, - MPIR_Win ** win_ptr) -{ - int mpi_errno = MPI_SUCCESS; - MPIR_Errflag_t errflag = MPIR_ERR_NONE; - int onNode = 0; - MPIR_Win *win = NULL; - int rank, prev_size; - - MPIDI_Win_info *winfo; - int comm_size,i; - int noncontig=FALSE; - MPI_Aint pageSize=0; - - /* Verify all ranks are on-node */ - mpi_errno=CheckRankOnNode(comm_ptr,&onNode); - if (mpi_errno) MPIU_ERR_POP(mpi_errno); - MPIU_ERR_CHKANDJUMP((onNode == 0), mpi_errno, MPI_ERR_RMA_SHARED, "**rmashared"); - - /* Initialize the window */ - mpi_errno =MPIDI_Win_init(size,disp_unit,win_ptr, info, comm_ptr, MPI_WIN_FLAVOR_SHARED, MPI_WIN_UNIFIED); - if (mpi_errno) MPIU_ERR_POP(mpi_errno); - win = *win_ptr; - win->mpid.shm = MPL_malloc(sizeof(MPIDI_Win_shm_t)); - MPID_assert(win->mpid.shm != NULL); - memset(win->mpid.shm, 0, sizeof(MPIDI_Win_shm_t)); - - rank = comm_ptr->rank; - win->mpid.info[rank].win = win; - win->mpid.info[rank].disp_unit = disp_unit; - - mpi_errno=CheckSpaceType(win_ptr,info,&noncontig); - comm_size = (*win_ptr)->comm_ptr->local_size; - mpi_errno = MPID_getSharedSegment(size, disp_unit,comm_ptr, win_ptr, &pageSize, &noncontig); - if (mpi_errno != MPI_SUCCESS) - return mpi_errno; - - mpi_errno = MPIDI_Win_allgather(size,win_ptr); - if (mpi_errno != MPI_SUCCESS) - return mpi_errno; - - win->mpid.info[0].base_addr = win->base; - char *cur_base = win->base; - prev_size=win->mpid.info[0].base_size; - for (i = 1; i < comm_size; ++i) { - if (win->mpid.info[i].base_size) { - if (prev_size) { - if (noncontig) { - /* Round up to next page size */ - win->mpid.info[i].base_addr = - (void *) ((MPI_Aint) cur_base + (MPI_Aint) MPIDI_ROUND_UP_PAGESIZE(prev_size,pageSize)); - } else { - win->mpid.info[i].base_addr = (void *) ((MPI_Aint) cur_base + (MPI_Aint) prev_size); - } - } else - win->mpid.info[i].base_addr = (void *) ((MPI_Aint) cur_base); - prev_size=win->mpid.info[i].base_size; - cur_base = win->mpid.info[i].base_addr; - } else { - win->mpid.info[i].base_addr = NULL; - } - } - - - *(void**) base_ptr = (void *) win->mpid.info[rank].base_addr; - - mpi_errno = MPIR_Barrier_impl(comm_ptr, &errflag); -fn_exit: - return mpi_errno; - /* --BEGIN ERROR HANDLING-- */ -fn_fail: - if (win != NULL) - if (win->mpid.shm != NULL) - MPL_free(win->mpid.shm); - goto fn_exit; - /* --END ERROR HANDLING-- */ - -} - diff --git a/src/mpid/pamid/src/onesided/mpid_win_attach.c b/src/mpid/pamid/src/onesided/mpid_win_attach.c deleted file mode 100644 index aae21a4..0000000 --- a/src/mpid/pamid/src/onesided/mpid_win_attach.c +++ /dev/null @@ -1,58 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/onesided/mpid_win_attach.c - * \brief attaches a local memory region beginning at base for remote - * access within the given window. - */ -#include "mpidi_onesided.h" - -/** - * \brief MPI-PAMI glue for MPI_Win_attach function - * - * Attaches a local memory region beginning at base for remote access - * within the given window. - * - * \param[in] win shared memory window object - * \param[in] base initial address of memory to be attached - * \param[in] length length of memory to be attached in bytes - * \return MPI_SUCCESS, MPI_ERR_RMA_FLAVOR - * - */ - -int -MPID_Win_attach(MPIR_Win *win, void *base, MPI_Aint size) -{ - int mpi_errno = MPI_SUCCESS; - static char FCNAME[] = "MPID_Win_attach"; - MPIU_ERR_CHKANDSTMT((win->create_flavor != MPI_WIN_FLAVOR_DYNAMIC), mpi_errno, - MPI_ERR_RMA_FLAVOR, return mpi_errno, "**rmaflavor"); - - - /* no op, all memory is exposed, the user is responsible for */ - /* ensuring that MPI_WIN_ATTACH at the target has returned */ - /* before a process attempts to target that memory with an */ - /* RMA call */ - - -fn_exit: - return mpi_errno; -fn_fail: - goto fn_exit; -} - diff --git a/src/mpid/pamid/src/onesided/mpid_win_compare_and_swap.c b/src/mpid/pamid/src/onesided/mpid_win_compare_and_swap.c deleted file mode 100644 index bfb48a0..0000000 --- a/src/mpid/pamid/src/onesided/mpid_win_compare_and_swap.c +++ /dev/null @@ -1,164 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/onesided/mpid_win_compare_and_swap.c - * \brief ??? - */ -#include "mpidi_onesided.h" - -extern pami_result_t -MPIDI_Atomic (pami_context_t context, - void * _req); - -#ifndef __BGQ__ -static pami_result_t -MPIDI_Compare_and_swap_using_pami_rmw(pami_context_t context, - void * _req) -{ - MPIDI_Win_request *req = (MPIDI_Win_request*)_req; - pami_result_t rc; - int target_rank; - - MPID_assert(req != NULL); - target_rank = req->target.rank; - - pami_rmw_t params; - params=zero_rmw_parms; - params.dest=req->dest; - params.cookie=(void *)req; - params.done_fn=MPIDI_Win_DoneCB; - params.type = req->pami_datatype; - params.operation = PAMI_ATOMIC_FETCH_COMPARE_SET; - params.local=req->user_buffer; /*result*/ - params.remote=req->win->mpid.info[target_rank].base_addr + req->offset + (size_t)req->origin.dt.map[0].DLOOP_VECTOR_BUF; - params.value=req->buffer; /* replaced value with origin */ - params.test=req->compare_buffer; - - rc = PAMI_Rmw(context, &params); - MPID_assert(rc == PAMI_SUCCESS); - return rc; -} -#endif - -#undef FUNCNAME -#define FUNCNAME MPIDI_Compare_and_swap -#undef FCNAME -#define FCNAME MPIU_QUOTE(FUNCNAME) -int MPID_Compare_and_swap(const void *origin_addr, const void *compare_addr, - void *result_addr, MPI_Datatype datatype, int target_rank, - MPI_Aint target_disp, MPIR_Win *win) -{ - int mpi_errno = MPI_SUCCESS; - MPIDI_Win_request *req; - int shm_locked=0; - - if(win->mpid.sync.origin_epoch_type == win->mpid.sync.target_epoch_type && - win->mpid.sync.origin_epoch_type == MPID_EPOTYPE_REFENCE){ - win->mpid.sync.origin_epoch_type = MPID_EPOTYPE_FENCE; - win->mpid.sync.target_epoch_type = MPID_EPOTYPE_FENCE; - } - - if(win->mpid.sync.origin_epoch_type == MPID_EPOTYPE_NONE || - win->mpid.sync.origin_epoch_type == MPID_EPOTYPE_POST){ - MPIU_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, - return mpi_errno, "**rmasync"); - } - - if (target_rank == MPI_PROC_NULL) - { - return MPI_SUCCESS; - } - /* Check if datatype is a C integer, Fortran Integer, - logical, or byte, per the classes given on page 165. */ - MPIR_ERRTEST_TYPE_RMA_ATOMIC(datatype, mpi_errno); - - req = (MPIDI_Win_request *) MPL_calloc0(1, MPIDI_Win_request); - req->win = win; - req->type = MPIDI_WIN_REQUEST_COMPARE_AND_SWAP; - - req->offset = target_disp * win->mpid.info[target_rank].disp_unit; -#ifdef __BGQ__ - /* PAMI limitation as it doesnt permit VA of 0 to be passed into - * memregion create, so we must pass base_va of heap computed from - * an SPI call instead. So the target offset must be adjusted */ - if (req->win->create_flavor == MPI_WIN_FLAVOR_DYNAMIC) - req->offset -= (size_t)req->win->mpid.info[target_rank].base_addr; -#endif - - MPIDI_Win_datatype_basic(1, datatype, &req->origin.dt); - - if (req->origin.dt.size == 0) - { - MPL_free(req); - return MPI_SUCCESS; - } - - req->target.rank = target_rank; - req->buffer = (void *) ((uintptr_t) origin_addr + req->origin.dt.true_lb); - req->user_buffer = result_addr + req->origin.dt.true_lb; - req->compare_buffer = (void *) ((uintptr_t) compare_addr + req->origin.dt.true_lb); - - pami_result_t rc; - pami_task_t task = MPID_VCR_GET_LPID(win->comm_ptr->vcr, target_rank); - if (win->mpid.sync.origin_epoch_type == MPID_EPOTYPE_START && - !MPIDI_valid_group_rank(task, win->mpid.sync.sc.group)) - { - MPIU_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, - return mpi_errno, "**rmasync"); - } - - rc = PAMI_Endpoint_create(MPIDI_Client, task, 0, &req->dest); - MPID_assert(rc == PAMI_SUCCESS); - - MPIDI_Win_datatype_map(&req->origin.dt); - win->mpid.sync.total += 1; - - MPI_Datatype basic_type = MPI_DATATYPE_NULL; - MPIDU_Datatype_get_basic_type(datatype, basic_type); - MPID_assert(basic_type != MPI_DATATYPE_NULL); - req->origin.datatype=basic_type; - - /* The pamid one-sided design requires context post in order to handle the - * case where the number of pending rma operation exceeds the - * 'PAMID_RMA_PENDING' threshold. When there are too many pending requests the - * work function remains on the context post queue (by returning PAMI_EAGAIN) - * so that the next time the context is advanced the work function will be - * invoked again. - * - * TODO - When context post is not required it would be better to attempt a - * direct context operation and then fail over to using context post if - * the rma pending threshold has been reached. This would result in - * better latency for one-sided operations. - */ - -#ifndef __BGQ__ - MPI_Op null_op=0; - pami_data_function pami_op; - pami_type_t pami_type; - if(MPIDI_Datatype_is_pami_rmw_supported(basic_type, &pami_type, null_op, &pami_op) ) { - req->pami_datatype = pami_type; - PAMI_Context_post(MPIDI_Context[0], &req->post_request, MPIDI_Compare_and_swap_using_pami_rmw, req); - } else -#endif - { - PAMI_Context_post(MPIDI_Context[0], &req->post_request, MPIDI_Atomic, req); - } - -fn_fail: - return mpi_errno; -} diff --git a/src/mpid/pamid/src/onesided/mpid_win_create.c b/src/mpid/pamid/src/onesided/mpid_win_create.c deleted file mode 100644 index ae7a062..0000000 --- a/src/mpid/pamid/src/onesided/mpid_win_create.c +++ /dev/null @@ -1,213 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/onesided/mpid_win_create.c - * \brief ??? - */ -#include "mpidi_onesided.h" - -/***************************************************************************/ -/* */ -/* allocate win_ptr (MPIDI_Win) */ -/* update win structure except for base address */ -/* */ -/***************************************************************************/ - -int -MPIDI_Win_init( MPI_Aint length, - int disp_unit, - MPIR_Win **win_ptr, - MPIR_Info *info, - MPIR_Comm *comm_ptr, - int create_flavor, - int model) -{ - int mpi_errno=MPI_SUCCESS; - size_t rank, size; - MPIDI_Win_info *winfo; - static char FCNAME[] = "MPIDI_Win_init"; - - /* ----------------------------------------- */ - /* Setup the common sections of the window */ - /* ----------------------------------------- */ - MPIR_Win *win = (MPIR_Win*)MPIR_Handle_obj_alloc(&MPIR_Win_mem); - - MPIU_ERR_CHKANDSTMT(win == NULL, mpi_errno, MPI_ERR_NO_MEM, - return mpi_errno, "**nomem"); - - *win_ptr = win; - memset(&win->mpid, 0, sizeof(struct MPIDI_Win)); - win->comm_ptr = comm_ptr; MPIR_Comm_add_ref(comm_ptr); - size = comm_ptr->local_size; - rank = comm_ptr->rank; - - win->mpid.info = MPL_malloc(size * sizeof(struct MPIDI_Win_info)); - MPID_assert(win->mpid.info != NULL); - memset((void *) win->mpid.info,0,(size * sizeof(struct MPIDI_Win_info))); - winfo = &win->mpid.info[rank]; - win->errhandler = NULL; - win->base = NULL; - win->size = length; - win->disp_unit = disp_unit; - win->create_flavor = create_flavor; - win->model = model; - win->copyCreateFlavor = 0; - win->copyModel = 0; - win->attributes = NULL; - win->comm_ptr = comm_ptr; - if ((info != NULL) && ((int *)info != (int *) MPI_INFO_NULL)) { - mpi_errno= MPIDI_Win_set_info(win, info); - MPID_assert(mpi_errno == 0); - } - MPID_assert(mpi_errno == 0); - - - /* Initialize the info (hint) flags per window */ - win->mpid.info_args.no_locks = 0; - win->mpid.info_args.accumulate_ordering = - (MPIDI_ACCU_ORDER_RAR | MPIDI_ACCU_ORDER_RAW | MPIDI_ACCU_ORDER_WAR | MPIDI_ACCU_ORDER_WAW); - win->mpid.info_args.accumulate_ops = MPIDI_ACCU_SAME_OP_NO_OP; /*default */ - win->mpid.info_args.same_size = 0; - win->mpid.info_args.alloc_shared_noncontig = 0; - - win->copyDispUnit=0; - win->copySize=0; - winfo->memregion_used = 0; - winfo->disp_unit = disp_unit; - - return mpi_errno; -} - -/***************************************************************************/ -/* */ -/* MPIDI_Win_allgather */ -/* */ -/* registers memory with PAMI if possible */ -/* calls Allgather to gather the information from all members in win. */ -/* */ -/***************************************************************************/ -int -MPIDI_Win_allgather( MPI_Aint size, MPIR_Win **win_ptr ) -{ - int mpi_errno = MPI_SUCCESS; - MPIR_Errflag_t errflag=MPIR_ERR_NONE; - MPIR_Win *win; - int rank; - MPIR_Comm *comm_ptr; - size_t length_out = 0; - pami_result_t rc; - MPIDI_Win_info *winfo; - static char FCNAME[] = "MPIDI_Win_allgather"; - - win = *win_ptr; - comm_ptr = win->comm_ptr; - rank = comm_ptr->rank; - winfo = &win->mpid.info[rank]; - - if (size != 0 && win->create_flavor != MPI_WIN_FLAVOR_SHARED) - { -#ifndef USE_PAMI_RDMA - if (!MPIDI_Process.mp_s_use_pami_get) - { -#endif - /* --------------------------------------- */ - /* Setup the PAMI sections of the window */ - /* --------------------------------------- */ - rc = PAMI_Memregion_create(MPIDI_Context[0], win->mpid.info[rank].base_addr, win->size, &length_out, &winfo->memregion); -#ifdef USE_PAMI_RDMA - MPIU_ERR_CHKANDJUMP((rc != PAMI_SUCCESS), mpi_errno, MPI_ERR_OTHER, "**nomem"); - MPIU_ERR_CHKANDJUMP((win->size < length_out), mpi_errno, MPI_ERR_OTHER, "**nomem"); -#else - if (rc == PAMI_SUCCESS) - { - winfo->memregion_used = 1; - MPID_assert(win->size == length_out); - } - } -#endif - } - - mpi_errno = MPIR_Allgather_impl(MPI_IN_PLACE, - 0, - MPI_DATATYPE_NULL, - win->mpid.info, - sizeof(struct MPIDI_Win_info), - MPI_BYTE, - comm_ptr, - &errflag); - -fn_fail: - return mpi_errno; -} - - - -/** - * \brief MPI-PAMI glue for MPI_Win_create function - * - * Create a window object. Allocates a MPIR_Win object and initializes it, - * then allocates the collective info array, initalizes our entry, and - * performs an Allgather to distribute/collect the rest of the array entries. - * - * ON first call, initializes (registers) protocol objects for locking, - * get, and send operations to message layer. Also creates datatype to - * represent the rma_sends element of the collective info array, - * used later to synchronize epoch end events. - * - * \param[in] base Local window buffer - * \param[in] size Local window size - * \param[in] disp_unit Displacement unit size - * \param[in] info Window hints (not used) - * \param[in] comm_ptr Communicator - * \param[out] win_ptr Window - * \return MPI_SUCCESS, MPI_ERR_OTHER, or error returned from - * MPI_Comm_dup or MPI_Allgather. - */ -int -MPID_Win_create(void * base, - MPI_Aint size, - int disp_unit, - MPIR_Info * info, - MPIR_Comm * comm_ptr, - MPIR_Win ** win_ptr) -{ - int mpi_errno = MPI_SUCCESS; - MPIR_Errflag_t errflag = MPIR_ERR_NONE; - int rc = MPI_SUCCESS; - MPIR_Win *win; - size_t rank; - MPIDI_Win_info *winfo; - - rc=MPIDI_Win_init(size,disp_unit,win_ptr, info, comm_ptr, MPI_WIN_FLAVOR_CREATE, MPI_WIN_UNIFIED); - win = *win_ptr; - win->base = base; - rank = comm_ptr->rank; - winfo = &win->mpid.info[rank]; - winfo->base_addr = base; - winfo->win = win; - winfo->disp_unit = disp_unit; - - rc= MPIDI_Win_allgather(size,win_ptr); - if (rc != MPI_SUCCESS) - return rc; - - - mpi_errno = MPIR_Barrier_impl(comm_ptr, (MPIR_Errflag_t *) &errflag); - - return mpi_errno; -} diff --git a/src/mpid/pamid/src/onesided/mpid_win_create_dynamic.c b/src/mpid/pamid/src/onesided/mpid_win_create_dynamic.c deleted file mode 100644 index 6b989ac..0000000 --- a/src/mpid/pamid/src/onesided/mpid_win_create_dynamic.c +++ /dev/null @@ -1,90 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/onesided/mpid_win_create.c - * \brief ??? - */ -#include "mpidi_onesided.h" - -/** - * \brief MPI-PAMI glue for MPI_Win_create_dynamic function - * - * Create a window object. Allocates a MPIR_Win object and initializes it, - * then allocates the collective info array, initalizes our entry, and - * performs an Allgather to distribute/collect the rest of the array entries. - * The function returns a window win without memory attached. - * - * Input Parameters: - * \param[in] info info argument - * \param[in] comm intra-Communicator (handle) - * \param[out] win_ptr window object returned by the call (handle) - * \return MPI_SUCCESS, MPI_ERR_ARG, MPI_ERR_COMM, MPI_ERR_INFO. MPI_ERR_OTHER, - * MPI_ERR_SIZE - */ - -int -MPID_Win_create_dynamic( MPIR_Info * info, - MPIR_Comm * comm_ptr, - MPIR_Win ** win_ptr) -{ - int mpi_errno = MPI_SUCCESS; - MPIR_Errflag_t errflag = MPIR_ERR_NONE; - int rc = MPI_SUCCESS; - MPIDI_Win_info *winfo; - MPIR_Win *win; - int rank; - - rc=MPIDI_Win_init(0,1,win_ptr, info, comm_ptr, MPI_WIN_FLAVOR_DYNAMIC, MPI_WIN_UNIFIED); - win = *win_ptr; - - rank = comm_ptr->rank; - win->base = MPI_BOTTOM; - winfo = &win->mpid.info[rank]; - winfo->win = win; - -#ifdef __BGQ__ - void *base = NULL; - size_t length = 0; - Kernel_MemoryRegion_t memregion; - void *tmpbuf = MPL_malloc(sizeof(int)); - Kernel_CreateMemoryRegion(&memregion, tmpbuf, sizeof(int)); - //Reset base to base VA of local heap - base = memregion.BaseVa; - length = memregion.Bytes; - MPL_free(tmpbuf); - - if (length != 0) - { - size_t length_out = 0; - pami_result_t rc; - rc = PAMI_Memregion_create(MPIDI_Context[0], base, length, &length_out, &winfo->memregion); - MPID_assert(rc == PAMI_SUCCESS); - MPID_assert(length == length_out); - } - win->base = base; - winfo->base_addr = base; -#endif - - rc= MPIDI_Win_allgather(0,win_ptr); - if (rc != MPI_SUCCESS) - return rc; - - mpi_errno = MPIR_Barrier_impl(comm_ptr, (MPIR_Errflag_t *) &errflag); - - return mpi_errno; -} diff --git a/src/mpid/pamid/src/onesided/mpid_win_detach.c b/src/mpid/pamid/src/onesided/mpid_win_detach.c deleted file mode 100644 index 4a84052..0000000 --- a/src/mpid/pamid/src/onesided/mpid_win_detach.c +++ /dev/null @@ -1,53 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/onesided/mpid_win_detach.c - * \brief detaches a previously attached memory region beginning at - * base - */ -#include "mpidi_onesided.h" - -/** - * \brief MPI-PAMI glue for MPI_Win_detach function - * - * Detaches a previously attached memory beginning at base. - * The arguments base and win must match the arguments passed - * to a previous call to MPI_Win_attach. - * \param[in] win window object - * \param[in] base initial address of emmory to be detached - * \return MPI_SUCCESS, MPI_ERR_RMA_FLAVOR - * - */ - -int -MPID_Win_detach(MPIR_Win *win, const void *base) -{ - int mpi_errno = MPI_SUCCESS; - static char FCNAME[] = "MPID_Win_detach"; - MPIU_ERR_CHKANDSTMT((win->create_flavor != MPI_WIN_FLAVOR_DYNAMIC), mpi_errno, - MPI_ERR_RMA_FLAVOR, return mpi_errno, "**rmaflavor"); - - - /* no op, all memory is exposed */ - -fn_exit: - return mpi_errno; -fn_fail: - goto fn_exit; -} - diff --git a/src/mpid/pamid/src/onesided/mpid_win_fence.c b/src/mpid/pamid/src/onesided/mpid_win_fence.c deleted file mode 100644 index 8ca4732..0000000 --- a/src/mpid/pamid/src/onesided/mpid_win_fence.c +++ /dev/null @@ -1,71 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/onesided/mpid_win_fence.c - * \brief ??? - */ -#include "mpidi_onesided.h" - - -int -MPID_Win_fence(int assert, - MPIR_Win *win) -{ - int mpi_errno = MPI_SUCCESS; - MPIR_Errflag_t errflag = MPIR_ERR_NONE; - static char FCNAME[] = "MPID_Win_fence"; - - if(win->mpid.sync.origin_epoch_type != win->mpid.sync.target_epoch_type){ - MPIU_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, - return mpi_errno, "**rmasync"); - } - - if (!(assert & MPI_MODE_NOPRECEDE) && - win->mpid.sync.origin_epoch_type != MPID_EPOTYPE_FENCE && - win->mpid.sync.origin_epoch_type != MPID_EPOTYPE_REFENCE && - win->mpid.sync.origin_epoch_type != MPID_EPOTYPE_NONE) { - /* --BEGIN ERROR HANDLING-- */ - MPIU_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, - return mpi_errno, "**rmasync"); - /* --END ERROR HANDLING-- */ - } - - - struct MPIDI_Win_sync* sync = &win->mpid.sync; - MPID_PROGRESS_WAIT_WHILE(sync->total != sync->complete); - sync->total = 0; - sync->started = 0; - sync->complete = 0; - - if(assert & MPI_MODE_NOSUCCEED) - { - win->mpid.sync.origin_epoch_type = MPID_EPOTYPE_NONE; - win->mpid.sync.target_epoch_type = MPID_EPOTYPE_NONE; - } - else{ - win->mpid.sync.origin_epoch_type = MPID_EPOTYPE_REFENCE; - win->mpid.sync.target_epoch_type = MPID_EPOTYPE_REFENCE; - } - - if (!(assert & MPI_MODE_NOPRECEDE)) - { - mpi_errno = MPIR_Barrier_impl(win->comm_ptr, &errflag); - } - - return mpi_errno; -} diff --git a/src/mpid/pamid/src/onesided/mpid_win_fetch_and_op.c b/src/mpid/pamid/src/onesided/mpid_win_fetch_and_op.c deleted file mode 100644 index c3e6583..0000000 --- a/src/mpid/pamid/src/onesided/mpid_win_fetch_and_op.c +++ /dev/null @@ -1,378 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/onesided/mpid_win_fetch_and_op.c.c - * \brief ??? - */ -#include "mpidi_onesided.h" - -static pami_result_t -MPIDI_Fetch_and_op_using_pami_rmw(pami_context_t context, - void * _req) -{ - MPIDI_Win_request *req = (MPIDI_Win_request*)_req; - pami_result_t rc; - int target_rank; - - MPID_assert(req != NULL); - target_rank = req->target.rank; - - pami_rmw_t params; - params=zero_rmw_parms; - params.dest=req->dest; - params.cookie=(void *)req; - params.done_fn=MPIDI_Win_DoneCB; - params.type = req->pami_datatype; - params.operation = req->pami_op; - params.local=req->user_buffer; /*result*/ - params.remote=req->win->mpid.info[target_rank].base_addr + req->offset + (size_t)req->origin.dt.map[0].DLOOP_VECTOR_BUF; - params.value=req->buffer; /* replaced value with origin */ - - rc = PAMI_Rmw(context, &params); - MPID_assert(rc == PAMI_SUCCESS); - return rc; -} - - -void -MPIDI_WinAtomicCB(pami_context_t context, - void * cookie, - const void * _hdr, - size_t size, - const void * sndbuf, - size_t sndlen, - pami_endpoint_t sender, - pami_recv_t * recv) -{ - MPIDI_AtomicHeader_t *ahdr = (MPIDI_AtomicHeader_t *) _hdr; - MPID_assert (ahdr != NULL); - MPID_assert (sizeof(MPIDI_AtomicHeader_t) == size); - MPIDI_AtomicHeader_t ack_hdr = *ahdr; - - void *dest_addr = ahdr->remote_addr; - int len; - len = MPIDU_Datatype_get_basic_size (ahdr->datatype); - - if (ahdr->atomic_type == MPIDI_WIN_REQUEST_COMPARE_AND_SWAP) { - - //overwrite value with result in ack_hdr - MPIR_Memcpy(ack_hdr.buf, dest_addr, len); - - if (MPIR_Compare_equal (&ahdr->test, dest_addr, ahdr->datatype)) - MPIR_Memcpy(dest_addr, ahdr->buf, len); - } - else if (ahdr->atomic_type == MPIDI_WIN_REQUEST_FETCH_AND_OP) { - //overwrite value with result - MPIR_Memcpy(ack_hdr.buf, dest_addr, len); - - MPI_User_function *uop; - int one = 1; - uop = MPIR_OP_HDL_TO_FN(ahdr->op); - - if (ahdr->op == MPI_REPLACE) - MPIR_Memcpy(dest_addr, ahdr->buf, len); - else if (ahdr->op == MPI_NO_OP); - else - (*uop) ((void *)ahdr->buf, dest_addr, &one, &ahdr->datatype); - } - else - MPID_abort(); - - pami_send_immediate_t params = { - .dispatch = MPIDI_Protocols_WinAtomicAck, - .dest = sender, - .header = { - .iov_base = &ack_hdr, - .iov_len = sizeof(MPIDI_AtomicHeader_t), - }, - .data = { - .iov_base = NULL, - .iov_len = 0, - }, - .hints = {0}, - }; - - pami_result_t rc = PAMI_Send_immediate(context, &params); - MPID_assert(rc == PAMI_SUCCESS); -} - -void -MPIDI_WinAtomicAckCB(pami_context_t context, - void * cookie, - const void * _hdr, - size_t size, - const void * sndbuf, - size_t sndlen, - pami_endpoint_t sender, - pami_recv_t * recv) -{ - int len; - MPIDI_AtomicHeader_t *ahdr = (MPIDI_AtomicHeader_t *) _hdr; - //We have a valid result addr - if (ahdr->result_addr != NULL) { - len = MPIDU_Datatype_get_basic_size (ahdr->datatype); - MPIR_Memcpy(ahdr->result_addr, ahdr->buf, len); - } - - MPIDI_Win_DoneCB(context, ahdr->request_addr, PAMI_SUCCESS); -} - - -pami_result_t -MPIDI_Atomic (pami_context_t context, - void * _req) -{ - MPIDI_Win_request *req = (MPIDI_Win_request*)_req; - pami_result_t rc; - MPIDI_AtomicHeader_t atomic_hdr; - int len; - - len = MPIDU_Datatype_get_basic_size (req->origin.datatype); - assert(len <= MAX_ATOMIC_TYPE_SIZE); - if (req->buffer) - MPIR_Memcpy(atomic_hdr.buf, req->buffer, len); - if (req->type == MPIDI_WIN_REQUEST_COMPARE_AND_SWAP) - MPIR_Memcpy(atomic_hdr.test, req->compare_buffer, len); - - atomic_hdr.result_addr = req->user_buffer; - atomic_hdr.remote_addr = req->win->mpid.info[req->target.rank].base_addr + req->offset; - atomic_hdr.request_addr = req; - atomic_hdr.datatype = req->origin.datatype; - atomic_hdr.atomic_type = req->type; - atomic_hdr.op = req->op; - - struct MPIDI_Win_sync* sync = &req->win->mpid.sync; - MPID_assert (req->origin.dt.num_contig == 1); - ++sync->started; - - pami_send_immediate_t params = { - .dispatch = MPIDI_Protocols_WinAtomic, - .dest = req->dest, - .header = { - .iov_base = &atomic_hdr, - .iov_len = sizeof(MPIDI_AtomicHeader_t), - }, - .data = { - .iov_base = NULL, - .iov_len = 0, - }, - .hints = {0}, - }; - - rc = PAMI_Send_immediate(context, &params); - MPID_assert(rc == PAMI_SUCCESS); - return PAMI_SUCCESS; -} - - -#define FUNCNAME MPIDI_Fetch_and_op -#undef FCNAME -#define FCNAME MPIU_QUOTE(FUNCNAME) -int MPID_Fetch_and_op(const void *origin_addr, void *result_addr, - MPI_Datatype datatype, int target_rank, - MPI_Aint target_disp, MPI_Op op, MPIR_Win *win) -{ - int mpi_errno = MPI_SUCCESS; - MPIDI_Win_request *req; - int good_for_rmw=0; - int count = 1; - int shm_locked = 0; - - if(win->mpid.sync.origin_epoch_type == win->mpid.sync.target_epoch_type && - win->mpid.sync.origin_epoch_type == MPID_EPOTYPE_REFENCE){ - win->mpid.sync.origin_epoch_type = MPID_EPOTYPE_FENCE; win->mpid.sync.target_epoch_type = MPID_EPOTYPE_FENCE; - } - - if(win->mpid.sync.origin_epoch_type == MPID_EPOTYPE_NONE || - win->mpid.sync.origin_epoch_type == MPID_EPOTYPE_POST){ - MPIU_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, - return mpi_errno, "**rmasync"); - } - - pami_type_t pami_type; - pami_atomic_t pami_op; - - if (target_rank == MPI_PROC_NULL) - return MPI_SUCCESS; - - MPI_Datatype basic_type = MPI_DATATYPE_NULL; - MPIDU_Datatype_get_basic_type(datatype, basic_type); - if ((datatype == MPI_FLOAT_INT) || - (datatype == MPI_DOUBLE_INT) || - (datatype == MPI_LONG_INT) || - (datatype == MPI_SHORT_INT) || - (datatype == MPI_LONG_DOUBLE_INT)) - { - MPID_assert(basic_type == MPI_DATATYPE_NULL); - basic_type = datatype; - } - MPID_assert(basic_type != MPI_DATATYPE_NULL); - - if(MPIDI_Datatype_is_pami_rmw_supported(basic_type, &pami_type, op, &pami_op) ) { -#ifndef __BGQ__ - good_for_rmw = 1; -#endif - } else { - if((op == MPI_NO_OP) && (origin_addr == NULL) && (win->create_flavor != MPI_WIN_FLAVOR_SHARED) ) { - /* essentially a MPI_Get to result buffer */ - MPID_Get(result_addr, 1, datatype, target_rank, - target_disp, 1, datatype, win); - return 0; - } - } - - req = (MPIDI_Win_request *) MPL_calloc0(1, MPIDI_Win_request); - req->win = win; - req->type = MPIDI_WIN_REQUEST_FETCH_AND_OP; - - req->offset = target_disp * win->mpid.info[target_rank].disp_unit; -#ifdef __BGQ__ - /* PAMI limitation as it doesnt permit VA of 0 to be passed into - * memregion create, so we must pass base_va of heap computed from - * an SPI call instead. So the target offset must be adjusted */ - if (req->win->create_flavor == MPI_WIN_FLAVOR_DYNAMIC) - req->offset -= (size_t)req->win->mpid.info[target_rank].base_addr; -#endif - - if (datatype == MPI_DOUBLE_INT) - { - MPIDI_Win_datatype_basic(count*2, - MPI_DOUBLE, - &req->origin.dt); - } - else if (datatype == MPI_LONG_DOUBLE_INT) - { - MPIDI_Win_datatype_basic(count*2, - MPI_LONG_DOUBLE, - &req->origin.dt); - } - else if (datatype == MPI_LONG_INT) - { - MPIDI_Win_datatype_basic(count*2, - MPI_LONG, - &req->origin.dt); - } - else if (datatype == MPI_SHORT_INT) - { - MPIDI_Win_datatype_basic(count*2, - MPI_INT, - &req->origin.dt); - } - else - { - MPIDI_Win_datatype_basic(count, - datatype, - &req->origin.dt); - } - - - if (req->origin.dt.size == 0) - { - MPL_free(req); - return MPI_SUCCESS; - } - - req->target.rank = target_rank; - - req->compare_buffer = NULL; - req->pami_op = pami_op; - req->op = op; - req->pami_datatype = pami_type; - /* MPI_Fetch_and_op only supports predefined datatype */ - req->buffer = (void *) ((uintptr_t) origin_addr + req->origin.dt.true_lb); - req->user_buffer = result_addr + req->origin.dt.true_lb; - - pami_result_t rc; - pami_task_t task = MPID_VCR_GET_LPID(win->comm_ptr->vcr, target_rank); - if (win->mpid.sync.origin_epoch_type == MPID_EPOTYPE_START && - !MPIDI_valid_group_rank(task, win->mpid.sync.sc.group)) - { - MPIU_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, - return mpi_errno, "**rmasync"); - } - - rc = PAMI_Endpoint_create(MPIDI_Client, task, 0, &req->dest); - MPID_assert(rc == PAMI_SUCCESS); - - MPIDI_Win_datatype_map(&req->origin.dt); - win->mpid.sync.total += req->origin.dt.num_contig; - req->origin.datatype= basic_type; - - /* The pamid one-sided design requires context post in order to handle the - * case where the number of pending rma operation exceeds the - * 'PAMID_RMA_PENDING' threshold. When there are too many pending requests the - * work function remains on the context post queue (by returning PAMI_EAGAIN) - * so that the next time the context is advanced the work function will be - * invoked again. - * - * TODO - When context post is not required it would be better to attempt a - * direct context operation and then fail over to using context post if - * the rma pending threshold has been reached. This would result in - * better latency for one-sided operations. - */ - if(good_for_rmw) { - PAMI_Context_post(MPIDI_Context[0], &req->post_request, MPIDI_Fetch_and_op_using_pami_rmw, req); - } else { - PAMI_Context_post(MPIDI_Context[0], &req->post_request, MPIDI_Atomic, req); - - } - -fn_fail: - return mpi_errno; -} - - -int MPIDI_Datatype_is_pami_rmw_supported(MPI_Datatype datatype, pami_type_t *pami_type, MPI_Op op, pami_atomic_t *pami_op) -{ - int null=0; - MPI_Op null_op=0; - int rc = FALSE; - pami_data_function pami_data_fn; - - MPIDI_Datatype_to_pami(datatype, pami_type, op, &pami_data_fn, &null); - - if(*pami_type == PAMI_TYPE_SIGNED_INT || - *pami_type == PAMI_TYPE_UNSIGNED_INT || - *pami_type == PAMI_TYPE_SIGNED_LONG || - *pami_type == PAMI_TYPE_UNSIGNED_LONG || - *pami_type == PAMI_TYPE_SIGNED_LONG_LONG || - *pami_type == PAMI_TYPE_SIGNED_LONG_LONG) { - if(op == null_op) { - rc = TRUE; - } else if (op == MPI_SUM) { - *pami_op = PAMI_ATOMIC_FETCH_ADD; - rc = TRUE; - } else if (op == MPI_BOR) { - *pami_op = PAMI_ATOMIC_FETCH_OR; - rc = TRUE; - } else if (op == MPI_BAND) { - *pami_op = PAMI_ATOMIC_FETCH_AND; - rc = TRUE; - } else if (op == MPI_BXOR) { - *pami_op = PAMI_ATOMIC_FETCH_XOR; - rc = TRUE; - } else if (op == MPI_REPLACE) { - *pami_op = PAMI_ATOMIC_FETCH_SET; - rc = TRUE; - } else if (op == MPI_NO_OP) { - *pami_op = PAMI_ATOMIC_FETCH; - rc = TRUE; - } - } - return rc; -} diff --git a/src/mpid/pamid/src/onesided/mpid_win_flush.c b/src/mpid/pamid/src/onesided/mpid_win_flush.c deleted file mode 100644 index 370bbd5..0000000 --- a/src/mpid/pamid/src/onesided/mpid_win_flush.c +++ /dev/null @@ -1,180 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/onesided/mpid_win_flush.c - * \brief returns a new info object containing the hints of the window - * associated with win. - */ -#include "mpidi_onesided.h" - -/** - * \brief MPI-PAMI glue for MPI_Win_flush function - * - * The funcion can be called only within passive target epochs such as - * MPI_Win_lock, MPI_Win_unlock, MPI_Win_lock_all and MPI_Win_unlock_all. - * - * The function completes all outstanding RMA operations initialized by - * the calling process to a specified target rank on the given window. - * The operations are completed both at the origin and the target. - * - * \param[in] rank rank of target window - * \param[in] win window object - * \return MPI_SUCCESS, MPI_ERR_OTHER - */ - - -int -MPID_Win_flush(int rank, - MPIR_Win *win) -{ - int mpi_errno = MPI_SUCCESS; - struct MPIDI_Win_sync* sync; - static char FCNAME[] = "MPID_Win_flush"; - - if((win->mpid.sync.origin_epoch_type != MPID_EPOTYPE_LOCK) && - (win->mpid.sync.origin_epoch_type != MPID_EPOTYPE_LOCK_ALL)) - { - MPIU_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, - return mpi_errno, "**rmasync"); - } - sync = &win->mpid.sync; - MPID_PROGRESS_WAIT_DO_WHILE(sync->total != sync->complete); - sync->total = 0; - sync->started = 0; - sync->complete = 0; - - return mpi_errno; -} -/** - * \brief MPI-PAMI glue for MPI_Win_flush_all function - * - * The funcion can be called only within passive target epochs such as - * MPI_Win_lock, MPI_Win_unlock, MPI_Win_lock_all and MPI_Win_unlock_all. - * - * All RMA opertions issued by the calling process to any target on the - * given window prior to this call and in the given window will have - * completed both at the origin and the target when the call returns. - * - * \param[in] win window object - * \return MPI_SUCCESS, MPI_ERR_OTHER - */ - - -int -MPID_Win_flush_all(MPIR_Win *win) -{ - int mpi_errno = MPI_SUCCESS; - struct MPIDI_Win_sync* sync; - static char FCNAME[] = "MPID_Win_flush_all"; - - if((win->mpid.sync.origin_epoch_type != MPID_EPOTYPE_LOCK) && - (win->mpid.sync.origin_epoch_type != MPID_EPOTYPE_LOCK_ALL)) - { - MPIU_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, - return mpi_errno, "**rmasync"); - } - - sync = &win->mpid.sync; - MPID_PROGRESS_WAIT_WHILE(sync->total != sync->complete); - sync->total = 0; - sync->started = 0; - sync->complete = 0; - return mpi_errno; - -} - -/** - * \brief MPI-PAMI glue for MPI_Win_flush_local function - * - * The funcion can be called only within passive target epochs such as - * MPI_Win_lock, MPI_Win_unlock, MPI_Win_lock_all and MPI_Win_unlock_all. - * - * Locally completes at the origin all outstanding RMA operations initiated - * the cng process to the target rank on the given window. The user may - * reuse any buffers after this routine returns. - * - * It has been determined that the routine uses only counters for each window - * and not for each rank because the overhead of tracking each rank could be great - * if a window group contains a large number of ranks. - * - * \param[in] rank rank of target window - * \param[in] win window object - * \return MPI_SUCCESS, MPI_ERR_OTHER - */ - -int -MPID_Win_flush_local(int rank, MPIR_Win *win) -{ - int mpi_errno = MPI_SUCCESS; - struct MPIDI_Win_sync* sync; - static char FCNAME[] = "MPID_Win_flush_local"; - - if((win->mpid.sync.origin_epoch_type != MPID_EPOTYPE_LOCK) && - (win->mpid.sync.origin_epoch_type != MPID_EPOTYPE_LOCK_ALL)) - { - MPIU_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, - return mpi_errno, "**rmasync"); - } - sync = &win->mpid.sync; - MPID_PROGRESS_WAIT_WHILE(sync->total != sync->complete); - sync->total = 0; - sync->started = 0; - sync->complete = 0; - - return mpi_errno; -} - -/** - * \brief MPI-PAMI glue for MPI_Win_flush_local_all function - * - * The funcion can be called only within passive target epochs such as - * MPI_Win_lock, MPI_Win_unlock, MPI_Win_lock_all and MPI_Win_unlock_all. - * - * All RMA operations issued to any target prior to this call in this window - * will have copleted at the origin when this function returns. - * - * \param[in] win window object - * \return MPI_SUCCESS, MPI_ERR_OTHER - */ - -int -MPID_Win_flush_local_all(MPIR_Win *win) -{ - int mpi_errno = MPI_SUCCESS; - struct MPIDI_Win_sync* sync; - static char FCNAME[] = "MPID_Win_flush"; - - if((win->mpid.sync.origin_epoch_type != MPID_EPOTYPE_LOCK) && - (win->mpid.sync.origin_epoch_type != MPID_EPOTYPE_LOCK_ALL)) - { - MPIU_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, - return mpi_errno, "**rmasync"); - } - sync = &win->mpid.sync; - MPID_PROGRESS_WAIT_WHILE(sync->total != sync->complete); - sync->total = 0; - sync->started = 0; - sync->complete = 0; - - return mpi_errno; -} - - - - - diff --git a/src/mpid/pamid/src/onesided/mpid_win_free.c b/src/mpid/pamid/src/onesided/mpid_win_free.c deleted file mode 100644 index 2533706..0000000 --- a/src/mpid/pamid/src/onesided/mpid_win_free.c +++ /dev/null @@ -1,132 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/onesided/mpid_win_free.c - * \brief ??? - */ -#include "mpidi_onesided.h" -#include <sys/shm.h> -#include <sys/ipc.h> -#include <sys/stat.h> -#ifdef USE_MMAP_SHM -#include <sys/mman.h> -#endif - - -int MPIDI_SHM_Win_free(MPIR_Win **win_ptr) -{ - static char FCNAME[] = "MPID_SHM_Win_free"; - int rc; - int mpi_errno = MPI_SUCCESS; - - /* Free shared memory region */ - /* free shm_base_addrs that's only used for shared memory windows */ - if ((*win_ptr)->mpid.shm->allocated) { - OPA_fetch_and_add_int((OPA_int_t *) &((*win_ptr)->mpid.shm->ctrl->shm_count),-1); - while((*win_ptr)->mpid.shm->ctrl->shm_count !=0) MPIDI_QUICKSLEEP; - if ((*win_ptr)->comm_ptr->rank == 0) { - MPIDI_SHM_MUTEX_DESTROY(*win_ptr); - } -#ifdef USE_SYSV_SHM - mpi_errno = shmdt((*win_ptr)->mpid.shm->base_addr); - if ((*win_ptr)->comm_ptr->rank == 0) { - rc=shmctl((*win_ptr)->mpid.shm->shm_id,IPC_RMID,NULL); - MPIU_ERR_CHKANDJUMP((rc == -1), errno,MPI_ERR_RMA_SHARED, "**shmctl"); - } -#elif USE_MMAP_SHM - munmap ((*win_ptr)->mpid.shm->base_addr, (*win_ptr)->mpid.shm->segment_len); - if (0 == (*win_ptr)->comm_ptr->rank) shm_unlink ((*win_ptr)->mpid.shm->shm_key); -#else - MPID_Abort(NULL, MPI_ERR_RMA_SHARED, -1, "MPI_Win_free error"); -#endif - } else {/* one task on a node */ - MPL_free((*win_ptr)->mpid.shm->base_addr); - } - MPL_free((*win_ptr)->mpid.shm); - (*win_ptr)->mpid.shm = NULL; - - fn_fail: - return mpi_errno; -} - -/** - * \brief MPI-PAMI glue for MPI_Win_free function - * - * Release all references and free memory associated with window. - * - * \param[in,out] win Window - * \return MPI_SUCCESS or error returned from MPI_Barrier. - */ -#undef FUNCNAME -#define FUNCNAME MPID_Win_free -#undef FCNAME -#define FCNAME MPIU_QUOTE(FUNCNAME) -int -MPID_Win_free(MPIR_Win **win_ptr) -{ - int mpi_errno = MPI_SUCCESS; - - MPIR_Win *win = *win_ptr; - size_t rank = win->comm_ptr->rank; - MPIR_Errflag_t errflag = MPIR_ERR_NONE; - - if(win->mpid.sync.origin_epoch_type != win->mpid.sync.target_epoch_type || - (win->mpid.sync.origin_epoch_type != MPID_EPOTYPE_NONE && - win->mpid.sync.origin_epoch_type != MPID_EPOTYPE_REFENCE)){ - MPIU_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, return mpi_errno, "**rmasync"); - } - - mpi_errno = MPIR_Barrier_impl(win->comm_ptr, &errflag); - MPIU_ERR_CHKANDJUMP(errflag, mpi_errno, MPI_ERR_OTHER, "**mpi_bcast"); - - if (win->create_flavor == MPI_WIN_FLAVOR_SHARED) - mpi_errno=MPIDI_SHM_Win_free(win_ptr); - - - - if (win->create_flavor == MPI_WIN_FLAVOR_ALLOCATE) - MPL_free(win->base); - - struct MPIDI_Win_info *winfo = &win->mpid.info[rank]; -#ifdef USE_PAMI_RDMA - if (win->size != 0) - { - pami_result_t rc; - rc = PAMI_Memregion_destroy(MPIDI_Context[0], &winfo->memregion); - MPID_assert(rc == PAMI_SUCCESS); - } -#else - if ( (!MPIDI_Process.mp_s_use_pami_get) && (win->size != 0) && (winfo->memregion_used) ) - { - pami_result_t rc; - rc = PAMI_Memregion_destroy(MPIDI_Context[0], &winfo->memregion); - MPID_assert(rc == PAMI_SUCCESS); - } -#endif - - MPL_free(win->mpid.info); - if (win->mpid.work.msgQ) - MPL_free(win->mpid.work.msgQ); - - MPIR_Comm_release(win->comm_ptr, 0); - - MPIR_Handle_obj_free(&MPIR_Win_mem, win); - -fn_fail: - return mpi_errno; -} diff --git a/src/mpid/pamid/src/onesided/mpid_win_get.c b/src/mpid/pamid/src/onesided/mpid_win_get.c deleted file mode 100644 index f9cd4b3..0000000 --- a/src/mpid/pamid/src/onesided/mpid_win_get.c +++ /dev/null @@ -1,434 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/onesided/mpid_win_get.c - * \brief ??? - */ -#include "mpidi_onesided.h" -#include "mpidi_util.h" - -static inline int -MPIDI_Get_use_pami_rget(pami_context_t context, MPIDI_Win_request * req) -__attribute__((__always_inline__)); -#ifdef RDMA_FAILOVER -static inline int -MPIDI_Get_use_pami_get(pami_context_t context, MPIDI_Win_request * req) -__attribute__((__always_inline__)); -#endif - -static pami_result_t -MPIDI_Get(pami_context_t context, - void * _req) -{ - MPIDI_Win_request *req = (MPIDI_Win_request*)_req; - pami_result_t rc; - -#ifdef USE_PAMI_RDMA - rc = MPIDI_Get_use_pami_rget(context,req); -#else - if( (req->origin.memregion_used) && - (req->win->mpid.info[req->target.rank].memregion_used) ) - { - rc = MPIDI_Get_use_pami_rget(context,req); - } else { - rc = MPIDI_Get_use_pami_get(context,req); - } -#endif - if(rc == PAMI_EAGAIN) - return rc; - - return PAMI_SUCCESS; -} - - -static inline int -MPIDI_Get_use_pami_rget(pami_context_t context, MPIDI_Win_request * req) -{ - int use_typed_rdma = 0; - - if (!req->target.dt.contig || !req->origin.dt.contig) { - use_typed_rdma = 0; - if (MPIDI_Process.typed_onesided == 1) - use_typed_rdma = 1; - } - - if (use_typed_rdma) { - pami_result_t rc; - pami_rget_typed_t params; - /* params need to zero out to avoid passing garbage to PAMI */ - params=zero_rget_typed_parms; - - params.rma.dest=req->dest; - params.rma.hints.buffer_registered = PAMI_HINT_ENABLE; - params.rma.hints.use_rdma = PAMI_HINT_ENABLE; - params.rma.bytes = req->target.dt.size; - params.rma.cookie = req; - params.rma.done_fn = MPIDI_Win_DoneCB; - params.rdma.local.mr=&req->origin.memregion; - params.rdma.remote.mr=&req->win->mpid.info[req->target.rank].memregion; - params.rdma.remote.offset= req->offset; - params.rdma.local.offset = req->state.local_offset; - - params.type.local = *(pami_type_t *)(req->origin.dt.pointer->device_datatype); - params.type.remote = *(pami_type_t *)(req->target.dt.pointer->device_datatype); - - - rc = PAMI_Rget_typed(context, &params); - MPID_assert(rc == PAMI_SUCCESS); - } - else { - pami_result_t rc; - pami_rget_simple_t params; - - params=zero_rget_parms; - - params.rma.dest=req->dest; - params.rma.hints.buffer_registered = PAMI_HINT_ENABLE; - params.rma.hints.use_rdma = PAMI_HINT_ENABLE; - params.rma.bytes = 0; - params.rma.cookie = req; - params.rma.done_fn = MPIDI_Win_DoneCB; - params.rdma.local.mr=&req->origin.memregion; - params.rdma.remote.mr=&req->win->mpid.info[req->target.rank].memregion; - params.rdma.remote.offset= req->offset; - - struct MPIDI_Win_sync* sync = &req->win->mpid.sync; - TRACE_ERR("Start index=%u/%d l-addr=%p r-base=%p r-offset=%zu (sync->started=%u sync->complete=%u)\n", - req->state.index, req->target.dt.num_contig, req->buffer, req->win->mpid.info[req->target.rank].base_addr, req->offset, sync->started, sync->complete); - while (req->state.index < req->target.dt.num_contig) { - if (sync->started > sync->complete + MPIDI_Process.rma_pending) - { - TRACE_ERR("Bailing out; index=%u/%d sync->started=%u sync->complete=%u\n", - req->state.index, req->target.dt.num_contig, sync->started, sync->complete); - return PAMI_EAGAIN; - } - ++sync->started; - - params.rma.bytes = req->target.dt.map[req->state.index].DLOOP_VECTOR_LEN; - params.rdma.remote.offset = req->offset + (size_t)req->target.dt.map[req->state.index].DLOOP_VECTOR_BUF; - params.rdma.local.offset = req->state.local_offset; - -#ifdef TRACE_ON - unsigned* buf = (unsigned*)(req->buffer + params.rdma.local.offset); -#endif - TRACE_ERR(" Sub index=%u bytes=%zu l-offset=%zu r-offset=%zu buf=%p *(int*)buf=0x%08x\n", req->state.index, params.rma.bytes, params.rdma.local.offset, params.rdma.remote.offset, buf, *buf); - - /** sync->total will be updated with every RMA and the complete - will not change till that RMA has completed. In the meanwhile - the rest of the RMAs will have memory leaks */ - if (req->target.dt.num_contig - req->state.index == 1) { - rc = PAMI_Rget(context, &params); - MPID_assert(rc == PAMI_SUCCESS); - return PAMI_SUCCESS; - } else { - rc = PAMI_Rget(context, &params); - MPID_assert(rc == PAMI_SUCCESS); - req->state.local_offset += params.rma.bytes; - ++req->state.index; - } - } - } - return PAMI_SUCCESS; -} - - -#ifdef RDMA_FAILOVER -static inline int -MPIDI_Get_use_pami_get(pami_context_t context, MPIDI_Win_request * req) -{ - pami_result_t rc; - pami_get_simple_t params; - - params=zero_get_parms; - - params.rma.dest=req->dest; - params.rma.hints.use_rdma = PAMI_HINT_DEFAULT; -#ifndef OUT_OF_ORDER_HANDLING - params.rma.hints.no_long_header= 1, -#endif - params.rma.bytes = 0; - params.rma.cookie = req; - params.rma.done_fn = MPIDI_Win_DoneCB; - params.addr.local=req->buffer; - params.addr.remote= req->win->mpid.info[req->target.rank].base_addr; - - struct MPIDI_Win_sync* sync = &req->win->mpid.sync; - TRACE_ERR("Start index=%u/%d l-addr=%p r-base=%p r-offset=%zu (sync->started=%u sync->complete=%u)\n", - req->state.index, req->target.dt.num_contig, req->buffer, req->win->mpid.info[req->target.rank].base_addr, req->offset, sync->started, sync->complete); - while (req->state.index < req->target.dt.num_contig) { - if (sync->started > sync->complete + MPIDI_Process.rma_pending) - { - TRACE_ERR("Bailing out; index=%u/%d sync->started=%u sync->complete=%u\n", - req->state.index, req->target.dt.num_contig, sync->started, sync->complete); - return PAMI_EAGAIN; - } - ++sync->started; - - - params.rma.bytes = req->target.dt.map[req->state.index].DLOOP_VECTOR_LEN; - params.addr.local = req->buffer+req->state.local_offset; - params.addr.remote = req->win->mpid.info[req->target.rank].base_addr+ req->offset + (size_t)req->target.dt.map[req->state.index].DLOOP_VECTOR_BUF; - -#ifdef TRACE_ON - unsigned* buf = (unsigned*)(req->buffer + params.rdma.local.offset); -#endif - TRACE_ERR(" Sub index=%u bytes=%zu l-offset=%zu r-offset=%zu buf=%p *(int*)buf=0x%08x\n", - req->state.index, params.rma.bytes, params.rdma.local.offset, params.rdma.remote.offset, buf, *buf); - - /** sync->total will be updated with every RMA and the complete - will not change till that RMA has completed. In the meanwhile - the rest of the RMAs will have memory leaks */ - if (req->target.dt.num_contig - req->state.index == 1) { - rc = PAMI_Get(context, &params); - MPID_assert(rc == PAMI_SUCCESS); - return PAMI_SUCCESS; - } else { - rc = PAMI_Get(context, &params); - MPID_assert(rc == PAMI_SUCCESS); - req->state.local_offset += params.rma.bytes; - ++req->state.index; - } - } - return PAMI_SUCCESS; -} -#endif - - -/** - * \brief MPI-PAMI glue for MPI_GET function - * - * \param[in] origin_addr Source buffer - * \param[in] origin_count Number of datatype elements - * \param[in] origin_datatype Source datatype - * \param[in] target_rank Destination rank (target) - * \param[in] target_disp Displacement factor in target buffer - * \param[in] target_count Number of target datatype elements - * \param[in] target_datatype Destination datatype - * \param[in] win Window - * \return MPI_SUCCESS - */ -#undef FUNCNAME -#define FUNCNAME MPID_Get -#undef FCNAME -#define FCNAME MPIU_QUOTE(FUNCNAME) -int -MPID_Get(void *origin_addr, - int origin_count, - MPI_Datatype origin_datatype, - int target_rank, - MPI_Aint target_disp, - int target_count, - MPI_Datatype target_datatype, - MPIR_Win *win) -{ - - int mpi_errno = MPI_SUCCESS; - int shm_locked=0; - void *target_addr; - MPIDI_Win_request *req = MPL_calloc0(1, MPIDI_Win_request); - req->win = win; - if(win->mpid.request_based != 1) - req->type = MPIDI_WIN_REQUEST_GET; - else { - req->req_handle = win->mpid.rreq; - req->type = MPIDI_WIN_REQUEST_RGET; - req->req_handle->mpid.win_req = req; - } - - if(win->mpid.sync.origin_epoch_type == win->mpid.sync.target_epoch_type && - win->mpid.sync.origin_epoch_type == MPID_EPOTYPE_REFENCE){ - win->mpid.sync.origin_epoch_type = MPID_EPOTYPE_FENCE; - win->mpid.sync.target_epoch_type = MPID_EPOTYPE_FENCE; - } - - if(win->mpid.sync.origin_epoch_type == MPID_EPOTYPE_NONE || - win->mpid.sync.origin_epoch_type == MPID_EPOTYPE_POST){ - MPIU_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, - return mpi_errno, "**rmasync"); - } - - req->offset = target_disp * win->mpid.info[target_rank].disp_unit; -#ifdef __BGQ__ - /* PAMI limitation as it doesnt permit VA of 0 to be passed into - * memregion create, so we must pass base_va of heap computed from - * an SPI call instead. So the target offset must be adjusted */ - if (req->win->create_flavor == MPI_WIN_FLAVOR_DYNAMIC) - req->offset -= (size_t)req->win->mpid.info[target_rank].base_addr; -#endif - - MPIDI_Win_datatype_basic(origin_count, - origin_datatype, - &req->origin.dt); - MPIDI_Win_datatype_basic(target_count, - target_datatype, - &req->target.dt); - #ifndef MPIDI_NO_ASSERT - MPID_assert(req->origin.dt.size == req->target.dt.size); - #else - MPIU_ERR_CHKANDJUMP((req->origin.dt.size != req->target.dt.size), mpi_errno, MPI_ERR_SIZE, "**rmasize"); - #endif - - if ( (req->origin.dt.size == 0) || - (target_rank == MPI_PROC_NULL)) - { - if(req->req_handle) - MPIR_cc_set(req->req_handle->cc_ptr, 0); - else - MPL_free(req); - return MPI_SUCCESS; - } - - /* If the get is a local operation,or shared window do it here */ - if ((target_rank == win->comm_ptr->rank) || (win->create_flavor == MPI_WIN_FLAVOR_SHARED)) - { - if (target_rank == win->comm_ptr->rank) - target_addr = win->base + req->offset; - else - target_addr = win->mpid.info[target_rank].base_addr + req->offset; - - - /* The operation is not complete until the local copy is performed */ - mpi_errno = MPIR_Localcopy(target_addr, - target_count, - target_datatype, - origin_addr, - origin_count, - origin_datatype); - - /* The instant this completion counter is set to zero another thread - * may notice the change and begin freeing request resources. The - * thread executing the code in this function must not touch any - * portion of the request structure after decrementing the completion - * counter. - * - * See MPID_Request_free_inline() - */ - if(req->req_handle) - MPIR_cc_set(req->req_handle->cc_ptr, 0); - else - MPL_free(req); - return mpi_errno; - } - req->target.rank = target_rank; - - - /* Only pack the origin data if the origin is non-contiguous and we are using the simple PAMI_Rget. - * If we are using the typed PAMI_Rget_typed use the origin address as is, if we are using the simple - * PAMI_Rget with contiguous data use the origin address with the lower-bound adjustment. - */ - if (req->origin.dt.contig || (!req->origin.dt.contig && (MPIDI_Process.typed_onesided == 1))) - { - req->buffer_free = 0; - if ((req->origin.dt.contig && req->target.dt.contig && (MPIDI_Process.typed_onesided == 1)) || (!(MPIDI_Process.typed_onesided == 1))) // use simple rput - req->buffer = (void *) ((uintptr_t) origin_addr + req->origin.dt.true_lb); - else - req->buffer = (void *) ((uintptr_t) origin_addr); - } - else - { - req->buffer_free = 1; - req->buffer = MPL_malloc(req->origin.dt.size); - MPID_assert(req->buffer != NULL); - - MPIDU_Datatype_add_ref(req->origin.dt.pointer); - req->origin.addr = origin_addr; - req->origin.count = origin_count; - req->origin.datatype = origin_datatype; - req->origin.completed = 0; - - - - - } - - - pami_result_t rc; - pami_task_t task = MPID_VCR_GET_LPID(win->comm_ptr->vcr, target_rank); - if (win->mpid.sync.origin_epoch_type == MPID_EPOTYPE_START && - !MPIDI_valid_group_rank(task, win->mpid.sync.sc.group)) - { - MPIU_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, - return mpi_errno, "**rmasync"); - } - - rc = PAMI_Endpoint_create(MPIDI_Client, task, 0, &req->dest); - MPID_assert(rc == PAMI_SUCCESS); - -#ifdef USE_PAMI_RDMA - size_t length_out; - rc = PAMI_Memregion_create(MPIDI_Context[0], - req->buffer, - req->origin.dt.size, - &length_out, - &req->origin.memregion); - MPID_assert(rc == PAMI_SUCCESS); - MPID_assert(req->origin.dt.size == length_out); -#else - if(!MPIDI_Process.mp_s_use_pami_get) - { - size_t length_out; - rc = PAMI_Memregion_create(MPIDI_Context[0], - req->buffer, - req->origin.dt.size, - &length_out, - &req->origin.memregion); - if(rc == PAMI_SUCCESS) - { - req->origin.memregion_used = 1; - MPID_assert(req->origin.dt.size == length_out); - } - } -#endif - - - if ((!req->target.dt.contig || !req->origin.dt.contig) && (MPIDI_Process.typed_onesided == 1)) - /* If the datatype is non-contiguous and the PAMID typed_onesided optimization - * is enabled then we will be using the typed interface and will only make 1 call. - */ - win->mpid.sync.total = 1; - else { - MPIDI_Win_datatype_map(&req->target.dt); - win->mpid.sync.total += req->target.dt.num_contig; - } - - if ((MPIDI_Process.typed_onesided == 1) && (!req->target.dt.contig || !req->origin.dt.contig)) { - /* We will use the PAMI_Rget_typed call so we need to make sure any MPI_Type_free before the context - * executes the get does not free the MPIDU_Datatype, which would also free the associated PAMI datatype which - * is still needed for communication -- decrement the ref in the callback to allow the MPIR_Datatype - * to be freed once the PAMI communication has completed. - */ - MPIDU_Datatype_add_ref(req->origin.dt.pointer); - MPIDU_Datatype_add_ref(req->target.dt.pointer); - } - /* The pamid one-sided design requires context post in order to handle the - * case where the number of pending rma operation exceeds the - * 'PAMID_RMA_PENDING' threshold. When there are too many pending requests the - * work function remains on the context post queue (by returning PAMI_EAGAIN) - * so that the next time the context is advanced the work function will be - * invoked again. - * - * TODO - When context post is not required it would be better to attempt a - * direct context operation and then fail over to using context post if - * the rma pending threshold has been reached. This would result in - * better latency for one-sided operations. - */ - PAMI_Context_post(MPIDI_Context[0], &req->post_request, MPIDI_Get, req); - -fn_fail: - return mpi_errno; -} diff --git a/src/mpid/pamid/src/onesided/mpid_win_get_accumulate.c b/src/mpid/pamid/src/onesided/mpid_win_get_accumulate.c deleted file mode 100644 index d1a6ae0..0000000 --- a/src/mpid/pamid/src/onesided/mpid_win_get_accumulate.c +++ /dev/null @@ -1,586 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/onesided/mpid_win_accumulate.c - * \brief ??? - */ -#include "mpidi_onesided.h" - -static void -MPIDI_Fetch_data_op(const void * origin_addr, - int origin_count, - MPI_Datatype origin_datatype, - void * result_addr, - int target_rank, - MPI_Aint target_disp, - int target_count, - MPI_Datatype target_datatype, - MPI_Op op, - MPIR_Win *win) -{ - static char FCNAME[] = "MPIDI_Fetch_data_op"; - int shm_locked=0; - MPI_User_function *uop; - void *base, *dest_addr; - int disp_unit; - int len, one; - - base = win->mpid.info[target_rank].base_addr; - disp_unit = win->mpid.info[target_rank].disp_unit; - dest_addr = (char *) base + disp_unit * target_disp; - - MPIDU_Datatype_get_size_macro(origin_datatype, len); - MPIR_Memcpy(result_addr, dest_addr, len); - if (op != MPI_NO_OP) { - uop = MPIR_OP_HDL_TO_FN(op); - one = 1; - (*uop)((void *) origin_addr, dest_addr, &one, &origin_datatype); - } - - fn_fail: return; -} - - -static void -MPIDI_Win_GetAccSendAckDoneCB(pami_context_t context, - void * _info, - pami_result_t result) -{ - MPIDI_Win_GetAccMsgInfo *msginfo = (MPIDI_Win_GetAccMsgInfo *) _info; - MPL_free(msginfo->tptr); - MPL_free(msginfo); -} - -static void -MPIDI_Win_GetAccumSendAck(pami_context_t context, - void * _info, - pami_result_t result) -{ - static char FCNAME[] = "MPID_Win_GetAccumSendAck"; - MPIDI_Win_GetAccMsgInfo *msginfo = (MPIDI_Win_GetAccMsgInfo *) _info; - pami_result_t rc = PAMI_SUCCESS; - - //Copy from msginfo->addr to a contiguous buffer - char *buffer = NULL; - - buffer = MPL_malloc(msginfo->size); - MPID_assert(buffer != NULL); - - if (msginfo->num_contig == 1) - memcpy(buffer, msginfo->addr, msginfo->size); - else - { - int mpi_errno = 0; - mpi_errno = MPIR_Localcopy(msginfo->addr, - msginfo->count, - msginfo->type, - buffer, - msginfo->size, - MPI_CHAR); - MPID_assert(mpi_errno == MPI_SUCCESS); - } - //Schedule sends to source to result buffer and trigger completion - //callback there - pami_send_t params = { - .send = { - .header = { - .iov_base = msginfo, - .iov_len = sizeof(MPIDI_Win_GetAccMsgInfo), - }, - .dispatch = MPIDI_Protocols_WinGetAccumAck, - .dest = msginfo->src_endpoint, - }, - .events = { - .cookie = msginfo, - .local_fn = MPIDI_Win_GetAccSendAckDoneCB, //cleanup buffer - }, - }; - - msginfo->tptr = buffer; - - params.send.data.iov_len = msginfo->size; - params.send.data.iov_base = buffer; - - rc = PAMI_Send(context, &params); - MPID_assert(rc == PAMI_SUCCESS); - fn_fail: return; -} - -void -MPIDI_WinGetAccumCB(pami_context_t context, - void * cookie, - const void * _msginfo, - size_t msginfo_size, - const void * sndbuf, - size_t sndlen, - pami_endpoint_t sender, - pami_recv_t * recv) -{ - MPID_assert(recv != NULL); - MPID_assert(sndbuf == NULL); - MPID_assert(msginfo_size == sizeof(MPIDI_Win_GetAccMsgInfo)); - MPID_assert(_msginfo != NULL); - MPIDI_Win_GetAccMsgInfo * msginfo = (MPIDI_Win_GetAccMsgInfo *) - MPL_malloc(sizeof(MPIDI_Win_GetAccMsgInfo)); - - *msginfo = *(MPIDI_Win_GetAccMsgInfo *)_msginfo; - msginfo->src_endpoint = sender; - - int null=0; - pami_type_t pami_type; - pami_data_function pami_op; - MPI_Op op = msginfo->op; - - MPIDI_Datatype_to_pami(msginfo->type, &pami_type, op, &pami_op, &null); - - recv->addr = msginfo->addr; - recv->type = pami_type; - recv->offset = 0; - recv->data_fn = pami_op; - recv->data_cookie = NULL; - recv->local_fn = NULL; - recv->cookie = NULL; - - if (msginfo->counter == 0) - //We will now allocate a tempbuf, copy local contents and start a - //send - MPIDI_Win_GetAccumSendAck (context, msginfo, PAMI_SUCCESS); - else - MPL_free(msginfo); -} - -static void -MPIDI_Win_GetAccDoneCB(pami_context_t context, - void * cookie, - pami_result_t result) -{ - MPIDI_Win_request *req = (MPIDI_Win_request*)cookie; - ++req->win->mpid.sync.complete; - ++req->origin.completed; - - if (req->origin.completed == req->target.dt.num_contig + 1) - { - MPIR_Request * req_handle = req->req_handle; - - if (req->buffer_free) { - MPL_free(req->buffer); - req->buffer_free = 0; - } - if (req->accum_headers) - MPL_free(req->accum_headers); - - MPIDI_Win_datatype_unmap(&req->target.dt); - MPIDI_Win_datatype_unmap(&req->result.dt); - - if( req->type != MPIDI_WIN_REQUEST_RGET_ACCUMULATE ) { - if (req_handle) { - req_handle->mpid.win_req = NULL; - } - MPL_free(req); - } - /* The instant this completion counter is set to zero another thread - * may notice the change and begin freeing request resources. The - * thread executing the code in this function must not touch any - * portion of the request structure after decrementing the completion - * counter. - * - * See MPID_Request_free_inline() - */ - if(req_handle) - MPIR_cc_set(req_handle->cc_ptr, 0); - } - MPIDI_Progress_signal(); -} - -void -MPIDI_Win_GetAccAckDoneCB(pami_context_t context, - void * _msginfo, - pami_result_t result) -{ - MPIDI_Win_GetAccMsgInfo * msginfo =(MPIDI_Win_GetAccMsgInfo *)_msginfo; - MPIDI_Win_request *req = (MPIDI_Win_request *) msginfo->request; - - if (req->result_num_contig > 1) { - MPIR_Localcopy(req->result.addr, - req->result.count, - req->result.datatype, - msginfo->tptr, - msginfo->size, - MPI_CHAR); - MPL_free(msginfo->tptr); - } - MPL_free(msginfo); - - MPIDI_Win_GetAccDoneCB(context, req, result); -} - - -void -MPIDI_WinGetAccumAckCB(pami_context_t context, - void * cookie, - const void * _msginfo, - size_t msginfo_size, - const void * sndbuf, - size_t sndlen, - pami_endpoint_t sender, - pami_recv_t * recv) -{ - MPID_assert(recv != NULL); - MPID_assert(sndbuf == NULL); - MPID_assert(_msginfo != NULL); - MPIDI_Win_GetAccMsgInfo * msginfo =MPL_malloc(sizeof(MPIDI_Win_GetAccMsgInfo)); - *msginfo = *(const MPIDI_Win_GetAccMsgInfo *)_msginfo; - MPIDI_Win_request *req = (MPIDI_Win_request *) msginfo->request; - - msginfo->tptr = NULL; - recv->addr = req->result.addr; - if (req->result_num_contig > 1) - recv->addr = msginfo->tptr = MPL_malloc(msginfo->size); - - recv->type = PAMI_TYPE_BYTE; - recv->offset = 0; - recv->data_fn = PAMI_DATA_COPY; - recv->data_cookie = NULL; - recv->local_fn = MPIDI_Win_GetAccAckDoneCB; - recv->cookie = msginfo; -} - -static pami_result_t -MPIDI_Get_accumulate(pami_context_t context, - void * _req) -{ - MPIDI_Win_request *req = (MPIDI_Win_request*)_req; - pami_result_t rc; - - pami_send_t params = { - .send = { - .header = { - .iov_len = sizeof(MPIDI_Win_GetAccMsgInfo), - }, - .dispatch = MPIDI_Protocols_WinGetAccum, - .dest = req->dest, - }, - .events = { - .cookie = req, - .remote_fn = MPIDI_Win_GetAccDoneCB, //When all accumulates have - //completed remotely - //complete accumulate - }, - }; - - struct MPIDI_Win_sync* sync = &req->win->mpid.sync; - TRACE_ERR("Start index=%u/%d l-addr=%p r-base=%p r-offset=%zu (sync->started=%u sync->complete=%u)\n", - req->state.index, req->target.dt.num_contig, req->buffer, req->win->mpid.info[req->target.rank].base_addr, req->offset, sync->started, sync->complete); - while (req->state.index < req->target.dt.num_contig) { - if (sync->started > sync->complete + MPIDI_Process.rma_pending) - { - TRACE_ERR("Bailing out; index=%u/%d sync->started=%u sync->complete=%u\n", - req->state.index, req->target.dt.num_contig, sync->started, sync->complete); - return PAMI_EAGAIN; - } - ++sync->started; - - params.send.header.iov_base = &(((MPIDI_Win_GetAccMsgInfo *)req->accum_headers)[req->state.index]); - params.send.data.iov_len = req->target.dt.map[req->state.index].DLOOP_VECTOR_LEN; - params.send.data.iov_base = req->buffer + req->state.local_offset; - - if (req->target.dt.num_contig - req->state.index == 1) { - rc = PAMI_Send(context, &params); - MPID_assert(rc == PAMI_SUCCESS); - return PAMI_SUCCESS; - } else { - rc = PAMI_Send(context, &params); - MPID_assert(rc == PAMI_SUCCESS); - req->state.local_offset += params.send.data.iov_len; - ++req->state.index; - } - } - return PAMI_SUCCESS; -} - - -/** - * \brief MPI-PAMI glue for MPI_GET_ACCUMULATE function - * - * According to the MPI Specification: - * - * Each datatype argument must be a predefined datatype or - * a derived datatype, where all basic components are of the - * same predefined datatype. Both datatype arguments must be - * constructed from the same predefined datatype. - * - * \param[in] origin_addr Source buffer - * \param[in] origin_count Number of datatype elements - * \param[in] origin_datatype Source datatype - * \param[in] target_rank Destination rank (target) - * \param[in] target_disp Displacement factor in target buffer - * \param[in] target_count Number of target datatype elements - * \param[in] target_datatype Destination datatype - * \param[in] op Operand to perform - * \param[in] win Window - * \return MPI_SUCCESS - */ -#undef FUNCNAME -#define FUNCNAME MPID_Get_accumulate -#undef FCNAME -#define FCNAME MPIU_QUOTE(FUNCNAME) -int -MPID_Get_accumulate(const void * origin_addr, - int origin_count, - MPI_Datatype origin_datatype, - void * result_addr, - int result_count, - MPI_Datatype result_datatype, - int target_rank, - MPI_Aint target_disp, - int target_count, - MPI_Datatype target_datatype, - MPI_Op op, - MPIR_Win *win) -{ - int mpi_errno = MPI_SUCCESS; - - if (op == MPI_NO_OP) {//we just need to fetch data - if (win->create_flavor == MPI_WIN_FLAVOR_SHARED) { - win->mpid.sync.total++; - MPIDI_Fetch_data_op(origin_addr, origin_count, origin_datatype, - result_addr, target_rank, target_disp, - target_count, target_datatype, op, win); - ++win->mpid.sync.complete; - } else { - mpi_errno = MPID_Get(result_addr, - result_count, - result_datatype, - target_rank, - target_disp, - target_count, - target_datatype, - win); - } - return mpi_errno; - } - - MPIDI_Win_request *req; - req = MPL_calloc0(1, MPIDI_Win_request); - req->win = win; - if(win->mpid.request_based != 1) - req->type = MPIDI_WIN_REQUEST_GET_ACCUMULATE; - else { - req->req_handle = win->mpid.rreq; - req->type = MPIDI_WIN_REQUEST_RGET_ACCUMULATE; - req->req_handle->mpid.win_req = req; - } - req->offset = target_disp*win->mpid.info[target_rank].disp_unit; -#ifdef __BGQ__ - /* PAMI limitation as it doesnt permit VA of 0 to be passed into - * memregion create, so we must pass base_va of heap computed from - * an SPI call instead. So the target offset must be adjusted */ - if (req->win->create_flavor == MPI_WIN_FLAVOR_DYNAMIC) - req->offset -= (size_t)req->win->mpid.info[target_rank].base_addr; -#endif - - if(win->mpid.sync.origin_epoch_type == win->mpid.sync.target_epoch_type && - win->mpid.sync.origin_epoch_type == MPID_EPOTYPE_REFENCE){ - win->mpid.sync.origin_epoch_type = MPID_EPOTYPE_FENCE; - win->mpid.sync.target_epoch_type = MPID_EPOTYPE_FENCE; - } - - if(win->mpid.sync.origin_epoch_type == MPID_EPOTYPE_NONE || - win->mpid.sync.origin_epoch_type == MPID_EPOTYPE_POST){ - MPIU_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, - return mpi_errno, "**rmasync"); - } - - req->offset = target_disp * win->mpid.info[target_rank].disp_unit; - - if (origin_datatype == MPI_DOUBLE_INT) { - MPIDI_Win_datatype_basic(origin_count*2, - MPI_DOUBLE, - &req->origin.dt); - MPIDI_Win_datatype_basic(target_count*2, - MPI_DOUBLE, - &req->target.dt); - } - else if (origin_datatype == MPI_LONG_DOUBLE_INT) - { - MPIDI_Win_datatype_basic(origin_count*2, - MPI_LONG_DOUBLE, - &req->origin.dt); - MPIDI_Win_datatype_basic(target_count*2, - MPI_LONG_DOUBLE, - &req->target.dt); - } - else if (origin_datatype == MPI_LONG_INT) - { - MPIDI_Win_datatype_basic(origin_count*2, - MPI_LONG, - &req->origin.dt); - MPIDI_Win_datatype_basic(target_count*2, - MPI_LONG, - &req->target.dt); - } - else if (origin_datatype == MPI_SHORT_INT) - { - MPIDI_Win_datatype_basic(origin_count*2, - MPI_INT, - &req->origin.dt); - MPIDI_Win_datatype_basic(target_count*2, - MPI_INT, - &req->target.dt); - } - else - { - MPIDI_Win_datatype_basic(origin_count, - origin_datatype, - &req->origin.dt); - MPIDI_Win_datatype_basic(target_count, - target_datatype, - &req->target.dt); - } - - MPID_assert(req->origin.dt.size == req->target.dt.size); - - if ( (req->origin.dt.size == 0) || - (target_rank == MPI_PROC_NULL)) - { - if(req->req_handle) - MPIR_cc_set(req->req_handle->cc_ptr, 0); - else - MPL_free(req); - return MPI_SUCCESS; - } - - req->target.rank = target_rank; - - - if (req->origin.dt.contig) - { - req->buffer_free = 0; - req->buffer = (char*)origin_addr + req->origin.dt.true_lb; - } - else - { - req->buffer_free = 1; - req->buffer = MPL_malloc(req->origin.dt.size); - MPID_assert(req->buffer != NULL); - - int mpi_errno = 0; - mpi_errno = MPIR_Localcopy(origin_addr, - origin_count, - origin_datatype, - req->buffer, - req->origin.dt.size, - MPI_CHAR); - MPID_assert(mpi_errno == MPI_SUCCESS); - } - - pami_result_t rc; - pami_task_t task = MPID_VCR_GET_LPID(win->comm_ptr->vcr, target_rank); - if (win->mpid.sync.origin_epoch_type == MPID_EPOTYPE_START && - !MPIDI_valid_group_rank(task, win->mpid.sync.sc.group)) - { - MPIU_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, - return mpi_errno, "**rmasync"); - } - - rc = PAMI_Endpoint_create(MPIDI_Client, task, 0, &req->dest); - MPID_assert(rc == PAMI_SUCCESS); - - - MPIDI_Win_datatype_map(&req->target.dt); - req->result.addr = result_addr; - req->result.count = result_count; - req->result.datatype = result_datatype; - MPIDI_Win_datatype_basic(result_count, result_datatype, &req->result.dt); - MPIDI_Win_datatype_map(&req->result.dt); - req->result_num_contig = req->result.dt.num_contig; - if (win->create_flavor == MPI_WIN_FLAVOR_SHARED) - { - win->mpid.sync.total++; - MPIDI_Fetch_data_op(origin_addr, origin_count, origin_datatype, - result_addr, target_rank, target_disp, - target_count, target_datatype, op, win); - ++win->mpid.sync.complete; - - if (req->buffer_free) { - MPL_free(req->buffer); - MPL_free(req->user_buffer); - req->buffer_free = 0; - } - MPIDI_Win_datatype_unmap(&req->target.dt); - MPIDI_Win_datatype_unmap(&req->result.dt); - - if(req->req_handle) { - MPIR_cc_set(req->req_handle->cc_ptr, 0); - } else { - MPL_free(req); - } - } else { /* non-shared */ - //We wait for #messages depending on target and result_datatype - win->mpid.sync.total += (1 + req->target.dt.num_contig); - { - MPI_Datatype basic_type = MPI_DATATYPE_NULL; - MPIDU_Datatype_get_basic_type(origin_datatype, basic_type); - /* MPIDU_Datatype_get_basic_type() doesn't handle the struct types */ - if ((origin_datatype == MPI_FLOAT_INT) || - (origin_datatype == MPI_DOUBLE_INT) || - (origin_datatype == MPI_LONG_INT) || - (origin_datatype == MPI_SHORT_INT) || - (origin_datatype == MPI_LONG_DOUBLE_INT)) - { - MPID_assert(basic_type == MPI_DATATYPE_NULL); - basic_type = origin_datatype; - } - MPID_assert(basic_type != MPI_DATATYPE_NULL); - - unsigned index; - MPIDI_Win_GetAccMsgInfo * headers = MPL_calloc0(req->target.dt.num_contig, MPIDI_Win_GetAccMsgInfo); - req->accum_headers = headers; - for (index=0; index < req->target.dt.num_contig; ++index) { - headers[index].addr = win->mpid.info[target_rank].base_addr + req->offset + - (size_t)req->target.dt.map[index].DLOOP_VECTOR_BUF; - headers[index].req = req; - headers[index].win = win; - headers[index].type = basic_type; - headers[index].op = op; - headers[index].count = target_count; - headers[index].counter = index; - headers[index].num_contig = req->target.dt.num_contig; - headers[index].size = req->target.dt.size; - headers[index].request = req; - } - - } - - /* The pamid one-sided design requires context post in order to handle the - * case where the number of pending rma operation exceeds the - * 'PAMID_RMA_PENDING' threshold. When there are too many pending requests the - * work function remains on the context post queue (by returning PAMI_EAGAIN) - * so that the next time the context is advanced the work function will be - * invoked again. - * - * TODO - When context post is not required it would be better to attempt a - * direct context operation and then fail over to using context post if - * the rma pending threshold has been reached. This would result in - * better latency for one-sided operations. - */ - PAMI_Context_post(MPIDI_Context[0], &req->post_request, MPIDI_Get_accumulate, req); - } -fn_fail: - return mpi_errno; -} diff --git a/src/mpid/pamid/src/onesided/mpid_win_get_info.c b/src/mpid/pamid/src/onesided/mpid_win_get_info.c deleted file mode 100644 index 5768e99..0000000 --- a/src/mpid/pamid/src/onesided/mpid_win_get_info.c +++ /dev/null @@ -1,104 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/onesided/mpid_win_get_info.c - * \brief returns a new info object containing the hints of the window - * associated with win. - */ -#include "mpidi_onesided.h" -#include "mpir_info.h" - -/** - * \brief MPI-PAMI glue for MPI_WIN_GET_INFO function - * - * \param[in] win Window - * \param[in] info_p_p Info hint - * \return MPI_SUCCESS - */ -int MPIDI_Win_get_info(MPIR_Win *win, MPIR_Info **info_used) -{ - int mpi_errno = MPI_SUCCESS; - - - /* Populate the predefined info keys */ - if (win->mpid.info_args.no_locks) - mpi_errno = MPIR_Info_set_impl(*info_used, "no_locks", "true"); - else - mpi_errno = MPIR_Info_set_impl(*info_used, "no_locks", "false"); - MPID_assert(mpi_errno == MPI_SUCCESS); - - { -#define BUFSIZE 32 - char buf[BUFSIZE]; - int c = 0; - if (win->mpid.info_args.accumulate_ordering & MPIDI_ACCU_ORDER_RAR) - c += snprintf(buf+c, BUFSIZE-c, "%srar", (c > 0) ? "," : ""); - if (win->mpid.info_args.accumulate_ordering & MPIDI_ACCU_ORDER_RAW) - c += snprintf(buf+c, BUFSIZE-c, "%sraw", (c > 0) ? "," : ""); - if (win->mpid.info_args.accumulate_ordering & MPIDI_ACCU_ORDER_WAR) - c += snprintf(buf+c, BUFSIZE-c, "%swar", (c > 0) ? "," : ""); - if (win->mpid.info_args.accumulate_ordering & MPIDI_ACCU_ORDER_WAW) - c += snprintf(buf+c, BUFSIZE-c, "%swaw", (c > 0) ? "," : ""); - - if (c == 0) { - memcpy(&buf[0],"not set ",10); - } - MPIR_Info_set_impl(*info_used, "accumulate_ordering", buf); - MPID_assert(mpi_errno == MPI_SUCCESS); -#undef BUFSIZE - } - if (win->mpid.info_args.accumulate_ops == (unsigned) MPIDI_ACCU_OPS_SAME_OP) - mpi_errno = MPIR_Info_set_impl(*info_used, "accumulate_ops", "same_op"); - else - mpi_errno = MPIR_Info_set_impl(*info_used, "accumulate_ops", "same_op_no_op"); - - MPID_assert(mpi_errno == MPI_SUCCESS); - - if (win->create_flavor == MPI_WIN_FLAVOR_SHARED) { - if (win->mpid.info_args.alloc_shared_noncontig) - mpi_errno = MPIR_Info_set_impl(*info_used, "alloc_shared_noncontig", "true"); - else - mpi_errno = MPIR_Info_set_impl(*info_used, "alloc_shared_noncontig", "false"); - - MPID_assert(mpi_errno == MPI_SUCCESS); - } - else if (win->create_flavor == MPI_WIN_FLAVOR_ALLOCATE) { - if (win->mpid.info_args.same_size) - mpi_errno = MPIR_Info_set_impl(*info_used, "same_size", "true"); - else - mpi_errno = MPIR_Info_set_impl(*info_used, "same_size", "false"); - - MPID_assert(mpi_errno == MPI_SUCCESS); - } - return mpi_errno; -} - - -int -MPID_Win_get_info(MPIR_Win *win, - MPIR_Info **info_p) -{ - int mpi_errno = MPI_SUCCESS; - - /* Allocate an empty info object */ - mpi_errno = MPIR_Info_alloc(info_p); - MPID_assert(mpi_errno == MPI_SUCCESS); - mpi_errno = MPIDI_Win_get_info(win, info_p); - MPID_assert(mpi_errno == MPI_SUCCESS); - return MPI_SUCCESS; -} diff --git a/src/mpid/pamid/src/onesided/mpid_win_lock.c b/src/mpid/pamid/src/onesided/mpid_win_lock.c deleted file mode 100644 index 6d7897c..0000000 --- a/src/mpid/pamid/src/onesided/mpid_win_lock.c +++ /dev/null @@ -1,230 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/onesided/mpid_win_lock.c - * \brief ??? - */ -#include "mpidi_onesided.h" - -void -MPIDI_WinLockAck_post(pami_context_t context, - unsigned peer, - MPIR_Win * win); - - -void -MPIDI_WinLockAdvance(pami_context_t context, - MPIR_Win * win) -{ - struct MPIDI_Win_sync_lock* slock = &win->mpid.sync.lock; - struct MPIDI_Win_queue* q = &slock->local.requested; - - if ( - (q->head != NULL ) && - ( (slock->local.count == 0) || - ( - (slock->local.type == MPI_LOCK_SHARED) && - (q->head->type == MPI_LOCK_SHARED) - ) - ) - ) - { - struct MPIDI_Win_lock* lock = q->head; - q->head = lock->next; - if (q->head == NULL) - q->tail = NULL; - - ++slock->local.count; - slock->local.type = lock->type; - if (lock->mtype == MPIDI_REQUEST_LOCK) - MPIDI_WinLockAck_post(context, lock->rank, win); - else if (lock->mtype == MPIDI_REQUEST_LOCKALL) - MPIDI_WinLockAllAck_post(context, lock->rank, win); - else - MPID_assert_always(0); - MPL_free(lock); - MPIDI_WinLockAdvance(context, win); - } -} - - -static pami_result_t -MPIDI_WinLockReq_post(pami_context_t context, - void * _info) -{ - MPIDI_WinLock_info* info = (MPIDI_WinLock_info*)_info; - MPIDI_Win_control_t msg = { - .type = MPIDI_WIN_MSGTYPE_LOCKREQ, - .data = { - .lock = { - .type = info->lock_type, - }, - }, - }; - - MPIDI_WinCtrlSend(context, &msg, info->peer, info->win); - return PAMI_SUCCESS; -} - - -void -MPIDI_WinLockReq_proc(pami_context_t context, - const MPIDI_Win_control_t * info, - unsigned peer) -{ - MPIR_Win * win = info->win; - struct MPIDI_Win_lock* lock = MPL_calloc0(1, struct MPIDI_Win_lock); - if (info->type == MPIDI_WIN_MSGTYPE_LOCKREQ) - lock->mtype = MPIDI_REQUEST_LOCK; - else if (info->type == MPIDI_WIN_MSGTYPE_LOCKALLREQ) { - lock->mtype = MPIDI_REQUEST_LOCKALL; - lock->flagAddr = (void *) info->flagAddr; - } - lock->rank = info->rank; - lock->type = info->data.lock.type; - - struct MPIDI_Win_queue* q = &win->mpid.sync.lock.local.requested; - MPID_assert( (q->head != NULL) ^ (q->tail == NULL) ); - if (q->tail == NULL) - q->head = lock; - else - q->tail->next = lock; - q->tail = lock; - - MPIDI_WinLockAdvance(context, win); -} - - -void -MPIDI_WinLockAck_post(pami_context_t context, - unsigned peer, - MPIR_Win * win) -{ - MPIDI_Win_control_t info = { - .type = MPIDI_WIN_MSGTYPE_LOCKACK, - }; - MPIDI_WinCtrlSend(context, &info, peer, win); -} - - -void -MPIDI_WinLockAck_proc(pami_context_t context, - const MPIDI_Win_control_t * info, - unsigned peer) -{ - if (info->type == MPIDI_WIN_MSGTYPE_LOCKACK) - info->win->mpid.sync.lock.remote.locked = 1; - else if (info->type == MPIDI_WIN_MSGTYPE_LOCKALLACK) - info->win->mpid.sync.lock.remote.allLocked += 1; - -} - - -static pami_result_t -MPIDI_WinUnlock_post(pami_context_t context, - void * _info) -{ - MPIDI_WinLock_info* info = (MPIDI_WinLock_info*)_info; - MPIDI_Win_control_t msg = { - .type = MPIDI_WIN_MSGTYPE_UNLOCK, - }; - MPIDI_WinCtrlSend(context, &msg, info->peer, info->win); - info->done = 1; - return PAMI_SUCCESS; -} - - -void -MPIDI_WinUnlock_proc(pami_context_t context, - const MPIDI_Win_control_t * info, - unsigned peer) -{ - MPIR_Win *win = info->win; - --win->mpid.sync.lock.local.count; - MPID_assert((int)win->mpid.sync.lock.local.count >= 0); - MPIDI_WinLockAdvance(context, win); -} - - -int -MPID_Win_lock(int lock_type, - int rank, - int assert, - MPIR_Win *win) -{ - int mpi_errno = MPI_SUCCESS; - struct MPIDI_Win_sync_lock* slock = &win->mpid.sync.lock; - static char FCNAME[] = "MPID_Win_lock"; - - if(win->mpid.sync.origin_epoch_type != MPID_EPOTYPE_NONE && - win->mpid.sync.origin_epoch_type != MPID_EPOTYPE_REFENCE){ - MPIU_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, - return mpi_errno, "**rmasync"); - } - if (rank == MPI_PROC_NULL) goto fn_exit; - - MPIDI_WinLock_info info = { - .done = 0, - .peer = rank, - .win = win, - .lock_type = lock_type - }; - - MPIDI_Context_post(MPIDI_Context[0], &info.work, MPIDI_WinLockReq_post, &info); - MPID_PROGRESS_WAIT_WHILE(!slock->remote.locked); -fn_exit: - win->mpid.sync.origin_epoch_type = MPID_EPOTYPE_LOCK; - - return mpi_errno; -} - - -int -MPID_Win_unlock(int rank, - MPIR_Win *win) -{ - int mpi_errno = MPI_SUCCESS; - static char FCNAME[] = "MPID_Win_unlock"; - - if(win->mpid.sync.origin_epoch_type != MPID_EPOTYPE_LOCK){ - MPIU_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, - return mpi_errno, "**rmasync"); - } - if (rank == MPI_PROC_NULL) goto fn_exit; - struct MPIDI_Win_sync* sync = &win->mpid.sync; - MPID_PROGRESS_WAIT_DO_WHILE(sync->total != sync->complete); - sync->total = 0; - sync->started = 0; - sync->complete = 0; - - MPIDI_WinLock_info info = { - .done = 0, - .peer = rank, - .win = win, - }; - MPIDI_Context_post(MPIDI_Context[0], &info.work, MPIDI_WinUnlock_post, &info); - MPID_PROGRESS_WAIT_WHILE(sync->lock.remote.locked); -fn_exit: - if(win->mpid.sync.target_epoch_type == MPID_EPOTYPE_REFENCE) - { - win->mpid.sync.origin_epoch_type = MPID_EPOTYPE_REFENCE; - }else{ - win->mpid.sync.origin_epoch_type = MPID_EPOTYPE_NONE; - } - return mpi_errno; -} diff --git a/src/mpid/pamid/src/onesided/mpid_win_lock_all.c b/src/mpid/pamid/src/onesided/mpid_win_lock_all.c deleted file mode 100644 index 5dfcd98..0000000 --- a/src/mpid/pamid/src/onesided/mpid_win_lock_all.c +++ /dev/null @@ -1,206 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/onesided/mpid_win_lock_all.c - * \brief ??? - */ -#include "mpidi_onesided.h" - - -static pami_result_t -MPIDI_WinLockAllReq_post(pami_context_t context, - void * _info) -{ - MPIDI_WinLock_info* info = (MPIDI_WinLock_info*)_info; - MPIDI_Win_control_t msg = { - .type = MPIDI_WIN_MSGTYPE_LOCKALLREQ, - .flagAddr = info, - .data = { - .lock = { - .type = info->lock_type, - }, - }, - }; - - MPIDI_WinCtrlSend(context, &msg, info->peer, info->win); - return PAMI_SUCCESS; -} - - -static pami_result_t -MPIDI_WinUnlockAllReq_post(pami_context_t context, - void * _info) -{ - MPIDI_WinLock_info* info = (MPIDI_WinLock_info*)_info; - MPIDI_Win_control_t msg = { - .type = MPIDI_WIN_MSGTYPE_UNLOCKALL, - .flagAddr = info, - }; - MPIDI_WinCtrlSend(context, &msg, info->peer, info->win); - return PAMI_SUCCESS; -} - -void -MPIDI_WinUnlockAll_proc(pami_context_t context, - const MPIDI_Win_control_t * info, - unsigned peer) -{ - MPIR_Win *win = info->win; - --win->mpid.sync.lock.local.count; - MPID_assert((int)win->mpid.sync.lock.local.count >= 0); - MPIDI_WinLockAdvance(context, win); -} - -void -MPIDI_WinLockAllAck_post(pami_context_t context, - unsigned peer, - MPIR_Win * win) -{ - MPIDI_Win_control_t info = { - .type = MPIDI_WIN_MSGTYPE_LOCKALLACK, - }; - MPIDI_WinCtrlSend(context, &info, peer, win); -} - - -int -MPID_Win_lock_all(int assert, - MPIR_Win *win) -{ - int mpi_errno = MPI_SUCCESS; - int i,size; - MPIDI_WinLock_info *lockQ; - char *cp=NULL; - int nMask,index; - struct MPIDI_Win_sync_lock* slock = &win->mpid.sync.lock; - static char FCNAME[] = "MPID_Win_lock_all"; - - if(win->mpid.sync.origin_epoch_type != MPID_EPOTYPE_NONE && - win->mpid.sync.origin_epoch_type != MPID_EPOTYPE_REFENCE){ - MPIU_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, - return mpi_errno, "**rmasync"); - } - size = (MPIR_Comm_size(win->comm_ptr)); - win->mpid.max_ctrlsends = MAX_NUM_CTRLSEND; - nMask= win->mpid.max_ctrlsends - 1; - if ( (cp=getenv("MP_MAX_NUM_CTRLSEND")) ) { - win->mpid.max_ctrlsends = atoi(cp); - } - nMask=(win->mpid.max_ctrlsends - 1); - if (!win->mpid.work.msgQ) { - if (size < (win->mpid.max_ctrlsends)) { - win->mpid.work.msgQ = (void *) MPL_calloc0(size, MPIDI_WinLock_info); - } else { - win->mpid.work.msgQ = (void *) MPL_calloc0((win->mpid.max_ctrlsends), MPIDI_WinLock_info); - } - MPID_assert(win->mpid.work.msgQ != NULL); - win->mpid.work.count=0; - } - lockQ = (MPIDI_WinLock_info *) win->mpid.work.msgQ; - if (size < win->mpid.max_ctrlsends) { - for (i = 0; i < size; i++) { - lockQ[i].done=0; - lockQ[i].peer=i; - lockQ[i].win=win; - lockQ[i].lock_type=MPI_LOCK_SHARED; - MPIDI_Context_post(MPIDI_Context[0], &lockQ[i].work, MPIDI_WinLockAllReq_post, &lockQ[i]); - } - } else { - for (i = 0; i < size; i++) { - if (i < win->mpid.max_ctrlsends) - index=i; - else { - index = i & nMask; - if (!lockQ[index].done) { - MPID_PROGRESS_WAIT_WHILE(lockQ[index].done == 0); - } - } - lockQ[index].done=0; - lockQ[index].peer=i; - lockQ[index].win=win; - lockQ[index].lock_type=MPI_LOCK_SHARED; - MPIDI_Context_post(MPIDI_Context[0], &lockQ[index].work, MPIDI_WinLockAllReq_post, &lockQ[index]); - } - } - /* wait for the lock is granted for all tasks in the window */ - MPID_PROGRESS_WAIT_WHILE(size != slock->remote.allLocked); - - win->mpid.sync.origin_epoch_type = MPID_EPOTYPE_LOCK_ALL; - - return mpi_errno; -} - - -int -MPID_Win_unlock_all(MPIR_Win *win) -{ - int mpi_errno = MPI_SUCCESS; - int i,size; - MPIDI_WinLock_info *lockQ; - int nMask,index; - struct MPIDI_Win_sync* sync; - static char FCNAME[] = "MPID_Win_unlock_all"; - - if (win->mpid.sync.origin_epoch_type != MPID_EPOTYPE_LOCK_ALL) { - MPIU_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, - return mpi_errno, "**rmasync"); - } - - sync = &win->mpid.sync; - MPID_PROGRESS_WAIT_WHILE(sync->total != sync->complete); - sync->total = 0; - sync->started = 0; - sync->complete = 0; - MPID_assert(win->mpid.work.msgQ != NULL); - lockQ = (MPIDI_WinLock_info *) win->mpid.work.msgQ; - size = MPIR_Comm_size(win->comm_ptr); - nMask = (win->mpid.max_ctrlsends - 1); - if (size < win->mpid.max_ctrlsends) { - for (i = 0; i < size; i++) { - lockQ[i].done=0; - lockQ[i].peer=i; - lockQ[i].win=win; - MPIDI_Context_post(MPIDI_Context[0], &lockQ[i].work, MPIDI_WinUnlockAllReq_post, &lockQ[i]); - } - } else { - for (i = 0; i < size; i++) { - if (i < win->mpid.max_ctrlsends) - index=i; - else { - index = (i & nMask); - if (!lockQ[index].done) { - MPID_PROGRESS_WAIT_WHILE(lockQ[index].done == 0); - } - } - lockQ[index].done=0; - lockQ[index].peer=i; - lockQ[index].win=win; - MPIDI_Context_post(MPIDI_Context[0], &lockQ[index].work, MPIDI_WinUnlockAllReq_post, &lockQ[index]); - } - } - - MPID_PROGRESS_WAIT_WHILE(sync->lock.remote.allLocked); - - if(win->mpid.sync.target_epoch_type == MPID_EPOTYPE_REFENCE) - { - win->mpid.sync.origin_epoch_type = MPID_EPOTYPE_REFENCE; - }else{ - win->mpid.sync.origin_epoch_type = MPID_EPOTYPE_NONE; - } - return mpi_errno; -} diff --git a/src/mpid/pamid/src/onesided/mpid_win_pscw.c b/src/mpid/pamid/src/onesided/mpid_win_pscw.c deleted file mode 100644 index 98d20d5..0000000 --- a/src/mpid/pamid/src/onesided/mpid_win_pscw.c +++ /dev/null @@ -1,270 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/onesided/mpid_win_pscw.c - * \brief ??? - */ -#include "mpidi_onesided.h" - - -typedef struct -{ - MPIR_Win * win; - - volatile unsigned done; - pami_work_t work; -} MPIDI_WinPSCW_info; - - -static pami_result_t -MPIDI_WinPost_post(pami_context_t context, - void * _info) -{ - MPIDI_WinPSCW_info * info = (MPIDI_WinPSCW_info*)_info; - unsigned peer, index, pid,i; - MPIR_Group *group = info->win->mpid.sync.pw.group; - MPID_assert(group != NULL); - MPIDI_Win_control_t msg = { - .type = MPIDI_WIN_MSGTYPE_POST, - }; - - for (index=0; index < group->size; ++index) { - pid = group->lrank_to_lpid[index].lpid; - for (i=0;i < ((info->win)->comm_ptr->local_size); i++) { - if ((info->win)->comm_ptr->local_group->lrank_to_lpid[i].lpid == pid) { - peer = ((info->win)->comm_ptr->local_group->lrank_to_lpid[i].lrank); - break; - } - } - MPIDI_WinCtrlSend(context, &msg, peer, info->win); - } - - info->done = 1; - return PAMI_SUCCESS; -} - - -void -MPIDI_WinPost_proc(pami_context_t context, - const MPIDI_Win_control_t * info, - unsigned peer) -{ - ++info->win->mpid.sync.pw.count; -} - - -static pami_result_t -MPIDI_WinComplete_post(pami_context_t context, - void * _info) -{ - MPIDI_WinPSCW_info * info = (MPIDI_WinPSCW_info*)_info; - unsigned peer, index,pid,i; - MPIR_Group *group = info->win->mpid.sync.sc.group; - MPID_assert(group != NULL); - MPIDI_Win_control_t msg = { - .type = MPIDI_WIN_MSGTYPE_COMPLETE, - }; - - for (index=0; index < group->size; ++index) { - pid = group->lrank_to_lpid[index].lpid; - for (i=0;i < ((info->win)->comm_ptr->local_size); i++) { - if ((info->win)->comm_ptr->local_group->lrank_to_lpid[i].lpid == pid) { - peer = ((info->win)->comm_ptr->local_group->lrank_to_lpid[i].lrank); - break; - } - } - MPIDI_WinCtrlSend(context, &msg, peer, info->win); - } - - info->done = 1; - return PAMI_SUCCESS; -} - - -void -MPIDI_WinComplete_proc(pami_context_t context, - const MPIDI_Win_control_t * info, - unsigned peer) -{ - ++info->win->mpid.sync.sc.count; -} - - -int -MPID_Win_start(MPIR_Group *group, - int assert, - MPIR_Win *win) -{ - int mpi_errno = MPI_SUCCESS; - static char FCNAME[] = "MPID_Win_start"; - if(win->mpid.sync.origin_epoch_type != MPID_EPOTYPE_NONE && - win->mpid.sync.origin_epoch_type != MPID_EPOTYPE_REFENCE) - { - MPIU_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, - return mpi_errno, "**rmasync"); - } - - MPIR_Group_add_ref(group); - - struct MPIDI_Win_sync* sync = &win->mpid.sync; - MPID_PROGRESS_WAIT_WHILE(group->size != sync->pw.count); - sync->pw.count = 0; - - MPIU_ERR_CHKANDJUMP((win->mpid.sync.sc.group != NULL), mpi_errno, MPI_ERR_GROUP, "**group"); - - win->mpid.sync.sc.group = group; - win->mpid.sync.origin_epoch_type = MPID_EPOTYPE_START; - -fn_exit: - return mpi_errno; -fn_fail: - goto fn_exit; -} - - -int -MPID_Win_complete(MPIR_Win *win) -{ - int mpi_errno = MPI_SUCCESS; - static char FCNAME[] = "MPID_Win_complete"; - if(win->mpid.sync.origin_epoch_type != MPID_EPOTYPE_START){ - MPIU_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, - return mpi_errno, "**rmasync"); - } - - struct MPIDI_Win_sync* sync = &win->mpid.sync; - MPID_PROGRESS_WAIT_WHILE(sync->total != sync->complete); - sync->total = 0; - sync->started = 0; - sync->complete = 0; - - MPIDI_WinPSCW_info info = { - .done = 0, - .win = win, - }; - MPIDI_Context_post(MPIDI_Context[0], &info.work, MPIDI_WinComplete_post, &info); - MPID_PROGRESS_WAIT_WHILE(!info.done); - - if(win->mpid.sync.target_epoch_type == MPID_EPOTYPE_REFENCE) - { - win->mpid.sync.origin_epoch_type = MPID_EPOTYPE_REFENCE; - }else{ - win->mpid.sync.origin_epoch_type = MPID_EPOTYPE_NONE; - } - - MPIR_Group_release(sync->sc.group); - sync->sc.group = NULL; - return mpi_errno; -} - - -int -MPID_Win_post(MPIR_Group *group, - int assert, - MPIR_Win *win) -{ - int mpi_errno = MPI_SUCCESS; - static char FCNAME[] = "MPID_Win_post"; - if(win->mpid.sync.target_epoch_type != MPID_EPOTYPE_NONE && - win->mpid.sync.target_epoch_type != MPID_EPOTYPE_REFENCE){ - MPIU_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, - return mpi_errno, "**rmasync"); - } - - MPIR_Group_add_ref(group); - - MPIU_ERR_CHKANDJUMP((win->mpid.sync.pw.group != NULL), mpi_errno, MPI_ERR_GROUP, "**group"); - - win->mpid.sync.pw.group = group; - - MPIDI_WinPSCW_info info = { - .done = 0, - .win = win, - }; - MPIDI_Context_post(MPIDI_Context[0], &info.work, MPIDI_WinPost_post, &info); - MPID_PROGRESS_WAIT_WHILE(!info.done); - - win->mpid.sync.target_epoch_type = MPID_EPOTYPE_POST; - -fn_fail: - return mpi_errno; -} - - -int -MPID_Win_wait(MPIR_Win *win) -{ - int mpi_errno = MPI_SUCCESS; - static char FCNAME[] = "MPID_Win_wait"; - struct MPIDI_Win_sync* sync = &win->mpid.sync; - - if(win->mpid.sync.target_epoch_type != MPID_EPOTYPE_POST){ - MPIU_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, - return mpi_errno, "**rmasync"); - } - - MPIR_Group *group = sync->pw.group; - MPID_PROGRESS_WAIT_WHILE(group->size != sync->sc.count); - sync->sc.count = 0; - sync->pw.group = NULL; - - MPIR_Group_release(group); - - if(win->mpid.sync.origin_epoch_type == MPID_EPOTYPE_REFENCE){ - win->mpid.sync.target_epoch_type = MPID_EPOTYPE_REFENCE; - }else{ - win->mpid.sync.target_epoch_type = MPID_EPOTYPE_NONE; - } - return mpi_errno; -} - - -int -MPID_Win_test(MPIR_Win *win, - int *flag) -{ - int mpi_errno = MPI_SUCCESS; - static char FCNAME[] = "MPID_Win_test"; - struct MPIDI_Win_sync* sync = &win->mpid.sync; - - if(win->mpid.sync.target_epoch_type != MPID_EPOTYPE_POST){ - MPIU_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, - return mpi_errno, "**rmasync"); - } - - MPIR_Group *group = sync->pw.group; - if (group->size == sync->sc.count) - { - sync->sc.count = 0; - sync->pw.group = NULL; - *flag = 1; - MPIR_Group_release(group); - if(win->mpid.sync.origin_epoch_type == MPID_EPOTYPE_REFENCE){ - win->mpid.sync.target_epoch_type = MPID_EPOTYPE_REFENCE; - }else{ - win->mpid.sync.target_epoch_type = MPID_EPOTYPE_NONE; - } - } - else - { - *flag = 0; - MPID_Progress_poke(); - } - - return mpi_errno; -} diff --git a/src/mpid/pamid/src/onesided/mpid_win_put.c b/src/mpid/pamid/src/onesided/mpid_win_put.c deleted file mode 100644 index 9e70f56..0000000 --- a/src/mpid/pamid/src/onesided/mpid_win_put.c +++ /dev/null @@ -1,434 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/onesided/mpid_win_put.c - * \brief ??? - */ -#include "mpidi_onesided.h" -#include "mpidi_util.h" - - -static inline int -MPIDI_Put_use_pami_rput(pami_context_t context, MPIDI_Win_request * req) -__attribute__((__always_inline__)); -#ifdef RDMA_FAILOVER -static inline int -MPIDI_Put_use_pami_put(pami_context_t context, MPIDI_Win_request * req) -__attribute__((__always_inline__)); -#endif - - -static pami_result_t -MPIDI_Put(pami_context_t context, - void * _req) -{ - MPIDI_Win_request *req = (MPIDI_Win_request*)_req; - pami_result_t rc; - -#ifdef USE_PAMI_RDMA - rc = MPIDI_Put_use_pami_rput(context, req); -#else - if( (req->origin.memregion_used) && (req->win->mpid.info[req->target.rank].memregion_used) ) - { - rc = MPIDI_Put_use_pami_rput(context, req); - } else { - rc = MPIDI_Put_use_pami_put(context, req); - } -#endif - if( rc == PAMI_EAGAIN) - return rc; - - - return PAMI_SUCCESS; -} - - -static inline int -MPIDI_Put_use_pami_rput(pami_context_t context, MPIDI_Win_request * req) -{ - int use_typed_rdma = 0; - if (!req->target.dt.contig || !req->origin.dt.contig) { - use_typed_rdma = 0; - if (MPIDI_Process.typed_onesided == 1) - use_typed_rdma = 1; - } - - if (use_typed_rdma) { - pami_result_t rc; - pami_rput_typed_t params; - /* params need to zero out to avoid passing garbage to PAMI */ - params=zero_rput_typed_parms; - - params.rma.dest=req->dest; - params.rma.hints.buffer_registered = PAMI_HINT_ENABLE; - params.rma.hints.use_rdma = PAMI_HINT_ENABLE; - params.rma.bytes = req->target.dt.size; - params.rma.cookie = req; - params.rma.done_fn = NULL; - params.rdma.local.mr=&req->origin.memregion; - params.rdma.remote.mr=&req->win->mpid.info[req->target.rank].memregion; - params.rdma.remote.offset= req->offset; - params.rdma.local.offset = req->state.local_offset; - params.put.rdone_fn= MPIDI_Win_DoneCB; - - params.type.local = *(pami_type_t *)(req->origin.dt.pointer->device_datatype); - params.type.remote = *(pami_type_t *)(req->target.dt.pointer->device_datatype); - - rc = PAMI_Rput_typed(context, &params); - MPID_assert(rc == PAMI_SUCCESS); - - } - else { - pami_result_t rc; - pami_rput_simple_t params; - /* params need to zero out to avoid passing garbage to PAMI */ - params=zero_rput_parms; - - params.rma.dest=req->dest; - params.rma.hints.buffer_registered = PAMI_HINT_ENABLE; - params.rma.hints.use_rdma = PAMI_HINT_ENABLE; - params.rma.bytes = 0; - params.rma.cookie = req; - params.rma.done_fn = NULL; - params.rdma.local.mr=&req->origin.memregion; - params.rdma.remote.mr=&req->win->mpid.info[req->target.rank].memregion; - params.rdma.remote.offset= req->offset; - params.put.rdone_fn= MPIDI_Win_DoneCB; - - struct MPIDI_Win_sync* sync = &req->win->mpid.sync; - TRACE_ERR("Start index=%u/%d l-addr=%p r-base=%p r-offset=%zu (sync->started=%u sync->complete=%u)\n", - req->state.index, req->target.dt.num_contig, req->buffer, req->win->mpid.info[req->target.rank].base_addr, req->offset, sync->started, sync->complete); - while (req->state.index < req->target.dt.num_contig) { - if (sync->started > sync->complete + MPIDI_Process.rma_pending) - { - TRACE_ERR("Bailing out; index=%u/%d sync->started=%u sync->complete=%u\n", - req->state.index, req->target.dt.num_contig, sync->started, sync->complete); - return PAMI_EAGAIN; - } - ++sync->started; - - - params.rma.bytes = req->target.dt.map[req->state.index].DLOOP_VECTOR_LEN; - params.rdma.remote.offset = req->offset + (size_t)req->target.dt.map[req->state.index].DLOOP_VECTOR_BUF; - params.rdma.local.offset = req->state.local_offset; -#ifdef TRACE_ON - unsigned* buf = (unsigned*)(req->buffer + params.rdma.local.offset); -#endif - TRACE_ERR(" Sub index=%u bytes=%zu l-offset=%zu r-offset=%zu buf=%p *(int*)buf=0x%08x\n", - req->state.index, params.rma.bytes, params.rdma.local.offset, params.rdma.remote.offset, buf, *buf); - - /** sync->total will be updated with every RMA and the complete - will not change till that RMA has completed. In the meanwhile - the rest of the RMAs will have memory leaks */ - if (req->target.dt.num_contig - req->state.index == 1) { - rc = PAMI_Rput(context, &params); - MPID_assert(rc == PAMI_SUCCESS); - return PAMI_SUCCESS; - } else { - rc = PAMI_Rput(context, &params); - MPID_assert(rc == PAMI_SUCCESS); - req->state.local_offset += params.rma.bytes; - ++req->state.index; - } - } - } - return PAMI_SUCCESS; -} - -#ifdef RDMA_FAILOVER -static inline int -MPIDI_Put_use_pami_put(pami_context_t context, MPIDI_Win_request * req) -{ - pami_result_t rc; - pami_put_simple_t params; - - params = zero_put_parms; - - params.rma.dest=req->dest; - params.rma.hints.use_rdma = PAMI_HINT_DEFAULT; -#ifndef OUT_OF_ORDER_HANDLING - params.rma.hints.no_long_header= 1, -#endif - params.rma.bytes = 0; - params.rma.cookie = req; - params.rma.done_fn = NULL; - params.addr.local=req->buffer; - params.addr.remote=req->win->mpid.info[req->target.rank].base_addr; - params.put.rdone_fn= MPIDI_Win_DoneCB; - - struct MPIDI_Win_sync* sync = &req->win->mpid.sync; - TRACE_ERR("Start index=%u/%d l-addr=%p r-base=%p r-offset=%zu (sync->started=%u sync->complete=%u)\n", - req->state.index, req->target.dt.num_contig, req->buffer, req->win->mpid.info[req->target.rank].base_addr, req->offset, sync->started, sync->complete); - while (req->state.index < req->target.dt.num_contig) { - if (sync->started > sync->complete + MPIDI_Process.rma_pending) - { - TRACE_ERR("Bailing out; index=%u/%d sync->started=%u sync->complete=%u\n", - req->state.index, req->target.dt.num_contig, sync->started, sync->complete); - return PAMI_EAGAIN; - } - ++sync->started; - - - params.rma.bytes = req->target.dt.map[req->state.index].DLOOP_VECTOR_LEN; - params.addr.local = req->buffer+req->state.local_offset; - params.addr.remote = req->win->mpid.info[req->target.rank].base_addr+ req->offset + (size_t)req->target.dt.map[req->state.index].DLOOP_VECTOR_BUF; -#ifdef TRACE_ON - unsigned* buf = (unsigned*)(req->buffer + req->state.local_offset); -#endif - TRACE_ERR(" Sub index=%u bytes=%zu l-offset=%zu r-offset=%zu buf=%p *(int*)buf=0x%08x\n", - req->state.index, params.rma.bytes, params.addr.local, params.addr.remote, buf, *buf); - - /** sync->total will be updated with every RMA and the complete - will not change till that RMA has completed. In the meanwhile - the rest of the RMAs will have memory leaks */ - if (req->target.dt.num_contig - req->state.index == 1) { - rc = PAMI_Put(context, &params); - MPID_assert(rc == PAMI_SUCCESS); - return PAMI_SUCCESS; - } else { - rc = PAMI_Put(context, &params); - MPID_assert(rc == PAMI_SUCCESS); - req->state.local_offset += params.rma.bytes; - ++req->state.index; - } - } - return PAMI_SUCCESS; -} -#endif - - -/** - * \brief MPI-PAMI glue for MPI_PUT function - * - * \param[in] origin_addr Source buffer - * \param[in] origin_count Number of datatype elements - * \param[in] origin_datatype Source datatype - * \param[in] target_rank Destination rank (target) - * \param[in] target_disp Displacement factor in target buffer - * \param[in] target_count Number of target datatype elements - * \param[in] target_datatype Destination datatype - * \param[in] win Window - * \return MPI_SUCCESS - */ -#undef FUNCNAME -#define FUNCNAME MPID_Put -#undef FCNAME -#define FCNAME MPIU_QUOTE(FUNCNAME) -int -MPID_Put(const void *origin_addr, - int origin_count, - MPI_Datatype origin_datatype, - int target_rank, - MPI_Aint target_disp, - int target_count, - MPI_Datatype target_datatype, - MPIR_Win *win) -{ - int mpi_errno = MPI_SUCCESS; - int shm_locked=0; - void * target_addr; - MPIDI_Win_request *req = MPL_calloc0(1, MPIDI_Win_request); - req->win = win; - if(win->mpid.request_based != 1) - req->type = MPIDI_WIN_REQUEST_PUT; - else { - req->req_handle = win->mpid.rreq; - req->type = MPIDI_WIN_REQUEST_RPUT; - req->req_handle->mpid.win_req = req; - } - - if(win->mpid.sync.origin_epoch_type == win->mpid.sync.target_epoch_type && - win->mpid.sync.origin_epoch_type == MPID_EPOTYPE_REFENCE){ - win->mpid.sync.origin_epoch_type = MPID_EPOTYPE_FENCE; - win->mpid.sync.target_epoch_type = MPID_EPOTYPE_FENCE; - } - - if(win->mpid.sync.origin_epoch_type == MPID_EPOTYPE_NONE || - win->mpid.sync.origin_epoch_type == MPID_EPOTYPE_POST){ - MPIU_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, - return mpi_errno, "**rmasync"); - } - - req->offset = target_disp * win->mpid.info[target_rank].disp_unit; -#ifdef __BGQ__ - /* PAMI limitation as it doesnt permit VA of 0 to be passed into - * memregion create, so we must pass base_va of heap computed from - * an SPI call instead. So the target offset must be adjusted */ - if (req->win->create_flavor == MPI_WIN_FLAVOR_DYNAMIC) - req->offset -= (size_t)req->win->mpid.info[target_rank].base_addr; -#endif - - MPIDI_Win_datatype_basic(origin_count, - origin_datatype, - &req->origin.dt); - MPIDI_Win_datatype_basic(target_count, - target_datatype, - &req->target.dt); - #ifndef MPIDI_NO_ASSERT - MPID_assert(req->origin.dt.size == req->target.dt.size); - #else - MPIU_ERR_CHKANDJUMP((req->origin.dt.size != req->target.dt.size), mpi_errno, MPI_ERR_SIZE, "**rmasize"); - #endif - - - - if ( (req->origin.dt.size == 0) || - (target_rank == MPI_PROC_NULL)) - { - if(req->req_handle) - MPIR_cc_set(req->req_handle->cc_ptr, 0); - else - MPL_free(req); - return MPI_SUCCESS; - } - - - if ((target_rank == win->comm_ptr->rank) || (win->create_flavor == MPI_WIN_FLAVOR_SHARED)) - { - size_t offset = req->offset; - if (target_rank == win->comm_ptr->rank) - target_addr = win->base + req->offset; - else - target_addr = win->mpid.info[target_rank].base_addr + req->offset; - - mpi_errno = MPIR_Localcopy(origin_addr, - origin_count, - origin_datatype, - target_addr, - target_count, - target_datatype); - - /* The instant this completion counter is set to zero another thread - * may notice the change and begin freeing request resources. The - * thread executing the code in this function must not touch any - * portion of the request structure after decrementing the completion - * counter. - * - * See MPID_Request_free_inline() - */ - if(req->req_handle) - MPIR_cc_set(req->req_handle->cc_ptr, 0); - else - MPL_free(req); - return mpi_errno; - } - req->target.rank = target_rank; - - - /* Only pack the origin data if the origin is non-contiguous and we are using the simple PAMI_Rput. - * If we are using the typed PAMI_Rput_typed use the origin address as-is, if we are using the simple - * PAMI_Rput with contiguous data use the origin address with the lower-bound adjustment. - */ - if (req->origin.dt.contig || (!req->origin.dt.contig && (MPIDI_Process.typed_onesided == 1))) - { - req->buffer_free = 0; - if ((req->origin.dt.contig && req->target.dt.contig && (MPIDI_Process.typed_onesided == 1)) || (!(MPIDI_Process.typed_onesided == 1))) - req->buffer = (void *) ((uintptr_t) origin_addr + req->origin.dt.true_lb); - else - req->buffer = (void *) ((uintptr_t) origin_addr); - } - else - { - req->buffer_free = 1; - req->buffer = MPL_malloc(req->origin.dt.size); - MPID_assert(req->buffer != NULL); - - int mpi_errno = 0; - mpi_errno = MPIR_Localcopy(origin_addr, - origin_count, - origin_datatype, - req->buffer, - req->origin.dt.size, - MPI_CHAR); - MPID_assert(mpi_errno == MPI_SUCCESS); - } - - - pami_result_t rc; - pami_task_t task = MPID_VCR_GET_LPID(win->comm_ptr->vcr, target_rank); - if (win->mpid.sync.origin_epoch_type == MPID_EPOTYPE_START && - !MPIDI_valid_group_rank(task, win->mpid.sync.sc.group)) - { - MPIU_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, - return mpi_errno, "**rmasync"); - } - - rc = PAMI_Endpoint_create(MPIDI_Client, task, 0, &req->dest); - MPID_assert(rc == PAMI_SUCCESS); - -#ifdef USE_PAMI_RDMA - size_t length_out; - rc = PAMI_Memregion_create(MPIDI_Context[0], - req->buffer, - req->origin.dt.size, - &length_out, - &req->origin.memregion); - MPID_assert(rc == PAMI_SUCCESS); - MPID_assert(req->origin.dt.size == length_out); -#else - if(!MPIDI_Process.mp_s_use_pami_get) - { - size_t length_out; - rc = PAMI_Memregion_create(MPIDI_Context[0], - req->buffer, - req->origin.dt.size, - &length_out, - &req->origin.memregion); - if(rc == PAMI_SUCCESS) { - MPID_assert(req->origin.dt.size == length_out); - req->origin.memregion_used = 1; - } - } -#endif - - - if ((!req->target.dt.contig || !req->origin.dt.contig) && (MPIDI_Process.typed_onesided == 1)) - /* If the datatype is non-contiguous and the PAMID typed_onesided optimization - * is enabled then we will be using the typed interface and will only make 1 call. - */ - win->mpid.sync.total = 1; - else { - MPIDI_Win_datatype_map(&req->target.dt); - win->mpid.sync.total += req->target.dt.num_contig; - } - if ((MPIDI_Process.typed_onesided == 1) && (!req->target.dt.contig || !req->origin.dt.contig)) { - /* We will use the PAMI_Rput_typed call so we need to make sure any MPI_Type_free before the context - * executes the put does not free the MPIDU_Datatype, which would also free the associated PAMI datatype which - * is still needed for communication -- decrement the ref in the callback to allow the MPIR_Datatype - * to be freed once the PAMI communication has completed. - */ - MPIDU_Datatype_add_ref(req->origin.dt.pointer); - MPIDU_Datatype_add_ref(req->target.dt.pointer); - } - /* The pamid one-sided design requires context post in order to handle the - * case where the number of pending rma operation exceeds the - * 'PAMID_RMA_PENDING' threshold. When there are too many pending requests the - * work function remains on the context post queue (by returning PAMI_EAGAIN) - * so that the next time the context is advanced the work function will be - * invoked again. - * - * TODO - When context post is not required it would be better to attempt a - * direct context operation and then fail over to using context post if - * the rma pending threshold has been reached. This would result in - * better latency for one-sided operations. - */ - PAMI_Context_post(MPIDI_Context[0], &req->post_request, MPIDI_Put, req); - -fn_fail: - return mpi_errno; -} diff --git a/src/mpid/pamid/src/onesided/mpid_win_reqops.c b/src/mpid/pamid/src/onesided/mpid_win_reqops.c deleted file mode 100644 index 36518d1..0000000 --- a/src/mpid/pamid/src/onesided/mpid_win_reqops.c +++ /dev/null @@ -1,253 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/onesided/mpid_win_reqops.c - * \brief ??? - */ -#include "mpidi_onesided.h" - - -/** - * \brief MPI-PAMI glue for MPI_PUT function - * - * \param[in] origin_addr Source buffer - * \param[in] origin_count Number of datatype elements - * \param[in] origin_datatype Source datatype - * \param[in] target_rank Destination rank (target) - * \param[in] target_disp Displacement factor in target buffer - * \param[in] target_count Number of target datatype elements - * \param[in] target_datatype Destination datatype - * \param[in] win Window - * \return MPI_SUCCESS - */ -#undef FUNCNAME -#define FUNCNAME MPID_Rput -#undef FCNAME -#define FCNAME MPIU_QUOTE(FUNCNAME) -int -MPID_Rput(const void *origin_addr, - int origin_count, - MPI_Datatype origin_datatype, - int target_rank, - MPI_Aint target_disp, - int target_count, - MPI_Datatype target_datatype, - MPIR_Win *win, - MPIR_Request **request) -{ - int mpi_errno = MPI_SUCCESS; - - if(win->mpid.sync.origin_epoch_type != MPID_EPOTYPE_LOCK && - win->mpid.sync.origin_epoch_type != MPID_EPOTYPE_LOCK_ALL){ - MPIU_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, - return mpi_errno, "**rmasync"); - } - - MPIR_Request *rreq = MPIDI_Request_create1(); - *request = rreq; - rreq->kind = MPIR_REQUEST_KIND__RMA; - win->mpid.rreq = rreq; - win->mpid.request_based = 1; - - /* Enqueue or perform the RMA operation */ - mpi_errno = MPID_Put(origin_addr, origin_count, - origin_datatype, target_rank, - target_disp, target_count, - target_datatype, win); - MPID_assert(mpi_errno == MPI_SUCCESS); - return (mpi_errno); -} - - -/** - * \brief MPI-PAMI glue for MPI_GET function - * - * \param[in] origin_addr Source buffer - * \param[in] origin_count Number of datatype elements - * \param[in] origin_datatype Source datatype - * \param[in] target_rank Destination rank (target) - * \param[in] target_disp Displacement factor in target buffer - * \param[in] target_count Number of target datatype elements - * \param[in] target_datatype Destination datatype - * \param[in] win Window - * \return MPI_SUCCESS - */ -#undef FUNCNAME -#define FUNCNAME MPID_Rget -#undef FCNAME -#define FCNAME MPIU_QUOTE(FUNCNAME) -int -MPID_Rget(void *origin_addr, - int origin_count, - MPI_Datatype origin_datatype, - int target_rank, - MPI_Aint target_disp, - int target_count, - MPI_Datatype target_datatype, - MPIR_Win *win, - MPIR_Request **request) -{ - int mpi_errno = MPI_SUCCESS; - - if(win->mpid.sync.origin_epoch_type != MPID_EPOTYPE_LOCK && - win->mpid.sync.origin_epoch_type != MPID_EPOTYPE_LOCK_ALL){ - MPIU_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, - return mpi_errno, "**rmasync"); - } - - MPIR_Request *rreq = MPIDI_Request_create1(); - rreq->kind = MPIR_REQUEST_KIND__RMA; - *request = rreq; - win->mpid.rreq = rreq; - win->mpid.request_based = 1; - - mpi_errno = MPID_Get(origin_addr, origin_count, - origin_datatype, target_rank, - target_disp, target_count, - target_datatype, win); - MPID_assert(mpi_errno == MPI_SUCCESS); - return(mpi_errno); -} - -/** - * \brief MPI-PAMI glue for MPI_ACCUMULATE function - * - * According to the MPI Specification: - * - * Each datatype argument must be a predefined datatype or - * a derived datatype, where all basic components are of the - * same predefined datatype. Both datatype arguments must be - * constructed from the same predefined datatype. - * - * \param[in] origin_addr Source buffer - * \param[in] origin_count Number of datatype elements - * \param[in] origin_datatype Source datatype - * \param[in] target_rank Destination rank (target) - * \param[in] target_disp Displacement factor in target buffer - * \param[in] target_count Number of target datatype elements - * \param[in] target_datatype Destination datatype - * \param[in] op Operand to perform - * \param[in] win Window - * \return MPI_SUCCESS - */ -#undef FUNCNAME -#define FUNCNAME MPID_Raccumulate -#undef FCNAME -#define FCNAME MPIU_QUOTE(FUNCNAME) -int -MPID_Raccumulate(const void *origin_addr, - int origin_count, - MPI_Datatype origin_datatype, - int target_rank, - MPI_Aint target_disp, - int target_count, - MPI_Datatype target_datatype, - MPI_Op op, - MPIR_Win *win, - MPIR_Request **request) -{ - int mpi_errno = MPI_SUCCESS; - - if(win->mpid.sync.origin_epoch_type != MPID_EPOTYPE_LOCK && - win->mpid.sync.origin_epoch_type != MPID_EPOTYPE_LOCK_ALL){ - MPIU_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, - return mpi_errno, "**rmasync"); - } - - MPIR_Request *rreq = MPIDI_Request_create1(); - rreq->kind = MPIR_REQUEST_KIND__RMA; - *request = rreq; - win->mpid.rreq = rreq; - win->mpid.request_based = 1; - - /* Enqueue or perform the RMA operation */ - mpi_errno = MPID_Accumulate(origin_addr, origin_count, - origin_datatype, target_rank, - target_disp, target_count, - target_datatype, op, win); - MPID_assert(mpi_errno == MPI_SUCCESS); - return(mpi_errno); -} - - -/** - * \brief MPI-PAMI glue for MPI_ACCUMULATE function - * - * According to the MPI Specification: - * - * Each datatype argument must be a predefined datatype or - * a derived datatype, where all basic components are of the - * same predefined datatype. Both datatype arguments must be - * constructed from the same predefined datatype. - * - * \param[in] origin_addr Source buffer - * \param[in] origin_count Number of datatype elements - * \param[in] origin_datatype Source datatype - * \param[in] target_rank Destination rank (target) - * \param[in] target_disp Displacement factor in target buffer - * \param[in] target_count Number of target datatype elements - * \param[in] target_datatype Destination datatype - * \param[in] op Operand to perform - * \param[in] win Window - * \return MPI_SUCCESS - */ -#undef FUNCNAME -#define FUNCNAME MPID_Rget_accumulate -#undef FCNAME -#define FCNAME MPIU_QUOTE(FUNCNAME) -int -MPID_Rget_accumulate(const void *origin_addr, - int origin_count, - MPI_Datatype origin_datatype, - void *result_addr, - int result_count, - MPI_Datatype result_datatype, - int target_rank, - MPI_Aint target_disp, - int target_count, - MPI_Datatype target_datatype, - MPI_Op op, - MPIR_Win *win, - MPIR_Request **request) -{ - - int mpi_errno = MPI_SUCCESS; - - if(win->mpid.sync.origin_epoch_type != MPID_EPOTYPE_LOCK && - win->mpid.sync.origin_epoch_type != MPID_EPOTYPE_LOCK_ALL){ - MPIU_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, - return mpi_errno, "**rmasync"); - } - - MPIR_Request *rreq = MPIDI_Request_create1(); - rreq->kind = MPIR_REQUEST_KIND__RMA; - *request = rreq; - win->mpid.rreq = rreq; - win->mpid.request_based = 1; - - /* Enqueue or perform the RMA operation */ - mpi_errno = MPID_Get_accumulate(origin_addr, origin_count, - origin_datatype, result_addr, - result_count, result_datatype, - target_rank, target_disp, - target_count, target_datatype, - op, win); - MPID_assert(mpi_errno == MPI_SUCCESS); - return (mpi_errno); -} - diff --git a/src/mpid/pamid/src/onesided/mpid_win_set_info.c b/src/mpid/pamid/src/onesided/mpid_win_set_info.c deleted file mode 100644 index 78ce569..0000000 --- a/src/mpid/pamid/src/onesided/mpid_win_set_info.c +++ /dev/null @@ -1,102 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/onesided/mpid_win_set_info.c - * \brief sets new values for the hints of the window of the window - * associated with win. - */ -#include "mpidi_onesided.h" - -/** - * \brief MPI-PAMI glue for MPI_WIN_SET_INFO function - * - * \param[in] win Window - * \param[in] info Info hint - * \return MPI_SUCCESS - */ - -int MPIDI_Win_set_info(MPIR_Win *win, MPIR_Info *info) -{ - - int mpi_errno = MPI_SUCCESS; - MPIR_Info *curr_ptr; - char *value, *token; - char *savePtr; - curr_ptr = info->next; - uint save_ordering; - - while (curr_ptr) { - if (!strcmp(curr_ptr->key,"no_locks")) { - if (!strcmp(curr_ptr->value,"true")) { - win->mpid.info_args.no_locks=1; - } else - win->mpid.info_args.no_locks=0; - } else if (!strcmp(curr_ptr->key,"accumulate_ordering")) - { - save_ordering=(uint) win->mpid.info_args.accumulate_ordering; - win->mpid.info_args.accumulate_ordering=0; - value = curr_ptr->value; - token = (char *) strtok_r(value,"," , &savePtr); - while (token) { - if (!memcmp(token,"rar",3)) - { - win->mpid.info_args.accumulate_ordering = - (win->mpid.info_args.accumulate_ordering | MPIDI_ACCU_ORDER_RAR); - } else if (!memcmp(token,"raw",3)) - { - win->mpid.info_args.accumulate_ordering = - (win->mpid.info_args.accumulate_ordering | MPIDI_ACCU_ORDER_RAW); - } else if (!memcmp(token,"war",3)) - { - win->mpid.info_args.accumulate_ordering = - (win->mpid.info_args.accumulate_ordering | MPIDI_ACCU_ORDER_WAR); - } else if (!memcmp(token,"waw",3)) - { - win->mpid.info_args.accumulate_ordering = - (win->mpid.info_args.accumulate_ordering | MPIDI_ACCU_ORDER_WAW); - } else - MPID_assert_always(0); - token = (char *) strtok_r(NULL,"," , &savePtr); - } - if (win->mpid.info_args.accumulate_ordering == 0) { - win->mpid.info_args.accumulate_ordering= - (MPIDI_Win_info_accumulate_ordering) save_ordering; - } - } else if (!strcmp(curr_ptr->key,"accumulate_ops")) - { - /* the default setting is MPIDI_ACCU_SAME_OP_NO_OP */ - if (!strcmp(curr_ptr->value,"same_op")) - win->mpid.info_args.accumulate_ops = MPIDI_ACCU_SAME_OP; - } - curr_ptr = curr_ptr->next; - } - - return mpi_errno; -} - -int -MPID_Win_set_info(MPIR_Win *win, MPIR_Info *info) -{ - int mpi_errno = MPI_SUCCESS; - MPIR_Errflag_t errflag = MPIR_ERR_NONE; - - mpi_errno = MPIDI_Win_set_info(win, info); - MPID_assert(mpi_errno == MPI_SUCCESS); - mpi_errno = MPIR_Barrier_impl(win->comm_ptr, &errflag); - return mpi_errno; -} diff --git a/src/mpid/pamid/src/onesided/mpid_win_shared_query.c b/src/mpid/pamid/src/onesided/mpid_win_shared_query.c deleted file mode 100644 index 30c5cd9..0000000 --- a/src/mpid/pamid/src/onesided/mpid_win_shared_query.c +++ /dev/null @@ -1,65 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/onesided/mpid_win_shared_query.c - * \brief queries the process-local address for remote memory segments - * created with MPI_Win_allocate_shared. - */ -#include "mpidi_onesided.h" - -/** - * \brief MPI-PAMI glue for MPI_Win_shared_query function - * - * Query the size and base pointer for a patch of a shared memory window - * - * \param[in] win shared memory window object - * \param[in] rank rank in the group of window win or MPI_PROC_NULL - * \param[out] size size of the window segment (non-negative integer) - * \param[out] disp_unit local unit size for displacements, in bytes - * \param[out] base_ptr address for load/store access to window segment - * \return MPI_SUCCESS, MPI_ERR_OTHER, or error returned from - */ - -int -MPID_Win_shared_query(MPIR_Win *win, int rank, MPI_Aint *size, - int *disp_unit, void *base_ptr) -{ - int i, mpi_errno = MPI_SUCCESS; - static char FCNAME[] = "MPID_Win_shared_query"; - MPIU_ERR_CHKANDSTMT((win->create_flavor != MPI_WIN_FLAVOR_SHARED), mpi_errno, - MPI_ERR_RMA_FLAVOR, return mpi_errno, "**rmaflavor"); - - if (rank == MPI_PROC_NULL) { - for (i=0; i<win->comm_ptr->local_size; ++i) { - if (win->mpid.info[i].base_size != 0) { - *size = win->mpid.info[i].base_size; - *disp_unit = win->mpid.info[i].disp_unit; - break; - } - } - *((void **) base_ptr) = win->base; /* return the begin. address of shared segment */ - } else { - *((void **) base_ptr) = win->mpid.info[rank].base_addr; /* return the begin. address of the rank */ - *size = win->mpid.info[rank].base_size; - *disp_unit = win->mpid.info[rank].disp_unit; - } - - - return mpi_errno; -} - diff --git a/src/mpid/pamid/src/onesided/mpid_win_sync.c b/src/mpid/pamid/src/onesided/mpid_win_sync.c deleted file mode 100644 index d014602..0000000 --- a/src/mpid/pamid/src/onesided/mpid_win_sync.c +++ /dev/null @@ -1,54 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/onesided/mpid_win_sync.c - * \brief returns a new info object containing the hints of the window - * associated with win. - */ -#include "mpidi_onesided.h" - -/** - * \brief MPI-PAMI glue for MPI_Win_sync function - * - * The funcion synchronizes the private and public window copies of win. - * For the purpose of synchronizing the private and public window, - * MPI_Win_sync has the effect of ending and reopening an access and - * exposure epoch on the window. (note that it does not actually end an - * epoch or complete any pending MPI RMA operations). - * - * \param[in] win window object - * \return MPI_SUCCESS, MPI_ERR_RMA_SYNC - */ - - -int -MPID_Win_sync(MPIR_Win *win) -{ - int mpi_errno = MPI_SUCCESS; - static char FCNAME[] = "MPID_Win_flush"; - - if((win->mpid.sync.origin_epoch_type != MPID_EPOTYPE_LOCK) && - (win->mpid.sync.origin_epoch_type != MPID_EPOTYPE_LOCK_ALL)) - { - MPIU_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, - return mpi_errno, "**rmasync"); - } - OPA_read_write_barrier(); - return mpi_errno; -} - diff --git a/src/mpid/pamid/src/onesided/mpidi_onesided.h b/src/mpid/pamid/src/onesided/mpidi_onesided.h deleted file mode 100644 index 369866f..0000000 --- a/src/mpid/pamid/src/onesided/mpidi_onesided.h +++ /dev/null @@ -1,389 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/onesided/mpidi_onesided.h - * \brief ??? - */ -#include <mpidimpl.h> -#include "opa_primitives.h" - -#ifndef __src_onesided_mpidi_onesided_h__ -#define __src_onesided_mpidi_onesided_h__ - -pami_rget_simple_t zero_rget_parms; -pami_get_simple_t zero_get_parms; -pami_rput_simple_t zero_rput_parms; -pami_put_simple_t zero_put_parms; -pami_rput_typed_t zero_rput_typed_parms; -pami_rget_typed_t zero_rget_typed_parms; -pami_send_t zero_send_parms; -pami_send_immediate_t zero_send_immediate_parms; -pami_recv_t zero_recv_parms; -pami_rmw_t zero_rmw_parms; - -#define MPIDI_QUICKSLEEP usleep(1); -#define MAX_NUM_CTRLSEND 1024 /* no more than 1024 outstanding control sends */ - -#define MPIDI_SHM_MUTEX_LOCK(win) { \ - int mpi_errno=0; \ - do { \ - pthread_mutex_t *shm_mutex = (pthread_mutex_t *) &win->mpid.shm->ctrl->mutex_lock; \ - int rval = pthread_mutex_lock(shm_mutex); \ - MPIU_ERR_CHKANDJUMP1(rval, mpi_errno, MPI_ERR_OTHER, "**pthread_lock", \ - "**pthread_lock %s", strerror(rval)); \ - } while (0); } - - -#define MPIDI_SHM_MUTEX_UNLOCK(win) { \ - int mpi_errno=0; \ - do { \ - pthread_mutex_t *shm_mutex = (pthread_mutex_t *) &win->mpid.shm->ctrl->mutex_lock; \ - int rval = pthread_mutex_unlock(shm_mutex); \ - MPIU_ERR_CHKANDJUMP1(rval, mpi_errno, MPI_ERR_OTHER, "**pthread_unlock", \ - "**pthread_unlock %s", strerror(rval)); \ - } while (0); } - - -#define MPIDI_SHM_MUTEX_INIT(win) \ - do { \ - int rval=0; \ - int mpi_errno=0; \ - pthread_mutexattr_t attr; \ - pthread_mutex_t *shm_mutex = (pthread_mutex_t *) &win->mpid.shm->ctrl->mutex_lock; \ - \ - rval = pthread_mutexattr_init(&attr); \ - MPIU_ERR_CHKANDJUMP1(rval, mpi_errno, MPI_ERR_OTHER, "**pthread_mutex", \ - "**pthread_mutex %s", strerror(rval)); \ - rval = pthread_mutexattr_setpshared(&attr, PTHREAD_PROCESS_SHARED); \ - MPIU_ERR_CHKANDJUMP1(rval, mpi_errno, MPI_ERR_OTHER, "**pthread_mutex", \ - "**pthread_mutex %s", strerror(rval)); \ - rval = pthread_mutex_init(shm_mutex, &attr); \ - MPIU_ERR_CHKANDJUMP1(rval, mpi_errno, MPI_ERR_OTHER, "**pthread_mutex", \ - "**pthread_mutex %s", strerror(rval)); \ - rval = pthread_mutexattr_destroy(&attr); \ - MPIU_ERR_CHKANDJUMP1(rval, mpi_errno, MPI_ERR_OTHER, "**pthread_mutex", \ - "**pthread_mutex %s", strerror(rval)); \ - } while (0); - -#define MPIDI_SHM_MUTEX_DESTROY(win) { \ - int mpi_errno=0; \ - do { \ - pthread_mutex_t *shm_mutex = (pthread_mutex_t *) &(win)->mpid.shm->ctrl->mutex_lock;\ - int rval = pthread_mutex_destroy(shm_mutex); \ - MPIU_ERR_CHKANDJUMP1(rval, mpi_errno, MPI_ERR_OTHER, "**pthread_mutex", \ - "**pthread_mutex %s", strerror(rval)); \ - } while (0); } - -#define MPIDI_ACQUIRE_SHARED_LOCK(win) { \ - MPIDI_SHM_MUTEX_LOCK(win); \ - OPA_fetch_and_add_int((OPA_int_t *)&win->mpid.shm->ctrl->active,(int) 1); \ -} - -#define MPIDI_RELEASE_SHARED_LOCK(win) { \ - MPIDI_SHM_MUTEX_UNLOCK(win); \ - OPA_fetch_and_add_int((OPA_int_t *)&(win->mpid.shm->ctrl->active),(int) -1); \ -} - - - - -/** - * \brief One-sided Message Types - */ -typedef enum - { - MPIDI_WIN_MSGTYPE_LOCKACK, /**< Lock acknowledge */ - MPIDI_WIN_MSGTYPE_LOCKALLACK, /**< Lock all acknowledge */ - MPIDI_WIN_MSGTYPE_LOCKREQ, /**< Lock window */ - MPIDI_WIN_MSGTYPE_LOCKALLREQ, /**< Lock all window */ - MPIDI_WIN_MSGTYPE_UNLOCK, /**< Unlock window */ - MPIDI_WIN_MSGTYPE_UNLOCKALL, /**< Unlock window */ - - MPIDI_WIN_MSGTYPE_COMPLETE, /**< End a START epoch */ - MPIDI_WIN_MSGTYPE_POST, /**< Begin POST epoch */ - } MPIDI_Win_msgtype_t; - -typedef enum - { - MPIDI_WIN_REQUEST_ACCUMULATE, - MPIDI_WIN_REQUEST_GET, - MPIDI_WIN_REQUEST_PUT, - MPIDI_WIN_REQUEST_GET_ACCUMULATE, - MPIDI_WIN_REQUEST_RACCUMULATE, - MPIDI_WIN_REQUEST_RGET, - MPIDI_WIN_REQUEST_RPUT, - MPIDI_WIN_REQUEST_RGET_ACCUMULATE, - MPIDI_WIN_REQUEST_COMPARE_AND_SWAP, - MPIDI_WIN_REQUEST_FETCH_AND_OP, - } MPIDI_Win_requesttype_t; - -typedef enum - { - MPIDI_ACCU_OPS_SAME_OP, - MPIDI_ACCU_OPS_SAME_OP_NO_OP - } MPIDI_Win_info_arg_vals_accumulate_ops; - - -typedef struct -{ - MPIDI_Win_msgtype_t type; - MPIR_Win *win; - int rank; /* MPI rank */ - void *flagAddr; - union - { - struct - { - int type; - } lock; - } data; -} MPIDI_Win_control_t; - - -#define MAX_ATOMIC_TYPE_SIZE 32 -typedef struct -{ - char buf[MAX_ATOMIC_TYPE_SIZE]; //Origin value or ack result value - char test[MAX_ATOMIC_TYPE_SIZE]; //Test element for CAS - void * result_addr; //Address on source to store output - void * remote_addr; //Address of target on destination - void * request_addr; //Address of the request object - MPI_Datatype datatype; - MPI_Op op; - int atomic_type; -} MPIDI_AtomicHeader_t; - - -typedef struct MPIDI_WinLock_info -{ - unsigned peer; - int lock_type; - struct MPIR_Win * win; - void *flagAddr; - volatile unsigned done; - pami_work_t work; -} MPIDI_WinLock_info; - - - -typedef struct -{ - MPIDU_Datatype* pointer; - MPI_Datatype type; - int count; - int contig; - MPI_Aint true_lb; - intptr_t size; - - int num_contig; - DLOOP_VECTOR * map; - DLOOP_VECTOR __map; -} MPIDI_Datatype; - - -typedef struct -{ - void * addr; - void * result_addr; /* anchor the result buffer address*/ - void * req; - MPIR_Win * win; - MPI_Datatype type; - MPI_Op op; - pami_endpoint_t origin_endpoint; /* needed for returning the result */ - /* for MPI_Get_accumulate */ - size_t len; /* length of the send data */ -} MPIDI_Win_MsgInfo; - -typedef struct -{ - void * addr; - void * req; - MPIR_Win * win; - MPI_Datatype type; - MPI_Op op; - int count; - int counter; - int num_contig; - int size; - void * request; - void * tptr; - pami_endpoint_t src_endpoint; -} MPIDI_Win_GetAccMsgInfo; - - -/** \todo make sure that the extra fields are removed */ -typedef struct MPIDI_Win_request -{ - MPIR_Win *win; - MPIDI_Win_requesttype_t type; - pami_endpoint_t dest; - size_t offset; - pami_work_t post_request; - - struct - { - unsigned index; - size_t local_offset; - } state; - - void * accum_headers; - - struct - { - pami_memregion_t memregion; -#ifdef RDMA_FAILOVER - uint32_t memregion_used; /**< memory region registered or not */ -#endif - void *addr; - void *result_addr; /** result buffer that holds target buffer data */ - int count; - MPI_Datatype datatype; - MPIDI_Datatype dt; - int completed; - } origin; - - struct - { - pami_task_t rank; /**< Comm-local rank */ - MPIDI_Datatype dt; - } target; - - struct - { - void *addr; - int count; - MPI_Datatype datatype; - MPIDI_Datatype dt; - } result; - - void *user_buffer; - void *compare_buffer; /* anchor of compare buffer for compare and swap */ - uint32_t buffer_free; - void *buffer; - struct MPIDI_Win_request *next; - MPI_Op op; - int result_num_contig; - - - /* for RMA atomic functions */ - - pami_atomic_t pami_op; - pami_type_t pami_datatype; - - int request_based; /* flag for request based rma */ - MPIR_Request *req_handle; /* anchor of MPIR_Request struc for request based rma*/ -} MPIDI_Win_request; - -MPIDI_Win_request zero_req; /* used for init. request structure to 0 */ - -void -MPIDI_Win_datatype_basic(int count, - MPI_Datatype datatype, - MPIDI_Datatype * dt); -void -MPIDI_Win_datatype_map (MPIDI_Datatype * dt); -void -MPIDI_Win_datatype_unmap(MPIDI_Datatype * dt); - -void -MPIDI_WinCtrlSend(pami_context_t context, - MPIDI_Win_control_t *control, - int rank, - MPIR_Win *win); - -void -MPIDI_WinLockReq_proc(pami_context_t context, - const MPIDI_Win_control_t * info, - unsigned peer); -void -MPIDI_WinLockAck_proc(pami_context_t context, - const MPIDI_Win_control_t * info, - unsigned peer); -void -MPIDI_WinUnlock_proc(pami_context_t context, - const MPIDI_Win_control_t * info, - unsigned peer); - -void -MPIDI_WinComplete_proc(pami_context_t context, - const MPIDI_Win_control_t * info, - unsigned peer); -void -MPIDI_WinPost_proc(pami_context_t context, - const MPIDI_Win_control_t * info, - unsigned peer); - -void -MPIDI_Win_DoneCB(pami_context_t context, - void * cookie, - pami_result_t result); -void -MPIDI_WinUnlockDoneCB(pami_context_t context, - void * cookie, - pami_result_t result); - -void -MPIDI_WinAccumCB(pami_context_t context, - void * cookie, - const void * _msginfo, - size_t msginfo_size, - const void * sndbuf, - size_t sndlen, - pami_endpoint_t sender, - pami_recv_t * recv); -int -MPIDI_Win_init( MPI_Aint length, - int disp_unit, - MPIR_Win **win_ptr, - MPIR_Info *info, - MPIR_Comm *comm_ptr, - int create_flavor, - int model); -int -MPIDI_Win_allgather(MPI_Aint size, - MPIR_Win **win_ptr); - -void -MPIDI_WinLockAdvance(pami_context_t context, - MPIR_Win * win); -void -MPIDI_WinLockAck_post(pami_context_t context, - unsigned peer, - MPIR_Win *win); - -void -MPIDI_WinLockReq_proc(pami_context_t context, - const MPIDI_Win_control_t * info, - unsigned peer); - -int -MPIDI_Datatype_is_pami_rmw_supported(MPI_Datatype datatype, - pami_type_t *pami_type, - MPI_Op op, - pami_atomic_t *pami_op); - -int -MPIDI_valid_group_rank(int lpid, - MPIR_Group *grp); -void -MPIDI_WinLockAllAck_post(pami_context_t context, - unsigned peer, - MPIR_Win * win); - - -#endif diff --git a/src/mpid/pamid/src/onesided/mpidi_win_control.c b/src/mpid/pamid/src/onesided/mpidi_win_control.c deleted file mode 100644 index f10d45d..0000000 --- a/src/mpid/pamid/src/onesided/mpidi_win_control.c +++ /dev/null @@ -1,149 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/onesided/mpidi_win_control.c - * \brief ??? - */ -#include "mpidi_onesided.h" - - -void -MPIDI_WinCtrlSend(pami_context_t context, - MPIDI_Win_control_t *control, - int rank, - MPIR_Win *win) -{ - pami_task_t taskid; - MPIDI_WinLock_info *winLock; - control->win = win->mpid.info[rank].win; - control->rank = win->comm_ptr->rank; - taskid=MPID_VCR_GET_LPID(win->comm_ptr->vcr,rank); - - pami_endpoint_t dest; - pami_result_t rc; - taskid=MPID_VCR_GET_LPID(win->comm_ptr->vcr,rank); - rc = PAMI_Endpoint_create(MPIDI_Client,taskid, 0, &dest); - MPID_assert(rc == PAMI_SUCCESS); - - if ((control->type == MPIDI_WIN_MSGTYPE_UNLOCK) || - (control->type == MPIDI_WIN_MSGTYPE_UNLOCKALL)) { - pami_send_t params = { - .send = { - .dispatch = MPIDI_Protocols_WinCtrl, - .dest = dest, - .header = { - .iov_base = control, - .iov_len = sizeof(MPIDI_Win_control_t), - }, - }, - .events = { - .cookie = win, - .local_fn = NULL, - .remote_fn= MPIDI_WinUnlockDoneCB, - }, - }; - rc = PAMI_Send(context, &params); - } else { - pami_send_immediate_t params = { - .dispatch = MPIDI_Protocols_WinCtrl, - .dest = dest, - .header = { - .iov_base = control, - .iov_len = sizeof(MPIDI_Win_control_t), - }, - .data = { - .iov_base = NULL, - .iov_len = 0, - }, - }; - rc = PAMI_Send_immediate(context, &params); - } - if ((control->type == MPIDI_WIN_MSGTYPE_LOCKALLREQ) || (control->type == MPIDI_WIN_MSGTYPE_UNLOCKALL)) { - winLock = (MPIDI_WinLock_info *) control->flagAddr; - winLock->done = 1; - } - MPID_assert(rc == PAMI_SUCCESS); - -} - -void -MPIDI_WinUnlockDoneCB(pami_context_t context, - void * cookie, - pami_result_t result) -{ - MPIR_Win *win = (MPIR_Win *)cookie; - if (win->mpid.sync.origin_epoch_type == MPID_EPOTYPE_LOCK) { - win->mpid.sync.lock.remote.locked = 0; - } - else if (win->mpid.sync.origin_epoch_type == MPID_EPOTYPE_LOCK_ALL) - { - MPID_assert((int)win->mpid.sync.lock.remote.allLocked > 0); - win->mpid.sync.lock.remote.allLocked -= 1; - } - else - MPID_assert_always(0); -} - - -void -MPIDI_WinControlCB(pami_context_t context, - void * cookie, - const void * _control, - size_t size, - const void * sndbuf, - size_t sndlen, - pami_endpoint_t sender, - pami_recv_t * recv) -{ - MPID_assert(recv == NULL); - MPID_assert(sndlen == 0); - MPID_assert(_control != NULL); - MPID_assert(size == sizeof(MPIDI_Win_control_t)); - const MPIDI_Win_control_t *control = (const MPIDI_Win_control_t *)_control; - pami_task_t senderrank = PAMIX_Endpoint_query(sender); - - switch (control->type) - { - case MPIDI_WIN_MSGTYPE_LOCKREQ: - case MPIDI_WIN_MSGTYPE_LOCKALLREQ: - MPIDI_WinLockReq_proc(context, control, senderrank); - break; - case MPIDI_WIN_MSGTYPE_LOCKACK: - case MPIDI_WIN_MSGTYPE_LOCKALLACK: - MPIDI_WinLockAck_proc(context, control, senderrank); - break; - case MPIDI_WIN_MSGTYPE_UNLOCK: - case MPIDI_WIN_MSGTYPE_UNLOCKALL: - MPIDI_WinUnlock_proc(context, control, senderrank); - break; - case MPIDI_WIN_MSGTYPE_COMPLETE: - MPIDI_WinComplete_proc(context, control, senderrank); - break; - case MPIDI_WIN_MSGTYPE_POST: - MPIDI_WinPost_proc(context, control, senderrank); - break; - - default: - fprintf(stderr, "Bad win-control type: 0x%08x %d\n", - control->type, - control->type); - MPID_abort(); - } - - MPIDI_Progress_signal(); -} diff --git a/src/mpid/pamid/src/pamix/Makefile.mk b/src/mpid/pamid/src/pamix/Makefile.mk deleted file mode 100644 index 0ab3a81..0000000 --- a/src/mpid/pamid/src/pamix/Makefile.mk +++ /dev/null @@ -1,29 +0,0 @@ -# begin_generated_IBM_copyright_prolog -# -# This is an automatically generated copyright prolog. -# After initializing, DO NOT MODIFY OR MOVE -# --------------------------------------------------------------- -# Licensed Materials - Property of IBM -# Blue Gene/Q 5765-PER 5765-PRP -# -# (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved -# US Government Users Restricted Rights - -# Use, duplication, or disclosure restricted -# by GSA ADP Schedule Contract with IBM Corp. -# -# --------------------------------------------------------------- -# -# end_generated_IBM_copyright_prolog -# -*- mode: makefile-gmake; -*- - -# note that the includes always happen but the effects of their contents are -# affected by "if BUILD_PAMID" -if BUILD_PAMID - - -mpi_core_sources += \ - src/mpid/pamid/src/pamix/pamix.c - - -endif BUILD_PAMID - diff --git a/src/mpid/pamid/src/pamix/pamix.c b/src/mpid/pamid/src/pamix/pamix.c deleted file mode 100644 index 4077318..0000000 --- a/src/mpid/pamid/src/pamix/pamix.c +++ /dev/null @@ -1,347 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/pamix/pamix.c - * \brief This file contains routines to make the PAMI API usable for MPI internals. It is less likely that - * MPI apps will use these routines. - */ - -#include <assert.h> -#include <limits.h> -#include "mpidimpl.h" -#include <pamix.h> - -#define PAMIX_assert_always(x) assert(x) -#if ASSERT_LEVEL==0 -#define PAMIX_assert(x) -#elif ASSERT_LEVEL>=1 -#define PAMIX_assert(x) assert(x) -#endif - -#ifdef __BGQ__ -#define __BG__ -#include <spi/include/kernel/location.h> -#endif - -typedef pami_result_t (*pamix_progress_register_fn) (pami_context_t context, - pamix_progress_function progress_fn, - pamix_progress_function suspend_fn, - pamix_progress_function resume_fn, - void * cookie); -typedef pami_result_t (*pamix_progress_enable_fn) (pami_context_t context, - pamix_progress_t event_type); -typedef pami_result_t (*pamix_progress_disable_fn) (pami_context_t context, - pamix_progress_t event_type); - -#if defined(__BG__) -typedef const pamix_torus_info_t* (*pamix_torus_info_fn) (); -typedef pami_result_t (*pamix_task2torus_fn) (pami_task_t, size_t[]); -typedef pami_result_t (*pamix_torus2task_fn) (size_t[], pami_task_t *); -#endif - - -static struct -{ - pamix_progress_register_fn progress_register; - pamix_progress_enable_fn progress_enable; - pamix_progress_disable_fn progress_disable; - -#if defined(__BG__) - pamix_torus_info_fn torus_info; - pamix_task2torus_fn task2torus; - pamix_torus2task_fn torus2task; -#endif -} PAMIX_Functions = {0}; - -pamix_extension_info_t PAMIX_Extensions; - - -#define PAMI_EXTENSION_OPEN(client, name, ext) \ -({ \ - pami_result_t rc; \ - rc = PAMI_Extension_open(client, name, ext); \ - PAMIX_assert_always(rc == PAMI_SUCCESS); \ -}) -#define PAMI_EXTENSION_FUNCTION(type, name, ext) \ -({ \ - void* fn; \ - fn = PAMI_Extension_symbol(ext, name); \ - (type)fn; \ -}) -void -PAMIX_Initialize(pami_client_t client) -{ - PAMI_EXTENSION_OPEN(client, "EXT_async_progress", &PAMIX_Extensions.progress); - PAMIX_Functions.progress_register = PAMI_EXTENSION_FUNCTION(pamix_progress_register_fn, "register", PAMIX_Extensions.progress); - PAMIX_Functions.progress_enable = PAMI_EXTENSION_FUNCTION(pamix_progress_enable_fn, "enable", PAMIX_Extensions.progress); - PAMIX_Functions.progress_disable = PAMI_EXTENSION_FUNCTION(pamix_progress_disable_fn, "disable", PAMIX_Extensions.progress); - -#if defined(__BG__) - PAMI_EXTENSION_OPEN(client, "EXT_torus_network", &PAMIX_Extensions.torus); - PAMIX_Functions.torus_info = PAMI_EXTENSION_FUNCTION(pamix_torus_info_fn, "information", PAMIX_Extensions.torus); - PAMIX_Functions.task2torus = PAMI_EXTENSION_FUNCTION(pamix_task2torus_fn, "task2torus", PAMIX_Extensions.torus); - PAMIX_Functions.torus2task = PAMI_EXTENSION_FUNCTION(pamix_torus2task_fn, "torus2task", PAMIX_Extensions.torus); -#endif - -#ifdef PAMIX_IS_LOCAL_TASK - { - PAMIX_Extensions.is_local_task.base = NULL; - PAMIX_Extensions.is_local_task.stride = 0; - PAMIX_Extensions.is_local_task.bitmask = 0; - PAMIX_Extensions.is_local_task.node_info = NULL; - PAMIX_Extensions.is_local_task.status = PAMI_Extension_open(client, "EXT_is_local_task", - &PAMIX_Extensions.is_local_task.extension); - - if (PAMIX_Extensions.is_local_task.status == PAMI_SUCCESS) - { - PAMIX_Extensions.is_local_task.base = PAMI_EXTENSION_FUNCTION(uint8_t *, "base", PAMIX_Extensions.is_local_task.extension); - PAMIX_Extensions.is_local_task.stride = PAMI_EXTENSION_FUNCTION(uintptr_t, "stride", PAMIX_Extensions.is_local_task.extension); - PAMIX_Extensions.is_local_task.bitmask = PAMI_EXTENSION_FUNCTION(uintptr_t, "bitmask", PAMIX_Extensions.is_local_task.extension); -#if defined(MPID_USE_NODE_IDS) - PAMIX_Extensions.is_local_task.node_info = PAMI_EXTENSION_FUNCTION(node_info_fn, "get_node_info", PAMIX_Extensions.is_local_task.extension); - if (PAMIX_Extensions.is_local_task.node_info == NULL) - { - /* The "node information" is not available via the "is_local_task" - * extension to pami - possibly due to a downlevel version of pami. - * */ - PAMIX_Extensions.is_local_task.node_info = (node_info_fn) PAMIX_is_local_task_get_node_info; - } -#endif - } - -#if defined(PAMIX_IS_LOCAL_TASK_STRIDE) && defined(PAMIX_IS_LOCAL_TASK_BITMASK) - /* - * If the compile-time stride and bitmask values are defined, then assert - * that the extension was open successfully and the base pointer is valid to - * avoid a null dereference in the PAMIX_Task_is_local macro and assert that - * the compile-time values match the values specified by the extension. - */ - PAMIX_assert_always(PAMIX_Extensions.is_local_task.status == PAMI_SUCCESS); - PAMIX_assert_always(PAMIX_Extensions.is_local_task.base != NULL); - PAMIX_assert_always(PAMIX_IS_LOCAL_TASK_STRIDE == PAMIX_Extensions.is_local_task.stride); - PAMIX_assert_always(PAMIX_IS_LOCAL_TASK_BITMASK == PAMIX_Extensions.is_local_task.bitmask); -#endif - } -#endif /* PAMIX_IS_LOCAL_TASK */ -} - - -void -PAMIX_Finalize(pami_client_t client) -{ - pami_result_t rc; - rc = PAMI_Extension_close(PAMIX_Extensions.progress); - PAMIX_assert_always(rc == PAMI_SUCCESS); - - if (PAMIX_Extensions.is_local_task.status == PAMI_SUCCESS) - { - rc = PAMI_Extension_close(PAMIX_Extensions.is_local_task.extension); - PAMIX_assert_always(rc == PAMI_SUCCESS); - - PAMIX_Extensions.is_local_task.base = NULL; - PAMIX_Extensions.is_local_task.stride = 0; - PAMIX_Extensions.is_local_task.bitmask = 0; - PAMIX_Extensions.is_local_task.status = PAMI_ERROR; - } - -#if defined(__BG__) - rc = PAMI_Extension_close(PAMIX_Extensions.torus); - PAMIX_assert_always(rc == PAMI_SUCCESS); -#endif -} - - -pami_configuration_t -PAMIX_Client_query(pami_client_t client, - pami_attribute_name_t name) -{ - pami_result_t rc; - pami_configuration_t query; - query.name = name; - rc = PAMI_Client_query(client, &query, 1); - PAMIX_assert_always(rc == PAMI_SUCCESS); - return query; -} - - -static inline pami_configuration_t -PAMIX_Dispatch_query(pami_context_t context, - size_t dispatch, - pami_attribute_name_t name) -{ - pami_result_t rc; - pami_configuration_t query; - query.name = name; - rc = PAMI_Dispatch_query(context, dispatch, &query, 1); - PAMIX_assert_always(rc == PAMI_SUCCESS); - return query; -} - - -void -PAMIX_Dispatch_set(pami_context_t context[], - size_t num_contexts, - size_t dispatch, - pami_dispatch_callback_function fn, - pami_dispatch_hint_t options, - size_t * immediate_max) -{ - pami_result_t rc; - size_t i; - size_t last_immediate_max = (size_t)-1; - for (i=0; i<num_contexts; ++i) { - rc = PAMI_Dispatch_set(context[i], - dispatch, - fn, - (void*)i, - options); - PAMIX_assert_always(rc == PAMI_SUCCESS); - - size_t size; - size = PAMIX_Dispatch_query(context[i], dispatch, PAMI_DISPATCH_SEND_IMMEDIATE_MAX).value.intval; - last_immediate_max = MPL_MIN(size, last_immediate_max); - size = PAMIX_Dispatch_query(context[i], dispatch, PAMI_DISPATCH_RECV_IMMEDIATE_MAX).value.intval; - last_immediate_max = MPL_MIN(size, last_immediate_max); - } - - if (immediate_max != NULL) - *immediate_max = last_immediate_max; -} - - -pami_task_t -PAMIX_Endpoint_query(pami_endpoint_t endpoint) -{ - pami_task_t rank; - size_t offset; - - pami_result_t rc; - rc = PAMI_Endpoint_query(endpoint, &rank, &offset); - PAMIX_assert(rc == PAMI_SUCCESS); - - return rank; -} - - -void -PAMIX_Progress_register(pami_context_t context, - pamix_progress_function progress_fn, - pamix_progress_function suspend_fn, - pamix_progress_function resume_fn, - void * cookie) -{ - PAMIX_assert_always(PAMIX_Functions.progress_register != NULL); - pami_result_t rc; - rc = PAMIX_Functions.progress_register(context, progress_fn,suspend_fn, resume_fn, cookie); - PAMIX_assert_always(rc == PAMI_SUCCESS); -} - - -void -PAMIX_Progress_enable(pami_context_t context, - pamix_progress_t event_type) -{ - PAMIX_assert(PAMIX_Functions.progress_enable != NULL); - pami_result_t rc; - rc = PAMIX_Functions.progress_enable(context, event_type); - PAMIX_assert(rc == PAMI_SUCCESS); -} - - -void -PAMIX_Progress_disable(pami_context_t context, - pamix_progress_t event_type) -{ - PAMIX_assert(PAMIX_Functions.progress_disable != NULL); - pami_result_t rc; - rc = PAMIX_Functions.progress_disable(context, event_type); - PAMIX_assert(rc == PAMI_SUCCESS); -} - - -pami_result_t -PAMIX_is_local_task_get_node_info(pami_task_t task, - uint32_t *node_id, - uint32_t *offset, - uint32_t *max_nodes) -{ -#if defined(__BG__) - size_t coords[5]; - if (PAMI_SUCCESS == PAMIX_Task2torus(task, coords)) - { - const pamix_torus_info_t * info = PAMIX_Torus_info(); - - *node_id = - coords[0] + - coords[1] * info->size[0] + - coords[2] * info->size[0] * info->size[1] + - coords[3] * info->size[0] * info->size[1] * info->size[2] + - coords[4] * info->size[0] * info->size[1] * info->size[2] * info->size[3]; - - *max_nodes = info->size[0] * info->size[1] * info->size[2] * info->size[3] * info->size[4]; - - *offset = 0; /* ???????????????????????????????? */ - } -#else - MPID_abort(); /* Other platforms should not need a fallback implementation */ -#endif - - return PAMI_SUCCESS; -} - -#if defined(__BG__) - -const pamix_torus_info_t * -PAMIX_Torus_info() -{ - PAMIX_assert(PAMIX_Functions.torus_info != NULL); - return PAMIX_Functions.torus_info(); -} - - -int -PAMIX_Task2torus(pami_task_t task_id, - size_t coords[]) -{ - PAMIX_assert(PAMIX_Functions.task2torus != NULL); - pami_result_t rc; - rc = PAMIX_Functions.task2torus(task_id, coords); - return rc; -} - - -#include <stdio.h> -int -PAMIX_Torus2task(size_t coords[], - pami_task_t * task_id) -{ - PAMIX_assert(PAMIX_Functions.torus2task != NULL); - pami_result_t rc; - rc = PAMIX_Functions.torus2task(coords, task_id); - return rc; -#if 0 - if(rc != PAMI_SUCCESS) - { - fprintf(stderr,"coords in: %zu %zu %zu %zu %zu, rc: %d\n", coords[0], coords[1], coords[2],coords[3],coords[4], rc); - PAMIX_assert(rc == PAMI_SUCCESS); - } - else return rc; -#endif -} - -#endif diff --git a/src/mpid/pamid/src/pt2pt/Makefile.mk b/src/mpid/pamid/src/pt2pt/Makefile.mk deleted file mode 100644 index c8dc0e4..0000000 --- a/src/mpid/pamid/src/pt2pt/Makefile.mk +++ /dev/null @@ -1,50 +0,0 @@ -# begin_generated_IBM_copyright_prolog -# -# This is an automatically generated copyright prolog. -# After initializing, DO NOT MODIFY OR MOVE -# --------------------------------------------------------------- -# Licensed Materials - Property of IBM -# Blue Gene/Q 5765-PER 5765-PRP -# -# (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved -# US Government Users Restricted Rights - -# Use, duplication, or disclosure restricted -# by GSA ADP Schedule Contract with IBM Corp. -# -# --------------------------------------------------------------- -# -# end_generated_IBM_copyright_prolog -# -*- mode: makefile-gmake; -*- - -# note that the includes always happen but the effects of their contents are -# affected by "if BUILD_PAMID" -if BUILD_PAMID - - -noinst_HEADERS += \ - src/mpid/pamid/src/pt2pt/mpid_send.h \ - src/mpid/pamid/src/pt2pt/mpidi_recv.h \ - src/mpid/pamid/src/pt2pt/mpidi_send.h \ - src/mpid/pamid/src/pt2pt/mpid_irecv.h \ - src/mpid/pamid/src/pt2pt/mpid_isend.h - -include $(top_srcdir)/src/mpid/pamid/src/pt2pt/persistent/Makefile.mk - -mpi_core_sources += \ - src/mpid/pamid/src/pt2pt/mpid_cancel.c \ - src/mpid/pamid/src/pt2pt/mpid_issend.c \ - src/mpid/pamid/src/pt2pt/mpid_recv.c \ - src/mpid/pamid/src/pt2pt/mpid_ssend.c \ - src/mpid/pamid/src/pt2pt/mpidi_callback_eager.c \ - src/mpid/pamid/src/pt2pt/mpidi_callback_rzv.c \ - src/mpid/pamid/src/pt2pt/mpidi_callback_short.c \ - src/mpid/pamid/src/pt2pt/mpidi_callback_util.c \ - src/mpid/pamid/src/pt2pt/mpidi_control.c \ - src/mpid/pamid/src/pt2pt/mpidi_done.c \ - src/mpid/pamid/src/pt2pt/mpidi_recvmsg.c \ - src/mpid/pamid/src/pt2pt/mpidi_rendezvous.c \ - src/mpid/pamid/src/pt2pt/mpidi_sendmsg.c - - -endif BUILD_PAMID - diff --git a/src/mpid/pamid/src/pt2pt/mpid_cancel.c b/src/mpid/pamid/src/pt2pt/mpid_cancel.c deleted file mode 100644 index 9b8d30f..0000000 --- a/src/mpid/pamid/src/pt2pt/mpid_cancel.c +++ /dev/null @@ -1,138 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/pt2pt/mpid_cancel.c - * \brief Device interface for canceling an MPI Recv - */ -#include <mpidimpl.h> - - -int -MPID_Cancel_recv(MPIR_Request * rreq) -{ - MPID_assert(rreq->kind == MPIR_REQUEST_KIND__RECV); - if (MPIDI_Recvq_FDPR(rreq)) - { - MPIR_STATUS_SET_CANCEL_BIT(rreq->status, TRUE); - MPIR_STATUS_SET_COUNT(rreq->status, 0); - MPIDI_Request_complete(rreq); - } - /* This is successful, even if the recv isn't cancelled */ - return MPI_SUCCESS; -} - - -/** - * \brief Cancel an MPI_Isend() - * - * \param[in] req The request element to cancel - * - * \return The same as MPIDI_CtrlSend() - */ -static inline pami_result_t -MPIDI_CancelReq_post(pami_context_t context, void * _req) -{ - MPIR_Request * req = (MPIR_Request*)_req; - MPID_assert(req != NULL); - - /* ------------------------------------------------- */ - /* Check if we already have a cancel request pending */ - /* ------------------------------------------------- */ - if (MPIDI_Request_cancel_pending(req)) - { - MPIDI_Request_complete(req); - return PAMI_SUCCESS; - } - - MPIDI_MsgInfo cancel = { - .MPItag = MPIDI_Request_getMatchTag(req), - .MPIrank = MPIDI_Request_getMatchRank(req), - .MPIctxt = MPIDI_Request_getMatchCtxt(req), - .req = req->handle, - }; - cancel.control = MPIDI_CONTROL_CANCEL_REQUEST; - - pami_endpoint_t dest; - MPIDI_Context_endpoint(req, &dest); - - pami_send_immediate_t params = { - .dispatch = MPIDI_Protocols_Cancel, - .dest = dest, - .header = { - .iov_base= &cancel, - .iov_len= sizeof(MPIDI_MsgInfo), - }, - }; - - TRACE_ERR("Running posted cancel for request=%p local=%u remote=%u\n", - req, MPIR_Process.comm_world->rank, MPIDI_Request_getPeerRank_pami(req)); - pami_result_t rc; - rc = PAMI_Send_immediate(context, &params); - MPID_assert(rc == PAMI_SUCCESS); - - return PAMI_SUCCESS; -} - - -int -MPID_Cancel_send(MPIR_Request * sreq) -{ - MPID_assert(sreq != NULL); - - if(!sreq->comm) - return MPI_SUCCESS; - - MPIDI_Request_uncomplete(sreq); - /* TRACE_ERR("Posting cancel for request=%p cc(curr)=%d ref(curr)=%d\n", sreq, val+1, MPIR_Object_get_ref(sreq)); */ - - pami_context_t context = MPIDI_Context_local(sreq); - -#if (MPICH_THREAD_GRANULARITY == MPICH_THREAD_GRANULARITY__POBJ) - if (likely(MPIDI_Process.perobj.context_post.active > 0)) - { - /* This leaks intentionally. At this time, the amount of work - * required to avoid a leak here just isn't worth it. - * Hopefully people aren't cancelling sends too much. - */ - pami_work_t * work = MPL_malloc(sizeof(pami_work_t)); - PAMI_Context_post(context, work, MPIDI_CancelReq_post, sreq); - } - else - { - /* Lock access to the context. For more information see discussion of the - * simplifying assumption that the "per object" mpich lock mode does not - * expect a completely single threaded run environment in the file - * src/mpid/pamid/src/mpid_progress.h - */ - PAMI_Context_lock(context); - MPIDI_CancelReq_post(context, sreq); - PAMI_Context_unlock(context); - } -#else /* (MPICH_THREAD_GRANULARITY != MPICH_THREAD_GRANULARITY__POBJ) */ - /* - * It is not necessary to lock the context before access in the "global" mpich - * lock mode because all application threads must first acquire the global - * mpich lock upon entry into the library, and any active async threads must - * first acquire the global mpich lock before accessing internal mpich data - * structures or accessing pami objects. This makes the context lock redundant. - */ - MPIDI_CancelReq_post(context, sreq); -#endif - - return MPI_SUCCESS; -} diff --git a/src/mpid/pamid/src/pt2pt/mpid_irecv.h b/src/mpid/pamid/src/pt2pt/mpid_irecv.h deleted file mode 100644 index a0b6daf..0000000 --- a/src/mpid/pamid/src/pt2pt/mpid_irecv.h +++ /dev/null @@ -1,67 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/pt2pt/mpid_irecv.h - * \brief ADI level implemenation of MPI_Irecv() - */ -#ifndef __src_pt2pt_mpid_irecv_h__ -#define __src_pt2pt_mpid_irecv_h__ - -#include <mpidimpl.h> -#include "mpidi_recv.h" - -#define MPID_Irecv MPID_Irecv_inline - -/** - * \brief ADI level implemenation of MPI_Irecv() - * - * \param[in] buf The buffer to receive into - * \param[in] count Number of expected elements in the buffer - * \param[in] datatype The datatype of each element - * \param[in] rank The sending rank - * \param[in] tag The message tag - * \param[in] comm Pointer to the communicator - * \param[in] context_offset Offset from the communicator context ID - * - * \param[out] request Return a pointer to the new request object - * - * \returns An MPI Error code - */ -static inline int -MPID_Irecv_inline(void * buf, - MPI_Aint count, - MPI_Datatype datatype, - int rank, - int tag, - MPIR_Comm * comm, - int context_offset, - MPIR_Request ** request) -{ - return MPIDI_Recv(buf, - count, - datatype, - rank, - tag, - comm, - context_offset, - 0, - MPI_STATUS_IGNORE, - request); -} - -#endif diff --git a/src/mpid/pamid/src/pt2pt/mpid_isend.h b/src/mpid/pamid/src/pt2pt/mpid_isend.h deleted file mode 100644 index 7928908..0000000 --- a/src/mpid/pamid/src/pt2pt/mpid_isend.h +++ /dev/null @@ -1,129 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/pt2pt/mpid_isend.h - * \brief ??? - */ - -#ifndef __src_pt2pt_mpid_isend_h__ -#define __src_pt2pt_mpid_isend_h__ - - -#define MPID_Isend MPID_Isend_inline - - -static inline unsigned -MPIDI_Context_hash(pami_task_t rank, unsigned ctxt, unsigned bias, unsigned ncontexts) -{ -#if (MPICH_THREAD_GRANULARITY == MPICH_THREAD_GRANULARITY__POBJ) - return (( rank + ctxt + bias ) & (ncontexts-1)); -#else - /* The 'global' mpich lock mode only supports a single context. See - * discussion in mpich/src/mpid/pamid/src/mpid_init.c for more information. - */ - return 0; -#endif -} -static inline void -MPIDI_Context_endpoint(MPIR_Request * req, pami_endpoint_t * e) -{ - pami_task_t remote = MPIDI_Request_getPeerRank_pami(req); - pami_task_t local = MPIR_Process.comm_world->rank; - unsigned rctxt = MPIDI_Context_hash(local, req->comm->context_id, MPIDI_Process.avail_contexts>>1, MPIDI_Process.avail_contexts); - - pami_result_t rc; - rc = PAMI_Endpoint_create(MPIDI_Client, remote, rctxt, e); - MPID_assert(rc == PAMI_SUCCESS); -} -static inline pami_context_t -MPIDI_Context_local(MPIR_Request * req) -{ - pami_task_t remote = MPIDI_Request_getPeerRank_comm(req); - unsigned lctxt = MPIDI_Context_hash(remote, req->comm->context_id, 0, MPIDI_Process.avail_contexts); - MPID_assert(lctxt < MPIDI_Process.avail_contexts); - return MPIDI_Context[lctxt]; -} - - -/** - * \brief ADI level implemenation of MPI_Isend() - * - * \param[in] buf The buffer to send - * \param[in] count Number of elements in the buffer - * \param[in] datatype The datatype of each element - * \param[in] rank The destination rank - * \param[in] tag The message tag - * \param[in] comm Pointer to the communicator - * \param[in] context_offset Offset from the communicator context ID - * \param[out] request Return a pointer to the new request object - * - * \returns An MPI Error code - * - * This is a slight variation on mpid_send.c - basically, we *always* - * want to return a send request even if the request is already - * complete (as is in the case of sending to a NULL rank). - */ -static inline int -MPID_Isend_inline(const void * buf, - MPI_Aint count, - MPI_Datatype datatype, - int rank, - int tag, - MPIR_Comm * comm, - int context_offset, - MPIR_Request ** request) -{ - /* ---------------------------------------------------- */ - /* special case: PROC null handled by handoff function */ - /* ---------------------------------------------------- */ - - /* --------------------- */ - /* create a send request */ - /* --------------------- */ - MPIR_Request * sreq = *request = MPIDI_Request_create2_fast(); - - unsigned context_id = comm->context_id; - /* match info */ - MPIDI_Request_setMatch(sreq, tag, comm->rank, context_id+context_offset); - - /* data buffer info */ - sreq->mpid.userbuf = (void*)buf; - sreq->mpid.userbufcount = count; - sreq->mpid.datatype = datatype; - - /* Enable passing in MPI_PROC_NULL, do the translation in the - handoff function */ - MPIDI_Request_setPeerRank_comm(sreq, rank); - - unsigned ncontexts = MPIDI_Process.avail_contexts; - /* communicator & destination info */ - sreq->comm = comm; - sreq->kind = MPIR_REQUEST_KIND__SEND; - MPIR_Comm_add_ref(comm); - - pami_context_t context = MPIDI_Context[MPIDI_Context_hash(rank, context_id, 0, ncontexts)]; - if (context_offset == 0) - MPIDI_Context_post(context, &sreq->mpid.post_request, MPIDI_Isend_handoff, sreq); - else - MPIDI_Context_post(context, &sreq->mpid.post_request, MPIDI_Isend_handoff_internal, sreq); - - return MPI_SUCCESS; -} - - -#endif diff --git a/src/mpid/pamid/src/pt2pt/mpid_issend.c b/src/mpid/pamid/src/pt2pt/mpid_issend.c deleted file mode 100644 index 21510a5..0000000 --- a/src/mpid/pamid/src/pt2pt/mpid_issend.c +++ /dev/null @@ -1,62 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/pt2pt/mpid_issend.c - * \brief ADI level implemenation of MPI_Issend() - */ -#include <mpidimpl.h> -#include "mpidi_send.h" - -/** - * \brief ADI level implemenation of MPI_Issend() - * - * \param[in] buf The buffer to send - * \param[in] count Number of elements in the buffer - * \param[in] datatype The datatype of each element - * \param[in] rank The destination rank - * \param[in] tag The message tag - * \param[in] comm Pointer to the communicator - * \param[in] context_offset Offset from the communicator context ID - * \param[out] request Return a pointer to the new request object - * - * \returns An MPI Error code - * - * This is a slight variation on mpid_ssend.c - basically, we *always* - * want to return a send request even if the request is already - * complete (as is in the case of sending to a NULL rank). - */ -int MPID_Issend(const void * buf, - int count, - MPI_Datatype datatype, - int rank, - int tag, - MPIR_Comm * comm, - int context_offset, - MPIR_Request ** request) -{ - return MPIDI_Send(buf, - count, - datatype, - rank, - tag, - comm, - context_offset, - 0, - 1, - request); -} diff --git a/src/mpid/pamid/src/pt2pt/mpid_recv.c b/src/mpid/pamid/src/pt2pt/mpid_recv.c deleted file mode 100644 index 15c88b9..0000000 --- a/src/mpid/pamid/src/pt2pt/mpid_recv.c +++ /dev/null @@ -1,60 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/pt2pt/mpid_recv.c - * \brief ADI level implemenation of MPI_Recv() - */ -#include <mpidimpl.h> -#include "mpidi_recv.h" - -/** - * \brief ADI level implemenation of MPI_Recv() - * - * \param[in] buf The buffer to receive into - * \param[in] count Number of expected elements in the buffer - * \param[in] datatype The datatype of each element - * \param[in] rank The sending rank - * \param[in] tag The message tag - * \param[in] comm Pointer to the communicator - * \param[in] context_offset Offset from the communicator context ID - * \param[out] status Update the status structure - * \param[out] request Return a pointer to the new request object - * - * \returns An MPI Error code - */ -int MPID_Recv(void * buf, - int count, - MPI_Datatype datatype, - int rank, - int tag, - MPIR_Comm * comm, - int context_offset, - MPI_Status * status, - MPIR_Request ** request) -{ - return MPIDI_Recv(buf, - count, - datatype, - rank, - tag, - comm, - context_offset, - 0, - status, - request); -} diff --git a/src/mpid/pamid/src/pt2pt/mpid_send.h b/src/mpid/pamid/src/pt2pt/mpid_send.h deleted file mode 100644 index 86c23f1..0000000 --- a/src/mpid/pamid/src/pt2pt/mpid_send.h +++ /dev/null @@ -1,67 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/pt2pt/mpid_send.h - * \brief ADI level implemenation of MPI_Send() - */ -#ifndef __src_pt2pt_mpid_send_h__ -#define __src_pt2pt_mpid_send_h__ - -#include <mpidimpl.h> -#include "mpidi_send.h" - -#define MPID_Send MPID_Send_inline - -/** - * \brief ADI level implemenation of MPI_Send() - * - * \param[in] buf The buffer to send - * \param[in] count Number of elements in the buffer - * \param[in] datatype The datatype of each element - * \param[in] rank The destination rank - * \param[in] tag The message tag - * \param[in] comm Pointer to the communicator - * \param[in] context_offset Offset from the communicator context ID - * \param[out] request Return a pointer to the new request object - * - * \returns An MPI Error code - */ -static inline int -MPID_Send_inline(const void * buf, - MPI_Aint count, - MPI_Datatype datatype, - int rank, - int tag, - MPIR_Comm * comm, - int context_offset, - MPIR_Request ** request) -{ - return MPIDI_Send(buf, - count, - datatype, - rank, - tag, - comm, - context_offset, - 1, - 0, - request); -} - - -#endif diff --git a/src/mpid/pamid/src/pt2pt/mpid_ssend.c b/src/mpid/pamid/src/pt2pt/mpid_ssend.c deleted file mode 100644 index 58ce5f9..0000000 --- a/src/mpid/pamid/src/pt2pt/mpid_ssend.c +++ /dev/null @@ -1,58 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/pt2pt/mpid_ssend.c - * \brief ADI level implemenation of MPI_Ssend() - */ -#include <mpidimpl.h> -#include "mpidi_send.h" - -/** - * \brief ADI level implemenation of MPI_Ssend() - * - * \param[in] buf The buffer to send - * \param[in] count Number of elements in the buffer - * \param[in] datatype The datatype of each element - * \param[in] rank The destination rank - * \param[in] tag The message tag - * \param[in] comm Pointer to the communicator - * \param[in] context_offset Offset from the communicator context ID - * \param[out] request Return a pointer to the new request object - * - * \returns An MPI Error code - */ -int MPID_Ssend(const void * buf, - int count, - MPI_Datatype datatype, - int rank, - int tag, - MPIR_Comm * comm, - int context_offset, - MPIR_Request ** request) -{ - return MPIDI_Send(buf, - count, - datatype, - rank, - tag, - comm, - context_offset, - 1, - 1, - request); -} diff --git a/src/mpid/pamid/src/pt2pt/mpidi_callback_eager.c b/src/mpid/pamid/src/pt2pt/mpidi_callback_eager.c deleted file mode 100644 index 51b876e..0000000 --- a/src/mpid/pamid/src/pt2pt/mpidi_callback_eager.c +++ /dev/null @@ -1,312 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/pt2pt/mpidi_callback_eager.c - * \brief The callback for a new eager message - */ -#include <mpidimpl.h> - - -/** - * \brief The standard callback for a new message - * - * \param[in] context The context on which the message is being received. - * \param[in] cookie Unused - * \param[in] _msginfo The header information - * \param[in] msginfo_size The size of the header information - * \param[in] sndbuf If the message is short, this is the data - * \param[in] sndlen The size of the incoming data - * \param[out] recv If the message is long, this tells the message layer how to handle the data. - */ -void -MPIDI_RecvCB(pami_context_t context, - void * cookie, - const void * _msginfo, - size_t msginfo_size, - const void * sndbuf, - size_t sndlen, - pami_endpoint_t sender, - pami_recv_t * recv) -{ - const MPIDI_MsgInfo *msginfo = (const MPIDI_MsgInfo *)_msginfo; -#if TOKEN_FLOW_CONTROL - int rettoks=0; - void *uebuf; -#endif - pami_task_t source; - if (recv == NULL) - { - if (msginfo->isSync) - MPIDI_RecvShortSyncCB(context, - cookie, - _msginfo, - msginfo_size, - sndbuf, - sndlen, - sender, - recv); - else - MPIDI_RecvShortAsyncCB(context, - cookie, - _msginfo, - msginfo_size, - sndbuf, - sndlen, - sender, - recv); - return; - } - - MPID_assert(sndbuf == NULL); - MPID_assert(_msginfo != NULL); - MPID_assert(msginfo_size == sizeof(MPIDI_MsgInfo)); - - MPIR_Request * rreq = NULL; - source=PAMIX_Endpoint_query(sender); - - /* -------------------- */ - /* Match the request. */ - /* -------------------- */ - unsigned rank = msginfo->MPIrank; - unsigned tag = msginfo->MPItag; - unsigned context_id = msginfo->MPIctxt; - - MPIU_THREAD_CS_ENTER(MSGQUEUE,0); - if (TOKEN_FLOW_CONTROL_ON) - { - #if TOKEN_FLOW_CONTROL - MPIDI_Receive_tokens(msginfo,source); - #else - MPID_assert_always(0); - #endif - } -#ifndef OUT_OF_ORDER_HANDLING - rreq = MPIDI_Recvq_FDP(rank, tag, context_id); -#else - rreq = MPIDI_Recvq_FDP(rank, source, tag, context_id, msginfo->MPIseqno); -#endif - - /* Match not found */ - if (unlikely(rreq == NULL)) - { - /* No request was found and hence no sync needed */ -#if (MPIDI_STATISTICS) - MPID_NSTAT(mpid_statp->earlyArrivals); -#endif - MPIU_THREAD_CS_EXIT(MSGQUEUE,0); - MPIR_Request *newreq = MPIDI_Request_create2(); - MPID_assert(newreq != NULL); - if (TOKEN_FLOW_CONTROL_ON) - { - #if TOKEN_FLOW_CONTROL - MPIU_THREAD_CS_ENTER(MSGQUEUE,0); - #else - MPID_assert_always(0); - #endif - } - - if (sndlen) - { - newreq->mpid.uebuflen = sndlen; - if (!(TOKEN_FLOW_CONTROL_ON)) - { - newreq->mpid.uebuf = MPL_malloc(sndlen); - newreq->mpid.uebuf_malloc = mpiuMalloc ; - } - else - { - #if TOKEN_FLOW_CONTROL - newreq->mpid.uebuf = MPIDI_mm_alloc(sndlen); - newreq->mpid.uebuf_malloc = mpidiBufMM; - #else - MPID_assert_always(0); - #endif - } - MPID_assert(newreq->mpid.uebuf != NULL); - } - if (!TOKEN_FLOW_CONTROL_ON) - { - MPIU_THREAD_CS_ENTER(MSGQUEUE,0); - } -#ifndef OUT_OF_ORDER_HANDLING - rreq = MPIDI_Recvq_FDP(rank, tag, context_id); -#else - rreq = MPIDI_Recvq_FDP(rank, source, tag, context_id, msginfo->MPIseqno); -#endif - - if (unlikely(rreq == NULL)) - { - MPIDI_Callback_process_unexp(newreq, context, msginfo, sndlen, sender, sndbuf, recv, msginfo->isSync); - int completed = MPIR_Request_is_complete(newreq); - if (TOKEN_FLOW_CONTROL_ON) - { - #if TOKEN_FLOW_CONTROL - MPIDI_Token_cntr[sender].unmatched++; - #else - MPID_assert_always(0); - #endif - } - MPIU_THREAD_CS_EXIT(MSGQUEUE,0); - if (completed) MPIR_Request_free(newreq); - goto fn_exit_eager; - } - else - { - MPIU_THREAD_CS_EXIT(MSGQUEUE,0); - MPID_Request_discard(newreq); - } - } - else - { -#if (MPIDI_STATISTICS) - MPID_NSTAT(mpid_statp->earlyArrivalsMatched); -#endif - if (TOKEN_FLOW_CONTROL_ON) - { - #if TOKEN_FLOW_CONTROL - MPIDI_Update_rettoks(source); - MPIDI_Must_return_tokens(context,source); - #else - MPID_assert_always(0); - #endif - } - MPIU_THREAD_CS_EXIT(MSGQUEUE,0); - } - - /* -------------------------------------------- */ - /* Figure out target buffer for request data. */ - /* -------------------------------------------- */ - - /* ---------------------- */ - /* Copy in information. */ - /* ---------------------- */ - rreq->status.MPI_SOURCE = rank; - rreq->status.MPI_TAG = tag; - MPIR_STATUS_SET_COUNT(rreq->status, sndlen); - MPIDI_Request_setCA (rreq, MPIDI_CA_COMPLETE); - MPIDI_Request_cpyPeerRequestH(rreq, msginfo); - MPIDI_Request_setSync (rreq, msginfo->isSync); - MPIDI_Request_setRzv (rreq, 0); - TRACE_SET_R_VAL(source,(rreq->mpid.PR_idx),len,sndlen); - /* --------------------------------------- */ - /* We have to fill in the callback info. */ - /* --------------------------------------- */ - recv->local_fn = MPIDI_RecvDoneCB_mutexed; - recv->cookie = rreq; -#if ASSERT_LEVEL > 0 - /* This ensures that the value is set, even to something impossible */ - recv->addr = NULL; -#endif - - /* ----------------------------- */ - /* Request was already posted. */ - /* ----------------------------- */ - - if (unlikely(msginfo->isSync)) - MPIDI_SyncAck_post(context, rreq, source); - - /* ----------------------------------------- */ - /* Calculate message length for reception. */ - /* ----------------------------------------- */ - unsigned dt_contig, dt_size; - MPIDU_Datatype*dt_ptr; - MPI_Aint dt_true_lb; - MPIDI_Datatype_get_info(rreq->mpid.userbufcount, - rreq->mpid.datatype, - dt_contig, - dt_size, - dt_ptr, - dt_true_lb); - - /* ----------------------------- */ - /* Test for truncated message. */ - /* ----------------------------- */ - if (unlikely(sndlen > dt_size)) - { - MPIDI_Callback_process_trunc(context, rreq, recv, sndbuf); - goto fn_exit_eager; - } - - /* --------------------------------------- */ - /* If buffer is contiguous, we are done. */ - /* --------------------------------------- */ - if (likely(dt_contig)) - { - /* - * This is to test that the fields don't need to be - * initialized. Remove after this doesn't fail for a while. - */ - MPID_assert(rreq->mpid.uebuf == NULL); - MPID_assert(rreq->mpid.uebuflen == 0); - /* rreq->mpid.uebuf = NULL; */ - /* rreq->mpid.uebuflen = 0; */ - void* rcvbuf = rreq->mpid.userbuf + dt_true_lb; - - recv->addr = rcvbuf; - } - - /* ----------------------------------------------- */ - /* Buffer is non-contiguous. we need to allocate */ - /* a temporary buffer, and unpack later. */ - /* ----------------------------------------------- */ - else - { - /* ----------------------------------------------- */ - /* Buffer is non-contiguous. the data is already */ - /* available, so we can just unpack it now. */ - /* ----------------------------------------------- */ - MPIDI_Request_setCA(rreq, MPIDI_CA_UNPACK_UEBUF_AND_COMPLETE); - rreq->mpid.uebuflen = sndlen; - if (sndlen) - { - if (!TOKEN_FLOW_CONTROL_ON) - { - rreq->mpid.uebuf = MPL_malloc(sndlen); - rreq->mpid.uebuf_malloc = mpiuMalloc; - } - else - { - #if TOKEN_FLOW_CONTROL - MPIDI_Alloc_lock(&rreq->mpid.uebuf,sndlen); - rreq->mpid.uebuf_malloc = mpidiBufMM; - #else - MPID_assert_always(0); - #endif - } - MPID_assert(rreq->mpid.uebuf != NULL); - } - /* -------------------------------------------------- */ - /* Let PAMI know where to put the rest of the data. */ - /* -------------------------------------------------- */ - recv->addr = rreq->mpid.uebuf; - } - TRACE_SET_R_VAL(source,(rreq->mpid.idx),fl.f.comp_in_HH,2); - TRACE_SET_R_VAL(source,(rreq->mpid.idx),bufadd,rreq->mpid.userbuf); - - fn_exit_eager: -#ifdef OUT_OF_ORDER_HANDLING - if (MPIDI_In_cntr[source].n_OutOfOrderMsgs > 0) { - MPIDI_Recvq_process_out_of_order_msgs(source, context); - } -#endif - - /* ---------------------------------------- */ - /* Signal that the recv has been started. */ - /* ---------------------------------------- */ - MPIDI_Progress_signal(); -} diff --git a/src/mpid/pamid/src/pt2pt/mpidi_callback_rzv.c b/src/mpid/pamid/src/pt2pt/mpidi_callback_rzv.c deleted file mode 100644 index 40ae04c..0000000 --- a/src/mpid/pamid/src/pt2pt/mpidi_callback_rzv.c +++ /dev/null @@ -1,234 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/pt2pt/mpidi_callback_rzv.c - * \brief The callback for a new RZV RTS - */ -#include <mpidimpl.h> - - -/** - * \brief The callback for a new RZV RTS - * \note Because this is a short message, the data is already received - * \param[in] context The context on which the message is being received. - * \param[in] sender The origin endpoint - * \param[in] _msginfo The extended header information - * \param[in] msginfo_size The size of the extended header information - * \param[in] is_zero_byte The rendezvous message is zero bytes in length. - */ -void -MPIDI_RecvRzvCB_impl(pami_context_t context, - pami_endpoint_t sender, - const void * _msginfo, - size_t msginfo_size, - const unsigned is_zero_byte) -{ - MPID_assert(_msginfo != NULL); - MPID_assert(msginfo_size == sizeof(MPIDI_MsgEnvelope)); - const MPIDI_MsgEnvelope * envelope = (const MPIDI_MsgEnvelope *)_msginfo; - const MPIDI_MsgInfo * msginfo = (const MPIDI_MsgInfo *)&envelope->msginfo; - - MPIR_Request * rreq = NULL; - int found; - pami_task_t source; -#if TOKEN_FLOW_CONTROL - int rettoks=0; -#endif - - /* -------------------- */ - /* Match the request. */ - /* -------------------- */ - unsigned rank = msginfo->MPIrank; - unsigned tag = msginfo->MPItag; - unsigned context_id = msginfo->MPIctxt; - - MPIR_Request *newreq = MPIDI_Request_create2(); - MPIU_THREAD_CS_ENTER(MSGQUEUE,0); - source = PAMIX_Endpoint_query(sender); - MPIDI_Receive_tokens(msginfo,source); -#ifndef OUT_OF_ORDER_HANDLING - rreq = MPIDI_Recvq_FDP_or_AEU(newreq, rank, tag, context_id, &found); -#else - rreq = MPIDI_Recvq_FDP_or_AEU(newreq, rank, source, tag, context_id, msginfo->MPIseqno, &found); -#endif - TRACE_ERR("RZV CB for req=%p remote-mr=0x%llx bytes=%zu (%sfound)\n", - rreq, - *(unsigned long long*)&envelope->envelope.memregion, - envelope->envelope.length, - found?"":"not "); - - /* ---------------------- */ - /* Copy in information. */ - /* ---------------------- */ - rreq->status.MPI_SOURCE = rank; - rreq->status.MPI_TAG = tag; - MPIR_STATUS_SET_COUNT(rreq->status, envelope->length); - MPIDI_Request_setPeerRank_comm(rreq, rank); - MPIDI_Request_setPeerRank_pami(rreq, source); - MPIDI_Request_cpyPeerRequestH (rreq, msginfo); - MPIDI_Request_setSync (rreq, msginfo->isSync); - MPIDI_Request_setRzv (rreq, 1); - - /* ----------------------------------------------------- */ - /* Save the rendezvous information for when the target */ - /* node calls a receive function and the data is */ - /* retreived from the origin node. */ - /* ----------------------------------------------------- */ - if (is_zero_byte) - { - rreq->mpid.envelope.length = 0; - rreq->mpid.envelope.data = NULL; - } - else - { -#ifdef USE_PAMI_RDMA - memcpy(&rreq->mpid.envelope.memregion, - &envelope->memregion, - sizeof(pami_memregion_t)); -#else - rreq->mpid.envelope.memregion_used = envelope->memregion_used; - if(envelope->memregion_used) - { - memcpy(&rreq->mpid.envelope.memregion, - &envelope->memregion, - sizeof(pami_memregion_t)); - } - rreq->mpid.envelope.data = envelope->data; -#endif - rreq->mpid.envelope.length = envelope->length; - TRACE_SET_R_VAL(source,(rreq->mpid.idx),req,rreq); - TRACE_SET_R_VAL(source,(rreq->mpid.idx),rlen,envelope->length); - TRACE_SET_R_VAL(source,(rreq->mpid.idx),fl.f.sync,msginfo->isSync); - TRACE_SET_R_BIT(source,(rreq->mpid.idx),fl.f.rzv); - if (TOKEN_FLOW_CONTROL_ON) - { - #if TOKEN_FLOW_CONTROL - MPIDI_Must_return_tokens(context,source); - #else - MPID_assert_always(0); - #endif - } - } - /* ----------------------------------------- */ - /* figure out target buffer for request data */ - /* ----------------------------------------- */ - if (found) - { -#if (MPIDI_STATISTICS) - MPID_NSTAT(mpid_statp->earlyArrivalsMatched); -#endif - /* --------------------------- */ - /* if synchronized, post ack. */ - /* --------------------------- */ - if (unlikely(MPIDI_Request_isSync(rreq))) - MPIDI_SyncAck_post(context, rreq, MPIDI_Request_getPeerRank_pami(rreq)); - - MPIU_THREAD_CS_EXIT(MSGQUEUE,0); - - if (is_zero_byte) - MPIDI_RecvRzvDoneCB_zerobyte(context, rreq, PAMI_SUCCESS); - else - { - MPIDI_RendezvousTransfer(context, rreq); - TRACE_SET_R_BIT(source,(rreq->mpid.idx),fl.f.sync_com_in_HH); - TRACE_SET_R_BIT(source,(rreq->mpid.idx),fl.f.matchedInHH); - TRACE_SET_R_VAL(source,(rreq->mpid.idx),bufadd,rreq->mpid.userbuf); - } - MPID_Request_discard(newreq); - } - - /* ------------------------------------------------------------- */ - /* Request was not posted. */ - /* ------------------------------------------------------------- */ - else - { -#if (MPIDI_STATISTICS) - MPID_NSTAT(mpid_statp->earlyArrivals); -#endif - /* - * This is to test that the fields don't need to be - * initialized. Remove after this doesn't fail for a while. - */ - MPID_assert(rreq->mpid.uebuf == NULL); - MPID_assert(rreq->mpid.uebuflen == 0); - /* rreq->mpid.uebuf = NULL; */ - /* rreq->mpid.uebuflen = 0; */ -#ifdef OUT_OF_ORDER_HANDLING - if (MPIDI_In_cntr[source].n_OutOfOrderMsgs > 0) { - MPIDI_Recvq_process_out_of_order_msgs(source, context); - } -#endif - MPIU_THREAD_CS_EXIT(MSGQUEUE,0); - } - /* ---------------------------------------- */ - /* Signal that the recv has been started. */ - /* ---------------------------------------- */ - MPIDI_Progress_signal(); -} - -/** - * \brief The callback for a new RZV RTS - * \param[in] context The context on which the message is being received. - * \param[in] cookie Unused - * \param[in] _msginfo The extended header information - * \param[in] msginfo_size The size of the extended header information - * \param[in] sndbuf Unused - * \param[in] sndlen Unused - * \param[in] sender The origin endpoint - * \param[out] recv Unused - */ -void -MPIDI_RecvRzvCB(pami_context_t context, - void * cookie, - const void * _msginfo, - size_t msginfo_size, - const void * sndbuf, - size_t sndlen, - pami_endpoint_t sender, - pami_recv_t * recv) -{ - MPID_assert(recv == NULL); - MPID_assert(sndlen == 0); - MPIDI_RecvRzvCB_impl (context, sender, _msginfo, msginfo_size, 0); -} - -/** - * \brief The callback for a new "zero byte" RZV RTS - * \param[in] context The context on which the message is being received. - * \param[in] cookie Unused - * \param[in] _msginfo The extended header information - * \param[in] msginfo_size The size of the extended header information - * \param[in] sndbuf Unused - * \param[in] sndlen Unused - * \param[in] sender The origin endpoint - * \param[out] recv Unused - */ -void -MPIDI_RecvRzvCB_zerobyte(pami_context_t context, - void * cookie, - const void * _msginfo, - size_t msginfo_size, - const void * sndbuf, - size_t sndlen, - pami_endpoint_t sender, - pami_recv_t * recv) -{ - MPID_assert(recv == NULL); - MPID_assert(sndlen == 0); - MPIDI_RecvRzvCB_impl (context, sender, _msginfo, msginfo_size, 1); -} diff --git a/src/mpid/pamid/src/pt2pt/mpidi_callback_short.c b/src/mpid/pamid/src/pt2pt/mpidi_callback_short.c deleted file mode 100644 index 8ecbef2..0000000 --- a/src/mpid/pamid/src/pt2pt/mpidi_callback_short.c +++ /dev/null @@ -1,268 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/pt2pt/mpidi_callback_short.c - * \brief The callback for a new short message - */ -#include <mpidimpl.h> - - -/** - * \brief The standard callback for a new message - * - * \param[in] context The context on which the message is being received. - * \param[in] cookie Unused - * \param[in] _msginfo The header information - * \param[in] msginfo_size The size of the header information - * \param[in] sndbuf If the message is short, this is the data - * \param[in] sndlen The size of the incoming data - * \param[out] recv If the message is long, this tells the message layer how to handle the data. - */ -static inline void -MPIDI_RecvShortCB(pami_context_t context, - const void * _msginfo, - const void * sndbuf, - size_t sndlen, - pami_endpoint_t sender, - unsigned isSync) - __attribute__((__always_inline__)); -static inline void -MPIDI_RecvShortCB(pami_context_t context, - const void * _msginfo, - const void * sndbuf, - size_t sndlen, - pami_endpoint_t sender, - unsigned isSync) -{ - MPID_assert(_msginfo != NULL); - - const MPIDI_MsgInfo *msginfo = (const MPIDI_MsgInfo *)_msginfo; - MPIR_Request * rreq = NULL; - pami_task_t source; -#if TOKEN_FLOW_CONTROL - int rettoks=0; -#endif - - /* -------------------- */ - /* Match the request. */ - /* -------------------- */ - unsigned rank = msginfo->MPIrank; - unsigned tag = msginfo->MPItag; - unsigned context_id = msginfo->MPIctxt; - - MPIU_THREAD_CS_ENTER(MSGQUEUE,0); - source = PAMIX_Endpoint_query(sender); - MPIDI_Receive_tokens(msginfo,source); -#ifndef OUT_OF_ORDER_HANDLING - rreq = MPIDI_Recvq_FDP(rank, tag, context_id); -#else - rreq = MPIDI_Recvq_FDP(rank, source, tag, context_id, msginfo->MPIseqno); -#endif - - /* Match not found */ - if (unlikely(rreq == NULL)) - { -#if (MPIDI_STATISTICS) - MPID_NSTAT(mpid_statp->earlyArrivals); -#endif - MPIU_THREAD_CS_EXIT(MSGQUEUE,0); - MPIR_Request *newreq = MPIDI_Request_create2(); - MPID_assert(newreq != NULL); - if (sndlen) - { - newreq->mpid.uebuflen = sndlen; - if (!TOKEN_FLOW_CONTROL_ON) - { - newreq->mpid.uebuf = MPL_malloc(sndlen); - newreq->mpid.uebuf_malloc = mpiuMalloc; - } - else - { - #if TOKEN_FLOW_CONTROL - MPIU_THREAD_CS_ENTER(MSGQUEUE,0); - newreq->mpid.uebuf = MPIDI_mm_alloc(sndlen); - newreq->mpid.uebuf_malloc = mpidiBufMM; - MPIU_THREAD_CS_EXIT(MSGQUEUE,0); - #else - MPID_assert_always(0); - #endif - } - MPID_assert(newreq->mpid.uebuf != NULL); - } - MPIU_THREAD_CS_ENTER(MSGQUEUE,0); -#ifndef OUT_OF_ORDER_HANDLING - rreq = MPIDI_Recvq_FDP(rank, tag, context_id); -#else - rreq = MPIDI_Recvq_FDP(rank, PAMIX_Endpoint_query(sender), tag, context_id, msginfo->MPIseqno); -#endif - - if (unlikely(rreq == NULL)) - { - MPIDI_Callback_process_unexp(newreq, context, msginfo, sndlen, sender, sndbuf, NULL, isSync); - /* request is always complete now */ - if (TOKEN_FLOW_CONTROL_ON && sndlen) - { - #if TOKEN_FLOW_CONTROL - MPIDI_Token_cntr[source].unmatched++; - #else - MPID_assert_always(0); - #endif - } - MPIU_THREAD_CS_EXIT(MSGQUEUE,0); - MPIR_Request_free(newreq); - goto fn_exit_short; - } - else - { - MPIU_THREAD_CS_EXIT(MSGQUEUE,0); - MPID_Request_discard(newreq); - } - } - else - { -#if (MPIDI_STATISTICS) - MPID_NSTAT(mpid_statp->earlyArrivalsMatched); -#endif - if (TOKEN_FLOW_CONTROL_ON && sndlen) - { - #if TOKEN_FLOW_CONTROL - MPIDI_Update_rettoks(source); - MPIDI_Must_return_tokens(context,source); - #else - MPID_assert_always(0); - #endif - } - MPIU_THREAD_CS_EXIT(MSGQUEUE,0); - } - - /* the receive queue processing has been completed and we found match*/ - - /* ---------------------- */ - /* Copy in information. */ - /* ---------------------- */ - rreq->status.MPI_SOURCE = rank; - rreq->status.MPI_TAG = tag; - MPIR_STATUS_SET_COUNT(rreq->status, sndlen); - MPIDI_Request_setCA (rreq, MPIDI_CA_COMPLETE); - MPIDI_Request_cpyPeerRequestH(rreq, msginfo); - MPIDI_Request_setSync (rreq, isSync); - MPIDI_Request_setRzv (rreq, 0); - - /* ----------------------------- */ - /* Request was already posted. */ - /* ----------------------------- */ - if (unlikely(isSync)) - MPIDI_SyncAck_post(context, rreq, PAMIX_Endpoint_query(sender)); - - if (unlikely(HANDLE_GET_KIND(rreq->mpid.datatype) != HANDLE_KIND_BUILTIN)) - { - MPIDI_Callback_process_userdefined_dt(context, sndbuf, sndlen, rreq); - goto fn_exit_short; - } - - size_t dt_size = rreq->mpid.userbufcount * MPIDU_Datatype_get_basic_size(rreq->mpid.datatype); - - /* ----------------------------- */ - /* Test for truncated message. */ - /* ----------------------------- */ - if (unlikely(sndlen > dt_size)) - { -#if ASSERT_LEVEL > 0 - MPIDI_Callback_process_trunc(context, rreq, NULL, sndbuf); - goto fn_exit_short; -#else - sndlen = dt_size; -#endif - } - - MPID_assert(rreq->mpid.uebuf == NULL); - MPID_assert(rreq->mpid.uebuflen == 0); - void* rcvbuf = rreq->mpid.userbuf; - - if (sndlen > 0) - { -#if CUDA_AWARE_SUPPORT - if(MPIDI_Process.cuda_aware_support_on && MPIDI_cuda_is_device_buf(rcvbuf)) - { - cudaError_t cudaerr = CudaMemcpy(rcvbuf, sndbuf, (size_t)sndlen, cudaMemcpyHostToDevice); - } - else -#endif - memcpy(rcvbuf, sndbuf, sndlen); - } - TRACE_SET_R_VAL(source,(rreq->mpid.idx),rlen,sndlen); - TRACE_SET_R_BIT(source,(rreq->mpid.idx),fl.f.comp_in_HH); - TRACE_SET_R_VAL(source,(rreq->mpid.idx),bufadd,rreq->mpid.userbuf); - MPIDI_Request_complete(rreq); - - fn_exit_short: -#ifdef OUT_OF_ORDER_HANDLING - MPIU_THREAD_CS_ENTER(MSGQUEUE,0); - if (MPIDI_In_cntr[source].n_OutOfOrderMsgs>0) { - MPIDI_Recvq_process_out_of_order_msgs(source, context); - } - MPIU_THREAD_CS_EXIT(MSGQUEUE,0); -#endif - - /* ---------------------------------------- */ - /* Signal that the recv has been started. */ - /* ---------------------------------------- */ - MPIDI_Progress_signal(); -} - - -void -MPIDI_RecvShortAsyncCB(pami_context_t context, - void * cookie, - const void * _msginfo, - size_t msginfo_size, - const void * sndbuf, - size_t sndlen, - pami_endpoint_t sender, - pami_recv_t * recv) -{ - MPID_assert(recv == NULL); - MPID_assert(msginfo_size == sizeof(MPIDI_MsgInfo)); - MPIDI_RecvShortCB(context, - _msginfo, - sndbuf, - sndlen, - sender, - 0); -} - - -void -MPIDI_RecvShortSyncCB(pami_context_t context, - void * cookie, - const void * _msginfo, - size_t msginfo_size, - const void * sndbuf, - size_t sndlen, - pami_endpoint_t sender, - pami_recv_t * recv) -{ - MPID_assert(recv == NULL); - MPID_assert(msginfo_size == sizeof(MPIDI_MsgInfo)); - MPIDI_RecvShortCB(context, - _msginfo, - sndbuf, - sndlen, - sender, - 1); -} diff --git a/src/mpid/pamid/src/pt2pt/mpidi_callback_util.c b/src/mpid/pamid/src/pt2pt/mpidi_callback_util.c deleted file mode 100644 index ca34ec5..0000000 --- a/src/mpid/pamid/src/pt2pt/mpidi_callback_util.c +++ /dev/null @@ -1,195 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/pt2pt/mpidi_callback_util.c - * \brief Utility functions to help uncommon cases in the new-message callbacks - */ -#include <mpidimpl.h> - - -/* MSGQUEUE lock must be held by caller */ -void -MPIDI_Callback_process_unexp(MPIR_Request *newreq, - pami_context_t context, - const MPIDI_MsgInfo * msginfo, - size_t sndlen, - pami_endpoint_t sender, - const void * sndbuf, - pami_recv_t * recv, - unsigned isSync) -{ - MPIR_Request *rreq = NULL; - - /* ---------------------------------------------------- */ - /* Fallback position: */ - /* + Request was not posted, or */ - /* + Request was long & not contiguous. */ - /* We must allocate enough space to hold the message. */ - /* The temporary buffer will be unpacked later. */ - /* ---------------------------------------------------- */ - unsigned rank = msginfo->MPIrank; - unsigned tag = msginfo->MPItag; - unsigned context_id = msginfo->MPIctxt; -#ifndef OUT_OF_ORDER_HANDLING - rreq = MPIDI_Recvq_AEU(newreq, rank, tag, context_id); -#else - unsigned msg_seqno = msginfo->MPIseqno; - rreq = MPIDI_Recvq_AEU(newreq, rank, PAMIX_Endpoint_query(sender), tag, context_id, msg_seqno); -#endif - /* ---------------------- */ - /* Copy in information. */ - /* ---------------------- */ - rreq->status.MPI_SOURCE = rank; - rreq->status.MPI_TAG = tag; - MPIR_STATUS_SET_COUNT(rreq->status, sndlen); - MPIDI_Request_setCA (rreq, MPIDI_CA_COMPLETE); - MPIDI_Request_cpyPeerRequestH(rreq, msginfo); - MPIDI_Request_setSync (rreq, isSync); - - /* Set the rank of the sender if a sync msg. */ -#ifndef OUT_OF_ORDER_HANDLING - if (isSync) - { -#endif - MPIDI_Request_setPeerRank_comm(rreq, rank); - MPIDI_Request_setPeerRank_pami(rreq, PAMIX_Endpoint_query(sender)); -#ifndef OUT_OF_ORDER_HANDLING - } -#endif - - MPID_assert(!sndlen || rreq->mpid.uebuf != NULL); - TRACE_MEMSET_R(PAMIX_Endpoint_query(sender),msg_seqno,recv_status); - TRACE_SET_R_VAL(PAMIX_Endpoint_query(sender),(msginfo->MPIseqno & SEQMASK),msgid,msginfo->MPIseqno); - TRACE_SET_R_VAL(PAMIX_Endpoint_query(sender),(msginfo->MPIseqno & SEQMASK),rtag,tag); - TRACE_SET_R_VAL(PAMIX_Endpoint_query(sender),(msginfo->MPIseqno & SEQMASK),rctx,msginfo->MPIctxt); - TRACE_SET_R_VAL(PAMIX_Endpoint_query(sender),(msginfo->MPIseqno & SEQMASK),rlen,sndlen); - TRACE_SET_R_VAL(PAMIX_Endpoint_query(sender),(msginfo->MPIseqno & SEQMASK),fl.f.sync,isSync); - TRACE_SET_R_VAL(PAMIX_Endpoint_query(sender),(msginfo->MPIseqno & SEQMASK),rsource,PAMIX_Endpoint_query(sender)); - TRACE_SET_REQ_VAL(rreq->mpid.idx,(msginfo->MPIseqno & SEQMASK)); - - if (recv != NULL) - { - recv->local_fn = MPIDI_RecvDoneCB_mutexed; - recv->cookie = rreq; - /* -------------------------------------------------- */ - /* Let PAMI know where to put the rest of the data. */ - /* -------------------------------------------------- */ - recv->addr = rreq->mpid.uebuf; - } - else - { - /* ------------------------------------------------- */ - /* We have the data; copy it and complete the msg. */ - /* ------------------------------------------------- */ - memcpy(rreq->mpid.uebuf, sndbuf, sndlen); - MPIDI_RecvDoneCB(context, rreq, PAMI_SUCCESS); - /* caller must release rreq, after unlocking MSGQUEUE */ - } -} - - -/* MSGQUEUE lock is not held */ -void -MPIDI_Callback_process_trunc(pami_context_t context, - MPIR_Request *rreq, - pami_recv_t *recv, - const void *sndbuf) -{ - rreq->status.MPI_ERROR = MPI_ERR_TRUNCATE; - - /* -------------------------------------------------------------- */ - /* The data is already available, so we can just unpack it now. */ - /* -------------------------------------------------------------- */ - if (recv) - { - MPIDI_Request_setCA(rreq, MPIDI_CA_UNPACK_UEBUF_AND_COMPLETE); - rreq->mpid.uebuflen = MPIR_STATUS_GET_COUNT(rreq->status); - rreq->mpid.uebuf = MPL_malloc(MPIR_STATUS_GET_COUNT(rreq->status)); - MPID_assert(rreq->mpid.uebuf != NULL); - rreq->mpid.uebuf_malloc = mpiuMalloc; - - recv->addr = rreq->mpid.uebuf; - } - else - { - MPIDI_Request_setCA(rreq, MPIDI_CA_UNPACK_UEBUF_AND_COMPLETE); - rreq->mpid.uebuflen = MPIR_STATUS_GET_COUNT(rreq->status); - rreq->mpid.uebuf = (void*)sndbuf; - MPIDI_RecvDoneCB(context, rreq, PAMI_SUCCESS); - MPIR_Request_free(rreq); - } -} - - -/* MSGQUEUE lock is not held */ -void -MPIDI_Callback_process_userdefined_dt(pami_context_t context, - const void * sndbuf, - size_t sndlen, - MPIR_Request * rreq) -{ - unsigned dt_contig, dt_size; - MPIDU_Datatype*dt_ptr; - MPI_Aint dt_true_lb; - MPIDI_Datatype_get_info(rreq->mpid.userbufcount, - rreq->mpid.datatype, - dt_contig, - dt_size, - dt_ptr, - dt_true_lb); - - /* ----------------------------- */ - /* Test for truncated message. */ - /* ----------------------------- */ - if (unlikely(sndlen > dt_size)) - { -#if ASSERT_LEVEL > 0 - MPIDI_Callback_process_trunc(context, rreq, NULL, sndbuf); - return; -#else - sndlen = dt_size; -#endif - } - - /* - * This is to test that the fields don't need to be - * initialized. Remove after this doesn't fail for a while. - */ - if (likely (dt_contig)) - { - MPID_assert(rreq->mpid.uebuf == NULL); - MPID_assert(rreq->mpid.uebuflen == 0); - void* rcvbuf = rreq->mpid.userbuf + dt_true_lb;; -#if CUDA_AWARE_SUPPORT - if(MPIDI_Process.cuda_aware_support_on && MPIDI_cuda_is_device_buf(rcvbuf)) - { - cudaError_t cudaerr = CudaMemcpy(rcvbuf, sndbuf, (size_t)sndlen, cudaMemcpyHostToDevice); - } - else -#endif - memcpy(rcvbuf, sndbuf, sndlen); - MPIDI_Request_complete(rreq); - return; - } - - MPIDI_Request_setCA(rreq, MPIDI_CA_UNPACK_UEBUF_AND_COMPLETE); - rreq->mpid.uebuflen = sndlen; - rreq->mpid.uebuf = (void*)sndbuf; - MPIDI_RecvDoneCB(context, rreq, PAMI_SUCCESS); - MPIR_Request_free(rreq); -} diff --git a/src/mpid/pamid/src/pt2pt/mpidi_control.c b/src/mpid/pamid/src/pt2pt/mpidi_control.c deleted file mode 100644 index d2e2b87..0000000 --- a/src/mpid/pamid/src/pt2pt/mpidi_control.c +++ /dev/null @@ -1,414 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/pt2pt/mpidi_control.c - * \brief Interface to the control protocols used by MPID pt2pt - */ -#include <mpidimpl.h> - - -/** - * \brief Send a high-priority msginfo struct (control data) - * - * \param[in] control The pointer to the msginfo structure - * \param[in] peerrank The node to whom the control message is to be sent - */ -static inline void -MPIDI_CtrlSend(pami_context_t context, - MPIDI_MsgInfo * msginfo, - pami_task_t peerrank) -{ - pami_endpoint_t dest; - PAMI_Endpoint_create(MPIDI_Client, peerrank, 0, &dest); - - TRACE_ERR("CtrlSend: type=%d local=%u remote=%u\n", msginfo->control, MPIR_Process.comm_world->rank, peerrank); - pami_send_immediate_t params = { - .dispatch = MPIDI_Protocols_Control, - .dest = dest, - .header = { - .iov_base = msginfo, - .iov_len = sizeof(MPIDI_MsgInfo), - }, - .data = { - .iov_base = NULL, - .iov_len = 0, - }, - }; - - pami_result_t rc; - rc = PAMI_Send_immediate(context, &params); - MPID_assert(rc == PAMI_SUCCESS); -} - - -/** - * \brief Message layer callback which is invoked on the target node - * of a flow-control rendezvous operation. - * - * This callback is invoked when the data buffer on the origin node - * has been completely transfered to the target node. The target node - * must acknowledge the completion of the transfer to the origin node - * with a control message and then complete the receive by releasing - * the request object. - * - * \param[in,out] rreq MPI receive request object - */ -void -MPIDI_RecvRzvDoneCB(pami_context_t context, - void * cookie, - pami_result_t result) -{ - MPIR_Request * rreq = (MPIR_Request*)cookie; - MPID_assert(rreq != NULL); - - TRACE_ERR("RZV Done for req=%p addr=%p *addr[0]=%#016llx *addr[1]=%#016llx\n", - rreq, - rreq->mpid.userbuf, - *(((unsigned long long*)rreq->mpid.userbuf)+0), - *(((unsigned long long*)rreq->mpid.userbuf)+1)); - - /* Is it neccesary to save the original value of the 'type' field ?? */ - unsigned original_value = MPIDI_Request_getControl(rreq); - MPIDI_Request_setControl(rreq, MPIDI_CONTROL_RENDEZVOUS_ACKNOWLEDGE); - MPIDI_CtrlSend(context, - &rreq->mpid.envelope.msginfo, - MPIDI_Request_getPeerRank_pami(rreq)); - MPIDI_Request_setControl(rreq, original_value); - -#ifdef USE_PAMI_RDMA - pami_result_t rc; - rc = PAMI_Memregion_destroy(context, &rreq->mpid.memregion); - MPID_assert(rc == PAMI_SUCCESS); -#else - if( (!MPIDI_Process.mp_s_use_pami_get) && (rreq->mpid.memregion_used) ) - { - pami_result_t rc; - rc = PAMI_Memregion_destroy(context, &rreq->mpid.memregion); - MPID_assert(rc == PAMI_SUCCESS); - } -#endif - - MPIDI_RecvDoneCB(context, rreq, PAMI_SUCCESS); - MPIR_Request_free(rreq); -} - -/** - * \brief Message layer callback which is invoked on the target node - * of a 'zero-byte' flow-control rendezvous operation. - * - * \param[in] context Communication context - * \param[in] cookie Completion callback cookie - MPI receive request object - * \param[in] result Status - */ -void -MPIDI_RecvRzvDoneCB_zerobyte(pami_context_t context, - void * cookie, - pami_result_t result) -{ - MPIR_Request * rreq = (MPIR_Request*)cookie; - MPID_assert(rreq != NULL); - - /* Is it neccesary to save the original value of the 'type' field ?? */ - unsigned original_value = MPIDI_Request_getControl(rreq); - MPIDI_Request_setControl(rreq, MPIDI_CONTROL_RENDEZVOUS_ACKNOWLEDGE); - MPIDI_CtrlSend(context, - &rreq->mpid.envelope.msginfo, - MPIDI_Request_getPeerRank_pami(rreq)); - MPIDI_Request_setControl(rreq, original_value); - - MPIDI_RecvDoneCB(context, rreq, PAMI_SUCCESS); - TRACE_SET_R_BIT(MPIDI_Request_getPeerRank_pami(rreq),(rreq->mpid.idx),fl.f.sync_com_in_HH); - TRACE_SET_R_BIT(MPIDI_Request_getPeerRank_pami(rreq),(rreq->mpid.idx),fl.f.matchedInHH); - TRACE_SET_R_VAL(MPIDI_Request_getPeerRank_pami(rreq),(rreq->mpid.idx),bufadd,rreq->mpid.userbuf); - MPIR_Request_free(rreq); -} - -/** - * \brief Acknowledge an MPI_Ssend() - * - * \param[in] req The request element to acknowledge - * - * \return The same as MPIDI_CtrlSend() - */ -void -MPIDI_SyncAck_post(pami_context_t context, - MPIR_Request * req, - unsigned peer) -{ - MPIDI_Request_setControl(req, MPIDI_CONTROL_SSEND_ACKNOWLEDGE); - MPIDI_MsgInfo * info = &req->mpid.envelope.msginfo; - MPIDI_CtrlSend(context, info, peer); -} - - -/** - * \brief Acknowledge an MPI_Ssend() - * - * This is the handoff side, executing in advance(). - * The send is performed, and then the request is - * completed. The send is "immediate" so the payload - * has been copied upon return from MPIDI_CtrlSend(), - * so it is safe to complete the request. - * - * \param[in] context The PAMI context - * \param[in] req The request element to acknowledge - * - * \returns The PAMI return code - */ -pami_result_t -MPIDI_SyncAck_handoff(pami_context_t context, - void * inputReq) -{ - MPIR_Request *req = inputReq; - MPIDI_Request_setControl(req, MPIDI_CONTROL_SSEND_ACKNOWLEDGE); - MPIDI_MsgInfo * info = &req->mpid.envelope.msginfo; - unsigned peer = MPIDI_Request_getPeerRank_pami(req); - MPIDI_CtrlSend(context, info, peer); - MPIDI_Request_complete(req); - return PAMI_SUCCESS; -} - - -/** - * \brief Process an incoming MPI_Ssend() acknowledgment - * - * \param[in] info The contents of the control message as a MPIDI_MsgInfo struct - * \param[in] peer The rank of the node sending the data - */ -static inline void -MPIDI_SyncAck_proc(pami_context_t context, - const MPIDI_MsgInfo * info, - unsigned peer) -{ - MPID_assert(info != NULL); - MPIR_Request *req = MPIDI_Msginfo_getPeerRequest(info); - MPID_assert(req != NULL); - MPIDI_Request_complete(req); -} - - -/** - * \brief - * - * \param[in] context - * \param[in] req - */ -static inline void -MPIDI_RzvAck_proc_req(pami_context_t context, - MPIR_Request * req) -{ -#ifdef USE_PAMI_RDMA - pami_result_t rc; - rc = PAMI_Memregion_destroy(context, &req->mpid.envelope.memregion); - MPID_assert(rc == PAMI_SUCCESS); -#else - if( (!MPIDI_Process.mp_s_use_pami_get) && (req->mpid.envelope.memregion_used) ) - { - pami_result_t rc; - rc = PAMI_Memregion_destroy(context, &req->mpid.envelope.memregion); - MPID_assert(rc == PAMI_SUCCESS); - } -#endif - TRACE_SET_S_BIT(req->mpid.partner_id,(req->mpid.idx),fl.f.recvAck); - MPIDI_SendDoneCB(context, req, PAMI_SUCCESS); -} - - -/** - * \brief Process an incoming rendezvous acknowledgment from the - * target (remote) node and complete the MPI_Send() on the origin - * (local) node. - * - * \param[in] context - * \param[in] info The contents of the control message as a MPIDI_MsgInfo struct - * \param[in] peer The rank of the node sending the data - */ -static inline void -MPIDI_RzvAck_proc(pami_context_t context, - const MPIDI_MsgInfo * info, - pami_task_t peer) -{ - MPID_assert(info != NULL); - MPIR_Request *req = MPIDI_Msginfo_getPeerRequest(info); - MPID_assert(req != NULL); - MPIDI_RzvAck_proc_req(context, req); -} - - -/** - * \brief Process an incoming MPI_Send() cancelation - * - * \param[in] info The contents of the control message as a MPIDI_MsgInfo struct - * \param[in] peer The rank of the node sending the data - */ -static inline void -MPIDI_CancelReq_proc(pami_context_t context, - const MPIDI_MsgInfo * info, - pami_task_t peer) -{ - MPIDI_CONTROL type; - MPIDI_MsgInfo ackinfo; - MPIR_Request * sreq; - - MPID_assert(info != NULL); - - sreq=MPIDI_Recvq_FDUR(MPIDI_Msginfo_getPeerRequestH(info), - info->MPIrank, - info->MPItag, - info->MPIctxt); - if(sreq) - { - MPIR_Request_free(sreq); - type = MPIDI_CONTROL_CANCEL_ACKNOWLEDGE; - } - else - { - type = MPIDI_CONTROL_CANCEL_NOT_ACKNOWLEDGE; - } - - TRACE_ERR("Cancel search: {rank=%d:tag=%d:ctxt=%d:req=%d} my_request=%p result=%s\n", - info->MPIrank, - info->MPItag, - info->MPIctxt, - MPIDI_Msginfo_getPeerRequestH(info), - sreq, - (type==MPIDI_CONTROL_CANCEL_ACKNOWLEDGE) ? "ACK" : "NAK"); - - ackinfo.control = type; - MPIDI_Msginfo_cpyPeerRequestH(&ackinfo, info); - MPIDI_CtrlSend(context, &ackinfo, peer); -} - - -/** - * \brief Process an incoming MPI_Send() cancelation result - * - * \param[in] info The contents of the control message as a MPIDI_MsgInfo struct - * \param[in] peer The rank of the node sending the data - */ -static inline void -MPIDI_CancelAck_proc(pami_context_t context, - const MPIDI_MsgInfo * info, - pami_task_t peer) -{ - MPID_assert(info != NULL); - MPIR_Request *req = MPIDI_Msginfo_getPeerRequest(info); - MPID_assert(req != NULL); - - TRACE_ERR("Cancel result: my_request=%p result=%s\n", - req, - (info->control==MPIDI_CONTROL_CANCEL_ACKNOWLEDGE) ? "ACK" : "NAK"); - - if(info->control == MPIDI_CONTROL_CANCEL_NOT_ACKNOWLEDGE) - { - req->mpid.cancel_pending = FALSE; - } - else - { - MPID_assert(info->control == MPIDI_CONTROL_CANCEL_ACKNOWLEDGE); - MPID_assert(req->mpid.cancel_pending == TRUE); - - MPIR_STATUS_SET_CANCEL_BIT(req->status, TRUE); - - /* - * Rendezvous-Sends wait until a rzv ack is received to complete - * the send. Since this request was canceled, no rzv ack will be - * sent from the target node; fake the response here. - */ - if (MPIDI_Request_isRzv(req)) - { - TRACE_ERR("RZV\n"); - MPIDI_RzvAck_proc_req(context, req); - } - /* - * A canceled Sync-Send hasn't been ACKed (and now will never be - * acked). We call complete now to simulate an ACKed message. - * This is the entirety of the sync-ack processing, so it hasn't - * been made into a new function. - */ - if (MPIDI_Request_isSync(req)) - { - TRACE_ERR("Sync\n"); - MPIDI_Request_complete(req); - } - - /* - * Finally, this request has been faux-Sync-ACKed and - * faux-RZV-ACKed. We just do a normal completion. The user - * can call MPI_Wait()/MPI_Test() to finish it off (unless the - * message hasn't finished sending, in which case the done - * callback will finish it off). - */ - } - - TRACE_ERR("Completing request\n"); - MPIDI_Request_complete(req); -} - - -/** - * \brief This is the general PT2PT control message call-back - */ -void -MPIDI_ControlCB(pami_context_t context, - void * cookie, - const void * _msginfo, - size_t size, - const void * sndbuf, - size_t sndlen, - pami_endpoint_t sender, - pami_recv_t * recv) -{ - MPID_assert(recv == NULL); - MPID_assert(sndlen == 0); - MPID_assert(_msginfo != NULL); - MPID_assert(size == sizeof(MPIDI_MsgInfo)); - const MPIDI_MsgInfo *msginfo = (const MPIDI_MsgInfo *)_msginfo; - pami_task_t senderrank = PAMIX_Endpoint_query(sender); - - TRACE_ERR("CtrlRecv: type=%d local=%u remote=%u\n", msginfo->control, MPIR_Process.comm_world->rank, senderrank); - switch (msginfo->control) - { - case MPIDI_CONTROL_SSEND_ACKNOWLEDGE: - MPIDI_SyncAck_proc(context, msginfo, senderrank); - break; - case MPIDI_CONTROL_CANCEL_REQUEST: - MPIDI_CancelReq_proc(context, msginfo, senderrank); - break; - case MPIDI_CONTROL_CANCEL_ACKNOWLEDGE: - case MPIDI_CONTROL_CANCEL_NOT_ACKNOWLEDGE: - MPIDI_CancelAck_proc(context, msginfo, senderrank); - break; - case MPIDI_CONTROL_RENDEZVOUS_ACKNOWLEDGE: - MPIDI_RzvAck_proc(context, msginfo, senderrank); - break; -#if TOKEN_FLOW_CONTROL - case MPIDI_CONTROL_RETURN_TOKENS: - MPIU_THREAD_CS_ENTER(MSGQUEUE,0); - MPIDI_Token_cntr[sender].tokens += msginfo->alltokens; - MPIU_THREAD_CS_EXIT(MSGQUEUE,0); - break; -#endif - default: - fprintf(stderr, "Bad msginfo type: 0x%08x %d\n", - msginfo->control, - msginfo->control); - MPID_abort(); - } - MPIDI_Progress_signal(); -} diff --git a/src/mpid/pamid/src/pt2pt/mpidi_done.c b/src/mpid/pamid/src/pt2pt/mpidi_done.c deleted file mode 100644 index 8034b43..0000000 --- a/src/mpid/pamid/src/pt2pt/mpidi_done.c +++ /dev/null @@ -1,316 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/pt2pt/mpidi_done.c - * \brief "Done" call-backs provided to the message layer for signaling completion - */ -#include <mpidimpl.h> - - -/** - * \brief Message layer callback which is invoked on the origin node - * when the send of the message is done - * - * \param[in,out] sreq MPI receive request object - */ -void -MPIDI_SendDoneCB(pami_context_t context, - void * clientdata, - pami_result_t result) -{ - TRACE_SET_S_BIT((((MPIR_Request *) clientdata)->mpid.partner_id),(((MPIR_Request *) clientdata)->mpid.idx),fl.f.sendComp); - MPIDI_SendDoneCB_inline(context, - clientdata, - result); -} - - -static inline void -MPIDI_RecvDoneCB_copy(MPIR_Request * rreq) -{ - int smpi_errno; - MPID_assert(rreq->mpid.uebuf != NULL); - intptr_t _count=0; - MPIDI_Buffer_copy(rreq->mpid.uebuf, /* source buffer */ - rreq->mpid.uebuflen, - MPI_CHAR, - &smpi_errno, - rreq->mpid.userbuf, /* dest buffer */ - rreq->mpid.userbufcount, /* dest count */ - rreq->mpid.datatype, /* dest type */ - &_count, - &rreq->status.MPI_ERROR); - MPIR_STATUS_SET_COUNT(rreq->status, _count); -} - - -/** - * \brief Message layer callback which is invoked on the target node - * when the incoming message is complete. - * - * The MSGQUEUE lock may or may not be held. - * - * \param[in,out] rreq MPI receive request object - */ -void -MPIDI_RecvDoneCB(pami_context_t context, - void * clientdata, - pami_result_t result) -{ - MPIR_Request * rreq = (MPIR_Request*)clientdata; - MPID_assert(rreq != NULL); - switch (MPIDI_Request_getCA(rreq)) - { - case MPIDI_CA_UNPACK_UEBUF_AND_COMPLETE: - { - MPIDI_RecvDoneCB_copy(rreq); - /* free the unexpected data buffer later */ - break; - } - case MPIDI_CA_COMPLETE: - { - break; - } - default: - { - MPID_Abort(NULL, MPI_ERR_OTHER, -1, "Internal: unknown CA"); - break; - } - } -#ifdef OUT_OF_ORDER_HANDLING - MPIR_Request * oo_peer = rreq->mpid.oo_peer; - if (oo_peer) { - MPIR_STATUS_SET_COUNT(oo_peer->status, MPIR_STATUS_GET_COUNT(rreq->status)); - MPIDI_Request_complete(oo_peer); - } -#endif - MPIDI_Request_complete_norelease(rreq); - /* caller must release rreq, after unlocking MSGQUEUE (if held) */ -#ifdef OUT_OF_ORDER_HANDLING - pami_task_t source; - source = MPIDI_Request_getPeerRank_pami(rreq); - if (MPIDI_In_cntr[source].n_OutOfOrderMsgs > 0) { - MPIDI_Recvq_process_out_of_order_msgs(source, context); - } -#endif -} - - -/** - * \brief Thread-safe message layer callback which is invoked on the - * target node when the incoming message is complete. - * - * \param[in,out] rreq MPI receive request object - */ -void -MPIDI_RecvDoneCB_mutexed(pami_context_t context, - void * clientdata, - pami_result_t result) -{ - MPIR_Request * rreq = (MPIR_Request*)clientdata; - MPIU_THREAD_CS_ENTER(MSGQUEUE, 0); - - MPIDI_RecvDoneCB(context, clientdata, result); - - MPIU_THREAD_CS_EXIT(MSGQUEUE, 0); - MPIR_Request_free(rreq); -} - - -#ifdef OUT_OF_ORDER_HANDLING -/** - * \brief Checks if any of the messages in the out of order list is ready - * to be processed, if so, process it - */ -void MPIDI_Recvq_process_out_of_order_msgs(pami_task_t src, pami_context_t context) -{ - /*******************************************************/ - /* If the next message to be processed in the */ - /* a recv is posted, then process the message. */ - /*******************************************************/ - MPIDI_In_cntr_t *in_cntr; - MPIR_Request *ooreq, *rreq, *prev_rreq; - pami_get_simple_t xferP; - intptr_t _count=0; - int matched; - void * it; - - in_cntr = &MPIDI_In_cntr[src]; - prev_rreq = NULL; - ooreq = in_cntr->OutOfOrderList; - while((in_cntr->n_OutOfOrderMsgs !=0) && ((MPIDI_Request_getMatchSeq(ooreq) == (in_cntr->nMsgs+1)) || (MPIDI_Request_getMatchSeq(ooreq) == in_cntr->nMsgs))) - { - matched=0; - matched=MPIDI_Search_recv_posting_queue(MPIDI_Request_getMatchRank(ooreq),MPIDI_Request_getMatchTag(ooreq),MPIDI_Request_getMatchCtxt(ooreq),&rreq, &it); - - if (matched) { - /* process a completed message i.e. data is in EA */ - if (TOKEN_FLOW_CONTROL_ON) { - #if TOKEN_FLOW_CONTROL - if ((ooreq->mpid.uebuflen) && (!(ooreq->mpid.envelope.msginfo.isRzv))) { - MPIDI_Token_cntr[src].unmatched--; - MPIDI_Update_rettoks(src); - } - MPIDI_Must_return_tokens(context,src); - #else - MPID_assert_always(0); - #endif - } - if (MPIDI_Request_getMatchSeq(ooreq) == (in_cntr->nMsgs+ 1)) - in_cntr->nMsgs++; - - if (ooreq->mpid.nextR != NULL) { /* recv is in the out of order list */ - MPIDI_Recvq_remove_req_from_ool(ooreq,in_cntr); - } - - /* ----------------------------------------- */ - /* Calculate message length for reception. */ - /* ----------------------------------------- */ - unsigned dt_contig, dt_size; - MPIDU_Datatype*dt_ptr; - MPI_Aint dt_true_lb; - MPIDI_Datatype_get_info(rreq->mpid.userbufcount, - rreq->mpid.datatype, - dt_contig, - dt_size, - dt_ptr, - dt_true_lb); - if (unlikely(ooreq->mpid.uebuflen > dt_size)) - { - MPIR_STATUS_SET_COUNT(rreq->status, dt_size); - rreq->status.MPI_ERROR = MPI_ERR_TRUNCATE; - } - - TRACE_SET_REQ_VAL(rreq->mpid.idx,ooreq->mpid.idx); - TRACE_SET_R_BIT(src,(rreq->mpid.idx),fl.f.matchedInOOL); - TRACE_SET_R_VAL(src,(rreq->mpid.idx),rlen,dt_size); - ooreq->comm = rreq->comm; - MPIR_Comm_add_ref(ooreq->comm); - ooreq->mpid.userbuf = rreq->mpid.userbuf; - ooreq->mpid.userbufcount = rreq->mpid.userbufcount; - ooreq->mpid.datatype = rreq->mpid.datatype; - if (HANDLE_GET_KIND(ooreq->mpid.datatype) != HANDLE_KIND_BUILTIN) - { - MPIDU_Datatype_get_ptr(ooreq->mpid.datatype, ooreq->mpid.datatype_ptr); - MPIDU_Datatype_add_ref(ooreq->mpid.datatype_ptr); - } -#ifdef QUEUE_BINARY_SEARCH_SUPPORT - if(MPIDI_Process.queue_binary_search_support_on) - MPIDI_Recvq_remove_uexp_noit(MPIDI_Request_getMatchRank(ooreq),MPIDI_Request_getMatchTag(ooreq),MPIDI_Request_getMatchCtxt(ooreq), MPIDI_Request_getMatchSeq(ooreq)); - else -#endif - MPIDI_Recvq_remove(MPIDI_Recvq.unexpected, ooreq, ooreq->mpid.prev); - if (!MPIR_cc_is_complete(&ooreq->cc)) { - ooreq->mpid.oo_peer = rreq; - MPIDI_RecvMsg_Unexp(ooreq, rreq->mpid.userbuf, rreq->mpid.userbufcount, rreq->mpid.datatype); - } else { - MPIDI_RecvMsg_Unexp(ooreq, rreq->mpid.userbuf, rreq->mpid.userbufcount, rreq->mpid.datatype); - MPIR_STATUS_SET_COUNT(rreq->status, MPIR_STATUS_GET_COUNT(ooreq->status)); - rreq->status.MPI_SOURCE = ooreq->status.MPI_SOURCE; - rreq->status.MPI_TAG = ooreq->status.MPI_TAG; - rreq->mpid.envelope.msginfo.MPIseqno = ooreq->mpid.envelope.msginfo.MPIseqno; - MPIDI_Request_complete(rreq); - } - MPIR_Request_free(ooreq); - - } else { - if (MPIDI_Request_getMatchSeq(ooreq) == (in_cntr->nMsgs+ 1)) - in_cntr->nMsgs++; - if (ooreq->mpid.nextR != NULL) { /* recv is in the out of order list */ - MPIDI_Recvq_remove_req_from_ool(ooreq,in_cntr); - } - } - if (in_cntr->n_OutOfOrderMsgs > 0) - ooreq = in_cntr->OutOfOrderList; - } /* while */ -} - - -/** - * \brief search the posted recv queue and if found eliminate the - * element from the queue and return in *request; else return NULL - */ -int MPIDI_Search_recv_posting_queue(int src, int tag, int context_id, - MPIR_Request **request, void** it ) -{ - MPIR_Request * rreq; - MPIR_Request * prev_rreq = NULL; - - *request = NULL; -#ifdef QUEUE_BINARY_SEARCH_SUPPORT - if(MPIDI_Process.queue_binary_search_support_on) - { - MPIDI_Recvq_find_in_post(src, tag, context_id, &rreq, it); - if (rreq != NULL) - { - /* The communicator test is not yet correct */ - MPIDI_Recvq_remove_post(src, tag, context_id, *it); - *request = rreq; -#if (MPIDI_STATISTICS) - MPID_NSTAT(mpid_statp->earlyArrivalsMatched); -#endif - return 1; - } - else - { - MPIDI_Recvq_find_in_post(src, MPI_ANY_TAG, context_id, &rreq, it); - if (rreq == NULL) - { - MPIDI_Recvq_find_in_post(MPI_ANY_SOURCE, tag, context_id, &rreq, it); - if (rreq == NULL) - MPIDI_Recvq_find_in_post(MPI_ANY_SOURCE, MPI_ANY_TAG, context_id, &rreq, it); - } - if (rreq != NULL) - { - /* The communicator test is not yet correct */ - MPIDI_Recvq_remove_post(MPIDI_Request_getMatchRank(rreq), MPIDI_Request_getMatchTag(rreq), context_id, *it); - *request = rreq; -#if (MPIDI_STATISTICS) - MPID_NSTAT(mpid_statp->earlyArrivalsMatched); -#endif - return 1; - } - } - } - else - { -#endif - rreq = MPIDI_Recvq.posted_head; - while (rreq != NULL) - { - /* The communicator test is not yet correct */ - if ((MPIDI_Request_getMatchRank(rreq)==src || MPIDI_Request_getMatchRank(rreq)==MPI_ANY_SOURCE) && - (MPIDI_Request_getMatchCtxt(rreq)==context_id) && - (MPIDI_Request_getMatchTag(rreq) == tag || MPIDI_Request_getMatchTag(rreq) == MPI_ANY_TAG) - ) { - MPIDI_Recvq_remove(MPIDI_Recvq.posted, rreq, prev_rreq); - *request = rreq; -#if (MPIDI_STATISTICS) - MPID_NSTAT(mpid_statp->earlyArrivalsMatched); -#endif - return 1; - } - prev_rreq = rreq; - rreq = rreq->mpid.next; - } -#ifdef QUEUE_BINARY_SEARCH_SUPPORT - } -#endif - return 0; -} -#endif diff --git a/src/mpid/pamid/src/pt2pt/mpidi_recv.h b/src/mpid/pamid/src/pt2pt/mpidi_recv.h deleted file mode 100644 index f2c1594..0000000 --- a/src/mpid/pamid/src/pt2pt/mpidi_recv.h +++ /dev/null @@ -1,257 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/pt2pt/mpidi_recv.h - * \brief ADI level implemenation of MPI_Irecv() - */ - -#ifndef __src_pt2pt_mpidi_recv_h__ -#define __src_pt2pt_mpidi_recv_h__ - -#include <mpidimpl.h> -#include "../mpid_recvq.h" -#include "mpidu_datatype.h" -/*#ifdef MPIDI_STATISTICS - #include "../../include/mpidi_datatypes.h" -#endif*/ - - -#if TOKEN_FLOW_CONTROL -extern MPIDI_Out_cntr_t *MPIDI_Out_cntr; -extern int MPIDI_tfctrl_hwmark; -extern void *MPIDI_mm_alloc(size_t); -extern void MPIDI_mm_free(void *, size_t); -extern int tfctrl_enabled; -extern char *EagerLimit; -#define MPIDI_Return_tokens MPIDI_Return_tokens_inline -#define MPIDI_Receive_tokens MPIDI_Receive_tokens_inline -#define MPIDI_Update_rettoks MPIDI_Update_rettoks_inline -#define MPIDI_Alloc_lock MPIDI_Alloc_lock_inline -#define MPIDI_Must_return_tokens MPIDI_Must_return_tokens_inline - -static inline void * -MPIDI_Return_tokens_inline(pami_context_t context, int dest, int tokens) -{ - MPIDI_MsgInfo tokenInfo; - if (tokens) { - memset(&tokenInfo,0, sizeof(MPIDI_MsgInfo)); - tokenInfo.control=MPIDI_CONTROL_RETURN_TOKENS; - tokenInfo.alltokens=tokens; - pami_send_immediate_t params = { - .dispatch = MPIDI_Protocols_Control, - .dest = dest, - .header = { - .iov_base = &tokenInfo, - .iov_len = sizeof(MPIDI_MsgInfo), - }, - .data = { - .iov_base = NULL, - .iov_len = 0, - }, - }; - pami_result_t rc; - rc = PAMI_Send_immediate(context, &params); - MPID_assert(rc == PAMI_SUCCESS); - } -} - - -static inline void * -MPIDI_Must_return_tokens_inline(pami_context_t context,int dest) -{ - int rettoks=0; - - if (MPIDI_Token_cntr[dest].rettoks - && (MPIDI_Token_cntr[dest].rettoks + MPIDI_Token_cntr[dest].unmatched - >= MPIDI_tfctrl_hwmark)) - { - rettoks=MPIDI_Token_cntr[dest].rettoks; - MPIDI_Token_cntr[dest].rettoks=0; - MPIDI_Return_tokens_inline(context,dest,rettoks); - - } -} - -static inline void * -MPIDI_Receive_tokens_inline(const MPIDI_MsgInfo *m, int dest) - { - if ((m)->tokens) - { - MPIDI_Token_cntr[dest].tokens += (m)->tokens; - } - } - -static inline void * -MPIDI_Update_rettoks_inline(int source) - { - MPIDI_Token_cntr[source].rettoks++; - } - -static inline void * -MPIDI_Alloc_lock_inline(void **buf,size_t size) - { - MPIU_THREAD_CS_ENTER(MSGQUEUE,0); - (*buf) = (void *) MPIDI_mm_alloc(size); - MPIU_THREAD_CS_EXIT(MSGQUEUE,0); - } - -#else -#define MPIDI_Return_tokens(x,y,z) -#define MPIDI_Receive_tokens(x,y) -#define MPIDI_Update_rettoks(x) -#define MPIDI_Must_return_tokens(x,y) (0) -#define MPIDI_Alloc_lock(x,y) -#endif - -/** - * \brief ADI level implemenation of MPI_(I)Recv() - * - * \param[in] buf The buffer to receive into - * \param[in] count Number of expected elements in the buffer - * \param[in] datatype The datatype of each element - * \param[in] rank The sending rank - * \param[in] tag The message tag - * \param[in] comm Pointer to the communicator - * \param[in] context_offset Offset from the communicator context ID - * \param[out] status Update the status structure - * \param[out] request Return a pointer to the new request object - * - * \returns An MPI Error code - */ -static inline int -MPIDI_Recv(void * buf, - MPI_Aint count, - MPI_Datatype datatype, - int rank, - int tag, - MPIR_Comm * comm, - int context_offset, - unsigned is_blocking, - MPI_Status * status, - MPIR_Request ** request) -{ - MPIR_Request * rreq; - int found; - int mpi_errno = MPI_SUCCESS; - - /* ---------------------------------------- */ - /* NULL rank means empty request */ - /* ---------------------------------------- */ - if (unlikely(rank == MPI_PROC_NULL)) - { - MPIDI_RecvMsg_procnull(comm, is_blocking, status, request); - return MPI_SUCCESS; - } -#if (MPIDI_STATISTICS) - MPID_NSTAT(mpid_statp->recvs); -#endif - MPIR_Comm_add_ref(comm); - - /* ---------------------------------------- */ - /* find our request in the unexpected queue */ - /* or allocate one in the posted queue */ - /* ---------------------------------------- */ - MPIR_Request *newreq = MPIDI_Request_create2(); - MPIU_THREAD_CS_ENTER(MSGQUEUE,0); -#ifndef OUT_OF_ORDER_HANDLING - rreq = MPIDI_Recvq_FDU_or_AEP(newreq, rank, - tag, - comm->recvcontext_id + context_offset, - &found); -#else - int pami_source; - if(rank != MPI_ANY_SOURCE) { - pami_source = MPID_VCR_GET_LPID(comm->vcr, rank); - } else { - pami_source = MPI_ANY_SOURCE; - } - if ((pami_source != MPI_ANY_SOURCE) && (MPIDI_In_cntr[pami_source].n_OutOfOrderMsgs>0)) { - /* returns unlock */ - MPIDI_Recvq_process_out_of_order_msgs(pami_source, MPIDI_Context[0]); - } - rreq = MPIDI_Recvq_FDU_or_AEP(newreq, rank, - pami_source, - tag, - comm->recvcontext_id + context_offset, - &found); -#endif - MPIDI_SET_PR_REC(rreq,buf,count,datatype,pami_source,rank,tag,comm,is_blocking); - - /* ----------------------------------------------------------------- */ - /* populate request with our data */ - /* We can do this because this is not a multithreaded implementation */ - /* ----------------------------------------------------------------- */ - - rreq->comm = comm; - rreq->mpid.userbuf = buf; - rreq->mpid.userbufcount = count; - rreq->mpid.datatype = datatype; - /* We don't need this because MPIDI_CA_COMPLETE is the initialized default */ - /* MPIDI_Request_setCA(rreq, MPIDI_CA_COMPLETE); */ - - if (unlikely(found)) - { - TRACE_SET_R_VALX(pami_source,rreq,len,rreq->mpid.uebuflen); - MPIDI_RecvMsg_Unexp(rreq, buf, count, datatype); - mpi_errno = rreq->status.MPI_ERROR; - if (TOKEN_FLOW_CONTROL_ON) { - #if TOKEN_FLOW_CONTROL - if ((rreq->mpid.uebuflen) && (!(rreq->mpid.envelope.msginfo.isRzv))) { - MPIDI_Token_cntr[(rreq->mpid.peer_pami)].unmatched--; - MPIDI_Update_rettoks(rreq->mpid.peer_pami); - } - MPIDI_Must_return_tokens(MPIDI_Context[0],(rreq->mpid.peer_pami)); - #else - MPID_assert_always(0); - #endif - } - MPIU_THREAD_CS_EXIT(MSGQUEUE,0); - MPID_Request_discard(newreq); -#ifdef OUT_OF_ORDER_HANDLING - if ((MPIDI_In_cntr[rreq->mpid.peer_pami].n_OutOfOrderMsgs>0)) - MPIDI_Recvq_process_out_of_order_msgs(rreq->mpid.peer_pami, MPIDI_Context[0]); -#endif - } - else - { - /* ----------------------------------------------------------- */ - /* request not found in unexpected queue, allocated and posted */ - /* ----------------------------------------------------------- */ - if (HANDLE_GET_KIND(datatype) != HANDLE_KIND_BUILTIN) - { - MPIDU_Datatype_get_ptr(datatype, rreq->mpid.datatype_ptr); - MPIDU_Datatype_add_ref(rreq->mpid.datatype_ptr); - } - MPIU_THREAD_CS_EXIT(MSGQUEUE,0); - } - - /* mutex has been dropped... */ - *request = rreq; - if (status != MPI_STATUS_IGNORE) - *status = rreq->status; -#ifdef MPIDI_STATISTICS - if (!(MPIR_cc_is_complete(&rreq->cc))) - { - MPID_NSTAT(mpid_statp->recvWaitsComplete); - } -#endif - - return mpi_errno; -} - -#endif diff --git a/src/mpid/pamid/src/pt2pt/mpidi_recvmsg.c b/src/mpid/pamid/src/pt2pt/mpidi_recvmsg.c deleted file mode 100644 index 16092e6..0000000 --- a/src/mpid/pamid/src/pt2pt/mpidi_recvmsg.c +++ /dev/null @@ -1,161 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/pt2pt/mpidi_recvmsg.c - * \brief ADI level implemenation of common recv code. - */ -#include <mpidimpl.h> -#include <mpidi_macros.h> - - -void -MPIDI_RecvMsg_Unexp(MPIR_Request * rreq, - void * buf, - MPI_Aint count, - MPI_Datatype datatype) -{ - /* ------------------------------------------------------------ */ - /* message was found in unexpected queue */ - /* ------------------------------------------------------------ */ - /* We must acknowledge synchronous send requests */ - /* The recvnew callback will acknowledge the posted messages */ - /* Recv functions will ack the messages that are unexpected */ - /* ------------------------------------------------------------ */ - TRACE_SET_R_BIT((rreq->mpid.partner_id),(rreq->mpid.idx),fl.f.matchedInUQ); - - if (MPIDI_Request_isRzv(rreq)) - { - const unsigned is_sync = MPIDI_Request_isSync(rreq); - const unsigned is_zero = (rreq->mpid.envelope.length==0); - - /* -------------------------------------------------------- */ - /* Received an expected flow-control rendezvous RTS. */ - /* This is very similar to the found/incomplete case */ - /* -------------------------------------------------------- */ - if (HANDLE_GET_KIND(datatype) != HANDLE_KIND_BUILTIN) - { - MPIDU_Datatype_get_ptr(datatype, rreq->mpid.datatype_ptr); - MPIDU_Datatype_add_ref(rreq->mpid.datatype_ptr); - } - - if (likely((is_sync+is_zero) == 0)) - MPIDI_Context_post(MPIDI_Context_local(rreq), &rreq->mpid.post_request, MPIDI_RendezvousTransfer, rreq); - else if (is_sync != 0) - MPIDI_Context_post(MPIDI_Context_local(rreq), &rreq->mpid.post_request, MPIDI_RendezvousTransfer_SyncAck, rreq); - else - MPIDI_Context_post(MPIDI_Context_local(rreq), &rreq->mpid.post_request, MPIDI_RendezvousTransfer_zerobyte, rreq); - } - else - { - if (MPIR_cc_is_complete(&rreq->cc)) - { - if (unlikely(MPIDI_Request_isSync(rreq))) - { - /* Post this to the context for asynchronous progresss. We cannot do - * the send-immediate inline here because we may not have the - * context locked (its is being asynchrously advanced). - * Must "uncomplete" the message (increment the ref and completion counts) so we - * hold onto this request object until this send has completed. When MPIDI_SyncAck_handoff - * finishes sending the ack, it will complete the request, decrementing the ref and - * completion counts. - */ - MPIDI_Request_uncomplete(rreq); - MPIDI_Send_post(MPIDI_SyncAck_handoff, rreq); - } - /* -------------------------------- */ - /* request is complete */ - /* -------------------------------- */ - if (rreq->mpid.uebuf != NULL) - { - if (likely(MPIR_STATUS_GET_CANCEL_BIT(rreq->status) == FALSE)) - { - intptr_t _count=0; - MPIDI_Buffer_copy(rreq->mpid.uebuf, - rreq->mpid.uebuflen, - MPI_CHAR, - &rreq->status.MPI_ERROR, - buf, - count, - datatype, - &_count, - &rreq->status.MPI_ERROR); - MPIR_STATUS_SET_COUNT(rreq->status, _count); - } - } - else - { - MPID_assert(rreq->mpid.uebuflen == 0); - MPIR_STATUS_SET_COUNT(rreq->status, 0); - } - } - else - { - /* -------------------------------- */ - /* request is incomplete */ - /* -------------------------------- */ - if (unlikely(MPIDI_Request_isSync(rreq))) - { - /* Post this to the context for asynchronous progresss. We cannot do - * the send-immediate inline here because we may not have the - * context locked (its is being asynchrously advanced). - * Must "uncomplete" the message (increment the ref and completion counts) so we - * hold onto this request object until this send has completed. When MPIDI_SyncAck_handoff - * finishes sending the ack, it will complete the request, decrementing the ref and - * completion counts. - */ - MPIDI_Request_uncomplete(rreq); - MPIDI_Send_post(MPIDI_SyncAck_handoff, rreq); - } - if (MPIR_STATUS_GET_CANCEL_BIT(rreq->status) == FALSE) - { - MPIDI_Request_setCA(rreq, MPIDI_CA_UNPACK_UEBUF_AND_COMPLETE); - } - if (HANDLE_GET_KIND(datatype) != HANDLE_KIND_BUILTIN) - { - MPIDU_Datatype_get_ptr(datatype, rreq->mpid.datatype_ptr); - MPIDU_Datatype_add_ref(rreq->mpid.datatype_ptr); - } - } - } -} - - - -void -MPIDI_RecvMsg_procnull(MPIR_Comm * comm, - unsigned is_blocking, - MPI_Status * status, - MPIR_Request ** request) -{ - if (is_blocking) - { - MPIR_Status_set_procnull(status); - *request = NULL; - } - else - { - MPIR_Request * rreq; - rreq = MPIDI_Request_create2(); - MPIR_Status_set_procnull(&rreq->status); - rreq->kind = MPIR_REQUEST_KIND__RECV; - rreq->comm = comm; - MPIR_Comm_add_ref(comm); - MPIDI_Request_complete(rreq); - *request = rreq; - } -} diff --git a/src/mpid/pamid/src/pt2pt/mpidi_rendezvous.c b/src/mpid/pamid/src/pt2pt/mpidi_rendezvous.c deleted file mode 100644 index a218865..0000000 --- a/src/mpid/pamid/src/pt2pt/mpidi_rendezvous.c +++ /dev/null @@ -1,266 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/pt2pt/mpidi_rendezvous.c - * \brief Provide for a flow-control rendezvous-based send - */ -#include <mpidimpl.h> - -inline void -MPIDI_RendezvousTransfer_use_pami_rget(pami_context_t context, - pami_endpoint_t dest, - MPIR_Request *rreq) -__attribute__((__always_inline__)); -#ifdef RDMA_FAILOVER -inline void -MPIDI_RendezvousTransfer_use_pami_get(pami_context_t context, - pami_endpoint_t dest, - void *rcvbuf, - MPIR_Request *rreq) -__attribute__((__always_inline__)); -#endif - - -pami_result_t -MPIDI_RendezvousTransfer(pami_context_t context, - void * _rreq) -{ - MPIR_Request * rreq = (MPIR_Request*) _rreq; - - void *rcvbuf; - size_t rcvlen; - - /* -------------------------------------- */ - /* calculate message length for reception */ - /* calculate receive message "count" */ - /* -------------------------------------- */ - unsigned dt_contig; - size_t dt_size; - MPIDU_Datatype*dt_ptr; - MPI_Aint dt_true_lb; - MPIDI_Datatype_get_info(rreq->mpid.userbufcount, - rreq->mpid.datatype, - dt_contig, - dt_size, - dt_ptr, - dt_true_lb); - - /* -------------------------------------- */ - /* test for truncated message. */ - /* -------------------------------------- */ - if (rreq->mpid.envelope.length > dt_size) - { - rcvlen = dt_size; - rreq->status.MPI_ERROR = MPI_ERR_TRUNCATE; - MPIR_STATUS_SET_COUNT(rreq->status, rcvlen); - } - else - { - rcvlen = rreq->mpid.envelope.length; - } - - /* -------------------------------------- */ - /* if buffer is contiguous ... */ - /* -------------------------------------- */ - if (dt_contig) - { - MPIDI_Request_setCA(rreq, MPIDI_CA_COMPLETE); - rreq->mpid.uebuf = NULL; - rreq->mpid.uebuflen = 0; - rcvbuf = rreq->mpid.userbuf + dt_true_lb; - } - - /* --------------------------------------------- */ - /* buffer is non-contiguous. we need to allocate */ - /* a temporary buffer, and unpack later. */ - /* --------------------------------------------- */ - else - { - MPIDI_Request_setCA(rreq, MPIDI_CA_UNPACK_UEBUF_AND_COMPLETE); - rcvbuf = MPL_malloc(rcvlen); - MPID_assert(rcvbuf != NULL); - rreq->mpid.uebuf = rcvbuf; - rreq->mpid.uebuflen = rcvlen; - rreq->mpid.uebuf_malloc = mpiuMalloc; - } - - /* ---------------------------------------------------------------- */ - /* Get the data from the origin node. */ - /* ---------------------------------------------------------------- */ - - pami_result_t rc; - pami_endpoint_t dest; - MPIDI_Context_endpoint(rreq, &dest); - -#if CUDA_AWARE_SUPPORT - if(MPIDI_Process.cuda_aware_support_on && MPIDI_cuda_is_device_buf(rcvbuf)) - { - MPIDI_RendezvousTransfer_use_pami_get(context,dest,rcvbuf,rreq); - } - else - { -#endif - - -#ifdef USE_PAMI_RDMA - size_t rcvlen_out; - rc = PAMI_Memregion_create(context, - rcvbuf, - rcvlen, - &rcvlen_out, - &rreq->mpid.memregion); - MPID_assert(rc == PAMI_SUCCESS); - MPID_assert(rcvlen == rcvlen_out); - - TRACE_ERR("RZV Xfer for req=%p addr=%p *addr[0]=%#016llx *addr[1]=%#016llx\n", - rreq, - rcvbuf, - *(((unsigned long long*)rcvbuf)+0), - *(((unsigned long long*)rcvbuf)+1)); - - MPIDI_RendezvousTransfer_use_pami_rget(context,dest,rreq); -#else - rreq->mpid.memregion_used=0; - if( (!MPIDI_Process.mp_s_use_pami_get) && (rreq->mpid.envelope.memregion_used) ) - { - size_t rcvlen_out; - rc = PAMI_Memregion_create(context, - rcvbuf, - rcvlen, - &rcvlen_out, - &rreq->mpid.memregion); - if (rc == PAMI_SUCCESS) - { - rreq->mpid.memregion_used=1; - MPID_assert(rcvlen == rcvlen_out); - - TRACE_ERR("RZV Xfer for req=%p addr=%p *addr[0]=%#016llx *addr[1]=%#016llx\n", - rreq, - rcvbuf, - *(((unsigned long long*)rcvbuf)+0), - *(((unsigned long long*)rcvbuf)+1)); - MPIDI_RendezvousTransfer_use_pami_rget(context,dest,rreq); - } else { - MPIDI_RendezvousTransfer_use_pami_get(context,dest,rcvbuf,rreq); - } - } else { - MPIDI_RendezvousTransfer_use_pami_get(context,dest,rcvbuf,rreq); - } -#endif - -#if CUDA_AWARE_SUPPORT - } -#endif - - return PAMI_SUCCESS; -} - - -inline void -MPIDI_RendezvousTransfer_use_pami_rget(pami_context_t context, - pami_endpoint_t dest, - MPIR_Request *rreq) -{ - pami_result_t rc; - pami_rget_simple_t params = { - .rma = { - .dest = dest, - .hints = { - .buffer_registered = PAMI_HINT_ENABLE, - .use_rdma = PAMI_HINT_ENABLE, - .remote_async_progress = PAMI_HINT_DEFAULT, - .use_shmem = PAMI_HINT_DEFAULT, - }, - .bytes = rreq->mpid.envelope.length, - .cookie = rreq, - .done_fn = MPIDI_RecvRzvDoneCB, - }, - .rdma = { - .local = { - .mr = &rreq->mpid.memregion, - .offset = 0, - }, - .remote = { - .mr = &rreq->mpid.envelope.memregion, - .offset = 0, - }, - }, - }; - - rc = PAMI_Rget(context, &params); - MPID_assert(rc == PAMI_SUCCESS); -} - - -#ifdef RDMA_FAILOVER -inline void -MPIDI_RendezvousTransfer_use_pami_get(pami_context_t context, - pami_endpoint_t dest, - void *rcvbuf, - MPIR_Request *rreq) -{ - pami_result_t rc; - int val=0; - - if (MPIDI_Process.mp_s_use_pami_get) - val=PAMI_HINT_DEFAULT; - else - val=PAMI_HINT_DISABLE; - pami_get_simple_t params = { - .rma = { - .dest = dest, - .hints = { - .buffer_registered = PAMI_HINT_DEFAULT, - .use_rdma = val, - .remote_async_progress = PAMI_HINT_DEFAULT, - .use_shmem = PAMI_HINT_DEFAULT, -#ifndef OUT_OF_ORDER_HANDLING - .no_long_header= 1, -#endif - }, - .bytes = rreq->mpid.envelope.length, - .cookie = rreq, - .done_fn = MPIDI_RecvRzvDoneCB, - }, - .addr = { - .local = rcvbuf, - .remote = rreq->mpid.envelope.data, - }, - }; - - rc = PAMI_Get(context, &params); - MPID_assert(rc == PAMI_SUCCESS); -} -#endif - -pami_result_t MPIDI_RendezvousTransfer_SyncAck (pami_context_t context, void * _rreq) -{ - MPIR_Request *rreq = (MPIR_Request*)_rreq; - - // Do the sync ack transfer here. - MPIDI_SyncAck_post (context, rreq, MPIDI_Request_getPeerRank_pami(rreq)); - - // Continue on to the rendezvous transfer part. - return MPIDI_RendezvousTransfer(context, _rreq); -} - -pami_result_t MPIDI_RendezvousTransfer_zerobyte (pami_context_t context, void * _rreq) -{ - MPIDI_RecvRzvDoneCB_zerobyte (context, _rreq, PAMI_SUCCESS); - return PAMI_SUCCESS; -} diff --git a/src/mpid/pamid/src/pt2pt/mpidi_send.h b/src/mpid/pamid/src/pt2pt/mpidi_send.h deleted file mode 100644 index b061771..0000000 --- a/src/mpid/pamid/src/pt2pt/mpidi_send.h +++ /dev/null @@ -1,128 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/pt2pt/mpidi_send.h - * - * This is a generic inline send operator. It groups code previous - * found in mpid_{,i}{,s}send.c. The hope is that the common code - * will be easier to maintain, and that the inline nature of the flags - * will ensure high performance. - */ -#include <mpidimpl.h> - -#ifndef __src_pt2pt_mpidi_send_h__ -#define __src_pt2pt_mpidi_send_h__ - -#include <mpidi_macros.h> - -static inline void -MPIDI_SendDoneCB_inline(pami_context_t context, - void * clientdata, - pami_result_t result) -{ - MPIR_Request * sreq = (MPIR_Request*)clientdata; - MPID_assert(sreq != NULL); - MPIDI_Request_complete(sreq); -} - - -/** - * \brief This is a generic inline verion of the various send functions. - * - * This is the same function prototype as MPID_{,I}{,S}send.c, but it - * inludes two inline flags: is_blocking and is_sync. These are not - * meant to be truely variable, but are expected to be known at - * compile time. That *should* allow the compiler to remove any extra - * code from this function that won't be used in the specific inlined - * version. - * - * The PAMI_Context_post function may queue this send operation for - * later handling and then return to the user. Therefore, this - * function has only two goals: - * - * + Fill in the request oject with all relevant information, and - * + Do any reference counting that must be done before the function returns. - * - * - * \param[in] buf The buffer to send - * \param[in] count Number of elements in the buffer - * \param[in] datatype The datatype of each element - * \param[in] rank The destination rank - * \param[in] tag The message tag - * \param[in] comm Pointer to the communicator - * \param[in] context_offset Offset from the communicator context ID - * \param[in] is_blocking Is this a blocking request? (i.e. one of MPI_Send or MPI_Ssend) - * \param[in] is_sync Is this a synchronous request? (i.e. of of MPI_Ssend or MPI_Issend) - * \param[out] request Return a pointer to the new request object - * - * \returns An MPI Error code - */ -static inline int -MPIDI_Send(const void * buf, - MPI_Aint count, - MPI_Datatype datatype, - int rank, - int tag, - MPIR_Comm * comm, - int context_offset, - unsigned is_blocking, - unsigned is_sync, - MPIR_Request ** request) -{ - MPIR_Request * sreq = NULL; - - /* --------------------- */ - /* create a send request */ - /* --------------------- */ - - sreq = MPIDI_Request_create2(); - *request = sreq; - - /* match info */ - MPIDI_Request_setMatch(sreq, tag, comm->rank, comm->context_id+context_offset); - - /* data buffer info */ - sreq->mpid.userbuf = (void*)buf; - sreq->mpid.userbufcount = count; - sreq->mpid.datatype = datatype; - - /* communicator & destination info */ - sreq->comm = comm; MPIR_Comm_add_ref(comm); - MPIDI_Request_setPeerRank_comm(sreq, rank); - - /* message type info */ - sreq->kind = MPIR_REQUEST_KIND__SEND; - /* ----------------------------------------- */ - /* start the message */ - /* ----------------------------------------- */ - if (is_sync) - { - MPIDI_Request_uncomplete(sreq); - MPIDI_Send_post(MPIDI_Ssend_handoff, sreq); - } - else - { - MPIDI_Send_post(MPIDI_Send_handoff, sreq); - } - - return MPI_SUCCESS; -} - - - -#endif diff --git a/src/mpid/pamid/src/pt2pt/mpidi_sendmsg.c b/src/mpid/pamid/src/pt2pt/mpidi_sendmsg.c deleted file mode 100644 index a26b6fd..0000000 --- a/src/mpid/pamid/src/pt2pt/mpidi_sendmsg.c +++ /dev/null @@ -1,743 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/pt2pt/mpidi_sendmsg.c - * \brief Funnel point for starting all MPI messages - */ -#include <mpidimpl.h> -#ifndef min -#define min(a,b) (((a) < (b)) ? (a) : (b)) -#endif - -#if TOKEN_FLOW_CONTROL -#define MPIDI_Piggy_back_tokens MPIDI_Piggy_back_tokens_inline -static inline void * -MPIDI_Piggy_back_tokens_inline(int dest,MPIR_Request *shd,size_t len) - { - int rettoks=0; - if (MPIDI_Token_cntr[dest].rettoks) - { - rettoks=min(MPIDI_Token_cntr[dest].rettoks, TOKENS_BITMASK); - MPIDI_Token_cntr[dest].rettoks -= rettoks; - shd->mpid.envelope.msginfo.tokens = rettoks; - } - } -#endif - -static inline void -MPIDI_SendMsg_short(pami_context_t context, - MPIR_Request * sreq, - pami_endpoint_t dest, - void * sndbuf, - unsigned sndlen, - unsigned isSync) -{ - MPIDI_MsgInfo * msginfo = &sreq->mpid.envelope.msginfo; - - pami_send_immediate_t params = { - .dispatch = MPIDI_Protocols_Short, - .dest = dest, - .header = { - .iov_base = msginfo, - .iov_len = sizeof(MPIDI_MsgInfo), - }, - .data = { - .iov_base = sndbuf, - .iov_len = sndlen, - }, - }; - if (isSync) - params.dispatch = MPIDI_Protocols_ShortSync; - - pami_result_t rc; - rc = PAMI_Send_immediate(context, &params); -#ifdef TRACE_ON - if (rc) - { - TRACE_ERR("sizeof(msginfo)=%zu sizeof(data)=%u\n", sizeof(MPIDI_MsgInfo), sndlen); - } -#endif - MPID_assert(rc == PAMI_SUCCESS); - TRACE_SET_S_VAL(dest,(sreq->mpid.envelope.msginfo.MPIseqno & SEQMASK),mode,params.dispatch); - if (!isSync) { - TRACE_SET_S_BIT(dest,(sreq->mpid.envelope.msginfo.MPIseqno & SEQMASK),fl.f.NoComp); - TRACE_SET_S_BIT(dest,(sreq->mpid.envelope.msginfo.MPIseqno & SEQMASK),fl.f.sendShort); - } else { - TRACE_SET_S_BIT(dest,(sreq->mpid.envelope.msginfo.MPIseqno & SEQMASK),fl.f.sendEnvelop); - } - - MPIDI_SendDoneCB_inline(context, sreq, PAMI_SUCCESS); -#if (MPIDI_STATISTICS) - MPID_NSTAT(mpid_statp->sendsComplete); -#endif -} - -static void -MPIDI_SendMsg_eager(pami_context_t context, - MPIR_Request * sreq, - pami_endpoint_t dest, - void * sndbuf, - unsigned sndlen) - __attribute__((__noinline__)); -static void -MPIDI_SendMsg_eager(pami_context_t context, - MPIR_Request * sreq, - pami_endpoint_t dest, - void * sndbuf, - unsigned sndlen) -{ - MPIDI_MsgInfo * msginfo = &sreq->mpid.envelope.msginfo; - - pami_send_t params = { - .send = { - .dispatch = MPIDI_Protocols_Eager, - .dest = dest, - .header = { - .iov_base = msginfo, - .iov_len = sizeof(MPIDI_MsgInfo), - }, - .data = { - .iov_base = sndbuf, - .iov_len = sndlen, - }, - }, - .events = { - .cookie = sreq, - .local_fn = MPIDI_SendDoneCB, - .remote_fn= NULL, - }, - }; - - pami_result_t rc; - rc = PAMI_Send(context, &params); - MPID_assert(rc == PAMI_SUCCESS); - TRACE_SET_S_VAL(dest,(sreq->mpid.idx),mode,MPIDI_Protocols_Eager); - TRACE_SET_S_BIT(dest,(sreq->mpid.idx),fl.f.sendEager); -} - - -static void -MPIDI_SendMsg_rzv(pami_context_t context, - MPIR_Request * sreq, - pami_endpoint_t dest, - void * sndbuf, - size_t sndlen) - __attribute__((__noinline__)); -static void -MPIDI_SendMsg_rzv(pami_context_t context, - MPIR_Request * sreq, - pami_endpoint_t dest, - void * sndbuf, - size_t sndlen) -{ - pami_result_t rc; - - /* Set the isRzv bit in the SEND request. This is important for - * canceling requests. - */ - MPIDI_Request_setRzv(sreq, 1); - - /* The rendezvous information, such as the origin/local/sender - * node's send buffer and the number of bytes the origin node wishes - * to send, is sent as the payload of the request-to-send (RTS) - * message. - */ -#ifdef USE_PAMI_RDMA - size_t sndlen_out; - rc = PAMI_Memregion_create(context, - sndbuf, - sndlen, - &sndlen_out, - &sreq->mpid.envelope.memregion); - MPID_assert(rc == PAMI_SUCCESS); - MPID_assert(sndlen == sndlen_out); - TRACE_ERR("RZV send for mr=%#llx addr=%p *addr[0]=%#016llx *addr[1]=%#016llx bytes=%zu\n", - *(unsigned long long*)&sreq->mpid.envelope.memregion, - sndbuf, - *(((unsigned long long*)sndbuf)+0), - *(((unsigned long long*)sndbuf)+1), - sndlen); -#else - sreq->mpid.envelope.memregion_used = 0; -#ifdef OUT_OF_ORDER_HANDLING - if ((!MPIDI_Process.mp_s_use_pami_get) && (!sreq->mpid.envelope.msginfo.noRDMA)) -#else - if (!MPIDI_Process.mp_s_use_pami_get) -#endif - { - size_t sndlen_out; - rc = PAMI_Memregion_create(context, - sndbuf, - sndlen, - &sndlen_out, - &sreq->mpid.envelope.memregion); - if(rc == PAMI_SUCCESS) - { - MPID_assert(sndlen == sndlen_out); - TRACE_ERR("RZV send for mr=%#llx addr=%p *addr[0]=%#016llx *addr[1]=%#016llx bytes=%zu\n", - *(unsigned long long*)&sreq->mpid.envelope.memregion, - sndbuf, - *(((unsigned long long*)sndbuf)+0), - *(((unsigned long long*)sndbuf)+1), - sndlen); - sreq->mpid.envelope.memregion_used = 1; - } - sreq->mpid.envelope.data = sndbuf; - } - else - { - TRACE_ERR("RZV send (failed registration for sreq=%p addr=%p *addr[0]=%#016llx *addr[1]=%#016llx bytes=%zu\n", - sreq,sndbuf, - *(((unsigned long long*)sndbuf)+0), - *(((unsigned long long*)sndbuf)+1), - sndlen); - sreq->mpid.envelope.data = sndbuf; - } -#endif - sreq->mpid.envelope.length = sndlen; - - /* Do not specify a callback function to be invoked when the RTS - * message has been sent. The MPI_Send is completed only when the - * target/remote/receiver node has completed an PAMI_Get from the - * origin node and has then sent a rendezvous acknowledgement (ACK) - * to the origin node to signify the end of the transfer. When the - * ACK message is received by the origin node the same callback - * function is used to complete the MPI_Send as the non-rendezvous - * case. - */ - pami_send_immediate_t params = { - .dispatch = MPIDI_Protocols_RVZ, - .dest = dest, - .header = { - .iov_base = &sreq->mpid.envelope, - .iov_len = sizeof(MPIDI_MsgEnvelope), - }, - .data = { - .iov_base = NULL, - .iov_len = 0, - }, - }; - - rc = PAMI_Send_immediate(context, &params); - MPID_assert(rc == PAMI_SUCCESS); - TRACE_SET_S_VAL(dest,(sreq->mpid.idx),bufaddr,sreq->mpid.envelope.data); - TRACE_SET_S_VAL(dest,(sreq->mpid.idx),mode,MPIDI_Protocols_RVZ); - TRACE_SET_S_BIT(dest,(sreq->mpid.idx),fl.f.sendRzv); - TRACE_SET_S_BIT(dest,(sreq->mpid.idx),fl.f.sendEnvelop); - TRACE_SET_S_VAL(dest,(sreq->mpid.idx),fl.f.memRegion,sreq->mpid.envelope.memregion_used); - TRACE_SET_S_VAL(dest,(sreq->mpid.idx),fl.f.use_pami_get,MPIDI_Process.mp_s_use_pami_get); -} - - -static void -MPIDI_SendMsg_rzv_zerobyte(pami_context_t context, - MPIR_Request * sreq, - pami_endpoint_t dest) - __attribute__((__noinline__)); -static void -MPIDI_SendMsg_rzv_zerobyte(pami_context_t context, - MPIR_Request * sreq, - pami_endpoint_t dest) -{ - pami_result_t rc; - - /* Set the isRzv bit in the SEND request. This is important for - * canceling requests. - */ - MPIDI_Request_setRzv(sreq, 1); - - /* The rendezvous information, such as the origin/local/sender - * node's send buffer and the number of bytes the origin node wishes - * to send, is sent as the payload of the request-to-send (RTS) - * message. - */ - - sreq->mpid.envelope.data = NULL; - sreq->mpid.envelope.length = 0; - - /* Do not specify a callback function to be invoked when the RTS - * message has been sent. The MPI_Send is completed only when the - * target/remote/receiver node has matched the receive and has then - * sent a rendezvous acknowledgement (ACK) to the origin node to - * signify the end of the transfer. When the ACK message is received - * by the origin node the same callback function is used to complete - * the MPI_Send as the non-rendezvous case. - */ - pami_send_immediate_t params = { - .dispatch = MPIDI_Protocols_RVZ_zerobyte, - .dest = dest, - .header = { - .iov_base = &sreq->mpid.envelope, - .iov_len = sizeof(MPIDI_MsgEnvelope), - }, - .data = { - .iov_base = NULL, - .iov_len = 0, - }, - }; - - rc = PAMI_Send_immediate(context, &params); - MPID_assert(rc == PAMI_SUCCESS); - TRACE_SET_S_VAL(dest,(sreq->mpid.idx),bufaddr,sreq->mpid.envelope.data); - TRACE_SET_S_VAL(dest,(sreq->mpid.idx),mode,MPIDI_Protocols_RVZ_zerobyte); - TRACE_SET_S_VAL(dest,(sreq->mpid.idx),fl.f.memRegion,sreq->mpid.envelope.memregion_used); - TRACE_SET_S_VAL(dest,(sreq->mpid.idx),fl.f.use_pami_get,MPIDI_Process.mp_s_use_pami_get); - TRACE_SET_S_BIT(dest,(sreq->mpid.idx),fl.f.sendRzv); - TRACE_SET_S_BIT(dest,(sreq->mpid.idx),fl.f.sendEnvelop); -} - - - -static void -MPIDI_SendMsg_process_userdefined_dt(MPIR_Request * sreq, - void ** sndbuf, - size_t * data_sz) - __attribute__((__noinline__)); -static void -MPIDI_SendMsg_process_userdefined_dt(MPIR_Request * sreq, - void ** _sndbuf, - size_t * _data_sz) -{ - size_t data_sz; - int dt_contig; - MPI_Aint dt_true_lb; - MPIDU_Datatype* dt_ptr; - void * sndbuf; - - /* - * Get the datatype info - */ - MPIDI_Datatype_get_info(sreq->mpid.userbufcount, - sreq->mpid.datatype, - dt_contig, - data_sz, - dt_ptr, - dt_true_lb); - - MPID_assert(sreq->mpid.uebuf == NULL); - - /* - * Contiguous data type - */ - if (likely(dt_contig)) - { - sndbuf = sreq->mpid.userbuf + dt_true_lb; - } - - /* - * Non-contiguous data type; allocate and populate temporary send - * buffer - */ - else - { - char *buf = NULL; -#if CUDA_AWARE_SUPPORT - // This will need to be done in two steps: - // 1 - Allocate a temp buffer which is the same size as user buffer and copy in it. - // 2 - Pack data into ue buffer from temp buffer. - int on_device = MPIDI_cuda_is_device_buf(sreq->mpid.userbuf); - if(MPIDI_Process.cuda_aware_support_on && on_device) - { - MPI_Aint dt_extent; - MPIDU_Datatype_get_extent_macro(sreq->mpid.datatype, dt_extent); - buf = MPL_malloc(dt_extent * sreq->mpid.userbufcount); - - cudaError_t cudaerr = CudaMemcpy(buf, sreq->mpid.userbuf, dt_extent * sreq->mpid.userbufcount, cudaMemcpyDeviceToHost); - if (cudaSuccess != cudaerr) { - fprintf(stderr, "cudaMalloc failed: %s\n", CudaGetErrorString(cudaerr)); - } - - } -#endif - - MPIDU_Segment segment; - - if(data_sz != 0) { - sreq->mpid.uebuf = sndbuf = MPL_malloc(data_sz); - if (unlikely(sndbuf == NULL)) - { - sreq->status.MPI_ERROR = MPI_ERR_NO_SPACE; - MPIR_STATUS_SET_COUNT(sreq->status, 0); - MPID_Abort(NULL, MPI_ERR_NO_SPACE, -1, - "Unable to allocate non-contiguous buffer"); - } - sreq->mpid.uebuf_malloc = mpiuMalloc; - - DLOOP_Offset last = data_sz; -#if CUDA_AWARE_SUPPORT - if(!MPIDI_Process.cuda_aware_support_on || !on_device) - buf = sreq->mpid.userbuf; -#else - buf = sreq->mpid.userbuf; -#endif - MPID_assert(buf != NULL); - - MPIDU_Segment_init(buf, - sreq->mpid.userbufcount, - sreq->mpid.datatype, - &segment, - 0); - MPIDU_Segment_pack(&segment, 0, &last, sndbuf); - MPID_assert(last == data_sz); -#if CUDA_AWARE_SUPPORT - if(MPIDI_Process.cuda_aware_support_on && on_device) - MPL_free(buf); -#endif - } else { - sndbuf = NULL; - } - } - - *_sndbuf = sndbuf; - *_data_sz = data_sz; -} - - -static inline void -MPIDI_SendMsg(pami_context_t context, - MPIR_Request * sreq, - unsigned isSync, - const unsigned isInternal) -{ - /* ------------------------------ */ - /* special case: NULL destination */ - /* ------------------------------ */ - int rank = MPIDI_Request_getPeerRank_comm(sreq); - if (unlikely(rank == MPI_PROC_NULL)) - { - if (isSync) - MPIDI_Request_complete(sreq); - MPIDI_Request_complete(sreq); - return; - } - else - { - MPIDI_Request_setPeerRank_pami(sreq, MPID_VCR_GET_LPID(sreq->comm->vcr, rank)); - } - - MPIDI_Request_setSync(sreq, isSync); - MPIDI_Request_setPeerRequestH(sreq); - - /* - * Create the destination endpoint - */ - pami_endpoint_t dest; - MPIDI_Context_endpoint(sreq, &dest); - pami_task_t dest_tid; - dest_tid=sreq->comm->vcr[rank]->taskid; -#if (MPIDI_STATISTICS) - MPID_NSTAT(mpid_statp->sends); -#endif -#ifdef OUT_OF_ORDER_HANDLING - MPIDI_Out_cntr_t *out_cntr; - - MPIU_THREAD_CS_ENTER(MSGQUEUE,0); - out_cntr = &MPIDI_Out_cntr[dest_tid]; - out_cntr->nMsgs++; - MPIU_THREAD_CS_EXIT(MSGQUEUE,0); - MPIDI_Request_setMatchSeq(sreq, out_cntr->nMsgs); -#endif -if (!TOKEN_FLOW_CONTROL_ON) { - size_t data_sz; - void * sndbuf; - if (likely(HANDLE_GET_KIND(sreq->mpid.datatype) == HANDLE_KIND_BUILTIN)) - { - sndbuf = sreq->mpid.userbuf; - data_sz = sreq->mpid.userbufcount * MPIDU_Datatype_get_basic_size(sreq->mpid.datatype); - } - else - { - MPIDI_SendMsg_process_userdefined_dt(sreq, &sndbuf, &data_sz); - } - MPIDI_GET_S_REC(dest_tid,sreq,context,isSync,data_sz); - -#ifdef OUT_OF_ORDER_HANDLING - sreq->mpid.envelope.msginfo.noRDMA=0; -#endif - - const unsigned isLocal = PAMIX_Task_is_local(dest_tid); - const size_t data_sz_limit = isSync?ULONG_MAX:data_sz; - - /* - * Always use the short protocol when data_sz is small. - */ - if (likely(data_sz < MPIDI_PT2PT_SHORT_LIMIT(isInternal,isLocal))) - { - TRACE_ERR("Sending(short%s%s) bytes=%zu (short_limit=%u)\n", isInternal==1?",internal":"", isLocal==1?",intranode":"", data_sz, MPIDI_PT2PT_SHORT_LIMIT(isInternal,isLocal)); - MPIDI_SendMsg_short(context, - sreq, - dest, - sndbuf, - data_sz, - isSync); - } - /* - * Use the eager protocol when data_sz is less than the eager limit. - */ - else if (data_sz_limit < MPIDI_PT2PT_EAGER_LIMIT(isInternal,isLocal)) - { - TRACE_ERR("Sending(eager%s%s) bytes=%zu (eager_limit=%u)\n", isInternal==1?",internal":"", isLocal==1?",intranode":"", data_sz, MPIDI_PT2PT_EAGER_LIMIT(isInternal,isLocal)); - MPIDI_SendMsg_eager(context, - sreq, - dest, - sndbuf, - data_sz); -#ifdef MPIDI_STATISTICS - if (!isLocal && MPIR_cc_is_complete(&sreq->cc)) - { - MPID_NSTAT(mpid_statp->sendsComplete); - } -#endif - } - /* - * Use the default rendezvous protocol implementation that guarantees - * no unexpected data and does not complete the send until the remote - * receive is posted. - */ - else - { - TRACE_ERR("Sending(rendezvous%s%s) bytes=%zu (eager_limit=%u)\n", isInternal==1?",internal":"", isLocal==1?",intranode":"", data_sz, MPIDI_PT2PT_EAGER_LIMIT(isInternal,isLocal)); -#ifdef OUT_OF_ORDER_HANDLING - sreq->mpid.envelope.msginfo.noRDMA=isLocal; -#endif - if (likely(data_sz > 0)) - { - MPIDI_SendMsg_rzv(context, - sreq, - dest, - sndbuf, - data_sz); - } - else - { - MPIDI_SendMsg_rzv_zerobyte(context, sreq, dest); - } - -#ifdef MPIDI_STATISTICS - if (!isLocal && MPIR_cc_is_complete(&sreq->cc)) - { - MPID_NSTAT(mpid_statp->sendsComplete); - } -#endif - } - } - else - { /* TOKEN_FLOW_CONTROL_ON */ - #if TOKEN_FLOW_CONTROL - if (!(sreq->mpid.userbufcount)) - { - MPIDI_GET_S_REC(dest_tid,sreq,context,isSync,0); - TRACE_ERR("Sending(short,intranode) bytes=%u (short_limit=%u)\n", data_sz, MPIDI_Process.short_limit); - MPIU_THREAD_CS_ENTER(MSGQUEUE,0); - MPIDI_Piggy_back_tokens(dest,sreq,0); - MPIU_THREAD_CS_EXIT(MSGQUEUE,0); - MPIDI_SendMsg_short(context, - sreq, - dest, - sreq->mpid.userbuf, - 0, - isSync); - } - else - { - size_t data_sz; - void * sndbuf; - int noRDMA=0; - if (likely(HANDLE_GET_KIND(sreq->mpid.datatype) == HANDLE_KIND_BUILTIN)) - { - sndbuf = sreq->mpid.userbuf; - data_sz = sreq->mpid.userbufcount * MPIDU_Datatype_get_basic_size(sreq->mpid.datatype); - } - else - { - MPIDI_SendMsg_process_userdefined_dt(sreq, &sndbuf, &data_sz); - } - MPIDI_GET_S_REC(dest_tid,sreq,context,isSync,data_sz); - if (unlikely(PAMIX_Task_is_local(dest_tid) != 0)) noRDMA=1; - - MPIU_THREAD_CS_ENTER(MSGQUEUE,0); - if ((!isSync) && MPIDI_Token_cntr[dest].tokens >= 1) - { - if (data_sz <= MPIDI_Process.pt2pt.limits.application.immediate.remote) - { - TRACE_ERR("Sending(short,intranode) bytes=%u (short_limit=%u)\n", data_sz,MPIDI_Process.pt2pt.limits.application.immediate.remote); - --MPIDI_Token_cntr[dest].tokens; - MPIDI_Piggy_back_tokens(dest,sreq,data_sz); - MPIU_THREAD_CS_EXIT(MSGQUEUE,0); - MPIDI_SendMsg_short(context, - sreq, - dest, - sndbuf, - data_sz, - isSync); - } - else if (data_sz <= MPIDI_Process.pt2pt.limits.application.eager.remote) - { - TRACE_ERR("Sending(eager) bytes=%u (eager_limit=%u)\n", data_sz, MPIDI_Process.pt2pt.limits.application.eager.remote); - --MPIDI_Token_cntr[dest].tokens; - MPIDI_Piggy_back_tokens(dest,sreq,data_sz); - MPIU_THREAD_CS_EXIT(MSGQUEUE,0); - MPIDI_SendMsg_eager(context, - sreq, - dest, - sndbuf, - data_sz); -#ifdef MPIDI_STATISTICS - if (MPIR_cc_is_complete(&sreq->cc)) { - MPID_NSTAT(mpid_statp->sendsComplete); - } -#endif - - } - else /* rendezvous message */ - { - TRACE_ERR("Sending(RZV) bytes=%u (eager_limit=%u)\n", data_sz, MPIDI_Process.pt2pt.limits.application.eager.remote); - MPIDI_Piggy_back_tokens(dest,sreq,data_sz); - MPIU_THREAD_CS_EXIT(MSGQUEUE,0); - sreq->mpid.envelope.msginfo.noRDMA=noRDMA; - MPIDI_SendMsg_rzv(context, - sreq, - dest, - sndbuf, - data_sz); -#ifdef MPIDI_STATISTICS - if (MPIR_cc_is_complete(&sreq->cc)) - { - MPID_NSTAT(mpid_statp->sendsComplete); - } -#endif - } - } - else - { /* no tokens, all messages use rendezvous protocol */ - if ((data_sz <= MPIDI_Process.pt2pt.limits.application.eager.remote) && (!isSync)) { - ++MPIDI_Token_cntr[dest].n_tokenStarved; - sreq->mpid.envelope.msginfo.noRDMA=1; - } - else sreq->mpid.envelope.msginfo.noRDMA=noRDMA; - MPIDI_Piggy_back_tokens(dest,sreq,data_sz); - MPIU_THREAD_CS_EXIT(MSGQUEUE,0); - TRACE_ERR("Sending(RZV) bytes=%u (eager_limit=%u)\n", data_sz, MPIDI_Process.pt2pt.limits.application.eager.remote); - if (likely(data_sz > 0)) - { - MPIDI_SendMsg_rzv(context, - sreq, - dest, - sndbuf, - data_sz); - } - else - { - MPIDI_SendMsg_rzv_zerobyte(context, sreq, dest); - } -#ifdef MPIDI_STATISTICS - if (MPIR_cc_is_complete(&sreq->cc)) - { - MPID_NSTAT(mpid_statp->sendsComplete); - } -#endif - } - } - #else - MPID_assert_always(0); - #endif /* TOKEN_FLOW_CONTROL */ - } -} - - -/* - * \brief Central function for all low-level sends. - * - * This is assumed to have been posted to a context, and is now being - * called from inside advance. This has (unspecified) locking - * implications. - * - * Prerequisites: - * + Not sending to a NULL rank - * + Request already allocated - * - * \param[in] context The PAMI context on which to do the send operation - * \param[in,out] sreq Structure containing all relevant info about the message. - */ -pami_result_t -MPIDI_Send_handoff(pami_context_t context, - void * _sreq) -{ - MPIR_Request * sreq = (MPIR_Request*)_sreq; - MPID_assert(sreq != NULL); - - MPIDI_SendMsg(context, sreq, 0, 0); - return PAMI_SUCCESS; -} - - -pami_result_t -MPIDI_Ssend_handoff(pami_context_t context, - void * _sreq) -{ - MPIR_Request * sreq = (MPIR_Request*)_sreq; - MPID_assert(sreq != NULL); - - MPIDI_SendMsg(context, sreq, 1, 0); - return PAMI_SUCCESS; -} - - -/* - * \brief Central function for all low-level sends. - * - * This is assumed to have been posted to a context, and is now being - * called from inside advance. This has (unspecified) locking - * implications. - * - * Prerequisites: - * + Not sending to a NULL rank - * + Request already allocated - * - * \param[in] context The PAMI context on which to do the send operation - * \param[in,out] sreq Structure containing all relevant info about the message. - */ -pami_result_t -MPIDI_Isend_handoff(pami_context_t context, - void * _sreq) -{ - MPIR_Request * sreq = (MPIR_Request*)_sreq; - MPID_assert(sreq != NULL); - - /* This initializes all the fields not set in MPI_Isend() */ - MPIDI_Request_initialize(sreq); - - /* Since this is only called from MPI_Isend(), it is not synchronous */ - MPIDI_SendMsg(context, sreq, 0, 0); - return PAMI_SUCCESS; -} - -pami_result_t -MPIDI_Isend_handoff_internal(pami_context_t context, - void * _sreq) -{ - MPIR_Request * sreq = (MPIR_Request*)_sreq; - MPID_assert(sreq != NULL); - - /* This initializes all the fields not set in MPI_Isend() */ - MPIDI_Request_initialize(sreq); - - /* Since this is only called from MPI_Isend(), it is not synchronous */ - MPIDI_SendMsg(context, sreq, 0, 1); - return PAMI_SUCCESS; -} diff --git a/src/mpid/pamid/src/pt2pt/persistent/Makefile.mk b/src/mpid/pamid/src/pt2pt/persistent/Makefile.mk deleted file mode 100644 index e2894a6..0000000 --- a/src/mpid/pamid/src/pt2pt/persistent/Makefile.mk +++ /dev/null @@ -1,30 +0,0 @@ -# begin_generated_IBM_copyright_prolog -# -# This is an automatically generated copyright prolog. -# After initializing, DO NOT MODIFY OR MOVE -# --------------------------------------------------------------- -# Licensed Materials - Property of IBM -# Blue Gene/Q 5765-PER 5765-PRP -# -# (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved -# US Government Users Restricted Rights - -# Use, duplication, or disclosure restricted -# by GSA ADP Schedule Contract with IBM Corp. -# -# --------------------------------------------------------------- -# -# end_generated_IBM_copyright_prolog -# -*- mode: makefile-gmake; -*- - -# note that the includes always happen but the effects of their contents are -# affected by "if BUILD_PAMID" -if BUILD_PAMID - - -mpi_core_sources += \ - src/mpid/pamid/src/pt2pt/persistent/mpid_recv_init.c \ - src/mpid/pamid/src/pt2pt/persistent/mpid_send_init.c \ - src/mpid/pamid/src/pt2pt/persistent/mpid_startall.c - -endif BUILD_PAMID - diff --git a/src/mpid/pamid/src/pt2pt/persistent/mpid_recv_init.c b/src/mpid/pamid/src/pt2pt/persistent/mpid_recv_init.c deleted file mode 100644 index 63ebbeb..0000000 --- a/src/mpid/pamid/src/pt2pt/persistent/mpid_recv_init.c +++ /dev/null @@ -1,56 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/pt2pt/persistent/mpid_recv_init.c - * \brief ??? - */ - -/* This creates and initializes a persistent recv request */ - -#include <mpidimpl.h> - -int MPID_Recv_init(void * buf, - int count, - MPI_Datatype datatype, - int rank, - int tag, - MPIR_Comm * comm, - int context_offset, - MPIR_Request ** request) -{ - MPIR_Request * rreq = *request = MPIDI_Request_create2(); - - rreq->kind = MPIR_REQUEST_KIND__PREQUEST_RECV; - rreq->comm = comm; - MPIR_Comm_add_ref(comm); - MPIDI_Request_setMatch(rreq, tag, rank, comm->recvcontext_id+context_offset); - rreq->mpid.userbuf = buf; - rreq->mpid.userbufcount = count; - rreq->mpid.datatype = datatype; - rreq->partner_request = NULL; - MPIDI_Request_complete(rreq); - - MPIDI_Request_setPType(rreq, MPIDI_REQUEST_PTYPE_RECV); - if (HANDLE_GET_KIND(datatype) != HANDLE_KIND_BUILTIN) - { - MPIDU_Datatype_get_ptr(datatype, rreq->mpid.datatype_ptr); - MPIDU_Datatype_add_ref(rreq->mpid.datatype_ptr); - } - - return MPI_SUCCESS; -} diff --git a/src/mpid/pamid/src/pt2pt/persistent/mpid_send_init.c b/src/mpid/pamid/src/pt2pt/persistent/mpid_send_init.c deleted file mode 100644 index 35080e5..0000000 --- a/src/mpid/pamid/src/pt2pt/persistent/mpid_send_init.c +++ /dev/null @@ -1,148 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/pt2pt/persistent/mpid_send_init.c - * \brief ??? - */ -/* This creates and initializes a persistent send request */ - -#include <mpidimpl.h> - -/** - * *************************************************************************** - * create a persistent send template - * *************************************************************************** - */ - -static inline int -MPID_PSendRequest(const void * buf, - int count, - MPI_Datatype datatype, - int rank, - int tag, - MPIR_Comm * comm, - int context_offset, - MPIR_Request ** request) -{ - MPIR_Request* sreq = *request = MPIDI_Request_create2(); - - sreq->kind = MPIR_REQUEST_KIND__PREQUEST_SEND; - sreq->comm = comm; - MPIR_Comm_add_ref(comm); - MPIDI_Request_setMatch(sreq, tag, rank, comm->context_id+context_offset); - sreq->mpid.userbuf = (void*)buf; - sreq->mpid.userbufcount = count; - sreq->mpid.datatype = datatype; - sreq->partner_request = NULL; - MPIDI_Request_complete(sreq); - - if (HANDLE_GET_KIND(datatype) != HANDLE_KIND_BUILTIN) - { - MPIDU_Datatype_get_ptr(datatype, sreq->mpid.datatype_ptr); - MPIDU_Datatype_add_ref(sreq->mpid.datatype_ptr); - } - - return MPI_SUCCESS; -} - -/** - * *************************************************************************** - * simple persistent send - * *************************************************************************** - */ - -int MPID_Send_init(const void * buf, - int count, - MPI_Datatype datatype, - int rank, - int tag, - MPIR_Comm * comm, - int context_offset, - MPIR_Request ** request) -{ - int mpi_errno = MPID_PSendRequest(buf, - count, - datatype, - rank, - tag, - comm, - context_offset, - request); - if (mpi_errno != MPI_SUCCESS) - return mpi_errno; - MPIDI_Request_setPType((*request), MPIDI_REQUEST_PTYPE_SEND); - return MPI_SUCCESS; -} - -/** - * *************************************************************************** - * persistent synchronous send - * *************************************************************************** - */ - -int MPID_Ssend_init(const void * buf, - int count, - MPI_Datatype datatype, - int rank, - int tag, - MPIR_Comm * comm, - int context_offset, - MPIR_Request ** request) -{ - int mpi_errno = MPID_PSendRequest(buf, - count, - datatype, - rank, - tag, - comm, - context_offset, - request); - if (mpi_errno != MPI_SUCCESS) - return mpi_errno; - MPIDI_Request_setPType((*request), MPIDI_REQUEST_PTYPE_SSEND); - return MPI_SUCCESS; -} - -/** - * *************************************************************************** - * persistent buffered send - * *************************************************************************** - */ - -int MPID_Bsend_init(const void * buf, - int count, - MPI_Datatype datatype, - int rank, - int tag, - MPIR_Comm * comm, - int context_offset, - MPIR_Request ** request) -{ - int mpi_errno = MPID_PSendRequest(buf, - count, - datatype, - rank, - tag, - comm, - context_offset, - request); - if (mpi_errno != MPI_SUCCESS) - return mpi_errno; - MPIDI_Request_setPType((*request), MPIDI_REQUEST_PTYPE_BSEND); - return MPI_SUCCESS; -} diff --git a/src/mpid/pamid/src/pt2pt/persistent/mpid_startall.c b/src/mpid/pamid/src/pt2pt/persistent/mpid_startall.c deleted file mode 100644 index dbb1c3a..0000000 --- a/src/mpid/pamid/src/pt2pt/persistent/mpid_startall.c +++ /dev/null @@ -1,140 +0,0 @@ -/* begin_generated_IBM_copyright_prolog */ -/* */ -/* This is an automatically generated copyright prolog. */ -/* After initializing, DO NOT MODIFY OR MOVE */ -/* --------------------------------------------------------------- */ -/* Licensed Materials - Property of IBM */ -/* Blue Gene/Q 5765-PER 5765-PRP */ -/* */ -/* (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved */ -/* US Government Users Restricted Rights - */ -/* Use, duplication, or disclosure restricted */ -/* by GSA ADP Schedule Contract with IBM Corp. */ -/* */ -/* --------------------------------------------------------------- */ -/* */ -/* end_generated_IBM_copyright_prolog */ -/* (C)Copyright IBM Corp. 2007, 2011 */ -/** - * \file src/pt2pt/persistent/mpid_startall.c - * \brief ??? - */ -#include <mpidimpl.h> -#include <../mpi/pt2pt/bsendutil.h> - -int MPID_Startall(int count, MPIR_Request * requests[]) -{ - int rc=MPI_SUCCESS, i; - for (i = 0; i < count; i++) - { - MPIR_Request * const preq = requests[i]; - switch(MPIDI_Request_getPType(preq)) - { - case MPIDI_REQUEST_PTYPE_RECV: - { - rc = MPID_Irecv(preq->mpid.userbuf, - preq->mpid.userbufcount, - preq->mpid.datatype, - MPIDI_Request_getMatchRank(preq), - MPIDI_Request_getMatchTag(preq), - preq->comm, - MPIDI_Request_getMatchCtxt(preq) - preq->comm->recvcontext_id, - &preq->partner_request); - break; - } - case MPIDI_REQUEST_PTYPE_SEND: - { - rc = MPID_Isend(preq->mpid.userbuf, - preq->mpid.userbufcount, - preq->mpid.datatype, - MPIDI_Request_getMatchRank(preq), - MPIDI_Request_getMatchTag(preq), - preq->comm, - MPIDI_Request_getMatchCtxt(preq) - preq->comm->context_id, - &preq->partner_request); - break; - } - case MPIDI_REQUEST_PTYPE_SSEND: - { - rc = MPID_Issend(preq->mpid.userbuf, - preq->mpid.userbufcount, - preq->mpid.datatype, - MPIDI_Request_getMatchRank(preq), - MPIDI_Request_getMatchTag(preq), - preq->comm, - MPIDI_Request_getMatchCtxt(preq) - preq->comm->context_id, - &preq->partner_request); - break; - } - case MPIDI_REQUEST_PTYPE_BSEND: - { - rc = MPIR_Bsend_isend(preq->mpid.userbuf, - preq->mpid.userbufcount, - preq->mpid.datatype, - MPIDI_Request_getMatchRank(preq), - MPIDI_Request_getMatchTag(preq), - preq->comm, - BSEND_INIT, - &preq->partner_request); - /* - * MPICH maintains an independant reference to the child, - * but doesn't refcount it. Since they actually call - * MPI_Test() on the child request (which will release a - * ref iff the request is complete), we have to increment - * the ref_count so that it doesn't get freed from under - * us. - */ - if (preq->partner_request != NULL) - MPIR_Object_add_ref(preq->partner_request); - break; - } - - default: - { - rc = MPIR_Err_create_code(MPI_SUCCESS, - MPIR_ERR_FATAL, - __FUNCTION__, - __LINE__, - MPI_ERR_INTERN, - "**ch3|badreqtype", - "**ch3|badreqtype %d", - MPIDI_Request_getPType(preq)); - } - - } /* switch should end here, bug fixed. */ - - if (rc == MPI_SUCCESS) - { - preq->status.MPI_ERROR = MPI_SUCCESS; - if (MPIDI_Request_getPType(preq) == MPIDI_REQUEST_PTYPE_BSEND) - { - /* - * Complete a persistent Bsend immediately. - * - * Because the child of a persistent Bsend is just a - * normal Isend on a temp buffer, we don't need to wait on - * the child when the user calls MPI_Wait on the parent. - * Therefore, disconnect the cc_ptr link to the child and - * mark the parent complete. - */ - preq->cc_ptr = &preq->cc; - MPID_Request_set_completed(preq); - } - else - preq->cc_ptr = &preq->partner_request->cc; - } - else - { - /* If a failure occurs attempting to start the request, - then we assume that partner request was not created, - and stuff the error code in the persistent request. - The wait and test routines will look at the error code - in the persistent request if a partner request is not present. */ - preq->partner_request = NULL; - preq->status.MPI_ERROR = rc; - preq->cc_ptr = &preq->cc; - MPID_Request_set_completed(preq); - } - } /* for */ - return rc; -} diff --git a/src/mpid/pamid/subconfigure.m4 b/src/mpid/pamid/subconfigure.m4 deleted file mode 100644 index 817b240..0000000 --- a/src/mpid/pamid/subconfigure.m4 +++ /dev/null @@ -1,351 +0,0 @@ -[#] start of __file__ -dnl begin_generated_IBM_copyright_prolog -dnl -dnl This is an automatically generated copyright prolog. -dnl After initializing, DO NOT MODIFY OR MOVE -dnl --------------------------------------------------------------- -dnl Licensed Materials - Property of IBM -dnl Blue Gene/Q 5765-PER 5765-PRP -dnl -dnl (C) Copyright IBM Corp. 2011, 2012 All Rights Reserved -dnl US Government Users Restricted Rights - -dnl Use, duplication, or disclosure restricted -dnl by GSA ADP Schedule Contract with IBM Corp. -dnl -dnl --------------------------------------------------------------- -dnl -dnl end_generated_IBM_copyright_prolog -dnl -*- mode: makefile-gmake; -*- - -dnl MPICH_SUBCFG_BEFORE=src/mpid/common/sched -dnl MPICH_SUBCFG_BEFORE=src/mpid/common/datatype -dnl MPICH_SUBCFG_BEFORE=src/mpid/common/thread -dnl MPICH_SUBCFG_BEFORE=src/mpid/common/shm - -dnl _PREREQ handles the former role of mpichprereq, setup_device, etc -[#] expansion is: PAC_SUBCFG_PREREQ_[]PAC_SUBCFG_AUTO_SUFFIX -AC_DEFUN([PAC_SUBCFG_PREREQ_]PAC_SUBCFG_AUTO_SUFFIX,[ -AM_CONDITIONAL([BUILD_PAMID],[test "$device_name" = "pamid"]) -AM_CONDITIONAL([QUEUE_BINARY_SEARCH_SUPPORT],[test "$enable_queue_binary_search" = "yes"]) - -AC_ARG_VAR([PAMILIBNAME],[can be used to override the name of the PAMI library (default: "pami")]) -original_PAMILIBNAME=${PAMILIBNAME} -PAMILIBNAME=${PAMILIBNAME:-"pami"} -AC_SUBST(PAMILIBNAME) -export PAMILIBNAME - -dnl this subconfigure.m4 handles the configure work for the ftb subdir too -dnl this AM_CONDITIONAL only works because enable_ftb is set very early on by -dnl autoconf's argument parsing code. The "action-if-given" from the -dnl AC_ARG_ENABLE has not yet run -dnl AM_CONDITIONAL([BUILD_CH3_UTIL_FTB],[test "x$enable_ftb" = "xyes"]) - -AM_COND_IF([BUILD_PAMID],[ - -pamid_platform=${device_args} -if test x"$pamid_platform" == "x"; then - AS_CASE([$host], - [*-bgq-*],[pamid_platform=BGQ]) -fi - -AC_MSG_NOTICE([Using the pamid platform '$pamid_platform']) - - -dnl Set a value for the maximum processor name. -MPID_MAX_PROCESSOR_NAME=128 -PM_REQUIRES_PMI=pmi2 -if test "${pamid_platform}" = "PE" ; then - with_shared_memory=sysv - PM_REQUIRES_PMI=pmi2/poe -elif test "${pamid_platform}" = "BGQ" ; then - with_shared_memory=mmap - MPID_DEFAULT_CROSS_FILE=${master_top_srcdir}/src/mpid/pamid/cross/bgq8 - MPID_DEFAULT_PM=no -fi - -MPID_MAX_THREAD_LEVEL=MPI_THREAD_MULTIPLE - -dnl the PAMID device depends on the common NBC scheduler SHM code -build_mpid_common_sched=yes -build_mpid_common_datatype=yes -build_mpid_common_thread=yes -build_mpid_common_shm=yes - - -])dnl end AM_COND_IF(BUILD_PAMID,...) -])dnl end PREREQ -AC_DEFUN([PAC_SUBCFG_BODY_]PAC_SUBCFG_AUTO_SUFFIX,[ -AM_COND_IF([BUILD_PAMID],[ -AC_MSG_NOTICE([RUNNING CONFIGURE FOR PAMI DEVICE]) - - -ASSERT_LEVEL=2 -AC_ARG_WITH(assert-level, - AS_HELP_STRING([--with-assert-level={0 1 2}],[pamid build assert-level (default: 2)]), - [ ASSERT_LEVEL=$withval ]) -AC_SUBST(ASSERT_LEVEL) -AC_DEFINE_UNQUOTED([ASSERT_LEVEL], $ASSERT_LEVEL, [The pamid assert level]) - -dnl -dnl This macro adds the -I to CPPFLAGS and/or the -L to LDFLAGS -dnl -PAC_SET_HEADER_LIB_PATH(pami) - -dnl -dnl Set the pamid platform define. -dnl -PAC_APPEND_FLAG([-D__${pamid_platform}__], [CPPFLAGS]) - -dnl Check for available shared memory functions -PAC_ARG_SHARED_MEMORY -if test "$with_shared_memory" != "mmap" -a "$with_shared_memory" != "sysv"; then - AC_MSG_ERROR([cannot support shared memory: need either sysv shared memory functions or mmap in order to support shared memory]) -fi - -dnl -dnl The default is to enable the use of the recv queue binary search -dnl ... except on BGQ -dnl -enable_queue_binary_search=yes -if test "${pamid_platform}" = "BGQ" ; then - enable_queue_binary_search=no -fi - -AM_CONDITIONAL([QUEUE_BINARY_SEARCH_SUPPORT],[test "$enable_queue_binary_search" = "yes"]) - -dnl -dnl This configure option allows "sandbox" bgq system software to be used. -dnl -AC_ARG_WITH(bgq-install-dir, - AS_HELP_STRING([--with-bgq-install-dir=PATH],[specify path where bgq system software can be found; - may also be specified with the 'BGQ_INSTALL_DIR' - environment variable]), - [ BGQ_INSTALL_DIR=$withval ]) -AC_SUBST(BGQ_INSTALL_DIR) - -dnl -dnl Add bgq-specific build options. -dnl -if test "${pamid_platform}" = "BGQ" ; then - - AC_MSG_CHECKING([for BGQ system software directory]) - - dnl - dnl Specify the default bgq system software paths - dnl - bgq_driver_search_path="${BGQ_INSTALL_DIR} " - for bgq_version in `echo 4 3 2 1`; do - for bgq_release in `echo 4 3 2 1`; do - for bgq_mod in `echo 4 3 2 1 0`; do - bgq_driver_search_path+="/bgsys/drivers/V${bgq_version}R${bgq_release}M${bgq_mod}/ppc64 " - done - done - done - - dnl Look for a bgq driver to use. - for bgq_driver in $bgq_driver_search_path ; do - if test -d ${bgq_driver}/spi/include ; then - - found_bgq_driver=yes - - PAC_APPEND_FLAG([-I${bgq_driver}], [CPPFLAGS]) - PAC_APPEND_FLAG([-I${bgq_driver}/spi/include/kernel/cnk], [CPPFLAGS]) - PAC_APPEND_FLAG([-L${bgq_driver}/spi/lib], [LDFLAGS]) - PAC_APPEND_FLAG([-L${bgq_driver}/spi/lib], [WRAPPER_LDFLAGS]) - - dnl If the '--with-pami' and the '--with-pami-include' configure options - dnl were NOT specified then test for a V1R2M1+ comm include directory. - dnl - AS_IF([test "x${with_pami_include}" = "x" ], - [AS_IF([test "x${with_pami}" = "x" ], - [AS_IF([test -d ${bgq_driver}/comm/include ], - [PAC_APPEND_FLAG([-I${bgq_driver}/comm/include], [CPPFLAGS])], - [PAC_APPEND_FLAG([-I${bgq_driver}/comm/sys/include],[CPPFLAGS])])])]) - - dnl If the '--with-pami' and the '--with-pami-lib' configure options were - dnl NOT specified then - dnl - dnl if a custom pami library name was NOT specified then test for a - dnl V1R2M1+ pami lib in a V1R2M1+ comm lib directory then test for a - dnl pre-V1R2M1 pami lib in a pre-V1R2M1 comm lib directory; otherwise - dnl - dnl if a custom pami library name WAS specified then test for a custom - dnl pami lib in a V1R2M1+ comm lib directory then test for a custom - dnl pami lib in a pre-V1R2M1 comm lib directory - dnl - AS_IF([test "x${with_pami_lib}" = "x" ], - [AS_IF([test "x${with_pami}" = "x" ], - [AS_IF([test "x${original_PAMILIBNAME}" = "x" ], - [AS_IF([test -f ${bgq_driver}/comm/lib/libpami-gcc.a ], - [PAMILIBNAME=pami-gcc - PAC_APPEND_FLAG([-L${bgq_driver}/comm/lib], [LDFLAGS]) - PAC_APPEND_FLAG([-L${bgq_driver}/comm/lib], [WRAPPER_LDFLAGS])], - [AS_IF([test -f ${bgq_driver}/comm/sys/lib/libpami.a ], - [PAC_APPEND_FLAG([-L${bgq_driver}/comm/sys/lib],[LDFLAGS]) - PAC_APPEND_FLAG([-L${bgq_driver}/comm/sys/lib],[WRAPPER_LDFLAGS])])])], - [AS_IF([test -f ${bgq_driver}/comm/lib/lib${PAMILIBNAME}.a ], - [PAC_APPEND_FLAG([-L${bgq_driver}/comm/lib], [LDFLAGS]) - PAC_APPEND_FLAG([-L${bgq_driver}/comm/lib], [WRAPPER_LDFLAGS])], - [AS_IF([test -f ${bgq_driver}/comm/sys/lib/lib${PAMILIBNAME}.a ], - [PAC_APPEND_FLAG([-L${bgq_driver}/comm/sys/lib],[LDFLAGS]) - PAC_APPEND_FLAG([-L${bgq_driver}/comm/sys/lib],[WRAPPER_LDFLAGS])])])])])]) - - break - fi - done - - if test "x${found_bgq_driver}" = "xyes"; then - AC_MSG_RESULT('$bgq_driver') - else - AC_MSG_RESULT('no') - fi - - dnl - dnl The bgq compile requires these libraries. - dnl - PAC_APPEND_FLAG([-lSPI], [LIBS]) - PAC_APPEND_FLAG([-lSPI_cnk], [LIBS]) - PAC_APPEND_FLAG([-lrt], [LIBS]) - PAC_APPEND_FLAG([-lpthread], [LIBS]) - PAC_APPEND_FLAG([-lstdc++], [LIBS]) - - dnl - dnl The wrapper scripts require these libraries. - dnl - PAC_APPEND_FLAG([-l${PAMILIBNAME}], [WRAPPER_LIBS]) - PAC_APPEND_FLAG([-lSPI], [WRAPPER_LIBS]) - PAC_APPEND_FLAG([-lSPI_cnk], [WRAPPER_LIBS]) - PAC_APPEND_FLAG([-lrt], [WRAPPER_LIBS]) - PAC_APPEND_FLAG([-lpthread], [WRAPPER_LIBS]) - PAC_APPEND_FLAG([-lstdc++], [WRAPPER_LIBS]) - - - AC_SEARCH_LIBS([PAMI_Send], [${PAMILIBNAME} pami-gcc]) - - - dnl For some reason, on bgq, libtool will incorrectly attempt a static link - dnl of libstdc++.so unless this '-all-static' option is used. This seems to - dnl be a problem specific to libstdc++. - dnl - dnl Only the 'cpi', 'mpivars', and 'mpichversion' executables have this problem. - MPID_LIBTOOL_STATIC_FLAG="-all-static" - - dnl Another bgq special case. The default linker behavior is to use static versions - dnl of libraries. This makes supporting interlibrary dependencies difficult. Just - dnl disable them to make our lives easier. - if test "$INTERLIB_DEPS" = "yes"; then - INTERLIB_DEPS="no" - fi -fi - -if test "${pamid_platform}" = "PE" ; then - MPID_MAX_ERROR_STRING=512 -fi - -PAC_APPEND_FLAG([-I${master_top_srcdir}/src/include], [CPPFLAGS]) -PAC_APPEND_FLAG([-I${master_top_srcdir}/src/util/wrappers], [CPPFLAGS]) -PAC_APPEND_FLAG([-I${master_top_srcdir}/src/mpid/pamid/include], [CPPFLAGS]) -PAC_APPEND_FLAG([-I${master_top_srcdir}/src/mpid/common/datatype], [CPPFLAGS]) -PAC_APPEND_FLAG([-I${master_top_srcdir}/src/mpid/common/locks], [CPPFLAGS]) -PAC_APPEND_FLAG([-I${master_top_srcdir}/src/mpid/common/thread], [CPPFLAGS]) -PAC_APPEND_FLAG([-I${master_top_srcdir}/src/mpid/common/sched], [CPPFLAGS]) - -dnl -dnl Check for PAMI_IN_PLACE -dnl -AC_MSG_CHECKING([for PAMI_IN_PLACE support]) -have_pami_in_place=0 -AC_COMPILE_IFELSE( - [AC_LANG_PROGRAM([[#include "pami.h"]], - [[void * foo = PAMI_IN_PLACE;]])], - have_pami_in_place=1 -) -if test "$have_pami_in_place" != "0"; then - AC_DEFINE(HAVE_PAMI_IN_PLACE,1,[Define if PAMI_IN_PLACE is defined in pami.h]) - AC_MSG_RESULT('yes') -else - AC_DEFINE(PAMI_IN_PLACE,((void *) -1L),[Define if PAMI_IN_PLACE is not defined in pami.h]) - AC_MSG_RESULT('no') -fi - -dnl -dnl Check for PAMI_CLIENT_NONCONTIG -dnl -AC_MSG_CHECKING([for PAMI_CLIENT_NONCONTIG support]) -have_pami_client_noncontig=0 -AC_COMPILE_IFELSE( - [AC_LANG_PROGRAM([[#include "pami.h"]], - [[int foo = PAMI_CLIENT_NONCONTIG;]])], - have_pami_client_noncontig=1 -) -if test "$have_pami_client_noncontig" != "0"; then - AC_DEFINE(HAVE_PAMI_CLIENT_NONCONTIG,1,[Define if PAMI_CLIENT_NONCONTIG is defined in pami.h]) - AC_MSG_RESULT('yes') -else - AC_MSG_RESULT('no') -fi - -dnl -dnl Check for PAMI_CLIENT_MEMORY_OPTIMIZE -dnl -AC_MSG_CHECKING([for PAMI_CLIENT_MEMORY_OPTIMIZE support]) -have_pami_client_memory_optimize=0 -AC_COMPILE_IFELSE( - [AC_LANG_PROGRAM([[#include "pami.h"]], - [[int foo = PAMI_CLIENT_MEMORY_OPTIMIZE;]])], - have_pami_client_memory_optimize=1 -) -if test "$have_pami_client_memory_optimize" != "0"; then - AC_DEFINE(HAVE_PAMI_CLIENT_MEMORY_OPTIMIZE,1,[Define if PAMI_CLIENT_MEMORY_OPTIMIZE is defined in pami.h]) - AC_MSG_RESULT('yes') -else - AC_MSG_RESULT('no') -fi - -dnl -dnl Check for PAMI_GEOMETRY_NONCONTIG -dnl -AC_MSG_CHECKING([for PAMI_GEOMETRY_NONCONTIG support]) -have_pami_geometry_noncontig=0 -AC_COMPILE_IFELSE( - [AC_LANG_PROGRAM([[#include "pami.h"]], - [[int foo = PAMI_GEOMETRY_NONCONTIG;]])], - have_pami_geometry_noncontig=1 -) -if test "$have_pami_geometry_noncontig" != "0"; then - AC_DEFINE(HAVE_PAMI_GEOMETRY_NONCONTIG,1,[Define if PAMI_GEOMETRY_NONCONTIG is defined in pami.h]) - AC_MSG_RESULT('yes') -else - AC_MSG_RESULT('no') -fi - -dnl -dnl Check for PAMI_GEOMETRY_MEMORY_OPTIMIZE -dnl -AC_MSG_CHECKING([for PAMI_GEOMETRY_MEMORY_OPTIMIZE support]) -have_pami_geometry_memory_optimize=0 -AC_COMPILE_IFELSE( - [AC_LANG_PROGRAM([[#include "pami.h"]], - [[int foo = PAMI_GEOMETRY_MEMORY_OPTIMIZE;]])], - have_pami_geometry_memory_optimize=1 -) -if test "$have_pami_geometry_memory_optimize" != "0"; then - AC_DEFINE(HAVE_PAMI_GEOMETRY_MEMORY_OPTIMIZE,1,[Define if PAMI_GEOMETRY_MEMORY_OPTIMIZE is defined in pami.h]) - AC_MSG_RESULT('yes') -else - AC_MSG_RESULT('no') -fi - - - - - - - - - - -])dnl end AM_COND_IF(BUILD_PAMID,...) -])dnl end _BODY - -[#] end of __file__ diff --git a/src/mpl/configure.ac b/src/mpl/configure.ac index e07f2db..cf34469 100644 --- a/src/mpl/configure.ac +++ b/src/mpl/configure.ac @@ -234,7 +234,6 @@ dnl Unix gettimeofday (one of two versions) dnl dnl Also available are various hardware time stamps dnl Linux-x86 cycle counter -dnl BG/Q PAMI-based timer dnl dnl We also allow --enable-timer-type=name to select a timer type AC_ARG_ENABLE(timer-type, diff --git a/src/pm/hydra/mpichprereq b/src/pm/hydra/mpichprereq index bfbf778..d6aa9f6 100755 --- a/src/pm/hydra/mpichprereq +++ b/src/pm/hydra/mpichprereq @@ -10,7 +10,7 @@ if test -z "$PM_REQUIRES_PMI" ; then else PM_REQUIRES_PMI=simple fi -elif test "$PM_REQUIRES_PMI" != "simple" -a "$PM_REQUIRES_PMI" != "pmi2/simple" -a "$PM_REQUIRES_PMI" != "pmi2/poe"; then +elif test "$PM_REQUIRES_PMI" != "simple" -a "$PM_REQUIRES_PMI" != "pmi2/simple" ; then echo "hydra requires the \"simple\" or \"pmi2\" PMI implementation; \"$PM_REQUIRES_PMI\" has already been selected" exit 1 fi diff --git a/src/pmi/pmi2/Makefile.mk b/src/pmi/pmi2/Makefile.mk index fd3d7f4..460f67e 100644 --- a/src/pmi/pmi2/Makefile.mk +++ b/src/pmi/pmi2/Makefile.mk @@ -5,5 +5,4 @@ ## See COPYRIGHT in top-level directory. ## -include $(top_srcdir)/src/pmi/pmi2/poe/Makefile.mk include $(top_srcdir)/src/pmi/pmi2/simple/Makefile.mk diff --git a/src/pmi/pmi2/poe/Makefile.mk b/src/pmi/pmi2/poe/Makefile.mk deleted file mode 100644 index 26707a9..0000000 --- a/src/pmi/pmi2/poe/Makefile.mk +++ /dev/null @@ -1,15 +0,0 @@ -## -*- Mode: Makefile; -*- -## vim: set ft=automake : -## -## (C) 2011 by Argonne National Laboratory. -## See COPYRIGHT in top-level directory. -## - -if BUILD_PMI_PMI2_POE - -mpi_core_sources += \ - src/pmi/pmi2/poe/poe2pmi.c - -AM_CPPFLAGS += -I$(top_srcdir)/src/pmi/pmi2/poe - -endif BUILD_PMI_PMI2_POE diff --git a/src/pmi/pmi2/poe/poe2pmi.c b/src/pmi/pmi2/poe/poe2pmi.c deleted file mode 100644 index 28cc16f..0000000 --- a/src/pmi/pmi2/poe/poe2pmi.c +++ /dev/null @@ -1,449 +0,0 @@ -/* -*- Mode: C; c-basic-offset:4 ; -*- */ -/* - * (C) 2007 by Argonne National Laboratory. - * See COPYRIGHT in top-level directory. - */ -#include <dlfcn.h> -#include "mpichconf.h" -#include "pmi2.h" -#include "mpiimpl.h" - -#include <stdio.h> -#ifdef HAVE_UNISTD_H -#include <unistd.h> -#endif -#ifdef HAVE_STDLIB_H -#include <stdlib.h> -#endif -#ifdef HAVE_STRING_H -#include <string.h> -#endif -#ifdef HAVE_STRINGS_H -#include <strings.h> -#endif -#if defined(HAVE_SYS_SOCKET_H) -#include <sys/socket.h> -#endif - -#ifdef USE_PMI_PORT -#ifndef MAXHOSTNAME -#define MAXHOSTNAME 256 -#endif -#endif - -#define PMII_EXIT_CODE -1 - -#define PMI_VERSION 2 -#define PMI_SUBVERSION 0 - -#define MAX_INT_STR_LEN 11 /* number of digits in MAX_UINT + 1 */ - -struct worldExitReq { - pami_work_t work; - int world_id; -}; - -int (*mp_world_exiting_handler)(int) = NULL; - -typedef enum { PMI2_UNINITIALIZED = 0, NORMAL_INIT_WITH_PM = 1 } PMI2State; -static PMI2State PMI2_initialized = PMI2_UNINITIALIZED; - -static int PMI2_debug = 0; -static int PMI2_fd = -1; -static int PMI2_size = 1; -static int PMI2_rank = 0; - -static int PMI2_debug_init = 0; /* Set this to true to debug the init */ - -int PMI2_pmiverbose = 0; /* Set this to true to print PMI debugging info */ - -extern MPIDI_PG_t *pg_world; -extern int world_rank; -extern int world_size; - -#ifdef MPICH_IS_THREADED -static MPID_Thread_mutex_t mutex; -static int blocked = FALSE; -static MPID_Thread_cond_t cond; -#endif - -extern int mpidi_finalized; -int _mpi_world_exiting_handler(int); - -void *poeptr = NULL; - -/* ------------------------------------------------------------------------- */ -/* PMI API Routines */ -/* ------------------------------------------------------------------------- */ -int PMI2_Init(int *spawned, int *size, int *rank, int *appnum) -{ - int pmi2_errno = PMI2_SUCCESS; - char *p; - char *jobid; - char *pmiid; - int ret; - - int (*pmi2_init)(int*, int*, int *, int*); - char *poelibname; - int poeflags; -#ifndef __AIX__ - poelibname = "libpoe.so"; - poeflags = RTLD_NOW|RTLD_GLOBAL; -#else - poeflags = RTLD_NOW|RTLD_GLOBAL|RTLD_MEMBER; - if(sizeof(void *) == 8) - poelibname = "libpoe_r.a(poe64_r.o)"; - else - poelibname = "libpoe_r.a(poe_r.o)"; -#endif - poeptr = dlopen(poelibname, poeflags); - if (poeptr == NULL) { - TRACE_ERR("failed to open %s error=%s\n", poelibname, dlerror()); - } - - pmi2_init = (int (*)())dlsym(poeptr, "PMI2_Init"); - if (pmi2_init == NULL) { - TRACE_ERR("failed to dlsym PMI2_Init\n"); - } - - ret = (*pmi2_init)(spawned, size, rank, appnum); - /*mp_world_exiting_handler = &(_mpi_world_exiting_handler); */ - return ret; -} - -int PMI2_Finalize(void) -{ - int pmi2_errno = PMI2_SUCCESS; - int rc; - const char *errmsg; - - int (*pmi2_finalize)(void); - - pmi2_finalize = (int (*)())dlsym(poeptr, "PMI2_Finalize"); - if (pmi2_finalize == NULL) { - TRACE_ERR("failed to dlsym PMI2_Finalize\n"); - } - - return (*pmi2_finalize)(); - -} - -int PMI2_Initialized(void) -{ - /* Turn this into a logical value (1 or 0) . This allows us - to use PMI2_initialized to distinguish between initialized with - an PMI service (e.g., via mpiexec) and the singleton init, - which has no PMI service */ - return PMI2_initialized != 0; -} - -int PMI2_Abort( int flag, const char msg[] ) -{ - int (*pmi2_abort)(int, const char*); - - pmi2_abort = (int (*)())dlsym(poeptr, "PMI2_Abort"); - if (pmi2_abort == NULL) { - TRACE_ERR("failed to dlsym pmi2_abort\n"); - } - - return (*pmi2_abort)(flag, msg); -} - -int PMI2_Job_Spawn(int count, const char * cmds[], - int argcs[], const char ** argvs[], - const int maxprocs[], - const int info_keyval_sizes[], - const struct MPIR_Info *info_keyval_vectors[], - int preput_keyval_size, - const struct MPIR_Info *preput_keyval_vector[], - char jobId[], int jobIdSize, - int errors[]) -{ - int i,rc,spawncnt,total_num_processes,num_errcodes_found; - int found; - const char *jid; - int jidlen; - char *lead, *lag; - int spawn_rc; - const char *errmsg = NULL; - int pmi2_errno = 0; - - int (*pmi2_job_spawn)(int , const char * [], int [], const char ** [],const int [],const int [],const struct MPIR_Info *[],int ,const struct MPIR_Info *[],char jobId[],int ,int []); - - pmi2_job_spawn = (int (*)())dlsym(poeptr, "PMI2_Job_Spawn"); - if (pmi2_job_spawn == NULL) { - TRACE_ERR("failed to dlsym pmi2_job_spawn\n"); - } - - return (*pmi2_job_spawn)(count, cmds, argcs, argvs, maxprocs, - info_keyval_sizes, info_keyval_vectors, - preput_keyval_size, preput_keyval_vector, - jobId, jobIdSize, errors); - -} - -int PMI2_Job_GetId(char jobid[], int jobid_size) -{ - int pmi2_errno = PMI2_SUCCESS; - int found; - const char *jid; - int jidlen; - int rc; - const char *errmsg; - - int (*pmi2_job_getid)(char*, int); - - pmi2_job_getid = (int (*)())dlsym(poeptr, "PMI2_Job_GetId"); - if (pmi2_job_getid == NULL) { - TRACE_ERR("failed to dlsym pmi2_job_getid\n"); - } - - return (*pmi2_job_getid)(jobid, jobid_size); -} - - -int PMI2_KVS_Put(const char key[], const char value[]) -{ - int pmi2_errno = PMI2_SUCCESS; - int rc; - const char *errmsg; - - int (*pmi2_kvs_put)(const char*, const char*); - - pmi2_kvs_put = (int (*)())dlsym(poeptr, "PMI2_KVS_Put"); - if (pmi2_kvs_put == NULL) { - TRACE_ERR("failed to dlsym pmi2_kvs_put\n"); - } - - return (*pmi2_kvs_put)(key, value); -} - -int PMI2_KVS_Fence(void) -{ - int pmi2_errno = PMI2_SUCCESS; - int rc; - const char *errmsg; - - int (*pmi2_kvs_fence)(void); - - pmi2_kvs_fence = (int (*)())dlsym(poeptr, "PMI2_KVS_Fence"); - if (pmi2_kvs_fence == NULL) { - TRACE_ERR("failed to dlsym pmi2_kvs_fence\n"); - } - - return (*pmi2_kvs_fence)(); -} - -int PMI2_KVS_Get(const char *jobid, int src_pmi_id, const char key[], char value [], int maxValue, int *valLen) -{ - int pmi2_errno = PMI2_SUCCESS; - int found, keyfound; - const char *kvsvalue; - int kvsvallen; - int ret; - int rc; - char src_pmi_id_str[256]; - const char *errmsg; - - int (*pmi2_kvs_get)(const char*, int, const char *, char *, int, int*); - - pmi2_kvs_get = (int (*)())dlsym(poeptr, "PMI2_KVS_Get"); - if (pmi2_kvs_get == NULL) { - TRACE_ERR("failed to dlsym pmi2_kvs_get\n"); - } - - return (*pmi2_kvs_get)(jobid, src_pmi_id, key, value, maxValue, valLen); -} - - -int PMI2_Info_GetJobAttr(const char name[], char value[], int valuelen, int *flag) -{ - int pmi2_errno = PMI2_SUCCESS; - int found; - const char *kvsvalue; - int kvsvallen; - int rc; - const char *errmsg; - - int (*pmi2_info_getjobattr)(const char*, char *, int, int*); - - pmi2_info_getjobattr = (int (*)())dlsym(poeptr, "PMI2_Info_GetJobAttr"); - if (pmi2_info_getjobattr == NULL) { - TRACE_ERR("failed to dlsym pmi2_info_getjobattr\n"); - } - - return (*pmi2_info_getjobattr)(name, value, valuelen, flag); -} - - -/** - * This is the mpi level of callback that get invoked when a task get notified - * of a world's exiting - */ -int _mpi_world_exiting_handler(int world_id) -{ - /* check the reference count associated with that remote world - if the reference count is zero, the task will call LAPI_Purge_totask on - all tasks in that world,reset MPCI. It would also remove the world - structure corresponding to that world ID - if the reference count is not zero, it should call STOPALL - */ - int rc,ref_count = -1; - int *taskid_list = NULL; - int i; - int my_state=FALSE,reduce_state=FALSE; - char world_id_str[32]; - int mpi_errno = MPI_SUCCESS; - pami_endpoint_t dest; -/* struct worldExitReq *req = (struct worldExitReq *)cookie; */ - MPIR_Comm *comm = MPIR_Process.comm_world; - - MPID_THREAD_CS_ENTER(GLOBAL, MPIR_THREAD_GLOBAL_ALLFUNC_MUTEX); - ref_count = MPIDI_get_refcnt_of_world(world_id); - TRACE_ERR("_mpi_world_exiting_handler: invoked for world %d exiting ref_count=%d my comm_word_size=%d\n", world_id, ref_count, world_size); - if(ref_count == 0) { - taskid_list = MPIDI_get_taskids_in_world_id(world_id); - if(taskid_list != NULL) { - for(i=0;taskid_list[i]!=-1;i++) { - PAMI_Endpoint_create(MPIDI_Client, taskid_list[i], 0, &dest); - MPIDI_OpState_reset(taskid_list[i]); - MPIDI_IpState_reset(taskid_list[i]); - TRACE_ERR("PAMI_Purge on taskid_list[%d]=%d\n", i,taskid_list[i]); - PAMI_Purge(MPIDI_Context[0], &dest, 1); - } - MPIDI_delete_conn_record(world_id); - } - rc = -1; - } - my_state = TRUE; - - rc = _mpi_reduce_for_dyntask(&my_state, &reduce_state); - if(rc) return rc; - - TRACE_ERR("_mpi_world_exiting_handler: Out of _mpi_reduce_for_dyntask for exiting world %d reduce_state=%d\n",world_id, reduce_state); - - MPID_THREAD_CS_EXIT(GLOBAL, MPIR_THREAD_GLOBAL_ALLFUNC_MUTEX); - if(comm->rank == 0) { - MPL_snprintf(world_id_str, sizeof(world_id_str), "%d", world_id); - PMI2_Abort(0, world_id_str); - if((reduce_state != world_size)) { - TRACE_ERR("root is exiting with error\n"); - exit(-1); - } - TRACE_ERR("_mpi_world_exiting_handler: Root finished sending SSM_WORLD_EXITING to POE for exiting world %d\n",world_id); - } - - if(ref_count != 0) { - TRACE_ERR("STOPALL is sent by task %d\n", PAMIX_Client_query(MPIDI_Client, PAMI_CLIENT_TASK_ID ).value.intval); - PMI2_Abort(1, "STOPALL should be sent"); - rc = -2; - } - -/* if(cookie) MPL_free(cookie);*/ - return PAMI_SUCCESS; -} - - -int getchildren(int iam, double alpha,int gsize, int *children, - int *blocks, int *numchildren, int *parent) -{ - int fakeme=iam,i; - int p=gsize,pbig,bflag=0,blocks_from_children=0; - - *numchildren=0; - - if( blocks != NULL ) - bflag=1; - - while( p > 1 ) { - - pbig = MPL_MAX(1,MPL_MIN((int) (alpha*(double)p), p-1)); - - if ( fakeme == 0 ) { - - (children)[*numchildren] = (iam+pbig+gsize)%gsize; - if(bflag) - (blocks)[*numchildren] = p -pbig; - - *numchildren +=1; - } - if ( fakeme == pbig ) { - *parent = (iam-pbig+gsize)%gsize; - if(bflag) - blocks_from_children = p - pbig; - } - if( pbig > fakeme) { - p = pbig; - } else { - p -=pbig; - fakeme -=pbig; - } - } - if(bflag) - (blocks)[*numchildren] = blocks_from_children; -} - -int _mpi_reduce_for_dyntask(int *sendbuf, int *recvbuf) -{ - int *children, gid, child_rank, parent_rank, rc; - int numchildren, parent=0, i, result=0,tag, remaining_child_count; - MPIR_Comm *comm_ptr; - int mpi_errno; - MPIR_Errflag_t errflag = MPIR_ERR_NONE; - - int TASKS= world_size; - children = MPL_malloc(TASKS*sizeof(int)); - - comm_ptr = MPIR_Process.comm_world; - - if(pg_world && pg_world->id) - tag = (-1) * (atoi(pg_world->id)); - else { - TRACE_ERR("pg_world hasn't been created, should skip the rest of the handler and return\n"); - return -1; - } - - result = *sendbuf; - - getchildren(world_rank, 0.5, TASKS, children, NULL, &numchildren, &parent); - - TRACE_ERR("_mpi_reduce_for_dyntask - numchildren=%d parent=%d world_rank=%d\n", numchildren, parent, world_rank); - for(i=numchildren-1;i>=0;i--) - { - remaining_child_count = i; - child_rank = (children[i])% TASKS; - TRACE_ERR("_mpi_reduce_for_dyntask - recv from child_rank%d child_taskid=%d\n", child_rank, pg_world->vct[child_rank].taskid); - mpi_errno = MPIC_Recv(recvbuf, sizeof(int),MPI_BYTE, child_rank, tag, comm_ptr, MPI_STATUS_IGNORE, &errflag); - TRACE_ERR("_mpi_reduce_for_dyntask - recv DONE from child_rank%d child_taskid=%d\n", child_rank, pg_world->vct[child_rank].taskid); - - if(world_rank != parent) - { - if(remaining_child_count == 0) { - parent_rank = (parent) % TASKS; - result += *recvbuf; - TRACE_ERR("_mpi_reduce_for_dyntask - send to parent_rank=%d parent taskid=%d \n", parent_rank, pg_world->vct[parent_rank].taskid); - MPIC_Send(&result, sizeof(int), MPI_BYTE, parent_rank, tag, comm_ptr, &errflag); - } - else - { - result += *recvbuf; - } - } - if(world_rank == 0) - { - result += *recvbuf; - } - } - - if(world_rank != parent && numchildren == 0) { - parent_rank = (parent) % TASKS; - TRACE_ERR("_mpi_reduce_for_dyntask - send to parent_rank=%d parent_task_id=%d\n", parent_rank, pg_world->vct[parent_rank].taskid); - MPIC_Send(sendbuf, sizeof(int), MPI_BYTE, parent_rank, tag, comm_ptr, &errflag); - } - - if(world_rank == 0) { - *recvbuf = result; - } - MPL_free(children); - return 0; -} diff --git a/src/pmi/pmi2/poe/subconfigure.m4 b/src/pmi/pmi2/poe/subconfigure.m4 deleted file mode 100644 index 5e2ce7d..0000000 --- a/src/pmi/pmi2/poe/subconfigure.m4 +++ /dev/null @@ -1,24 +0,0 @@ -[#] start of __file__ -dnl MPICH_SUBCFG_AFTER=src/pmi - -AC_DEFUN([PAC_SUBCFG_PREREQ_]PAC_SUBCFG_AUTO_SUFFIX,[ -]) - -AC_DEFUN([PAC_SUBCFG_BODY_]PAC_SUBCFG_AUTO_SUFFIX,[ - -AM_CONDITIONAL([BUILD_PMI_PMI2_POE],[test "x$pmi_name" = "xpmi2/poe"]) - -AM_COND_IF([BUILD_PMI_PMI2_POE],[ -if test "$enable_pmiport" != "no" ; then - enable_pmiport=yes -fi - -dnl causes USE_PMI2_API to be AC_DEFINE'ed by the top-level configure.ac -USE_PMI2_API=yes - -PAC_C_GNU_ATTRIBUTE -])dnl end COND_IF - -])dnl end BODY macro - -[#] end of __file__ diff --git a/src/pmi/pmi2/subconfigure.m4 b/src/pmi/pmi2/subconfigure.m4 index d1dc441..c516051 100644 --- a/src/pmi/pmi2/subconfigure.m4 +++ b/src/pmi/pmi2/subconfigure.m4 @@ -16,7 +16,6 @@ fi dnl causes USE_PMI2_API to be AC_DEFINE'ed by the top-level configure.ac USE_PMI2_API=yes -# common ARG_ENABLE, shared by "simple" and "poe" AC_ARG_ENABLE(pmiport, [--enable-pmiport - Allow PMI interface to use a host-port pair to contact for PMI services],,enable_pmiport=default) ----------------------------------------------------------------------- Summary of changes: CHANGES | 22 + README.vin | 27 - maint/version.m4 | 4 +- src/mpid/Makefile.mk | 1 - src/mpid/pamid/.gitattributes | 15 - src/mpid/pamid/.gitignore | 3 - src/mpid/pamid/Makefile.mk | 43 - src/mpid/pamid/cross/bgq8 | 27 - src/mpid/pamid/cross/pe4 | 27 - src/mpid/pamid/cross/pe4-aix | 28 - src/mpid/pamid/cross/pe8 | 27 - src/mpid/pamid/cross/pe8-aix | 28 - src/mpid/pamid/include/.gitignore | 1 - src/mpid/pamid/include/mpidi_constants.h | 128 -- src/mpid/pamid/include/mpidi_datatypes.h | 658 ------- src/mpid/pamid/include/mpidi_externs.h | 43 - src/mpid/pamid/include/mpidi_hooks.h | 66 - src/mpid/pamid/include/mpidi_macros.h | 217 -- src/mpid/pamid/include/mpidi_mutex.h | 374 ---- src/mpid/pamid/include/mpidi_platform.h | 196 -- src/mpid/pamid/include/mpidi_prototypes.h | 460 ----- src/mpid/pamid/include/mpidi_thread.h | 184 -- src/mpid/pamid/include/mpidi_trace.h | 266 --- src/mpid/pamid/include/mpidi_util.h | 187 -- src/mpid/pamid/include/mpidimpl.h | 197 -- src/mpid/pamid/include/mpidpost.h | 104 - src/mpid/pamid/include/mpidpre.h | 94 - src/mpid/pamid/include/mpix.h | 298 --- src/mpid/pamid/include/pamix.h | 147 -- src/mpid/pamid/src/Makefile.mk | 73 - src/mpid/pamid/src/coll/Makefile.mk | 47 - src/mpid/pamid/src/coll/allgather/Makefile.mk | 30 - .../pamid/src/coll/allgather/mpido_allgather.c | 820 -------- .../pamid/src/coll/allgather/mpido_iallgather.c | 73 - src/mpid/pamid/src/coll/allgatherv/Makefile.mk | 30 - .../pamid/src/coll/allgatherv/mpido_allgatherv.c | 921 --------- .../pamid/src/coll/allgatherv/mpido_iallgatherv.c | 77 - src/mpid/pamid/src/coll/allreduce/Makefile.mk | 30 - .../pamid/src/coll/allreduce/mpido_allreduce.c | 562 ------ .../pamid/src/coll/allreduce/mpido_iallreduce.c | 73 - src/mpid/pamid/src/coll/alltoall/Makefile.mk | 30 - src/mpid/pamid/src/coll/alltoall/mpido_alltoall.c | 449 ----- src/mpid/pamid/src/coll/alltoall/mpido_ialltoall.c | 76 - src/mpid/pamid/src/coll/alltoallv/Makefile.mk | 30 - .../pamid/src/coll/alltoallv/mpido_alltoallv.c | 734 ------- .../pamid/src/coll/alltoallv/mpido_ialltoallv.c | 79 - src/mpid/pamid/src/coll/alltoallw/Makefile.mk | 28 - .../pamid/src/coll/alltoallw/mpido_ialltoallw.c | 79 - src/mpid/pamid/src/coll/barrier/Makefile.mk | 30 - src/mpid/pamid/src/coll/barrier/mpido_barrier.c | 146 -- src/mpid/pamid/src/coll/barrier/mpido_ibarrier.c | 62 - src/mpid/pamid/src/coll/bcast/Makefile.mk | 30 - src/mpid/pamid/src/coll/bcast/mpido_bcast.c | 394 ---- src/mpid/pamid/src/coll/bcast/mpido_ibcast.c | 67 - src/mpid/pamid/src/coll/coll_utils.c | 405 ---- src/mpid/pamid/src/coll/exscan/Makefile.mk | 28 - src/mpid/pamid/src/coll/exscan/mpido_iexscan.c | 65 - src/mpid/pamid/src/coll/gather/Makefile.mk | 30 - src/mpid/pamid/src/coll/gather/mpido_gather.c | 642 ------ src/mpid/pamid/src/coll/gather/mpido_igather.c | 72 - src/mpid/pamid/src/coll/gatherv/Makefile.mk | 30 - src/mpid/pamid/src/coll/gatherv/mpido_gatherv.c | 573 ------ src/mpid/pamid/src/coll/gatherv/mpido_igatherv.c | 78 - src/mpid/pamid/src/coll/ired_scat/Makefile.mk | 28 - .../pamid/src/coll/ired_scat/mpido_ired_scat.c | 72 - .../pamid/src/coll/ired_scat_block/Makefile.mk | 28 - .../coll/ired_scat_block/mpido_ired_scat_block.c | 72 - src/mpid/pamid/src/coll/red_scat/Makefile.mk | 29 - src/mpid/pamid/src/coll/red_scat/mpido_red_scat.c | 185 -- src/mpid/pamid/src/coll/reduce/Makefile.mk | 30 - src/mpid/pamid/src/coll/reduce/mpido_ireduce.c | 72 - src/mpid/pamid/src/coll/reduce/mpido_reduce.c | 422 ---- src/mpid/pamid/src/coll/scan/Makefile.mk | 29 - src/mpid/pamid/src/coll/scan/mpido_scan.c | 452 ----- src/mpid/pamid/src/coll/scatter/Makefile.mk | 30 - src/mpid/pamid/src/coll/scatter/mpido_iscatter.c | 74 - src/mpid/pamid/src/coll/scatter/mpido_scatter.c | 591 ------ src/mpid/pamid/src/coll/scatterv/Makefile.mk | 30 - src/mpid/pamid/src/coll/scatterv/mpido_iscatterv.c | 73 - src/mpid/pamid/src/coll/scatterv/mpido_scatterv.c | 812 -------- src/mpid/pamid/src/comm/Makefile.mk | 31 - src/mpid/pamid/src/comm/mpid_comm.c | 432 ---- src/mpid/pamid/src/comm/mpid_optcolls.c | 1047 ---------- src/mpid/pamid/src/comm/mpid_selectcolls.c | 746 ------- src/mpid/pamid/src/dyntask/Makefile.mk | 32 - src/mpid/pamid/src/dyntask/mpid_comm_disconnect.c | 541 ----- .../pamid/src/dyntask/mpid_comm_spawn_multiple.c | 414 ---- src/mpid/pamid/src/dyntask/mpid_port.c | 317 --- src/mpid/pamid/src/dyntask/mpidi_pg.c | 1063 ---------- src/mpid/pamid/src/dyntask/mpidi_port.c | 1766 ----------------- src/mpid/pamid/src/misc/Makefile.mk | 33 - src/mpid/pamid/src/misc/mpid_abort.c | 103 - src/mpid/pamid/src/misc/mpid_get_universe_size.c | 96 - src/mpid/pamid/src/misc/mpid_getpname.c | 46 - src/mpid/pamid/src/misc/mpid_mem.c | 42 - src/mpid/pamid/src/misc/mpid_unimpl.c | 112 -- src/mpid/pamid/src/mpid_aint.c | 61 - src/mpid/pamid/src/mpid_buffer.c | 250 --- src/mpid/pamid/src/mpid_finalize.c | 149 -- src/mpid/pamid/src/mpid_improbe.c | 49 - src/mpid/pamid/src/mpid_imrecv.c | 91 - src/mpid/pamid/src/mpid_init.c | 1703 ---------------- src/mpid/pamid/src/mpid_iprobe.c | 47 - src/mpid/pamid/src/mpid_mprobe.c | 41 - src/mpid/pamid/src/mpid_mrecv.c | 34 - src/mpid/pamid/src/mpid_probe.c | 41 - src/mpid/pamid/src/mpid_progress.c | 355 ---- src/mpid/pamid/src/mpid_progress.h | 297 --- src/mpid/pamid/src/mpid_recvq.c | 945 --------- src/mpid/pamid/src/mpid_recvq.h | 425 ---- src/mpid/pamid/src/mpid_recvq_mmap.cpp | 419 ---- src/mpid/pamid/src/mpid_request.c | 88 - src/mpid/pamid/src/mpid_request.h | 354 ---- src/mpid/pamid/src/mpid_time.c | 112 -- src/mpid/pamid/src/mpid_vc.c | 360 ---- src/mpid/pamid/src/mpidi_bufmm.c | 716 ------- src/mpid/pamid/src/mpidi_env.c | 1314 ------------- src/mpid/pamid/src/mpidi_mutex.c | 88 - src/mpid/pamid/src/mpidi_nbc_sched.c | 65 - src/mpid/pamid/src/mpidi_pami_datatype.c | 155 -- src/mpid/pamid/src/mpidi_util.c | 2057 -------------------- src/mpid/pamid/src/mpix/Makefile.mk | 29 - src/mpid/pamid/src/mpix/mpix.c | 1210 ------------ src/mpid/pamid/src/onesided/Makefile.mk | 57 - src/mpid/pamid/src/onesided/mpid_1s.c | 151 -- src/mpid/pamid/src/onesided/mpid_win_accumulate.c | 364 ---- src/mpid/pamid/src/onesided/mpid_win_allocate.c | 99 - .../pamid/src/onesided/mpid_win_allocate_shared.c | 561 ------ src/mpid/pamid/src/onesided/mpid_win_attach.c | 58 - .../pamid/src/onesided/mpid_win_compare_and_swap.c | 164 -- src/mpid/pamid/src/onesided/mpid_win_create.c | 213 -- .../pamid/src/onesided/mpid_win_create_dynamic.c | 90 - src/mpid/pamid/src/onesided/mpid_win_detach.c | 53 - src/mpid/pamid/src/onesided/mpid_win_fence.c | 71 - .../pamid/src/onesided/mpid_win_fetch_and_op.c | 378 ---- src/mpid/pamid/src/onesided/mpid_win_flush.c | 180 -- src/mpid/pamid/src/onesided/mpid_win_free.c | 132 -- src/mpid/pamid/src/onesided/mpid_win_get.c | 434 ----- .../pamid/src/onesided/mpid_win_get_accumulate.c | 586 ------ src/mpid/pamid/src/onesided/mpid_win_get_info.c | 104 - src/mpid/pamid/src/onesided/mpid_win_lock.c | 230 --- src/mpid/pamid/src/onesided/mpid_win_lock_all.c | 206 -- src/mpid/pamid/src/onesided/mpid_win_pscw.c | 270 --- src/mpid/pamid/src/onesided/mpid_win_put.c | 434 ----- src/mpid/pamid/src/onesided/mpid_win_reqops.c | 253 --- src/mpid/pamid/src/onesided/mpid_win_set_info.c | 102 - .../pamid/src/onesided/mpid_win_shared_query.c | 65 - src/mpid/pamid/src/onesided/mpid_win_sync.c | 54 - src/mpid/pamid/src/onesided/mpidi_onesided.h | 389 ---- src/mpid/pamid/src/onesided/mpidi_win_control.c | 149 -- src/mpid/pamid/src/pamix/Makefile.mk | 29 - src/mpid/pamid/src/pamix/pamix.c | 347 ---- src/mpid/pamid/src/pt2pt/Makefile.mk | 50 - src/mpid/pamid/src/pt2pt/mpid_cancel.c | 138 -- src/mpid/pamid/src/pt2pt/mpid_irecv.h | 67 - src/mpid/pamid/src/pt2pt/mpid_isend.h | 129 -- src/mpid/pamid/src/pt2pt/mpid_issend.c | 62 - src/mpid/pamid/src/pt2pt/mpid_recv.c | 60 - src/mpid/pamid/src/pt2pt/mpid_send.h | 67 - src/mpid/pamid/src/pt2pt/mpid_ssend.c | 58 - src/mpid/pamid/src/pt2pt/mpidi_callback_eager.c | 312 --- src/mpid/pamid/src/pt2pt/mpidi_callback_rzv.c | 234 --- src/mpid/pamid/src/pt2pt/mpidi_callback_short.c | 268 --- src/mpid/pamid/src/pt2pt/mpidi_callback_util.c | 195 -- src/mpid/pamid/src/pt2pt/mpidi_control.c | 414 ---- src/mpid/pamid/src/pt2pt/mpidi_done.c | 316 --- src/mpid/pamid/src/pt2pt/mpidi_recv.h | 257 --- src/mpid/pamid/src/pt2pt/mpidi_recvmsg.c | 161 -- src/mpid/pamid/src/pt2pt/mpidi_rendezvous.c | 266 --- src/mpid/pamid/src/pt2pt/mpidi_send.h | 128 -- src/mpid/pamid/src/pt2pt/mpidi_sendmsg.c | 743 ------- src/mpid/pamid/src/pt2pt/persistent/Makefile.mk | 30 - .../pamid/src/pt2pt/persistent/mpid_recv_init.c | 56 - .../pamid/src/pt2pt/persistent/mpid_send_init.c | 148 -- .../pamid/src/pt2pt/persistent/mpid_startall.c | 140 -- src/mpid/pamid/subconfigure.m4 | 351 ---- src/mpl/configure.ac | 1 - src/pm/hydra/mpichprereq | 2 +- src/pmi/pmi2/Makefile.mk | 1 - src/pmi/pmi2/poe/Makefile.mk | 15 - src/pmi/pmi2/poe/poe2pmi.c | 449 ----- src/pmi/pmi2/poe/subconfigure.m4 | 24 - src/pmi/pmi2/subconfigure.m4 | 1 - 183 files changed, 25 insertions(+), 43411 deletions(-) delete mode 100644 src/mpid/pamid/.gitattributes delete mode 100644 src/mpid/pamid/.gitignore delete mode 100644 src/mpid/pamid/Makefile.mk delete mode 100644 src/mpid/pamid/cross/bgq8 delete mode 100644 src/mpid/pamid/cross/pe4 delete mode 100644 src/mpid/pamid/cross/pe4-aix delete mode 100644 src/mpid/pamid/cross/pe8 delete mode 100644 src/mpid/pamid/cross/pe8-aix delete mode 100644 src/mpid/pamid/include/.gitignore delete mode 100644 src/mpid/pamid/include/mpidi_constants.h delete mode 100644 src/mpid/pamid/include/mpidi_datatypes.h delete mode 100644 src/mpid/pamid/include/mpidi_externs.h delete mode 100644 src/mpid/pamid/include/mpidi_hooks.h delete mode 100644 src/mpid/pamid/include/mpidi_macros.h delete mode 100644 src/mpid/pamid/include/mpidi_mutex.h delete mode 100644 src/mpid/pamid/include/mpidi_platform.h delete mode 100644 src/mpid/pamid/include/mpidi_prototypes.h delete mode 100644 src/mpid/pamid/include/mpidi_thread.h delete mode 100644 src/mpid/pamid/include/mpidi_trace.h delete mode 100644 src/mpid/pamid/include/mpidi_util.h delete mode 100644 src/mpid/pamid/include/mpidimpl.h delete mode 100644 src/mpid/pamid/include/mpidpost.h delete mode 100644 src/mpid/pamid/include/mpidpre.h delete mode 100644 src/mpid/pamid/include/mpix.h delete mode 100644 src/mpid/pamid/include/pamix.h delete mode 100644 src/mpid/pamid/src/Makefile.mk delete mode 100644 src/mpid/pamid/src/coll/Makefile.mk delete mode 100644 src/mpid/pamid/src/coll/allgather/Makefile.mk delete mode 100644 src/mpid/pamid/src/coll/allgather/mpido_allgather.c delete mode 100644 src/mpid/pamid/src/coll/allgather/mpido_iallgather.c delete mode 100644 src/mpid/pamid/src/coll/allgatherv/Makefile.mk delete mode 100644 src/mpid/pamid/src/coll/allgatherv/mpido_allgatherv.c delete mode 100644 src/mpid/pamid/src/coll/allgatherv/mpido_iallgatherv.c delete mode 100644 src/mpid/pamid/src/coll/allreduce/Makefile.mk delete mode 100644 src/mpid/pamid/src/coll/allreduce/mpido_allreduce.c delete mode 100644 src/mpid/pamid/src/coll/allreduce/mpido_iallreduce.c delete mode 100644 src/mpid/pamid/src/coll/alltoall/Makefile.mk delete mode 100644 src/mpid/pamid/src/coll/alltoall/mpido_alltoall.c delete mode 100644 src/mpid/pamid/src/coll/alltoall/mpido_ialltoall.c delete mode 100644 src/mpid/pamid/src/coll/alltoallv/Makefile.mk delete mode 100644 src/mpid/pamid/src/coll/alltoallv/mpido_alltoallv.c delete mode 100644 src/mpid/pamid/src/coll/alltoallv/mpido_ialltoallv.c delete mode 100644 src/mpid/pamid/src/coll/alltoallw/Makefile.mk delete mode 100644 src/mpid/pamid/src/coll/alltoallw/mpido_ialltoallw.c delete mode 100644 src/mpid/pamid/src/coll/barrier/Makefile.mk delete mode 100644 src/mpid/pamid/src/coll/barrier/mpido_barrier.c delete mode 100644 src/mpid/pamid/src/coll/barrier/mpido_ibarrier.c delete mode 100644 src/mpid/pamid/src/coll/bcast/Makefile.mk delete mode 100644 src/mpid/pamid/src/coll/bcast/mpido_bcast.c delete mode 100644 src/mpid/pamid/src/coll/bcast/mpido_ibcast.c delete mode 100644 src/mpid/pamid/src/coll/coll_utils.c delete mode 100644 src/mpid/pamid/src/coll/exscan/Makefile.mk delete mode 100644 src/mpid/pamid/src/coll/exscan/mpido_iexscan.c delete mode 100644 src/mpid/pamid/src/coll/gather/Makefile.mk delete mode 100644 src/mpid/pamid/src/coll/gather/mpido_gather.c delete mode 100644 src/mpid/pamid/src/coll/gather/mpido_igather.c delete mode 100644 src/mpid/pamid/src/coll/gatherv/Makefile.mk delete mode 100644 src/mpid/pamid/src/coll/gatherv/mpido_gatherv.c delete mode 100644 src/mpid/pamid/src/coll/gatherv/mpido_igatherv.c delete mode 100644 src/mpid/pamid/src/coll/ired_scat/Makefile.mk delete mode 100644 src/mpid/pamid/src/coll/ired_scat/mpido_ired_scat.c delete mode 100644 src/mpid/pamid/src/coll/ired_scat_block/Makefile.mk delete mode 100644 src/mpid/pamid/src/coll/ired_scat_block/mpido_ired_scat_block.c delete mode 100644 src/mpid/pamid/src/coll/red_scat/Makefile.mk delete mode 100644 src/mpid/pamid/src/coll/red_scat/mpido_red_scat.c delete mode 100644 src/mpid/pamid/src/coll/reduce/Makefile.mk delete mode 100644 src/mpid/pamid/src/coll/reduce/mpido_ireduce.c delete mode 100644 src/mpid/pamid/src/coll/reduce/mpido_reduce.c delete mode 100644 src/mpid/pamid/src/coll/scan/Makefile.mk delete mode 100644 src/mpid/pamid/src/coll/scan/mpido_scan.c delete mode 100644 src/mpid/pamid/src/coll/scatter/Makefile.mk delete mode 100644 src/mpid/pamid/src/coll/scatter/mpido_iscatter.c delete mode 100644 src/mpid/pamid/src/coll/scatter/mpido_scatter.c delete mode 100644 src/mpid/pamid/src/coll/scatterv/Makefile.mk delete mode 100644 src/mpid/pamid/src/coll/scatterv/mpido_iscatterv.c delete mode 100644 src/mpid/pamid/src/coll/scatterv/mpido_scatterv.c delete mode 100644 src/mpid/pamid/src/comm/Makefile.mk delete mode 100644 src/mpid/pamid/src/comm/mpid_comm.c delete mode 100644 src/mpid/pamid/src/comm/mpid_optcolls.c delete mode 100644 src/mpid/pamid/src/comm/mpid_selectcolls.c delete mode 100644 src/mpid/pamid/src/dyntask/Makefile.mk delete mode 100644 src/mpid/pamid/src/dyntask/mpid_comm_disconnect.c delete mode 100644 src/mpid/pamid/src/dyntask/mpid_comm_spawn_multiple.c delete mode 100644 src/mpid/pamid/src/dyntask/mpid_port.c delete mode 100644 src/mpid/pamid/src/dyntask/mpidi_pg.c delete mode 100644 src/mpid/pamid/src/dyntask/mpidi_port.c delete mode 100644 src/mpid/pamid/src/misc/Makefile.mk delete mode 100644 src/mpid/pamid/src/misc/mpid_abort.c delete mode 100644 src/mpid/pamid/src/misc/mpid_get_universe_size.c delete mode 100644 src/mpid/pamid/src/misc/mpid_getpname.c delete mode 100644 src/mpid/pamid/src/misc/mpid_mem.c delete mode 100644 src/mpid/pamid/src/misc/mpid_unimpl.c delete mode 100644 src/mpid/pamid/src/mpid_aint.c delete mode 100644 src/mpid/pamid/src/mpid_buffer.c delete mode 100644 src/mpid/pamid/src/mpid_finalize.c delete mode 100644 src/mpid/pamid/src/mpid_improbe.c delete mode 100644 src/mpid/pamid/src/mpid_imrecv.c delete mode 100644 src/mpid/pamid/src/mpid_init.c delete mode 100644 src/mpid/pamid/src/mpid_iprobe.c delete mode 100644 src/mpid/pamid/src/mpid_mprobe.c delete mode 100644 src/mpid/pamid/src/mpid_mrecv.c delete mode 100644 src/mpid/pamid/src/mpid_probe.c delete mode 100644 src/mpid/pamid/src/mpid_progress.c delete mode 100644 src/mpid/pamid/src/mpid_progress.h delete mode 100644 src/mpid/pamid/src/mpid_recvq.c delete mode 100644 src/mpid/pamid/src/mpid_recvq.h delete mode 100644 src/mpid/pamid/src/mpid_recvq_mmap.cpp delete mode 100644 src/mpid/pamid/src/mpid_request.c delete mode 100644 src/mpid/pamid/src/mpid_request.h delete mode 100644 src/mpid/pamid/src/mpid_time.c delete mode 100644 src/mpid/pamid/src/mpid_vc.c delete mode 100644 src/mpid/pamid/src/mpidi_bufmm.c delete mode 100644 src/mpid/pamid/src/mpidi_env.c delete mode 100644 src/mpid/pamid/src/mpidi_mutex.c delete mode 100644 src/mpid/pamid/src/mpidi_nbc_sched.c delete mode 100644 src/mpid/pamid/src/mpidi_pami_datatype.c delete mode 100644 src/mpid/pamid/src/mpidi_util.c delete mode 100644 src/mpid/pamid/src/mpix/Makefile.mk delete mode 100644 src/mpid/pamid/src/mpix/mpix.c delete mode 100644 src/mpid/pamid/src/onesided/Makefile.mk delete mode 100644 src/mpid/pamid/src/onesided/mpid_1s.c delete mode 100644 src/mpid/pamid/src/onesided/mpid_win_accumulate.c delete mode 100644 src/mpid/pamid/src/onesided/mpid_win_allocate.c delete mode 100644 src/mpid/pamid/src/onesided/mpid_win_allocate_shared.c delete mode 100644 src/mpid/pamid/src/onesided/mpid_win_attach.c delete mode 100644 src/mpid/pamid/src/onesided/mpid_win_compare_and_swap.c delete mode 100644 src/mpid/pamid/src/onesided/mpid_win_create.c delete mode 100644 src/mpid/pamid/src/onesided/mpid_win_create_dynamic.c delete mode 100644 src/mpid/pamid/src/onesided/mpid_win_detach.c delete mode 100644 src/mpid/pamid/src/onesided/mpid_win_fence.c delete mode 100644 src/mpid/pamid/src/onesided/mpid_win_fetch_and_op.c delete mode 100644 src/mpid/pamid/src/onesided/mpid_win_flush.c delete mode 100644 src/mpid/pamid/src/onesided/mpid_win_free.c delete mode 100644 src/mpid/pamid/src/onesided/mpid_win_get.c delete mode 100644 src/mpid/pamid/src/onesided/mpid_win_get_accumulate.c delete mode 100644 src/mpid/pamid/src/onesided/mpid_win_get_info.c delete mode 100644 src/mpid/pamid/src/onesided/mpid_win_lock.c delete mode 100644 src/mpid/pamid/src/onesided/mpid_win_lock_all.c delete mode 100644 src/mpid/pamid/src/onesided/mpid_win_pscw.c delete mode 100644 src/mpid/pamid/src/onesided/mpid_win_put.c delete mode 100644 src/mpid/pamid/src/onesided/mpid_win_reqops.c delete mode 100644 src/mpid/pamid/src/onesided/mpid_win_set_info.c delete mode 100644 src/mpid/pamid/src/onesided/mpid_win_shared_query.c delete mode 100644 src/mpid/pamid/src/onesided/mpid_win_sync.c delete mode 100644 src/mpid/pamid/src/onesided/mpidi_onesided.h delete mode 100644 src/mpid/pamid/src/onesided/mpidi_win_control.c delete mode 100644 src/mpid/pamid/src/pamix/Makefile.mk delete mode 100644 src/mpid/pamid/src/pamix/pamix.c delete mode 100644 src/mpid/pamid/src/pt2pt/Makefile.mk delete mode 100644 src/mpid/pamid/src/pt2pt/mpid_cancel.c delete mode 100644 src/mpid/pamid/src/pt2pt/mpid_irecv.h delete mode 100644 src/mpid/pamid/src/pt2pt/mpid_isend.h delete mode 100644 src/mpid/pamid/src/pt2pt/mpid_issend.c delete mode 100644 src/mpid/pamid/src/pt2pt/mpid_recv.c delete mode 100644 src/mpid/pamid/src/pt2pt/mpid_send.h delete mode 100644 src/mpid/pamid/src/pt2pt/mpid_ssend.c delete mode 100644 src/mpid/pamid/src/pt2pt/mpidi_callback_eager.c delete mode 100644 src/mpid/pamid/src/pt2pt/mpidi_callback_rzv.c delete mode 100644 src/mpid/pamid/src/pt2pt/mpidi_callback_short.c delete mode 100644 src/mpid/pamid/src/pt2pt/mpidi_callback_util.c delete mode 100644 src/mpid/pamid/src/pt2pt/mpidi_control.c delete mode 100644 src/mpid/pamid/src/pt2pt/mpidi_done.c delete mode 100644 src/mpid/pamid/src/pt2pt/mpidi_recv.h delete mode 100644 src/mpid/pamid/src/pt2pt/mpidi_recvmsg.c delete mode 100644 src/mpid/pamid/src/pt2pt/mpidi_rendezvous.c delete mode 100644 src/mpid/pamid/src/pt2pt/mpidi_send.h delete mode 100644 src/mpid/pamid/src/pt2pt/mpidi_sendmsg.c delete mode 100644 src/mpid/pamid/src/pt2pt/persistent/Makefile.mk delete mode 100644 src/mpid/pamid/src/pt2pt/persistent/mpid_recv_init.c delete mode 100644 src/mpid/pamid/src/pt2pt/persistent/mpid_send_init.c delete mode 100644 src/mpid/pamid/src/pt2pt/persistent/mpid_startall.c delete mode 100644 src/mpid/pamid/subconfigure.m4 delete mode 100644 src/pmi/pmi2/poe/Makefile.mk delete mode 100644 src/pmi/pmi2/poe/poe2pmi.c delete mode 100644 src/pmi/pmi2/poe/subconfigure.m4 hooks/post-receive -- MPICH primary repository

1 0

[mpich] MPICH primary repository branch, master, updated. v3.2-403-g398f4af
by noreply＠mpich.org 25 Aug '16

25 Aug '16

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "MPICH primary repository". The branch, master has been updated via 398f4af12cf6d05176886515c8f42df870c92631 (commit) via 2299f39c89dbcf3bf0a825be7ca02469cc8c825a (commit) via 4f2fb4708e7190bb98e5b5b55d0f27d44013aa32 (commit) via 9a89293b84fa160b3d5d767bc1e8be3bed27789b (commit) via 3359484d7216e598203b8f9655c73ceefaeb844a (commit) via 0b72320b7a842e16f3e19dcbacf900a57a6a1c61 (commit) via 181430bda15e745f82f761e28643e71eb42bcbb2 (commit) via af99ecc1e04a32f9224f2d803751c1149e0a97c5 (commit) via 7910b821f8e6d816e281d42120737e63c63a83b9 (commit) via 3257bbea93f661d54b8d43e0d06bc4841d496b13 (commit) via 02cae1085c508fe494a14970e9dce541ee3c0c07 (commit) via 6ff49c7d01617b757fc4b9450d74496aaf774d80 (commit) via 5d2ded70d81461d9db6661e2fb32c8a1b4e97db9 (commit) via 7040a6289af4de7ace66ad71248c1ef01f411855 (commit) via 21dd483c6c5a075cbe86b0a979af481dfddf9329 (commit) via 3dd1f56608e155242d8118cb2391d87f6bb33091 (commit) via daa80957db99eea196c920bf465a67972ad5af9f (commit) via e22e938f22f4af2f432ca3011074c3e630922f2c (commit) from cd846d2c0f3349032f981b297a28eef8af9a5995 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- http://git.mpich.org/mpich.git/commitdiff/398f4af12cf6d05176886515c8f42df87… commit 398f4af12cf6d05176886515c8f42df870c92631 Author: Pavan Balaji <balaji(a)anl.gov> Date: Thu Aug 25 10:08:08 2016 -0500 CH4: Fix resource leak in any source receive When we get a message from the network, clean up the shared memory request correctly. Also, look for request completion rather than relying on the netmod setting the request source (which is not necessary, as long as the status object is correctly set). Signed-off-by: Ken Raffenetti <raffenet(a)mcs.anl.gov> diff --git a/src/mpid/ch4/src/ch4_recv.h b/src/mpid/ch4/src/ch4_recv.h index 50370bd..74aafb3 100644 --- a/src/mpid/ch4/src/ch4_recv.h +++ b/src/mpid/ch4/src/ch4_recv.h @@ -60,11 +60,12 @@ MPL_STATIC_INLINE_PREFIX int MPIDI_Recv(void *buf, } /* cancel the shm request if netmod/am handles the request from unexpected queue. */ else if (*request) { - if (MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(*request)->status.MPI_SOURCE != MPI_UNDEFINED) { + if (MPIR_Request_is_complete(MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(*request))) { mpi_errno = MPIDI_SHM_cancel_recv(*request); if (MPIR_STATUS_GET_CANCEL_BIT((*request)->status)) { (*request)->status = MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(*request)->status; } + MPIR_Request_free(MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(*request)); goto fn_exit; } MPIDI_CH4I_REQUEST(*request, is_local) = 1; http://git.mpich.org/mpich.git/commitdiff/2299f39c89dbcf3bf0a825be7ca02469c… commit 2299f39c89dbcf3bf0a825be7ca02469cc8c825a Author: Pavan Balaji <balaji(a)anl.gov> Date: Wed Aug 24 22:36:21 2016 -0500 CH4: Fix bug in handling zero-byte AM sends Signed-off-by: Ken Raffenetti <raffenet(a)mcs.anl.gov> diff --git a/src/mpid/ch4/src/ch4r_callbacks.h b/src/mpid/ch4/src/ch4r_callbacks.h index 521d509..460adb9 100644 --- a/src/mpid/ch4/src/ch4r_callbacks.h +++ b/src/mpid/ch4/src/ch4r_callbacks.h @@ -1365,6 +1365,9 @@ static inline int MPIDI_CH4I_do_send_target_handler(void **data, *cmpl_handler_fn = MPIDI_CH4U_recv_cmpl_handler; MPIDI_CH4U_REQUEST(rreq, req->seq_no) = OPA_fetch_and_add_int(&MPIDI_CH4_Global.nxt_seq_no, 1); + if (p_data_sz == NULL) + return MPI_SUCCESS; + MPIDI_Datatype_get_info(MPIDI_CH4U_REQUEST(rreq, count), MPIDI_CH4U_REQUEST(rreq, datatype), dt_contig, data_sz, dt_ptr, dt_true_lb); @@ -1437,9 +1440,15 @@ static inline int MPIDI_CH4U_send_target_handler(int handler_id, void *am_hdr, if (rreq == NULL) { rreq = MPIDI_CH4I_am_request_create(MPIR_REQUEST_KIND__RECV); - MPIDI_CH4U_REQUEST(rreq, buffer) = (char *) MPL_malloc(*p_data_sz); MPIDI_CH4U_REQUEST(rreq, datatype) = MPI_BYTE; - MPIDI_CH4U_REQUEST(rreq, count) = *p_data_sz; + if (p_data_sz) { + MPIDI_CH4U_REQUEST(rreq, buffer) = (char *) MPL_malloc(*p_data_sz); + MPIDI_CH4U_REQUEST(rreq, count) = *p_data_sz; + } + else { + MPIDI_CH4U_REQUEST(rreq, buffer) = NULL; + MPIDI_CH4U_REQUEST(rreq, count) = 0; + } MPIDI_CH4U_REQUEST(rreq, tag) = hdr->msg_tag; MPIDI_CH4U_REQUEST(rreq, src_rank) = hdr->src_rank; MPIDI_CH4U_REQUEST(rreq, req->status) |= MPIDI_CH4U_REQ_BUSY; @@ -2574,9 +2583,11 @@ static inline int MPIDI_CH4U_handle_acc_request(int handler_id, void *am_hdr, MPIDI_CH4U_acc_cmpl_handler; MPIDI_CH4U_REQUEST(rreq, req->seq_no) = OPA_fetch_and_add_int(&MPIDI_CH4_Global.nxt_seq_no, 1); - *is_contig = 1; - *p_data_sz = data_sz; - *data = p_data; + if (is_contig) { + *is_contig = 1; + *p_data_sz = data_sz; + *data = p_data; + } MPL_HASH_FIND(dev.ch4u.hash_handle, MPIDI_CH4_Global.win_hash, &msg_hdr->win_id, sizeof(uint64_t), win); http://git.mpich.org/mpich.git/commitdiff/4f2fb4708e7190bb98e5b5b55d0f27d44… commit 4f2fb4708e7190bb98e5b5b55d0f27d44013aa32 Author: Pavan Balaji <balaji(a)anl.gov> Date: Wed Aug 24 17:56:11 2016 -0500 CH4: Fixes to improve strict builds. This patch is incomplete and does not fully fix the build when configured with --enable-strict. Signed-off-by: Ken Raffenetti <raffenet(a)mcs.anl.gov> diff --git a/src/mpid/ch4/include/mpidch4.h b/src/mpid/ch4/include/mpidch4.h index bffa964..b0f718d 100644 --- a/src/mpid/ch4/include/mpidch4.h +++ b/src/mpid/ch4/include/mpidch4.h @@ -143,8 +143,8 @@ MPIDI_CH4I_API(MPI_Aint, Aint_diff, MPI_Aint, MPI_Aint); MPIDI_CH4I_API(int, GPID_GetAllInComm, MPIR_Comm *, int, MPIR_Gpid[], int *); MPIDI_CH4I_API(int, GPID_ToLpidArray, int, MPIR_Gpid[], int[]); MPIDI_CH4I_API(int, Create_intercomm_from_lpids, MPIR_Comm *, int, const int[]); -MPIDI_CH4I_API(int, Comm_create, MPIR_Comm *); -MPIDI_CH4I_API(int, Comm_destroy, MPIR_Comm *); +MPIDI_CH4I_API(int, Comm_create_hook, MPIR_Comm *); +MPIDI_CH4I_API(int, Comm_free_hook, MPIR_Comm *); MPIDI_CH4I_API(int, Barrier, MPIR_Comm *, MPIR_Errflag_t *); MPIDI_CH4I_API(int, Bcast, void *, int, MPI_Datatype, int, MPIR_Comm *, MPIR_Errflag_t *); MPIDI_CH4I_API(int, Allreduce, const void *, void *, int, MPI_Datatype, MPI_Op, MPIR_Comm *, diff --git a/src/mpid/ch4/src/ch4_init.h b/src/mpid/ch4/src/ch4_init.h index 7d8c10e..ca88320 100644 --- a/src/mpid/ch4/src/ch4_init.h +++ b/src/mpid/ch4/src/ch4_init.h @@ -14,6 +14,7 @@ #include "ch4_impl.h" #include "ch4r_proc.h" #include "ch4i_comm.h" +#include "strings.h" /* === BEGIN_MPI_T_CVAR_INFO_BLOCK === diff --git a/src/mpid/ch4/src/ch4_recv.h b/src/mpid/ch4/src/ch4_recv.h index a6ab84a..50370bd 100644 --- a/src/mpid/ch4/src/ch4_recv.h +++ b/src/mpid/ch4/src/ch4_recv.h @@ -190,7 +190,7 @@ MPL_STATIC_INLINE_PREFIX int MPIDI_Mrecv(void *buf, } if (unlikely(message->status.MPI_SOURCE == MPI_PROC_NULL)) { - MPIR_Request *rreq = message; + rreq = message; rreq->status.MPI_SOURCE = message->status.MPI_SOURCE; rreq->status.MPI_TAG = message->status.MPI_TAG; MPIDI_CH4U_request_complete(rreq); http://git.mpich.org/mpich.git/commitdiff/9a89293b84fa160b3d5d767bc1e8be3be… commit 9a89293b84fa160b3d5d767bc1e8be3bed27789b Author: Pavan Balaji <balaji(a)anl.gov> Date: Fri Aug 19 18:53:46 2016 -0500 CH4: Remove "msg type" field for RMA control messages Use a single packet handle type to convey this information. Signed-off-by: Ken Raffenetti <raffenet(a)mcs.anl.gov> diff --git a/src/mpid/ch4/netmod/ofi/ofi_types.h b/src/mpid/ch4/netmod/ofi/ofi_types.h index 045cc96..9ba98ec 100644 --- a/src/mpid/ch4/netmod/ofi/ofi_types.h +++ b/src/mpid/ch4/netmod/ofi/ofi_types.h @@ -38,7 +38,7 @@ #define MPIDI_OFI_BUF_POOL_SIZE (1024) #define MPIDI_OFI_BUF_POOL_NUM (1024) #define MPIDI_OFI_NUM_CQ_BUFFERED (1024) -#define MPIDI_OFI_MAX_AM_HANDLERS_TOTAL (24) +#define MPIDI_OFI_MAX_AM_HANDLERS_TOTAL (100) #define MPIDI_OFI_INTERNAL_HANDLER_CONTROL (MPIDI_OFI_MAX_AM_HANDLERS_TOTAL-1) #define MPIDI_OFI_INTERNAL_HANDLER_NEXT (MPIDI_OFI_MAX_AM_HANDLERS_TOTAL-2) #define MPIDI_OFI_MAX_AM_HANDLERS (MPIDI_OFI_INTERNAL_HANDLER_NEXT-1) diff --git a/src/mpid/ch4/src/ch4_types.h b/src/mpid/ch4/src/ch4_types.h index 061122e..93b5a17 100644 --- a/src/mpid/ch4/src/ch4_types.h +++ b/src/mpid/ch4/src/ch4_types.h @@ -49,7 +49,7 @@ typedef struct progress_hook_slot { int active; } progress_hook_slot_t; -typedef enum { +enum { MPIDI_CH4U_SEND = 0, /* Eager send */ MPIDI_CH4U_SEND_LONG_REQ, /* Rendezvous send RTS (request to send) */ @@ -59,8 +59,6 @@ typedef enum { MPIDI_CH4U_SSEND_REQ, MPIDI_CH4U_SSEND_ACK, - MPIDI_CH4U_WIN_CTRL, - MPIDI_CH4U_PUT_REQ, MPIDI_CH4U_PUT_ACK, MPIDI_CH4U_PUT_IOV_REQ, @@ -79,10 +77,8 @@ typedef enum { MPIDI_CH4U_CSWAP_REQ, MPIDI_CH4U_CSWAP_ACK, - MPIDI_CH4U_FETCH_OP -} MPIDI_CH4U_TYPE; + MPIDI_CH4U_FETCH_OP, -typedef enum { MPIDI_CH4U_WIN_COMPLETE, MPIDI_CH4U_WIN_POST, MPIDI_CH4U_WIN_LOCK, @@ -93,7 +89,7 @@ typedef enum { MPIDI_CH4U_WIN_LOCKALL_ACK, MPIDI_CH4U_WIN_UNLOCKALL, MPIDI_CH4U_WIN_UNLOCKALL_ACK -} MPIDI_CH4U_WIN_CTRL_MSG_TYPE; +}; enum { MPIDI_CH4U_EPOTYPE_NONE = 0, /**< No epoch in affect */ @@ -144,7 +140,6 @@ typedef struct MPIDI_CH4U_win_cntrl_msg_t { uint64_t win_id; uint32_t origin_rank; int16_t lock_type; - int16_t type; } MPIDI_CH4U_win_cntrl_msg_t; typedef struct MPIDI_CH4U_put_msg_t { diff --git a/src/mpid/ch4/src/ch4r_callbacks.h b/src/mpid/ch4/src/ch4r_callbacks.h index be2291f..521d509 100644 --- a/src/mpid/ch4/src/ch4r_callbacks.h +++ b/src/mpid/ch4/src/ch4r_callbacks.h @@ -1885,18 +1885,19 @@ static inline int MPIDI_CH4U_win_lock_advance(MPIR_Win * win) slock->local.type = lock->type; MPIDI_CH4U_win_cntrl_msg_t msg; + int handler_id; msg.win_id = MPIDI_CH4U_WIN(win, win_id); msg.origin_rank = win->comm_ptr->rank; if (lock->mtype == MPIDI_CH4U_WIN_LOCK) - msg.type = MPIDI_CH4U_WIN_LOCK_ACK; + handler_id = MPIDI_CH4U_WIN_LOCK_ACK; else if (lock->mtype == MPIDI_CH4U_WIN_LOCKALL) - msg.type = MPIDI_CH4U_WIN_LOCKALL_ACK; + handler_id = MPIDI_CH4U_WIN_LOCKALL_ACK; else MPIR_ERR_SETANDJUMP(mpi_errno, MPI_ERR_OTHER, "**rmasync"); mpi_errno = MPIDI_NM_am_send_hdr_reply(MPIDI_CH4U_win_to_context(win), - lock->rank, MPIDI_CH4U_WIN_CTRL, &msg, sizeof(msg)); + lock->rank, handler_id, &msg, sizeof(msg)); if (mpi_errno) MPIR_ERR_POP(mpi_errno); MPL_free(lock); @@ -1917,7 +1918,8 @@ static inline int MPIDI_CH4U_win_lock_advance(MPIR_Win * win) #define FUNCNAME MPIDI_CH4U_win_lock_req_proc #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline void MPIDI_CH4U_win_lock_req_proc(const MPIDI_CH4U_win_cntrl_msg_t * info, +static inline void MPIDI_CH4U_win_lock_req_proc(int handler_id, + const MPIDI_CH4U_win_cntrl_msg_t * info, MPIR_Win * win) { MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_WIN_LOCK_REQ_PROC); @@ -1926,11 +1928,7 @@ static inline void MPIDI_CH4U_win_lock_req_proc(const MPIDI_CH4U_win_cntrl_msg_t struct MPIDI_CH4U_win_lock *lock = (struct MPIDI_CH4U_win_lock *) MPL_calloc(1, sizeof(struct MPIDI_CH4U_win_lock)); - if (info->type == MPIDI_CH4U_WIN_LOCK) - lock->mtype = MPIDI_CH4U_WIN_LOCK; - else if (info->type == MPIDI_CH4U_WIN_LOCKALL) - lock->mtype = MPIDI_CH4U_WIN_LOCKALL; - + lock->mtype = handler_id; lock->rank = info->origin_rank; lock->type = info->lock_type; struct MPIDI_CH4U_win_queue *q = &MPIDI_CH4U_WIN(win, sync).lock.local.requested; @@ -1952,15 +1950,16 @@ static inline void MPIDI_CH4U_win_lock_req_proc(const MPIDI_CH4U_win_cntrl_msg_t #define FUNCNAME MPIDI_CH4U_win_lock_ack_proc #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline void MPIDI_CH4U_win_lock_ack_proc(const MPIDI_CH4U_win_cntrl_msg_t * info, +static inline void MPIDI_CH4U_win_lock_ack_proc(int handler_id, + const MPIDI_CH4U_win_cntrl_msg_t * info, MPIR_Win * win) { MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_WIN_LOCK_ACK_PROC); MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_WIN_LOCK_ACK_PROC); - if (info->type == MPIDI_CH4U_WIN_LOCK_ACK) + if (handler_id == MPIDI_CH4U_WIN_LOCK_ACK) MPIDI_CH4U_WIN(win, sync).lock.remote.locked += 1; - else if (info->type == MPIDI_CH4U_WIN_LOCKALL_ACK) + else if (handler_id == MPIDI_CH4U_WIN_LOCKALL_ACK) MPIDI_CH4U_WIN(win, sync).lock.remote.allLocked += 1; MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_WIN_LOCK_ACK_PROC); @@ -1985,11 +1984,10 @@ static inline void MPIDI_CH4U_win_unlock_proc(const MPIDI_CH4U_win_cntrl_msg_t * MPIDI_CH4U_win_cntrl_msg_t msg; msg.win_id = MPIDI_CH4U_WIN(win, win_id); msg.origin_rank = win->comm_ptr->rank; - msg.type = MPIDI_CH4U_WIN_UNLOCK_ACK; mpi_errno = MPIDI_NM_am_send_hdr_reply(MPIDI_CH4U_win_to_context(win), info->origin_rank, - MPIDI_CH4U_WIN_CTRL, &msg, sizeof(msg)); + MPIDI_CH4U_WIN_UNLOCK_ACK, &msg, sizeof(msg)); if (mpi_errno) MPIR_ERR_POP(mpi_errno); fn_exit: @@ -2074,17 +2072,17 @@ static inline int MPIDI_CH4U_win_ctrl_target_handler(int handler_id, void *am_hd &msg_hdr->win_id, sizeof(uint64_t), win); /* TODO: check output win ptr */ - switch (msg_hdr->type) { + switch (handler_id) { char buff[32]; case MPIDI_CH4U_WIN_LOCK: case MPIDI_CH4U_WIN_LOCKALL: - MPIDI_CH4U_win_lock_req_proc(msg_hdr, win); + MPIDI_CH4U_win_lock_req_proc(handler_id, msg_hdr, win); break; case MPIDI_CH4U_WIN_LOCK_ACK: case MPIDI_CH4U_WIN_LOCKALL_ACK: - MPIDI_CH4U_win_lock_ack_proc(msg_hdr, win); + MPIDI_CH4U_win_lock_ack_proc(handler_id, msg_hdr, win); break; case MPIDI_CH4U_WIN_UNLOCK: @@ -2106,7 +2104,7 @@ static inline int MPIDI_CH4U_win_ctrl_target_handler(int handler_id, void *am_hd break; default: - MPL_snprintf(buff, sizeof(buff), "Invalid message type: %d\n", msg_hdr->type); + MPL_snprintf(buff, sizeof(buff), "Invalid message type: %d\n", handler_id); MPID_Abort(NULL, MPI_ERR_INTERN, 1, buff); } diff --git a/src/mpid/ch4/src/ch4r_init.h b/src/mpid/ch4/src/ch4r_init.h index f692720..0089861 100644 --- a/src/mpid/ch4/src/ch4r_init.h +++ b/src/mpid/ch4/src/ch4r_init.h @@ -215,11 +215,55 @@ MPL_STATIC_INLINE_PREFIX int MPIDI_CH4U_init(MPIR_Comm * comm_world, MPIR_Comm * if (mpi_errno) MPIR_ERR_POP(mpi_errno); - mpi_errno = MPIDI_NM_am_reg_handler(MPIDI_CH4U_WIN_CTRL, + mpi_errno = MPIDI_NM_am_reg_handler(MPIDI_CH4U_WIN_COMPLETE, NULL, &MPIDI_CH4U_win_ctrl_target_handler); if (mpi_errno) MPIR_ERR_POP(mpi_errno); + mpi_errno = MPIDI_NM_am_reg_handler(MPIDI_CH4U_WIN_POST, + NULL, &MPIDI_CH4U_win_ctrl_target_handler); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + mpi_errno = MPIDI_NM_am_reg_handler(MPIDI_CH4U_WIN_LOCK, + NULL, &MPIDI_CH4U_win_ctrl_target_handler); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + mpi_errno = MPIDI_NM_am_reg_handler(MPIDI_CH4U_WIN_LOCK_ACK, + NULL, &MPIDI_CH4U_win_ctrl_target_handler); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + mpi_errno = MPIDI_NM_am_reg_handler(MPIDI_CH4U_WIN_UNLOCK, + NULL, &MPIDI_CH4U_win_ctrl_target_handler); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + mpi_errno = MPIDI_NM_am_reg_handler(MPIDI_CH4U_WIN_UNLOCK_ACK, + NULL, &MPIDI_CH4U_win_ctrl_target_handler); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + mpi_errno = MPIDI_NM_am_reg_handler(MPIDI_CH4U_WIN_LOCKALL, + NULL, &MPIDI_CH4U_win_ctrl_target_handler); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + mpi_errno = MPIDI_NM_am_reg_handler(MPIDI_CH4U_WIN_LOCKALL_ACK, + NULL, &MPIDI_CH4U_win_ctrl_target_handler); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + mpi_errno = MPIDI_NM_am_reg_handler(MPIDI_CH4U_WIN_UNLOCKALL, + NULL, &MPIDI_CH4U_win_ctrl_target_handler); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + mpi_errno = MPIDI_NM_am_reg_handler(MPIDI_CH4U_WIN_UNLOCKALL_ACK, + NULL, &MPIDI_CH4U_win_ctrl_target_handler); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); mpi_errno = MPIDI_NM_am_reg_handler(MPIDI_CH4U_PUT_IOV_REQ, &MPIDI_CH4U_put_iov_origin_cmpl_handler, diff --git a/src/mpid/ch4/src/ch4r_win.h b/src/mpid/ch4/src/ch4r_win.h index ebf3299..035e125 100644 --- a/src/mpid/ch4/src/ch4r_win.h +++ b/src/mpid/ch4/src/ch4r_win.h @@ -273,7 +273,6 @@ static inline int MPIDI_CH4R_win_complete(MPIR_Win * win) msg.win_id = MPIDI_CH4U_WIN(win, win_id); msg.origin_rank = win->comm_ptr->rank; - msg.type = MPIDI_CH4U_WIN_COMPLETE; ranks_in_win_grp = (int *) MPL_malloc(sizeof(int) * group->size); MPIR_Assert(ranks_in_win_grp); @@ -285,7 +284,7 @@ static inline int MPIDI_CH4R_win_complete(MPIR_Win * win) for (index = 0; index < group->size; ++index) { peer = ranks_in_win_grp[index]; mpi_errno = MPIDI_NM_am_send_hdr(peer, win->comm_ptr, - MPIDI_CH4U_WIN_CTRL, &msg, sizeof(msg), NULL); + MPIDI_CH4U_WIN_COMPLETE, &msg, sizeof(msg), NULL); if (mpi_errno != MPI_SUCCESS) MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, goto fn_fail, "**rmasync"); } @@ -327,7 +326,6 @@ static inline int MPIDI_CH4R_win_post(MPIR_Group * group, int assert, MPIR_Win * msg.win_id = MPIDI_CH4U_WIN(win, win_id); msg.origin_rank = win->comm_ptr->rank; - msg.type = MPIDI_CH4U_WIN_POST; ranks_in_win_grp = (int *) MPL_malloc(sizeof(int) * group->size); MPIR_Assert(ranks_in_win_grp); @@ -339,7 +337,7 @@ static inline int MPIDI_CH4R_win_post(MPIR_Group * group, int assert, MPIR_Win * for (index = 0; index < group->size; ++index) { peer = ranks_in_win_grp[index]; mpi_errno = MPIDI_NM_am_send_hdr(peer, win->comm_ptr, - MPIDI_CH4U_WIN_CTRL, &msg, sizeof(msg), NULL); + MPIDI_CH4U_WIN_POST, &msg, sizeof(msg), NULL); if (mpi_errno != MPI_SUCCESS) MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, goto fn_fail, "**rmasync"); } @@ -436,12 +434,11 @@ static inline int MPIDI_CH4R_win_lock(int lock_type, int rank, int assert, MPIR_ MPIDI_CH4U_win_cntrl_msg_t msg; msg.win_id = MPIDI_CH4U_WIN(win, win_id); msg.origin_rank = win->comm_ptr->rank; - msg.type = MPIDI_CH4U_WIN_LOCK; msg.lock_type = lock_type; locked = slock->remote.locked + 1; mpi_errno = MPIDI_NM_am_send_hdr(rank, win->comm_ptr, - MPIDI_CH4U_WIN_CTRL, &msg, sizeof(msg), NULL); + MPIDI_CH4U_WIN_LOCK, &msg, sizeof(msg), NULL); if (mpi_errno != MPI_SUCCESS) MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, goto fn_fail, "**rmasync"); MPIDI_CH4R_PROGRESS_WHILE(slock->remote.locked != locked); @@ -479,11 +476,10 @@ static inline int MPIDI_CH4R_win_unlock(int rank, MPIR_Win * win) msg.win_id = MPIDI_CH4U_WIN(win, win_id); msg.origin_rank = win->comm_ptr->rank; - msg.type = MPIDI_CH4U_WIN_UNLOCK; unlocked = MPIDI_CH4U_WIN(win, sync).lock.remote.locked - 1; mpi_errno = MPIDI_NM_am_send_hdr(rank, win->comm_ptr, - MPIDI_CH4U_WIN_CTRL, &msg, sizeof(msg), NULL); + MPIDI_CH4U_WIN_UNLOCK, &msg, sizeof(msg), NULL); if (mpi_errno != MPI_SUCCESS) MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, goto fn_fail, "**rmasync"); @@ -1060,14 +1056,13 @@ static inline int MPIDI_CH4R_win_unlock_all(MPIR_Win * win) MPIDI_CH4U_win_cntrl_msg_t msg; msg.win_id = MPIDI_CH4U_WIN(win, win_id); msg.origin_rank = win->comm_ptr->rank; - msg.type = MPIDI_CH4U_WIN_UNLOCKALL; lockQ[i].done = 0; lockQ[i].peer = i; lockQ[i].win = win; mpi_errno = MPIDI_NM_am_send_hdr(i, win->comm_ptr, - MPIDI_CH4U_WIN_CTRL, &msg, sizeof(msg), NULL); + MPIDI_CH4U_WIN_UNLOCKALL, &msg, sizeof(msg), NULL); if (mpi_errno != MPI_SUCCESS) MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, goto fn_fail, "**rmasync"); @@ -1220,7 +1215,6 @@ static inline int MPIDI_CH4R_win_lock_all(int assert, MPIR_Win * win) MPIDI_CH4U_win_cntrl_msg_t msg; msg.win_id = MPIDI_CH4U_WIN(win, win_id); msg.origin_rank = win->comm_ptr->rank; - msg.type = MPIDI_CH4U_WIN_LOCKALL; msg.lock_type = MPI_LOCK_SHARED; lockQ[i].done = 0; @@ -1229,7 +1223,7 @@ static inline int MPIDI_CH4R_win_lock_all(int assert, MPIR_Win * win) lockQ[i].lock_type = MPI_LOCK_SHARED; mpi_errno = MPIDI_NM_am_send_hdr(i, win->comm_ptr, - MPIDI_CH4U_WIN_CTRL, &msg, sizeof(msg), NULL); + MPIDI_CH4U_WIN_LOCKALL, &msg, sizeof(msg), NULL); if (mpi_errno != MPI_SUCCESS) MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, goto fn_fail, "**rmasync"); http://git.mpich.org/mpich.git/commitdiff/3359484d7216e598203b8f9655c73ceef… commit 3359484d7216e598203b8f9655c73ceefaeb844a Author: Pavan Balaji <balaji(a)anl.gov> Date: Wed Aug 24 17:12:09 2016 -0500 CH4: Pass packet handler IDs back to the handlers This allows the target to use the same function for multiple packet types that differ slightly from each other. The target already has this information. This patch just passes it back to the packet handler. Signed-off-by: Ken Raffenetti <raffenet(a)mcs.anl.gov> diff --git a/src/mpid/ch4/netmod/include/netmod.h b/src/mpid/ch4/netmod/include/netmod.h index 04f192b..6e3457d 100644 --- a/src/mpid/ch4/netmod/include/netmod.h +++ b/src/mpid/ch4/netmod/include/netmod.h @@ -22,7 +22,7 @@ typedef int (*MPIDI_NM_am_origin_handler_fn) (MPIR_Request * req); /* Callback function setup by handler register function */ /* for short cases, output arguments are NULL */ typedef int (*MPIDI_NM_am_target_handler_fn) - (void *am_hdr, void **data, /* data should be iovs if *is_contig is false */ + (int handler_id, void *am_hdr, void **data, /* data should be iovs if *is_contig is false */ size_t * data_sz, int *is_contig, MPIDI_NM_am_completion_handler_fn * cmpl_handler_fn, /* completion handler */ MPIR_Request ** req); /* if allocated, need pointer to completion function */ diff --git a/src/mpid/ch4/netmod/ofi/ofi_am_events.h b/src/mpid/ch4/netmod/ofi/ofi_am_events.h index a309a02..1f83f40 100644 --- a/src/mpid/ch4/netmod/ofi/ofi_am_events.h +++ b/src/mpid/ch4/netmod/ofi/ofi_am_events.h @@ -36,7 +36,7 @@ static inline int MPIDI_OFI_handle_short_am(MPIDI_OFI_am_header_t * msg_hdr) p_data = in_data = (char *) msg_hdr->payload + msg_hdr->am_hdr_sz; in_data_sz = data_sz = msg_hdr->data_sz; - MPIDI_Global.am_handlers[msg_hdr->handler_id] (msg_hdr->payload, + MPIDI_Global.am_handlers[msg_hdr->handler_id] (msg_hdr->handler_id, msg_hdr->payload, &p_data, &data_sz, &is_contig, &cmpl_handler_fn, &rreq); @@ -106,7 +106,7 @@ static inline int MPIDI_OFI_handle_short_am_hdr(MPIDI_OFI_am_header_t * msg_hdr, MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_HANDLE_SHORT_AM_HDR); MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_HANDLE_SHORT_AM_HDR); - MPIDI_Global.am_handlers[msg_hdr->handler_id] (am_hdr, + MPIDI_Global.am_handlers[msg_hdr->handler_id] (msg_hdr->handler_id, am_hdr, NULL, NULL, NULL, &cmpl_handler_fn, &rreq); if (!rreq) @@ -225,7 +225,7 @@ static inline int MPIDI_OFI_do_handle_long_am(MPIDI_OFI_am_header_t * msg_hdr, MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_DO_HANDLE_LONG_AM); in_data_sz = data_sz = msg_hdr->data_sz; - MPIDI_Global.am_handlers[msg_hdr->handler_id] (am_hdr, + MPIDI_Global.am_handlers[msg_hdr->handler_id] (msg_hdr->handler_id, am_hdr, &p_data, &data_sz, &is_contig, &cmpl_handler_fn, &rreq); diff --git a/src/mpid/ch4/netmod/ofi/ofi_impl.h b/src/mpid/ch4/netmod/ofi/ofi_impl.h index 9e6ff5c..95e3117 100644 --- a/src/mpid/ch4/netmod/ofi/ofi_impl.h +++ b/src/mpid/ch4/netmod/ofi/ofi_impl.h @@ -303,7 +303,7 @@ __ALWAYS_INLINE__ void MPIDI_OFI_cntr_incr() /* Externs: see util.c for definition */ extern int MPIDI_OFI_handle_cq_error_util(ssize_t ret); extern int MPIDI_OFI_progress_test_no_inline(); -extern int MPIDI_OFI_control_handler(void *am_hdr, +extern int MPIDI_OFI_control_handler(int handler_id, void *am_hdr, void **data, size_t * data_sz, int *is_contig, MPIDI_NM_am_completion_handler_fn * cmpl_handler_fn, MPIR_Request ** req); diff --git a/src/mpid/ch4/netmod/ofi/util.c b/src/mpid/ch4/netmod/ofi/util.c index 6c30026..f2b2b54 100644 --- a/src/mpid/ch4/netmod/ofi/util.c +++ b/src/mpid/ch4/netmod/ofi/util.c @@ -376,7 +376,7 @@ static inline void MPIDI_OFI_get_huge(MPIDI_OFI_send_control_t * info) MPIDI_OFI_get_huge_event(NULL, (MPIR_Request *) recv); } -int MPIDI_OFI_control_handler(void *am_hdr, +int MPIDI_OFI_control_handler(int handler_id, void *am_hdr, void **data, size_t * data_sz, int *is_contig, diff --git a/src/mpid/ch4/netmod/ucx/ucx_progress.h b/src/mpid/ch4/netmod/ucx/ucx_progress.h index c12801d..0467099 100644 --- a/src/mpid/ch4/netmod/ucx/ucx_progress.h +++ b/src/mpid/ch4/netmod/ucx/ucx_progress.h @@ -28,7 +28,7 @@ static inline int MPIDI_UCX_am_handler(void *msg, size_t msg_sz) p_data = in_data = (char *) msg_hdr->payload + (msg_sz - msg_hdr->data_sz - sizeof(*msg_hdr)); in_data_sz = data_sz = msg_hdr->data_sz; - MPIDI_UCX_global.am_handlers[msg_hdr->handler_id] (msg_hdr->payload, + MPIDI_UCX_global.am_handlers[msg_hdr->handler_id] (msg_hdr->handler_id, msg_hdr->payload, &p_data, &data_sz, &is_contig, &cmpl_handler_fn, &rreq); diff --git a/src/mpid/ch4/src/ch4r_callbacks.h b/src/mpid/ch4/src/ch4r_callbacks.h index 0f311d7..be2291f 100644 --- a/src/mpid/ch4/src/ch4r_callbacks.h +++ b/src/mpid/ch4/src/ch4r_callbacks.h @@ -1413,7 +1413,7 @@ static inline int MPIDI_CH4I_do_send_target_handler(void **data, #define FUNCNAME MPIDI_CH4U_send_target_handler #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_CH4U_send_target_handler(void *am_hdr, +static inline int MPIDI_CH4U_send_target_handler(int handler_id, void *am_hdr, void **data, size_t * p_data_sz, int *is_contig, @@ -1476,7 +1476,7 @@ static inline int MPIDI_CH4U_send_target_handler(void *am_hdr, #define FUNCNAME MPIDI_CH4U_send_long_req_target_handler #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_CH4U_send_long_req_target_handler(void *am_hdr, +static inline int MPIDI_CH4U_send_long_req_target_handler(int handler_id, void *am_hdr, void **data, size_t * p_data_sz, int *is_contig, @@ -1545,7 +1545,7 @@ static inline int MPIDI_CH4U_send_long_req_target_handler(void *am_hdr, #define FUNCNAME MPIDI_CH4U_send_long_lmt_target_handler #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_CH4U_send_long_lmt_target_handler(void *am_hdr, +static inline int MPIDI_CH4U_send_long_lmt_target_handler(int handler_id, void *am_hdr, void **data, size_t * p_data_sz, int *is_contig, @@ -1574,7 +1574,7 @@ static inline int MPIDI_CH4U_send_long_lmt_target_handler(void *am_hdr, #define FUNCNAME MPIDI_CH4U_ssend_target_handler #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_CH4U_ssend_target_handler(void *am_hdr, +static inline int MPIDI_CH4U_ssend_target_handler(int handler_id, void *am_hdr, void **data, size_t * p_data_sz, int *is_contig, @@ -1587,7 +1587,7 @@ static inline int MPIDI_CH4U_ssend_target_handler(void *am_hdr, MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_SSEND_HANDLER); MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_SSEND_HANDLER); - mpi_errno = MPIDI_CH4U_send_target_handler(am_hdr, + mpi_errno = MPIDI_CH4U_send_target_handler(handler_id, am_hdr, data, p_data_sz, is_contig, cmpl_handler_fn, req); if (mpi_errno) MPIR_ERR_POP(mpi_errno); @@ -1606,7 +1606,7 @@ static inline int MPIDI_CH4U_ssend_target_handler(void *am_hdr, #define FUNCNAME MPIDI_CH4U_ssend_ack_target_handler #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_CH4U_ssend_ack_target_handler(void *am_hdr, +static inline int MPIDI_CH4U_ssend_ack_target_handler(int handler_id, void *am_hdr, void **data, size_t * p_data_sz, int *is_contig, MPIDI_NM_am_completion_handler_fn * @@ -1634,7 +1634,7 @@ static inline int MPIDI_CH4U_ssend_ack_target_handler(void *am_hdr, #define FUNCNAME MPIDI_CH4U_send_long_ack_target_handler #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_CH4U_send_long_ack_target_handler(void *am_hdr, +static inline int MPIDI_CH4U_send_long_ack_target_handler(int handler_id, void *am_hdr, void **data, size_t * p_data_sz, int *is_contig, MPIDI_NM_am_completion_handler_fn * @@ -1679,7 +1679,7 @@ static inline int MPIDI_CH4U_send_long_ack_target_handler(void *am_hdr, #define FUNCNAME MPIDI_CH4U_put_ack_target_handler #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_CH4U_put_ack_target_handler(void *am_hdr, +static inline int MPIDI_CH4U_put_ack_target_handler(int handler_id, void *am_hdr, void **data, size_t * p_data_sz, int *is_contig, MPIDI_NM_am_completion_handler_fn * @@ -1719,7 +1719,7 @@ static inline int MPIDI_CH4U_put_ack_target_handler(void *am_hdr, #define FUNCNAME MPIDI_CH4U_acc_ack_target_handler #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_CH4U_acc_ack_target_handler(void *am_hdr, +static inline int MPIDI_CH4U_acc_ack_target_handler(int handler_id, void *am_hdr, void **data, size_t * p_data_sz, int *is_contig, MPIDI_NM_am_completion_handler_fn * @@ -1759,7 +1759,7 @@ static inline int MPIDI_CH4U_acc_ack_target_handler(void *am_hdr, #define FUNCNAME MPIDI_CH4U_get_acc_ack_target_handler #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_CH4U_get_acc_ack_target_handler(void *am_hdr, +static inline int MPIDI_CH4U_get_acc_ack_target_handler(int handler_id, void *am_hdr, void **data, size_t * p_data_sz, int *is_contig, MPIDI_NM_am_completion_handler_fn * @@ -1831,7 +1831,7 @@ static inline int MPIDI_CH4U_get_acc_ack_target_handler(void *am_hdr, #define FUNCNAME MPIDI_CH4U_cswap_ack_target_handler #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_CH4U_cswap_ack_target_handler(void *am_hdr, +static inline int MPIDI_CH4U_cswap_ack_target_handler(int handler_id, void *am_hdr, void **data, size_t * p_data_sz, int *is_contig, MPIDI_NM_am_completion_handler_fn * @@ -2057,7 +2057,7 @@ static inline void MPIDI_CH4U_win_unlock_done_cb(const MPIDI_CH4U_win_cntrl_msg_ #define FUNCNAME MPIDI_CH4U_win_ctrl_target_handler #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_CH4U_win_ctrl_target_handler(void *am_hdr, +static inline int MPIDI_CH4U_win_ctrl_target_handler(int handler_id, void *am_hdr, void **data, size_t * p_data_sz, int *is_contig, MPIDI_NM_am_completion_handler_fn * @@ -2123,7 +2123,7 @@ static inline int MPIDI_CH4U_win_ctrl_target_handler(void *am_hdr, #define FUNCNAME MPIDI_CH4U_put_target_handler #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_CH4U_put_target_handler(void *am_hdr, +static inline int MPIDI_CH4U_put_target_handler(int handler_id, void *am_hdr, void **data, size_t * p_data_sz, int *is_contig, @@ -2228,7 +2228,7 @@ static inline int MPIDI_CH4U_put_target_handler(void *am_hdr, #define FUNCNAME MPIDI_CH4U_put_iov_target_handler #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_CH4U_put_iov_target_handler(void *am_hdr, +static inline int MPIDI_CH4U_put_iov_target_handler(int handler_id, void *am_hdr, void **data, size_t * p_data_sz, int *is_contig, @@ -2284,7 +2284,7 @@ static inline int MPIDI_CH4U_put_iov_target_handler(void *am_hdr, #define FUNCNAME MPIDI_CH4U_put_iov_ack_target_handler #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_CH4U_put_iov_ack_target_handler(void *am_hdr, +static inline int MPIDI_CH4U_put_iov_ack_target_handler(int handler_id, void *am_hdr, void **data, size_t * p_data_sz, int *is_contig, @@ -2333,7 +2333,7 @@ static inline int MPIDI_CH4U_put_iov_ack_target_handler(void *am_hdr, #define FUNCNAME MPIDI_CH4U_acc_iov_ack_target_handler #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_CH4U_acc_iov_ack_target_handler(void *am_hdr, +static inline int MPIDI_CH4U_acc_iov_ack_target_handler(int handler_id, void *am_hdr, void **data, size_t * p_data_sz, int *is_contig, @@ -2383,7 +2383,7 @@ static inline int MPIDI_CH4U_acc_iov_ack_target_handler(void *am_hdr, #define FUNCNAME MPIDI_CH4U_put_data_target_handler #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_CH4U_put_data_target_handler(void *am_hdr, +static inline int MPIDI_CH4U_put_data_target_handler(int handler_id, void *am_hdr, void **data, size_t * p_data_sz, int *is_contig, @@ -2424,7 +2424,7 @@ static inline int MPIDI_CH4U_put_data_target_handler(void *am_hdr, #define FUNCNAME MPIDI_CH4U_acc_data_target_handler #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_CH4U_acc_data_target_handler(void *am_hdr, +static inline int MPIDI_CH4U_acc_data_target_handler(int handler_id, void *am_hdr, void **data, size_t * p_data_sz, int *is_contig, @@ -2475,7 +2475,7 @@ static inline int MPIDI_CH4U_acc_data_target_handler(void *am_hdr, #define FUNCNAME MPIDI_CH4U_cswap_target_handler #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_CH4U_cswap_target_handler(void *am_hdr, +static inline int MPIDI_CH4U_cswap_target_handler(int handler_id, void *am_hdr, void **data, size_t * p_data_sz, int *is_contig, @@ -2540,7 +2540,7 @@ static inline int MPIDI_CH4U_cswap_target_handler(void *am_hdr, #define FUNCNAME MPIDI_CH4U_handle_acc_request #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_CH4U_handle_acc_request(void *am_hdr, +static inline int MPIDI_CH4U_handle_acc_request(int handler_id, void *am_hdr, void **data, size_t * p_data_sz, int *is_contig, @@ -2627,7 +2627,7 @@ static inline int MPIDI_CH4U_handle_acc_request(void *am_hdr, #define FUNCNAME MPIDI_CH4U_acc_iov_target_handler #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_CH4U_acc_iov_target_handler(void *am_hdr, +static inline int MPIDI_CH4U_acc_iov_target_handler(int handler_id, void *am_hdr, void **data, size_t * p_data_sz, int *is_contig, @@ -2695,7 +2695,7 @@ static inline int MPIDI_CH4U_acc_iov_target_handler(void *am_hdr, #define FUNCNAME MPIDI_CH4U_get_target_handler #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_CH4U_get_target_handler(void *am_hdr, +static inline int MPIDI_CH4U_get_target_handler(int handler_id, void *am_hdr, void **data, size_t * p_data_sz, int *is_contig, @@ -2759,7 +2759,7 @@ static inline int MPIDI_CH4U_get_target_handler(void *am_hdr, #define FUNCNAME MPIDI_CH4U_get_ack_target_handler #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_CH4U_get_ack_target_handler(void *am_hdr, +static inline int MPIDI_CH4U_get_ack_target_handler(int handler_id, void *am_hdr, void **data, size_t * p_data_sz, int *is_contig, http://git.mpich.org/mpich.git/commitdiff/0b72320b7a842e16f3e19dcbacf900a57… commit 0b72320b7a842e16f3e19dcbacf900a57a6a1c61 Author: Pavan Balaji <balaji(a)anl.gov> Date: Wed Aug 24 17:04:14 2016 -0500 CH4: Remove __CH4_INLINE__ Signed-off-by: Ken Raffenetti <raffenet(a)mcs.anl.gov> diff --git a/src/mpid/ch4/include/mpidch4.h b/src/mpid/ch4/include/mpidch4.h index 7c408bd..bffa964 100644 --- a/src/mpid/ch4/include/mpidch4.h +++ b/src/mpid/ch4/include/mpidch4.h @@ -11,16 +11,13 @@ #ifndef MPIDCH4_H_INCLUDED #define MPIDCH4_H_INCLUDED -#define __CH4_INLINE__ __attribute__((always_inline)) static inline - /* We need to define the static inlines right away to avoid * any implicit prototype generation and subsequent warnings * This allows us to make ADI up calls from within a direct * netmod. */ #define MPIDI_CH4I_API(rc,fcnname,...) \ - __CH4_INLINE__ rc MPIDI_##fcnname(__VA_ARGS__) \ - __attribute__((always_inline)) + MPL_STATIC_INLINE_PREFIX rc MPIDI_##fcnname(__VA_ARGS__) MPL_STATIC_INLINE_SUFFIX MPIDI_CH4I_API(int, Init, int *, char ***, int, int *, int *, int *); MPIDI_CH4I_API(int, InitCompleted, void); @@ -239,7 +236,7 @@ MPIDI_CH4I_API(int, Iscatterv, const void *, const int *, const int *, MPI_Datat * similar to the functions above. Other CH4-level functions should call this * function to query locality. This function will determine whether to call the * netmod or CH4U locality functions. */ -__CH4_INLINE__ int MPIDI_CH4_rank_is_local(int rank, MPIR_Comm * comm); +MPL_STATIC_INLINE_PREFIX int MPIDI_CH4_rank_is_local(int rank, MPIR_Comm * comm); /* Include netmod prototypes */ #include <netmod.h> diff --git a/src/mpid/ch4/src/ch4_coll.h b/src/mpid/ch4/src/ch4_coll.h index 03dae8f..595b176 100644 --- a/src/mpid/ch4/src/ch4_coll.h +++ b/src/mpid/ch4/src/ch4_coll.h @@ -14,362 +14,382 @@ #include "ch4_impl.h" #include "ch4r_proc.h" -__CH4_INLINE__ int MPIDI_Barrier(MPIR_Comm * comm, MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_Barrier(MPIR_Comm * comm, MPIR_Errflag_t * errflag) { return MPIDI_NM_barrier(comm, errflag); } -__CH4_INLINE__ int MPIDI_Bcast(void *buffer, int count, MPI_Datatype datatype, - int root, MPIR_Comm * comm, MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_Bcast(void *buffer, int count, MPI_Datatype datatype, + int root, MPIR_Comm * comm, MPIR_Errflag_t * errflag) { return MPIDI_NM_bcast(buffer, count, datatype, root, comm, errflag); } -__CH4_INLINE__ int MPIDI_Allreduce(const void *sendbuf, void *recvbuf, int count, - MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm, - MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_Allreduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm, + MPIR_Errflag_t * errflag) { return MPIDI_NM_allreduce(sendbuf, recvbuf, count, datatype, op, comm, errflag); } -__CH4_INLINE__ int MPIDI_Allgather(const void *sendbuf, int sendcount, MPI_Datatype sendtype, - void *recvbuf, int recvcount, MPI_Datatype recvtype, - MPIR_Comm * comm, MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_Allgather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, int recvcount, + MPI_Datatype recvtype, MPIR_Comm * comm, + MPIR_Errflag_t * errflag) { return MPIDI_NM_allgather(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, comm, errflag); } -__CH4_INLINE__ int MPIDI_Allgatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype, - void *recvbuf, const int *recvcounts, const int *displs, - MPI_Datatype recvtype, MPIR_Comm * comm, - MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_Allgatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, const int *displs, + MPI_Datatype recvtype, MPIR_Comm * comm, + MPIR_Errflag_t * errflag) { return MPIDI_NM_allgatherv(sendbuf, sendcount, sendtype, recvbuf, recvcounts, displs, recvtype, comm, errflag); } -__CH4_INLINE__ int MPIDI_Scatter(const void *sendbuf, int sendcount, MPI_Datatype sendtype, - void *recvbuf, int recvcount, MPI_Datatype recvtype, - int root, MPIR_Comm * comm, MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_Scatter(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, int recvcount, + MPI_Datatype recvtype, int root, MPIR_Comm * comm, + MPIR_Errflag_t * errflag) { return MPIDI_NM_scatter(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, root, comm, errflag); } -__CH4_INLINE__ int MPIDI_Scatterv(const void *sendbuf, const int *sendcounts, - const int *displs, MPI_Datatype sendtype, - void *recvbuf, int recvcount, MPI_Datatype recvtype, - int root, MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_Scatterv(const void *sendbuf, const int *sendcounts, + const int *displs, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + int root, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) { return MPIDI_NM_scatterv(sendbuf, sendcounts, displs, sendtype, recvbuf, recvcount, recvtype, root, comm_ptr, errflag); } -__CH4_INLINE__ int MPIDI_Gather(const void *sendbuf, int sendcount, MPI_Datatype sendtype, - void *recvbuf, int recvcount, MPI_Datatype recvtype, - int root, MPIR_Comm * comm, MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_Gather(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + int root, MPIR_Comm * comm, MPIR_Errflag_t * errflag) { return MPIDI_NM_gather(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, root, comm, errflag); } -__CH4_INLINE__ int MPIDI_Gatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype, - void *recvbuf, const int *recvcounts, - const int *displs, MPI_Datatype recvtype, - int root, MPIR_Comm * comm, MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_Gatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, const int *displs, + MPI_Datatype recvtype, int root, MPIR_Comm * comm, + MPIR_Errflag_t * errflag) { return MPIDI_NM_gatherv(sendbuf, sendcount, sendtype, recvbuf, recvcounts, displs, recvtype, root, comm, errflag); } -__CH4_INLINE__ int MPIDI_Alltoall(const void *sendbuf, int sendcount, MPI_Datatype sendtype, - void *recvbuf, int recvcount, MPI_Datatype recvtype, - MPIR_Comm * comm, MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_Alltoall(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, int recvcount, + MPI_Datatype recvtype, MPIR_Comm * comm, + MPIR_Errflag_t * errflag) { return MPIDI_NM_alltoall(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, comm, errflag); } -__CH4_INLINE__ int MPIDI_Alltoallv(const void *sendbuf, const int *sendcounts, - const int *sdispls, MPI_Datatype sendtype, - void *recvbuf, const int *recvcounts, - const int *rdispls, MPI_Datatype recvtype, - MPIR_Comm * comm, MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_Alltoallv(const void *sendbuf, const int *sendcounts, + const int *sdispls, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, + const int *rdispls, MPI_Datatype recvtype, + MPIR_Comm * comm, MPIR_Errflag_t * errflag) { return MPIDI_NM_alltoallv(sendbuf, sendcounts, sdispls, sendtype, recvbuf, recvcounts, rdispls, recvtype, comm, errflag); } -__CH4_INLINE__ int MPIDI_Alltoallw(const void *sendbuf, const int sendcounts[], - const int sdispls[], const MPI_Datatype sendtypes[], - void *recvbuf, const int recvcounts[], - const int rdispls[], const MPI_Datatype recvtypes[], - MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_Alltoallw(const void *sendbuf, const int sendcounts[], + const int sdispls[], const MPI_Datatype sendtypes[], + void *recvbuf, const int recvcounts[], + const int rdispls[], const MPI_Datatype recvtypes[], + MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) { return MPIDI_NM_alltoallw(sendbuf, sendcounts, sdispls, sendtypes, recvbuf, recvcounts, rdispls, recvtypes, comm_ptr, errflag); } -__CH4_INLINE__ int MPIDI_Reduce(const void *sendbuf, void *recvbuf, - int count, MPI_Datatype datatype, MPI_Op op, - int root, MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_Reduce(const void *sendbuf, void *recvbuf, + int count, MPI_Datatype datatype, MPI_Op op, + int root, MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) { return MPIDI_NM_reduce(sendbuf, recvbuf, count, datatype, op, root, comm_ptr, errflag); } -__CH4_INLINE__ int MPIDI_Reduce_scatter(const void *sendbuf, void *recvbuf, - const int recvcounts[], MPI_Datatype datatype, - MPI_Op op, MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_Reduce_scatter(const void *sendbuf, void *recvbuf, + const int recvcounts[], MPI_Datatype datatype, + MPI_Op op, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) { return MPIDI_NM_reduce_scatter(sendbuf, recvbuf, recvcounts, datatype, op, comm_ptr, errflag); } -__CH4_INLINE__ int MPIDI_Reduce_scatter_block(const void *sendbuf, void *recvbuf, int recvcount, - MPI_Datatype datatype, MPI_Op op, - MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_Reduce_scatter_block(const void *sendbuf, void *recvbuf, + int recvcount, MPI_Datatype datatype, + MPI_Op op, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) { return MPIDI_NM_reduce_scatter_block(sendbuf, recvbuf, recvcount, datatype, op, comm_ptr, errflag); } -__CH4_INLINE__ int MPIDI_Scan(const void *sendbuf, void *recvbuf, int count, - MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm, - MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_Scan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm, + MPIR_Errflag_t * errflag) { return MPIDI_NM_scan(sendbuf, recvbuf, count, datatype, op, comm, errflag); } -__CH4_INLINE__ int MPIDI_Exscan(const void *sendbuf, void *recvbuf, int count, - MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm, - MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_Exscan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm, + MPIR_Errflag_t * errflag) { return MPIDI_NM_exscan(sendbuf, recvbuf, count, datatype, op, comm, errflag); } -__CH4_INLINE__ int MPIDI_Neighbor_allgather(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, int recvcount, - MPI_Datatype recvtype, MPIR_Comm * comm) +MPL_STATIC_INLINE_PREFIX int MPIDI_Neighbor_allgather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm) { return MPIDI_NM_neighbor_allgather(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, comm); } -__CH4_INLINE__ int MPIDI_Neighbor_allgatherv(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - const int *recvcounts, const int *displs, - MPI_Datatype recvtype, MPIR_Comm * comm) +MPL_STATIC_INLINE_PREFIX int MPIDI_Neighbor_allgatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, const int *displs, + MPI_Datatype recvtype, MPIR_Comm * comm) { return MPIDI_NM_neighbor_allgatherv(sendbuf, sendcount, sendtype, recvbuf, recvcounts, displs, recvtype, comm); } -__CH4_INLINE__ int MPIDI_Neighbor_alltoallv(const void *sendbuf, const int *sendcounts, - const int *sdispls, MPI_Datatype sendtype, - void *recvbuf, const int *recvcounts, - const int *rdispls, MPI_Datatype recvtype, - MPIR_Comm * comm) +MPL_STATIC_INLINE_PREFIX int MPIDI_Neighbor_alltoallv(const void *sendbuf, const int *sendcounts, + const int *sdispls, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, + const int *rdispls, MPI_Datatype recvtype, + MPIR_Comm * comm) { return MPIDI_NM_neighbor_alltoallv(sendbuf, sendcounts, sdispls, sendtype, recvbuf, recvcounts, rdispls, recvtype, comm); } -__CH4_INLINE__ int MPIDI_Neighbor_alltoallw(const void *sendbuf, const int *sendcounts, - const MPI_Aint * sdispls, - const MPI_Datatype * sendtypes, void *recvbuf, - const int *recvcounts, const MPI_Aint * rdispls, - const MPI_Datatype * recvtypes, MPIR_Comm * comm) +MPL_STATIC_INLINE_PREFIX int MPIDI_Neighbor_alltoallw(const void *sendbuf, const int *sendcounts, + const MPI_Aint * sdispls, + const MPI_Datatype * sendtypes, void *recvbuf, + const int *recvcounts, + const MPI_Aint * rdispls, + const MPI_Datatype * recvtypes, + MPIR_Comm * comm) { return MPIDI_NM_neighbor_alltoallw(sendbuf, sendcounts, sdispls, sendtypes, recvbuf, recvcounts, rdispls, recvtypes, comm); } -__CH4_INLINE__ int MPIDI_Neighbor_alltoall(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, int recvcount, - MPI_Datatype recvtype, MPIR_Comm * comm) +MPL_STATIC_INLINE_PREFIX int MPIDI_Neighbor_alltoall(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm) { return MPIDI_NM_neighbor_alltoall(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, comm); } -__CH4_INLINE__ int MPIDI_Ineighbor_allgather(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - int recvcount, MPI_Datatype recvtype, - MPIR_Comm * comm, MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_Ineighbor_allgather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, MPI_Request * req) { return MPIDI_NM_ineighbor_allgather(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, comm, req); } -__CH4_INLINE__ int MPIDI_Ineighbor_allgatherv(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - const int *recvcounts, const int *displs, - MPI_Datatype recvtype, MPIR_Comm * comm, - MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_Ineighbor_allgatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, const int *displs, + MPI_Datatype recvtype, MPIR_Comm * comm, + MPI_Request * req) { return MPIDI_NM_ineighbor_allgatherv(sendbuf, sendcount, sendtype, recvbuf, recvcounts, displs, recvtype, comm, req); } -__CH4_INLINE__ int MPIDI_Ineighbor_alltoall(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - int recvcount, MPI_Datatype recvtype, - MPIR_Comm * comm, MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_Ineighbor_alltoall(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, MPI_Request * req) { return MPIDI_NM_ineighbor_alltoall(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, comm, req); } -__CH4_INLINE__ int MPIDI_Ineighbor_alltoallv(const void *sendbuf, const int *sendcounts, - const int *sdispls, MPI_Datatype sendtype, - void *recvbuf, const int *recvcounts, - const int *rdispls, MPI_Datatype recvtype, - MPIR_Comm * comm, MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_Ineighbor_alltoallv(const void *sendbuf, const int *sendcounts, + const int *sdispls, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, + const int *rdispls, MPI_Datatype recvtype, + MPIR_Comm * comm, MPI_Request * req) { return MPIDI_NM_ineighbor_alltoallv(sendbuf, sendcounts, sdispls, sendtype, recvbuf, recvcounts, rdispls, recvtype, comm, req); } -__CH4_INLINE__ int MPIDI_Ineighbor_alltoallw(const void *sendbuf, const int *sendcounts, - const MPI_Aint * sdispls, - const MPI_Datatype * sendtypes, void *recvbuf, - const int *recvcounts, const MPI_Aint * rdispls, - const MPI_Datatype * recvtypes, MPIR_Comm * comm, - MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_Ineighbor_alltoallw(const void *sendbuf, const int *sendcounts, + const MPI_Aint * sdispls, + const MPI_Datatype * sendtypes, + void *recvbuf, const int *recvcounts, + const MPI_Aint * rdispls, + const MPI_Datatype * recvtypes, + MPIR_Comm * comm, MPI_Request * req) { return MPIDI_NM_ineighbor_alltoallw(sendbuf, sendcounts, sdispls, sendtypes, recvbuf, recvcounts, rdispls, recvtypes, comm, req); } -__CH4_INLINE__ int MPIDI_Ibarrier(MPIR_Comm * comm, MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_Ibarrier(MPIR_Comm * comm, MPI_Request * req) { return MPIDI_NM_ibarrier(comm, req); } -__CH4_INLINE__ int MPIDI_Ibcast(void *buffer, int count, MPI_Datatype datatype, - int root, MPIR_Comm * comm, MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_Ibcast(void *buffer, int count, MPI_Datatype datatype, + int root, MPIR_Comm * comm, MPI_Request * req) { return MPIDI_NM_ibcast(buffer, count, datatype, root, comm, req); } -__CH4_INLINE__ int MPIDI_Iallgather(const void *sendbuf, int sendcount, MPI_Datatype sendtype, - void *recvbuf, int recvcount, MPI_Datatype recvtype, - MPIR_Comm * comm, MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_Iallgather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, int recvcount, + MPI_Datatype recvtype, MPIR_Comm * comm, + MPI_Request * req) { return MPIDI_NM_iallgather(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, comm, req); } -__CH4_INLINE__ int MPIDI_Iallgatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype, - void *recvbuf, const int *recvcounts, const int *displs, - MPI_Datatype recvtype, MPIR_Comm * comm, MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_Iallgatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, const int *displs, + MPI_Datatype recvtype, MPIR_Comm * comm, + MPI_Request * req) { return MPIDI_NM_iallgatherv(sendbuf, sendcount, sendtype, recvbuf, recvcounts, displs, recvtype, comm, req); } -__CH4_INLINE__ int MPIDI_Iallreduce(const void *sendbuf, void *recvbuf, int count, - MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm, - MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_Iallreduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm, + MPI_Request * req) { return MPIDI_NM_iallreduce(sendbuf, recvbuf, count, datatype, op, comm, req); } -__CH4_INLINE__ int MPIDI_Ialltoall(const void *sendbuf, int sendcount, MPI_Datatype sendtype, - void *recvbuf, int recvcount, MPI_Datatype recvtype, - MPIR_Comm * comm, MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_Ialltoall(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, int recvcount, + MPI_Datatype recvtype, MPIR_Comm * comm, + MPI_Request * req) { return MPIDI_NM_ialltoall(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, comm, req); } -__CH4_INLINE__ int MPIDI_Ialltoallv(const void *sendbuf, const int *sendcounts, - const int *sdispls, MPI_Datatype sendtype, - void *recvbuf, const int *recvcounts, - const int *rdispls, MPI_Datatype recvtype, - MPIR_Comm * comm, MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_Ialltoallv(const void *sendbuf, const int *sendcounts, + const int *sdispls, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, + const int *rdispls, MPI_Datatype recvtype, + MPIR_Comm * comm, MPI_Request * req) { return MPIDI_NM_ialltoallv(sendbuf, sendcounts, sdispls, sendtype, recvbuf, recvcounts, rdispls, recvtype, comm, req); } -__CH4_INLINE__ int MPIDI_Ialltoallw(const void *sendbuf, const int *sendcounts, - const int *sdispls, const MPI_Datatype * sendtypes, - void *recvbuf, const int *recvcounts, - const int *rdispls, const MPI_Datatype * recvtypes, - MPIR_Comm * comm, MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_Ialltoallw(const void *sendbuf, const int *sendcounts, + const int *sdispls, const MPI_Datatype * sendtypes, + void *recvbuf, const int *recvcounts, + const int *rdispls, const MPI_Datatype * recvtypes, + MPIR_Comm * comm, MPI_Request * req) { return MPIDI_NM_ialltoallw(sendbuf, sendcounts, sdispls, sendtypes, recvbuf, recvcounts, rdispls, recvtypes, comm, req); } -__CH4_INLINE__ int MPIDI_Iexscan(const void *sendbuf, void *recvbuf, int count, - MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm, - MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_Iexscan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm, + MPI_Request * req) { return MPIDI_NM_iexscan(sendbuf, recvbuf, count, datatype, op, comm, req); } -__CH4_INLINE__ int MPIDI_Igather(const void *sendbuf, int sendcount, MPI_Datatype sendtype, - void *recvbuf, int recvcount, MPI_Datatype recvtype, - int root, MPIR_Comm * comm, MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_Igather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, int recvcount, + MPI_Datatype recvtype, int root, MPIR_Comm * comm, + MPI_Request * req) { return MPIDI_NM_igather(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, root, comm, req); } -__CH4_INLINE__ int MPIDI_Igatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype, - void *recvbuf, const int *recvcounts, const int *displs, - MPI_Datatype recvtype, int root, MPIR_Comm * comm, - MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_Igatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, const int *displs, + MPI_Datatype recvtype, int root, MPIR_Comm * comm, + MPI_Request * req) { return MPIDI_NM_igatherv(sendbuf, sendcount, sendtype, recvbuf, recvcounts, displs, recvtype, root, comm, req); } -__CH4_INLINE__ int MPIDI_Ireduce_scatter_block(const void *sendbuf, void *recvbuf, int recvcount, - MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm, - MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_Ireduce_scatter_block(const void *sendbuf, void *recvbuf, + int recvcount, MPI_Datatype datatype, + MPI_Op op, MPIR_Comm * comm, + MPI_Request * req) { return MPIDI_NM_ireduce_scatter_block(sendbuf, recvbuf, recvcount, datatype, op, comm, req); } -__CH4_INLINE__ int MPIDI_Ireduce_scatter(const void *sendbuf, void *recvbuf, - const int *recvcounts, MPI_Datatype datatype, - MPI_Op op, MPIR_Comm * comm, MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_Ireduce_scatter(const void *sendbuf, void *recvbuf, + const int *recvcounts, MPI_Datatype datatype, + MPI_Op op, MPIR_Comm * comm, MPI_Request * req) { return MPIDI_NM_ireduce_scatter(sendbuf, recvbuf, recvcounts, datatype, op, comm, req); } -__CH4_INLINE__ int MPIDI_Ireduce(const void *sendbuf, void *recvbuf, int count, - MPI_Datatype datatype, MPI_Op op, int root, - MPIR_Comm * comm, MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_Ireduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, int root, + MPIR_Comm * comm, MPI_Request * req) { return MPIDI_NM_ireduce(sendbuf, recvbuf, count, datatype, op, root, comm, req); } -__CH4_INLINE__ int MPIDI_Iscan(const void *sendbuf, void *recvbuf, int count, - MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm, - MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_Iscan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm, + MPI_Request * req) { return MPIDI_NM_iscan(sendbuf, recvbuf, count, datatype, op, comm, req); } -__CH4_INLINE__ int MPIDI_Iscatter(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, int recvcount, - MPI_Datatype recvtype, int root, MPIR_Comm * comm, - MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_Iscatter(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, int recvcount, + MPI_Datatype recvtype, int root, MPIR_Comm * comm, + MPI_Request * req) { return MPIDI_NM_iscatter(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, root, comm, req); } -__CH4_INLINE__ int MPIDI_Iscatterv(const void *sendbuf, const int *sendcounts, - const int *displs, MPI_Datatype sendtype, - void *recvbuf, int recvcount, MPI_Datatype recvtype, - int root, MPIR_Comm * comm, MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_Iscatterv(const void *sendbuf, const int *sendcounts, + const int *displs, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + int root, MPIR_Comm * comm, MPI_Request * req) { return MPIDI_NM_iscatterv(sendbuf, sendcounts, displs, sendtype, recvbuf, recvcount, recvtype, root, comm, req); diff --git a/src/mpid/ch4/src/ch4_comm.h b/src/mpid/ch4/src/ch4_comm.h index c5a9520..f849329 100644 --- a/src/mpid/ch4/src/ch4_comm.h +++ b/src/mpid/ch4/src/ch4_comm.h @@ -14,51 +14,54 @@ #include "ch4_impl.h" #include "ch4i_comm.h" -__CH4_INLINE__ int MPIDI_Comm_AS_enabled(MPIR_Comm * comm) +MPL_STATIC_INLINE_PREFIX int MPIDI_Comm_AS_enabled(MPIR_Comm * comm) { MPIR_Assert(0); return MPI_SUCCESS; } -__CH4_INLINE__ int MPIDI_Comm_reenable_anysource(MPIR_Comm * comm, MPIR_Group ** failed_group_ptr) +MPL_STATIC_INLINE_PREFIX int MPIDI_Comm_reenable_anysource(MPIR_Comm * comm, + MPIR_Group ** failed_group_ptr) { MPIR_Assert(0); return MPI_SUCCESS; } -__CH4_INLINE__ int MPIDI_Comm_remote_group_failed(MPIR_Comm * comm, MPIR_Group ** failed_group_ptr) +MPL_STATIC_INLINE_PREFIX int MPIDI_Comm_remote_group_failed(MPIR_Comm * comm, + MPIR_Group ** failed_group_ptr) { MPIR_Assert(0); return MPI_SUCCESS; } -__CH4_INLINE__ int MPIDI_Comm_group_failed(MPIR_Comm * comm_ptr, MPIR_Group ** failed_group_ptr) +MPL_STATIC_INLINE_PREFIX int MPIDI_Comm_group_failed(MPIR_Comm * comm_ptr, + MPIR_Group ** failed_group_ptr) { MPIR_Assert(0); return MPI_SUCCESS; } -__CH4_INLINE__ int MPIDI_Comm_failure_ack(MPIR_Comm * comm_ptr) +MPL_STATIC_INLINE_PREFIX int MPIDI_Comm_failure_ack(MPIR_Comm * comm_ptr) { MPIR_Assert(0); return 0; } -__CH4_INLINE__ int MPIDI_Comm_failure_get_acked(MPIR_Comm * comm_ptr, - MPIR_Group ** failed_group_ptr) +MPL_STATIC_INLINE_PREFIX int MPIDI_Comm_failure_get_acked(MPIR_Comm * comm_ptr, + MPIR_Group ** failed_group_ptr) { MPIR_Assert(0); return 0; } -__CH4_INLINE__ int MPIDI_Comm_revoke(MPIR_Comm * comm_ptr, int is_remote) +MPL_STATIC_INLINE_PREFIX int MPIDI_Comm_revoke(MPIR_Comm * comm_ptr, int is_remote) { MPIR_Assert(0); return 0; } -__CH4_INLINE__ int MPIDI_Comm_get_all_failed_procs(MPIR_Comm * comm_ptr, MPIR_Group ** failed_group, - int tag) +MPL_STATIC_INLINE_PREFIX int MPIDI_Comm_get_all_failed_procs(MPIR_Comm * comm_ptr, + MPIR_Group ** failed_group, int tag) { MPIR_Assert(0); return 0; @@ -68,9 +71,10 @@ __CH4_INLINE__ int MPIDI_Comm_get_all_failed_procs(MPIR_Comm * comm_ptr, MPIR_Gr #define FUNCNAME MPIDI_Comm_split_type #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Comm_split_type(MPIR_Comm * comm_ptr, - int split_type, - int key, MPIR_Info * info_ptr, MPIR_Comm ** newcomm_ptr) +MPL_STATIC_INLINE_PREFIX int MPIDI_Comm_split_type(MPIR_Comm * comm_ptr, + int split_type, + int key, MPIR_Info * info_ptr, + MPIR_Comm ** newcomm_ptr) { int mpi_errno = MPI_SUCCESS; int idx; @@ -95,7 +99,7 @@ __CH4_INLINE__ int MPIDI_Comm_split_type(MPIR_Comm * comm_ptr, #define FUNCNAME MPIDI_Comm_create_hook #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Comm_create_hook(MPIR_Comm * comm) +MPL_STATIC_INLINE_PREFIX int MPIDI_Comm_create_hook(MPIR_Comm * comm) { int mpi_errno; int i, *uniq_avtids; @@ -167,7 +171,7 @@ __CH4_INLINE__ int MPIDI_Comm_create_hook(MPIR_Comm * comm) #define FUNCNAME MPIDI_Comm_free_hook #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Comm_free_hook(MPIR_Comm * comm) +MPL_STATIC_INLINE_PREFIX int MPIDI_Comm_free_hook(MPIR_Comm * comm) { int mpi_errno; int i, *uniq_avtids; diff --git a/src/mpid/ch4/src/ch4_impl.h b/src/mpid/ch4/src/ch4_impl.h index f8a8b75..301d561 100644 --- a/src/mpid/ch4/src/ch4_impl.h +++ b/src/mpid/ch4/src/ch4_impl.h @@ -80,7 +80,7 @@ static inline MPIR_Context_id_t MPIDI_CH4U_win_to_context(const MPIR_Win * win) #define FUNCNAME MPIDI_CH4U_request_release #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ void MPIDI_CH4U_request_release(MPIR_Request * req) +MPL_STATIC_INLINE_PREFIX void MPIDI_CH4U_request_release(MPIR_Request * req) { MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4R_REQUEST_RELEASE); MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4R_REQEUST_RELEASE); @@ -97,7 +97,7 @@ __CH4_INLINE__ void MPIDI_CH4U_request_release(MPIR_Request * req) #define FUNCNAME MPIDI_CH4U_request_complete #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ void MPIDI_CH4U_request_complete(MPIR_Request * req) +MPL_STATIC_INLINE_PREFIX void MPIDI_CH4U_request_complete(MPIR_Request * req) { int incomplete; MPIR_cc_decr(req->cc_ptr, &incomplete); diff --git a/src/mpid/ch4/src/ch4_init.h b/src/mpid/ch4/src/ch4_init.h index 2365b85..7d8c10e 100644 --- a/src/mpid/ch4/src/ch4_init.h +++ b/src/mpid/ch4/src/ch4_init.h @@ -145,9 +145,9 @@ static inline int MPIDI_choose_shm(void) #define FUNCNAME MPIDI_Init #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Init(int *argc, - char ***argv, - int requested, int *provided, int *has_args, int *has_env) +MPL_STATIC_INLINE_PREFIX int MPIDI_Init(int *argc, + char ***argv, + int requested, int *provided, int *has_args, int *has_env) { int pmi_errno, mpi_errno = MPI_SUCCESS, rank, has_parent, size, appnum, thr_err; void *netmod_contexts; @@ -326,7 +326,7 @@ __CH4_INLINE__ int MPIDI_Init(int *argc, #define FUNCNAME MPIDI_InitCompleted #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_InitCompleted(void) +MPL_STATIC_INLINE_PREFIX int MPIDI_InitCompleted(void) { MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_INITCOMPLETED); MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_INITCOMPLETED); @@ -339,7 +339,7 @@ __CH4_INLINE__ int MPIDI_InitCompleted(void) #define FUNCNAME MPIDI_Finalize #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Finalize(void) +MPL_STATIC_INLINE_PREFIX int MPIDI_Finalize(void) { int mpi_errno, thr_err; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_FINALIZE); @@ -380,7 +380,7 @@ __CH4_INLINE__ int MPIDI_Finalize(void) #define FUNCNAME MPIDI_Get_universe_size #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Get_universe_size(int *universe_size) +MPL_STATIC_INLINE_PREFIX int MPIDI_Get_universe_size(int *universe_size) { int mpi_errno = MPI_SUCCESS; int pmi_errno = PMI_SUCCESS; @@ -408,7 +408,7 @@ __CH4_INLINE__ int MPIDI_Get_universe_size(int *universe_size) #define FUNCNAME MPIDI_Get_processor_name #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Get_processor_name(char *name, int namelen, int *resultlen) +MPL_STATIC_INLINE_PREFIX int MPIDI_Get_processor_name(char *name, int namelen, int *resultlen) { int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_GET_PROCESSOR_NAME); @@ -451,8 +451,8 @@ __CH4_INLINE__ int MPIDI_Get_processor_name(char *name, int namelen, int *result #define FUNCNAME MPIDI_Abort #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Abort(MPIR_Comm * comm, - int mpi_errno, int exit_code, const char *error_msg) +MPL_STATIC_INLINE_PREFIX int MPIDI_Abort(MPIR_Comm * comm, + int mpi_errno, int exit_code, const char *error_msg) { char sys_str[MPI_MAX_ERROR_STRING + 5] = ""; char comm_str[MPI_MAX_ERROR_STRING] = ""; @@ -495,7 +495,7 @@ __CH4_INLINE__ int MPIDI_Abort(MPIR_Comm * comm, #define FUNCNAME MPIDI_Alloc_mem #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ void *MPIDI_Alloc_mem(size_t size, MPIR_Info * info_ptr) +MPL_STATIC_INLINE_PREFIX void *MPIDI_Alloc_mem(size_t size, MPIR_Info * info_ptr) { void *p; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_ALLOC_MEM); @@ -511,7 +511,7 @@ __CH4_INLINE__ void *MPIDI_Alloc_mem(size_t size, MPIR_Info * info_ptr) #define FUNCNAME MPIDI_Free_mem #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Free_mem(void *ptr) +MPL_STATIC_INLINE_PREFIX int MPIDI_Free_mem(void *ptr) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_FREE_MEM); @@ -533,8 +533,8 @@ __CH4_INLINE__ int MPIDI_Free_mem(void *ptr) #define FUNCNAME MPIDI_Comm_get_lpid #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Comm_get_lpid(MPIR_Comm * comm_ptr, - int idx, int *lpid_ptr, MPL_bool is_remote) +MPL_STATIC_INLINE_PREFIX int MPIDI_Comm_get_lpid(MPIR_Comm * comm_ptr, + int idx, int *lpid_ptr, MPL_bool is_remote) { int mpi_errno = MPI_SUCCESS; int avtid = 0, lpid = 0; @@ -559,7 +559,7 @@ __CH4_INLINE__ int MPIDI_Comm_get_lpid(MPIR_Comm * comm_ptr, #define FUNCNAME MPIDI_GPID_Get #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_GPID_Get(MPIR_Comm * comm_ptr, int rank, MPIR_Gpid * gpid) +MPL_STATIC_INLINE_PREFIX int MPIDI_GPID_Get(MPIR_Comm * comm_ptr, int rank, MPIR_Gpid * gpid) { int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_GPID_GET); @@ -583,7 +583,7 @@ __CH4_INLINE__ int MPIDI_GPID_Get(MPIR_Comm * comm_ptr, int rank, MPIR_Gpid * gp #define FUNCNAME MPIDI_Get_node_id #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Get_node_id(MPIR_Comm * comm, int rank, MPID_Node_id_t * id_p) +MPL_STATIC_INLINE_PREFIX int MPIDI_Get_node_id(MPIR_Comm * comm, int rank, MPID_Node_id_t * id_p) { int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_GET_NODE_ID); @@ -599,7 +599,7 @@ __CH4_INLINE__ int MPIDI_Get_node_id(MPIR_Comm * comm, int rank, MPID_Node_id_t #define FUNCNAME MPIDI_Get_max_node_id #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Get_max_node_id(MPIR_Comm * comm, MPID_Node_id_t * max_id_p) +MPL_STATIC_INLINE_PREFIX int MPIDI_Get_max_node_id(MPIR_Comm * comm, MPID_Node_id_t * max_id_p) { int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_GET_MAX_NODE_ID); @@ -615,8 +615,9 @@ __CH4_INLINE__ int MPIDI_Get_max_node_id(MPIR_Comm * comm, MPID_Node_id_t * max_ #define FUNCNAME MPIDI_GetAllInComm #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_GPID_GetAllInComm(MPIR_Comm * comm_ptr, - int local_size, MPIR_Gpid local_gpids[], int *singleAVT) +MPL_STATIC_INLINE_PREFIX int MPIDI_GPID_GetAllInComm(MPIR_Comm * comm_ptr, + int local_size, MPIR_Gpid local_gpids[], + int *singleAVT) { int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_GETALLINCOMM); @@ -639,7 +640,7 @@ __CH4_INLINE__ int MPIDI_GPID_GetAllInComm(MPIR_Comm * comm_ptr, #define FUNCNAME MPIDI_GPID_ToLpidArray #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_GPID_ToLpidArray(int size, MPIR_Gpid gpid[], int lpid[]) +MPL_STATIC_INLINE_PREFIX int MPIDI_GPID_ToLpidArray(int size, MPIR_Gpid gpid[], int lpid[]) { int mpi_errno = MPI_SUCCESS, i; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_GPID_TOLPIDARRAY); @@ -683,8 +684,8 @@ __CH4_INLINE__ int MPIDI_GPID_ToLpidArray(int size, MPIR_Gpid gpid[], int lpid[] #define FUNCNAME MPIDI_Create_intercomm_from_lpids #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Create_intercomm_from_lpids(MPIR_Comm * newcomm_ptr, - int size, const int lpids[]) +MPL_STATIC_INLINE_PREFIX int MPIDI_Create_intercomm_from_lpids(MPIR_Comm * newcomm_ptr, + int size, const int lpids[]) { int mpi_errno = MPI_SUCCESS, i; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_CREATE_INTERCOMM_FROM_LPIDS); @@ -722,7 +723,7 @@ __CH4_INLINE__ int MPIDI_Create_intercomm_from_lpids(MPIR_Comm * newcomm_ptr, #define FUNCNAME MPIDI_Aint_add #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ MPI_Aint MPIDI_Aint_add(MPI_Aint base, MPI_Aint disp) +MPL_STATIC_INLINE_PREFIX MPI_Aint MPIDI_Aint_add(MPI_Aint base, MPI_Aint disp) { MPI_Aint result; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_AINT_ADD); @@ -736,7 +737,7 @@ __CH4_INLINE__ MPI_Aint MPIDI_Aint_add(MPI_Aint base, MPI_Aint disp) #define FUNCNAME MPIDI_Aint_diff #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ MPI_Aint MPIDI_Aint_diff(MPI_Aint addr1, MPI_Aint addr2) +MPL_STATIC_INLINE_PREFIX MPI_Aint MPIDI_Aint_diff(MPI_Aint addr1, MPI_Aint addr2) { MPI_Aint result; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_AINT_DIFF); @@ -752,7 +753,7 @@ __CH4_INLINE__ MPI_Aint MPIDI_Aint_diff(MPI_Aint addr1, MPI_Aint addr2) #define FUNCNAME MPIDI_Type_create_hook #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Type_create_hook(MPIR_Datatype * type) +MPL_STATIC_INLINE_PREFIX int MPIDI_Type_create_hook(MPIR_Datatype * type) { int mpi_errno; @@ -782,7 +783,7 @@ __CH4_INLINE__ int MPIDI_Type_create_hook(MPIR_Datatype * type) #define FUNCNAME MPIDI_Type_free_hook #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Type_free_hook(MPIR_Datatype * type) +MPL_STATIC_INLINE_PREFIX int MPIDI_Type_free_hook(MPIR_Datatype * type) { int mpi_errno; @@ -812,7 +813,7 @@ __CH4_INLINE__ int MPIDI_Type_free_hook(MPIR_Datatype * type) #define FUNCNAME MPIDI_Op_create_hook #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Op_create_hook(MPIR_Op * op) +MPL_STATIC_INLINE_PREFIX int MPIDI_Op_create_hook(MPIR_Op * op) { int mpi_errno; @@ -842,7 +843,7 @@ __CH4_INLINE__ int MPIDI_Op_create_hook(MPIR_Op * op) #define FUNCNAME MPIDI_Op_free_hook #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Op_free_hook(MPIR_Op * op) +MPL_STATIC_INLINE_PREFIX int MPIDI_Op_free_hook(MPIR_Op * op) { int mpi_errno; diff --git a/src/mpid/ch4/src/ch4_probe.h b/src/mpid/ch4/src/ch4_probe.h index 89b7bc9..2e43468 100644 --- a/src/mpid/ch4/src/ch4_probe.h +++ b/src/mpid/ch4/src/ch4_probe.h @@ -17,8 +17,9 @@ #define FUNCNAME MPIDI_Probe #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Probe(int source, - int tag, MPIR_Comm * comm, int context_offset, MPI_Status * status) +MPL_STATIC_INLINE_PREFIX int MPIDI_Probe(int source, + int tag, MPIR_Comm * comm, int context_offset, + MPI_Status * status) { int mpi_errno, flag = 0; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_PROBE); @@ -72,10 +73,11 @@ __CH4_INLINE__ int MPIDI_Probe(int source, #define FUNCNAME MPIDI_Mprobe #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Mprobe(int source, - int tag, - MPIR_Comm * comm, - int context_offset, MPIR_Request ** message, MPI_Status * status) +MPL_STATIC_INLINE_PREFIX int MPIDI_Mprobe(int source, + int tag, + MPIR_Comm * comm, + int context_offset, MPIR_Request ** message, + MPI_Status * status) { int mpi_errno = MPI_SUCCESS, flag = 0; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_MPROBE); @@ -132,11 +134,11 @@ __CH4_INLINE__ int MPIDI_Mprobe(int source, #define FUNCNAME MPIDI_Improbe #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Improbe(int source, - int tag, - MPIR_Comm * comm, - int context_offset, - int *flag, MPIR_Request ** message, MPI_Status * status) +MPL_STATIC_INLINE_PREFIX int MPIDI_Improbe(int source, + int tag, + MPIR_Comm * comm, + int context_offset, + int *flag, MPIR_Request ** message, MPI_Status * status) { int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_IMPROBE); @@ -177,10 +179,10 @@ __CH4_INLINE__ int MPIDI_Improbe(int source, #define FUNCNAME MPIDI_Iprobe #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Iprobe(int source, - int tag, - MPIR_Comm * comm, - int context_offset, int *flag, MPI_Status * status) +MPL_STATIC_INLINE_PREFIX int MPIDI_Iprobe(int source, + int tag, + MPIR_Comm * comm, + int context_offset, int *flag, MPI_Status * status) { int mpi_errno; diff --git a/src/mpid/ch4/src/ch4_proc.h b/src/mpid/ch4/src/ch4_proc.h index 4aefcf0..7666a82 100644 --- a/src/mpid/ch4/src/ch4_proc.h +++ b/src/mpid/ch4/src/ch4_proc.h @@ -17,7 +17,7 @@ #define FUNCNAME MPIDI_CH4_rank_is_local #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_CH4_rank_is_local(int rank, MPIR_Comm * comm) +MPL_STATIC_INLINE_PREFIX int MPIDI_CH4_rank_is_local(int rank, MPIR_Comm * comm) { int ret; MPIR_FUNC_VERBOSE_STATE_DECL(MPIDI_STATE_CH4_RANK_IS_LOCAL); diff --git a/src/mpid/ch4/src/ch4_progress.h b/src/mpid/ch4/src/ch4_progress.h index 37bffef..66a022d 100644 --- a/src/mpid/ch4/src/ch4_progress.h +++ b/src/mpid/ch4/src/ch4_progress.h @@ -17,7 +17,7 @@ #define FUNCNAME MPIDI_Progress_test #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Progress_test(void) +MPL_STATIC_INLINE_PREFIX int MPIDI_Progress_test(void) { int mpi_errno, made_progress, i; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_PROGRESS_TEST); @@ -56,22 +56,22 @@ __CH4_INLINE__ int MPIDI_Progress_test(void) goto fn_exit;; } -__CH4_INLINE__ int MPIDI_Progress_poke(void) +MPL_STATIC_INLINE_PREFIX int MPIDI_Progress_poke(void) { return MPIDI_Progress_test(); } -__CH4_INLINE__ void MPIDI_Progress_start(MPID_Progress_state * state) +MPL_STATIC_INLINE_PREFIX void MPIDI_Progress_start(MPID_Progress_state * state) { return; } -__CH4_INLINE__ void MPIDI_Progress_end(MPID_Progress_state * state) +MPL_STATIC_INLINE_PREFIX void MPIDI_Progress_end(MPID_Progress_state * state) { return; } -__CH4_INLINE__ int MPIDI_Progress_wait(MPID_Progress_state * state) +MPL_STATIC_INLINE_PREFIX int MPIDI_Progress_wait(MPID_Progress_state * state) { return MPIDI_Progress_test(); } @@ -81,7 +81,7 @@ __CH4_INLINE__ int MPIDI_Progress_wait(MPID_Progress_state * state) #define FUNCNAME MPIDI_Progress_register #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Progress_register(int (*progress_fn) (int *), int *id) +MPL_STATIC_INLINE_PREFIX int MPIDI_Progress_register(int (*progress_fn) (int *), int *id) { int mpi_errno = MPI_SUCCESS; int i; @@ -119,7 +119,7 @@ __CH4_INLINE__ int MPIDI_Progress_register(int (*progress_fn) (int *), int *id) #define FUNCNAME MPIDI_Progress_deregister #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Progress_deregister(int id) +MPL_STATIC_INLINE_PREFIX int MPIDI_Progress_deregister(int id) { int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDI_PROGRESS_DEREGISTER); @@ -142,7 +142,7 @@ __CH4_INLINE__ int MPIDI_Progress_deregister(int id) #define FUNCNAME MPIDI_Progress_activate #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Progress_activate(int id) +MPL_STATIC_INLINE_PREFIX int MPIDI_Progress_activate(int id) { int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDI_PROGRESS_ACTIVATE); @@ -164,7 +164,7 @@ __CH4_INLINE__ int MPIDI_Progress_activate(int id) #define FUNCNAME MPIDI_Progress_deactivate #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Progress_deactivate(int id) +MPL_STATIC_INLINE_PREFIX int MPIDI_Progress_deactivate(int id) { int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDI_PROGRESS_DEACTIVATE); diff --git a/src/mpid/ch4/src/ch4_recv.h b/src/mpid/ch4/src/ch4_recv.h index 71af9db..a6ab84a 100644 --- a/src/mpid/ch4/src/ch4_recv.h +++ b/src/mpid/ch4/src/ch4_recv.h @@ -17,13 +17,14 @@ #define FUNCNAME MPIDI_Recv #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Recv(void *buf, - int count, - MPI_Datatype datatype, - int rank, - int tag, - MPIR_Comm * comm, - int context_offset, MPI_Status * status, MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_Recv(void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, + int context_offset, MPI_Status * status, + MPIR_Request ** request) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_RECV); @@ -104,12 +105,13 @@ __CH4_INLINE__ int MPIDI_Recv(void *buf, #define FUNCNAME MPIDI_Recv_init #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Recv_init(void *buf, - int count, - MPI_Datatype datatype, - int rank, - int tag, - MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_Recv_init(void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_RECV_INIT); @@ -167,9 +169,10 @@ __CH4_INLINE__ int MPIDI_Recv_init(void *buf, #define FUNCNAME MPIDI_Mrecv #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Mrecv(void *buf, - int count, - MPI_Datatype datatype, MPIR_Request * message, MPI_Status * status) +MPL_STATIC_INLINE_PREFIX int MPIDI_Mrecv(void *buf, + int count, + MPI_Datatype datatype, MPIR_Request * message, + MPI_Status * status) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_MRECV); @@ -256,10 +259,10 @@ __CH4_INLINE__ int MPIDI_Mrecv(void *buf, #define FUNCNAME MPIDI_Imrecv #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Imrecv(void *buf, - int count, - MPI_Datatype datatype, - MPIR_Request * message, MPIR_Request ** rreqp) +MPL_STATIC_INLINE_PREFIX int MPIDI_Imrecv(void *buf, + int count, + MPI_Datatype datatype, + MPIR_Request * message, MPIR_Request ** rreqp) { int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_IMRECV); @@ -311,12 +314,13 @@ __CH4_INLINE__ int MPIDI_Imrecv(void *buf, #define FUNCNAME MPIDI_Irecv #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Irecv(void *buf, - int count, - MPI_Datatype datatype, - int rank, - int tag, - MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_Irecv(void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_IRECV); @@ -385,7 +389,7 @@ __CH4_INLINE__ int MPIDI_Irecv(void *buf, #define FUNCNAME MPIDI_Cancel_Recv #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Cancel_recv(MPIR_Request * rreq) +MPL_STATIC_INLINE_PREFIX int MPIDI_Cancel_recv(MPIR_Request * rreq) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_CANCEL_RECV); diff --git a/src/mpid/ch4/src/ch4_request.h b/src/mpid/ch4/src/ch4_request.h index ac11dac..a38a5fa 100644 --- a/src/mpid/ch4/src/ch4_request.h +++ b/src/mpid/ch4/src/ch4_request.h @@ -14,31 +14,31 @@ #include "ch4_impl.h" #include "ch4r_buf.h" -__CH4_INLINE__ int MPIDI_Request_is_anysource(MPIR_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_Request_is_anysource(MPIR_Request * req) { MPIR_Assert(0); return MPI_SUCCESS; } -__CH4_INLINE__ int MPIDI_Request_is_pending_failure(MPIR_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_Request_is_pending_failure(MPIR_Request * req) { MPIR_Assert(0); return MPI_SUCCESS; } -__CH4_INLINE__ void MPIDI_Request_set_completed(MPIR_Request * req) +MPL_STATIC_INLINE_PREFIX void MPIDI_Request_set_completed(MPIR_Request * req) { MPIR_cc_set(&req->cc, 0); return; } -__CH4_INLINE__ void MPIDI_Request_add_ref(MPIR_Request * req) +MPL_STATIC_INLINE_PREFIX void MPIDI_Request_add_ref(MPIR_Request * req) { MPIR_Request_add_ref(req); return; } -__CH4_INLINE__ void MPIDI_Request_release_ref(MPIR_Request * req) +MPL_STATIC_INLINE_PREFIX void MPIDI_Request_release_ref(MPIR_Request * req) { int inuse; MPIR_Request_release_ref(req, &inuse); @@ -80,7 +80,7 @@ __CH4_INLINE__ void MPIDI_Request_release_ref(MPIR_Request * req) #define FUNCNAME MPIDI_request_complete #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Request_complete(MPIR_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_Request_complete(MPIR_Request * req) { int incomplete; MPIR_cc_decr(req->cc_ptr, &incomplete); diff --git a/src/mpid/ch4/src/ch4_rma.h b/src/mpid/ch4/src/ch4_rma.h index a85cdb8..7eabd9d 100644 --- a/src/mpid/ch4/src/ch4_rma.h +++ b/src/mpid/ch4/src/ch4_rma.h @@ -17,12 +17,13 @@ #define FUNCNAME MPIDI_Put #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Put(const void *origin_addr, - int origin_count, - MPI_Datatype origin_datatype, - int target_rank, - MPI_Aint target_disp, - int target_count, MPI_Datatype target_datatype, MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_Put(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, MPI_Datatype target_datatype, + MPIR_Win * win) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_PUT); @@ -43,12 +44,13 @@ __CH4_INLINE__ int MPIDI_Put(const void *origin_addr, #define FUNCNAME MPIDI_Get #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Get(void *origin_addr, - int origin_count, - MPI_Datatype origin_datatype, - int target_rank, - MPI_Aint target_disp, - int target_count, MPI_Datatype target_datatype, MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_Get(void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, MPI_Datatype target_datatype, + MPIR_Win * win) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_GET); @@ -69,13 +71,14 @@ __CH4_INLINE__ int MPIDI_Get(void *origin_addr, #define FUNCNAME MPIDI_Accumulate #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Accumulate(const void *origin_addr, - int origin_count, - MPI_Datatype origin_datatype, - int target_rank, - MPI_Aint target_disp, - int target_count, - MPI_Datatype target_datatype, MPI_Op op, MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_Accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, MPI_Op op, + MPIR_Win * win) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_ACCUMULATE); @@ -97,11 +100,12 @@ __CH4_INLINE__ int MPIDI_Accumulate(const void *origin_addr, #define FUNCNAME MPIDI_Compare_and_swap #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Compare_and_swap(const void *origin_addr, - const void *compare_addr, - void *result_addr, - MPI_Datatype datatype, - int target_rank, MPI_Aint target_disp, MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_Compare_and_swap(const void *origin_addr, + const void *compare_addr, + void *result_addr, + MPI_Datatype datatype, + int target_rank, MPI_Aint target_disp, + MPIR_Win * win) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_COMPARE_AND_SWAP); @@ -122,14 +126,14 @@ __CH4_INLINE__ int MPIDI_Compare_and_swap(const void *origin_addr, #define FUNCNAME MPIDI_Raccumulate #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Raccumulate(const void *origin_addr, - int origin_count, - MPI_Datatype origin_datatype, - int target_rank, - MPI_Aint target_disp, - int target_count, - MPI_Datatype target_datatype, - MPI_Op op, MPIR_Win * win, MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_Raccumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPI_Op op, MPIR_Win * win, MPIR_Request ** request) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_RACCUMULATE); @@ -151,17 +155,18 @@ __CH4_INLINE__ int MPIDI_Raccumulate(const void *origin_addr, #define FUNCNAME MPIDI_Rget_accumulate #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Rget_accumulate(const void *origin_addr, - int origin_count, - MPI_Datatype origin_datatype, - void *result_addr, - int result_count, - MPI_Datatype result_datatype, - int target_rank, - MPI_Aint target_disp, - int target_count, - MPI_Datatype target_datatype, - MPI_Op op, MPIR_Win * win, MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_Rget_accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + void *result_addr, + int result_count, + MPI_Datatype result_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPI_Op op, MPIR_Win * win, + MPIR_Request ** request) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_RGET_ACCUMULATE); @@ -184,11 +189,11 @@ __CH4_INLINE__ int MPIDI_Rget_accumulate(const void *origin_addr, #define FUNCNAME MPIDI_Fetch_and_op #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Fetch_and_op(const void *origin_addr, - void *result_addr, - MPI_Datatype datatype, - int target_rank, - MPI_Aint target_disp, MPI_Op op, MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_Fetch_and_op(const void *origin_addr, + void *result_addr, + MPI_Datatype datatype, + int target_rank, + MPI_Aint target_disp, MPI_Op op, MPIR_Win * win) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_FETCH_AND_OP); @@ -210,13 +215,14 @@ __CH4_INLINE__ int MPIDI_Fetch_and_op(const void *origin_addr, #define FUNCNAME MPIDI_Rget #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Rget(void *origin_addr, - int origin_count, - MPI_Datatype origin_datatype, - int target_rank, - MPI_Aint target_disp, - int target_count, - MPI_Datatype target_datatype, MPIR_Win * win, MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_Rget(void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, MPIR_Win * win, + MPIR_Request ** request) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_RGET); @@ -238,13 +244,14 @@ __CH4_INLINE__ int MPIDI_Rget(void *origin_addr, #define FUNCNAME MPIDI_Rput #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Rput(const void *origin_addr, - int origin_count, - MPI_Datatype origin_datatype, - int target_rank, - MPI_Aint target_disp, - int target_count, - MPI_Datatype target_datatype, MPIR_Win * win, MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_Rput(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, MPIR_Win * win, + MPIR_Request ** request) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_RPUT); @@ -266,16 +273,17 @@ __CH4_INLINE__ int MPIDI_Rput(const void *origin_addr, #define FUNCNAME MPIDI_Get_accumulate #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Get_accumulate(const void *origin_addr, - int origin_count, - MPI_Datatype origin_datatype, - void *result_addr, - int result_count, - MPI_Datatype result_datatype, - int target_rank, - MPI_Aint target_disp, - int target_count, - MPI_Datatype target_datatype, MPI_Op op, MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_Get_accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + void *result_addr, + int result_count, + MPI_Datatype result_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, MPI_Op op, + MPIR_Win * win) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_GET_ACCUMULATE); diff --git a/src/mpid/ch4/src/ch4_send.h b/src/mpid/ch4/src/ch4_send.h index 318e79b..c719115 100644 --- a/src/mpid/ch4/src/ch4_send.h +++ b/src/mpid/ch4/src/ch4_send.h @@ -17,12 +17,13 @@ #define FUNCNAME MPIDI_Send #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Send(const void *buf, - int count, - MPI_Datatype datatype, - int rank, - int tag, - MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_Send(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_SEND); @@ -62,12 +63,13 @@ __CH4_INLINE__ int MPIDI_Send(const void *buf, #define FUNCNAME MPIDI_Isend #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Isend(const void *buf, - int count, - MPI_Datatype datatype, - int rank, - int tag, - MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_Isend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_ISEND); @@ -108,12 +110,13 @@ __CH4_INLINE__ int MPIDI_Isend(const void *buf, #define FUNCNAME MPIDI_Rsend #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Rsend(const void *buf, - int count, - MPI_Datatype datatype, - int rank, - int tag, - MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_Rsend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_RSEND); @@ -154,12 +157,13 @@ __CH4_INLINE__ int MPIDI_Rsend(const void *buf, #define FUNCNAME MPIDI_Irsend #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Irsend(const void *buf, - int count, - MPI_Datatype datatype, - int rank, - int tag, - MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_Irsend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_IRSEND); @@ -199,12 +203,13 @@ __CH4_INLINE__ int MPIDI_Irsend(const void *buf, #define FUNCNAME MPIDI_Ssend #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Ssend(const void *buf, - int count, - MPI_Datatype datatype, - int rank, - int tag, - MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_Ssend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_SSEND); @@ -244,12 +249,13 @@ __CH4_INLINE__ int MPIDI_Ssend(const void *buf, #define FUNCNAME MPIDI_Issend #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Issend(const void *buf, - int count, - MPI_Datatype datatype, - int rank, - int tag, - MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_Issend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_ISSEND); @@ -290,7 +296,7 @@ __CH4_INLINE__ int MPIDI_Issend(const void *buf, #define FUNCNAME MPIDI_Startall #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Startall(int count, MPIR_Request * requests[]) +MPL_STATIC_INLINE_PREFIX int MPIDI_Startall(int count, MPIR_Request * requests[]) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_STARTALL); @@ -333,12 +339,13 @@ __CH4_INLINE__ int MPIDI_Startall(int count, MPIR_Request * requests[]) #define FUNCNAME MPIDI_Send_init #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Send_init(const void *buf, - int count, - MPI_Datatype datatype, - int rank, - int tag, - MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_Send_init(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_SEND_INIT); @@ -371,12 +378,13 @@ __CH4_INLINE__ int MPIDI_Send_init(const void *buf, #define FUNCNAME MPIDI_Ssend_init #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Ssend_init(const void *buf, - int count, - MPI_Datatype datatype, - int rank, - int tag, - MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_Ssend_init(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_SSEND_INIT); @@ -410,12 +418,13 @@ __CH4_INLINE__ int MPIDI_Ssend_init(const void *buf, #define FUNCNAME MPIDI_Bsend_init #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Bsend_init(const void *buf, - int count, - MPI_Datatype datatype, - int rank, - int tag, - MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_Bsend_init(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_BSEND_INIT); @@ -449,12 +458,13 @@ __CH4_INLINE__ int MPIDI_Bsend_init(const void *buf, #define FUNCNAME MPIDI_Rsend_init #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Rsend_init(const void *buf, - int count, - MPI_Datatype datatype, - int rank, - int tag, - MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_Rsend_init(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_RSEND_INIT); @@ -489,7 +499,7 @@ __CH4_INLINE__ int MPIDI_Rsend_init(const void *buf, #define FUNCNAME MPIDI_Cancel_send #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Cancel_send(MPIR_Request * sreq) +MPL_STATIC_INLINE_PREFIX int MPIDI_Cancel_send(MPIR_Request * sreq) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_CANCEL_SEND); diff --git a/src/mpid/ch4/src/ch4_spawn.h b/src/mpid/ch4/src/ch4_spawn.h index d4968a1..07290cc 100644 --- a/src/mpid/ch4/src/ch4_spawn.h +++ b/src/mpid/ch4/src/ch4_spawn.h @@ -70,14 +70,14 @@ static inline void MPIDI_free_pmi_keyvals(PMI_keyval_t ** kv, int size, int *cou #define FUNCNAME MPIDI_Comm_spawn_multiple #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Comm_spawn_multiple(int count, - char *commands[], - char **argvs[], - const int maxprocs[], - MPIR_Info * info_ptrs[], - int root, - MPIR_Comm * comm_ptr, - MPIR_Comm ** intercomm, int errcodes[]) +MPL_STATIC_INLINE_PREFIX int MPIDI_Comm_spawn_multiple(int count, + char *commands[], + char **argvs[], + const int maxprocs[], + MPIR_Info * info_ptrs[], + int root, + MPIR_Comm * comm_ptr, + MPIR_Comm ** intercomm, int errcodes[]) { char port_name[MPI_MAX_PORT_NAME]; int *info_keyval_sizes = 0, i, mpi_errno = MPI_SUCCESS; @@ -206,9 +206,10 @@ __CH4_INLINE__ int MPIDI_Comm_spawn_multiple(int count, #define FUNCNAME MPIDI_Comm_connect #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Comm_connect(const char *port_name, - MPIR_Info * info, - int root, MPIR_Comm * comm, MPIR_Comm ** newcomm_ptr) +MPL_STATIC_INLINE_PREFIX int MPIDI_Comm_connect(const char *port_name, + MPIR_Info * info, + int root, MPIR_Comm * comm, + MPIR_Comm ** newcomm_ptr) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_COMM_CONNECT); @@ -230,7 +231,7 @@ __CH4_INLINE__ int MPIDI_Comm_connect(const char *port_name, #define FUNCNAME MPIDI_Comm_disconnect #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Comm_disconnect(MPIR_Comm * comm_ptr) +MPL_STATIC_INLINE_PREFIX int MPIDI_Comm_disconnect(MPIR_Comm * comm_ptr) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_COMM_DISCONNECT); @@ -252,7 +253,7 @@ __CH4_INLINE__ int MPIDI_Comm_disconnect(MPIR_Comm * comm_ptr) #define FUNCNAME MPIDI_Open_port #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Open_port(MPIR_Info * info_ptr, char *port_name) +MPL_STATIC_INLINE_PREFIX int MPIDI_Open_port(MPIR_Info * info_ptr, char *port_name) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_OPEN_PORT); @@ -274,7 +275,7 @@ __CH4_INLINE__ int MPIDI_Open_port(MPIR_Info * info_ptr, char *port_name) #define FUNCNAME MPIDI_Close_port #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Close_port(const char *port_name) +MPL_STATIC_INLINE_PREFIX int MPIDI_Close_port(const char *port_name) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_CLOSE_PORT); @@ -296,9 +297,9 @@ __CH4_INLINE__ int MPIDI_Close_port(const char *port_name) #define FUNCNAME MPIDI_Comm_accept #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Comm_accept(const char *port_name, - MPIR_Info * info, - int root, MPIR_Comm * comm, MPIR_Comm ** newcomm_ptr) +MPL_STATIC_INLINE_PREFIX int MPIDI_Comm_accept(const char *port_name, + MPIR_Info * info, + int root, MPIR_Comm * comm, MPIR_Comm ** newcomm_ptr) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_COMM_ACCEPT); diff --git a/src/mpid/ch4/src/ch4_win.h b/src/mpid/ch4/src/ch4_win.h index 222f7fe..12f899d 100644 --- a/src/mpid/ch4/src/ch4_win.h +++ b/src/mpid/ch4/src/ch4_win.h @@ -17,7 +17,7 @@ #define FUNCNAME MPIDI_Win_set_info #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Win_set_info(MPIR_Win * win, MPIR_Info * info) +MPL_STATIC_INLINE_PREFIX int MPIDI_Win_set_info(MPIR_Win * win, MPIR_Info * info) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_WIN_SET_INFO); @@ -37,7 +37,7 @@ __CH4_INLINE__ int MPIDI_Win_set_info(MPIR_Win * win, MPIR_Info * info) #define FUNCNAME MPIDI_Win_start #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Win_start(MPIR_Group * group, int assert, MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_Win_start(MPIR_Group * group, int assert, MPIR_Win * win) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_WIN_START); @@ -57,7 +57,7 @@ __CH4_INLINE__ int MPIDI_Win_start(MPIR_Group * group, int assert, MPIR_Win * wi #define FUNCNAME MPIDI_Win_complete #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Win_complete(MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_Win_complete(MPIR_Win * win) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_WIN_COMPLETE); @@ -77,7 +77,7 @@ __CH4_INLINE__ int MPIDI_Win_complete(MPIR_Win * win) #define FUNCNAME MPIDI_Win_post #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Win_post(MPIR_Group * group, int assert, MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_Win_post(MPIR_Group * group, int assert, MPIR_Win * win) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_WIN_POST); @@ -97,7 +97,7 @@ __CH4_INLINE__ int MPIDI_Win_post(MPIR_Group * group, int assert, MPIR_Win * win #define FUNCNAME MPIDI_Win_wait #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Win_wait(MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_Win_wait(MPIR_Win * win) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_WIN_WAIT); @@ -118,7 +118,7 @@ __CH4_INLINE__ int MPIDI_Win_wait(MPIR_Win * win) #define FUNCNAME MPIDI_Win_test #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Win_test(MPIR_Win * win, int *flag) +MPL_STATIC_INLINE_PREFIX int MPIDI_Win_test(MPIR_Win * win, int *flag) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_WIN_TEST); @@ -138,7 +138,7 @@ __CH4_INLINE__ int MPIDI_Win_test(MPIR_Win * win, int *flag) #define FUNCNAME MPIDI_Win_lock #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Win_lock(int lock_type, int rank, int assert, MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_Win_lock(int lock_type, int rank, int assert, MPIR_Win * win) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_WIN_LOCK); @@ -158,7 +158,7 @@ __CH4_INLINE__ int MPIDI_Win_lock(int lock_type, int rank, int assert, MPIR_Win #define FUNCNAME MPIDI_Win_unlock #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Win_unlock(int rank, MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_Win_unlock(int rank, MPIR_Win * win) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_WIN_UNLOCK); @@ -178,7 +178,7 @@ __CH4_INLINE__ int MPIDI_Win_unlock(int rank, MPIR_Win * win) #define FUNCNAME MPIDI_Win_get_info #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Win_get_info(MPIR_Win * win, MPIR_Info ** info_p_p) +MPL_STATIC_INLINE_PREFIX int MPIDI_Win_get_info(MPIR_Win * win, MPIR_Info ** info_p_p) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_WIN_GET_INFO); @@ -198,7 +198,7 @@ __CH4_INLINE__ int MPIDI_Win_get_info(MPIR_Win * win, MPIR_Info ** info_p_p) #define FUNCNAME MPIDI_Win_free #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Win_free(MPIR_Win ** win_ptr) +MPL_STATIC_INLINE_PREFIX int MPIDI_Win_free(MPIR_Win ** win_ptr) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_WIN_FREE); @@ -218,7 +218,7 @@ __CH4_INLINE__ int MPIDI_Win_free(MPIR_Win ** win_ptr) #define FUNCNAME MPIDI_Win_fence #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Win_fence(int assert, MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_Win_fence(int assert, MPIR_Win * win) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_WIN_FENCE); @@ -238,10 +238,11 @@ __CH4_INLINE__ int MPIDI_Win_fence(int assert, MPIR_Win * win) #define FUNCNAME MPIDI_Win_create #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Win_create(void *base, - MPI_Aint length, - int disp_unit, - MPIR_Info * info, MPIR_Comm * comm_ptr, MPIR_Win ** win_ptr) +MPL_STATIC_INLINE_PREFIX int MPIDI_Win_create(void *base, + MPI_Aint length, + int disp_unit, + MPIR_Info * info, MPIR_Comm * comm_ptr, + MPIR_Win ** win_ptr) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_WIN_CREATE); @@ -261,7 +262,7 @@ __CH4_INLINE__ int MPIDI_Win_create(void *base, #define FUNCNAME MPIDI_Win_attach #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Win_attach(MPIR_Win * win, void *base, MPI_Aint size) +MPL_STATIC_INLINE_PREFIX int MPIDI_Win_attach(MPIR_Win * win, void *base, MPI_Aint size) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_WIN_ATTACH); @@ -281,11 +282,11 @@ __CH4_INLINE__ int MPIDI_Win_attach(MPIR_Win * win, void *base, MPI_Aint size) #define FUNCNAME MPIDI_Win_allocate_shared #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Win_allocate_shared(MPI_Aint size, - int disp_unit, - MPIR_Info * info_ptr, - MPIR_Comm * comm_ptr, - void **base_ptr, MPIR_Win ** win_ptr) +MPL_STATIC_INLINE_PREFIX int MPIDI_Win_allocate_shared(MPI_Aint size, + int disp_unit, + MPIR_Info * info_ptr, + MPIR_Comm * comm_ptr, + void **base_ptr, MPIR_Win ** win_ptr) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_WIN_ALLOCATE_SHARED); @@ -306,7 +307,7 @@ __CH4_INLINE__ int MPIDI_Win_allocate_shared(MPI_Aint size, #define FUNCNAME MPIDI_Win_flush_local #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Win_flush_local(int rank, MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_Win_flush_local(int rank, MPIR_Win * win) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_WIN_FLUSH_LOCAL); @@ -326,7 +327,7 @@ __CH4_INLINE__ int MPIDI_Win_flush_local(int rank, MPIR_Win * win) #define FUNCNAME MPIDI_Win_detach #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Win_detach(MPIR_Win * win, const void *base) +MPL_STATIC_INLINE_PREFIX int MPIDI_Win_detach(MPIR_Win * win, const void *base) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_WIN_DETACH); @@ -347,8 +348,9 @@ __CH4_INLINE__ int MPIDI_Win_detach(MPIR_Win * win, const void *base) #define FUNCNAME MPIDI_Win_shared_query #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Win_shared_query(MPIR_Win * win, - int rank, MPI_Aint * size, int *disp_unit, void *baseptr) +MPL_STATIC_INLINE_PREFIX int MPIDI_Win_shared_query(MPIR_Win * win, + int rank, MPI_Aint * size, int *disp_unit, + void *baseptr) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_WIN_SHARED_QUERY); @@ -368,10 +370,10 @@ __CH4_INLINE__ int MPIDI_Win_shared_query(MPIR_Win * win, #define FUNCNAME MPIDI_Win_allocate #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Win_allocate(MPI_Aint size, - int disp_unit, - MPIR_Info * info, - MPIR_Comm * comm, void *baseptr, MPIR_Win ** win) +MPL_STATIC_INLINE_PREFIX int MPIDI_Win_allocate(MPI_Aint size, + int disp_unit, + MPIR_Info * info, + MPIR_Comm * comm, void *baseptr, MPIR_Win ** win) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_WIN_ALLOCATE); @@ -391,7 +393,7 @@ __CH4_INLINE__ int MPIDI_Win_allocate(MPI_Aint size, #define FUNCNAME MPIDI_Win_flush #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Win_flush(int rank, MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_Win_flush(int rank, MPIR_Win * win) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_WIN_FLUSH); @@ -411,7 +413,7 @@ __CH4_INLINE__ int MPIDI_Win_flush(int rank, MPIR_Win * win) #define FUNCNAME MPIDI_Win_flush_local_all #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Win_flush_local_all(MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_Win_flush_local_all(MPIR_Win * win) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_WIN_FLUSH_LOCAL_ALL); @@ -431,7 +433,7 @@ __CH4_INLINE__ int MPIDI_Win_flush_local_all(MPIR_Win * win) #define FUNCNAME MPIDI_Win_unlock_all #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Win_unlock_all(MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_Win_unlock_all(MPIR_Win * win) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_WIN_UNLOCK_ALL); @@ -451,7 +453,8 @@ __CH4_INLINE__ int MPIDI_Win_unlock_all(MPIR_Win * win) #define FUNCNAME MPIDI_Win_create_dynamic #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Win_create_dynamic(MPIR_Info * info, MPIR_Comm * comm, MPIR_Win ** win) +MPL_STATIC_INLINE_PREFIX int MPIDI_Win_create_dynamic(MPIR_Info * info, MPIR_Comm * comm, + MPIR_Win ** win) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_WIN_CREATE_DYNAMIC); @@ -471,7 +474,7 @@ __CH4_INLINE__ int MPIDI_Win_create_dynamic(MPIR_Info * info, MPIR_Comm * comm, #define FUNCNAME MPIDI_Win_sync #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Win_sync(MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_Win_sync(MPIR_Win * win) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_WIN_SYNC); @@ -491,7 +494,7 @@ __CH4_INLINE__ int MPIDI_Win_sync(MPIR_Win * win) #define FUNCNAME MPIDI_Win_flush_all #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Win_flush_all(MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_Win_flush_all(MPIR_Win * win) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_WIN_FLUSH_ALL); @@ -511,7 +514,7 @@ __CH4_INLINE__ int MPIDI_Win_flush_all(MPIR_Win * win) #define FUNCNAME MPIDI_Win_lock_all #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Win_lock_all(int assert, MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_Win_lock_all(int assert, MPIR_Win * win) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_WIN_LOCK_ALL); diff --git a/src/mpid/ch4/src/ch4i_comm.h b/src/mpid/ch4/src/ch4i_comm.h index b34405c..a5575d7 100644 --- a/src/mpid/ch4/src/ch4i_comm.h +++ b/src/mpid/ch4/src/ch4i_comm.h @@ -977,7 +977,7 @@ static inline int MPIDII_set_map(MPIDII_rank_map_t * src_rmap, #define FUNCNAME MPIDII_comm_create_rank_map #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDII_comm_create_rank_map(MPIR_Comm * comm) +MPL_STATIC_INLINE_PREFIX int MPIDII_comm_create_rank_map(MPIR_Comm * comm) { int mpi_errno = MPI_SUCCESS; MPIR_Comm_map_t *mapper; diff --git a/src/mpid/ch4/src/ch4r_init.h b/src/mpid/ch4/src/ch4r_init.h index 9126e17..f692720 100644 --- a/src/mpid/ch4/src/ch4r_init.h +++ b/src/mpid/ch4/src/ch4r_init.h @@ -21,7 +21,7 @@ #define FUNCNAME MPIDI_CH4U_init_comm #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_CH4U_init_comm(MPIR_Comm * comm) +MPL_STATIC_INLINE_PREFIX int MPIDI_CH4U_init_comm(MPIR_Comm * comm) { int mpi_errno = MPI_SUCCESS, comm_idx, subcomm_type, is_localcomm; MPIDI_CH4U_rreq_t **uelist; @@ -72,7 +72,7 @@ __CH4_INLINE__ int MPIDI_CH4U_init_comm(MPIR_Comm * comm) #define FUNCNAME MPIDI_CH4U_destroy_comm #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_CH4U_destroy_comm(MPIR_Comm * comm) +MPL_STATIC_INLINE_PREFIX int MPIDI_CH4U_destroy_comm(MPIR_Comm * comm) { int mpi_errno = MPI_SUCCESS, comm_idx, subcomm_type, is_localcomm; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_DESTROY_COMM); @@ -104,8 +104,8 @@ __CH4_INLINE__ int MPIDI_CH4U_destroy_comm(MPIR_Comm * comm) #define FUNCNAME MPIDI_CH4U_init #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_CH4U_init(MPIR_Comm * comm_world, MPIR_Comm * comm_self, - int num_contexts, void **netmod_contexts) +MPL_STATIC_INLINE_PREFIX int MPIDI_CH4U_init(MPIR_Comm * comm_world, MPIR_Comm * comm_self, + int num_contexts, void **netmod_contexts) { int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_INIT); @@ -278,7 +278,7 @@ __CH4_INLINE__ int MPIDI_CH4U_init(MPIR_Comm * comm_world, MPIR_Comm * comm_self #define FUNCNAME MPIDI_CH4U_init #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ void MPIDI_CH4U_finalize() +MPL_STATIC_INLINE_PREFIX void MPIDI_CH4U_finalize() { MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_FINALIZE); MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_FINALIZE); @@ -293,7 +293,7 @@ __CH4_INLINE__ void MPIDI_CH4U_finalize() #define FUNCNAME MPIDI_CH4U_alloc_mem #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ void *MPIDI_CH4U_alloc_mem(size_t size, MPIR_Info * info_ptr) +MPL_STATIC_INLINE_PREFIX void *MPIDI_CH4U_alloc_mem(size_t size, MPIR_Info * info_ptr) { MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_ALLOC_MEM); MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_ALLOC_MEM); @@ -307,7 +307,7 @@ __CH4_INLINE__ void *MPIDI_CH4U_alloc_mem(size_t size, MPIR_Info * info_ptr) #define FUNCNAME MPIDI_CH4U_free_mem #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_CH4U_free_mem(void *ptr) +MPL_STATIC_INLINE_PREFIX int MPIDI_CH4U_free_mem(void *ptr) { int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_FREE_MEM); diff --git a/src/mpid/ch4/src/ch4r_probe.h b/src/mpid/ch4/src/ch4r_probe.h index 20753db..015a274 100644 --- a/src/mpid/ch4/src/ch4r_probe.h +++ b/src/mpid/ch4/src/ch4r_probe.h @@ -17,10 +17,10 @@ #define FUNCNAME MPIDI_CH4U_iprobe #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_CH4U_iprobe(int source, - int tag, - MPIR_Comm * comm, - int context_offset, int *flag, MPI_Status * status) +MPL_STATIC_INLINE_PREFIX int MPIDI_CH4U_iprobe(int source, + int tag, + MPIR_Comm * comm, + int context_offset, int *flag, MPI_Status * status) { int mpi_errno = MPI_SUCCESS; MPIR_Comm *root_comm; @@ -69,9 +69,10 @@ __CH4_INLINE__ int MPIDI_CH4U_iprobe(int source, #define FUNCNAME MPIDI_Probe #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_CH4U_probe(int source, - int tag, - MPIR_Comm * comm, int context_offset, MPI_Status * status) +MPL_STATIC_INLINE_PREFIX int MPIDI_CH4U_probe(int source, + int tag, + MPIR_Comm * comm, int context_offset, + MPI_Status * status) { int mpi_errno, flag = 0; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_PROBE); @@ -94,11 +95,12 @@ __CH4_INLINE__ int MPIDI_CH4U_probe(int source, #define FUNCNAME MPIDI_CH4U_improbe #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_CH4U_improbe(int source, - int tag, - MPIR_Comm * comm, - int context_offset, - int *flag, MPIR_Request ** message, MPI_Status * status) +MPL_STATIC_INLINE_PREFIX int MPIDI_CH4U_improbe(int source, + int tag, + MPIR_Comm * comm, + int context_offset, + int *flag, MPIR_Request ** message, + MPI_Status * status) { int mpi_errno = MPI_SUCCESS; MPIR_Comm *root_comm; @@ -157,11 +159,11 @@ __CH4_INLINE__ int MPIDI_CH4U_improbe(int source, #define FUNCNAME MPIDI_CH4U_mprobe #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_CH4U_mprobe(int source, - int tag, - MPIR_Comm * comm, - int context_offset, - MPIR_Request ** message, MPI_Status * status) +MPL_STATIC_INLINE_PREFIX int MPIDI_CH4U_mprobe(int source, + int tag, + MPIR_Comm * comm, + int context_offset, + MPIR_Request ** message, MPI_Status * status) { int mpi_errno, flag = 0; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_MPROBE); diff --git a/src/mpid/ch4/src/ch4r_recv.h b/src/mpid/ch4/src/ch4r_recv.h index c77040c..4672198 100644 --- a/src/mpid/ch4/src/ch4r_recv.h +++ b/src/mpid/ch4/src/ch4r_recv.h @@ -213,13 +213,14 @@ static inline int MPIDI_CH4I_do_irecv(void *buf, #define FUNCNAME MPIDI_CH4U_recv #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_CH4U_recv(void *buf, - int count, - MPI_Datatype datatype, - int rank, - int tag, - MPIR_Comm * comm, - int context_offset, MPI_Status * status, MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_CH4U_recv(void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, + int context_offset, MPI_Status * status, + MPIR_Request ** request) { int mpi_errno; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_RECV); @@ -241,13 +242,13 @@ __CH4_INLINE__ int MPIDI_CH4U_recv(void *buf, #define FUNCNAME MPIDI_CH4U_recv_init #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_CH4U_recv_init(void *buf, - int count, - MPI_Datatype datatype, - int rank, - int tag, - MPIR_Comm * comm, - int context_offset, MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_CH4U_recv_init(void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, + int context_offset, MPIR_Request ** request) { int mpi_errno = MPI_SUCCESS; MPIR_Request *rreq; @@ -279,10 +280,10 @@ __CH4_INLINE__ int MPIDI_CH4U_recv_init(void *buf, #define FUNCNAME MPIDI_CH4U_imrecv #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_CH4U_imrecv(void *buf, - int count, - MPI_Datatype datatype, - MPIR_Request * message, MPIR_Request ** rreqp) +MPL_STATIC_INLINE_PREFIX int MPIDI_CH4U_imrecv(void *buf, + int count, + MPI_Datatype datatype, + MPIR_Request * message, MPIR_Request ** rreqp) { int mpi_errno = MPI_SUCCESS; MPIR_Request *rreq; @@ -332,10 +333,10 @@ __CH4_INLINE__ int MPIDI_CH4U_imrecv(void *buf, #define FUNCNAME MPIDI_CH4U_mrecv #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_CH4U_mrecv(void *buf, - int count, - MPI_Datatype datatype, - MPIR_Request * message, MPI_Status * status) +MPL_STATIC_INLINE_PREFIX int MPIDI_CH4U_mrecv(void *buf, + int count, + MPI_Datatype datatype, + MPIR_Request * message, MPI_Status * status) { int mpi_errno = MPI_SUCCESS, active_flag; MPI_Request req_handle; @@ -369,12 +370,13 @@ __CH4_INLINE__ int MPIDI_CH4U_mrecv(void *buf, #define FUNCNAME MPIDI_CH4U_irecv #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_CH4U_irecv(void *buf, - int count, - MPI_Datatype datatype, - int rank, - int tag, - MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_CH4U_irecv(void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) { int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_IRECV); @@ -395,7 +397,7 @@ __CH4_INLINE__ int MPIDI_CH4U_irecv(void *buf, #define FUNCNAME MPIDI_CH4U_cancel_recv #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_CH4U_cancel_recv(MPIR_Request * rreq) +MPL_STATIC_INLINE_PREFIX int MPIDI_CH4U_cancel_recv(MPIR_Request * rreq) { int mpi_errno = MPI_SUCCESS, found; MPIR_Comm *root_comm; diff --git a/src/mpid/ch4/src/ch4r_recvq.h b/src/mpid/ch4/src/ch4r_recvq.h index ff98e36..03e41d7 100644 --- a/src/mpid/ch4/src/ch4r_recvq.h +++ b/src/mpid/ch4/src/ch4r_recvq.h @@ -21,7 +21,8 @@ #define FUNCNAME MPIDI_CH4U_enqueue_posted #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ void MPIDI_CH4U_enqueue_posted(MPIR_Request * req, MPIDI_CH4U_rreq_t ** list) +MPL_STATIC_INLINE_PREFIX void MPIDI_CH4U_enqueue_posted(MPIR_Request * req, + MPIDI_CH4U_rreq_t ** list) { MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_ENQUEUE_POSTED); MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_ENQUEUE_POSTED); @@ -34,7 +35,8 @@ __CH4_INLINE__ void MPIDI_CH4U_enqueue_posted(MPIR_Request * req, MPIDI_CH4U_rre #define FUNCNAME MPIDI_CH4U_enqueue_unexp #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ void MPIDI_CH4U_enqueue_unexp(MPIR_Request * req, MPIDI_CH4U_rreq_t ** list) +MPL_STATIC_INLINE_PREFIX void MPIDI_CH4U_enqueue_unexp(MPIR_Request * req, + MPIDI_CH4U_rreq_t ** list) { MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_ENQUEUE_UNEXP); MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_ENQUEUE_UNEXP); @@ -47,7 +49,7 @@ __CH4_INLINE__ void MPIDI_CH4U_enqueue_unexp(MPIR_Request * req, MPIDI_CH4U_rreq #define FUNCNAME MPIDI_CH4U_delete_unexp #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ void MPIDI_CH4U_delete_unexp(MPIR_Request * req, MPIDI_CH4U_rreq_t ** list) +MPL_STATIC_INLINE_PREFIX void MPIDI_CH4U_delete_unexp(MPIR_Request * req, MPIDI_CH4U_rreq_t ** list) { MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_DELETE_UNEXP); MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_DELETE_UNEXP); @@ -59,8 +61,9 @@ __CH4_INLINE__ void MPIDI_CH4U_delete_unexp(MPIR_Request * req, MPIDI_CH4U_rreq_ #define FUNCNAME MPIDI_CH4U_dequeue_unexp_strict #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ MPIR_Request *MPIDI_CH4U_dequeue_unexp_strict(uint64_t tag, uint64_t ignore, - MPIDI_CH4U_rreq_t ** list) +MPL_STATIC_INLINE_PREFIX MPIR_Request *MPIDI_CH4U_dequeue_unexp_strict(uint64_t tag, + uint64_t ignore, + MPIDI_CH4U_rreq_t ** list) { MPIDI_CH4U_rreq_t *curr, *tmp; MPIR_Request *req = NULL; @@ -84,8 +87,8 @@ __CH4_INLINE__ MPIR_Request *MPIDI_CH4U_dequeue_unexp_strict(uint64_t tag, uint6 #define FUNCNAME MPIDI_CH4U_dequeue_unexp #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ MPIR_Request *MPIDI_CH4U_dequeue_unexp(uint64_t tag, uint64_t ignore, - MPIDI_CH4U_rreq_t ** list) +MPL_STATIC_INLINE_PREFIX MPIR_Request *MPIDI_CH4U_dequeue_unexp(uint64_t tag, uint64_t ignore, + MPIDI_CH4U_rreq_t ** list) { MPIDI_CH4U_rreq_t *curr, *tmp; MPIR_Request *req = NULL; @@ -108,8 +111,8 @@ __CH4_INLINE__ MPIR_Request *MPIDI_CH4U_dequeue_unexp(uint64_t tag, uint64_t ign #define FUNCNAME MPIDI_CH4U_find_unexp #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ MPIR_Request *MPIDI_CH4U_find_unexp(uint64_t tag, uint64_t ignore, - MPIDI_CH4U_rreq_t ** list) +MPL_STATIC_INLINE_PREFIX MPIR_Request *MPIDI_CH4U_find_unexp(uint64_t tag, uint64_t ignore, + MPIDI_CH4U_rreq_t ** list) { MPIDI_CH4U_rreq_t *curr, *tmp; MPIR_Request *req = NULL; @@ -131,7 +134,8 @@ __CH4_INLINE__ MPIR_Request *MPIDI_CH4U_find_unexp(uint64_t tag, uint64_t ignore #define FUNCNAME MPIDI_CH4U_dequeue_posted #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ MPIR_Request *MPIDI_CH4U_dequeue_posted(uint64_t tag, MPIDI_CH4U_rreq_t ** list) +MPL_STATIC_INLINE_PREFIX MPIR_Request *MPIDI_CH4U_dequeue_posted(uint64_t tag, + MPIDI_CH4U_rreq_t ** list) { MPIR_Request *req = NULL; MPIDI_CH4U_rreq_t *curr, *tmp; @@ -155,7 +159,8 @@ __CH4_INLINE__ MPIR_Request *MPIDI_CH4U_dequeue_posted(uint64_t tag, MPIDI_CH4U_ #define FUNCNAME MPIDI_CH4U_delete_posted #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_CH4U_delete_posted(MPIDI_CH4U_rreq_t * req, MPIDI_CH4U_rreq_t ** list) +MPL_STATIC_INLINE_PREFIX int MPIDI_CH4U_delete_posted(MPIDI_CH4U_rreq_t * req, + MPIDI_CH4U_rreq_t ** list) { int found = 0; MPIDI_CH4U_rreq_t *curr, *tmp; @@ -180,7 +185,8 @@ __CH4_INLINE__ int MPIDI_CH4U_delete_posted(MPIDI_CH4U_rreq_t * req, MPIDI_CH4U_ #define FUNCNAME MPIDI_CH4U_enqueue_posted #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ void MPIDI_CH4U_enqueue_posted(MPIR_Request * req, MPIDI_CH4U_rreq_t ** list) +MPL_STATIC_INLINE_PREFIX void MPIDI_CH4U_enqueue_posted(MPIR_Request * req, + MPIDI_CH4U_rreq_t ** list) { MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_ENQUEUE_POSTED); MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_ENQUEUE_POSTED); @@ -193,7 +199,8 @@ __CH4_INLINE__ void MPIDI_CH4U_enqueue_posted(MPIR_Request * req, MPIDI_CH4U_rre #define FUNCNAME MPIDI_CH4U_enqueue_unexp #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ void MPIDI_CH4U_enqueue_unexp(MPIR_Request * req, MPIDI_CH4U_rreq_t ** list) +MPL_STATIC_INLINE_PREFIX void MPIDI_CH4U_enqueue_unexp(MPIR_Request * req, + MPIDI_CH4U_rreq_t ** list) { MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_ENQUEUE_UNEXP); MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_ENQUEUE_UNEXP); @@ -206,7 +213,7 @@ __CH4_INLINE__ void MPIDI_CH4U_enqueue_unexp(MPIR_Request * req, MPIDI_CH4U_rreq #define FUNCNAME MPIDI_CH4U_delete_unexp #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ void MPIDI_CH4U_delete_unexp(MPIR_Request * req, MPIDI_CH4U_rreq_t ** list) +MPL_STATIC_INLINE_PREFIX void MPIDI_CH4U_delete_unexp(MPIR_Request * req, MPIDI_CH4U_rreq_t ** list) { MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_DELETE_UNEXP); MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_DELETE_UNEXP); @@ -218,8 +225,9 @@ __CH4_INLINE__ void MPIDI_CH4U_delete_unexp(MPIR_Request * req, MPIDI_CH4U_rreq_ #define FUNCNAME MPIDI_CH4U_dequeue_unexp_strict #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ MPIR_Request *MPIDI_CH4U_dequeue_unexp_strict(uint64_t tag, uint64_t ignore, - MPIDI_CH4U_rreq_t ** list) +MPL_STATIC_INLINE_PREFIX MPIR_Request *MPIDI_CH4U_dequeue_unexp_strict(uint64_t tag, + uint64_t ignore, + MPIDI_CH4U_rreq_t ** list) { MPIDI_CH4U_rreq_t *curr, *tmp; MPIR_Request *req = NULL; @@ -243,8 +251,8 @@ __CH4_INLINE__ MPIR_Request *MPIDI_CH4U_dequeue_unexp_strict(uint64_t tag, uint6 #define FUNCNAME MPIDI_CH4U_dequeue_unexp #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ MPIR_Request *MPIDI_CH4U_dequeue_unexp(uint64_t tag, uint64_t ignore, - MPIDI_CH4U_rreq_t ** list) +MPL_STATIC_INLINE_PREFIX MPIR_Request *MPIDI_CH4U_dequeue_unexp(uint64_t tag, uint64_t ignore, + MPIDI_CH4U_rreq_t ** list) { MPIDI_CH4U_rreq_t *curr, *tmp; MPIR_Request *req = NULL; @@ -267,8 +275,8 @@ __CH4_INLINE__ MPIR_Request *MPIDI_CH4U_dequeue_unexp(uint64_t tag, uint64_t ign #define FUNCNAME MPIDI_CH4U_find_unexp #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ MPIR_Request *MPIDI_CH4U_find_unexp(uint64_t tag, uint64_t ignore, - MPIDI_CH4U_rreq_t ** list) +MPL_STATIC_INLINE_PREFIX MPIR_Request *MPIDI_CH4U_find_unexp(uint64_t tag, uint64_t ignore, + MPIDI_CH4U_rreq_t ** list) { MPIDI_CH4U_rreq_t *curr, *tmp; MPIR_Request *req = NULL; @@ -290,7 +298,8 @@ __CH4_INLINE__ MPIR_Request *MPIDI_CH4U_find_unexp(uint64_t tag, uint64_t ignore #define FUNCNAME MPIDI_CH4U_dequeue_posted #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ MPIR_Request *MPIDI_CH4U_dequeue_posted(uint64_t tag, MPIDI_CH4U_rreq_t ** list) +MPL_STATIC_INLINE_PREFIX MPIR_Request *MPIDI_CH4U_dequeue_posted(uint64_t tag, + MPIDI_CH4U_rreq_t ** list) { MPIR_Request *req = NULL; MPIDI_CH4U_rreq_t *curr, *tmp; @@ -314,7 +323,8 @@ __CH4_INLINE__ MPIR_Request *MPIDI_CH4U_dequeue_posted(uint64_t tag, MPIDI_CH4U_ #define FUNCNAME MPIDI_CH4U_delete_posted #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_CH4U_delete_posted(MPIDI_CH4U_rreq_t * req, MPIDI_CH4U_rreq_t ** list) +MPL_STATIC_INLINE_PREFIX int MPIDI_CH4U_delete_posted(MPIDI_CH4U_rreq_t * req, + MPIDI_CH4U_rreq_t ** list) { int found = 0; MPIDI_CH4U_rreq_t *curr, *tmp; diff --git a/src/mpid/ch4/src/ch4r_request.h b/src/mpid/ch4/src/ch4r_request.h index 68399c5..0308b09 100644 --- a/src/mpid/ch4/src/ch4r_request.h +++ b/src/mpid/ch4/src/ch4r_request.h @@ -68,7 +68,7 @@ static inline MPIR_Request *MPIDI_CH4I_am_win_request_create() #define FUNCNAME MPIDI_CH4I_am_request_complete #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ void MPIDI_CH4I_am_request_complete(MPIR_Request * req) +MPL_STATIC_INLINE_PREFIX void MPIDI_CH4I_am_request_complete(MPIR_Request * req) { int incomplete; MPIR_cc_decr(req->cc_ptr, &incomplete); diff --git a/src/mpid/ch4/src/ch4r_rma.h b/src/mpid/ch4/src/ch4r_rma.h index 637ba56..39bf2cf 100644 --- a/src/mpid/ch4/src/ch4r_rma.h +++ b/src/mpid/ch4/src/ch4r_rma.h @@ -253,12 +253,13 @@ static inline int MPIDI_CH4I_do_get(void *origin_addr, #define FUNCNAME MPIDI_CH4U_put #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_CH4U_put(const void *origin_addr, - int origin_count, - MPI_Datatype origin_datatype, - int target_rank, - MPI_Aint target_disp, - int target_count, MPI_Datatype target_datatype, MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_CH4U_put(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, MPI_Datatype target_datatype, + MPIR_Win * win) { int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_PUT); @@ -280,14 +281,14 @@ __CH4_INLINE__ int MPIDI_CH4U_put(const void *origin_addr, #define FUNCNAME MPIDI_CH4U_rput #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_CH4U_rput(const void *origin_addr, - int origin_count, - MPI_Datatype origin_datatype, - int target_rank, - MPI_Aint target_disp, - int target_count, - MPI_Datatype target_datatype, - MPIR_Win * win, MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_CH4U_rput(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPIR_Win * win, MPIR_Request ** request) { int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_RPUT); @@ -310,12 +311,13 @@ __CH4_INLINE__ int MPIDI_CH4U_rput(const void *origin_addr, #define FUNCNAME MPIDI_CH4U_get #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_CH4U_get(void *origin_addr, - int origin_count, - MPI_Datatype origin_datatype, - int target_rank, - MPI_Aint target_disp, - int target_count, MPI_Datatype target_datatype, MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_CH4U_get(void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, MPI_Datatype target_datatype, + MPIR_Win * win) { int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_GET); @@ -337,14 +339,14 @@ __CH4_INLINE__ int MPIDI_CH4U_get(void *origin_addr, #define FUNCNAME MPIDI_CH4U_rget #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_CH4U_rget(void *origin_addr, - int origin_count, - MPI_Datatype origin_datatype, - int target_rank, - MPI_Aint target_disp, - int target_count, - MPI_Datatype target_datatype, - MPIR_Win * win, MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_CH4U_rget(void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPIR_Win * win, MPIR_Request ** request) { int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_RGET); @@ -366,15 +368,15 @@ __CH4_INLINE__ int MPIDI_CH4U_rget(void *origin_addr, #define FUNCNAME MPIDI_CH4I_do_accumulate #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_CH4I_do_accumulate(const void *origin_addr, - int origin_count, - MPI_Datatype origin_datatype, - int target_rank, - MPI_Aint target_disp, - int target_count, - MPI_Datatype target_datatype, - MPI_Op op, MPIR_Win * win, - int do_get, MPIR_Request * sreq) +MPL_STATIC_INLINE_PREFIX int MPIDI_CH4I_do_accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPI_Op op, MPIR_Win * win, + int do_get, MPIR_Request * sreq) { int mpi_errno = MPI_SUCCESS, c, n_iov; size_t basic_type_size; @@ -510,14 +512,15 @@ __CH4_INLINE__ int MPIDI_CH4I_do_accumulate(const void *origin_addr, #define FUNCNAME MPIDI_CH4U_raccumulate #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_CH4U_raccumulate(const void *origin_addr, - int origin_count, - MPI_Datatype origin_datatype, - int target_rank, - MPI_Aint target_disp, - int target_count, - MPI_Datatype target_datatype, - MPI_Op op, MPIR_Win * win, MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_CH4U_raccumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPI_Op op, MPIR_Win * win, + MPIR_Request ** request) { int mpi_errno = MPI_SUCCESS; MPIR_Request *sreq; @@ -551,13 +554,14 @@ __CH4_INLINE__ int MPIDI_CH4U_raccumulate(const void *origin_addr, #define FUNCNAME MPIDI_CH4U_accumulate #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_CH4U_accumulate(const void *origin_addr, - int origin_count, - MPI_Datatype origin_datatype, - int target_rank, - MPI_Aint target_disp, - int target_count, - MPI_Datatype target_datatype, MPI_Op op, MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_CH4U_accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, MPI_Op op, + MPIR_Win * win) { int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_ACCUMULATE); @@ -582,17 +586,18 @@ __CH4_INLINE__ int MPIDI_CH4U_accumulate(const void *origin_addr, #define FUNCNAME MPIDI_CH4U_rget_accumulate #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_CH4U_rget_accumulate(const void *origin_addr, - int origin_count, - MPI_Datatype origin_datatype, - void *result_addr, - int result_count, - MPI_Datatype result_datatype, - int target_rank, - MPI_Aint target_disp, - int target_count, - MPI_Datatype target_datatype, - MPI_Op op, MPIR_Win * win, MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_CH4U_rget_accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + void *result_addr, + int result_count, + MPI_Datatype result_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPI_Op op, MPIR_Win * win, + MPIR_Request ** request) { int mpi_errno = MPI_SUCCESS; MPIR_Request *sreq; @@ -632,17 +637,17 @@ __CH4_INLINE__ int MPIDI_CH4U_rget_accumulate(const void *origin_addr, #define FUNCNAME MPIDI_CH4U_get_accumulate #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_CH4U_get_accumulate(const void *origin_addr, - int origin_count, - MPI_Datatype origin_datatype, - void *result_addr, - int result_count, - MPI_Datatype result_datatype, - int target_rank, - MPI_Aint target_disp, - int target_count, - MPI_Datatype target_datatype, - MPI_Op op, MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_CH4U_get_accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + void *result_addr, + int result_count, + MPI_Datatype result_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPI_Op op, MPIR_Win * win) { int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_GET_ACCUMULATE); @@ -670,12 +675,12 @@ __CH4_INLINE__ int MPIDI_CH4U_get_accumulate(const void *origin_addr, #define FUNCNAME MPIDI_CH4U_compare_and_swap #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_CH4U_compare_and_swap(const void *origin_addr, - const void *compare_addr, - void *result_addr, - MPI_Datatype datatype, - int target_rank, - MPI_Aint target_disp, MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_CH4U_compare_and_swap(const void *origin_addr, + const void *compare_addr, + void *result_addr, + MPI_Datatype datatype, + int target_rank, + MPI_Aint target_disp, MPIR_Win * win) { int mpi_errno = MPI_SUCCESS, c; MPIR_Request *sreq = NULL; @@ -738,11 +743,12 @@ __CH4_INLINE__ int MPIDI_CH4U_compare_and_swap(const void *origin_addr, #define FUNCNAME MPIDI_CH4U_fetch_and_op #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_CH4U_fetch_and_op(const void *origin_addr, - void *result_addr, - MPI_Datatype datatype, - int target_rank, - MPI_Aint target_disp, MPI_Op op, MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_CH4U_fetch_and_op(const void *origin_addr, + void *result_addr, + MPI_Datatype datatype, + int target_rank, + MPI_Aint target_disp, MPI_Op op, + MPIR_Win * win) { int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_FETCH_AND_OP); diff --git a/src/mpid/ch4/src/ch4r_send.h b/src/mpid/ch4/src/ch4r_send.h index dc1f3e7..57f50cb 100644 --- a/src/mpid/ch4/src/ch4r_send.h +++ b/src/mpid/ch4/src/ch4r_send.h @@ -140,12 +140,13 @@ static inline int MPIDI_CH4I_psend(const void *buf, #define FUNCNAME MPIDI_CH4U_send #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_CH4U_send(const void *buf, - int count, - MPI_Datatype datatype, - int rank, - int tag, - MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_CH4U_send(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) { int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_SEND); @@ -160,12 +161,13 @@ __CH4_INLINE__ int MPIDI_CH4U_send(const void *buf, #define FUNCNAME MPIDI_CH4U_isend #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_CH4U_isend(const void *buf, - int count, - MPI_Datatype datatype, - int rank, - int tag, - MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_CH4U_isend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) { int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_ISEND); @@ -182,12 +184,13 @@ __CH4_INLINE__ int MPIDI_CH4U_isend(const void *buf, #define FUNCNAME MPIDI_CH4U_rsend #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_CH4U_rsend(const void *buf, - int count, - MPI_Datatype datatype, - int rank, - int tag, - MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_CH4U_rsend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) { int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_RSEND); @@ -203,12 +206,13 @@ __CH4_INLINE__ int MPIDI_CH4U_rsend(const void *buf, #define FUNCNAME MPIDI_CH4U_irsend #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_CH4U_irsend(const void *buf, - int count, - MPI_Datatype datatype, - int rank, - int tag, - MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_CH4U_irsend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) { int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_IRSEND); @@ -223,12 +227,13 @@ __CH4_INLINE__ int MPIDI_CH4U_irsend(const void *buf, #define FUNCNAME MPIDI_CH4U_ssend #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_CH4U_ssend(const void *buf, - int count, - MPI_Datatype datatype, - int rank, - int tag, - MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_CH4U_ssend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) { int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_SSEND); @@ -243,12 +248,13 @@ __CH4_INLINE__ int MPIDI_CH4U_ssend(const void *buf, #define FUNCNAME MPIDI_CH4U_issend #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_CH4U_issend(const void *buf, - int count, - MPI_Datatype datatype, - int rank, - int tag, - MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_CH4U_issend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) { int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_ISSEND); @@ -263,7 +269,7 @@ __CH4_INLINE__ int MPIDI_CH4U_issend(const void *buf, #define FUNCNAME MPIDI_CH4U_startall #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_CH4U_startall(int count, MPIR_Request * requests[]) +MPL_STATIC_INLINE_PREFIX int MPIDI_CH4U_startall(int count, MPIR_Request * requests[]) { int mpi_errno = MPI_SUCCESS, i; int rank, tag, context_offset; @@ -371,13 +377,13 @@ __CH4_INLINE__ int MPIDI_CH4U_startall(int count, MPIR_Request * requests[]) #define FUNCNAME MPIDI_CH4U_send_init #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_CH4U_send_init(const void *buf, - int count, - MPI_Datatype datatype, - int rank, - int tag, - MPIR_Comm * comm, - int context_offset, MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_CH4U_send_init(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, + int context_offset, MPIR_Request ** request) { int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_SEND_INIT); @@ -392,13 +398,13 @@ __CH4_INLINE__ int MPIDI_CH4U_send_init(const void *buf, #define FUNCNAME MPIDI_CH4U_ssend_init #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_CH4U_ssend_init(const void *buf, - int count, - MPI_Datatype datatype, - int rank, - int tag, - MPIR_Comm * comm, - int context_offset, MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_CH4U_ssend_init(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, + int context_offset, MPIR_Request ** request) { int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_SSEND_INIT); @@ -413,13 +419,13 @@ __CH4_INLINE__ int MPIDI_CH4U_ssend_init(const void *buf, #define FUNCNAME MPIDI_CH4U_bsend_init #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_CH4U_bsend_init(const void *buf, - int count, - MPI_Datatype datatype, - int rank, - int tag, - MPIR_Comm * comm, - int context_offset, MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_CH4U_bsend_init(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, + int context_offset, MPIR_Request ** request) { int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_BSEND_INIT); @@ -434,13 +440,13 @@ __CH4_INLINE__ int MPIDI_CH4U_bsend_init(const void *buf, #define FUNCNAME MPIDI_CH4U_rsend_init #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_CH4U_rsend_init(const void *buf, - int count, - MPI_Datatype datatype, - int rank, - int tag, - MPIR_Comm * comm, - int context_offset, MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_CH4U_rsend_init(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, + int context_offset, MPIR_Request ** request) { int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_RSEND_INIT); @@ -455,7 +461,7 @@ __CH4_INLINE__ int MPIDI_CH4U_rsend_init(const void *buf, #define FUNCNAME MPIDI_CH4U_cancel_send #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_CH4U_cancel_send(MPIR_Request * sreq) +MPL_STATIC_INLINE_PREFIX int MPIDI_CH4U_cancel_send(MPIR_Request * sreq) { int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_CANCEL_SEND); http://git.mpich.org/mpich.git/commitdiff/181430bda15e745f82f761e28643e71eb… commit 181430bda15e745f82f761e28643e71eb42bcbb2 Author: Pavan Balaji <balaji(a)anl.gov> Date: Wed Aug 24 17:00:47 2016 -0500 CH4: Rename send/inject to isend/send_hdr Signed-off-by: Ken Raffenetti <raffenet(a)mcs.anl.gov> diff --git a/src/mpid/ch4/netmod/include/netmod.h b/src/mpid/ch4/netmod/include/netmod.h index 6230eac..04f192b 100644 --- a/src/mpid/ch4/netmod/include/netmod.h +++ b/src/mpid/ch4/netmod/include/netmod.h @@ -41,24 +41,22 @@ typedef int (*MPIDI_NM_open_port_t) (MPIR_Info * info_ptr, char *port_name); typedef int (*MPIDI_NM_close_port_t) (const char *port_name); typedef int (*MPIDI_NM_comm_accept_t) (const char *port_name, MPIR_Info * info, int root, MPIR_Comm * comm, MPIR_Comm ** newcomm_ptr); -typedef int (*MPIDI_NM_am_inject_hdr_t) (int rank, MPIR_Comm * comm, int handler_id, - const void *am_hdr, size_t am_hdr_sz, - void *netmod_context); -typedef int (*MPIDI_NM_am_send_t) (int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, - size_t am_hdr_sz, const void *data, MPI_Count count, - MPI_Datatype datatype, MPIR_Request * sreq, - void *netmod_context); -typedef int (*MPIDI_NM_am_sendv_t) (int rank, MPIR_Comm * comm, int handler_id, - struct iovec * am_hdrs, size_t iov_len, const void *data, - MPI_Count count, MPI_Datatype datatype, MPIR_Request * sreq, +typedef int (*MPIDI_NM_am_send_hdr_t) (int rank, MPIR_Comm * comm, int handler_id, + const void *am_hdr, size_t am_hdr_sz, void *netmod_context); +typedef int (*MPIDI_NM_am_isend_t) (int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, + size_t am_hdr_sz, const void *data, MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq, void *netmod_context); -typedef int (*MPIDI_NM_am_inject_hdr_reply_t) (MPIR_Context_id_t context_id, int src_rank, - int handler_id, const void *am_hdr, - size_t am_hdr_sz); -typedef int (*MPIDI_NM_am_send_reply_t) (MPIR_Context_id_t context_id, int src_rank, int handler_id, - const void *am_hdr, size_t am_hdr_sz, const void *data, - MPI_Count count, MPI_Datatype datatype, - MPIR_Request * sreq); +typedef int (*MPIDI_NM_am_isendv_t) (int rank, MPIR_Comm * comm, int handler_id, + struct iovec * am_hdrs, size_t iov_len, const void *data, + MPI_Count count, MPI_Datatype datatype, MPIR_Request * sreq, + void *netmod_context); +typedef int (*MPIDI_NM_am_send_hdr_reply_t) (MPIR_Context_id_t context_id, int src_rank, + int handler_id, const void *am_hdr, size_t am_hdr_sz); +typedef int (*MPIDI_NM_am_isend_reply_t) (MPIR_Context_id_t context_id, int src_rank, + int handler_id, const void *am_hdr, size_t am_hdr_sz, + const void *data, MPI_Count count, MPI_Datatype datatype, + MPIR_Request * sreq); typedef size_t(*MPIDI_NM_am_hdr_max_sz_t) (void); typedef int (*MPIDI_NM_am_recv_t) (MPIR_Request * req); typedef int (*MPIDI_NM_comm_get_lpid_t) (MPIR_Comm * comm_ptr, int idx, int *lpid_ptr, @@ -366,11 +364,11 @@ typedef struct MPIDI_NM_funcs { MPIDI_NM_am_request_finalize_t am_request_finalize; /* Active Message Routines */ MPIDI_NM_am_reg_handler_t am_reg_handler; - MPIDI_NM_am_inject_hdr_t am_inject_hdr; - MPIDI_NM_am_send_t am_send; - MPIDI_NM_am_sendv_t am_sendv; - MPIDI_NM_am_inject_hdr_reply_t am_inject_hdr_reply; - MPIDI_NM_am_send_reply_t am_send_reply; + MPIDI_NM_am_send_hdr_t am_send_hdr; + MPIDI_NM_am_isend_t am_isend; + MPIDI_NM_am_isendv_t am_isendv; + MPIDI_NM_am_send_hdr_reply_t am_send_hdr_reply; + MPIDI_NM_am_isend_reply_t am_isend_reply; MPIDI_NM_am_hdr_max_sz_t am_hdr_max_sz; MPIDI_NM_am_recv_t am_recv; } MPIDI_NM_funcs_t; @@ -517,29 +515,28 @@ MPL_STATIC_INLINE_PREFIX int MPIDI_NM_comm_accept(const char *port_name, MPIR_In int root, MPIR_Comm * comm, MPIR_Comm ** newcomm_ptr) MPL_STATIC_INLINE_SUFFIX; -MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_inject_hdr(int rank, MPIR_Comm * comm, int handler_id, - const void *am_hdr, size_t am_hdr_sz, - void *netmod_context) MPL_STATIC_INLINE_SUFFIX; -MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_send(int rank, MPIR_Comm * comm, int handler_id, - const void *am_hdr, size_t am_hdr_sz, - const void *data, MPI_Count count, - MPI_Datatype datatype, MPIR_Request * sreq, - void *netmod_context) MPL_STATIC_INLINE_SUFFIX; -MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_sendv(int rank, MPIR_Comm * comm, int handler_id, - struct iovec *am_hdrs, size_t iov_len, +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_send_hdr(int rank, MPIR_Comm * comm, int handler_id, + const void *am_hdr, size_t am_hdr_sz, + void *netmod_context) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_isend(int rank, MPIR_Comm * comm, int handler_id, + const void *am_hdr, size_t am_hdr_sz, const void *data, MPI_Count count, MPI_Datatype datatype, MPIR_Request * sreq, void *netmod_context) MPL_STATIC_INLINE_SUFFIX; -MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_inject_hdr_reply(MPIR_Context_id_t context_id, - int src_rank, int handler_id, - const void *am_hdr, - size_t am_hdr_sz) - MPL_STATIC_INLINE_SUFFIX; -MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_send_reply(MPIR_Context_id_t context_id, int src_rank, - int handler_id, const void *am_hdr, - size_t am_hdr_sz, const void *data, - MPI_Count count, MPI_Datatype datatype, - MPIR_Request * sreq) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_isendv(int rank, MPIR_Comm * comm, int handler_id, + struct iovec *am_hdrs, size_t iov_len, + const void *data, MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq, + void *netmod_context) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_send_hdr_reply(MPIR_Context_id_t context_id, + int src_rank, int handler_id, + const void *am_hdr, + size_t am_hdr_sz) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_isend_reply(MPIR_Context_id_t context_id, int src_rank, + int handler_id, const void *am_hdr, + size_t am_hdr_sz, const void *data, + MPI_Count count, MPI_Datatype datatype, + MPIR_Request * sreq) MPL_STATIC_INLINE_SUFFIX; MPL_STATIC_INLINE_PREFIX size_t MPIDI_NM_am_hdr_max_sz(void) MPL_STATIC_INLINE_SUFFIX; MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_recv(MPIR_Request * req) MPL_STATIC_INLINE_SUFFIX; MPL_STATIC_INLINE_PREFIX int MPIDI_NM_comm_get_lpid(MPIR_Comm * comm_ptr, int idx, diff --git a/src/mpid/ch4/netmod/include/netmod_impl.h b/src/mpid/ch4/netmod/include/netmod_impl.h index dc4f022..781da99 100644 --- a/src/mpid/ch4/netmod/include/netmod_impl.h +++ b/src/mpid/ch4/netmod/include/netmod_impl.h @@ -72,48 +72,48 @@ MPL_STATIC_INLINE_PREFIX int MPIDI_NM_comm_accept(const char *port_name, MPIR_In return MPIDI_NM_func->comm_accept(port_name, info, root, comm, newcomm_ptr); }; -MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_inject_hdr(int rank, MPIR_Comm * comm, int handler_id, - const void *am_hdr, size_t am_hdr_sz, - void *netmod_context) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_send_hdr(int rank, MPIR_Comm * comm, int handler_id, + const void *am_hdr, size_t am_hdr_sz, + void *netmod_context) { - return MPIDI_NM_func->am_inject_hdr(rank, comm, handler_id, am_hdr, am_hdr_sz, netmod_context); + return MPIDI_NM_func->am_send_hdr(rank, comm, handler_id, am_hdr, am_hdr_sz, netmod_context); }; -MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_send(int rank, MPIR_Comm * comm, int handler_id, - const void *am_hdr, size_t am_hdr_sz, - const void *data, MPI_Count count, - MPI_Datatype datatype, MPIR_Request * sreq, - void *netmod_context) -{ - return MPIDI_NM_func->am_send(rank, comm, handler_id, am_hdr, am_hdr_sz, data, count, datatype, - sreq, netmod_context); -}; - -MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_sendv(int rank, MPIR_Comm * comm, int handler_id, - struct iovec *am_hdrs, size_t iov_len, +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_isend(int rank, MPIR_Comm * comm, int handler_id, + const void *am_hdr, size_t am_hdr_sz, const void *data, MPI_Count count, MPI_Datatype datatype, MPIR_Request * sreq, void *netmod_context) { - return MPIDI_NM_func->am_sendv(rank, comm, handler_id, am_hdrs, iov_len, data, count, datatype, + return MPIDI_NM_func->am_isend(rank, comm, handler_id, am_hdr, am_hdr_sz, data, count, datatype, sreq, netmod_context); }; -MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_inject_hdr_reply(MPIR_Context_id_t context_id, - int src_rank, int handler_id, - const void *am_hdr, size_t am_hdr_sz) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_isendv(int rank, MPIR_Comm * comm, int handler_id, + struct iovec *am_hdrs, size_t iov_len, + const void *data, MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq, + void *netmod_context) +{ + return MPIDI_NM_func->am_isendv(rank, comm, handler_id, am_hdrs, iov_len, data, count, datatype, + sreq, netmod_context); +}; + +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_send_hdr_reply(MPIR_Context_id_t context_id, + int src_rank, int handler_id, + const void *am_hdr, size_t am_hdr_sz) { - return MPIDI_NM_func->am_inject_hdr_reply(context_id, src_rank, handler_id, am_hdr, am_hdr_sz); + return MPIDI_NM_func->am_send_hdr_reply(context_id, src_rank, handler_id, am_hdr, am_hdr_sz); }; -MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_send_reply(MPIR_Context_id_t context_id, int src_rank, - int handler_id, const void *am_hdr, - size_t am_hdr_sz, const void *data, - MPI_Count count, MPI_Datatype datatype, - MPIR_Request * sreq) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_isend_reply(MPIR_Context_id_t context_id, int src_rank, + int handler_id, const void *am_hdr, + size_t am_hdr_sz, const void *data, + MPI_Count count, MPI_Datatype datatype, + MPIR_Request * sreq) { - return MPIDI_NM_func->am_send_reply(context_id, src_rank, handler_id, am_hdr, am_hdr_sz, data, - count, datatype, sreq); + return MPIDI_NM_func->am_isend_reply(context_id, src_rank, handler_id, am_hdr, am_hdr_sz, data, + count, datatype, sreq); }; MPL_STATIC_INLINE_PREFIX size_t MPIDI_NM_am_hdr_max_sz(void) diff --git a/src/mpid/ch4/netmod/ofi/func_table.c b/src/mpid/ch4/netmod/ofi/func_table.c index ae06b7b..bed0f63 100644 --- a/src/mpid/ch4/netmod/ofi/func_table.c +++ b/src/mpid/ch4/netmod/ofi/func_table.c @@ -32,11 +32,11 @@ MPIDI_NM_funcs_t MPIDI_NM_ofi_funcs = { MPIDI_NM_am_request_init, MPIDI_NM_am_request_finalize, MPIDI_NM_am_reg_handler, - MPIDI_NM_am_inject_hdr, - MPIDI_NM_am_send, - MPIDI_NM_am_sendv, - MPIDI_NM_am_inject_hdr_reply, - MPIDI_NM_am_send_reply, + MPIDI_NM_am_send_hdr, + MPIDI_NM_am_isend, + MPIDI_NM_am_isendv, + MPIDI_NM_am_send_hdr_reply, + MPIDI_NM_am_isend_reply, MPIDI_NM_am_hdr_max_sz, MPIDI_NM_am_recv }; diff --git a/src/mpid/ch4/netmod/ofi/ofi_am.h b/src/mpid/ch4/netmod/ofi/ofi_am.h index f139399..c0f1257 100644 --- a/src/mpid/ch4/netmod/ofi/ofi_am.h +++ b/src/mpid/ch4/netmod/ofi/ofi_am.h @@ -44,7 +44,7 @@ static inline int MPIDI_NM_am_reg_handler(int handler_id, } MPIDI_Global.am_handlers[handler_id] = target_handler_fn; - MPIDI_Global.am_send_cmpl_handlers[handler_id] = origin_handler_fn; + MPIDI_Global.am_isend_cmpl_handlers[handler_id] = origin_handler_fn; fn_exit: MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_REG_HDR_HANDLER); return mpi_errno; @@ -53,45 +53,46 @@ static inline int MPIDI_NM_am_reg_handler(int handler_id, } #undef FUNCNAME -#define FUNCNAME MPIDI_NM_am_send +#define FUNCNAME MPIDI_NM_am_isend #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_NM_am_send(int rank, - MPIR_Comm * comm, - int handler_id, - const void *am_hdr, - size_t am_hdr_sz, - const void *data, - MPI_Count count, - MPI_Datatype datatype, MPIR_Request * sreq, void *netmod_context) +static inline int MPIDI_NM_am_isend(int rank, + MPIR_Comm * comm, + int handler_id, + const void *am_hdr, + size_t am_hdr_sz, + const void *data, + MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq, + void *netmod_context) { int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_SEND_AM); MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_SEND_AM); if (count) - mpi_errno = MPIDI_OFI_do_am_send(rank, comm, handler_id, - am_hdr, am_hdr_sz, data, count, datatype, sreq, FALSE); + mpi_errno = MPIDI_OFI_do_am_isend(rank, comm, handler_id, + am_hdr, am_hdr_sz, data, count, datatype, sreq, FALSE); else - mpi_errno = MPIDI_OFI_do_am_send_header(rank, comm, handler_id, - am_hdr, am_hdr_sz, sreq, FALSE); + mpi_errno = MPIDI_OFI_do_am_isend_header(rank, comm, handler_id, + am_hdr, am_hdr_sz, sreq, FALSE); MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_SEND_AM); return mpi_errno; } #undef FUNCNAME -#define FUNCNAME MPIDI_NM_am_sendv +#define FUNCNAME MPIDI_NM_am_isendv #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_NM_am_sendv(int rank, - MPIR_Comm * comm, - int handler_id, - struct iovec *am_hdr, - size_t iov_len, - const void *data, - MPI_Count count, - MPI_Datatype datatype, - MPIR_Request * sreq, void *netmod_context) +static inline int MPIDI_NM_am_isendv(int rank, + MPIR_Comm * comm, + int handler_id, + struct iovec *am_hdr, + size_t iov_len, + const void *data, + MPI_Count count, + MPI_Datatype datatype, + MPIR_Request * sreq, void *netmod_context) { int mpi_errno = MPI_SUCCESS, is_allocated; size_t am_hdr_sz = 0, i; @@ -121,8 +122,8 @@ static inline int MPIDI_NM_am_sendv(int rank, am_hdr_sz += am_hdr[i].iov_len; } - mpi_errno = MPIDI_NM_am_send(rank, comm, handler_id, am_hdr_buf, am_hdr_sz, - data, count, datatype, sreq, netmod_context); + mpi_errno = MPIDI_NM_am_isend(rank, comm, handler_id, am_hdr_buf, am_hdr_sz, + data, count, datatype, sreq, netmod_context); if (is_allocated) MPL_free(am_hdr_buf); @@ -134,30 +135,30 @@ static inline int MPIDI_NM_am_sendv(int rank, } #undef FUNCNAME -#define FUNCNAME MPIDI_NM_am_send_reply +#define FUNCNAME MPIDI_NM_am_isend_reply #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_NM_am_send_reply(MPIR_Context_id_t context_id, - int src_rank, - int handler_id, - const void *am_hdr, - size_t am_hdr_sz, - const void *data, - MPI_Count count, - MPI_Datatype datatype, MPIR_Request * sreq) +static inline int MPIDI_NM_am_isend_reply(MPIR_Context_id_t context_id, + int src_rank, + int handler_id, + const void *am_hdr, + size_t am_hdr_sz, + const void *data, + MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq) { int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_SEND_AM_REPLY); MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_SEND_AM_REPLY); if (count) - mpi_errno = MPIDI_OFI_do_am_send(src_rank, - MPIDI_CH4U_context_id_to_comm(context_id), - handler_id, - am_hdr, am_hdr_sz, data, count, datatype, sreq, TRUE); + mpi_errno = MPIDI_OFI_do_am_isend(src_rank, + MPIDI_CH4U_context_id_to_comm(context_id), + handler_id, + am_hdr, am_hdr_sz, data, count, datatype, sreq, TRUE); else - mpi_errno = MPIDI_OFI_do_am_send_header(src_rank, - MPIDI_CH4U_context_id_to_comm(context_id), - handler_id, am_hdr, am_hdr_sz, sreq, TRUE); + mpi_errno = MPIDI_OFI_do_am_isend_header(src_rank, + MPIDI_CH4U_context_id_to_comm(context_id), + handler_id, am_hdr, am_hdr_sz, sreq, TRUE); MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_SEND_AM_REPLY); return mpi_errno; } @@ -173,10 +174,10 @@ static inline size_t MPIDI_NM_am_hdr_max_sz(void) return MPL_MIN(max_shortsend, max_representable); } -static inline int MPIDI_NM_am_inject_hdr(int rank, - MPIR_Comm * comm, - int handler_id, - const void *am_hdr, size_t am_hdr_sz, void *netmod_context) +static inline int MPIDI_NM_am_send_hdr(int rank, + MPIR_Comm * comm, + int handler_id, + const void *am_hdr, size_t am_hdr_sz, void *netmod_context) { int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_INJECT_AM_HDR); @@ -195,9 +196,9 @@ static inline int MPIDI_NM_am_inject_hdr(int rank, goto fn_exit; } -static inline int MPIDI_NM_am_inject_hdr_reply(MPIR_Context_id_t context_id, - int src_rank, - int handler_id, const void *am_hdr, size_t am_hdr_sz) +static inline int MPIDI_NM_am_send_hdr_reply(MPIR_Context_id_t context_id, + int src_rank, + int handler_id, const void *am_hdr, size_t am_hdr_sz) { int mpi_errno = MPI_SUCCESS; @@ -228,9 +229,9 @@ static inline int MPIDI_NM_am_recv(MPIR_Request * req) msg.sreq_ptr = (MPIDI_CH4U_REQUEST(req, req->rreq.peer_req_ptr)); msg.rreq_ptr = (uint64_t) req; MPIR_Assert((void *) msg.sreq_ptr != NULL); - mpi_errno = MPIDI_NM_am_inject_hdr_reply(MPIDI_CH4U_get_context(MPIDI_CH4U_REQUEST(req, tag)), - MPIDI_CH4U_REQUEST(req, src_rank), - MPIDI_CH4U_SEND_LONG_ACK, &msg, sizeof(msg)); + mpi_errno = MPIDI_NM_am_send_hdr_reply(MPIDI_CH4U_get_context(MPIDI_CH4U_REQUEST(req, tag)), + MPIDI_CH4U_REQUEST(req, src_rank), + MPIDI_CH4U_SEND_LONG_ACK, &msg, sizeof(msg)); if (mpi_errno) MPIR_ERR_POP(mpi_errno); diff --git a/src/mpid/ch4/netmod/ofi/ofi_am_events.h b/src/mpid/ch4/netmod/ofi/ofi_am_events.h index fdc920c..a309a02 100644 --- a/src/mpid/ch4/netmod/ofi/ofi_am_events.h +++ b/src/mpid/ch4/netmod/ofi/ofi_am_events.h @@ -367,7 +367,7 @@ static inline int MPIDI_OFI_handle_lmt_ack(MPIDI_OFI_am_header_t * msg_hdr) handler_id = MPIDI_OFI_AMREQUEST_HDR(sreq, msg_hdr).handler_id; MPIDI_OFI_am_request_complete(sreq); - mpi_errno = MPIDI_Global.am_send_cmpl_handlers[handler_id] (sreq); + mpi_errno = MPIDI_Global.am_isend_cmpl_handlers[handler_id] (sreq); if (mpi_errno) MPIR_ERR_POP(mpi_errno); diff --git a/src/mpid/ch4/netmod/ofi/ofi_am_impl.h b/src/mpid/ch4/netmod/ofi/ofi_am_impl.h index 058c936..1f0537a 100644 --- a/src/mpid/ch4/netmod/ofi/ofi_am_impl.h +++ b/src/mpid/ch4/netmod/ofi/ofi_am_impl.h @@ -20,10 +20,10 @@ static inline int MPIDI_OFI_progress_do_queue(void *netmod_context); * When calling OFI function MPIDI_OFI_THREAD_FI_MUTEX must be held. * When being called from the MPI layer (app), we must grab the lock. - This is the case for regular (non-reply) functions such as am_send. + This is the case for regular (non-reply) functions such as am_isend. * When being called from callback function or progress engine, we must not grab the lock because the progress engine is already holding the lock. - This is the case for reply functions such as am_send_reply. + This is the case for reply functions such as am_isend_reply. */ #define MPIDI_OFI_CALL_RETRY_AM(FUNC,LOCK,STR) \ do { \ @@ -192,14 +192,14 @@ static inline int MPIDI_OFI_progress_do_queue(void *netmod_context) } #undef FUNCNAME -#define FUNCNAME MPIDI_OFI_do_am_send_header +#define FUNCNAME MPIDI_OFI_do_am_isend_header #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_OFI_do_am_send_header(int rank, - MPIR_Comm * comm, - int handler_id, - const void *am_hdr, - size_t am_hdr_sz, MPIR_Request * sreq, int is_reply) +static inline int MPIDI_OFI_do_am_isend_header(int rank, + MPIR_Comm * comm, + int handler_id, + const void *am_hdr, + size_t am_hdr_sz, MPIR_Request * sreq, int is_reply) { struct iovec iov[2]; MPIDI_OFI_am_header_t *msg_hdr; @@ -246,16 +246,16 @@ static inline int MPIDI_OFI_do_am_send_header(int rank, } #undef FUNCNAME -#define FUNCNAME MPIDI_OFI_am_send_long +#define FUNCNAME MPIDI_OFI_am_isend_long #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_OFI_am_send_long(int rank, - MPIR_Comm * comm, - int handler_id, - const void *am_hdr, - size_t am_hdr_sz, - const void *data, - size_t data_sz, MPIR_Request * sreq, int need_lock) +static inline int MPIDI_OFI_am_isend_long(int rank, + MPIR_Comm * comm, + int handler_id, + const void *am_hdr, + size_t am_hdr_sz, + const void *data, + size_t data_sz, MPIR_Request * sreq, int need_lock) { int mpi_errno = MPI_SUCCESS, c; MPIDI_OFI_am_header_t *msg_hdr; @@ -337,16 +337,16 @@ static inline int MPIDI_OFI_am_send_long(int rank, } #undef FUNCNAME -#define FUNCNAME MPIDI_OFI_am_send_short +#define FUNCNAME MPIDI_OFI_am_isend_short #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_OFI_am_send_short(int rank, - MPIR_Comm * comm, - int handler_id, - const void *am_hdr, - size_t am_hdr_sz, - const void *data, - MPI_Count count, MPIR_Request * sreq, int need_lock) +static inline int MPIDI_OFI_am_isend_short(int rank, + MPIR_Comm * comm, + int handler_id, + const void *am_hdr, + size_t am_hdr_sz, + const void *data, + MPI_Count count, MPIR_Request * sreq, int need_lock) { int mpi_errno = MPI_SUCCESS, c; MPIDI_OFI_am_header_t *msg_hdr; @@ -388,17 +388,17 @@ static inline int MPIDI_OFI_am_send_short(int rank, } #undef FUNCNAME -#define FUNCNAME MPIDI_OFI_do_am_send +#define FUNCNAME MPIDI_OFI_do_am_isend #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_OFI_do_am_send(int rank, - MPIR_Comm * comm, - int handler_id, - const void *am_hdr, - size_t am_hdr_sz, - const void *buf, - size_t count, - MPI_Datatype datatype, MPIR_Request * sreq, int is_reply) +static inline int MPIDI_OFI_do_am_isend(int rank, + MPIR_Comm * comm, + int handler_id, + const void *am_hdr, + size_t am_hdr_sz, + const void *buf, + size_t count, + MPI_Datatype datatype, MPIR_Request * sreq, int is_reply) { int dt_contig, mpi_errno = MPI_SUCCESS; char *send_buf; @@ -426,8 +426,8 @@ static inline int MPIDI_OFI_do_am_send(int rank, MPIDI_CH4U_REQUEST(sreq, req->lreq).datatype = datatype; MPIDI_CH4U_REQUEST(sreq, req->lreq).msg_tag = lreq_hdr.hdr.msg_tag; MPIDI_CH4U_REQUEST(sreq, src_rank) = rank; - mpi_errno = MPIDI_NM_am_inject_hdr(rank, comm, MPIDI_CH4U_SEND_LONG_REQ, - &lreq_hdr, sizeof(lreq_hdr), NULL); + mpi_errno = MPIDI_NM_am_send_hdr(rank, comm, MPIDI_CH4U_SEND_LONG_REQ, + &lreq_hdr, sizeof(lreq_hdr), NULL); if (mpi_errno) MPIR_ERR_POP(mpi_errno); goto fn_exit; @@ -461,13 +461,13 @@ static inline int MPIDI_OFI_do_am_send(int rank, if (am_hdr_sz + data_sz + sizeof(MPIDI_OFI_am_header_t) <= MPIDI_OFI_DEFAULT_SHORT_SEND_SIZE) { mpi_errno = - MPIDI_OFI_am_send_short(rank, comm, handler_id, MPIDI_OFI_AMREQUEST_HDR(sreq, am_hdr), - am_hdr_sz, send_buf, data_sz, sreq, need_lock); + MPIDI_OFI_am_isend_short(rank, comm, handler_id, MPIDI_OFI_AMREQUEST_HDR(sreq, am_hdr), + am_hdr_sz, send_buf, data_sz, sreq, need_lock); } else { mpi_errno = - MPIDI_OFI_am_send_long(rank, comm, handler_id, MPIDI_OFI_AMREQUEST_HDR(sreq, am_hdr), - am_hdr_sz, send_buf, data_sz, sreq, need_lock); + MPIDI_OFI_am_isend_long(rank, comm, handler_id, MPIDI_OFI_AMREQUEST_HDR(sreq, am_hdr), + am_hdr_sz, send_buf, data_sz, sreq, need_lock); } if (mpi_errno) MPIR_ERR_POP(mpi_errno); diff --git a/src/mpid/ch4/netmod/ofi/ofi_events.h b/src/mpid/ch4/netmod/ofi/ofi_events.h index 3ddb664..191bc51 100644 --- a/src/mpid/ch4/netmod/ofi/ofi_events.h +++ b/src/mpid/ch4/netmod/ofi/ofi_events.h @@ -444,10 +444,10 @@ __ALWAYS_INLINE__ int MPIDI_OFI_dynproc_done_event(struct fi_cq_tagged_entry *wc } #undef FUNCNAME -#define FUNCNAME MPIDI_OFI_am_send_event +#define FUNCNAME MPIDI_OFI_am_isend_event #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__ALWAYS_INLINE__ int MPIDI_OFI_am_send_event(struct fi_cq_tagged_entry *wc, MPIR_Request * sreq) +__ALWAYS_INLINE__ int MPIDI_OFI_am_isend_event(struct fi_cq_tagged_entry *wc, MPIR_Request * sreq) { int mpi_errno = MPI_SUCCESS; MPIDI_OFI_am_header_t *msg_hdr; @@ -472,7 +472,7 @@ __ALWAYS_INLINE__ int MPIDI_OFI_am_send_event(struct fi_cq_tagged_entry *wc, MPI MPIDI_OFI_AMREQUEST_HDR(sreq, pack_buffer) = NULL; } - mpi_errno = MPIDI_Global.am_send_cmpl_handlers[msg_hdr->handler_id] (sreq); + mpi_errno = MPIDI_Global.am_isend_cmpl_handlers[msg_hdr->handler_id] (sreq); if (mpi_errno) MPIR_ERR_POP(mpi_errno); @@ -615,7 +615,7 @@ __ALWAYS_INLINE__ int MPIDI_OFI_dispatch_function(struct fi_cq_tagged_entry *wc, goto fn_exit; } else if (likely(MPIDI_OFI_REQUEST(req, event_id) == MPIDI_OFI_EVENT_AM_SEND)) { - mpi_errno = MPIDI_OFI_am_send_event(wc, req); + mpi_errno = MPIDI_OFI_am_isend_event(wc, req); goto fn_exit; } else if (likely(MPIDI_OFI_REQUEST(req, event_id) == MPIDI_OFI_EVENT_AM_RECV)) { diff --git a/src/mpid/ch4/netmod/ofi/ofi_init.h b/src/mpid/ch4/netmod/ofi/ofi_init.h index f6655df..d8a9579 100644 --- a/src/mpid/ch4/netmod/ofi/ofi_init.h +++ b/src/mpid/ch4/netmod/ofi/ofi_init.h @@ -390,7 +390,7 @@ static inline int MPIDI_OFI_init_generic(int rank, /* ---------------------------------- */ if (do_am) { /* Maximum possible message size for short message send (=eager send) - * See MPIDI_OFI_do_am_send for short/long switching logic */ + * See MPIDI_OFI_do_am_isend for short/long switching logic */ MPIR_Assert(MPIDI_OFI_DEFAULT_SHORT_SEND_SIZE <= MPIDI_Global.max_send); MPIDI_Global.am_buf_pool = MPIDI_CH4U_create_buf_pool(MPIDI_OFI_BUF_POOL_NUM, MPIDI_OFI_BUF_POOL_SIZE); @@ -427,7 +427,7 @@ static inline int MPIDI_OFI_init_generic(int rank, /* Grow the header handlers down */ MPIDI_Global.am_handlers[MPIDI_OFI_INTERNAL_HANDLER_CONTROL] = MPIDI_OFI_control_handler; - MPIDI_Global.am_send_cmpl_handlers[MPIDI_OFI_INTERNAL_HANDLER_CONTROL] = NULL; + MPIDI_Global.am_isend_cmpl_handlers[MPIDI_OFI_INTERNAL_HANDLER_CONTROL] = NULL; } OPA_store_int(&MPIDI_Global.am_inflight_inject_emus, 0); OPA_store_int(&MPIDI_Global.am_inflight_rma_send_mrs, 0); diff --git a/src/mpid/ch4/netmod/ofi/ofi_recv.h b/src/mpid/ch4/netmod/ofi/ofi_recv.h index 2f9e0e8..bed547d 100644 --- a/src/mpid/ch4/netmod/ofi/ofi_recv.h +++ b/src/mpid/ch4/netmod/ofi/ofi_recv.h @@ -277,7 +277,7 @@ __ALWAYS_INLINE__ int MPIDI_NM_cancel_recv(MPIR_Request * rreq) return mpi_errno; #ifndef MPIDI_BUILD_CH4_SHM -fn_fail: + fn_fail: goto fn_exit; #endif } diff --git a/src/mpid/ch4/netmod/ofi/ofi_types.h b/src/mpid/ch4/netmod/ofi/ofi_types.h index a501d7e..045cc96 100644 --- a/src/mpid/ch4/netmod/ofi/ofi_types.h +++ b/src/mpid/ch4/netmod/ofi/ofi_types.h @@ -380,7 +380,7 @@ typedef struct { void *am_bufs[MPIDI_OFI_NUM_AM_BUFFERS]; MPIDI_OFI_am_repost_request_t am_reqs[MPIDI_OFI_NUM_AM_BUFFERS]; MPIDI_NM_am_target_handler_fn am_handlers[MPIDI_OFI_MAX_AM_HANDLERS_TOTAL]; - MPIDI_NM_am_origin_handler_fn am_send_cmpl_handlers[MPIDI_OFI_MAX_AM_HANDLERS_TOTAL]; + MPIDI_NM_am_origin_handler_fn am_isend_cmpl_handlers[MPIDI_OFI_MAX_AM_HANDLERS_TOTAL]; MPIU_buf_pool_t *am_buf_pool; OPA_int_t am_inflight_inject_emus; OPA_int_t am_inflight_rma_send_mrs; diff --git a/src/mpid/ch4/netmod/portals4/func_table.c b/src/mpid/ch4/netmod/portals4/func_table.c index eade09e..8d39e10 100644 --- a/src/mpid/ch4/netmod/portals4/func_table.c +++ b/src/mpid/ch4/netmod/portals4/func_table.c @@ -32,11 +32,11 @@ MPIDI_NM_funcs_t MPIDI_NM_portals4_funcs = { MPIDI_NM_am_request_init, MPIDI_NM_am_request_finalize, MPIDI_NM_am_reg_handler, - MPIDI_NM_am_inject_hdr, - MPIDI_NM_am_send, - MPIDI_NM_am_sendv, - MPIDI_NM_am_inject_hdr_reply, - MPIDI_NM_am_send_reply, + MPIDI_NM_am_send_hdr, + MPIDI_NM_am_isend, + MPIDI_NM_am_isendv, + MPIDI_NM_am_send_hdr_reply, + MPIDI_NM_am_isend_reply, MPIDI_NM_am_hdr_max_sz, MPIDI_NM_am_recv, }; diff --git a/src/mpid/ch4/netmod/portals4/ptl_am.h b/src/mpid/ch4/netmod/portals4/ptl_am.h index 0ff7db0..cbb9d14 100644 --- a/src/mpid/ch4/netmod/portals4/ptl_am.h +++ b/src/mpid/ch4/netmod/portals4/ptl_am.h @@ -34,14 +34,15 @@ static inline int MPIDI_NM_am_reg_handler(int handler_id, goto fn_exit; } -static inline int MPIDI_NM_am_send(int rank, - MPIR_Comm * comm, - int handler_id, - const void *am_hdr, - size_t am_hdr_sz, - const void *data, - MPI_Count count, - MPI_Datatype datatype, MPIR_Request * sreq, void *netmod_context) +static inline int MPIDI_NM_am_isend(int rank, + MPIR_Comm * comm, + int handler_id, + const void *am_hdr, + size_t am_hdr_sz, + const void *data, + MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq, + void *netmod_context) { int mpi_errno = MPI_SUCCESS, ret, c; size_t data_sz; @@ -129,28 +130,28 @@ static inline int MPIDI_NM_am_send(int rank, goto fn_exit; } -static inline int MPIDI_NM_am_sendv(int rank, - MPIR_Comm * comm, - int handler_id, - struct iovec *am_hdr, - size_t iov_len, - const void *data, - MPI_Count count, - MPI_Datatype datatype, - MPIR_Request * sreq, void *netmod_context) +static inline int MPIDI_NM_am_isendv(int rank, + MPIR_Comm * comm, + int handler_id, + struct iovec *am_hdr, + size_t iov_len, + const void *data, + MPI_Count count, + MPI_Datatype datatype, + MPIR_Request * sreq, void *netmod_context) { MPIR_Assert(0); return MPI_SUCCESS; } -static inline int MPIDI_NM_am_send_reply(MPIR_Context_id_t context_id, - int src_rank, - int handler_id, - const void *am_hdr, - size_t am_hdr_sz, - const void *data, - MPI_Count count, - MPI_Datatype datatype, MPIR_Request * sreq) +static inline int MPIDI_NM_am_isend_reply(MPIR_Context_id_t context_id, + int src_rank, + int handler_id, + const void *am_hdr, + size_t am_hdr_sz, + const void *data, + MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq) { int mpi_errno = MPI_SUCCESS, ret, c; size_t data_sz; @@ -231,10 +232,10 @@ static inline size_t MPIDI_NM_am_hdr_max_sz(void) return 0; } -static inline int MPIDI_NM_am_inject_hdr(int rank, - MPIR_Comm * comm, - int handler_id, - const void *am_hdr, size_t am_hdr_sz, void *netmod_context) +static inline int MPIDI_NM_am_send_hdr(int rank, + MPIR_Comm * comm, + int handler_id, + const void *am_hdr, size_t am_hdr_sz, void *netmod_context) { int mpi_errno = MPI_SUCCESS, ret, c; ptl_hdr_data_t ptl_hdr; @@ -266,9 +267,9 @@ static inline int MPIDI_NM_am_inject_hdr(int rank, goto fn_exit; } -static inline int MPIDI_NM_am_inject_hdr_reply(MPIR_Context_id_t context_id, - int src_rank, - int handler_id, const void *am_hdr, size_t am_hdr_sz) +static inline int MPIDI_NM_am_send_hdr_reply(MPIR_Context_id_t context_id, + int src_rank, + int handler_id, const void *am_hdr, size_t am_hdr_sz) { int mpi_errno = MPI_SUCCESS, ret, c; ptl_hdr_data_t ptl_hdr; diff --git a/src/mpid/ch4/netmod/stubnm/globals.c b/src/mpid/ch4/netmod/stubnm/globals.c index 89a5cc9..9dc545b 100644 --- a/src/mpid/ch4/netmod/stubnm/globals.c +++ b/src/mpid/ch4/netmod/stubnm/globals.c @@ -32,11 +32,11 @@ MPIDI_NM_funcs_t MPIDI_NM_stubnm_funcs = { MPIDI_NM_am_request_init, MPIDI_NM_am_request_finalize, MPIDI_NM_am_reg_handler, - MPIDI_NM_am_inject_hdr, - MPIDI_NM_am_send, - MPIDI_NM_am_sendv, - MPIDI_NM_am_inject_hdr_reply, - MPIDI_NM_am_send_reply, + MPIDI_NM_am_send_hdr, + MPIDI_NM_am_isend, + MPIDI_NM_am_isendv, + MPIDI_NM_am_send_hdr_reply, + MPIDI_NM_am_isend_reply, MPIDI_NM_am_hdr_max_sz, MPIDI_NM_am_recv, }; diff --git a/src/mpid/ch4/netmod/stubnm/stubnm_am.h b/src/mpid/ch4/netmod/stubnm/stubnm_am.h index f33e1c3..e3986d2 100644 --- a/src/mpid/ch4/netmod/stubnm/stubnm_am.h +++ b/src/mpid/ch4/netmod/stubnm/stubnm_am.h @@ -21,40 +21,41 @@ static inline int MPIDI_NM_am_reg_handler(int handler_id, return MPI_SUCCESS; } -static inline int MPIDI_NM_am_send(int rank, - MPIR_Comm * comm, - int handler_id, - const void *am_hdr, - size_t am_hdr_sz, - const void *data, - MPI_Count count, - MPI_Datatype datatype, MPIR_Request * sreq, void *netmod_context) +static inline int MPIDI_NM_am_isend(int rank, + MPIR_Comm * comm, + int handler_id, + const void *am_hdr, + size_t am_hdr_sz, + const void *data, + MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq, + void *netmod_context) { MPIR_Assert(0); return MPI_SUCCESS; } -static inline int MPIDI_NM_am_sendv(int rank, - MPIR_Comm * comm, - int handler_id, - struct iovec *am_hdr, - size_t iov_len, - const void *data, - MPI_Count count, - MPI_Datatype datatype, - MPIR_Request * sreq, void *netmod_context) +static inline int MPIDI_NM_am_isendv(int rank, + MPIR_Comm * comm, + int handler_id, + struct iovec *am_hdr, + size_t iov_len, + const void *data, + MPI_Count count, + MPI_Datatype datatype, + MPIR_Request * sreq, void *netmod_context) { MPIR_Assert(0); return MPI_SUCCESS; } -static inline int MPIDI_NM_am_send_reply(MPIR_Context_id_t context_id, int src_rank, - int handler_id, - const void *am_hdr, - size_t am_hdr_sz, - const void *data, - MPI_Count count, - MPI_Datatype datatype, MPIR_Request * sreq) +static inline int MPIDI_NM_am_isend_reply(MPIR_Context_id_t context_id, int src_rank, + int handler_id, + const void *am_hdr, + size_t am_hdr_sz, + const void *data, + MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq) { MPIR_Assert(0); return MPI_SUCCESS; @@ -66,17 +67,17 @@ static inline size_t MPIDI_NM_am_hdr_max_sz(void) return 0; } -static inline int MPIDI_NM_am_inject_hdr(int rank, - MPIR_Comm * comm, - int handler_id, - const void *am_hdr, size_t am_hdr_sz, void *netmod_context) +static inline int MPIDI_NM_am_send_hdr(int rank, + MPIR_Comm * comm, + int handler_id, + const void *am_hdr, size_t am_hdr_sz, void *netmod_context) { MPIR_Assert(0); return MPI_SUCCESS; } -static inline int MPIDI_NM_am_inject_hdr_reply(MPIR_Context_id_t context_id, int src_rank, - int handler_id, const void *am_hdr, size_t am_hdr_sz) +static inline int MPIDI_NM_am_send_hdr_reply(MPIR_Context_id_t context_id, int src_rank, + int handler_id, const void *am_hdr, size_t am_hdr_sz) { MPIR_Assert(0); return MPI_SUCCESS; diff --git a/src/mpid/ch4/netmod/ucx/func_table.c b/src/mpid/ch4/netmod/ucx/func_table.c index 121a197..e4801eb 100644 --- a/src/mpid/ch4/netmod/ucx/func_table.c +++ b/src/mpid/ch4/netmod/ucx/func_table.c @@ -30,11 +30,11 @@ MPIDI_NM_funcs_t MPIDI_NM_ucx_funcs = { MPIDI_NM_am_request_init, MPIDI_NM_am_request_finalize, MPIDI_NM_am_reg_handler, - MPIDI_NM_am_inject_hdr, - MPIDI_NM_am_send, - MPIDI_NM_am_sendv, - MPIDI_NM_am_inject_hdr_reply, - MPIDI_NM_am_send_reply, + MPIDI_NM_am_send_hdr, + MPIDI_NM_am_isend, + MPIDI_NM_am_isendv, + MPIDI_NM_am_send_hdr_reply, + MPIDI_NM_am_isend_reply, MPIDI_NM_am_hdr_max_sz, MPIDI_NM_am_recv }; diff --git a/src/mpid/ch4/netmod/ucx/ucx_am.h b/src/mpid/ch4/netmod/ucx/ucx_am.h index 3277010..4d1f8ec 100644 --- a/src/mpid/ch4/netmod/ucx/ucx_am.h +++ b/src/mpid/ch4/netmod/ucx/ucx_am.h @@ -33,7 +33,7 @@ static inline int MPIDI_NM_am_reg_handler(int handler_id, } -static inline void MPIDI_UCX_am_send_callback(void *request, ucs_status_t status) +static inline void MPIDI_UCX_am_isend_callback(void *request, ucs_status_t status) { MPIDI_UCX_ucp_request_t *ucp_request = (MPIDI_UCX_ucp_request_t *) request; @@ -76,17 +76,18 @@ static inline void MPIDI_UCX_inject_am_callback(void *request, ucs_status_t stat } #undef FUNCNAME -#define FUNCNAME MPIDI_NM_am_send +#define FUNCNAME MPIDI_NM_am_isend #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_NM_am_send(int rank, - MPIR_Comm * comm, - int handler_id, - const void *am_hdr, - size_t am_hdr_sz, - const void *data, - MPI_Count count, - MPI_Datatype datatype, MPIR_Request * sreq, void *netmod_context) +static inline int MPIDI_NM_am_isend(int rank, + MPIR_Comm * comm, + int handler_id, + const void *am_hdr, + size_t am_hdr_sz, + const void *data, + MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq, + void *netmod_context) { int mpi_errno = MPI_SUCCESS, c; MPIDI_UCX_ucp_request_t *ucp_request; @@ -116,8 +117,8 @@ static inline int MPIDI_NM_am_send(int rank, MPIDI_CH4U_REQUEST(sreq, req->lreq).datatype = datatype; MPIDI_CH4U_REQUEST(sreq, req->lreq).msg_tag = lreq_hdr.hdr.msg_tag; MPIDI_CH4U_REQUEST(sreq, src_rank) = rank; - mpi_errno = MPIDI_NM_am_inject_hdr(rank, comm, MPIDI_CH4U_SEND_LONG_REQ, - &lreq_hdr, sizeof(lreq_hdr), NULL); + mpi_errno = MPIDI_NM_am_send_hdr(rank, comm, MPIDI_CH4U_SEND_LONG_REQ, + &lreq_hdr, sizeof(lreq_hdr), NULL); if (mpi_errno) MPIR_ERR_POP(mpi_errno); goto fn_exit; @@ -158,7 +159,7 @@ static inline int MPIDI_NM_am_send(int rank, ucp_request = (MPIDI_UCX_ucp_request_t *) ucp_tag_send_nb(ep, send_buf, data_sz + am_hdr_sz + sizeof(ucx_hdr), ucp_dt_make_contig(1), ucx_tag, - &MPIDI_UCX_am_send_callback); + &MPIDI_UCX_am_isend_callback); MPIDI_CH4_UCX_REQUEST(ucp_request, tag_send_nb); /* send is done. free all resources and complete the request */ if (ucp_request == NULL) { @@ -191,18 +192,18 @@ static inline int MPIDI_NM_am_send(int rank, } #undef FUNCNAME -#define FUNCNAME MPIDI_NM_am_sendv +#define FUNCNAME MPIDI_NM_am_isendv #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_NM_am_sendv(int rank, - MPIR_Comm * comm, - int handler_id, - struct iovec *am_hdr, - size_t iov_len, - const void *data, - MPI_Count count, - MPI_Datatype datatype, - MPIR_Request * sreq, void *netmod_context) +static inline int MPIDI_NM_am_isendv(int rank, + MPIR_Comm * comm, + int handler_id, + struct iovec *am_hdr, + size_t iov_len, + const void *data, + MPI_Count count, + MPI_Datatype datatype, + MPIR_Request * sreq, void *netmod_context) { int mpi_errno = MPI_SUCCESS; size_t am_hdr_sz = 0, i; @@ -224,8 +225,8 @@ static inline int MPIDI_NM_am_sendv(int rank, am_hdr_sz += am_hdr[i].iov_len; } - mpi_errno = MPIDI_NM_am_send(rank, comm, handler_id, am_hdr_buf, am_hdr_sz, - data, count, datatype, sreq, netmod_context); + mpi_errno = MPIDI_NM_am_isend(rank, comm, handler_id, am_hdr_buf, am_hdr_sz, + data, count, datatype, sreq, netmod_context); MPL_free(am_hdr_buf); MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_UCX_SEND_AMV); @@ -234,16 +235,16 @@ static inline int MPIDI_NM_am_sendv(int rank, #undef FUNCNAME -#define FUNCNAME MPIDI_NM_am_send_reply +#define FUNCNAME MPIDI_NM_am_isend_reply #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_NM_am_send_reply(MPIR_Context_id_t context_id, - int src_rank, - int handler_id, - const void *am_hdr, - size_t am_hdr_sz, - const void *data, MPI_Count count, - MPI_Datatype datatype, MPIR_Request * sreq) +static inline int MPIDI_NM_am_isend_reply(MPIR_Context_id_t context_id, + int src_rank, + int handler_id, + const void *am_hdr, + size_t am_hdr_sz, + const void *data, MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq) { int mpi_errno = MPI_SUCCESS, c; MPIDI_UCX_ucp_request_t *ucp_request; @@ -281,7 +282,7 @@ static inline int MPIDI_NM_am_send_reply(MPIR_Context_id_t context_id, data_sz + am_hdr_sz + sizeof(ucx_hdr), ucp_dt_make_contig(1), ucx_tag, - &MPIDI_UCX_am_send_callback); + &MPIDI_UCX_am_isend_callback); MPIDI_CH4_UCX_REQUEST(ucp_request, tag_send_nb); } @@ -321,10 +322,10 @@ static inline size_t MPIDI_NM_am_hdr_max_sz(void) return (MPIDI_UCX_MAX_AM_EAGER_SZ - sizeof(MPIDI_UCX_am_header_t)); } -static inline int MPIDI_NM_am_inject_hdr(int rank, - MPIR_Comm * comm, - int handler_id, - const void *am_hdr, size_t am_hdr_sz, void *netmod_context) +static inline int MPIDI_NM_am_send_hdr(int rank, + MPIR_Comm * comm, + int handler_id, + const void *am_hdr, size_t am_hdr_sz, void *netmod_context) { int mpi_errno = MPI_SUCCESS, c; MPIDI_UCX_ucp_request_t *ucp_request; @@ -374,9 +375,9 @@ static inline int MPIDI_NM_am_inject_hdr(int rank, goto fn_exit; } -static inline int MPIDI_NM_am_inject_hdr_reply(MPIR_Context_id_t context_id, - int src_rank, - int handler_id, const void *am_hdr, size_t am_hdr_sz) +static inline int MPIDI_NM_am_send_hdr_reply(MPIR_Context_id_t context_id, + int src_rank, + int handler_id, const void *am_hdr, size_t am_hdr_sz) { int mpi_errno = MPI_SUCCESS, c; MPIDI_UCX_ucp_request_t *ucp_request; @@ -437,9 +438,9 @@ static inline int MPIDI_NM_am_recv(MPIR_Request * req) msg.sreq_ptr = (MPIDI_CH4U_REQUEST(req, req->rreq.peer_req_ptr)); msg.rreq_ptr = (uint64_t) req; MPIR_Assert((void *) msg.sreq_ptr != NULL); - mpi_errno = MPIDI_NM_am_inject_hdr_reply(MPIDI_CH4U_get_context(MPIDI_CH4U_REQUEST(req, tag)), - MPIDI_CH4U_REQUEST(req, src_rank), - MPIDI_CH4U_SEND_LONG_ACK, &msg, sizeof(msg)); + mpi_errno = MPIDI_NM_am_send_hdr_reply(MPIDI_CH4U_get_context(MPIDI_CH4U_REQUEST(req, tag)), + MPIDI_CH4U_REQUEST(req, src_rank), + MPIDI_CH4U_SEND_LONG_ACK, &msg, sizeof(msg)); if (mpi_errno) MPIR_ERR_POP(mpi_errno); diff --git a/src/mpid/ch4/shm/include/shm.h b/src/mpid/ch4/shm/include/shm.h index 1013990..7b90ecb 100644 --- a/src/mpid/ch4/shm/include/shm.h +++ b/src/mpid/ch4/shm/include/shm.h @@ -40,28 +40,27 @@ typedef int (*MPIDI_SHM_open_port_t) (MPIR_Info * info_ptr, char *port_name); typedef int (*MPIDI_SHM_close_port_t) (const char *port_name); typedef int (*MPIDI_SHM_comm_accept_t) (const char *port_name, MPIR_Info * info, int root, MPIR_Comm * comm, MPIR_Comm ** newcomm_ptr); -typedef int (*MPIDI_SHM_am_inject_hdr_t) (int rank, MPIR_Comm * comm, int handler_id, - const void *am_hdr, size_t am_hdr_sz, void *shm_context); -typedef int (*MPIDI_SHM_am_send_t) (int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, - size_t am_hdr_sz, const void *data, MPI_Count count, - MPI_Datatype datatype, MPIR_Request * sreq, void *shm_context); +typedef int (*MPIDI_SHM_am_send_hdr_t) (int rank, MPIR_Comm * comm, int handler_id, + const void *am_hdr, size_t am_hdr_sz, void *shm_context); +typedef int (*MPIDI_SHM_am_isend_t) (int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, + size_t am_hdr_sz, const void *data, MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq, void *shm_context); typedef int (*MPIDI_SHM_inject_am_t) (int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, size_t am_hdr_sz, const void *data, MPI_Count count, MPI_Datatype datatype, void *shm_context); -typedef int (*MPIDI_SHM_am_sendv_t) (int rank, MPIR_Comm * comm, int handler_id, - struct iovec * am_hdrs, size_t iov_len, const void *data, - MPI_Count count, MPI_Datatype datatype, MPIR_Request * sreq, - void *shm_context); +typedef int (*MPIDI_SHM_am_isendv_t) (int rank, MPIR_Comm * comm, int handler_id, + struct iovec * am_hdrs, size_t iov_len, const void *data, + MPI_Count count, MPI_Datatype datatype, MPIR_Request * sreq, + void *shm_context); typedef int (*MPIDI_SHM_inject_amv_t) (int rank, MPIR_Comm * comm, int handler_id, struct iovec * am_hdrs, size_t iov_len, const void *data, MPI_Count count, MPI_Datatype datatype, void *shm_context); -typedef int (*MPIDI_SHM_am_inject_hdr_reply_t) (MPIR_Context_id_t context_id, int src_rank, - int handler_id, const void *am_hdr, - size_t am_hdr_sz); -typedef int (*MPIDI_SHM_am_send_reply_t) (MPIR_Context_id_t context_id, int src_rank, - int handler_id, const void *am_hdr, size_t am_hdr_sz, - const void *data, MPI_Count count, MPI_Datatype datatype, - MPIR_Request * sreq); +typedef int (*MPIDI_SHM_am_send_hdr_reply_t) (MPIR_Context_id_t context_id, int src_rank, + int handler_id, const void *am_hdr, size_t am_hdr_sz); +typedef int (*MPIDI_SHM_am_isend_reply_t) (MPIR_Context_id_t context_id, int src_rank, + int handler_id, const void *am_hdr, size_t am_hdr_sz, + const void *data, MPI_Count count, MPI_Datatype datatype, + MPIR_Request * sreq); typedef int (*MPIDI_SHM_inject_am_reply_t) (MPIR_Context_id_t context_id, int src_rank, int handler_id, const void *am_hdr, size_t am_hdr_sz, const void *data, MPI_Count count, @@ -372,13 +371,13 @@ typedef struct MPIDI_SHM_funcs { MPIDI_SHM_open_port_t open_port; MPIDI_SHM_close_port_t close_port; MPIDI_SHM_comm_accept_t comm_accept; - MPIDI_SHM_am_inject_hdr_t am_inject_hdr; - MPIDI_SHM_am_send_t am_send; + MPIDI_SHM_am_send_hdr_t am_send_hdr; + MPIDI_SHM_am_isend_t am_isend; MPIDI_SHM_inject_am_t inject_am; - MPIDI_SHM_am_sendv_t am_sendv; + MPIDI_SHM_am_isendv_t am_isendv; MPIDI_SHM_inject_amv_t inject_amv; - MPIDI_SHM_am_inject_hdr_reply_t am_inject_hdr_reply; - MPIDI_SHM_am_send_reply_t am_send_reply; + MPIDI_SHM_am_send_hdr_reply_t am_send_hdr_reply; + MPIDI_SHM_am_isend_reply_t am_isend_reply; MPIDI_SHM_inject_am_reply_t inject_am_reply; MPIDI_SHM_inject_amv_reply_t inject_amv_reply; MPIDI_SHM_am_hdr_max_sz_t am_hdr_max_sz; @@ -533,41 +532,40 @@ MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_accept(const char *port_name, MPIR_I int root, MPIR_Comm * comm, MPIR_Comm ** newcomm_ptr) MPL_STATIC_INLINE_SUFFIX; -MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_am_inject_hdr(int rank, MPIR_Comm * comm, - int handler_id, const void *am_hdr, - size_t am_hdr_sz, - void *shm_context) MPL_STATIC_INLINE_SUFFIX; -MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_am_send(int rank, MPIR_Comm * comm, int handler_id, - const void *am_hdr, size_t am_hdr_sz, - const void *data, MPI_Count count, - MPI_Datatype datatype, MPIR_Request * sreq, - void *shm_context) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_am_send_hdr(int rank, MPIR_Comm * comm, + int handler_id, const void *am_hdr, + size_t am_hdr_sz, + void *shm_context) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_am_isend(int rank, MPIR_Comm * comm, int handler_id, + const void *am_hdr, size_t am_hdr_sz, + const void *data, MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq, + void *shm_context) MPL_STATIC_INLINE_SUFFIX; MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_am(int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, size_t am_hdr_sz, const void *data, MPI_Count count, MPI_Datatype datatype, void *shm_context) MPL_STATIC_INLINE_SUFFIX; -MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_am_sendv(int rank, MPIR_Comm * comm, int handler_id, - struct iovec *am_hdrs, size_t iov_len, - const void *data, MPI_Count count, - MPI_Datatype datatype, MPIR_Request * sreq, - void *shm_context) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_am_isendv(int rank, MPIR_Comm * comm, int handler_id, + struct iovec *am_hdrs, size_t iov_len, + const void *data, MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq, + void *shm_context) MPL_STATIC_INLINE_SUFFIX; MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_amv(int rank, MPIR_Comm * comm, int handler_id, struct iovec *am_hdrs, size_t iov_len, const void *data, MPI_Count count, MPI_Datatype datatype, void *shm_context) MPL_STATIC_INLINE_SUFFIX; -MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_am_inject_hdr_reply(MPIR_Context_id_t context_id, - int src_rank, int handler_id, - const void *am_hdr, - size_t am_hdr_sz) - MPL_STATIC_INLINE_SUFFIX; -MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_am_send_reply(MPIR_Context_id_t context_id, - int src_rank, int handler_id, - const void *am_hdr, size_t am_hdr_sz, - const void *data, MPI_Count count, - MPI_Datatype datatype, - MPIR_Request * sreq) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_am_send_hdr_reply(MPIR_Context_id_t context_id, + int src_rank, int handler_id, + const void *am_hdr, + size_t am_hdr_sz) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_am_isend_reply(MPIR_Context_id_t context_id, + int src_rank, int handler_id, + const void *am_hdr, size_t am_hdr_sz, + const void *data, MPI_Count count, + MPI_Datatype datatype, + MPIR_Request * sreq) MPL_STATIC_INLINE_SUFFIX; MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_am_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, const void *am_hdr, size_t am_hdr_sz, diff --git a/src/mpid/ch4/shm/include/shm_impl.h b/src/mpid/ch4/shm/include/shm_impl.h index dd7a495..b563ae2 100644 --- a/src/mpid/ch4/shm/include/shm_impl.h +++ b/src/mpid/ch4/shm/include/shm_impl.h @@ -68,21 +68,21 @@ MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_accept(const char *port_name, MPIR_I return MPIDI_SHM_func->comm_accept(port_name, info, root, comm, newcomm_ptr); }; -MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_am_inject_hdr(int rank, MPIR_Comm * comm, - int handler_id, const void *am_hdr, - size_t am_hdr_sz, void *shm_context) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_am_send_hdr(int rank, MPIR_Comm * comm, + int handler_id, const void *am_hdr, + size_t am_hdr_sz, void *shm_context) { - return MPIDI_SHM_func->am_inject_hdr(rank, comm, handler_id, am_hdr, am_hdr_sz, shm_context); + return MPIDI_SHM_func->am_send_hdr(rank, comm, handler_id, am_hdr, am_hdr_sz, shm_context); }; -MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_am_send(int rank, MPIR_Comm * comm, int handler_id, - const void *am_hdr, size_t am_hdr_sz, - const void *data, MPI_Count count, - MPI_Datatype datatype, MPIR_Request * sreq, - void *shm_context) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_am_isend(int rank, MPIR_Comm * comm, int handler_id, + const void *am_hdr, size_t am_hdr_sz, + const void *data, MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq, + void *shm_context) { - return MPIDI_SHM_func->am_send(rank, comm, handler_id, am_hdr, am_hdr_sz, data, count, datatype, - sreq, shm_context); + return MPIDI_SHM_func->am_isend(rank, comm, handler_id, am_hdr, am_hdr_sz, data, count, + datatype, sreq, shm_context); }; MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_am(int rank, MPIR_Comm * comm, int handler_id, @@ -94,14 +94,14 @@ MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_am(int rank, MPIR_Comm * comm, int datatype, shm_context); }; -MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_am_sendv(int rank, MPIR_Comm * comm, int handler_id, - struct iovec *am_hdrs, size_t iov_len, - const void *data, MPI_Count count, - MPI_Datatype datatype, MPIR_Request * sreq, - void *shm_context) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_am_isendv(int rank, MPIR_Comm * comm, int handler_id, + struct iovec *am_hdrs, size_t iov_len, + const void *data, MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq, + void *shm_context) { - return MPIDI_SHM_func->am_sendv(rank, comm, handler_id, am_hdrs, iov_len, data, count, datatype, - sreq, shm_context); + return MPIDI_SHM_func->am_isendv(rank, comm, handler_id, am_hdrs, iov_len, data, count, + datatype, sreq, shm_context); }; MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_amv(int rank, MPIR_Comm * comm, int handler_id, @@ -113,21 +113,21 @@ MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_amv(int rank, MPIR_Comm * comm, in datatype, shm_context); }; -MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_am_inject_hdr_reply(MPIR_Context_id_t context_id, - int src_rank, int handler_id, - const void *am_hdr, size_t am_hdr_sz) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_am_send_hdr_reply(MPIR_Context_id_t context_id, + int src_rank, int handler_id, + const void *am_hdr, size_t am_hdr_sz) { - return MPIDI_SHM_func->am_inject_hdr_reply(context_id, src_rank, handler_id, am_hdr, am_hdr_sz); + return MPIDI_SHM_func->am_send_hdr_reply(context_id, src_rank, handler_id, am_hdr, am_hdr_sz); }; -MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_am_send_reply(MPIR_Context_id_t context_id, - int src_rank, int handler_id, - const void *am_hdr, size_t am_hdr_sz, - const void *data, MPI_Count count, - MPI_Datatype datatype, MPIR_Request * sreq) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_am_isend_reply(MPIR_Context_id_t context_id, + int src_rank, int handler_id, + const void *am_hdr, size_t am_hdr_sz, + const void *data, MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq) { - return MPIDI_SHM_func->am_send_reply(context_id, src_rank, handler_id, am_hdr, am_hdr_sz, data, - count, datatype, sreq); + return MPIDI_SHM_func->am_isend_reply(context_id, src_rank, handler_id, am_hdr, am_hdr_sz, data, + count, datatype, sreq); }; MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_am_reply(MPIR_Context_id_t context_id, diff --git a/src/mpid/ch4/shm/posix/func_table.c b/src/mpid/ch4/shm/posix/func_table.c index f6d884f..3319e3a 100644 --- a/src/mpid/ch4/shm/posix/func_table.c +++ b/src/mpid/ch4/shm/posix/func_table.c @@ -22,13 +22,13 @@ MPIDI_SHM_funcs_t MPIDI_SHM_posix_funcs = { MPIDI_SHM_open_port, MPIDI_SHM_close_port, MPIDI_SHM_comm_accept, - MPIDI_SHM_am_inject_hdr, - MPIDI_SHM_am_send, + MPIDI_SHM_am_send_hdr, + MPIDI_SHM_am_isend, MPIDI_SHM_inject_am, - MPIDI_SHM_am_sendv, + MPIDI_SHM_am_isendv, MPIDI_SHM_inject_amv, - MPIDI_SHM_am_inject_hdr_reply, - MPIDI_SHM_am_send_reply, + MPIDI_SHM_am_send_hdr_reply, + MPIDI_SHM_am_isend_reply, MPIDI_SHM_inject_am_reply, MPIDI_SHM_inject_amv_reply, MPIDI_SHM_am_hdr_max_sz, diff --git a/src/mpid/ch4/shm/posix/posix_am.h b/src/mpid/ch4/shm/posix/posix_am.h index 6b41b54..63a44c0 100644 --- a/src/mpid/ch4/shm/posix/posix_am.h +++ b/src/mpid/ch4/shm/posix/posix_am.h @@ -20,24 +20,11 @@ static inline int MPIDI_SHM_am_reg_handler(int handler_id, return MPI_SUCCESS; } -static inline int MPIDI_SHM_am_send(int rank, - MPIR_Comm * comm, - int handler_id, - const void *am_hdr, - size_t am_hdr_sz, - const void *data, - MPI_Count count, - MPI_Datatype datatype, MPIR_Request * sreq, void *shm_context) -{ - MPIR_Assert(0); - return MPI_SUCCESS; -} - -static inline int MPIDI_SHM_am_sendv(int rank, +static inline int MPIDI_SHM_am_isend(int rank, MPIR_Comm * comm, int handler_id, - struct iovec *am_hdr, - size_t iov_len, + const void *am_hdr, + size_t am_hdr_sz, const void *data, MPI_Count count, MPI_Datatype datatype, MPIR_Request * sreq, void *shm_context) @@ -46,13 +33,26 @@ static inline int MPIDI_SHM_am_sendv(int rank, return MPI_SUCCESS; } -static inline int MPIDI_SHM_am_send_reply(MPIR_Context_id_t context_id, int src_rank, - int handler_id, - const void *am_hdr, - size_t am_hdr_sz, - const void *data, - MPI_Count count, - MPI_Datatype datatype, MPIR_Request * sreq) +static inline int MPIDI_SHM_am_isendv(int rank, + MPIR_Comm * comm, + int handler_id, + struct iovec *am_hdr, + size_t iov_len, + const void *data, + MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq, void *shm_context) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_am_isend_reply(MPIR_Context_id_t context_id, int src_rank, + int handler_id, + const void *am_hdr, + size_t am_hdr_sz, + const void *data, + MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq) { MPIR_Assert(0); return MPI_SUCCESS; @@ -64,10 +64,10 @@ static inline size_t MPIDI_SHM_am_hdr_max_sz(void) return 0; } -static inline int MPIDI_SHM_am_inject_hdr(int rank, - MPIR_Comm * comm, - int handler_id, - const void *am_hdr, size_t am_hdr_sz, void *shm_context) +static inline int MPIDI_SHM_am_send_hdr(int rank, + MPIR_Comm * comm, + int handler_id, + const void *am_hdr, size_t am_hdr_sz, void *shm_context) { MPIR_Assert(0); return MPI_SUCCESS; @@ -97,9 +97,8 @@ static inline int MPIDI_SHM_inject_amv(int rank, return MPI_SUCCESS; } -static inline int MPIDI_SHM_am_inject_hdr_reply(MPIR_Context_id_t context_id, int src_rank, - int handler_id, - const void *am_hdr, size_t am_hdr_sz) +static inline int MPIDI_SHM_am_send_hdr_reply(MPIR_Context_id_t context_id, int src_rank, + int handler_id, const void *am_hdr, size_t am_hdr_sz) { MPIR_Assert(0); return MPI_SUCCESS; diff --git a/src/mpid/ch4/shm/stubshm/func_table.c b/src/mpid/ch4/shm/stubshm/func_table.c index 3b8253c..44a89c5 100644 --- a/src/mpid/ch4/shm/stubshm/func_table.c +++ b/src/mpid/ch4/shm/stubshm/func_table.c @@ -22,13 +22,13 @@ MPIDI_SHM_funcs_t MPIDI_SHM_stubshm_funcs = { MPIDI_SHM_open_port, MPIDI_SHM_close_port, MPIDI_SHM_comm_accept, - MPIDI_SHM_am_inject_hdr, - MPIDI_SHM_am_send, + MPIDI_SHM_am_send_hdr, + MPIDI_SHM_am_isend, MPIDI_SHM_inject_am, - MPIDI_SHM_am_sendv, + MPIDI_SHM_am_isendv, MPIDI_SHM_inject_amv, - MPIDI_SHM_am_inject_hdr_reply, - MPIDI_SHM_am_send_reply, + MPIDI_SHM_am_send_hdr_reply, + MPIDI_SHM_am_isend_reply, MPIDI_SHM_inject_am_reply, MPIDI_SHM_inject_amv_reply, MPIDI_SHM_am_hdr_max_sz, diff --git a/src/mpid/ch4/shm/stubshm/stubshm_am.h b/src/mpid/ch4/shm/stubshm/stubshm_am.h index 2e29e5d..0191b50 100644 --- a/src/mpid/ch4/shm/stubshm/stubshm_am.h +++ b/src/mpid/ch4/shm/stubshm/stubshm_am.h @@ -20,24 +20,11 @@ static inline int MPIDI_SHM_am_reg_handler(int handler_id, return MPI_SUCCESS; } -static inline int MPIDI_SHM_am_send(int rank, - MPIR_Comm * comm, - int handler_id, - const void *am_hdr, - size_t am_hdr_sz, - const void *data, - MPI_Count count, - MPI_Datatype datatype, MPIR_Request * sreq, void *shm_context) -{ - MPIR_Assert(0); - return MPI_SUCCESS; -} - -static inline int MPIDI_SHM_am_sendv(int rank, +static inline int MPIDI_SHM_am_isend(int rank, MPIR_Comm * comm, int handler_id, - struct iovec *am_hdr, - size_t iov_len, + const void *am_hdr, + size_t am_hdr_sz, const void *data, MPI_Count count, MPI_Datatype datatype, MPIR_Request * sreq, void *shm_context) @@ -46,13 +33,26 @@ static inline int MPIDI_SHM_am_sendv(int rank, return MPI_SUCCESS; } -static inline int MPIDI_SHM_am_send_reply(MPIR_Context_id_t context_id, int src_rank, - int handler_id, - const void *am_hdr, - size_t am_hdr_sz, - const void *data, - MPI_Count count, - MPI_Datatype datatype, MPIR_Request * sreq) +static inline int MPIDI_SHM_am_isendv(int rank, + MPIR_Comm * comm, + int handler_id, + struct iovec *am_hdr, + size_t iov_len, + const void *data, + MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq, void *shm_context) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_am_isend_reply(MPIR_Context_id_t context_id, int src_rank, + int handler_id, + const void *am_hdr, + size_t am_hdr_sz, + const void *data, + MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq) { MPIR_Assert(0); return MPI_SUCCESS; @@ -64,10 +64,10 @@ static inline size_t MPIDI_SHM_am_hdr_max_sz(void) return 0; } -static inline int MPIDI_SHM_am_inject_hdr(int rank, - MPIR_Comm * comm, - int handler_id, - const void *am_hdr, size_t am_hdr_sz, void *shm_context) +static inline int MPIDI_SHM_am_send_hdr(int rank, + MPIR_Comm * comm, + int handler_id, + const void *am_hdr, size_t am_hdr_sz, void *shm_context) { MPIR_Assert(0); return MPI_SUCCESS; @@ -97,9 +97,8 @@ static inline int MPIDI_SHM_inject_amv(int rank, return MPI_SUCCESS; } -static inline int MPIDI_SHM_am_inject_hdr_reply(MPIR_Context_id_t context_id, int src_rank, - int handler_id, - const void *am_hdr, size_t am_hdr_sz) +static inline int MPIDI_SHM_am_send_hdr_reply(MPIR_Context_id_t context_id, int src_rank, + int handler_id, const void *am_hdr, size_t am_hdr_sz) { MPIR_Assert(0); return MPI_SUCCESS; diff --git a/src/mpid/ch4/src/ch4r_callbacks.h b/src/mpid/ch4/src/ch4r_callbacks.h index 3ab5c1d..0f311d7 100644 --- a/src/mpid/ch4/src/ch4r_callbacks.h +++ b/src/mpid/ch4/src/ch4r_callbacks.h @@ -152,13 +152,13 @@ static inline int MPIDI_CH4U_get_cmpl_handler(MPIR_Request * req) win = MPIDI_CH4U_REQUEST(req, req->greq.win_ptr); context_id = MPIDI_CH4U_win_to_context(win); if (MPIDI_CH4U_REQUEST(req, req->greq.n_iov) == 0) { - mpi_errno = MPIDI_NM_am_send_reply(context_id, - MPIDI_CH4U_REQUEST(req, src_rank), - MPIDI_CH4U_GET_ACK, - &get_ack, sizeof(get_ack), - (void *) MPIDI_CH4U_REQUEST(req, req->greq.addr), - MPIDI_CH4U_REQUEST(req, req->greq.count), - MPIDI_CH4U_REQUEST(req, req->greq.datatype), req); + mpi_errno = MPIDI_NM_am_isend_reply(context_id, + MPIDI_CH4U_REQUEST(req, src_rank), + MPIDI_CH4U_GET_ACK, + &get_ack, sizeof(get_ack), + (void *) MPIDI_CH4U_REQUEST(req, req->greq.addr), + MPIDI_CH4U_REQUEST(req, req->greq.count), + MPIDI_CH4U_REQUEST(req, req->greq.datatype), req); MPIDI_CH4I_am_request_complete(req); if (mpi_errno) MPIR_ERR_POP(mpi_errno); @@ -186,10 +186,10 @@ static inline int MPIDI_CH4U_get_cmpl_handler(MPIR_Request * req) MPL_free(MPIDI_CH4U_REQUEST(req, req->greq.dt_iov)); MPIDI_CH4U_REQUEST(req, req->greq.dt_iov) = (void *) p_data; - mpi_errno = MPIDI_NM_am_send_reply(context_id, - MPIDI_CH4U_REQUEST(req, src_rank), - MPIDI_CH4U_GET_ACK, - &get_ack, sizeof(get_ack), p_data, data_sz, MPI_BYTE, req); + mpi_errno = MPIDI_NM_am_isend_reply(context_id, + MPIDI_CH4U_REQUEST(req, src_rank), + MPIDI_CH4U_GET_ACK, + &get_ack, sizeof(get_ack), p_data, data_sz, MPI_BYTE, req); MPIDI_CH4I_am_request_complete(req); if (mpi_errno) MPIR_ERR_POP(mpi_errno); @@ -414,10 +414,10 @@ static inline int MPIDI_CH4U_reply_ssend(MPIR_Request * rreq) MPIR_cc_incr(rreq->cc_ptr, &c); ack_msg.sreq_ptr = MPIDI_CH4U_REQUEST(rreq, req->rreq.peer_req_ptr); - mpi_errno = MPIDI_NM_am_send_reply(MPIDI_CH4U_get_context(MPIDI_CH4U_REQUEST(rreq, tag)), - MPIDI_CH4U_REQUEST(rreq, src_rank), - MPIDI_CH4U_SSEND_ACK, &ack_msg, sizeof(ack_msg), - NULL, 0, MPI_DATATYPE_NULL, rreq); + mpi_errno = MPIDI_NM_am_isend_reply(MPIDI_CH4U_get_context(MPIDI_CH4U_REQUEST(rreq, tag)), + MPIDI_CH4U_REQUEST(rreq, src_rank), + MPIDI_CH4U_SSEND_ACK, &ack_msg, sizeof(ack_msg), + NULL, 0, MPI_DATATYPE_NULL, rreq); if (mpi_errno) MPIR_ERR_POP(mpi_errno); fn_exit: @@ -440,10 +440,10 @@ static inline int MPIDI_CH4U_ack_put(MPIR_Request * rreq) ack_msg.preq_ptr = MPIDI_CH4U_REQUEST(rreq, req->preq.preq_ptr); mpi_errno = - MPIDI_NM_am_inject_hdr_reply(MPIDI_CH4U_win_to_context - (MPIDI_CH4U_REQUEST(rreq, req->preq.win_ptr)), - MPIDI_CH4U_REQUEST(rreq, src_rank), MPIDI_CH4U_PUT_ACK, - &ack_msg, sizeof(ack_msg)); + MPIDI_NM_am_send_hdr_reply(MPIDI_CH4U_win_to_context + (MPIDI_CH4U_REQUEST(rreq, req->preq.win_ptr)), + MPIDI_CH4U_REQUEST(rreq, src_rank), MPIDI_CH4U_PUT_ACK, + &ack_msg, sizeof(ack_msg)); if (mpi_errno) MPIR_ERR_POP(mpi_errno); fn_exit: @@ -474,12 +474,13 @@ static inline int MPIDI_CH4U_ack_cswap(MPIR_Request * rreq) ack_msg.req_ptr = MPIDI_CH4U_REQUEST(rreq, req->creq.creq_ptr); mpi_errno = - MPIDI_NM_am_send_reply(MPIDI_CH4U_win_to_context - (MPIDI_CH4U_REQUEST(rreq, req->creq.win_ptr)), - MPIDI_CH4U_REQUEST(rreq, src_rank), MPIDI_CH4U_CSWAP_ACK, &ack_msg, - sizeof(ack_msg), result_addr, 1, MPIDI_CH4U_REQUEST(rreq, - req->creq. - datatype), rreq); + MPIDI_NM_am_isend_reply(MPIDI_CH4U_win_to_context + (MPIDI_CH4U_REQUEST(rreq, req->creq.win_ptr)), + MPIDI_CH4U_REQUEST(rreq, src_rank), MPIDI_CH4U_CSWAP_ACK, &ack_msg, + sizeof(ack_msg), result_addr, 1, MPIDI_CH4U_REQUEST(rreq, + req->creq. + datatype), + rreq); if (mpi_errno) MPIR_ERR_POP(mpi_errno); fn_exit: @@ -504,10 +505,10 @@ static inline int MPIDI_CH4U_ack_acc(MPIR_Request * rreq) ack_msg.req_ptr = MPIDI_CH4U_REQUEST(rreq, req->areq.req_ptr); mpi_errno = - MPIDI_NM_am_inject_hdr_reply(MPIDI_CH4U_win_to_context - (MPIDI_CH4U_REQUEST(rreq, req->areq.win_ptr)), - MPIDI_CH4U_REQUEST(rreq, src_rank), MPIDI_CH4U_ACC_ACK, - &ack_msg, sizeof(ack_msg)); + MPIDI_NM_am_send_hdr_reply(MPIDI_CH4U_win_to_context + (MPIDI_CH4U_REQUEST(rreq, req->areq.win_ptr)), + MPIDI_CH4U_REQUEST(rreq, src_rank), MPIDI_CH4U_ACC_ACK, + &ack_msg, sizeof(ack_msg)); win = MPIDI_CH4U_REQUEST(rreq, req->areq.win_ptr); /* MPIDI_CS_ENTER(); */ @@ -539,11 +540,11 @@ static inline int MPIDI_CH4U_ack_get_acc(MPIR_Request * rreq) ack_msg.req_ptr = MPIDI_CH4U_REQUEST(rreq, req->areq.req_ptr); mpi_errno = - MPIDI_NM_am_send_reply(MPIDI_CH4U_win_to_context - (MPIDI_CH4U_REQUEST(rreq, req->areq.win_ptr)), - MPIDI_CH4U_REQUEST(rreq, src_rank), MPIDI_CH4U_GET_ACC_ACK, &ack_msg, - sizeof(ack_msg), MPIDI_CH4U_REQUEST(rreq, req->areq.data), - MPIDI_CH4U_REQUEST(rreq, req->areq.data_sz), MPI_BYTE, rreq); + MPIDI_NM_am_isend_reply(MPIDI_CH4U_win_to_context + (MPIDI_CH4U_REQUEST(rreq, req->areq.win_ptr)), + MPIDI_CH4U_REQUEST(rreq, src_rank), MPIDI_CH4U_GET_ACC_ACK, + &ack_msg, sizeof(ack_msg), MPIDI_CH4U_REQUEST(rreq, req->areq.data), + MPIDI_CH4U_REQUEST(rreq, req->areq.data_sz), MPI_BYTE, rreq); if (mpi_errno) MPIR_ERR_POP(mpi_errno); fn_exit: @@ -900,10 +901,10 @@ static inline int MPIDI_CH4U_put_iov_cmpl_handler(MPIR_Request * rreq) ack_msg.target_preq_ptr = (uint64_t) rreq; mpi_errno = - MPIDI_NM_am_inject_hdr_reply(MPIDI_CH4U_win_to_context - (MPIDI_CH4U_REQUEST(rreq, req->preq.win_ptr)), - MPIDI_CH4U_REQUEST(rreq, src_rank), MPIDI_CH4U_PUT_IOV_ACK, - &ack_msg, sizeof(ack_msg)); + MPIDI_NM_am_send_hdr_reply(MPIDI_CH4U_win_to_context + (MPIDI_CH4U_REQUEST(rreq, req->preq.win_ptr)), + MPIDI_CH4U_REQUEST(rreq, src_rank), MPIDI_CH4U_PUT_IOV_ACK, + &ack_msg, sizeof(ack_msg)); if (mpi_errno) MPIR_ERR_POP(mpi_errno); @@ -930,10 +931,10 @@ static inline int MPIDI_CH4U_acc_iov_cmpl_handler(MPIR_Request * rreq) ack_msg.target_preq_ptr = (uint64_t) rreq; mpi_errno = - MPIDI_NM_am_inject_hdr_reply(MPIDI_CH4U_win_to_context - (MPIDI_CH4U_REQUEST(rreq, req->areq.win_ptr)), - MPIDI_CH4U_REQUEST(rreq, src_rank), MPIDI_CH4U_ACC_IOV_ACK, - &ack_msg, sizeof(ack_msg)); + MPIDI_NM_am_send_hdr_reply(MPIDI_CH4U_win_to_context + (MPIDI_CH4U_REQUEST(rreq, req->areq.win_ptr)), + MPIDI_CH4U_REQUEST(rreq, src_rank), MPIDI_CH4U_ACC_IOV_ACK, + &ack_msg, sizeof(ack_msg)); if (mpi_errno) MPIR_ERR_POP(mpi_errno); @@ -1652,14 +1653,14 @@ static inline int MPIDI_CH4U_send_long_ack_target_handler(void *am_hdr, /* Start the main data transfer */ send_hdr.rreq_ptr = msg_hdr->rreq_ptr; mpi_errno = - MPIDI_NM_am_send_reply(MPIDI_CH4U_get_context(MPIDI_CH4U_REQUEST(sreq, req->lreq).msg_tag), - MPIDI_CH4U_REQUEST(sreq, src_rank), MPIDI_CH4U_SEND_LONG_LMT, - &send_hdr, sizeof(send_hdr), MPIDI_CH4U_REQUEST(sreq, - req->lreq).src_buf, - MPIDI_CH4U_REQUEST(sreq, req->lreq).count, MPIDI_CH4U_REQUEST(sreq, - req-> - lreq). - datatype, sreq); + MPIDI_NM_am_isend_reply(MPIDI_CH4U_get_context(MPIDI_CH4U_REQUEST(sreq, req->lreq).msg_tag), + MPIDI_CH4U_REQUEST(sreq, src_rank), MPIDI_CH4U_SEND_LONG_LMT, + &send_hdr, sizeof(send_hdr), MPIDI_CH4U_REQUEST(sreq, + req->lreq).src_buf, + MPIDI_CH4U_REQUEST(sreq, req->lreq).count, MPIDI_CH4U_REQUEST(sreq, + req-> + lreq). + datatype, sreq); if (mpi_errno) MPIR_ERR_POP(mpi_errno); @@ -1894,9 +1895,8 @@ static inline int MPIDI_CH4U_win_lock_advance(MPIR_Win * win) else MPIR_ERR_SETANDJUMP(mpi_errno, MPI_ERR_OTHER, "**rmasync"); - mpi_errno = MPIDI_NM_am_inject_hdr_reply(MPIDI_CH4U_win_to_context(win), - lock->rank, - MPIDI_CH4U_WIN_CTRL, &msg, sizeof(msg)); + mpi_errno = MPIDI_NM_am_send_hdr_reply(MPIDI_CH4U_win_to_context(win), + lock->rank, MPIDI_CH4U_WIN_CTRL, &msg, sizeof(msg)); if (mpi_errno) MPIR_ERR_POP(mpi_errno); MPL_free(lock); @@ -1987,9 +1987,9 @@ static inline void MPIDI_CH4U_win_unlock_proc(const MPIDI_CH4U_win_cntrl_msg_t * msg.origin_rank = win->comm_ptr->rank; msg.type = MPIDI_CH4U_WIN_UNLOCK_ACK; - mpi_errno = MPIDI_NM_am_inject_hdr_reply(MPIDI_CH4U_win_to_context(win), - info->origin_rank, - MPIDI_CH4U_WIN_CTRL, &msg, sizeof(msg)); + mpi_errno = MPIDI_NM_am_send_hdr_reply(MPIDI_CH4U_win_to_context(win), + info->origin_rank, + MPIDI_CH4U_WIN_CTRL, &msg, sizeof(msg)); if (mpi_errno) MPIR_ERR_POP(mpi_errno); fn_exit: @@ -2307,14 +2307,14 @@ static inline int MPIDI_CH4U_put_iov_ack_target_handler(void *am_hdr, origin_req = (MPIR_Request *) msg_hdr->origin_preq_ptr; dat_msg.preq_ptr = msg_hdr->target_preq_ptr; win = MPIDI_CH4U_REQUEST(origin_req, req->preq.win_ptr); - mpi_errno = MPIDI_NM_am_send_reply(MPIDI_CH4U_win_to_context(win), - MPIDI_CH4U_REQUEST(origin_req, src_rank), - MPIDI_CH4U_PUT_DAT_REQ, - &dat_msg, sizeof(dat_msg), - MPIDI_CH4U_REQUEST(origin_req, req->preq.origin_addr), - MPIDI_CH4U_REQUEST(origin_req, req->preq.origin_count), - MPIDI_CH4U_REQUEST(origin_req, req->preq.origin_datatype), - rreq); + mpi_errno = MPIDI_NM_am_isend_reply(MPIDI_CH4U_win_to_context(win), + MPIDI_CH4U_REQUEST(origin_req, src_rank), + MPIDI_CH4U_PUT_DAT_REQ, + &dat_msg, sizeof(dat_msg), + MPIDI_CH4U_REQUEST(origin_req, req->preq.origin_addr), + MPIDI_CH4U_REQUEST(origin_req, req->preq.origin_count), + MPIDI_CH4U_REQUEST(origin_req, req->preq.origin_datatype), + rreq); if (mpi_errno) MPIR_ERR_POP(mpi_errno); dtype_release_if_not_builtin(MPIDI_CH4U_REQUEST(origin_req, req->preq.origin_datatype)); @@ -2356,14 +2356,14 @@ static inline int MPIDI_CH4U_acc_iov_ack_target_handler(void *am_hdr, origin_req = (MPIR_Request *) msg_hdr->origin_preq_ptr; dat_msg.preq_ptr = msg_hdr->target_preq_ptr; win = MPIDI_CH4U_REQUEST(origin_req, req->areq.win_ptr); - mpi_errno = MPIDI_NM_am_send_reply(MPIDI_CH4U_win_to_context(win), - MPIDI_CH4U_REQUEST(origin_req, src_rank), - MPIDI_CH4U_ACC_DAT_REQ, - &dat_msg, sizeof(dat_msg), - MPIDI_CH4U_REQUEST(origin_req, req->areq.origin_addr), - MPIDI_CH4U_REQUEST(origin_req, req->areq.origin_count), - MPIDI_CH4U_REQUEST(origin_req, req->areq.origin_datatype), - rreq); + mpi_errno = MPIDI_NM_am_isend_reply(MPIDI_CH4U_win_to_context(win), + MPIDI_CH4U_REQUEST(origin_req, src_rank), + MPIDI_CH4U_ACC_DAT_REQ, + &dat_msg, sizeof(dat_msg), + MPIDI_CH4U_REQUEST(origin_req, req->areq.origin_addr), + MPIDI_CH4U_REQUEST(origin_req, req->areq.origin_count), + MPIDI_CH4U_REQUEST(origin_req, req->areq.origin_datatype), + rreq); if (mpi_errno) MPIR_ERR_POP(mpi_errno); dtype_release_if_not_builtin(MPIDI_CH4U_REQUEST(origin_req, req->areq.origin_datatype)); diff --git a/src/mpid/ch4/src/ch4r_rma.h b/src/mpid/ch4/src/ch4r_rma.h index adaaa0c..637ba56 100644 --- a/src/mpid/ch4/src/ch4r_rma.h +++ b/src/mpid/ch4/src/ch4r_rma.h @@ -81,9 +81,9 @@ static inline int MPIDI_CH4I_do_put(const void *origin_addr, am_hdr.n_iov = 0; MPIDI_CH4U_REQUEST(sreq, req->preq.dt_iov) = NULL; - mpi_errno = MPIDI_NM_am_send(target_rank, win->comm_ptr, MPIDI_CH4U_PUT_REQ, - &am_hdr, sizeof(am_hdr), origin_addr, - origin_count, origin_datatype, sreq, NULL); + mpi_errno = MPIDI_NM_am_isend(target_rank, win->comm_ptr, MPIDI_CH4U_PUT_REQ, + &am_hdr, sizeof(am_hdr), origin_addr, + origin_count, origin_datatype, sreq, NULL); if (mpi_errno) MPIR_ERR_POP(mpi_errno); goto fn_exit; @@ -114,9 +114,9 @@ static inline int MPIDI_CH4I_do_put(const void *origin_addr, MPIDI_CH4U_REQUEST(sreq, req->preq.dt_iov) = dt_iov; if ((am_iov[0].iov_len + am_iov[1].iov_len) <= MPIDI_NM_am_hdr_max_sz()) { - mpi_errno = MPIDI_NM_am_sendv(target_rank, win->comm_ptr, MPIDI_CH4U_PUT_REQ, - &am_iov[0], 2, origin_addr, origin_count, origin_datatype, - sreq, NULL); + mpi_errno = MPIDI_NM_am_isendv(target_rank, win->comm_ptr, MPIDI_CH4U_PUT_REQ, + &am_iov[0], 2, origin_addr, origin_count, origin_datatype, + sreq, NULL); } else { MPIDI_CH4U_REQUEST(sreq, req->preq.origin_addr) = (void *) origin_addr; @@ -125,9 +125,9 @@ static inline int MPIDI_CH4I_do_put(const void *origin_addr, MPIDI_CH4U_REQUEST(sreq, src_rank) = target_rank; dtype_add_ref_if_not_builtin(origin_datatype); - mpi_errno = MPIDI_NM_am_send(target_rank, win->comm_ptr, MPIDI_CH4U_PUT_IOV_REQ, - &am_hdr, sizeof(am_hdr), am_iov[1].iov_base, - am_iov[1].iov_len, MPI_BYTE, sreq, NULL); + mpi_errno = MPIDI_NM_am_isend(target_rank, win->comm_ptr, MPIDI_CH4U_PUT_IOV_REQ, + &am_hdr, sizeof(am_hdr), am_iov[1].iov_base, + am_iov[1].iov_len, MPI_BYTE, sreq, NULL); } if (mpi_errno) MPIR_ERR_POP(mpi_errno); @@ -210,9 +210,9 @@ static inline int MPIDI_CH4I_do_get(void *origin_addr, am_hdr.n_iov = 0; MPIDI_CH4U_REQUEST(sreq, req->greq.dt_iov) = NULL; - mpi_errno = MPIDI_NM_am_send(target_rank, win->comm_ptr, - MPIDI_CH4U_GET_REQ, &am_hdr, sizeof(am_hdr), - NULL, 0, MPI_DATATYPE_NULL, sreq, NULL); + mpi_errno = MPIDI_NM_am_isend(target_rank, win->comm_ptr, + MPIDI_CH4U_GET_REQ, &am_hdr, sizeof(am_hdr), + NULL, 0, MPI_DATATYPE_NULL, sreq, NULL); if (mpi_errno) MPIR_ERR_POP(mpi_errno); goto fn_exit; @@ -236,9 +236,9 @@ static inline int MPIDI_CH4I_do_get(void *origin_addr, MPL_free(segment_ptr); MPIDI_CH4U_REQUEST(sreq, req->greq.dt_iov) = dt_iov; - mpi_errno = MPIDI_NM_am_send(target_rank, win->comm_ptr, MPIDI_CH4U_GET_REQ, - &am_hdr, sizeof(am_hdr), dt_iov, - sizeof(struct iovec) * am_hdr.n_iov, MPI_BYTE, sreq, NULL); + mpi_errno = MPIDI_NM_am_isend(target_rank, win->comm_ptr, MPIDI_CH4U_GET_REQ, + &am_hdr, sizeof(am_hdr), dt_iov, + sizeof(struct iovec) * am_hdr.n_iov, MPI_BYTE, sreq, NULL); if (mpi_errno) MPIR_ERR_POP(mpi_errno); @@ -442,10 +442,10 @@ __CH4_INLINE__ int MPIDI_CH4I_do_accumulate(const void *origin_addr, am_hdr.n_iov = 0; MPIDI_CH4U_REQUEST(sreq, req->areq.dt_iov) = NULL; - mpi_errno = MPIDI_NM_am_send(target_rank, win->comm_ptr, MPIDI_CH4U_ACC_REQ, - &am_hdr, sizeof(am_hdr), origin_addr, - (op == MPI_NO_OP) ? 0 : origin_count, - origin_datatype, sreq, NULL); + mpi_errno = MPIDI_NM_am_isend(target_rank, win->comm_ptr, MPIDI_CH4U_ACC_REQ, + &am_hdr, sizeof(am_hdr), origin_addr, + (op == MPI_NO_OP) ? 0 : origin_count, + origin_datatype, sreq, NULL); if (mpi_errno) MPIR_ERR_POP(mpi_errno); goto fn_exit; @@ -480,10 +480,10 @@ __CH4_INLINE__ int MPIDI_CH4I_do_accumulate(const void *origin_addr, MPIDI_CH4U_REQUEST(sreq, req->areq.dt_iov) = dt_iov; if ((am_iov[0].iov_len + am_iov[1].iov_len) <= MPIDI_NM_am_hdr_max_sz()) { - mpi_errno = MPIDI_NM_am_sendv(target_rank, win->comm_ptr, MPIDI_CH4U_ACC_REQ, - &am_iov[0], 2, origin_addr, - (op == MPI_NO_OP) ? 0 : origin_count, - origin_datatype, sreq, NULL); + mpi_errno = MPIDI_NM_am_isendv(target_rank, win->comm_ptr, MPIDI_CH4U_ACC_REQ, + &am_iov[0], 2, origin_addr, + (op == MPI_NO_OP) ? 0 : origin_count, + origin_datatype, sreq, NULL); } else { MPIDI_CH4U_REQUEST(sreq, req->areq.origin_addr) = (void *) origin_addr; @@ -492,9 +492,9 @@ __CH4_INLINE__ int MPIDI_CH4I_do_accumulate(const void *origin_addr, MPIDI_CH4U_REQUEST(sreq, src_rank) = target_rank; dtype_add_ref_if_not_builtin(origin_datatype); - mpi_errno = MPIDI_NM_am_send(target_rank, win->comm_ptr, MPIDI_CH4U_ACC_IOV_REQ, - &am_hdr, sizeof(am_hdr), am_iov[1].iov_base, - am_iov[1].iov_len, MPI_BYTE, sreq, NULL); + mpi_errno = MPIDI_NM_am_isend(target_rank, win->comm_ptr, MPIDI_CH4U_ACC_IOV_REQ, + &am_hdr, sizeof(am_hdr), am_iov[1].iov_base, + am_iov[1].iov_len, MPI_BYTE, sreq, NULL); } if (mpi_errno) MPIR_ERR_POP(mpi_errno); @@ -722,8 +722,9 @@ __CH4_INLINE__ int MPIDI_CH4U_compare_and_swap(const void *origin_addr, OPA_incr_int(&MPIDI_CH4U_WIN(win, outstanding_ops)); /* MPIDI_CS_EXIT(); */ - mpi_errno = MPIDI_NM_am_send(target_rank, win->comm_ptr, MPIDI_CH4U_CSWAP_REQ, - &am_hdr, sizeof(am_hdr), (char *) p_data, 2, datatype, sreq, NULL); + mpi_errno = MPIDI_NM_am_isend(target_rank, win->comm_ptr, MPIDI_CH4U_CSWAP_REQ, + &am_hdr, sizeof(am_hdr), (char *) p_data, 2, datatype, sreq, + NULL); if (mpi_errno) MPIR_ERR_POP(mpi_errno); fn_exit: diff --git a/src/mpid/ch4/src/ch4r_send.h b/src/mpid/ch4/src/ch4r_send.h index 30f4f4e..dc1f3e7 100644 --- a/src/mpid/ch4/src/ch4r_send.h +++ b/src/mpid/ch4/src/ch4r_send.h @@ -49,15 +49,15 @@ static inline int MPIDI_CH4I_do_send(const void *buf, ssend_req.sreq_ptr = (uint64_t) sreq; MPIR_cc_incr(sreq->cc_ptr, &c); - mpi_errno = MPIDI_NM_am_send(rank, comm, MPIDI_CH4U_SSEND_REQ, - &ssend_req, sizeof(ssend_req), - buf, count, datatype, sreq, NULL); + mpi_errno = MPIDI_NM_am_isend(rank, comm, MPIDI_CH4U_SSEND_REQ, + &ssend_req, sizeof(ssend_req), + buf, count, datatype, sreq, NULL); if (mpi_errno) MPIR_ERR_POP(mpi_errno); } else { - mpi_errno = MPIDI_NM_am_send(rank, comm, MPIDI_CH4U_SEND, - &am_hdr, sizeof(am_hdr), buf, count, datatype, sreq, NULL); + mpi_errno = MPIDI_NM_am_isend(rank, comm, MPIDI_CH4U_SEND, + &am_hdr, sizeof(am_hdr), buf, count, datatype, sreq, NULL); if (mpi_errno) MPIR_ERR_POP(mpi_errno); } diff --git a/src/mpid/ch4/src/ch4r_win.h b/src/mpid/ch4/src/ch4r_win.h index 4bcd90d..ebf3299 100644 --- a/src/mpid/ch4/src/ch4r_win.h +++ b/src/mpid/ch4/src/ch4r_win.h @@ -284,8 +284,8 @@ static inline int MPIDI_CH4R_win_complete(MPIR_Win * win) for (index = 0; index < group->size; ++index) { peer = ranks_in_win_grp[index]; - mpi_errno = MPIDI_NM_am_inject_hdr(peer, win->comm_ptr, - MPIDI_CH4U_WIN_CTRL, &msg, sizeof(msg), NULL); + mpi_errno = MPIDI_NM_am_send_hdr(peer, win->comm_ptr, + MPIDI_CH4U_WIN_CTRL, &msg, sizeof(msg), NULL); if (mpi_errno != MPI_SUCCESS) MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, goto fn_fail, "**rmasync"); } @@ -338,8 +338,8 @@ static inline int MPIDI_CH4R_win_post(MPIR_Group * group, int assert, MPIR_Win * for (index = 0; index < group->size; ++index) { peer = ranks_in_win_grp[index]; - mpi_errno = MPIDI_NM_am_inject_hdr(peer, win->comm_ptr, - MPIDI_CH4U_WIN_CTRL, &msg, sizeof(msg), NULL); + mpi_errno = MPIDI_NM_am_send_hdr(peer, win->comm_ptr, + MPIDI_CH4U_WIN_CTRL, &msg, sizeof(msg), NULL); if (mpi_errno != MPI_SUCCESS) MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, goto fn_fail, "**rmasync"); } @@ -440,8 +440,8 @@ static inline int MPIDI_CH4R_win_lock(int lock_type, int rank, int assert, MPIR_ msg.lock_type = lock_type; locked = slock->remote.locked + 1; - mpi_errno = MPIDI_NM_am_inject_hdr(rank, win->comm_ptr, - MPIDI_CH4U_WIN_CTRL, &msg, sizeof(msg), NULL); + mpi_errno = MPIDI_NM_am_send_hdr(rank, win->comm_ptr, + MPIDI_CH4U_WIN_CTRL, &msg, sizeof(msg), NULL); if (mpi_errno != MPI_SUCCESS) MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, goto fn_fail, "**rmasync"); MPIDI_CH4R_PROGRESS_WHILE(slock->remote.locked != locked); @@ -482,8 +482,8 @@ static inline int MPIDI_CH4R_win_unlock(int rank, MPIR_Win * win) msg.type = MPIDI_CH4U_WIN_UNLOCK; unlocked = MPIDI_CH4U_WIN(win, sync).lock.remote.locked - 1; - mpi_errno = MPIDI_NM_am_inject_hdr(rank, win->comm_ptr, - MPIDI_CH4U_WIN_CTRL, &msg, sizeof(msg), NULL); + mpi_errno = MPIDI_NM_am_send_hdr(rank, win->comm_ptr, + MPIDI_CH4U_WIN_CTRL, &msg, sizeof(msg), NULL); if (mpi_errno != MPI_SUCCESS) MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, goto fn_fail, "**rmasync"); @@ -1066,8 +1066,8 @@ static inline int MPIDI_CH4R_win_unlock_all(MPIR_Win * win) lockQ[i].peer = i; lockQ[i].win = win; - mpi_errno = MPIDI_NM_am_inject_hdr(i, win->comm_ptr, - MPIDI_CH4U_WIN_CTRL, &msg, sizeof(msg), NULL); + mpi_errno = MPIDI_NM_am_send_hdr(i, win->comm_ptr, + MPIDI_CH4U_WIN_CTRL, &msg, sizeof(msg), NULL); if (mpi_errno != MPI_SUCCESS) MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, goto fn_fail, "**rmasync"); @@ -1228,8 +1228,8 @@ static inline int MPIDI_CH4R_win_lock_all(int assert, MPIR_Win * win) lockQ[i].win = win; lockQ[i].lock_type = MPI_LOCK_SHARED; - mpi_errno = MPIDI_NM_am_inject_hdr(i, win->comm_ptr, - MPIDI_CH4U_WIN_CTRL, &msg, sizeof(msg), NULL); + mpi_errno = MPIDI_NM_am_send_hdr(i, win->comm_ptr, + MPIDI_CH4U_WIN_CTRL, &msg, sizeof(msg), NULL); if (mpi_errno != MPI_SUCCESS) MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, goto fn_fail, "**rmasync"); http://git.mpich.org/mpich.git/commitdiff/af99ecc1e04a32f9224f2d803751c1149… commit af99ecc1e04a32f9224f2d803751c1149e0a97c5 Author: Pavan Balaji <balaji(a)anl.gov> Date: Fri Aug 19 10:40:32 2016 -0500 CH4/OFI: Build fix Signed-off-by: Ken Raffenetti <raffenet(a)mcs.anl.gov> diff --git a/src/mpid/ch4/netmod/ofi/ofi_recv.h b/src/mpid/ch4/netmod/ofi/ofi_recv.h index ecc93b5..2f9e0e8 100644 --- a/src/mpid/ch4/netmod/ofi/ofi_recv.h +++ b/src/mpid/ch4/netmod/ofi/ofi_recv.h @@ -275,6 +275,11 @@ __ALWAYS_INLINE__ int MPIDI_NM_cancel_recv(MPIR_Request * rreq) fn_exit: MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_CANCEL_RECV); return mpi_errno; + +#ifndef MPIDI_BUILD_CH4_SHM +fn_fail: + goto fn_exit; +#endif } #endif /* NETMOD_OFI_RECV_H_INCLUDED */ http://git.mpich.org/mpich.git/commitdiff/7910b821f8e6d816e281d42120737e63c… commit 7910b821f8e6d816e281d42120737e63c63a83b9 Author: Pavan Balaji <balaji(a)anl.gov> Date: Wed Aug 24 16:43:08 2016 -0500 CH4: Remove hdr-only AM API Merge am_send and am_send_hdr into a single function. Remove sendv_hdr and sendv_reply, since they are unused. Signed-off-by: Ken Raffenetti <raffenet(a)mcs.anl.gov> diff --git a/src/mpid/ch4/netmod/include/netmod.h b/src/mpid/ch4/netmod/include/netmod.h index f013ed3..6230eac 100644 --- a/src/mpid/ch4/netmod/include/netmod.h +++ b/src/mpid/ch4/netmod/include/netmod.h @@ -41,9 +41,6 @@ typedef int (*MPIDI_NM_open_port_t) (MPIR_Info * info_ptr, char *port_name); typedef int (*MPIDI_NM_close_port_t) (const char *port_name); typedef int (*MPIDI_NM_comm_accept_t) (const char *port_name, MPIR_Info * info, int root, MPIR_Comm * comm, MPIR_Comm ** newcomm_ptr); -typedef int (*MPIDI_NM_am_send_hdr_t) (int rank, MPIR_Comm * comm, int handler_id, - const void *am_hdr, size_t am_hdr_sz, MPIR_Request * sreq, - void *netmod_context); typedef int (*MPIDI_NM_am_inject_hdr_t) (int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, size_t am_hdr_sz, void *netmod_context); @@ -55,12 +52,6 @@ typedef int (*MPIDI_NM_am_sendv_t) (int rank, MPIR_Comm * comm, int handler_id, struct iovec * am_hdrs, size_t iov_len, const void *data, MPI_Count count, MPI_Datatype datatype, MPIR_Request * sreq, void *netmod_context); -typedef int (*MPIDI_NM_am_sendv_hdr_t) (int rank, MPIR_Comm * comm, int handler_id, - struct iovec * am_hdrs, size_t iov_len, MPIR_Request * sreq, - void *netmod_context); -typedef int (*MPIDI_NM_am_send_hdr_reply_t) (MPIR_Context_id_t context_id, int src_rank, - int handler_id, const void *am_hdr, size_t am_hdr_sz, - MPIR_Request * sreq); typedef int (*MPIDI_NM_am_inject_hdr_reply_t) (MPIR_Context_id_t context_id, int src_rank, int handler_id, const void *am_hdr, size_t am_hdr_sz); @@ -68,10 +59,6 @@ typedef int (*MPIDI_NM_am_send_reply_t) (MPIR_Context_id_t context_id, int src_r const void *am_hdr, size_t am_hdr_sz, const void *data, MPI_Count count, MPI_Datatype datatype, MPIR_Request * sreq); -typedef int (*MPIDI_NM_am_sendv_reply_t) (MPIR_Context_id_t context_id, int src_rank, - int handler_id, struct iovec * am_hdr, size_t iov_len, - const void *data, MPI_Count count, MPI_Datatype datatype, - MPIR_Request * sreq); typedef size_t(*MPIDI_NM_am_hdr_max_sz_t) (void); typedef int (*MPIDI_NM_am_recv_t) (MPIR_Request * req); typedef int (*MPIDI_NM_comm_get_lpid_t) (MPIR_Comm * comm_ptr, int idx, int *lpid_ptr, @@ -379,15 +366,11 @@ typedef struct MPIDI_NM_funcs { MPIDI_NM_am_request_finalize_t am_request_finalize; /* Active Message Routines */ MPIDI_NM_am_reg_handler_t am_reg_handler; - MPIDI_NM_am_send_hdr_t am_send_hdr; MPIDI_NM_am_inject_hdr_t am_inject_hdr; MPIDI_NM_am_send_t am_send; MPIDI_NM_am_sendv_t am_sendv; - MPIDI_NM_am_sendv_hdr_t am_sendv_hdr; - MPIDI_NM_am_send_hdr_reply_t am_send_hdr_reply; MPIDI_NM_am_inject_hdr_reply_t am_inject_hdr_reply; MPIDI_NM_am_send_reply_t am_send_reply; - MPIDI_NM_am_sendv_reply_t am_sendv_reply; MPIDI_NM_am_hdr_max_sz_t am_hdr_max_sz; MPIDI_NM_am_recv_t am_recv; } MPIDI_NM_funcs_t; @@ -534,10 +517,6 @@ MPL_STATIC_INLINE_PREFIX int MPIDI_NM_comm_accept(const char *port_name, MPIR_In int root, MPIR_Comm * comm, MPIR_Comm ** newcomm_ptr) MPL_STATIC_INLINE_SUFFIX; -MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_send_hdr(int rank, MPIR_Comm * comm, int handler_id, - const void *am_hdr, size_t am_hdr_sz, - MPIR_Request * sreq, - void *netmod_context) MPL_STATIC_INLINE_SUFFIX; MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_inject_hdr(int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, size_t am_hdr_sz, void *netmod_context) MPL_STATIC_INLINE_SUFFIX; @@ -551,15 +530,6 @@ MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_sendv(int rank, MPIR_Comm * comm, int h const void *data, MPI_Count count, MPI_Datatype datatype, MPIR_Request * sreq, void *netmod_context) MPL_STATIC_INLINE_SUFFIX; -MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_sendv_hdr(int rank, MPIR_Comm * comm, int handler_id, - struct iovec *am_hdrs, size_t iov_len, - MPIR_Request * sreq, - void *netmod_context) MPL_STATIC_INLINE_SUFFIX; -MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_send_hdr_reply(MPIR_Context_id_t context_id, - int src_rank, int handler_id, - const void *am_hdr, size_t am_hdr_sz, - MPIR_Request * - sreq) MPL_STATIC_INLINE_SUFFIX; MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_inject_hdr_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, const void *am_hdr, @@ -570,12 +540,6 @@ MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_send_reply(MPIR_Context_id_t context_id size_t am_hdr_sz, const void *data, MPI_Count count, MPI_Datatype datatype, MPIR_Request * sreq) MPL_STATIC_INLINE_SUFFIX; -MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_sendv_reply(MPIR_Context_id_t context_id, - int src_rank, int handler_id, - struct iovec *am_hdr, size_t iov_len, - const void *data, MPI_Count count, - MPI_Datatype datatype, - MPIR_Request * sreq) MPL_STATIC_INLINE_SUFFIX; MPL_STATIC_INLINE_PREFIX size_t MPIDI_NM_am_hdr_max_sz(void) MPL_STATIC_INLINE_SUFFIX; MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_recv(MPIR_Request * req) MPL_STATIC_INLINE_SUFFIX; MPL_STATIC_INLINE_PREFIX int MPIDI_NM_comm_get_lpid(MPIR_Comm * comm_ptr, int idx, diff --git a/src/mpid/ch4/netmod/include/netmod_impl.h b/src/mpid/ch4/netmod/include/netmod_impl.h index d143059..dc4f022 100644 --- a/src/mpid/ch4/netmod/include/netmod_impl.h +++ b/src/mpid/ch4/netmod/include/netmod_impl.h @@ -72,14 +72,6 @@ MPL_STATIC_INLINE_PREFIX int MPIDI_NM_comm_accept(const char *port_name, MPIR_In return MPIDI_NM_func->comm_accept(port_name, info, root, comm, newcomm_ptr); }; -MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_send_hdr(int rank, MPIR_Comm * comm, int handler_id, - const void *am_hdr, size_t am_hdr_sz, - MPIR_Request * sreq, void *netmod_context) -{ - return MPIDI_NM_func->am_send_hdr(rank, comm, handler_id, am_hdr, am_hdr_sz, sreq, - netmod_context); -}; - MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_inject_hdr(int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, size_t am_hdr_sz, void *netmod_context) @@ -107,23 +99,6 @@ MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_sendv(int rank, MPIR_Comm * comm, int h sreq, netmod_context); }; -MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_sendv_hdr(int rank, MPIR_Comm * comm, int handler_id, - struct iovec *am_hdrs, size_t iov_len, - MPIR_Request * sreq, void *netmod_context) -{ - return MPIDI_NM_func->am_sendv_hdr(rank, comm, handler_id, am_hdrs, iov_len, sreq, - netmod_context); -}; - -MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_send_hdr_reply(MPIR_Context_id_t context_id, - int src_rank, int handler_id, - const void *am_hdr, size_t am_hdr_sz, - MPIR_Request * sreq) -{ - return MPIDI_NM_func->am_send_hdr_reply(context_id, src_rank, handler_id, am_hdr, am_hdr_sz, - sreq); -}; - MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_inject_hdr_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, const void *am_hdr, size_t am_hdr_sz) @@ -141,16 +116,6 @@ MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_send_reply(MPIR_Context_id_t context_id count, datatype, sreq); }; -MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_sendv_reply(MPIR_Context_id_t context_id, - int src_rank, int handler_id, - struct iovec *am_hdr, size_t iov_len, - const void *data, MPI_Count count, - MPI_Datatype datatype, MPIR_Request * sreq) -{ - return MPIDI_NM_func->am_sendv_reply(context_id, src_rank, handler_id, am_hdr, iov_len, data, - count, datatype, sreq); -}; - MPL_STATIC_INLINE_PREFIX size_t MPIDI_NM_am_hdr_max_sz(void) { return MPIDI_NM_func->am_hdr_max_sz(); diff --git a/src/mpid/ch4/netmod/ofi/func_table.c b/src/mpid/ch4/netmod/ofi/func_table.c index c96f47d..ae06b7b 100644 --- a/src/mpid/ch4/netmod/ofi/func_table.c +++ b/src/mpid/ch4/netmod/ofi/func_table.c @@ -32,15 +32,11 @@ MPIDI_NM_funcs_t MPIDI_NM_ofi_funcs = { MPIDI_NM_am_request_init, MPIDI_NM_am_request_finalize, MPIDI_NM_am_reg_handler, - MPIDI_NM_am_send_hdr, MPIDI_NM_am_inject_hdr, MPIDI_NM_am_send, MPIDI_NM_am_sendv, - MPIDI_NM_am_sendv_hdr, - MPIDI_NM_am_send_hdr_reply, MPIDI_NM_am_inject_hdr_reply, MPIDI_NM_am_send_reply, - MPIDI_NM_am_sendv_reply, MPIDI_NM_am_hdr_max_sz, MPIDI_NM_am_recv }; diff --git a/src/mpid/ch4/netmod/ofi/ofi_am.h b/src/mpid/ch4/netmod/ofi/ofi_am.h index f624d28..f139399 100644 --- a/src/mpid/ch4/netmod/ofi/ofi_am.h +++ b/src/mpid/ch4/netmod/ofi/ofi_am.h @@ -53,26 +53,6 @@ static inline int MPIDI_NM_am_reg_handler(int handler_id, } #undef FUNCNAME -#define FUNCNAME MPIDI_NM_am_send_hdr -#undef FCNAME -#define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_NM_am_send_hdr(int rank, - MPIR_Comm * comm, - int handler_id, - const void *am_hdr, - size_t am_hdr_sz, MPIR_Request * sreq, void *netmod_context) -{ - int mpi_errno = MPI_SUCCESS; - MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_SEND_AM_HDR); - MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_SEND_AM_HDR); - - mpi_errno = MPIDI_OFI_do_am_send_header(rank, comm, handler_id, am_hdr, am_hdr_sz, sreq, FALSE); - MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_SEND_AM_HDR); - return mpi_errno; -} - - -#undef FUNCNAME #define FUNCNAME MPIDI_NM_am_send #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) @@ -88,8 +68,12 @@ static inline int MPIDI_NM_am_send(int rank, int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_SEND_AM); MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_SEND_AM); - mpi_errno = MPIDI_OFI_do_am_send(rank, comm, handler_id, - am_hdr, am_hdr_sz, data, count, datatype, sreq, FALSE); + if (count) + mpi_errno = MPIDI_OFI_do_am_send(rank, comm, handler_id, + am_hdr, am_hdr_sz, data, count, datatype, sreq, FALSE); + else + mpi_errno = MPIDI_OFI_do_am_send_header(rank, comm, handler_id, + am_hdr, am_hdr_sz, sreq, FALSE); MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_SEND_AM); return mpi_errno; @@ -150,77 +134,6 @@ static inline int MPIDI_NM_am_sendv(int rank, } #undef FUNCNAME -#define FUNCNAME MPIDI_NM_am_sendv_hdr -#undef FCNAME -#define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_NM_am_sendv_hdr(int rank, - MPIR_Comm * comm, - int handler_id, - struct iovec *am_hdr, - size_t iov_len, MPIR_Request * sreq, void *netmod_context) -{ - int mpi_errno = MPI_SUCCESS, is_allocated; - size_t am_hdr_sz = 0, i; - char *am_hdr_buf; - - MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_SEND_AMV_HDR); - MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_SEND_AMV_HDR); - - for (i = 0; i < iov_len; i++) { - am_hdr_sz += am_hdr[i].iov_len; - } - - /* TODO: avoid the malloc here, use the am_hdr directly */ - if (am_hdr_sz > MPIDI_OFI_BUF_POOL_SIZE) { - am_hdr_buf = (char *) MPL_malloc(am_hdr_sz); - is_allocated = 1; - } - else { - am_hdr_buf = (char *) MPIDI_CH4R_get_buf(MPIDI_Global.am_buf_pool); - is_allocated = 0; - } - - MPIR_Assert(am_hdr_buf); - am_hdr_sz = 0; - - for (i = 0; i < iov_len; i++) { - MPIR_Memcpy(am_hdr_buf + am_hdr_sz, am_hdr[i].iov_base, am_hdr[i].iov_len); - am_hdr_sz += am_hdr[i].iov_len; - } - - mpi_errno = MPIDI_NM_am_send_hdr(rank, comm, handler_id, am_hdr_buf, am_hdr_sz, - sreq, netmod_context); - - if (is_allocated) - MPL_free(am_hdr_buf); - else - MPIDI_CH4R_release_buf(am_hdr_buf); - - MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_SEND_AMV_HDR); - return mpi_errno; -} - -#undef FUNCNAME -#define FUNCNAME MPIDI_NM_am_send_hdr_reply -#undef FCNAME -#define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_NM_am_send_hdr_reply(MPIR_Context_id_t context_id, - int src_rank, - int handler_id, - const void *am_hdr, - size_t am_hdr_sz, MPIR_Request * sreq) -{ - int mpi_errno = MPI_SUCCESS; - MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_SEND_AM_HDR_REPLY); - MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_SEND_AM_HDR_REPLY); - mpi_errno = MPIDI_OFI_do_am_send_header(src_rank, - MPIDI_CH4U_context_id_to_comm(context_id), - handler_id, am_hdr, am_hdr_sz, sreq, TRUE); - MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_SEND_AM_HDR_REPLY); - return mpi_errno; -} - -#undef FUNCNAME #define FUNCNAME MPIDI_NM_am_send_reply #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) @@ -236,61 +149,16 @@ static inline int MPIDI_NM_am_send_reply(MPIR_Context_id_t context_id, int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_SEND_AM_REPLY); MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_SEND_AM_REPLY); - mpi_errno = MPIDI_OFI_do_am_send(src_rank, - MPIDI_CH4U_context_id_to_comm(context_id), - handler_id, - am_hdr, am_hdr_sz, data, count, datatype, sreq, TRUE); - MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_SEND_AM_REPLY); - return mpi_errno; -} - -static inline int MPIDI_NM_am_sendv_reply(MPIR_Context_id_t context_id, - int src_rank, - int handler_id, - struct iovec *am_hdr, - size_t iov_len, - const void *data, - MPI_Count count, - MPI_Datatype datatype, MPIR_Request * sreq) -{ - int mpi_errno = MPI_SUCCESS, is_allocated; - size_t am_hdr_sz = 0, i; - char *am_hdr_buf; - - MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_SEND_AMV_REPLY); - MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_SEND_AMV_REPLY); - - for (i = 0; i < iov_len; i++) { - am_hdr_sz += am_hdr[i].iov_len; - } - - /* TODO: avoid the malloc here, use the am_hdr directly */ - if (am_hdr_sz > MPIDI_OFI_BUF_POOL_SIZE) { - am_hdr_buf = (char *) MPL_malloc(am_hdr_sz); - is_allocated = 1; - } - else { - am_hdr_buf = (char *) MPIDI_CH4R_get_buf(MPIDI_Global.am_buf_pool); - is_allocated = 0; - } - - MPIR_Assert(am_hdr_buf); - am_hdr_sz = 0; - - for (i = 0; i < iov_len; i++) { - MPIR_Memcpy(am_hdr_buf + am_hdr_sz, am_hdr[i].iov_base, am_hdr[i].iov_len); - am_hdr_sz += am_hdr[i].iov_len; - } - - mpi_errno = MPIDI_NM_am_send_reply(context_id, src_rank, handler_id, am_hdr_buf, am_hdr_sz, - data, count, datatype, sreq); - - if (is_allocated) - MPL_free(am_hdr_buf); + if (count) + mpi_errno = MPIDI_OFI_do_am_send(src_rank, + MPIDI_CH4U_context_id_to_comm(context_id), + handler_id, + am_hdr, am_hdr_sz, data, count, datatype, sreq, TRUE); else - MPIDI_CH4R_release_buf(am_hdr_buf); - - MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_SEND_AMV_REPLY); + mpi_errno = MPIDI_OFI_do_am_send_header(src_rank, + MPIDI_CH4U_context_id_to_comm(context_id), + handler_id, am_hdr, am_hdr_sz, sreq, TRUE); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_SEND_AM_REPLY); return mpi_errno; } diff --git a/src/mpid/ch4/netmod/portals4/func_table.c b/src/mpid/ch4/netmod/portals4/func_table.c index c5b92db..eade09e 100644 --- a/src/mpid/ch4/netmod/portals4/func_table.c +++ b/src/mpid/ch4/netmod/portals4/func_table.c @@ -32,15 +32,11 @@ MPIDI_NM_funcs_t MPIDI_NM_portals4_funcs = { MPIDI_NM_am_request_init, MPIDI_NM_am_request_finalize, MPIDI_NM_am_reg_handler, - MPIDI_NM_am_send_hdr, MPIDI_NM_am_inject_hdr, MPIDI_NM_am_send, MPIDI_NM_am_sendv, - MPIDI_NM_am_sendv_hdr, - MPIDI_NM_am_send_hdr_reply, MPIDI_NM_am_inject_hdr_reply, MPIDI_NM_am_send_reply, - MPIDI_NM_am_sendv_reply, MPIDI_NM_am_hdr_max_sz, MPIDI_NM_am_recv, }; diff --git a/src/mpid/ch4/netmod/portals4/ptl_am.h b/src/mpid/ch4/netmod/portals4/ptl_am.h index c2a560c..0ff7db0 100644 --- a/src/mpid/ch4/netmod/portals4/ptl_am.h +++ b/src/mpid/ch4/netmod/portals4/ptl_am.h @@ -34,41 +34,6 @@ static inline int MPIDI_NM_am_reg_handler(int handler_id, goto fn_exit; } -static inline int MPIDI_NM_am_send_hdr(int rank, - MPIR_Comm * comm, - int handler_id, - const void *am_hdr, - size_t am_hdr_sz, MPIR_Request * sreq, void *netmod_context) -{ - int mpi_errno = MPI_SUCCESS, ret, c; - ptl_hdr_data_t ptl_hdr; - ptl_match_bits_t match_bits; - char *send_buf = NULL; - - MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_SEND_AM); - MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_SEND_AM); - - ptl_hdr = MPIDI_PTL_init_am_hdr(handler_id, 0); - match_bits = MPIDI_PTL_init_tag(comm->context_id, MPIDI_PTL_AM_TAG); - sreq->dev.ch4.ch4u.netmod_am.portals4.handler_id = handler_id; - - MPIR_cc_incr(sreq->cc_ptr, &c); - - send_buf = MPL_malloc(am_hdr_sz); - MPIR_Memcpy(send_buf, am_hdr, am_hdr_sz); - sreq->dev.ch4.ch4u.netmod_am.portals4.pack_buffer = send_buf; - - ret = PtlPut(MPIDI_PTL_global.md, (ptl_size_t) send_buf, am_hdr_sz, - PTL_ACK_REQ, MPIDI_PTL_global.addr_table[rank].process, - MPIDI_PTL_global.addr_table[rank].pt, match_bits, 0, sreq, ptl_hdr); - - fn_exit: - MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_SEND_AM); - return mpi_errno; - fn_fail: - goto fn_exit; -} - static inline int MPIDI_NM_am_send(int rank, MPIR_Comm * comm, int handler_id, @@ -90,13 +55,29 @@ static inline int MPIDI_NM_am_send(int rank, MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_SEND_AM); MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_SEND_AM); - MPIDI_Datatype_get_info(count, datatype, dt_contig, data_sz, dt_ptr, dt_true_lb); match_bits = MPIDI_PTL_init_tag(comm->context_id, MPIDI_PTL_AM_TAG); - ptl_hdr = MPIDI_PTL_init_am_hdr(handler_id, data_sz); sreq->dev.ch4.ch4u.netmod_am.portals4.handler_id = handler_id; MPIR_cc_incr(sreq->cc_ptr, &c); + /* fast path: there's no data to be sent */ + if (count == 0) { + send_buf = MPL_malloc(am_hdr_sz); + MPIR_Memcpy(send_buf, am_hdr, am_hdr_sz); + sreq->dev.ch4.ch4u.netmod_am.portals4.pack_buffer = send_buf; + + ptl_hdr = MPIDI_PTL_init_am_hdr(handler_id, 0); + + ret = PtlPut(MPIDI_PTL_global.md, (ptl_size_t) send_buf, am_hdr_sz, + PTL_ACK_REQ, MPIDI_PTL_global.addr_table[rank].process, + MPIDI_PTL_global.addr_table[rank].pt, match_bits, 0, sreq, ptl_hdr); + + goto fn_exit; + } + + MPIDI_Datatype_get_info(count, datatype, dt_contig, data_sz, dt_ptr, dt_true_lb); + ptl_hdr = MPIDI_PTL_init_am_hdr(handler_id, data_sz); + if (dt_contig) { /* create a two element iovec and send */ ptl_md_t md; @@ -162,26 +143,6 @@ static inline int MPIDI_NM_am_sendv(int rank, return MPI_SUCCESS; } -static inline int MPIDI_NM_am_sendv_hdr(int rank, - MPIR_Comm * comm, - int handler_id, - struct iovec *am_hdr, - size_t iov_len, MPIR_Request * sreq, void *netmod_context) -{ - MPIR_Assert(0); - return MPI_SUCCESS; -} - -static inline int MPIDI_NM_am_send_hdr_reply(MPIR_Context_id_t context_id, - int src_rank, - int handler_id, - const void *am_hdr, - size_t am_hdr_sz, MPIR_Request * sreq) -{ - MPIR_Assert(0); - return MPI_SUCCESS; -} - static inline int MPIDI_NM_am_send_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, @@ -264,19 +225,6 @@ static inline int MPIDI_NM_am_send_reply(MPIR_Context_id_t context_id, goto fn_exit; } -static inline int MPIDI_NM_am_sendv_reply(MPIR_Context_id_t context_id, - int src_rank, - int handler_id, - struct iovec *am_hdr, - size_t iov_len, - const void *data, - MPI_Count count, - MPI_Datatype datatype, MPIR_Request * sreq) -{ - MPIR_Assert(0); - return MPI_SUCCESS; -} - static inline size_t MPIDI_NM_am_hdr_max_sz(void) { MPIR_Assert(0); diff --git a/src/mpid/ch4/netmod/stubnm/globals.c b/src/mpid/ch4/netmod/stubnm/globals.c index 7ecbbb2..89a5cc9 100644 --- a/src/mpid/ch4/netmod/stubnm/globals.c +++ b/src/mpid/ch4/netmod/stubnm/globals.c @@ -32,15 +32,11 @@ MPIDI_NM_funcs_t MPIDI_NM_stubnm_funcs = { MPIDI_NM_am_request_init, MPIDI_NM_am_request_finalize, MPIDI_NM_am_reg_handler, - MPIDI_NM_am_send_hdr, MPIDI_NM_am_inject_hdr, MPIDI_NM_am_send, MPIDI_NM_am_sendv, - MPIDI_NM_am_sendv_hdr, - MPIDI_NM_am_send_hdr_reply, MPIDI_NM_am_inject_hdr_reply, MPIDI_NM_am_send_reply, - MPIDI_NM_am_sendv_reply, MPIDI_NM_am_hdr_max_sz, MPIDI_NM_am_recv, }; diff --git a/src/mpid/ch4/netmod/stubnm/stubnm_am.h b/src/mpid/ch4/netmod/stubnm/stubnm_am.h index 074c696..f33e1c3 100644 --- a/src/mpid/ch4/netmod/stubnm/stubnm_am.h +++ b/src/mpid/ch4/netmod/stubnm/stubnm_am.h @@ -21,16 +21,6 @@ static inline int MPIDI_NM_am_reg_handler(int handler_id, return MPI_SUCCESS; } -static inline int MPIDI_NM_am_send_hdr(int rank, - MPIR_Comm * comm, - int handler_id, - const void *am_hdr, - size_t am_hdr_sz, MPIR_Request * sreq, void *netmod_context) -{ - MPIR_Assert(0); - return MPI_SUCCESS; -} - static inline int MPIDI_NM_am_send(int rank, MPIR_Comm * comm, int handler_id, @@ -58,25 +48,6 @@ static inline int MPIDI_NM_am_sendv(int rank, return MPI_SUCCESS; } -static inline int MPIDI_NM_am_sendv_hdr(int rank, - MPIR_Comm * comm, - int handler_id, - struct iovec *am_hdr, - size_t iov_len, MPIR_Request * sreq, void *netmod_context) -{ - MPIR_Assert(0); - return MPI_SUCCESS; -} - -static inline int MPIDI_NM_am_send_hdr_reply(MPIR_Context_id_t context_id, int src_rank, - int handler_id, - const void *am_hdr, - size_t am_hdr_sz, MPIR_Request * sreq) -{ - MPIR_Assert(0); - return MPI_SUCCESS; -} - static inline int MPIDI_NM_am_send_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, const void *am_hdr, @@ -89,18 +60,6 @@ static inline int MPIDI_NM_am_send_reply(MPIR_Context_id_t context_id, int src_r return MPI_SUCCESS; } -static inline int MPIDI_NM_am_sendv_reply(MPIR_Context_id_t context_id, int src_rank, - int handler_id, - struct iovec *am_hdr, - size_t iov_len, - const void *data, - MPI_Count count, - MPI_Datatype datatype, MPIR_Request * sreq) -{ - MPIR_Assert(0); - return MPI_SUCCESS; -} - static inline size_t MPIDI_NM_am_hdr_max_sz(void) { MPIR_Assert(0); diff --git a/src/mpid/ch4/netmod/ucx/func_table.c b/src/mpid/ch4/netmod/ucx/func_table.c index f4a03fe..121a197 100644 --- a/src/mpid/ch4/netmod/ucx/func_table.c +++ b/src/mpid/ch4/netmod/ucx/func_table.c @@ -30,15 +30,11 @@ MPIDI_NM_funcs_t MPIDI_NM_ucx_funcs = { MPIDI_NM_am_request_init, MPIDI_NM_am_request_finalize, MPIDI_NM_am_reg_handler, - MPIDI_NM_am_send_hdr, MPIDI_NM_am_inject_hdr, MPIDI_NM_am_send, MPIDI_NM_am_sendv, - MPIDI_NM_am_sendv_hdr, - MPIDI_NM_am_send_hdr_reply, MPIDI_NM_am_inject_hdr_reply, MPIDI_NM_am_send_reply, - MPIDI_NM_am_sendv_reply, MPIDI_NM_am_hdr_max_sz, MPIDI_NM_am_recv }; diff --git a/src/mpid/ch4/netmod/ucx/ucx_am.h b/src/mpid/ch4/netmod/ucx/ucx_am.h index 64a0051..3277010 100644 --- a/src/mpid/ch4/netmod/ucx/ucx_am.h +++ b/src/mpid/ch4/netmod/ucx/ucx_am.h @@ -76,74 +76,6 @@ static inline void MPIDI_UCX_inject_am_callback(void *request, ucs_status_t stat } #undef FUNCNAME -#define FUNCNAME MPIDI_NM_am_send_hdr -#undef FCNAME -#define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_NM_am_send_hdr(int rank, - MPIR_Comm * comm, - int handler_id, - const void *am_hdr, - size_t am_hdr_sz, MPIR_Request * sreq, void *netmod_context) -{ - int mpi_errno = MPI_SUCCESS, c; - MPIDI_UCX_ucp_request_t *ucp_request; - ucp_ep_h ep; - uint64_t ucx_tag; - char *send_buf; - MPIDI_UCX_am_header_t ucx_hdr; - - MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_SEND_AM_HDR); - MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_SEND_AM_HDR); - - ep = MPIDI_UCX_COMM_TO_EP(comm, rank); - ucx_tag = MPIDI_UCX_init_tag(0, 0, MPIDI_UCX_AM_TAG); - - /* initialize our portion of the hdr */ - ucx_hdr.handler_id = handler_id; - ucx_hdr.data_sz = 0; - - /* just pack and send for now */ - send_buf = MPL_malloc(am_hdr_sz + sizeof(ucx_hdr)); - MPIR_Memcpy(send_buf, &ucx_hdr, sizeof(ucx_hdr)); - MPIR_Memcpy(send_buf + sizeof(ucx_hdr), am_hdr, am_hdr_sz); - - ucp_request = (MPIDI_UCX_ucp_request_t *) ucp_tag_send_nb(ep, send_buf, - am_hdr_sz + sizeof(ucx_hdr), - ucp_dt_make_contig(1), ucx_tag, - &MPIDI_UCX_am_send_callback); - MPIDI_CH4_UCX_REQUEST(ucp_request, tag_send_nb); - - /* send is done. free all resources and complete the request */ - if (ucp_request == NULL) { - MPL_free(send_buf); - MPIDI_UCX_global.send_cmpl_handlers[handler_id] (sreq); - goto fn_exit; - } - - /* request completed between the UCP call and now. free resources - * and complete the send request */ - if (ucp_request->req) { - MPL_free(send_buf); - MPIDI_UCX_global.send_cmpl_handlers[handler_id] (sreq); - ucp_request->req = NULL; - ucp_request_release(ucp_request); - } - else { - /* set the ch4r request inside the UCP request */ - sreq->dev.ch4.ch4u.netmod_am.ucx.pack_buffer = send_buf; - sreq->dev.ch4.ch4u.netmod_am.ucx.handler_id = handler_id; - ucp_request->req = sreq; - ucp_request_release(ucp_request); - } - - fn_exit: - MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_SEND_AM_HDR); - return mpi_errno; - fn_fail: - goto fn_exit; -} - -#undef FUNCNAME #define FUNCNAME MPIDI_NM_am_send #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) @@ -259,43 +191,6 @@ static inline int MPIDI_NM_am_send(int rank, } #undef FUNCNAME -#define FUNCNAME MPIDI_NM_am_sendv_hdr -#undef FCNAME -#define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_NM_am_sendv_hdr(int rank, - MPIR_Comm * comm, - int handler_id, - struct iovec *am_hdr, - size_t iov_len, MPIR_Request * sreq, void *netmod_context) -{ - int mpi_errno = MPI_SUCCESS; - size_t am_hdr_sz = 0, i; - char *am_hdr_buf; - - MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_UCX_SEND_AMV_HDR); - MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_UCX_SEND_AMV_HDR); - - for (i = 0; i < iov_len; i++) { - am_hdr_sz += am_hdr[i].iov_len; - } - - am_hdr_buf = (char *) MPL_malloc(am_hdr_sz); - MPIR_Assert(am_hdr_buf); - am_hdr_sz = 0; - - for (i = 0; i < iov_len; i++) { - MPIR_Memcpy(am_hdr_buf + am_hdr_sz, am_hdr[i].iov_base, am_hdr[i].iov_len); - am_hdr_sz += am_hdr[i].iov_len; - } - - mpi_errno = MPIDI_NM_am_send_hdr(rank, comm, handler_id, am_hdr_buf, am_hdr_sz, - sreq, netmod_context); - MPL_free(am_hdr_buf); - MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_UCX_SEND_AMV_HDR); - return mpi_errno; -} - -#undef FUNCNAME #define FUNCNAME MPIDI_NM_am_sendv #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) @@ -339,21 +234,6 @@ static inline int MPIDI_NM_am_sendv(int rank, #undef FUNCNAME -#define FUNCNAME MPIDI_NM_am_send_hdr_reply -#undef FCNAME -#define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_NM_am_send_hdr_reply(MPIR_Context_id_t context_id, - int src_rank, - int handler_id, - const void *am_hdr, - size_t am_hdr_sz, MPIR_Request * sreq) -{ - - return MPIDI_NM_am_send_hdr(src_rank, MPIDI_CH4U_context_id_to_comm(context_id), handler_id, - am_hdr, am_hdr_sz, sreq, NULL); -} - -#undef FUNCNAME #define FUNCNAME MPIDI_NM_am_send_reply #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) @@ -436,42 +316,6 @@ static inline int MPIDI_NM_am_send_reply(MPIR_Context_id_t context_id, goto fn_exit; } -static inline int MPIDI_NM_am_sendv_reply(MPIR_Context_id_t context_id, - int src_rank, - int handler_id, - struct iovec *am_hdr, - size_t iov_len, - const void *data, MPI_Count count, - MPI_Datatype datatype, MPIR_Request * sreq) -{ - int mpi_errno = MPI_SUCCESS; - size_t am_hdr_sz = 0, i; - char *am_hdr_buf; - - MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_UCX_SEND_AMV_REPLY); - MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_UCX_SEND_AMV_REPLY); - - for (i = 0; i < iov_len; i++) { - am_hdr_sz += am_hdr[i].iov_len; - } - - am_hdr_buf = (char *) MPL_malloc(am_hdr_sz); - - MPIR_Assert(am_hdr_buf); - am_hdr_sz = 0; - - for (i = 0; i < iov_len; i++) { - MPIR_Memcpy(am_hdr_buf + am_hdr_sz, am_hdr[i].iov_base, am_hdr[i].iov_len); - am_hdr_sz += am_hdr[i].iov_len; - } - - mpi_errno = MPIDI_NM_am_send_reply(context_id, src_rank, handler_id, am_hdr_buf, am_hdr_sz, - data, count, datatype, sreq); - MPL_free(am_hdr_buf); - MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_UCX_SEND_AMV_REPLY); - return mpi_errno; -} - static inline size_t MPIDI_NM_am_hdr_max_sz(void) { return (MPIDI_UCX_MAX_AM_EAGER_SZ - sizeof(MPIDI_UCX_am_header_t)); diff --git a/src/mpid/ch4/shm/include/shm.h b/src/mpid/ch4/shm/include/shm.h index ed7f0f3..1013990 100644 --- a/src/mpid/ch4/shm/include/shm.h +++ b/src/mpid/ch4/shm/include/shm.h @@ -40,9 +40,6 @@ typedef int (*MPIDI_SHM_open_port_t) (MPIR_Info * info_ptr, char *port_name); typedef int (*MPIDI_SHM_close_port_t) (const char *port_name); typedef int (*MPIDI_SHM_comm_accept_t) (const char *port_name, MPIR_Info * info, int root, MPIR_Comm * comm, MPIR_Comm ** newcomm_ptr); -typedef int (*MPIDI_SHM_am_send_hdr_t) (int rank, MPIR_Comm * comm, int handler_id, - const void *am_hdr, size_t am_hdr_sz, MPIR_Request * sreq, - void *shm_context); typedef int (*MPIDI_SHM_am_inject_hdr_t) (int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, size_t am_hdr_sz, void *shm_context); typedef int (*MPIDI_SHM_am_send_t) (int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, @@ -58,9 +55,6 @@ typedef int (*MPIDI_SHM_am_sendv_t) (int rank, MPIR_Comm * comm, int handler_id, typedef int (*MPIDI_SHM_inject_amv_t) (int rank, MPIR_Comm * comm, int handler_id, struct iovec * am_hdrs, size_t iov_len, const void *data, MPI_Count count, MPI_Datatype datatype, void *shm_context); -typedef int (*MPIDI_SHM_am_send_hdr_reply_t) (MPIR_Context_id_t context_id, int src_rank, - int handler_id, const void *am_hdr, size_t am_hdr_sz, - MPIR_Request * sreq); typedef int (*MPIDI_SHM_am_inject_hdr_reply_t) (MPIR_Context_id_t context_id, int src_rank, int handler_id, const void *am_hdr, size_t am_hdr_sz); @@ -72,10 +66,6 @@ typedef int (*MPIDI_SHM_inject_am_reply_t) (MPIR_Context_id_t context_id, int sr int handler_id, const void *am_hdr, size_t am_hdr_sz, const void *data, MPI_Count count, MPI_Datatype datatype); -typedef int (*MPIDI_SHM_am_sendv_reply_t) (MPIR_Context_id_t context_id, int src_rank, - int handler_id, struct iovec * am_hdr, size_t iov_len, - const void *data, MPI_Count count, MPI_Datatype datatype, - MPIR_Request * sreq); typedef int (*MPIDI_SHM_inject_amv_reply_t) (MPIR_Context_id_t context_id, int src_rank, int handler_id, struct iovec * am_hdrs, size_t iov_len, const void *data, MPI_Count count, @@ -382,17 +372,14 @@ typedef struct MPIDI_SHM_funcs { MPIDI_SHM_open_port_t open_port; MPIDI_SHM_close_port_t close_port; MPIDI_SHM_comm_accept_t comm_accept; - MPIDI_SHM_am_send_hdr_t am_send_hdr; MPIDI_SHM_am_inject_hdr_t am_inject_hdr; MPIDI_SHM_am_send_t am_send; MPIDI_SHM_inject_am_t inject_am; MPIDI_SHM_am_sendv_t am_sendv; MPIDI_SHM_inject_amv_t inject_amv; - MPIDI_SHM_am_send_hdr_reply_t am_send_hdr_reply; MPIDI_SHM_am_inject_hdr_reply_t am_inject_hdr_reply; MPIDI_SHM_am_send_reply_t am_send_reply; MPIDI_SHM_inject_am_reply_t inject_am_reply; - MPIDI_SHM_am_sendv_reply_t am_sendv_reply; MPIDI_SHM_inject_amv_reply_t inject_amv_reply; MPIDI_SHM_am_hdr_max_sz_t am_hdr_max_sz; MPIDI_SHM_am_inject_max_sz_t am_inject_max_sz; @@ -546,10 +533,6 @@ MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_accept(const char *port_name, MPIR_I int root, MPIR_Comm * comm, MPIR_Comm ** newcomm_ptr) MPL_STATIC_INLINE_SUFFIX; -MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_am_send_hdr(int rank, MPIR_Comm * comm, int handler_id, - const void *am_hdr, size_t am_hdr_sz, - MPIR_Request * sreq, - void *shm_context) MPL_STATIC_INLINE_SUFFIX; MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_am_inject_hdr(int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, size_t am_hdr_sz, @@ -574,11 +557,6 @@ MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_amv(int rank, MPIR_Comm * comm, in const void *data, MPI_Count count, MPI_Datatype datatype, void *shm_context) MPL_STATIC_INLINE_SUFFIX; -MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_am_send_hdr_reply(MPIR_Context_id_t context_id, - int src_rank, int handler_id, - const void *am_hdr, size_t am_hdr_sz, - MPIR_Request * - sreq) MPL_STATIC_INLINE_SUFFIX; MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_am_inject_hdr_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, const void *am_hdr, @@ -596,12 +574,6 @@ MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_am_reply(MPIR_Context_id_t context const void *data, MPI_Count count, MPI_Datatype datatype) MPL_STATIC_INLINE_SUFFIX; -MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_am_sendv_reply(MPIR_Context_id_t context_id, - int src_rank, int handler_id, - struct iovec *am_hdr, size_t iov_len, - const void *data, MPI_Count count, - MPI_Datatype datatype, - MPIR_Request * sreq) MPL_STATIC_INLINE_SUFFIX; MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_amv_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, struct iovec *am_hdrs, size_t iov_len, diff --git a/src/mpid/ch4/shm/include/shm_impl.h b/src/mpid/ch4/shm/include/shm_impl.h index 92a5e0b..dd7a495 100644 --- a/src/mpid/ch4/shm/include/shm_impl.h +++ b/src/mpid/ch4/shm/include/shm_impl.h @@ -68,14 +68,6 @@ MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_accept(const char *port_name, MPIR_I return MPIDI_SHM_func->comm_accept(port_name, info, root, comm, newcomm_ptr); }; -MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_am_send_hdr(int rank, MPIR_Comm * comm, int handler_id, - const void *am_hdr, size_t am_hdr_sz, - MPIR_Request * sreq, void *shm_context) -{ - return MPIDI_SHM_func->am_send_hdr(rank, comm, handler_id, am_hdr, am_hdr_sz, sreq, - shm_context); -}; - MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_am_inject_hdr(int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, size_t am_hdr_sz, void *shm_context) @@ -121,15 +113,6 @@ MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_amv(int rank, MPIR_Comm * comm, in datatype, shm_context); }; -MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_am_send_hdr_reply(MPIR_Context_id_t context_id, - int src_rank, int handler_id, - const void *am_hdr, size_t am_hdr_sz, - MPIR_Request * sreq) -{ - return MPIDI_SHM_func->am_send_hdr_reply(context_id, src_rank, handler_id, am_hdr, am_hdr_sz, - sreq); -}; - MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_am_inject_hdr_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, const void *am_hdr, size_t am_hdr_sz) @@ -157,16 +140,6 @@ MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_am_reply(MPIR_Context_id_t context data, count, datatype); }; -MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_am_sendv_reply(MPIR_Context_id_t context_id, - int src_rank, int handler_id, - struct iovec *am_hdr, size_t iov_len, - const void *data, MPI_Count count, - MPI_Datatype datatype, MPIR_Request * sreq) -{ - return MPIDI_SHM_func->am_sendv_reply(context_id, src_rank, handler_id, am_hdr, iov_len, data, - count, datatype, sreq); -}; - MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_amv_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, struct iovec *am_hdrs, size_t iov_len, diff --git a/src/mpid/ch4/shm/posix/func_table.c b/src/mpid/ch4/shm/posix/func_table.c index 4a39d50..f6d884f 100644 --- a/src/mpid/ch4/shm/posix/func_table.c +++ b/src/mpid/ch4/shm/posix/func_table.c @@ -22,17 +22,14 @@ MPIDI_SHM_funcs_t MPIDI_SHM_posix_funcs = { MPIDI_SHM_open_port, MPIDI_SHM_close_port, MPIDI_SHM_comm_accept, - MPIDI_SHM_am_send_hdr, MPIDI_SHM_am_inject_hdr, MPIDI_SHM_am_send, MPIDI_SHM_inject_am, MPIDI_SHM_am_sendv, MPIDI_SHM_inject_amv, - MPIDI_SHM_am_send_hdr_reply, MPIDI_SHM_am_inject_hdr_reply, MPIDI_SHM_am_send_reply, MPIDI_SHM_inject_am_reply, - MPIDI_SHM_am_sendv_reply, MPIDI_SHM_inject_amv_reply, MPIDI_SHM_am_hdr_max_sz, MPIDI_SHM_am_inject_max_sz, diff --git a/src/mpid/ch4/shm/posix/posix_am.h b/src/mpid/ch4/shm/posix/posix_am.h index 842bf6c..6b41b54 100644 --- a/src/mpid/ch4/shm/posix/posix_am.h +++ b/src/mpid/ch4/shm/posix/posix_am.h @@ -20,16 +20,6 @@ static inline int MPIDI_SHM_am_reg_handler(int handler_id, return MPI_SUCCESS; } -static inline int MPIDI_SHM_am_send_hdr(int rank, - MPIR_Comm * comm, - int handler_id, - const void *am_hdr, - size_t am_hdr_sz, MPIR_Request * sreq, void *shm_context) -{ - MPIR_Assert(0); - return MPI_SUCCESS; -} - static inline int MPIDI_SHM_am_send(int rank, MPIR_Comm * comm, int handler_id, @@ -56,15 +46,6 @@ static inline int MPIDI_SHM_am_sendv(int rank, return MPI_SUCCESS; } -static inline int MPIDI_SHM_am_send_hdr_reply(MPIR_Context_id_t context_id, int src_rank, - int handler_id, - const void *am_hdr, - size_t am_hdr_sz, MPIR_Request * sreq) -{ - MPIR_Assert(0); - return MPI_SUCCESS; -} - static inline int MPIDI_SHM_am_send_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, const void *am_hdr, @@ -77,18 +58,6 @@ static inline int MPIDI_SHM_am_send_reply(MPIR_Context_id_t context_id, int src_ return MPI_SUCCESS; } -static inline int MPIDI_SHM_am_sendv_reply(MPIR_Context_id_t context_id, int src_rank, - int handler_id, - struct iovec *am_hdr, - size_t iov_len, - const void *data, - MPI_Count count, - MPI_Datatype datatype, MPIR_Request * sreq) -{ - MPIR_Assert(0); - return MPI_SUCCESS; -} - static inline size_t MPIDI_SHM_am_hdr_max_sz(void) { MPIR_Assert(0); diff --git a/src/mpid/ch4/shm/stubshm/func_table.c b/src/mpid/ch4/shm/stubshm/func_table.c index 237869c..3b8253c 100644 --- a/src/mpid/ch4/shm/stubshm/func_table.c +++ b/src/mpid/ch4/shm/stubshm/func_table.c @@ -22,17 +22,14 @@ MPIDI_SHM_funcs_t MPIDI_SHM_stubshm_funcs = { MPIDI_SHM_open_port, MPIDI_SHM_close_port, MPIDI_SHM_comm_accept, - MPIDI_SHM_am_send_hdr, MPIDI_SHM_am_inject_hdr, MPIDI_SHM_am_send, MPIDI_SHM_inject_am, MPIDI_SHM_am_sendv, MPIDI_SHM_inject_amv, - MPIDI_SHM_am_send_hdr_reply, MPIDI_SHM_am_inject_hdr_reply, MPIDI_SHM_am_send_reply, MPIDI_SHM_inject_am_reply, - MPIDI_SHM_am_sendv_reply, MPIDI_SHM_inject_amv_reply, MPIDI_SHM_am_hdr_max_sz, MPIDI_SHM_am_inject_max_sz, diff --git a/src/mpid/ch4/shm/stubshm/stubshm_am.h b/src/mpid/ch4/shm/stubshm/stubshm_am.h index e837c9f..2e29e5d 100644 --- a/src/mpid/ch4/shm/stubshm/stubshm_am.h +++ b/src/mpid/ch4/shm/stubshm/stubshm_am.h @@ -20,16 +20,6 @@ static inline int MPIDI_SHM_am_reg_handler(int handler_id, return MPI_SUCCESS; } -static inline int MPIDI_SHM_am_send_hdr(int rank, - MPIR_Comm * comm, - int handler_id, - const void *am_hdr, - size_t am_hdr_sz, MPIR_Request * sreq, void *shm_context) -{ - MPIR_Assert(0); - return MPI_SUCCESS; -} - static inline int MPIDI_SHM_am_send(int rank, MPIR_Comm * comm, int handler_id, @@ -56,15 +46,6 @@ static inline int MPIDI_SHM_am_sendv(int rank, return MPI_SUCCESS; } -static inline int MPIDI_SHM_am_send_hdr_reply(MPIR_Context_id_t context_id, int src_rank, - int handler_id, - const void *am_hdr, - size_t am_hdr_sz, MPIR_Request * sreq) -{ - MPIR_Assert(0); - return MPI_SUCCESS; -} - static inline int MPIDI_SHM_am_send_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, const void *am_hdr, @@ -77,18 +58,6 @@ static inline int MPIDI_SHM_am_send_reply(MPIR_Context_id_t context_id, int src_ return MPI_SUCCESS; } -static inline int MPIDI_SHM_am_sendv_reply(MPIR_Context_id_t context_id, int src_rank, - int handler_id, - struct iovec *am_hdr, - size_t iov_len, - const void *data, - MPI_Count count, - MPI_Datatype datatype, MPIR_Request * sreq) -{ - MPIR_Assert(0); - return MPI_SUCCESS; -} - static inline size_t MPIDI_SHM_am_hdr_max_sz(void) { MPIR_Assert(0); diff --git a/src/mpid/ch4/src/ch4r_callbacks.h b/src/mpid/ch4/src/ch4r_callbacks.h index 471d3e3..3ab5c1d 100644 --- a/src/mpid/ch4/src/ch4r_callbacks.h +++ b/src/mpid/ch4/src/ch4r_callbacks.h @@ -414,9 +414,10 @@ static inline int MPIDI_CH4U_reply_ssend(MPIR_Request * rreq) MPIR_cc_incr(rreq->cc_ptr, &c); ack_msg.sreq_ptr = MPIDI_CH4U_REQUEST(rreq, req->rreq.peer_req_ptr); - mpi_errno = MPIDI_NM_am_send_hdr_reply(MPIDI_CH4U_get_context(MPIDI_CH4U_REQUEST(rreq, tag)), - MPIDI_CH4U_REQUEST(rreq, src_rank), - MPIDI_CH4U_SSEND_ACK, &ack_msg, sizeof(ack_msg), rreq); + mpi_errno = MPIDI_NM_am_send_reply(MPIDI_CH4U_get_context(MPIDI_CH4U_REQUEST(rreq, tag)), + MPIDI_CH4U_REQUEST(rreq, src_rank), + MPIDI_CH4U_SSEND_ACK, &ack_msg, sizeof(ack_msg), + NULL, 0, MPI_DATATYPE_NULL, rreq); if (mpi_errno) MPIR_ERR_POP(mpi_errno); fn_exit: diff --git a/src/mpid/ch4/src/ch4r_rma.h b/src/mpid/ch4/src/ch4r_rma.h index 1277ff5..adaaa0c 100644 --- a/src/mpid/ch4/src/ch4r_rma.h +++ b/src/mpid/ch4/src/ch4r_rma.h @@ -210,8 +210,9 @@ static inline int MPIDI_CH4I_do_get(void *origin_addr, am_hdr.n_iov = 0; MPIDI_CH4U_REQUEST(sreq, req->greq.dt_iov) = NULL; - mpi_errno = MPIDI_NM_am_send_hdr(target_rank, win->comm_ptr, - MPIDI_CH4U_GET_REQ, &am_hdr, sizeof(am_hdr), sreq, NULL); + mpi_errno = MPIDI_NM_am_send(target_rank, win->comm_ptr, + MPIDI_CH4U_GET_REQ, &am_hdr, sizeof(am_hdr), + NULL, 0, MPI_DATATYPE_NULL, sreq, NULL); if (mpi_errno) MPIR_ERR_POP(mpi_errno); goto fn_exit; http://git.mpich.org/mpich.git/commitdiff/3257bbea93f661d54b8d43e0d06bc4841… commit 3257bbea93f661d54b8d43e0d06bc4841d496b13 Author: Pavan Balaji <balaji(a)anl.gov> Date: Wed Aug 24 16:34:32 2016 -0500 CH4: Name AM functions consistently Signed-off-by: Ken Raffenetti <raffenet(a)mcs.anl.gov> diff --git a/src/mpid/ch4/netmod/include/netmod.h b/src/mpid/ch4/netmod/include/netmod.h index 1225705..f013ed3 100644 --- a/src/mpid/ch4/netmod/include/netmod.h +++ b/src/mpid/ch4/netmod/include/netmod.h @@ -31,9 +31,9 @@ typedef int (*MPIDI_NM_init_t) (int rank, int size, int appnum, int *tag_ub, MPI void **netmod_contexts); typedef int (*MPIDI_NM_finalize_t) (void); typedef int (*MPIDI_NM_progress_t) (void *netmod_context, int blocking); -typedef int (*MPIDI_NM_reg_hdr_handler_t) (int handler_id, - MPIDI_NM_am_origin_handler_fn origin_handler_fn, - MPIDI_NM_am_target_handler_fn target_handler_fn); +typedef int (*MPIDI_NM_am_reg_handler_t) (int handler_id, + MPIDI_NM_am_origin_handler_fn origin_handler_fn, + MPIDI_NM_am_target_handler_fn target_handler_fn); typedef int (*MPIDI_NM_comm_connect_t) (const char *port_name, MPIR_Info * info, int root, MPIR_Comm * comm, MPIR_Comm ** newcomm_ptr); typedef int (*MPIDI_NM_comm_disconnect_t) (MPIR_Comm * comm_ptr); @@ -41,34 +41,34 @@ typedef int (*MPIDI_NM_open_port_t) (MPIR_Info * info_ptr, char *port_name); typedef int (*MPIDI_NM_close_port_t) (const char *port_name); typedef int (*MPIDI_NM_comm_accept_t) (const char *port_name, MPIR_Info * info, int root, MPIR_Comm * comm, MPIR_Comm ** newcomm_ptr); -typedef int (*MPIDI_NM_send_am_hdr_t) (int rank, MPIR_Comm * comm, int handler_id, +typedef int (*MPIDI_NM_am_send_hdr_t) (int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, size_t am_hdr_sz, MPIR_Request * sreq, void *netmod_context); -typedef int (*MPIDI_NM_inject_am_hdr_t) (int rank, MPIR_Comm * comm, int handler_id, +typedef int (*MPIDI_NM_am_inject_hdr_t) (int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, size_t am_hdr_sz, void *netmod_context); -typedef int (*MPIDI_NM_send_am_t) (int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, +typedef int (*MPIDI_NM_am_send_t) (int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, size_t am_hdr_sz, const void *data, MPI_Count count, MPI_Datatype datatype, MPIR_Request * sreq, void *netmod_context); -typedef int (*MPIDI_NM_send_amv_t) (int rank, MPIR_Comm * comm, int handler_id, +typedef int (*MPIDI_NM_am_sendv_t) (int rank, MPIR_Comm * comm, int handler_id, struct iovec * am_hdrs, size_t iov_len, const void *data, MPI_Count count, MPI_Datatype datatype, MPIR_Request * sreq, void *netmod_context); -typedef int (*MPIDI_NM_send_amv_hdr_t) (int rank, MPIR_Comm * comm, int handler_id, +typedef int (*MPIDI_NM_am_sendv_hdr_t) (int rank, MPIR_Comm * comm, int handler_id, struct iovec * am_hdrs, size_t iov_len, MPIR_Request * sreq, void *netmod_context); -typedef int (*MPIDI_NM_send_am_hdr_reply_t) (MPIR_Context_id_t context_id, int src_rank, +typedef int (*MPIDI_NM_am_send_hdr_reply_t) (MPIR_Context_id_t context_id, int src_rank, int handler_id, const void *am_hdr, size_t am_hdr_sz, MPIR_Request * sreq); -typedef int (*MPIDI_NM_inject_am_hdr_reply_t) (MPIR_Context_id_t context_id, int src_rank, +typedef int (*MPIDI_NM_am_inject_hdr_reply_t) (MPIR_Context_id_t context_id, int src_rank, int handler_id, const void *am_hdr, size_t am_hdr_sz); -typedef int (*MPIDI_NM_send_am_reply_t) (MPIR_Context_id_t context_id, int src_rank, int handler_id, +typedef int (*MPIDI_NM_am_send_reply_t) (MPIR_Context_id_t context_id, int src_rank, int handler_id, const void *am_hdr, size_t am_hdr_sz, const void *data, MPI_Count count, MPI_Datatype datatype, MPIR_Request * sreq); -typedef int (*MPIDI_NM_send_amv_reply_t) (MPIR_Context_id_t context_id, int src_rank, +typedef int (*MPIDI_NM_am_sendv_reply_t) (MPIR_Context_id_t context_id, int src_rank, int handler_id, struct iovec * am_hdr, size_t iov_len, const void *data, MPI_Count count, MPI_Datatype datatype, MPIR_Request * sreq); @@ -378,16 +378,16 @@ typedef struct MPIDI_NM_funcs { MPIDI_NM_am_request_init_t am_request_init; MPIDI_NM_am_request_finalize_t am_request_finalize; /* Active Message Routines */ - MPIDI_NM_reg_hdr_handler_t reg_hdr_handler; - MPIDI_NM_send_am_hdr_t send_am_hdr; - MPIDI_NM_inject_am_hdr_t inject_am_hdr; - MPIDI_NM_send_am_t send_am; - MPIDI_NM_send_amv_t send_amv; - MPIDI_NM_send_amv_hdr_t send_amv_hdr; - MPIDI_NM_send_am_hdr_reply_t send_am_hdr_reply; - MPIDI_NM_inject_am_hdr_reply_t inject_am_hdr_reply; - MPIDI_NM_send_am_reply_t send_am_reply; - MPIDI_NM_send_amv_reply_t send_amv_reply; + MPIDI_NM_am_reg_handler_t am_reg_handler; + MPIDI_NM_am_send_hdr_t am_send_hdr; + MPIDI_NM_am_inject_hdr_t am_inject_hdr; + MPIDI_NM_am_send_t am_send; + MPIDI_NM_am_sendv_t am_sendv; + MPIDI_NM_am_sendv_hdr_t am_sendv_hdr; + MPIDI_NM_am_send_hdr_reply_t am_send_hdr_reply; + MPIDI_NM_am_inject_hdr_reply_t am_inject_hdr_reply; + MPIDI_NM_am_send_reply_t am_send_reply; + MPIDI_NM_am_sendv_reply_t am_sendv_reply; MPIDI_NM_am_hdr_max_sz_t am_hdr_max_sz; MPIDI_NM_am_recv_t am_recv; } MPIDI_NM_funcs_t; @@ -516,11 +516,11 @@ MPL_STATIC_INLINE_PREFIX int MPIDI_NM_init(int rank, int size, int appnum, int * MPL_STATIC_INLINE_PREFIX int MPIDI_NM_finalize(void) MPL_STATIC_INLINE_SUFFIX; MPL_STATIC_INLINE_PREFIX int MPIDI_NM_progress(void *netmod_context, int blocking) MPL_STATIC_INLINE_SUFFIX; -MPL_STATIC_INLINE_PREFIX int MPIDI_NM_reg_hdr_handler(int handler_id, - MPIDI_NM_am_origin_handler_fn - origin_handler_fn, - MPIDI_NM_am_target_handler_fn - target_handler_fn) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_reg_handler(int handler_id, + MPIDI_NM_am_origin_handler_fn + origin_handler_fn, + MPIDI_NM_am_target_handler_fn + target_handler_fn) MPL_STATIC_INLINE_SUFFIX; MPL_STATIC_INLINE_PREFIX int MPIDI_NM_comm_connect(const char *port_name, MPIR_Info * info, int root, MPIR_Comm * comm, MPIR_Comm ** @@ -534,43 +534,43 @@ MPL_STATIC_INLINE_PREFIX int MPIDI_NM_comm_accept(const char *port_name, MPIR_In int root, MPIR_Comm * comm, MPIR_Comm ** newcomm_ptr) MPL_STATIC_INLINE_SUFFIX; -MPL_STATIC_INLINE_PREFIX int MPIDI_NM_send_am_hdr(int rank, MPIR_Comm * comm, int handler_id, +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_send_hdr(int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, size_t am_hdr_sz, MPIR_Request * sreq, void *netmod_context) MPL_STATIC_INLINE_SUFFIX; -MPL_STATIC_INLINE_PREFIX int MPIDI_NM_inject_am_hdr(int rank, MPIR_Comm * comm, int handler_id, +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_inject_hdr(int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, size_t am_hdr_sz, void *netmod_context) MPL_STATIC_INLINE_SUFFIX; -MPL_STATIC_INLINE_PREFIX int MPIDI_NM_send_am(int rank, MPIR_Comm * comm, int handler_id, +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_send(int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, size_t am_hdr_sz, const void *data, MPI_Count count, MPI_Datatype datatype, MPIR_Request * sreq, void *netmod_context) MPL_STATIC_INLINE_SUFFIX; -MPL_STATIC_INLINE_PREFIX int MPIDI_NM_send_amv(int rank, MPIR_Comm * comm, int handler_id, +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_sendv(int rank, MPIR_Comm * comm, int handler_id, struct iovec *am_hdrs, size_t iov_len, const void *data, MPI_Count count, MPI_Datatype datatype, MPIR_Request * sreq, void *netmod_context) MPL_STATIC_INLINE_SUFFIX; -MPL_STATIC_INLINE_PREFIX int MPIDI_NM_send_amv_hdr(int rank, MPIR_Comm * comm, int handler_id, +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_sendv_hdr(int rank, MPIR_Comm * comm, int handler_id, struct iovec *am_hdrs, size_t iov_len, MPIR_Request * sreq, void *netmod_context) MPL_STATIC_INLINE_SUFFIX; -MPL_STATIC_INLINE_PREFIX int MPIDI_NM_send_am_hdr_reply(MPIR_Context_id_t context_id, +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_send_hdr_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, const void *am_hdr, size_t am_hdr_sz, MPIR_Request * sreq) MPL_STATIC_INLINE_SUFFIX; -MPL_STATIC_INLINE_PREFIX int MPIDI_NM_inject_am_hdr_reply(MPIR_Context_id_t context_id, +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_inject_hdr_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, const void *am_hdr, size_t am_hdr_sz) MPL_STATIC_INLINE_SUFFIX; -MPL_STATIC_INLINE_PREFIX int MPIDI_NM_send_am_reply(MPIR_Context_id_t context_id, int src_rank, +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_send_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, const void *am_hdr, size_t am_hdr_sz, const void *data, MPI_Count count, MPI_Datatype datatype, MPIR_Request * sreq) MPL_STATIC_INLINE_SUFFIX; -MPL_STATIC_INLINE_PREFIX int MPIDI_NM_send_amv_reply(MPIR_Context_id_t context_id, +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_sendv_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, struct iovec *am_hdr, size_t iov_len, const void *data, MPI_Count count, diff --git a/src/mpid/ch4/netmod/include/netmod_impl.h b/src/mpid/ch4/netmod/include/netmod_impl.h index 7b50bfb..d143059 100644 --- a/src/mpid/ch4/netmod/include/netmod_impl.h +++ b/src/mpid/ch4/netmod/include/netmod_impl.h @@ -34,13 +34,13 @@ MPL_STATIC_INLINE_PREFIX int MPIDI_NM_progress(void *netmod_context, int blockin return MPIDI_NM_func->progress(netmod_context, blocking); }; -MPL_STATIC_INLINE_PREFIX int MPIDI_NM_reg_hdr_handler(int handler_id, - MPIDI_NM_am_origin_handler_fn - origin_handler_fn, - MPIDI_NM_am_target_handler_fn - target_handler_fn) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_reg_handler(int handler_id, + MPIDI_NM_am_origin_handler_fn + origin_handler_fn, + MPIDI_NM_am_target_handler_fn + target_handler_fn) { - return MPIDI_NM_func->reg_hdr_handler(handler_id, origin_handler_fn, target_handler_fn); + return MPIDI_NM_func->am_reg_handler(handler_id, origin_handler_fn, target_handler_fn); }; MPL_STATIC_INLINE_PREFIX int MPIDI_NM_comm_connect(const char *port_name, MPIR_Info * info, @@ -72,82 +72,82 @@ MPL_STATIC_INLINE_PREFIX int MPIDI_NM_comm_accept(const char *port_name, MPIR_In return MPIDI_NM_func->comm_accept(port_name, info, root, comm, newcomm_ptr); }; -MPL_STATIC_INLINE_PREFIX int MPIDI_NM_send_am_hdr(int rank, MPIR_Comm * comm, int handler_id, +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_send_hdr(int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, size_t am_hdr_sz, MPIR_Request * sreq, void *netmod_context) { - return MPIDI_NM_func->send_am_hdr(rank, comm, handler_id, am_hdr, am_hdr_sz, sreq, + return MPIDI_NM_func->am_send_hdr(rank, comm, handler_id, am_hdr, am_hdr_sz, sreq, netmod_context); }; -MPL_STATIC_INLINE_PREFIX int MPIDI_NM_inject_am_hdr(int rank, MPIR_Comm * comm, int handler_id, +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_inject_hdr(int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, size_t am_hdr_sz, void *netmod_context) { - return MPIDI_NM_func->inject_am_hdr(rank, comm, handler_id, am_hdr, am_hdr_sz, netmod_context); + return MPIDI_NM_func->am_inject_hdr(rank, comm, handler_id, am_hdr, am_hdr_sz, netmod_context); }; -MPL_STATIC_INLINE_PREFIX int MPIDI_NM_send_am(int rank, MPIR_Comm * comm, int handler_id, +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_send(int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, size_t am_hdr_sz, const void *data, MPI_Count count, MPI_Datatype datatype, MPIR_Request * sreq, void *netmod_context) { - return MPIDI_NM_func->send_am(rank, comm, handler_id, am_hdr, am_hdr_sz, data, count, datatype, + return MPIDI_NM_func->am_send(rank, comm, handler_id, am_hdr, am_hdr_sz, data, count, datatype, sreq, netmod_context); }; -MPL_STATIC_INLINE_PREFIX int MPIDI_NM_send_amv(int rank, MPIR_Comm * comm, int handler_id, +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_sendv(int rank, MPIR_Comm * comm, int handler_id, struct iovec *am_hdrs, size_t iov_len, const void *data, MPI_Count count, MPI_Datatype datatype, MPIR_Request * sreq, void *netmod_context) { - return MPIDI_NM_func->send_amv(rank, comm, handler_id, am_hdrs, iov_len, data, count, datatype, + return MPIDI_NM_func->am_sendv(rank, comm, handler_id, am_hdrs, iov_len, data, count, datatype, sreq, netmod_context); }; -MPL_STATIC_INLINE_PREFIX int MPIDI_NM_send_amv_hdr(int rank, MPIR_Comm * comm, int handler_id, +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_sendv_hdr(int rank, MPIR_Comm * comm, int handler_id, struct iovec *am_hdrs, size_t iov_len, MPIR_Request * sreq, void *netmod_context) { - return MPIDI_NM_func->send_amv_hdr(rank, comm, handler_id, am_hdrs, iov_len, sreq, + return MPIDI_NM_func->am_sendv_hdr(rank, comm, handler_id, am_hdrs, iov_len, sreq, netmod_context); }; -MPL_STATIC_INLINE_PREFIX int MPIDI_NM_send_am_hdr_reply(MPIR_Context_id_t context_id, +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_send_hdr_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, const void *am_hdr, size_t am_hdr_sz, MPIR_Request * sreq) { - return MPIDI_NM_func->send_am_hdr_reply(context_id, src_rank, handler_id, am_hdr, am_hdr_sz, + return MPIDI_NM_func->am_send_hdr_reply(context_id, src_rank, handler_id, am_hdr, am_hdr_sz, sreq); }; -MPL_STATIC_INLINE_PREFIX int MPIDI_NM_inject_am_hdr_reply(MPIR_Context_id_t context_id, +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_inject_hdr_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, const void *am_hdr, size_t am_hdr_sz) { - return MPIDI_NM_func->inject_am_hdr_reply(context_id, src_rank, handler_id, am_hdr, am_hdr_sz); + return MPIDI_NM_func->am_inject_hdr_reply(context_id, src_rank, handler_id, am_hdr, am_hdr_sz); }; -MPL_STATIC_INLINE_PREFIX int MPIDI_NM_send_am_reply(MPIR_Context_id_t context_id, int src_rank, +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_send_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, const void *am_hdr, size_t am_hdr_sz, const void *data, MPI_Count count, MPI_Datatype datatype, MPIR_Request * sreq) { - return MPIDI_NM_func->send_am_reply(context_id, src_rank, handler_id, am_hdr, am_hdr_sz, data, + return MPIDI_NM_func->am_send_reply(context_id, src_rank, handler_id, am_hdr, am_hdr_sz, data, count, datatype, sreq); }; -MPL_STATIC_INLINE_PREFIX int MPIDI_NM_send_amv_reply(MPIR_Context_id_t context_id, +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_sendv_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, struct iovec *am_hdr, size_t iov_len, const void *data, MPI_Count count, MPI_Datatype datatype, MPIR_Request * sreq) { - return MPIDI_NM_func->send_amv_reply(context_id, src_rank, handler_id, am_hdr, iov_len, data, + return MPIDI_NM_func->am_sendv_reply(context_id, src_rank, handler_id, am_hdr, iov_len, data, count, datatype, sreq); }; diff --git a/src/mpid/ch4/netmod/ofi/func_table.c b/src/mpid/ch4/netmod/ofi/func_table.c index dff22ff..c96f47d 100644 --- a/src/mpid/ch4/netmod/ofi/func_table.c +++ b/src/mpid/ch4/netmod/ofi/func_table.c @@ -31,16 +31,16 @@ MPIDI_NM_funcs_t MPIDI_NM_ofi_funcs = { MPIDI_NM_comm_free_hook, MPIDI_NM_am_request_init, MPIDI_NM_am_request_finalize, - MPIDI_NM_reg_hdr_handler, - MPIDI_NM_send_am_hdr, - MPIDI_NM_inject_am_hdr, - MPIDI_NM_send_am, - MPIDI_NM_send_amv, - MPIDI_NM_send_amv_hdr, - MPIDI_NM_send_am_hdr_reply, - MPIDI_NM_inject_am_hdr_reply, - MPIDI_NM_send_am_reply, - MPIDI_NM_send_amv_reply, + MPIDI_NM_am_reg_handler, + MPIDI_NM_am_send_hdr, + MPIDI_NM_am_inject_hdr, + MPIDI_NM_am_send, + MPIDI_NM_am_sendv, + MPIDI_NM_am_sendv_hdr, + MPIDI_NM_am_send_hdr_reply, + MPIDI_NM_am_inject_hdr_reply, + MPIDI_NM_am_send_reply, + MPIDI_NM_am_sendv_reply, MPIDI_NM_am_hdr_max_sz, MPIDI_NM_am_recv }; diff --git a/src/mpid/ch4/netmod/ofi/ofi_am.h b/src/mpid/ch4/netmod/ofi/ofi_am.h index e3b6287..f624d28 100644 --- a/src/mpid/ch4/netmod/ofi/ofi_am.h +++ b/src/mpid/ch4/netmod/ofi/ofi_am.h @@ -27,12 +27,12 @@ static inline void MPIDI_NM_am_request_finalize(MPIR_Request * req) } #undef FUNCNAME -#define FUNCNAME MPIDI_NM_reg_hdr_handler +#define FUNCNAME MPIDI_NM_am_reg_handler #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_NM_reg_hdr_handler(int handler_id, - MPIDI_NM_am_origin_handler_fn origin_handler_fn, - MPIDI_NM_am_target_handler_fn target_handler_fn) +static inline int MPIDI_NM_am_reg_handler(int handler_id, + MPIDI_NM_am_origin_handler_fn origin_handler_fn, + MPIDI_NM_am_target_handler_fn target_handler_fn) { int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_REG_HDR_HANDLER); @@ -53,10 +53,10 @@ static inline int MPIDI_NM_reg_hdr_handler(int handler_id, } #undef FUNCNAME -#define FUNCNAME MPIDI_NM_send_am_hdr +#define FUNCNAME MPIDI_NM_am_send_hdr #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_NM_send_am_hdr(int rank, +static inline int MPIDI_NM_am_send_hdr(int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, @@ -66,17 +66,17 @@ static inline int MPIDI_NM_send_am_hdr(int rank, MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_SEND_AM_HDR); MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_SEND_AM_HDR); - mpi_errno = MPIDI_OFI_do_send_am_header(rank, comm, handler_id, am_hdr, am_hdr_sz, sreq, FALSE); + mpi_errno = MPIDI_OFI_do_am_send_header(rank, comm, handler_id, am_hdr, am_hdr_sz, sreq, FALSE); MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_SEND_AM_HDR); return mpi_errno; } #undef FUNCNAME -#define FUNCNAME MPIDI_NM_send_am +#define FUNCNAME MPIDI_NM_am_send #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_NM_send_am(int rank, +static inline int MPIDI_NM_am_send(int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, @@ -88,7 +88,7 @@ static inline int MPIDI_NM_send_am(int rank, int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_SEND_AM); MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_SEND_AM); - mpi_errno = MPIDI_OFI_do_send_am(rank, comm, handler_id, + mpi_errno = MPIDI_OFI_do_am_send(rank, comm, handler_id, am_hdr, am_hdr_sz, data, count, datatype, sreq, FALSE); MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_SEND_AM); @@ -96,10 +96,10 @@ static inline int MPIDI_NM_send_am(int rank, } #undef FUNCNAME -#define FUNCNAME MPIDI_NM_send_amv +#define FUNCNAME MPIDI_NM_am_sendv #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_NM_send_amv(int rank, +static inline int MPIDI_NM_am_sendv(int rank, MPIR_Comm * comm, int handler_id, struct iovec *am_hdr, @@ -137,7 +137,7 @@ static inline int MPIDI_NM_send_amv(int rank, am_hdr_sz += am_hdr[i].iov_len; } - mpi_errno = MPIDI_NM_send_am(rank, comm, handler_id, am_hdr_buf, am_hdr_sz, + mpi_errno = MPIDI_NM_am_send(rank, comm, handler_id, am_hdr_buf, am_hdr_sz, data, count, datatype, sreq, netmod_context); if (is_allocated) @@ -150,10 +150,10 @@ static inline int MPIDI_NM_send_amv(int rank, } #undef FUNCNAME -#define FUNCNAME MPIDI_NM_send_amv_hdr +#define FUNCNAME MPIDI_NM_am_sendv_hdr #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_NM_send_amv_hdr(int rank, +static inline int MPIDI_NM_am_sendv_hdr(int rank, MPIR_Comm * comm, int handler_id, struct iovec *am_hdr, @@ -188,7 +188,7 @@ static inline int MPIDI_NM_send_amv_hdr(int rank, am_hdr_sz += am_hdr[i].iov_len; } - mpi_errno = MPIDI_NM_send_am_hdr(rank, comm, handler_id, am_hdr_buf, am_hdr_sz, + mpi_errno = MPIDI_NM_am_send_hdr(rank, comm, handler_id, am_hdr_buf, am_hdr_sz, sreq, netmod_context); if (is_allocated) @@ -201,10 +201,10 @@ static inline int MPIDI_NM_send_amv_hdr(int rank, } #undef FUNCNAME -#define FUNCNAME MPIDI_NM_send_am_hdr_reply +#define FUNCNAME MPIDI_NM_am_send_hdr_reply #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_NM_send_am_hdr_reply(MPIR_Context_id_t context_id, +static inline int MPIDI_NM_am_send_hdr_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, const void *am_hdr, @@ -213,7 +213,7 @@ static inline int MPIDI_NM_send_am_hdr_reply(MPIR_Context_id_t context_id, int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_SEND_AM_HDR_REPLY); MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_SEND_AM_HDR_REPLY); - mpi_errno = MPIDI_OFI_do_send_am_header(src_rank, + mpi_errno = MPIDI_OFI_do_am_send_header(src_rank, MPIDI_CH4U_context_id_to_comm(context_id), handler_id, am_hdr, am_hdr_sz, sreq, TRUE); MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_SEND_AM_HDR_REPLY); @@ -221,10 +221,10 @@ static inline int MPIDI_NM_send_am_hdr_reply(MPIR_Context_id_t context_id, } #undef FUNCNAME -#define FUNCNAME MPIDI_NM_send_am_reply +#define FUNCNAME MPIDI_NM_am_send_reply #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_NM_send_am_reply(MPIR_Context_id_t context_id, +static inline int MPIDI_NM_am_send_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, const void *am_hdr, @@ -236,7 +236,7 @@ static inline int MPIDI_NM_send_am_reply(MPIR_Context_id_t context_id, int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_SEND_AM_REPLY); MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_SEND_AM_REPLY); - mpi_errno = MPIDI_OFI_do_send_am(src_rank, + mpi_errno = MPIDI_OFI_do_am_send(src_rank, MPIDI_CH4U_context_id_to_comm(context_id), handler_id, am_hdr, am_hdr_sz, data, count, datatype, sreq, TRUE); @@ -244,7 +244,7 @@ static inline int MPIDI_NM_send_am_reply(MPIR_Context_id_t context_id, return mpi_errno; } -static inline int MPIDI_NM_send_amv_reply(MPIR_Context_id_t context_id, +static inline int MPIDI_NM_am_sendv_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, struct iovec *am_hdr, @@ -282,7 +282,7 @@ static inline int MPIDI_NM_send_amv_reply(MPIR_Context_id_t context_id, am_hdr_sz += am_hdr[i].iov_len; } - mpi_errno = MPIDI_NM_send_am_reply(context_id, src_rank, handler_id, am_hdr_buf, am_hdr_sz, + mpi_errno = MPIDI_NM_am_send_reply(context_id, src_rank, handler_id, am_hdr_buf, am_hdr_sz, data, count, datatype, sreq); if (is_allocated) @@ -305,7 +305,7 @@ static inline size_t MPIDI_NM_am_hdr_max_sz(void) return MPL_MIN(max_shortsend, max_representable); } -static inline int MPIDI_NM_inject_am_hdr(int rank, +static inline int MPIDI_NM_am_inject_hdr(int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, size_t am_hdr_sz, void *netmod_context) @@ -327,7 +327,7 @@ static inline int MPIDI_NM_inject_am_hdr(int rank, goto fn_exit; } -static inline int MPIDI_NM_inject_am_hdr_reply(MPIR_Context_id_t context_id, +static inline int MPIDI_NM_am_inject_hdr_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, const void *am_hdr, size_t am_hdr_sz) { @@ -360,7 +360,7 @@ static inline int MPIDI_NM_am_recv(MPIR_Request * req) msg.sreq_ptr = (MPIDI_CH4U_REQUEST(req, req->rreq.peer_req_ptr)); msg.rreq_ptr = (uint64_t) req; MPIR_Assert((void *) msg.sreq_ptr != NULL); - mpi_errno = MPIDI_NM_inject_am_hdr_reply(MPIDI_CH4U_get_context(MPIDI_CH4U_REQUEST(req, tag)), + mpi_errno = MPIDI_NM_am_inject_hdr_reply(MPIDI_CH4U_get_context(MPIDI_CH4U_REQUEST(req, tag)), MPIDI_CH4U_REQUEST(req, src_rank), MPIDI_CH4U_SEND_LONG_ACK, &msg, sizeof(msg)); if (mpi_errno) diff --git a/src/mpid/ch4/netmod/ofi/ofi_am_impl.h b/src/mpid/ch4/netmod/ofi/ofi_am_impl.h index 1a9cba6..058c936 100644 --- a/src/mpid/ch4/netmod/ofi/ofi_am_impl.h +++ b/src/mpid/ch4/netmod/ofi/ofi_am_impl.h @@ -20,10 +20,10 @@ static inline int MPIDI_OFI_progress_do_queue(void *netmod_context); * When calling OFI function MPIDI_OFI_THREAD_FI_MUTEX must be held. * When being called from the MPI layer (app), we must grab the lock. - This is the case for regular (non-reply) functions such as send_am. + This is the case for regular (non-reply) functions such as am_send. * When being called from callback function or progress engine, we must not grab the lock because the progress engine is already holding the lock. - This is the case for reply functions such as send_am_reply. + This is the case for reply functions such as am_send_reply. */ #define MPIDI_OFI_CALL_RETRY_AM(FUNC,LOCK,STR) \ do { \ @@ -192,10 +192,10 @@ static inline int MPIDI_OFI_progress_do_queue(void *netmod_context) } #undef FUNCNAME -#define FUNCNAME MPIDI_OFI_do_send_am_header +#define FUNCNAME MPIDI_OFI_do_am_send_header #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_OFI_do_send_am_header(int rank, +static inline int MPIDI_OFI_do_am_send_header(int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, @@ -246,10 +246,10 @@ static inline int MPIDI_OFI_do_send_am_header(int rank, } #undef FUNCNAME -#define FUNCNAME MPIDI_OFI_send_am_long +#define FUNCNAME MPIDI_OFI_am_send_long #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_OFI_send_am_long(int rank, +static inline int MPIDI_OFI_am_send_long(int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, @@ -337,10 +337,10 @@ static inline int MPIDI_OFI_send_am_long(int rank, } #undef FUNCNAME -#define FUNCNAME MPIDI_OFI_send_am_short +#define FUNCNAME MPIDI_OFI_am_send_short #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_OFI_send_am_short(int rank, +static inline int MPIDI_OFI_am_send_short(int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, @@ -388,10 +388,10 @@ static inline int MPIDI_OFI_send_am_short(int rank, } #undef FUNCNAME -#define FUNCNAME MPIDI_OFI_do_send_am +#define FUNCNAME MPIDI_OFI_do_am_send #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_OFI_do_send_am(int rank, +static inline int MPIDI_OFI_do_am_send(int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, @@ -426,7 +426,7 @@ static inline int MPIDI_OFI_do_send_am(int rank, MPIDI_CH4U_REQUEST(sreq, req->lreq).datatype = datatype; MPIDI_CH4U_REQUEST(sreq, req->lreq).msg_tag = lreq_hdr.hdr.msg_tag; MPIDI_CH4U_REQUEST(sreq, src_rank) = rank; - mpi_errno = MPIDI_NM_inject_am_hdr(rank, comm, MPIDI_CH4U_SEND_LONG_REQ, + mpi_errno = MPIDI_NM_am_inject_hdr(rank, comm, MPIDI_CH4U_SEND_LONG_REQ, &lreq_hdr, sizeof(lreq_hdr), NULL); if (mpi_errno) MPIR_ERR_POP(mpi_errno); @@ -461,12 +461,12 @@ static inline int MPIDI_OFI_do_send_am(int rank, if (am_hdr_sz + data_sz + sizeof(MPIDI_OFI_am_header_t) <= MPIDI_OFI_DEFAULT_SHORT_SEND_SIZE) { mpi_errno = - MPIDI_OFI_send_am_short(rank, comm, handler_id, MPIDI_OFI_AMREQUEST_HDR(sreq, am_hdr), + MPIDI_OFI_am_send_short(rank, comm, handler_id, MPIDI_OFI_AMREQUEST_HDR(sreq, am_hdr), am_hdr_sz, send_buf, data_sz, sreq, need_lock); } else { mpi_errno = - MPIDI_OFI_send_am_long(rank, comm, handler_id, MPIDI_OFI_AMREQUEST_HDR(sreq, am_hdr), + MPIDI_OFI_am_send_long(rank, comm, handler_id, MPIDI_OFI_AMREQUEST_HDR(sreq, am_hdr), am_hdr_sz, send_buf, data_sz, sreq, need_lock); } if (mpi_errno) diff --git a/src/mpid/ch4/netmod/ofi/ofi_init.h b/src/mpid/ch4/netmod/ofi/ofi_init.h index 7b12704..f6655df 100644 --- a/src/mpid/ch4/netmod/ofi/ofi_init.h +++ b/src/mpid/ch4/netmod/ofi/ofi_init.h @@ -390,7 +390,7 @@ static inline int MPIDI_OFI_init_generic(int rank, /* ---------------------------------- */ if (do_am) { /* Maximum possible message size for short message send (=eager send) - * See MPIDI_OFI_do_send_am for short/long switching logic */ + * See MPIDI_OFI_do_am_send for short/long switching logic */ MPIR_Assert(MPIDI_OFI_DEFAULT_SHORT_SEND_SIZE <= MPIDI_Global.max_send); MPIDI_Global.am_buf_pool = MPIDI_CH4U_create_buf_pool(MPIDI_OFI_BUF_POOL_NUM, MPIDI_OFI_BUF_POOL_SIZE); diff --git a/src/mpid/ch4/netmod/portals4/func_table.c b/src/mpid/ch4/netmod/portals4/func_table.c index 4511394..c5b92db 100644 --- a/src/mpid/ch4/netmod/portals4/func_table.c +++ b/src/mpid/ch4/netmod/portals4/func_table.c @@ -31,16 +31,16 @@ MPIDI_NM_funcs_t MPIDI_NM_portals4_funcs = { MPIDI_NM_comm_free_hook, MPIDI_NM_am_request_init, MPIDI_NM_am_request_finalize, - MPIDI_NM_reg_hdr_handler, - MPIDI_NM_send_am_hdr, - MPIDI_NM_inject_am_hdr, - MPIDI_NM_send_am, - MPIDI_NM_send_amv, - MPIDI_NM_send_amv_hdr, - MPIDI_NM_send_am_hdr_reply, - MPIDI_NM_inject_am_hdr_reply, - MPIDI_NM_send_am_reply, - MPIDI_NM_send_amv_reply, + MPIDI_NM_am_reg_handler, + MPIDI_NM_am_send_hdr, + MPIDI_NM_am_inject_hdr, + MPIDI_NM_am_send, + MPIDI_NM_am_sendv, + MPIDI_NM_am_sendv_hdr, + MPIDI_NM_am_send_hdr_reply, + MPIDI_NM_am_inject_hdr_reply, + MPIDI_NM_am_send_reply, + MPIDI_NM_am_sendv_reply, MPIDI_NM_am_hdr_max_sz, MPIDI_NM_am_recv, }; diff --git a/src/mpid/ch4/netmod/portals4/ptl_am.h b/src/mpid/ch4/netmod/portals4/ptl_am.h index de0e607..c2a560c 100644 --- a/src/mpid/ch4/netmod/portals4/ptl_am.h +++ b/src/mpid/ch4/netmod/portals4/ptl_am.h @@ -14,12 +14,12 @@ #include "ptl_impl.h" #undef FUNCNAME -#define FUNCNAME MPIDI_NM_reg_hdr_handler +#define FUNCNAME MPIDI_NM_am_reg_handler #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_NM_reg_hdr_handler(int handler_id, - MPIDI_NM_am_origin_handler_fn origin_handler_fn, - MPIDI_NM_am_target_handler_fn target_handler_fn) +static inline int MPIDI_NM_am_reg_handler(int handler_id, + MPIDI_NM_am_origin_handler_fn origin_handler_fn, + MPIDI_NM_am_target_handler_fn target_handler_fn) { int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_REG_HDR_HANDLER); @@ -34,7 +34,7 @@ static inline int MPIDI_NM_reg_hdr_handler(int handler_id, goto fn_exit; } -static inline int MPIDI_NM_send_am_hdr(int rank, +static inline int MPIDI_NM_am_send_hdr(int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, @@ -69,7 +69,7 @@ static inline int MPIDI_NM_send_am_hdr(int rank, goto fn_exit; } -static inline int MPIDI_NM_send_am(int rank, +static inline int MPIDI_NM_am_send(int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, @@ -148,7 +148,7 @@ static inline int MPIDI_NM_send_am(int rank, goto fn_exit; } -static inline int MPIDI_NM_send_amv(int rank, +static inline int MPIDI_NM_am_sendv(int rank, MPIR_Comm * comm, int handler_id, struct iovec *am_hdr, @@ -162,7 +162,7 @@ static inline int MPIDI_NM_send_amv(int rank, return MPI_SUCCESS; } -static inline int MPIDI_NM_send_amv_hdr(int rank, +static inline int MPIDI_NM_am_sendv_hdr(int rank, MPIR_Comm * comm, int handler_id, struct iovec *am_hdr, @@ -172,7 +172,7 @@ static inline int MPIDI_NM_send_amv_hdr(int rank, return MPI_SUCCESS; } -static inline int MPIDI_NM_send_am_hdr_reply(MPIR_Context_id_t context_id, +static inline int MPIDI_NM_am_send_hdr_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, const void *am_hdr, @@ -182,7 +182,7 @@ static inline int MPIDI_NM_send_am_hdr_reply(MPIR_Context_id_t context_id, return MPI_SUCCESS; } -static inline int MPIDI_NM_send_am_reply(MPIR_Context_id_t context_id, +static inline int MPIDI_NM_am_send_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, const void *am_hdr, @@ -264,7 +264,7 @@ static inline int MPIDI_NM_send_am_reply(MPIR_Context_id_t context_id, goto fn_exit; } -static inline int MPIDI_NM_send_amv_reply(MPIR_Context_id_t context_id, +static inline int MPIDI_NM_am_sendv_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, struct iovec *am_hdr, @@ -283,7 +283,7 @@ static inline size_t MPIDI_NM_am_hdr_max_sz(void) return 0; } -static inline int MPIDI_NM_inject_am_hdr(int rank, +static inline int MPIDI_NM_am_inject_hdr(int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, size_t am_hdr_sz, void *netmod_context) @@ -318,7 +318,7 @@ static inline int MPIDI_NM_inject_am_hdr(int rank, goto fn_exit; } -static inline int MPIDI_NM_inject_am_hdr_reply(MPIR_Context_id_t context_id, +static inline int MPIDI_NM_am_inject_hdr_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, const void *am_hdr, size_t am_hdr_sz) { diff --git a/src/mpid/ch4/netmod/stubnm/globals.c b/src/mpid/ch4/netmod/stubnm/globals.c index b886803..7ecbbb2 100644 --- a/src/mpid/ch4/netmod/stubnm/globals.c +++ b/src/mpid/ch4/netmod/stubnm/globals.c @@ -31,16 +31,16 @@ MPIDI_NM_funcs_t MPIDI_NM_stubnm_funcs = { MPIDI_NM_comm_free_hook, MPIDI_NM_am_request_init, MPIDI_NM_am_request_finalize, - MPIDI_NM_reg_hdr_handler, - MPIDI_NM_send_am_hdr, - MPIDI_NM_inject_am_hdr, - MPIDI_NM_send_am, - MPIDI_NM_send_amv, - MPIDI_NM_send_amv_hdr, - MPIDI_NM_send_am_hdr_reply, - MPIDI_NM_inject_am_hdr_reply, - MPIDI_NM_send_am_reply, - MPIDI_NM_send_amv_reply, + MPIDI_NM_am_reg_handler, + MPIDI_NM_am_send_hdr, + MPIDI_NM_am_inject_hdr, + MPIDI_NM_am_send, + MPIDI_NM_am_sendv, + MPIDI_NM_am_sendv_hdr, + MPIDI_NM_am_send_hdr_reply, + MPIDI_NM_am_inject_hdr_reply, + MPIDI_NM_am_send_reply, + MPIDI_NM_am_sendv_reply, MPIDI_NM_am_hdr_max_sz, MPIDI_NM_am_recv, }; diff --git a/src/mpid/ch4/netmod/stubnm/stubnm_am.h b/src/mpid/ch4/netmod/stubnm/stubnm_am.h index e3bcda3..074c696 100644 --- a/src/mpid/ch4/netmod/stubnm/stubnm_am.h +++ b/src/mpid/ch4/netmod/stubnm/stubnm_am.h @@ -13,15 +13,15 @@ #include "stubnm_impl.h" -static inline int MPIDI_NM_reg_hdr_handler(int handler_id, - MPIDI_NM_am_origin_handler_fn origin_handler_fn, - MPIDI_NM_am_target_handler_fn target_handler_fn) +static inline int MPIDI_NM_am_reg_handler(int handler_id, + MPIDI_NM_am_origin_handler_fn origin_handler_fn, + MPIDI_NM_am_target_handler_fn target_handler_fn) { MPIR_Assert(0); return MPI_SUCCESS; } -static inline int MPIDI_NM_send_am_hdr(int rank, +static inline int MPIDI_NM_am_send_hdr(int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, @@ -31,7 +31,7 @@ static inline int MPIDI_NM_send_am_hdr(int rank, return MPI_SUCCESS; } -static inline int MPIDI_NM_send_am(int rank, +static inline int MPIDI_NM_am_send(int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, @@ -44,7 +44,7 @@ static inline int MPIDI_NM_send_am(int rank, return MPI_SUCCESS; } -static inline int MPIDI_NM_send_amv(int rank, +static inline int MPIDI_NM_am_sendv(int rank, MPIR_Comm * comm, int handler_id, struct iovec *am_hdr, @@ -58,7 +58,7 @@ static inline int MPIDI_NM_send_amv(int rank, return MPI_SUCCESS; } -static inline int MPIDI_NM_send_amv_hdr(int rank, +static inline int MPIDI_NM_am_sendv_hdr(int rank, MPIR_Comm * comm, int handler_id, struct iovec *am_hdr, @@ -68,7 +68,7 @@ static inline int MPIDI_NM_send_amv_hdr(int rank, return MPI_SUCCESS; } -static inline int MPIDI_NM_send_am_hdr_reply(MPIR_Context_id_t context_id, int src_rank, +static inline int MPIDI_NM_am_send_hdr_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, const void *am_hdr, size_t am_hdr_sz, MPIR_Request * sreq) @@ -77,7 +77,7 @@ static inline int MPIDI_NM_send_am_hdr_reply(MPIR_Context_id_t context_id, int s return MPI_SUCCESS; } -static inline int MPIDI_NM_send_am_reply(MPIR_Context_id_t context_id, int src_rank, +static inline int MPIDI_NM_am_send_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, const void *am_hdr, size_t am_hdr_sz, @@ -89,7 +89,7 @@ static inline int MPIDI_NM_send_am_reply(MPIR_Context_id_t context_id, int src_r return MPI_SUCCESS; } -static inline int MPIDI_NM_send_amv_reply(MPIR_Context_id_t context_id, int src_rank, +static inline int MPIDI_NM_am_sendv_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, struct iovec *am_hdr, size_t iov_len, @@ -107,7 +107,7 @@ static inline size_t MPIDI_NM_am_hdr_max_sz(void) return 0; } -static inline int MPIDI_NM_inject_am_hdr(int rank, +static inline int MPIDI_NM_am_inject_hdr(int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, size_t am_hdr_sz, void *netmod_context) @@ -116,7 +116,7 @@ static inline int MPIDI_NM_inject_am_hdr(int rank, return MPI_SUCCESS; } -static inline int MPIDI_NM_inject_am_hdr_reply(MPIR_Context_id_t context_id, int src_rank, +static inline int MPIDI_NM_am_inject_hdr_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, const void *am_hdr, size_t am_hdr_sz) { MPIR_Assert(0); diff --git a/src/mpid/ch4/netmod/ucx/func_table.c b/src/mpid/ch4/netmod/ucx/func_table.c index 8bd03fb..f4a03fe 100644 --- a/src/mpid/ch4/netmod/ucx/func_table.c +++ b/src/mpid/ch4/netmod/ucx/func_table.c @@ -29,16 +29,16 @@ MPIDI_NM_funcs_t MPIDI_NM_ucx_funcs = { MPIDI_NM_comm_free_hook, MPIDI_NM_am_request_init, MPIDI_NM_am_request_finalize, - MPIDI_NM_reg_hdr_handler, - MPIDI_NM_send_am_hdr, - MPIDI_NM_inject_am_hdr, - MPIDI_NM_send_am, - MPIDI_NM_send_amv, - MPIDI_NM_send_amv_hdr, - MPIDI_NM_send_am_hdr_reply, - MPIDI_NM_inject_am_hdr_reply, - MPIDI_NM_send_am_reply, - MPIDI_NM_send_amv_reply, + MPIDI_NM_am_reg_handler, + MPIDI_NM_am_send_hdr, + MPIDI_NM_am_inject_hdr, + MPIDI_NM_am_send, + MPIDI_NM_am_sendv, + MPIDI_NM_am_sendv_hdr, + MPIDI_NM_am_send_hdr_reply, + MPIDI_NM_am_inject_hdr_reply, + MPIDI_NM_am_send_reply, + MPIDI_NM_am_sendv_reply, MPIDI_NM_am_hdr_max_sz, MPIDI_NM_am_recv }; diff --git a/src/mpid/ch4/netmod/ucx/ucx_am.h b/src/mpid/ch4/netmod/ucx/ucx_am.h index 1a4c595..64a0051 100644 --- a/src/mpid/ch4/netmod/ucx/ucx_am.h +++ b/src/mpid/ch4/netmod/ucx/ucx_am.h @@ -12,12 +12,12 @@ #include "ucx_impl.h" #undef FUNCNAME -#define FUNCNAME MPIDI_NM_reg_hdr_handler +#define FUNCNAME MPIDI_NM_am_reg_handler #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_NM_reg_hdr_handler(int handler_id, - MPIDI_NM_am_origin_handler_fn origin_handler_fn, - MPIDI_NM_am_target_handler_fn target_handler_fn) +static inline int MPIDI_NM_am_reg_handler(int handler_id, + MPIDI_NM_am_origin_handler_fn origin_handler_fn, + MPIDI_NM_am_target_handler_fn target_handler_fn) { int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_REG_HDR_HANDLER); @@ -33,7 +33,7 @@ static inline int MPIDI_NM_reg_hdr_handler(int handler_id, } -static inline void MPIDI_UCX_send_am_callback(void *request, ucs_status_t status) +static inline void MPIDI_UCX_am_send_callback(void *request, ucs_status_t status) { MPIDI_UCX_ucp_request_t *ucp_request = (MPIDI_UCX_ucp_request_t *) request; @@ -76,10 +76,10 @@ static inline void MPIDI_UCX_inject_am_callback(void *request, ucs_status_t stat } #undef FUNCNAME -#define FUNCNAME MPIDI_NM_send_am_hdr +#define FUNCNAME MPIDI_NM_am_send_hdr #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_NM_send_am_hdr(int rank, +static inline int MPIDI_NM_am_send_hdr(int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, @@ -110,7 +110,7 @@ static inline int MPIDI_NM_send_am_hdr(int rank, ucp_request = (MPIDI_UCX_ucp_request_t *) ucp_tag_send_nb(ep, send_buf, am_hdr_sz + sizeof(ucx_hdr), ucp_dt_make_contig(1), ucx_tag, - &MPIDI_UCX_send_am_callback); + &MPIDI_UCX_am_send_callback); MPIDI_CH4_UCX_REQUEST(ucp_request, tag_send_nb); /* send is done. free all resources and complete the request */ @@ -144,10 +144,10 @@ static inline int MPIDI_NM_send_am_hdr(int rank, } #undef FUNCNAME -#define FUNCNAME MPIDI_NM_send_am +#define FUNCNAME MPIDI_NM_am_send #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_NM_send_am(int rank, +static inline int MPIDI_NM_am_send(int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, @@ -184,7 +184,7 @@ static inline int MPIDI_NM_send_am(int rank, MPIDI_CH4U_REQUEST(sreq, req->lreq).datatype = datatype; MPIDI_CH4U_REQUEST(sreq, req->lreq).msg_tag = lreq_hdr.hdr.msg_tag; MPIDI_CH4U_REQUEST(sreq, src_rank) = rank; - mpi_errno = MPIDI_NM_inject_am_hdr(rank, comm, MPIDI_CH4U_SEND_LONG_REQ, + mpi_errno = MPIDI_NM_am_inject_hdr(rank, comm, MPIDI_CH4U_SEND_LONG_REQ, &lreq_hdr, sizeof(lreq_hdr), NULL); if (mpi_errno) MPIR_ERR_POP(mpi_errno); @@ -226,7 +226,7 @@ static inline int MPIDI_NM_send_am(int rank, ucp_request = (MPIDI_UCX_ucp_request_t *) ucp_tag_send_nb(ep, send_buf, data_sz + am_hdr_sz + sizeof(ucx_hdr), ucp_dt_make_contig(1), ucx_tag, - &MPIDI_UCX_send_am_callback); + &MPIDI_UCX_am_send_callback); MPIDI_CH4_UCX_REQUEST(ucp_request, tag_send_nb); /* send is done. free all resources and complete the request */ if (ucp_request == NULL) { @@ -259,10 +259,10 @@ static inline int MPIDI_NM_send_am(int rank, } #undef FUNCNAME -#define FUNCNAME MPIDI_NM_send_amv_hdr +#define FUNCNAME MPIDI_NM_am_sendv_hdr #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_NM_send_amv_hdr(int rank, +static inline int MPIDI_NM_am_sendv_hdr(int rank, MPIR_Comm * comm, int handler_id, struct iovec *am_hdr, @@ -288,7 +288,7 @@ static inline int MPIDI_NM_send_amv_hdr(int rank, am_hdr_sz += am_hdr[i].iov_len; } - mpi_errno = MPIDI_NM_send_am_hdr(rank, comm, handler_id, am_hdr_buf, am_hdr_sz, + mpi_errno = MPIDI_NM_am_send_hdr(rank, comm, handler_id, am_hdr_buf, am_hdr_sz, sreq, netmod_context); MPL_free(am_hdr_buf); MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_UCX_SEND_AMV_HDR); @@ -296,10 +296,10 @@ static inline int MPIDI_NM_send_amv_hdr(int rank, } #undef FUNCNAME -#define FUNCNAME MPIDI_NM_send_amv +#define FUNCNAME MPIDI_NM_am_sendv #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_NM_send_amv(int rank, +static inline int MPIDI_NM_am_sendv(int rank, MPIR_Comm * comm, int handler_id, struct iovec *am_hdr, @@ -329,7 +329,7 @@ static inline int MPIDI_NM_send_amv(int rank, am_hdr_sz += am_hdr[i].iov_len; } - mpi_errno = MPIDI_NM_send_am(rank, comm, handler_id, am_hdr_buf, am_hdr_sz, + mpi_errno = MPIDI_NM_am_send(rank, comm, handler_id, am_hdr_buf, am_hdr_sz, data, count, datatype, sreq, netmod_context); MPL_free(am_hdr_buf); @@ -339,25 +339,25 @@ static inline int MPIDI_NM_send_amv(int rank, #undef FUNCNAME -#define FUNCNAME MPIDI_NM_send_am_hdr_reply +#define FUNCNAME MPIDI_NM_am_send_hdr_reply #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_NM_send_am_hdr_reply(MPIR_Context_id_t context_id, +static inline int MPIDI_NM_am_send_hdr_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, const void *am_hdr, size_t am_hdr_sz, MPIR_Request * sreq) { - return MPIDI_NM_send_am_hdr(src_rank, MPIDI_CH4U_context_id_to_comm(context_id), handler_id, + return MPIDI_NM_am_send_hdr(src_rank, MPIDI_CH4U_context_id_to_comm(context_id), handler_id, am_hdr, am_hdr_sz, sreq, NULL); } #undef FUNCNAME -#define FUNCNAME MPIDI_NM_send_am_reply +#define FUNCNAME MPIDI_NM_am_send_reply #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_NM_send_am_reply(MPIR_Context_id_t context_id, +static inline int MPIDI_NM_am_send_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, const void *am_hdr, @@ -401,7 +401,7 @@ static inline int MPIDI_NM_send_am_reply(MPIR_Context_id_t context_id, data_sz + am_hdr_sz + sizeof(ucx_hdr), ucp_dt_make_contig(1), ucx_tag, - &MPIDI_UCX_send_am_callback); + &MPIDI_UCX_am_send_callback); MPIDI_CH4_UCX_REQUEST(ucp_request, tag_send_nb); } @@ -436,7 +436,7 @@ static inline int MPIDI_NM_send_am_reply(MPIR_Context_id_t context_id, goto fn_exit; } -static inline int MPIDI_NM_send_amv_reply(MPIR_Context_id_t context_id, +static inline int MPIDI_NM_am_sendv_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, struct iovec *am_hdr, @@ -465,7 +465,7 @@ static inline int MPIDI_NM_send_amv_reply(MPIR_Context_id_t context_id, am_hdr_sz += am_hdr[i].iov_len; } - mpi_errno = MPIDI_NM_send_am_reply(context_id, src_rank, handler_id, am_hdr_buf, am_hdr_sz, + mpi_errno = MPIDI_NM_am_send_reply(context_id, src_rank, handler_id, am_hdr_buf, am_hdr_sz, data, count, datatype, sreq); MPL_free(am_hdr_buf); MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_UCX_SEND_AMV_REPLY); @@ -477,7 +477,7 @@ static inline size_t MPIDI_NM_am_hdr_max_sz(void) return (MPIDI_UCX_MAX_AM_EAGER_SZ - sizeof(MPIDI_UCX_am_header_t)); } -static inline int MPIDI_NM_inject_am_hdr(int rank, +static inline int MPIDI_NM_am_inject_hdr(int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, size_t am_hdr_sz, void *netmod_context) @@ -530,7 +530,7 @@ static inline int MPIDI_NM_inject_am_hdr(int rank, goto fn_exit; } -static inline int MPIDI_NM_inject_am_hdr_reply(MPIR_Context_id_t context_id, +static inline int MPIDI_NM_am_inject_hdr_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, const void *am_hdr, size_t am_hdr_sz) { @@ -593,7 +593,7 @@ static inline int MPIDI_NM_am_recv(MPIR_Request * req) msg.sreq_ptr = (MPIDI_CH4U_REQUEST(req, req->rreq.peer_req_ptr)); msg.rreq_ptr = (uint64_t) req; MPIR_Assert((void *) msg.sreq_ptr != NULL); - mpi_errno = MPIDI_NM_inject_am_hdr_reply(MPIDI_CH4U_get_context(MPIDI_CH4U_REQUEST(req, tag)), + mpi_errno = MPIDI_NM_am_inject_hdr_reply(MPIDI_CH4U_get_context(MPIDI_CH4U_REQUEST(req, tag)), MPIDI_CH4U_REQUEST(req, src_rank), MPIDI_CH4U_SEND_LONG_ACK, &msg, sizeof(msg)); if (mpi_errno) diff --git a/src/mpid/ch4/shm/include/shm.h b/src/mpid/ch4/shm/include/shm.h index 11acade..ed7f0f3 100644 --- a/src/mpid/ch4/shm/include/shm.h +++ b/src/mpid/ch4/shm/include/shm.h @@ -30,9 +30,9 @@ typedef int (*MPIDI_SHM_am_target_handler_fn) typedef int (*MPIDI_SHM_init_t) (int rank, int size); typedef int (*MPIDI_SHM_finalize_t) (void); typedef int (*MPIDI_SHM_progress_t) (int blocking); -typedef int (*MPIDI_SHM_reg_hdr_handler_t) (int handler_id, - MPIDI_SHM_am_origin_handler_fn origin_handler_fn, - MPIDI_SHM_am_target_handler_fn target_handler_fn); +typedef int (*MPIDI_SHM_am_reg_handler_t) (int handler_id, + MPIDI_SHM_am_origin_handler_fn origin_handler_fn, + MPIDI_SHM_am_target_handler_fn target_handler_fn); typedef int (*MPIDI_SHM_comm_connect_t) (const char *port_name, MPIR_Info * info, int root, MPIR_Comm * comm, MPIR_Comm ** newcomm_ptr); typedef int (*MPIDI_SHM_comm_disconnect_t) (MPIR_Comm * comm_ptr); @@ -40,31 +40,31 @@ typedef int (*MPIDI_SHM_open_port_t) (MPIR_Info * info_ptr, char *port_name); typedef int (*MPIDI_SHM_close_port_t) (const char *port_name); typedef int (*MPIDI_SHM_comm_accept_t) (const char *port_name, MPIR_Info * info, int root, MPIR_Comm * comm, MPIR_Comm ** newcomm_ptr); -typedef int (*MPIDI_SHM_send_am_hdr_t) (int rank, MPIR_Comm * comm, int handler_id, +typedef int (*MPIDI_SHM_am_send_hdr_t) (int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, size_t am_hdr_sz, MPIR_Request * sreq, void *shm_context); -typedef int (*MPIDI_SHM_inject_am_hdr_t) (int rank, MPIR_Comm * comm, int handler_id, +typedef int (*MPIDI_SHM_am_inject_hdr_t) (int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, size_t am_hdr_sz, void *shm_context); -typedef int (*MPIDI_SHM_send_am_t) (int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, +typedef int (*MPIDI_SHM_am_send_t) (int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, size_t am_hdr_sz, const void *data, MPI_Count count, MPI_Datatype datatype, MPIR_Request * sreq, void *shm_context); typedef int (*MPIDI_SHM_inject_am_t) (int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, size_t am_hdr_sz, const void *data, MPI_Count count, MPI_Datatype datatype, void *shm_context); -typedef int (*MPIDI_SHM_send_amv_t) (int rank, MPIR_Comm * comm, int handler_id, +typedef int (*MPIDI_SHM_am_sendv_t) (int rank, MPIR_Comm * comm, int handler_id, struct iovec * am_hdrs, size_t iov_len, const void *data, MPI_Count count, MPI_Datatype datatype, MPIR_Request * sreq, void *shm_context); typedef int (*MPIDI_SHM_inject_amv_t) (int rank, MPIR_Comm * comm, int handler_id, struct iovec * am_hdrs, size_t iov_len, const void *data, MPI_Count count, MPI_Datatype datatype, void *shm_context); -typedef int (*MPIDI_SHM_send_am_hdr_reply_t) (MPIR_Context_id_t context_id, int src_rank, +typedef int (*MPIDI_SHM_am_send_hdr_reply_t) (MPIR_Context_id_t context_id, int src_rank, int handler_id, const void *am_hdr, size_t am_hdr_sz, MPIR_Request * sreq); -typedef int (*MPIDI_SHM_inject_am_hdr_reply_t) (MPIR_Context_id_t context_id, int src_rank, +typedef int (*MPIDI_SHM_am_inject_hdr_reply_t) (MPIR_Context_id_t context_id, int src_rank, int handler_id, const void *am_hdr, size_t am_hdr_sz); -typedef int (*MPIDI_SHM_send_am_reply_t) (MPIR_Context_id_t context_id, int src_rank, +typedef int (*MPIDI_SHM_am_send_reply_t) (MPIR_Context_id_t context_id, int src_rank, int handler_id, const void *am_hdr, size_t am_hdr_sz, const void *data, MPI_Count count, MPI_Datatype datatype, MPIR_Request * sreq); @@ -72,7 +72,7 @@ typedef int (*MPIDI_SHM_inject_am_reply_t) (MPIR_Context_id_t context_id, int sr int handler_id, const void *am_hdr, size_t am_hdr_sz, const void *data, MPI_Count count, MPI_Datatype datatype); -typedef int (*MPIDI_SHM_send_amv_reply_t) (MPIR_Context_id_t context_id, int src_rank, +typedef int (*MPIDI_SHM_am_sendv_reply_t) (MPIR_Context_id_t context_id, int src_rank, int handler_id, struct iovec * am_hdr, size_t iov_len, const void *data, MPI_Count count, MPI_Datatype datatype, MPIR_Request * sreq); @@ -376,23 +376,23 @@ typedef struct MPIDI_SHM_funcs { MPIDI_SHM_init_t init; MPIDI_SHM_finalize_t finalize; MPIDI_SHM_progress_t progress; - MPIDI_SHM_reg_hdr_handler_t reg_hdr_handler; + MPIDI_SHM_am_reg_handler_t am_reg_handler; MPIDI_SHM_comm_connect_t comm_connect; MPIDI_SHM_comm_disconnect_t comm_disconnect; MPIDI_SHM_open_port_t open_port; MPIDI_SHM_close_port_t close_port; MPIDI_SHM_comm_accept_t comm_accept; - MPIDI_SHM_send_am_hdr_t send_am_hdr; - MPIDI_SHM_inject_am_hdr_t inject_am_hdr; - MPIDI_SHM_send_am_t send_am; + MPIDI_SHM_am_send_hdr_t am_send_hdr; + MPIDI_SHM_am_inject_hdr_t am_inject_hdr; + MPIDI_SHM_am_send_t am_send; MPIDI_SHM_inject_am_t inject_am; - MPIDI_SHM_send_amv_t send_amv; + MPIDI_SHM_am_sendv_t am_sendv; MPIDI_SHM_inject_amv_t inject_amv; - MPIDI_SHM_send_am_hdr_reply_t send_am_hdr_reply; - MPIDI_SHM_inject_am_hdr_reply_t inject_am_hdr_reply; - MPIDI_SHM_send_am_reply_t send_am_reply; + MPIDI_SHM_am_send_hdr_reply_t am_send_hdr_reply; + MPIDI_SHM_am_inject_hdr_reply_t am_inject_hdr_reply; + MPIDI_SHM_am_send_reply_t am_send_reply; MPIDI_SHM_inject_am_reply_t inject_am_reply; - MPIDI_SHM_send_amv_reply_t send_amv_reply; + MPIDI_SHM_am_sendv_reply_t am_sendv_reply; MPIDI_SHM_inject_amv_reply_t inject_amv_reply; MPIDI_SHM_am_hdr_max_sz_t am_hdr_max_sz; MPIDI_SHM_am_inject_max_sz_t am_inject_max_sz; @@ -528,11 +528,11 @@ extern char MPIDI_SHM_strings[][MPIDI_MAX_SHM_STRING_LEN]; MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_init(int rank, int size) MPL_STATIC_INLINE_SUFFIX; MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_finalize(void) MPL_STATIC_INLINE_SUFFIX; MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_progress(int blocking) MPL_STATIC_INLINE_SUFFIX; -MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_reg_hdr_handler(int handler_id, - MPIDI_SHM_am_origin_handler_fn - origin_handler_fn, - MPIDI_SHM_am_target_handler_fn - target_handler_fn) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_am_reg_handler(int handler_id, + MPIDI_SHM_am_origin_handler_fn + origin_handler_fn, + MPIDI_SHM_am_target_handler_fn + target_handler_fn) MPL_STATIC_INLINE_SUFFIX; MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_connect(const char *port_name, MPIR_Info * info, int root, MPIR_Comm * comm, MPIR_Comm ** @@ -546,15 +546,15 @@ MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_accept(const char *port_name, MPIR_I int root, MPIR_Comm * comm, MPIR_Comm ** newcomm_ptr) MPL_STATIC_INLINE_SUFFIX; -MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_send_am_hdr(int rank, MPIR_Comm * comm, int handler_id, +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_am_send_hdr(int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, size_t am_hdr_sz, MPIR_Request * sreq, void *shm_context) MPL_STATIC_INLINE_SUFFIX; -MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_am_hdr(int rank, MPIR_Comm * comm, +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_am_inject_hdr(int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, size_t am_hdr_sz, void *shm_context) MPL_STATIC_INLINE_SUFFIX; -MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_send_am(int rank, MPIR_Comm * comm, int handler_id, +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_am_send(int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, size_t am_hdr_sz, const void *data, MPI_Count count, MPI_Datatype datatype, MPIR_Request * sreq, @@ -564,7 +564,7 @@ MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_am(int rank, MPIR_Comm * comm, int const void *data, MPI_Count count, MPI_Datatype datatype, void *shm_context) MPL_STATIC_INLINE_SUFFIX; -MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_send_amv(int rank, MPIR_Comm * comm, int handler_id, +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_am_sendv(int rank, MPIR_Comm * comm, int handler_id, struct iovec *am_hdrs, size_t iov_len, const void *data, MPI_Count count, MPI_Datatype datatype, MPIR_Request * sreq, @@ -574,17 +574,17 @@ MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_amv(int rank, MPIR_Comm * comm, in const void *data, MPI_Count count, MPI_Datatype datatype, void *shm_context) MPL_STATIC_INLINE_SUFFIX; -MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_send_am_hdr_reply(MPIR_Context_id_t context_id, +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_am_send_hdr_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, const void *am_hdr, size_t am_hdr_sz, MPIR_Request * sreq) MPL_STATIC_INLINE_SUFFIX; -MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_am_hdr_reply(MPIR_Context_id_t context_id, +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_am_inject_hdr_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, const void *am_hdr, size_t am_hdr_sz) MPL_STATIC_INLINE_SUFFIX; -MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_send_am_reply(MPIR_Context_id_t context_id, +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_am_send_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, const void *am_hdr, size_t am_hdr_sz, const void *data, MPI_Count count, @@ -596,7 +596,7 @@ MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_am_reply(MPIR_Context_id_t context const void *data, MPI_Count count, MPI_Datatype datatype) MPL_STATIC_INLINE_SUFFIX; -MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_send_amv_reply(MPIR_Context_id_t context_id, +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_am_sendv_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, struct iovec *am_hdr, size_t iov_len, const void *data, MPI_Count count, diff --git a/src/mpid/ch4/shm/include/shm_impl.h b/src/mpid/ch4/shm/include/shm_impl.h index d1125cf..92a5e0b 100644 --- a/src/mpid/ch4/shm/include/shm_impl.h +++ b/src/mpid/ch4/shm/include/shm_impl.h @@ -30,13 +30,13 @@ MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_progress(int blocking) return MPIDI_SHM_func->progress(blocking); }; -MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_reg_hdr_handler(int handler_id, - MPIDI_SHM_am_origin_handler_fn - origin_handler_fn, - MPIDI_SHM_am_target_handler_fn - target_handler_fn) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_am_reg_handler(int handler_id, + MPIDI_SHM_am_origin_handler_fn + origin_handler_fn, + MPIDI_SHM_am_target_handler_fn + target_handler_fn) { - return MPIDI_SHM_func->reg_hdr_handler(handler_id, origin_handler_fn, target_handler_fn); + return MPIDI_SHM_func->am_reg_handler(handler_id, origin_handler_fn, target_handler_fn); }; MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_connect(const char *port_name, MPIR_Info * info, @@ -68,28 +68,28 @@ MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_accept(const char *port_name, MPIR_I return MPIDI_SHM_func->comm_accept(port_name, info, root, comm, newcomm_ptr); }; -MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_send_am_hdr(int rank, MPIR_Comm * comm, int handler_id, +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_am_send_hdr(int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, size_t am_hdr_sz, MPIR_Request * sreq, void *shm_context) { - return MPIDI_SHM_func->send_am_hdr(rank, comm, handler_id, am_hdr, am_hdr_sz, sreq, + return MPIDI_SHM_func->am_send_hdr(rank, comm, handler_id, am_hdr, am_hdr_sz, sreq, shm_context); }; -MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_am_hdr(int rank, MPIR_Comm * comm, +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_am_inject_hdr(int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, size_t am_hdr_sz, void *shm_context) { - return MPIDI_SHM_func->inject_am_hdr(rank, comm, handler_id, am_hdr, am_hdr_sz, shm_context); + return MPIDI_SHM_func->am_inject_hdr(rank, comm, handler_id, am_hdr, am_hdr_sz, shm_context); }; -MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_send_am(int rank, MPIR_Comm * comm, int handler_id, +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_am_send(int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, size_t am_hdr_sz, const void *data, MPI_Count count, MPI_Datatype datatype, MPIR_Request * sreq, void *shm_context) { - return MPIDI_SHM_func->send_am(rank, comm, handler_id, am_hdr, am_hdr_sz, data, count, datatype, + return MPIDI_SHM_func->am_send(rank, comm, handler_id, am_hdr, am_hdr_sz, data, count, datatype, sreq, shm_context); }; @@ -102,13 +102,13 @@ MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_am(int rank, MPIR_Comm * comm, int datatype, shm_context); }; -MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_send_amv(int rank, MPIR_Comm * comm, int handler_id, +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_am_sendv(int rank, MPIR_Comm * comm, int handler_id, struct iovec *am_hdrs, size_t iov_len, const void *data, MPI_Count count, MPI_Datatype datatype, MPIR_Request * sreq, void *shm_context) { - return MPIDI_SHM_func->send_amv(rank, comm, handler_id, am_hdrs, iov_len, data, count, datatype, + return MPIDI_SHM_func->am_sendv(rank, comm, handler_id, am_hdrs, iov_len, data, count, datatype, sreq, shm_context); }; @@ -121,29 +121,29 @@ MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_amv(int rank, MPIR_Comm * comm, in datatype, shm_context); }; -MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_send_am_hdr_reply(MPIR_Context_id_t context_id, +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_am_send_hdr_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, const void *am_hdr, size_t am_hdr_sz, MPIR_Request * sreq) { - return MPIDI_SHM_func->send_am_hdr_reply(context_id, src_rank, handler_id, am_hdr, am_hdr_sz, + return MPIDI_SHM_func->am_send_hdr_reply(context_id, src_rank, handler_id, am_hdr, am_hdr_sz, sreq); }; -MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_am_hdr_reply(MPIR_Context_id_t context_id, +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_am_inject_hdr_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, const void *am_hdr, size_t am_hdr_sz) { - return MPIDI_SHM_func->inject_am_hdr_reply(context_id, src_rank, handler_id, am_hdr, am_hdr_sz); + return MPIDI_SHM_func->am_inject_hdr_reply(context_id, src_rank, handler_id, am_hdr, am_hdr_sz); }; -MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_send_am_reply(MPIR_Context_id_t context_id, +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_am_send_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, const void *am_hdr, size_t am_hdr_sz, const void *data, MPI_Count count, MPI_Datatype datatype, MPIR_Request * sreq) { - return MPIDI_SHM_func->send_am_reply(context_id, src_rank, handler_id, am_hdr, am_hdr_sz, data, + return MPIDI_SHM_func->am_send_reply(context_id, src_rank, handler_id, am_hdr, am_hdr_sz, data, count, datatype, sreq); }; @@ -157,13 +157,13 @@ MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_am_reply(MPIR_Context_id_t context data, count, datatype); }; -MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_send_amv_reply(MPIR_Context_id_t context_id, +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_am_sendv_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, struct iovec *am_hdr, size_t iov_len, const void *data, MPI_Count count, MPI_Datatype datatype, MPIR_Request * sreq) { - return MPIDI_SHM_func->send_amv_reply(context_id, src_rank, handler_id, am_hdr, iov_len, data, + return MPIDI_SHM_func->am_sendv_reply(context_id, src_rank, handler_id, am_hdr, iov_len, data, count, datatype, sreq); }; diff --git a/src/mpid/ch4/shm/posix/func_table.c b/src/mpid/ch4/shm/posix/func_table.c index 39f2ea5..4a39d50 100644 --- a/src/mpid/ch4/shm/posix/func_table.c +++ b/src/mpid/ch4/shm/posix/func_table.c @@ -16,23 +16,23 @@ MPIDI_SHM_funcs_t MPIDI_SHM_posix_funcs = { MPIDI_SHM_init, MPIDI_SHM_finalize, MPIDI_SHM_progress, - MPIDI_SHM_reg_hdr_handler, + MPIDI_SHM_am_reg_handler, MPIDI_SHM_comm_connect, MPIDI_SHM_comm_disconnect, MPIDI_SHM_open_port, MPIDI_SHM_close_port, MPIDI_SHM_comm_accept, - MPIDI_SHM_send_am_hdr, - MPIDI_SHM_inject_am_hdr, - MPIDI_SHM_send_am, + MPIDI_SHM_am_send_hdr, + MPIDI_SHM_am_inject_hdr, + MPIDI_SHM_am_send, MPIDI_SHM_inject_am, - MPIDI_SHM_send_amv, + MPIDI_SHM_am_sendv, MPIDI_SHM_inject_amv, - MPIDI_SHM_send_am_hdr_reply, - MPIDI_SHM_inject_am_hdr_reply, - MPIDI_SHM_send_am_reply, + MPIDI_SHM_am_send_hdr_reply, + MPIDI_SHM_am_inject_hdr_reply, + MPIDI_SHM_am_send_reply, MPIDI_SHM_inject_am_reply, - MPIDI_SHM_send_amv_reply, + MPIDI_SHM_am_sendv_reply, MPIDI_SHM_inject_amv_reply, MPIDI_SHM_am_hdr_max_sz, MPIDI_SHM_am_inject_max_sz, diff --git a/src/mpid/ch4/shm/posix/posix_am.h b/src/mpid/ch4/shm/posix/posix_am.h index 06b2351..842bf6c 100644 --- a/src/mpid/ch4/shm/posix/posix_am.h +++ b/src/mpid/ch4/shm/posix/posix_am.h @@ -12,15 +12,15 @@ #include "posix_impl.h" -static inline int MPIDI_SHM_reg_hdr_handler(int handler_id, - MPIDI_SHM_am_origin_handler_fn origin_handler_fn, - MPIDI_SHM_am_target_handler_fn target_handler_fn) +static inline int MPIDI_SHM_am_reg_handler(int handler_id, + MPIDI_SHM_am_origin_handler_fn origin_handler_fn, + MPIDI_SHM_am_target_handler_fn target_handler_fn) { MPIR_Assert(0); return MPI_SUCCESS; } -static inline int MPIDI_SHM_send_am_hdr(int rank, +static inline int MPIDI_SHM_am_send_hdr(int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, @@ -30,7 +30,7 @@ static inline int MPIDI_SHM_send_am_hdr(int rank, return MPI_SUCCESS; } -static inline int MPIDI_SHM_send_am(int rank, +static inline int MPIDI_SHM_am_send(int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, @@ -43,7 +43,7 @@ static inline int MPIDI_SHM_send_am(int rank, return MPI_SUCCESS; } -static inline int MPIDI_SHM_send_amv(int rank, +static inline int MPIDI_SHM_am_sendv(int rank, MPIR_Comm * comm, int handler_id, struct iovec *am_hdr, @@ -56,7 +56,7 @@ static inline int MPIDI_SHM_send_amv(int rank, return MPI_SUCCESS; } -static inline int MPIDI_SHM_send_am_hdr_reply(MPIR_Context_id_t context_id, int src_rank, +static inline int MPIDI_SHM_am_send_hdr_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, const void *am_hdr, size_t am_hdr_sz, MPIR_Request * sreq) @@ -65,7 +65,7 @@ static inline int MPIDI_SHM_send_am_hdr_reply(MPIR_Context_id_t context_id, int return MPI_SUCCESS; } -static inline int MPIDI_SHM_send_am_reply(MPIR_Context_id_t context_id, int src_rank, +static inline int MPIDI_SHM_am_send_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, const void *am_hdr, size_t am_hdr_sz, @@ -77,7 +77,7 @@ static inline int MPIDI_SHM_send_am_reply(MPIR_Context_id_t context_id, int src_ return MPI_SUCCESS; } -static inline int MPIDI_SHM_send_amv_reply(MPIR_Context_id_t context_id, int src_rank, +static inline int MPIDI_SHM_am_sendv_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, struct iovec *am_hdr, size_t iov_len, @@ -95,7 +95,7 @@ static inline size_t MPIDI_SHM_am_hdr_max_sz(void) return 0; } -static inline int MPIDI_SHM_inject_am_hdr(int rank, +static inline int MPIDI_SHM_am_inject_hdr(int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, size_t am_hdr_sz, void *shm_context) @@ -128,7 +128,7 @@ static inline int MPIDI_SHM_inject_amv(int rank, return MPI_SUCCESS; } -static inline int MPIDI_SHM_inject_am_hdr_reply(MPIR_Context_id_t context_id, int src_rank, +static inline int MPIDI_SHM_am_inject_hdr_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, const void *am_hdr, size_t am_hdr_sz) { diff --git a/src/mpid/ch4/shm/stubshm/func_table.c b/src/mpid/ch4/shm/stubshm/func_table.c index 264b179..237869c 100644 --- a/src/mpid/ch4/shm/stubshm/func_table.c +++ b/src/mpid/ch4/shm/stubshm/func_table.c @@ -16,23 +16,23 @@ MPIDI_SHM_funcs_t MPIDI_SHM_stubshm_funcs = { MPIDI_SHM_init, MPIDI_SHM_finalize, MPIDI_SHM_progress, - MPIDI_SHM_reg_hdr_handler, + MPIDI_SHM_am_reg_handler, MPIDI_SHM_comm_connect, MPIDI_SHM_comm_disconnect, MPIDI_SHM_open_port, MPIDI_SHM_close_port, MPIDI_SHM_comm_accept, - MPIDI_SHM_send_am_hdr, - MPIDI_SHM_inject_am_hdr, - MPIDI_SHM_send_am, + MPIDI_SHM_am_send_hdr, + MPIDI_SHM_am_inject_hdr, + MPIDI_SHM_am_send, MPIDI_SHM_inject_am, - MPIDI_SHM_send_amv, + MPIDI_SHM_am_sendv, MPIDI_SHM_inject_amv, - MPIDI_SHM_send_am_hdr_reply, - MPIDI_SHM_inject_am_hdr_reply, - MPIDI_SHM_send_am_reply, + MPIDI_SHM_am_send_hdr_reply, + MPIDI_SHM_am_inject_hdr_reply, + MPIDI_SHM_am_send_reply, MPIDI_SHM_inject_am_reply, - MPIDI_SHM_send_amv_reply, + MPIDI_SHM_am_sendv_reply, MPIDI_SHM_inject_amv_reply, MPIDI_SHM_am_hdr_max_sz, MPIDI_SHM_am_inject_max_sz, diff --git a/src/mpid/ch4/shm/stubshm/stubshm_am.h b/src/mpid/ch4/shm/stubshm/stubshm_am.h index 1ff7dab..e837c9f 100644 --- a/src/mpid/ch4/shm/stubshm/stubshm_am.h +++ b/src/mpid/ch4/shm/stubshm/stubshm_am.h @@ -12,15 +12,15 @@ #include "stubshm_impl.h" -static inline int MPIDI_SHM_reg_hdr_handler(int handler_id, - MPIDI_SHM_am_origin_handler_fn origin_handler_fn, - MPIDI_SHM_am_target_handler_fn target_handler_fn) +static inline int MPIDI_SHM_am_reg_handler(int handler_id, + MPIDI_SHM_am_origin_handler_fn origin_handler_fn, + MPIDI_SHM_am_target_handler_fn target_handler_fn) { MPIR_Assert(0); return MPI_SUCCESS; } -static inline int MPIDI_SHM_send_am_hdr(int rank, +static inline int MPIDI_SHM_am_send_hdr(int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, @@ -30,7 +30,7 @@ static inline int MPIDI_SHM_send_am_hdr(int rank, return MPI_SUCCESS; } -static inline int MPIDI_SHM_send_am(int rank, +static inline int MPIDI_SHM_am_send(int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, @@ -43,7 +43,7 @@ static inline int MPIDI_SHM_send_am(int rank, return MPI_SUCCESS; } -static inline int MPIDI_SHM_send_amv(int rank, +static inline int MPIDI_SHM_am_sendv(int rank, MPIR_Comm * comm, int handler_id, struct iovec *am_hdr, @@ -56,7 +56,7 @@ static inline int MPIDI_SHM_send_amv(int rank, return MPI_SUCCESS; } -static inline int MPIDI_SHM_send_am_hdr_reply(MPIR_Context_id_t context_id, int src_rank, +static inline int MPIDI_SHM_am_send_hdr_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, const void *am_hdr, size_t am_hdr_sz, MPIR_Request * sreq) @@ -65,7 +65,7 @@ static inline int MPIDI_SHM_send_am_hdr_reply(MPIR_Context_id_t context_id, int return MPI_SUCCESS; } -static inline int MPIDI_SHM_send_am_reply(MPIR_Context_id_t context_id, int src_rank, +static inline int MPIDI_SHM_am_send_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, const void *am_hdr, size_t am_hdr_sz, @@ -77,7 +77,7 @@ static inline int MPIDI_SHM_send_am_reply(MPIR_Context_id_t context_id, int src_ return MPI_SUCCESS; } -static inline int MPIDI_SHM_send_amv_reply(MPIR_Context_id_t context_id, int src_rank, +static inline int MPIDI_SHM_am_sendv_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, struct iovec *am_hdr, size_t iov_len, @@ -95,7 +95,7 @@ static inline size_t MPIDI_SHM_am_hdr_max_sz(void) return 0; } -static inline int MPIDI_SHM_inject_am_hdr(int rank, +static inline int MPIDI_SHM_am_inject_hdr(int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, size_t am_hdr_sz, void *shm_context) @@ -128,7 +128,7 @@ static inline int MPIDI_SHM_inject_amv(int rank, return MPI_SUCCESS; } -static inline int MPIDI_SHM_inject_am_hdr_reply(MPIR_Context_id_t context_id, int src_rank, +static inline int MPIDI_SHM_am_inject_hdr_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, const void *am_hdr, size_t am_hdr_sz) { diff --git a/src/mpid/ch4/src/ch4r_callbacks.h b/src/mpid/ch4/src/ch4r_callbacks.h index db0ec20..471d3e3 100644 --- a/src/mpid/ch4/src/ch4r_callbacks.h +++ b/src/mpid/ch4/src/ch4r_callbacks.h @@ -152,7 +152,7 @@ static inline int MPIDI_CH4U_get_cmpl_handler(MPIR_Request * req) win = MPIDI_CH4U_REQUEST(req, req->greq.win_ptr); context_id = MPIDI_CH4U_win_to_context(win); if (MPIDI_CH4U_REQUEST(req, req->greq.n_iov) == 0) { - mpi_errno = MPIDI_NM_send_am_reply(context_id, + mpi_errno = MPIDI_NM_am_send_reply(context_id, MPIDI_CH4U_REQUEST(req, src_rank), MPIDI_CH4U_GET_ACK, &get_ack, sizeof(get_ack), @@ -186,7 +186,7 @@ static inline int MPIDI_CH4U_get_cmpl_handler(MPIR_Request * req) MPL_free(MPIDI_CH4U_REQUEST(req, req->greq.dt_iov)); MPIDI_CH4U_REQUEST(req, req->greq.dt_iov) = (void *) p_data; - mpi_errno = MPIDI_NM_send_am_reply(context_id, + mpi_errno = MPIDI_NM_am_send_reply(context_id, MPIDI_CH4U_REQUEST(req, src_rank), MPIDI_CH4U_GET_ACK, &get_ack, sizeof(get_ack), p_data, data_sz, MPI_BYTE, req); @@ -414,7 +414,7 @@ static inline int MPIDI_CH4U_reply_ssend(MPIR_Request * rreq) MPIR_cc_incr(rreq->cc_ptr, &c); ack_msg.sreq_ptr = MPIDI_CH4U_REQUEST(rreq, req->rreq.peer_req_ptr); - mpi_errno = MPIDI_NM_send_am_hdr_reply(MPIDI_CH4U_get_context(MPIDI_CH4U_REQUEST(rreq, tag)), + mpi_errno = MPIDI_NM_am_send_hdr_reply(MPIDI_CH4U_get_context(MPIDI_CH4U_REQUEST(rreq, tag)), MPIDI_CH4U_REQUEST(rreq, src_rank), MPIDI_CH4U_SSEND_ACK, &ack_msg, sizeof(ack_msg), rreq); if (mpi_errno) @@ -439,7 +439,7 @@ static inline int MPIDI_CH4U_ack_put(MPIR_Request * rreq) ack_msg.preq_ptr = MPIDI_CH4U_REQUEST(rreq, req->preq.preq_ptr); mpi_errno = - MPIDI_NM_inject_am_hdr_reply(MPIDI_CH4U_win_to_context + MPIDI_NM_am_inject_hdr_reply(MPIDI_CH4U_win_to_context (MPIDI_CH4U_REQUEST(rreq, req->preq.win_ptr)), MPIDI_CH4U_REQUEST(rreq, src_rank), MPIDI_CH4U_PUT_ACK, &ack_msg, sizeof(ack_msg)); @@ -473,7 +473,7 @@ static inline int MPIDI_CH4U_ack_cswap(MPIR_Request * rreq) ack_msg.req_ptr = MPIDI_CH4U_REQUEST(rreq, req->creq.creq_ptr); mpi_errno = - MPIDI_NM_send_am_reply(MPIDI_CH4U_win_to_context + MPIDI_NM_am_send_reply(MPIDI_CH4U_win_to_context (MPIDI_CH4U_REQUEST(rreq, req->creq.win_ptr)), MPIDI_CH4U_REQUEST(rreq, src_rank), MPIDI_CH4U_CSWAP_ACK, &ack_msg, sizeof(ack_msg), result_addr, 1, MPIDI_CH4U_REQUEST(rreq, @@ -503,7 +503,7 @@ static inline int MPIDI_CH4U_ack_acc(MPIR_Request * rreq) ack_msg.req_ptr = MPIDI_CH4U_REQUEST(rreq, req->areq.req_ptr); mpi_errno = - MPIDI_NM_inject_am_hdr_reply(MPIDI_CH4U_win_to_context + MPIDI_NM_am_inject_hdr_reply(MPIDI_CH4U_win_to_context (MPIDI_CH4U_REQUEST(rreq, req->areq.win_ptr)), MPIDI_CH4U_REQUEST(rreq, src_rank), MPIDI_CH4U_ACC_ACK, &ack_msg, sizeof(ack_msg)); @@ -538,7 +538,7 @@ static inline int MPIDI_CH4U_ack_get_acc(MPIR_Request * rreq) ack_msg.req_ptr = MPIDI_CH4U_REQUEST(rreq, req->areq.req_ptr); mpi_errno = - MPIDI_NM_send_am_reply(MPIDI_CH4U_win_to_context + MPIDI_NM_am_send_reply(MPIDI_CH4U_win_to_context (MPIDI_CH4U_REQUEST(rreq, req->areq.win_ptr)), MPIDI_CH4U_REQUEST(rreq, src_rank), MPIDI_CH4U_GET_ACC_ACK, &ack_msg, sizeof(ack_msg), MPIDI_CH4U_REQUEST(rreq, req->areq.data), @@ -899,7 +899,7 @@ static inline int MPIDI_CH4U_put_iov_cmpl_handler(MPIR_Request * rreq) ack_msg.target_preq_ptr = (uint64_t) rreq; mpi_errno = - MPIDI_NM_inject_am_hdr_reply(MPIDI_CH4U_win_to_context + MPIDI_NM_am_inject_hdr_reply(MPIDI_CH4U_win_to_context (MPIDI_CH4U_REQUEST(rreq, req->preq.win_ptr)), MPIDI_CH4U_REQUEST(rreq, src_rank), MPIDI_CH4U_PUT_IOV_ACK, &ack_msg, sizeof(ack_msg)); @@ -929,7 +929,7 @@ static inline int MPIDI_CH4U_acc_iov_cmpl_handler(MPIR_Request * rreq) ack_msg.target_preq_ptr = (uint64_t) rreq; mpi_errno = - MPIDI_NM_inject_am_hdr_reply(MPIDI_CH4U_win_to_context + MPIDI_NM_am_inject_hdr_reply(MPIDI_CH4U_win_to_context (MPIDI_CH4U_REQUEST(rreq, req->areq.win_ptr)), MPIDI_CH4U_REQUEST(rreq, src_rank), MPIDI_CH4U_ACC_IOV_ACK, &ack_msg, sizeof(ack_msg)); @@ -1651,7 +1651,7 @@ static inline int MPIDI_CH4U_send_long_ack_target_handler(void *am_hdr, /* Start the main data transfer */ send_hdr.rreq_ptr = msg_hdr->rreq_ptr; mpi_errno = - MPIDI_NM_send_am_reply(MPIDI_CH4U_get_context(MPIDI_CH4U_REQUEST(sreq, req->lreq).msg_tag), + MPIDI_NM_am_send_reply(MPIDI_CH4U_get_context(MPIDI_CH4U_REQUEST(sreq, req->lreq).msg_tag), MPIDI_CH4U_REQUEST(sreq, src_rank), MPIDI_CH4U_SEND_LONG_LMT, &send_hdr, sizeof(send_hdr), MPIDI_CH4U_REQUEST(sreq, req->lreq).src_buf, @@ -1893,7 +1893,7 @@ static inline int MPIDI_CH4U_win_lock_advance(MPIR_Win * win) else MPIR_ERR_SETANDJUMP(mpi_errno, MPI_ERR_OTHER, "**rmasync"); - mpi_errno = MPIDI_NM_inject_am_hdr_reply(MPIDI_CH4U_win_to_context(win), + mpi_errno = MPIDI_NM_am_inject_hdr_reply(MPIDI_CH4U_win_to_context(win), lock->rank, MPIDI_CH4U_WIN_CTRL, &msg, sizeof(msg)); if (mpi_errno) @@ -1986,7 +1986,7 @@ static inline void MPIDI_CH4U_win_unlock_proc(const MPIDI_CH4U_win_cntrl_msg_t * msg.origin_rank = win->comm_ptr->rank; msg.type = MPIDI_CH4U_WIN_UNLOCK_ACK; - mpi_errno = MPIDI_NM_inject_am_hdr_reply(MPIDI_CH4U_win_to_context(win), + mpi_errno = MPIDI_NM_am_inject_hdr_reply(MPIDI_CH4U_win_to_context(win), info->origin_rank, MPIDI_CH4U_WIN_CTRL, &msg, sizeof(msg)); if (mpi_errno) @@ -2306,7 +2306,7 @@ static inline int MPIDI_CH4U_put_iov_ack_target_handler(void *am_hdr, origin_req = (MPIR_Request *) msg_hdr->origin_preq_ptr; dat_msg.preq_ptr = msg_hdr->target_preq_ptr; win = MPIDI_CH4U_REQUEST(origin_req, req->preq.win_ptr); - mpi_errno = MPIDI_NM_send_am_reply(MPIDI_CH4U_win_to_context(win), + mpi_errno = MPIDI_NM_am_send_reply(MPIDI_CH4U_win_to_context(win), MPIDI_CH4U_REQUEST(origin_req, src_rank), MPIDI_CH4U_PUT_DAT_REQ, &dat_msg, sizeof(dat_msg), @@ -2355,7 +2355,7 @@ static inline int MPIDI_CH4U_acc_iov_ack_target_handler(void *am_hdr, origin_req = (MPIR_Request *) msg_hdr->origin_preq_ptr; dat_msg.preq_ptr = msg_hdr->target_preq_ptr; win = MPIDI_CH4U_REQUEST(origin_req, req->areq.win_ptr); - mpi_errno = MPIDI_NM_send_am_reply(MPIDI_CH4U_win_to_context(win), + mpi_errno = MPIDI_NM_am_send_reply(MPIDI_CH4U_win_to_context(win), MPIDI_CH4U_REQUEST(origin_req, src_rank), MPIDI_CH4U_ACC_DAT_REQ, &dat_msg, sizeof(dat_msg), diff --git a/src/mpid/ch4/src/ch4r_init.h b/src/mpid/ch4/src/ch4r_init.h index 124ac49..9126e17 100644 --- a/src/mpid/ch4/src/ch4r_init.h +++ b/src/mpid/ch4/src/ch4r_init.h @@ -129,129 +129,129 @@ __CH4_INLINE__ int MPIDI_CH4U_init(MPIR_Comm * comm_world, MPIR_Comm * comm_self MPIDI_CH4I_BUF_POOL_SZ); MPIR_Assert(MPIDI_CH4_Global.buf_pool); - mpi_errno = MPIDI_NM_reg_hdr_handler(MPIDI_CH4U_SEND, - &MPIDI_CH4U_send_origin_cmpl_handler, - &MPIDI_CH4U_send_target_handler); + mpi_errno = MPIDI_NM_am_reg_handler(MPIDI_CH4U_SEND, + &MPIDI_CH4U_send_origin_cmpl_handler, + &MPIDI_CH4U_send_target_handler); if (mpi_errno) MPIR_ERR_POP(mpi_errno); - mpi_errno = MPIDI_NM_reg_hdr_handler(MPIDI_CH4U_SEND_LONG_REQ, NULL /* Injection only */ , - &MPIDI_CH4U_send_long_req_target_handler); + mpi_errno = MPIDI_NM_am_reg_handler(MPIDI_CH4U_SEND_LONG_REQ, NULL /* Injection only */ , + &MPIDI_CH4U_send_long_req_target_handler); if (mpi_errno) MPIR_ERR_POP(mpi_errno); - mpi_errno = MPIDI_NM_reg_hdr_handler(MPIDI_CH4U_SEND_LONG_ACK, NULL /* Injection only */ , - &MPIDI_CH4U_send_long_ack_target_handler); + mpi_errno = MPIDI_NM_am_reg_handler(MPIDI_CH4U_SEND_LONG_ACK, NULL /* Injection only */ , + &MPIDI_CH4U_send_long_ack_target_handler); if (mpi_errno) MPIR_ERR_POP(mpi_errno); - mpi_errno = MPIDI_NM_reg_hdr_handler(MPIDI_CH4U_SEND_LONG_LMT, - &MPIDI_CH4U_send_long_lmt_origin_cmpl_handler, - &MPIDI_CH4U_send_long_lmt_target_handler); + mpi_errno = MPIDI_NM_am_reg_handler(MPIDI_CH4U_SEND_LONG_LMT, + &MPIDI_CH4U_send_long_lmt_origin_cmpl_handler, + &MPIDI_CH4U_send_long_lmt_target_handler); if (mpi_errno) MPIR_ERR_POP(mpi_errno); - mpi_errno = MPIDI_NM_reg_hdr_handler(MPIDI_CH4U_SSEND_REQ, - &MPIDI_CH4U_send_origin_cmpl_handler, - &MPIDI_CH4U_ssend_target_handler); + mpi_errno = MPIDI_NM_am_reg_handler(MPIDI_CH4U_SSEND_REQ, + &MPIDI_CH4U_send_origin_cmpl_handler, + &MPIDI_CH4U_ssend_target_handler); if (mpi_errno) MPIR_ERR_POP(mpi_errno); - mpi_errno = MPIDI_NM_reg_hdr_handler(MPIDI_CH4U_SSEND_ACK, - &MPIDI_CH4U_ssend_ack_origin_cmpl_handler, - &MPIDI_CH4U_ssend_ack_target_handler); + mpi_errno = MPIDI_NM_am_reg_handler(MPIDI_CH4U_SSEND_ACK, + &MPIDI_CH4U_ssend_ack_origin_cmpl_handler, + &MPIDI_CH4U_ssend_ack_target_handler); if (mpi_errno) MPIR_ERR_POP(mpi_errno); - mpi_errno = MPIDI_NM_reg_hdr_handler(MPIDI_CH4U_PUT_REQ, - &MPIDI_CH4U_put_origin_cmpl_handler, - &MPIDI_CH4U_put_target_handler); + mpi_errno = MPIDI_NM_am_reg_handler(MPIDI_CH4U_PUT_REQ, + &MPIDI_CH4U_put_origin_cmpl_handler, + &MPIDI_CH4U_put_target_handler); if (mpi_errno) MPIR_ERR_POP(mpi_errno); - mpi_errno = MPIDI_NM_reg_hdr_handler(MPIDI_CH4U_PUT_ACK, - NULL, &MPIDI_CH4U_put_ack_target_handler); + mpi_errno = MPIDI_NM_am_reg_handler(MPIDI_CH4U_PUT_ACK, + NULL, &MPIDI_CH4U_put_ack_target_handler); if (mpi_errno) MPIR_ERR_POP(mpi_errno); - mpi_errno = MPIDI_NM_reg_hdr_handler(MPIDI_CH4U_GET_REQ, - &MPIDI_CH4U_get_origin_cmpl_handler, - &MPIDI_CH4U_get_target_handler); + mpi_errno = MPIDI_NM_am_reg_handler(MPIDI_CH4U_GET_REQ, + &MPIDI_CH4U_get_origin_cmpl_handler, + &MPIDI_CH4U_get_target_handler); if (mpi_errno) MPIR_ERR_POP(mpi_errno); - mpi_errno = MPIDI_NM_reg_hdr_handler(MPIDI_CH4U_GET_ACK, - &MPIDI_CH4U_get_ack_origin_cmpl_handler, - &MPIDI_CH4U_get_ack_target_handler); + mpi_errno = MPIDI_NM_am_reg_handler(MPIDI_CH4U_GET_ACK, + &MPIDI_CH4U_get_ack_origin_cmpl_handler, + &MPIDI_CH4U_get_ack_target_handler); if (mpi_errno) MPIR_ERR_POP(mpi_errno); - mpi_errno = MPIDI_NM_reg_hdr_handler(MPIDI_CH4U_CSWAP_REQ, - &MPIDI_CH4U_cswap_origin_cmpl_handler, - &MPIDI_CH4U_cswap_target_handler); + mpi_errno = MPIDI_NM_am_reg_handler(MPIDI_CH4U_CSWAP_REQ, + &MPIDI_CH4U_cswap_origin_cmpl_handler, + &MPIDI_CH4U_cswap_target_handler); if (mpi_errno) MPIR_ERR_POP(mpi_errno); - mpi_errno = MPIDI_NM_reg_hdr_handler(MPIDI_CH4U_CSWAP_ACK, - &MPIDI_CH4U_cswap_ack_origin_cmpl_handler, - &MPIDI_CH4U_cswap_ack_target_handler); + mpi_errno = MPIDI_NM_am_reg_handler(MPIDI_CH4U_CSWAP_ACK, + &MPIDI_CH4U_cswap_ack_origin_cmpl_handler, + &MPIDI_CH4U_cswap_ack_target_handler); if (mpi_errno) MPIR_ERR_POP(mpi_errno); - mpi_errno = MPIDI_NM_reg_hdr_handler(MPIDI_CH4U_ACC_REQ, - &MPIDI_CH4U_acc_origin_cmpl_handler, - &MPIDI_CH4U_handle_acc_request); + mpi_errno = MPIDI_NM_am_reg_handler(MPIDI_CH4U_ACC_REQ, + &MPIDI_CH4U_acc_origin_cmpl_handler, + &MPIDI_CH4U_handle_acc_request); if (mpi_errno) MPIR_ERR_POP(mpi_errno); - mpi_errno = MPIDI_NM_reg_hdr_handler(MPIDI_CH4U_ACC_ACK, - NULL, &MPIDI_CH4U_acc_ack_target_handler); + mpi_errno = MPIDI_NM_am_reg_handler(MPIDI_CH4U_ACC_ACK, + NULL, &MPIDI_CH4U_acc_ack_target_handler); if (mpi_errno) MPIR_ERR_POP(mpi_errno); - mpi_errno = MPIDI_NM_reg_hdr_handler(MPIDI_CH4U_GET_ACC_ACK, - &MPIDI_CH4U_get_acc_ack_origin_cmpl_handler, - &MPIDI_CH4U_get_acc_ack_target_handler); + mpi_errno = MPIDI_NM_am_reg_handler(MPIDI_CH4U_GET_ACC_ACK, + &MPIDI_CH4U_get_acc_ack_origin_cmpl_handler, + &MPIDI_CH4U_get_acc_ack_target_handler); if (mpi_errno) MPIR_ERR_POP(mpi_errno); - mpi_errno = MPIDI_NM_reg_hdr_handler(MPIDI_CH4U_WIN_CTRL, - NULL, &MPIDI_CH4U_win_ctrl_target_handler); + mpi_errno = MPIDI_NM_am_reg_handler(MPIDI_CH4U_WIN_CTRL, + NULL, &MPIDI_CH4U_win_ctrl_target_handler); if (mpi_errno) MPIR_ERR_POP(mpi_errno); - mpi_errno = MPIDI_NM_reg_hdr_handler(MPIDI_CH4U_PUT_IOV_REQ, - &MPIDI_CH4U_put_iov_origin_cmpl_handler, - &MPIDI_CH4U_put_iov_target_handler); + mpi_errno = MPIDI_NM_am_reg_handler(MPIDI_CH4U_PUT_IOV_REQ, + &MPIDI_CH4U_put_iov_origin_cmpl_handler, + &MPIDI_CH4U_put_iov_target_handler); if (mpi_errno) MPIR_ERR_POP(mpi_errno); - mpi_errno = MPIDI_NM_reg_hdr_handler(MPIDI_CH4U_PUT_IOV_ACK, - NULL, &MPIDI_CH4U_put_iov_ack_target_handler); + mpi_errno = MPIDI_NM_am_reg_handler(MPIDI_CH4U_PUT_IOV_ACK, + NULL, &MPIDI_CH4U_put_iov_ack_target_handler); if (mpi_errno) MPIR_ERR_POP(mpi_errno); - mpi_errno = MPIDI_NM_reg_hdr_handler(MPIDI_CH4U_PUT_DAT_REQ, - &MPIDI_CH4U_put_data_origin_cmpl_handler, - &MPIDI_CH4U_put_data_target_handler); + mpi_errno = MPIDI_NM_am_reg_handler(MPIDI_CH4U_PUT_DAT_REQ, + &MPIDI_CH4U_put_data_origin_cmpl_handler, + &MPIDI_CH4U_put_data_target_handler); if (mpi_errno) MPIR_ERR_POP(mpi_errno); - mpi_errno = MPIDI_NM_reg_hdr_handler(MPIDI_CH4U_ACC_IOV_REQ, - &MPIDI_CH4U_acc_iov_origin_cmpl_handler, - &MPIDI_CH4U_acc_iov_target_handler); + mpi_errno = MPIDI_NM_am_reg_handler(MPIDI_CH4U_ACC_IOV_REQ, + &MPIDI_CH4U_acc_iov_origin_cmpl_handler, + &MPIDI_CH4U_acc_iov_target_handler); if (mpi_errno) MPIR_ERR_POP(mpi_errno); - mpi_errno = MPIDI_NM_reg_hdr_handler(MPIDI_CH4U_ACC_IOV_ACK, - NULL, &MPIDI_CH4U_acc_iov_ack_target_handler); + mpi_errno = MPIDI_NM_am_reg_handler(MPIDI_CH4U_ACC_IOV_ACK, + NULL, &MPIDI_CH4U_acc_iov_ack_target_handler); if (mpi_errno) MPIR_ERR_POP(mpi_errno); - mpi_errno = MPIDI_NM_reg_hdr_handler(MPIDI_CH4U_ACC_DAT_REQ, - &MPIDI_CH4U_acc_data_origin_cmpl_handler, - &MPIDI_CH4U_acc_data_target_handler); + mpi_errno = MPIDI_NM_am_reg_handler(MPIDI_CH4U_ACC_DAT_REQ, + &MPIDI_CH4U_acc_data_origin_cmpl_handler, + &MPIDI_CH4U_acc_data_target_handler); if (mpi_errno) MPIR_ERR_POP(mpi_errno); diff --git a/src/mpid/ch4/src/ch4r_rma.h b/src/mpid/ch4/src/ch4r_rma.h index b0411db..1277ff5 100644 --- a/src/mpid/ch4/src/ch4r_rma.h +++ b/src/mpid/ch4/src/ch4r_rma.h @@ -81,7 +81,7 @@ static inline int MPIDI_CH4I_do_put(const void *origin_addr, am_hdr.n_iov = 0; MPIDI_CH4U_REQUEST(sreq, req->preq.dt_iov) = NULL; - mpi_errno = MPIDI_NM_send_am(target_rank, win->comm_ptr, MPIDI_CH4U_PUT_REQ, + mpi_errno = MPIDI_NM_am_send(target_rank, win->comm_ptr, MPIDI_CH4U_PUT_REQ, &am_hdr, sizeof(am_hdr), origin_addr, origin_count, origin_datatype, sreq, NULL); if (mpi_errno) @@ -114,7 +114,7 @@ static inline int MPIDI_CH4I_do_put(const void *origin_addr, MPIDI_CH4U_REQUEST(sreq, req->preq.dt_iov) = dt_iov; if ((am_iov[0].iov_len + am_iov[1].iov_len) <= MPIDI_NM_am_hdr_max_sz()) { - mpi_errno = MPIDI_NM_send_amv(target_rank, win->comm_ptr, MPIDI_CH4U_PUT_REQ, + mpi_errno = MPIDI_NM_am_sendv(target_rank, win->comm_ptr, MPIDI_CH4U_PUT_REQ, &am_iov[0], 2, origin_addr, origin_count, origin_datatype, sreq, NULL); } @@ -125,7 +125,7 @@ static inline int MPIDI_CH4I_do_put(const void *origin_addr, MPIDI_CH4U_REQUEST(sreq, src_rank) = target_rank; dtype_add_ref_if_not_builtin(origin_datatype); - mpi_errno = MPIDI_NM_send_am(target_rank, win->comm_ptr, MPIDI_CH4U_PUT_IOV_REQ, + mpi_errno = MPIDI_NM_am_send(target_rank, win->comm_ptr, MPIDI_CH4U_PUT_IOV_REQ, &am_hdr, sizeof(am_hdr), am_iov[1].iov_base, am_iov[1].iov_len, MPI_BYTE, sreq, NULL); } @@ -210,7 +210,7 @@ static inline int MPIDI_CH4I_do_get(void *origin_addr, am_hdr.n_iov = 0; MPIDI_CH4U_REQUEST(sreq, req->greq.dt_iov) = NULL; - mpi_errno = MPIDI_NM_send_am_hdr(target_rank, win->comm_ptr, + mpi_errno = MPIDI_NM_am_send_hdr(target_rank, win->comm_ptr, MPIDI_CH4U_GET_REQ, &am_hdr, sizeof(am_hdr), sreq, NULL); if (mpi_errno) MPIR_ERR_POP(mpi_errno); @@ -235,7 +235,7 @@ static inline int MPIDI_CH4I_do_get(void *origin_addr, MPL_free(segment_ptr); MPIDI_CH4U_REQUEST(sreq, req->greq.dt_iov) = dt_iov; - mpi_errno = MPIDI_NM_send_am(target_rank, win->comm_ptr, MPIDI_CH4U_GET_REQ, + mpi_errno = MPIDI_NM_am_send(target_rank, win->comm_ptr, MPIDI_CH4U_GET_REQ, &am_hdr, sizeof(am_hdr), dt_iov, sizeof(struct iovec) * am_hdr.n_iov, MPI_BYTE, sreq, NULL); if (mpi_errno) @@ -441,7 +441,7 @@ __CH4_INLINE__ int MPIDI_CH4I_do_accumulate(const void *origin_addr, am_hdr.n_iov = 0; MPIDI_CH4U_REQUEST(sreq, req->areq.dt_iov) = NULL; - mpi_errno = MPIDI_NM_send_am(target_rank, win->comm_ptr, MPIDI_CH4U_ACC_REQ, + mpi_errno = MPIDI_NM_am_send(target_rank, win->comm_ptr, MPIDI_CH4U_ACC_REQ, &am_hdr, sizeof(am_hdr), origin_addr, (op == MPI_NO_OP) ? 0 : origin_count, origin_datatype, sreq, NULL); @@ -479,7 +479,7 @@ __CH4_INLINE__ int MPIDI_CH4I_do_accumulate(const void *origin_addr, MPIDI_CH4U_REQUEST(sreq, req->areq.dt_iov) = dt_iov; if ((am_iov[0].iov_len + am_iov[1].iov_len) <= MPIDI_NM_am_hdr_max_sz()) { - mpi_errno = MPIDI_NM_send_amv(target_rank, win->comm_ptr, MPIDI_CH4U_ACC_REQ, + mpi_errno = MPIDI_NM_am_sendv(target_rank, win->comm_ptr, MPIDI_CH4U_ACC_REQ, &am_iov[0], 2, origin_addr, (op == MPI_NO_OP) ? 0 : origin_count, origin_datatype, sreq, NULL); @@ -491,7 +491,7 @@ __CH4_INLINE__ int MPIDI_CH4I_do_accumulate(const void *origin_addr, MPIDI_CH4U_REQUEST(sreq, src_rank) = target_rank; dtype_add_ref_if_not_builtin(origin_datatype); - mpi_errno = MPIDI_NM_send_am(target_rank, win->comm_ptr, MPIDI_CH4U_ACC_IOV_REQ, + mpi_errno = MPIDI_NM_am_send(target_rank, win->comm_ptr, MPIDI_CH4U_ACC_IOV_REQ, &am_hdr, sizeof(am_hdr), am_iov[1].iov_base, am_iov[1].iov_len, MPI_BYTE, sreq, NULL); } @@ -721,7 +721,7 @@ __CH4_INLINE__ int MPIDI_CH4U_compare_and_swap(const void *origin_addr, OPA_incr_int(&MPIDI_CH4U_WIN(win, outstanding_ops)); /* MPIDI_CS_EXIT(); */ - mpi_errno = MPIDI_NM_send_am(target_rank, win->comm_ptr, MPIDI_CH4U_CSWAP_REQ, + mpi_errno = MPIDI_NM_am_send(target_rank, win->comm_ptr, MPIDI_CH4U_CSWAP_REQ, &am_hdr, sizeof(am_hdr), (char *) p_data, 2, datatype, sreq, NULL); if (mpi_errno) MPIR_ERR_POP(mpi_errno); diff --git a/src/mpid/ch4/src/ch4r_send.h b/src/mpid/ch4/src/ch4r_send.h index 84a67e3..30f4f4e 100644 --- a/src/mpid/ch4/src/ch4r_send.h +++ b/src/mpid/ch4/src/ch4r_send.h @@ -49,14 +49,14 @@ static inline int MPIDI_CH4I_do_send(const void *buf, ssend_req.sreq_ptr = (uint64_t) sreq; MPIR_cc_incr(sreq->cc_ptr, &c); - mpi_errno = MPIDI_NM_send_am(rank, comm, MPIDI_CH4U_SSEND_REQ, + mpi_errno = MPIDI_NM_am_send(rank, comm, MPIDI_CH4U_SSEND_REQ, &ssend_req, sizeof(ssend_req), buf, count, datatype, sreq, NULL); if (mpi_errno) MPIR_ERR_POP(mpi_errno); } else { - mpi_errno = MPIDI_NM_send_am(rank, comm, MPIDI_CH4U_SEND, + mpi_errno = MPIDI_NM_am_send(rank, comm, MPIDI_CH4U_SEND, &am_hdr, sizeof(am_hdr), buf, count, datatype, sreq, NULL); if (mpi_errno) MPIR_ERR_POP(mpi_errno); diff --git a/src/mpid/ch4/src/ch4r_win.h b/src/mpid/ch4/src/ch4r_win.h index 833066c..4bcd90d 100644 --- a/src/mpid/ch4/src/ch4r_win.h +++ b/src/mpid/ch4/src/ch4r_win.h @@ -284,7 +284,7 @@ static inline int MPIDI_CH4R_win_complete(MPIR_Win * win) for (index = 0; index < group->size; ++index) { peer = ranks_in_win_grp[index]; - mpi_errno = MPIDI_NM_inject_am_hdr(peer, win->comm_ptr, + mpi_errno = MPIDI_NM_am_inject_hdr(peer, win->comm_ptr, MPIDI_CH4U_WIN_CTRL, &msg, sizeof(msg), NULL); if (mpi_errno != MPI_SUCCESS) MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, goto fn_fail, "**rmasync"); @@ -338,7 +338,7 @@ static inline int MPIDI_CH4R_win_post(MPIR_Group * group, int assert, MPIR_Win * for (index = 0; index < group->size; ++index) { peer = ranks_in_win_grp[index]; - mpi_errno = MPIDI_NM_inject_am_hdr(peer, win->comm_ptr, + mpi_errno = MPIDI_NM_am_inject_hdr(peer, win->comm_ptr, MPIDI_CH4U_WIN_CTRL, &msg, sizeof(msg), NULL); if (mpi_errno != MPI_SUCCESS) MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, goto fn_fail, "**rmasync"); @@ -440,7 +440,7 @@ static inline int MPIDI_CH4R_win_lock(int lock_type, int rank, int assert, MPIR_ msg.lock_type = lock_type; locked = slock->remote.locked + 1; - mpi_errno = MPIDI_NM_inject_am_hdr(rank, win->comm_ptr, + mpi_errno = MPIDI_NM_am_inject_hdr(rank, win->comm_ptr, MPIDI_CH4U_WIN_CTRL, &msg, sizeof(msg), NULL); if (mpi_errno != MPI_SUCCESS) MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, goto fn_fail, "**rmasync"); @@ -482,7 +482,7 @@ static inline int MPIDI_CH4R_win_unlock(int rank, MPIR_Win * win) msg.type = MPIDI_CH4U_WIN_UNLOCK; unlocked = MPIDI_CH4U_WIN(win, sync).lock.remote.locked - 1; - mpi_errno = MPIDI_NM_inject_am_hdr(rank, win->comm_ptr, + mpi_errno = MPIDI_NM_am_inject_hdr(rank, win->comm_ptr, MPIDI_CH4U_WIN_CTRL, &msg, sizeof(msg), NULL); if (mpi_errno != MPI_SUCCESS) MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, goto fn_fail, "**rmasync"); @@ -1066,7 +1066,7 @@ static inline int MPIDI_CH4R_win_unlock_all(MPIR_Win * win) lockQ[i].peer = i; lockQ[i].win = win; - mpi_errno = MPIDI_NM_inject_am_hdr(i, win->comm_ptr, + mpi_errno = MPIDI_NM_am_inject_hdr(i, win->comm_ptr, MPIDI_CH4U_WIN_CTRL, &msg, sizeof(msg), NULL); if (mpi_errno != MPI_SUCCESS) MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, goto fn_fail, "**rmasync"); @@ -1228,7 +1228,7 @@ static inline int MPIDI_CH4R_win_lock_all(int assert, MPIR_Win * win) lockQ[i].win = win; lockQ[i].lock_type = MPI_LOCK_SHARED; - mpi_errno = MPIDI_NM_inject_am_hdr(i, win->comm_ptr, + mpi_errno = MPIDI_NM_am_inject_hdr(i, win->comm_ptr, MPIDI_CH4U_WIN_CTRL, &msg, sizeof(msg), NULL); if (mpi_errno != MPI_SUCCESS) MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, goto fn_fail, "**rmasync"); http://git.mpich.org/mpich.git/commitdiff/02cae1085c508fe494a14970e9dce541e… commit 02cae1085c508fe494a14970e9dce541ee3c0c07 Author: Pavan Balaji <balaji(a)anl.gov> Date: Thu Aug 18 12:23:14 2016 -0500 CH4: Cleanup comm, type, op, and attr hooks Name functions similar to their MPI counterparts, and use _hooks in the names to make it clear that these are just hooks that get called from the upper layer. Signed-off-by: Ken Raffenetti <raffenet(a)mcs.anl.gov> diff --git a/src/include/mpir_attr_generic.h b/src/include/mpir_attr_generic.h index 72d7aeb..07fbdd8 100644 --- a/src/include/mpir_attr_generic.h +++ b/src/include/mpir_attr_generic.h @@ -81,7 +81,7 @@ * ADI defines a collection of routines that are used by the implementation * of the MPI attribute routines (such as 'MPI_Comm_get_attr'). * In addition, the MPI routines involving attributes will invoke the - * corresponding 'hook' functions (e.g., 'MPID_Dev_comm_attr_set_hook') + * corresponding 'hook' functions (e.g., 'MPID_Comm_attr_hook') * should the device define them. * * Attributes on windows and datatypes are defined by MPI but not of diff --git a/src/mpi/attr/comm_set_attr.c b/src/mpi/attr/comm_set_attr.c index db47c40..c66a495 100644 --- a/src/mpi/attr/comm_set_attr.c +++ b/src/mpi/attr/comm_set_attr.c @@ -91,7 +91,7 @@ int MPIR_Comm_set_attr_impl(MPIR_Comm *comm_ptr, int comm_keyval, void *attribut /* Here is where we could add a hook for the device to detect attribute value changes, using something like - MPID_Dev_comm_attr_hook( comm_ptr, keyval, attribute_val ); + MPID_Comm_attr_hook( comm_ptr, keyval, attribute_val ); */ diff --git a/src/mpi/attr/type_set_attr.c b/src/mpi/attr/type_set_attr.c index 0673df9..b9d2b1d 100644 --- a/src/mpi/attr/type_set_attr.c +++ b/src/mpi/attr/type_set_attr.c @@ -134,7 +134,7 @@ int MPII_Type_set_attr(MPI_Datatype datatype, int type_keyval, void *attribute_v /* Here is where we could add a hook for the device to detect attribute value changes, using something like - MPID_Dev_type_attr_hook( type_ptr, keyval, attribute_val ); + MPID_Type_attr_hook( type_ptr, keyval, attribute_val ); */ /* ... end of body of routine ... */ diff --git a/src/mpi/attr/win_set_attr.c b/src/mpi/attr/win_set_attr.c index d9fa7c7..ed4f9e5 100644 --- a/src/mpi/attr/win_set_attr.c +++ b/src/mpi/attr/win_set_attr.c @@ -140,7 +140,7 @@ int MPII_Win_set_attr( MPI_Win win, int win_keyval, void *attribute_val, /* Here is where we could add a hook for the device to detect attribute value changes, using something like - MPID_Dev_win_attr_hook( win_ptr, keyval, attribute_val ); + MPID_Win_attr_hook( win_ptr, keyval, attribute_val ); */ /* ... end of body of routine ... */ diff --git a/src/mpi/coll/op_create.c b/src/mpi/coll/op_create.c index cd2cae5..6089172 100644 --- a/src/mpi/coll/op_create.c +++ b/src/mpi/coll/op_create.c @@ -131,8 +131,8 @@ int MPI_Op_create(MPI_User_function *user_fn, int commute, MPI_Op *op) MPIR_OBJ_PUBLISH_HANDLE(*op, op_ptr->handle); -#ifdef MPID_Dev_op_commit_hook - MPID_Dev_op_commit_hook(op_ptr); +#ifdef MPID_Op_commit_hook + MPID_Op_commit_hook(op_ptr); #endif /* ... end of body of routine ... */ diff --git a/src/mpi/coll/op_free.c b/src/mpi/coll/op_free.c index 2d013ab..4872e66 100644 --- a/src/mpi/coll/op_free.c +++ b/src/mpi/coll/op_free.c @@ -91,8 +91,8 @@ int MPI_Op_free(MPI_Op *op) MPIR_Op_release_ref( op_ptr, &in_use); if (!in_use) { MPIR_Handle_obj_free( &MPIR_Op_mem, op_ptr ); -#ifdef MPID_Dev_op_destroy_hook - MPID_Dev_op_destroy_hook(op_ptr); +#ifdef MPID_Op_free_hook + MPID_Op_free_hook(op_ptr); #endif } *op = MPI_OP_NULL; diff --git a/src/mpi/comm/commutil.c b/src/mpi/comm/commutil.c index c365612..10840bc 100644 --- a/src/mpi/comm/commutil.c +++ b/src/mpi/comm/commutil.c @@ -543,7 +543,7 @@ int MPIR_Comm_commit(MPIR_Comm * comm) MPIR_ERR_POP(mpi_errno); /* Notify device of communicator creation */ - mpi_errno = MPID_Dev_comm_create_hook(comm); + mpi_errno = MPID_Comm_create_hook(comm); if (mpi_errno) MPIR_ERR_POP(mpi_errno); @@ -612,7 +612,7 @@ int MPIR_Comm_commit(MPIR_Comm * comm) MPIR_ERR_POP(mpi_errno); /* Notify device of communicator creation */ - mpi_errno = MPID_Dev_comm_create_hook(comm->node_comm); + mpi_errno = MPID_Comm_create_hook(comm->node_comm); if (mpi_errno) MPIR_ERR_POP(mpi_errno); /* don't call MPIR_Comm_commit here */ @@ -645,7 +645,7 @@ int MPIR_Comm_commit(MPIR_Comm * comm) MPIR_ERR_POP(mpi_errno); /* Notify device of communicator creation */ - mpi_errno = MPID_Dev_comm_create_hook(comm->node_roots_comm); + mpi_errno = MPID_Comm_create_hook(comm->node_roots_comm); if (mpi_errno) MPIR_ERR_POP(mpi_errno); /* don't call MPIR_Comm_commit here */ @@ -953,7 +953,7 @@ int MPIR_Comm_delete_internal(MPIR_Comm * comm_ptr) /* Notify the device that the communicator is about to be * destroyed */ - mpi_errno = MPID_Dev_comm_destroy_hook(comm_ptr); + mpi_errno = MPID_Comm_free_hook(comm_ptr); if (mpi_errno) MPIR_ERR_POP(mpi_errno); diff --git a/src/mpid/ch3/include/mpidpre.h b/src/mpid/ch3/include/mpidpre.h index 7b05e75..926612e 100644 --- a/src/mpid/ch3/include/mpidpre.h +++ b/src/mpid/ch3/include/mpidpre.h @@ -163,8 +163,8 @@ typedef union { * by the channel instance. */ -#define MPID_Dev_comm_create_hook(comm_) MPIDI_CH3I_Comm_create_hook(comm_) -#define MPID_Dev_comm_destroy_hook(comm_) MPIDI_CH3I_Comm_destroy_hook(comm_) +#define MPID_Comm_create_hook(comm_) MPIDI_CH3I_Comm_create_hook(comm_) +#define MPID_Comm_free_hook(comm_) MPIDI_CH3I_Comm_destroy_hook(comm_) #ifndef HAVE_MPIDI_VCRT #define HAVE_MPIDI_VCRT diff --git a/src/mpid/ch3/src/ch3u_port.c b/src/mpid/ch3/src/ch3u_port.c index ac38034..a24d350 100644 --- a/src/mpid/ch3/src/ch3u_port.c +++ b/src/mpid/ch3/src/ch3u_port.c @@ -301,7 +301,7 @@ static int MPIDI_CH3I_Initialize_tmp_comm(MPIR_Comm **comm_pptr, /* Even though this is a tmp comm and we don't call MPI_Comm_commit, we still need to call the creation hook because the destruction hook will be called in comm_release */ - mpi_errno = MPID_Dev_comm_create_hook(tmp_comm); + mpi_errno = MPID_Comm_create_hook(tmp_comm); if (mpi_errno) MPIR_ERR_POP(mpi_errno); *comm_pptr = tmp_comm; diff --git a/src/mpid/ch4/include/mpidch4.h b/src/mpid/ch4/include/mpidch4.h index 8110d81..7c408bd 100644 --- a/src/mpid/ch4/include/mpidch4.h +++ b/src/mpid/ch4/include/mpidch4.h @@ -374,8 +374,6 @@ __CH4_INLINE__ int MPIDI_CH4_rank_is_local(int rank, MPIR_Comm * comm); #define MPID_GPID_ToLpidArray MPIDI_GPID_ToLpidArray #define MPID_Create_intercomm_from_lpids MPIDI_Create_intercomm_from_lpids /* Variables */ -#define MPID_Comm_create MPIDI_Comm_create -#define MPID_Comm_destroy MPIDI_Comm_destroy #define MPID_Barrier MPIDI_Barrier #define MPID_Bcast MPIDI_Bcast #define MPID_Allreduce MPIDI_Allreduce diff --git a/src/mpid/ch4/include/mpidpre.h b/src/mpid/ch4/include/mpidpre.h index 6fdd3f8..9990fb1 100644 --- a/src/mpid/ch4/include/mpidpre.h +++ b/src/mpid/ch4/include/mpidpre.h @@ -435,14 +435,14 @@ extern MPIDII_av_table_t *MPIDII_av_table0; #define MPID_Progress_deactivate_hook(id_) MPID_Progress_deactivate(id_) #define HAVE_DEV_COMM_HOOK -#define MPID_Dev_comm_create_hook(a) (MPID_Comm_create(a)) -#define MPID_Dev_comm_destroy_hook(a) (MPID_Comm_destroy(a)) +#define MPID_Comm_create_hook MPIDI_Comm_create_hook +#define MPID_Comm_free_hook MPIDI_Comm_free_hook -#define MPID_Dev_datatype_commit_hook MPIDI_NM_datatype_commit -#define MPID_Dev_datatype_destroy_hook MPIDI_NM_datatype_destroy +#define MPID_Type_create_hook MPIDI_Type_create_hook +#define MPID_Type_free_hook MPIDI_Type_free_hook -#define MPID_Dev_op_commit_hook MPIDI_NM_op_commit -#define MPID_Dev_op_destroy_hook MPIDI_NM_op_destroy +#define MPID_Op_create_hook MPIDI_Op_create_hook +#define MPID_Op_free_hook MPIDI_Op_free_hook /* operation for (avtid, lpid) to/from "lpid64" */ /* hard code limit on number of live comm worlds. This should be fixed by future diff --git a/src/mpid/ch4/netmod/include/netmod.h b/src/mpid/ch4/netmod/include/netmod.h index 6a5cbc3..1225705 100644 --- a/src/mpid/ch4/netmod/include/netmod.h +++ b/src/mpid/ch4/netmod/include/netmod.h @@ -82,8 +82,8 @@ typedef int (*MPIDI_NM_getallincomm_t) (MPIR_Comm * comm_ptr, int local_size, typedef int (*MPIDI_NM_gpid_tolpidarray_t) (int size, MPIR_Gpid gpid[], int lpid[]); typedef int (*MPIDI_NM_create_intercomm_from_lpids_t) (MPIR_Comm * newcomm_ptr, int size, const int lpids[]); -typedef int (*MPIDI_NM_comm_create_t) (MPIR_Comm * comm); -typedef int (*MPIDI_NM_comm_destroy_t) (MPIR_Comm * comm); +typedef int (*MPIDI_NM_comm_create_hook_t) (MPIR_Comm * comm); +typedef int (*MPIDI_NM_comm_free_hook_t) (MPIR_Comm * comm); typedef void (*MPIDI_NM_am_request_init_t) (MPIR_Request * req); typedef void (*MPIDI_NM_am_request_finalize_t) (MPIR_Request * req); typedef int (*MPIDI_NM_send_t) (const void *buf, int count, MPI_Datatype datatype, int rank, @@ -350,12 +350,12 @@ typedef int (*MPIDI_NM_iscatterv_t) (const void *sendbuf, const int *sendcounts, MPI_Datatype sendtype, void *recvbuf, int recvcount, MPI_Datatype recvtype, int root, MPIR_Comm * comm_ptr, MPI_Request * req); -typedef void (*MPIDI_NM_datatype_commit_t) (MPIR_Datatype * datatype_p); -typedef void (*MPIDI_NM_datatype_dup_t) (MPIR_Datatype * old_datatype_p, - MPIR_Datatype * new_datatype_p); -typedef void (*MPIDI_NM_datatype_destroy_t) (MPIR_Datatype * datatype_p); -typedef void (*MPIDI_NM_op_commit_t) (MPIR_Op * op_p); -typedef void (*MPIDI_NM_op_destroy_t) (MPIR_Op * op_p); +typedef void (*MPIDI_NM_type_dup_hook_t) (MPIR_Datatype * old_datatype_p, + MPIR_Datatype * new_datatype_p); +typedef int (*MPIDI_NM_type_create_hook_t) (MPIR_Datatype * datatype_p); +typedef int (*MPIDI_NM_type_free_hook_t) (MPIR_Datatype * datatype_p); +typedef int (*MPIDI_NM_op_create_hook_t) (MPIR_Op * op_p); +typedef int (*MPIDI_NM_op_free_hook_t) (MPIR_Op * op_p); typedef struct MPIDI_NM_funcs { MPIDI_NM_init_t init; @@ -372,8 +372,8 @@ typedef struct MPIDI_NM_funcs { MPIDI_NM_getallincomm_t getallincomm; MPIDI_NM_gpid_tolpidarray_t gpid_tolpidarray; MPIDI_NM_create_intercomm_from_lpids_t create_intercomm_from_lpids; - MPIDI_NM_comm_create_t comm_create; - MPIDI_NM_comm_destroy_t comm_destroy; + MPIDI_NM_comm_create_hook_t comm_create_hook; + MPIDI_NM_comm_free_hook_t comm_free_hook; /* Request allocation routines */ MPIDI_NM_am_request_init_t am_request_init; MPIDI_NM_am_request_finalize_t am_request_finalize; @@ -494,12 +494,12 @@ typedef struct MPIDI_NM_native_funcs { MPIDI_NM_iscatter_t iscatter; MPIDI_NM_iscatterv_t iscatterv; /* Datatype hooks */ - MPIDI_NM_datatype_commit_t datatype_commit; - MPIDI_NM_datatype_dup_t datatype_dup; - MPIDI_NM_datatype_destroy_t datatype_destroy; + MPIDI_NM_type_create_hook_t type_create_hook; + MPIDI_NM_type_dup_hook_t type_dup_hook; + MPIDI_NM_type_free_hook_t type_free_hook; /* Op hooks */ - MPIDI_NM_op_commit_t op_commit; - MPIDI_NM_op_destroy_t op_destroy; + MPIDI_NM_op_create_hook_t op_create_hook; + MPIDI_NM_op_free_hook_t op_free_hook; } MPIDI_NM_native_funcs_t; extern MPIDI_NM_funcs_t *MPIDI_NM_funcs[]; @@ -592,8 +592,8 @@ MPL_STATIC_INLINE_PREFIX int MPIDI_NM_create_intercomm_from_lpids(MPIR_Comm * ne int size, const int lpids[]) MPL_STATIC_INLINE_SUFFIX; -MPL_STATIC_INLINE_PREFIX int MPIDI_NM_comm_create(MPIR_Comm * comm) MPL_STATIC_INLINE_SUFFIX; -MPL_STATIC_INLINE_PREFIX int MPIDI_NM_comm_destroy(MPIR_Comm * comm) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_comm_create_hook(MPIR_Comm * comm) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_comm_free_hook(MPIR_Comm * comm) MPL_STATIC_INLINE_SUFFIX; MPL_STATIC_INLINE_PREFIX void MPIDI_NM_am_request_init(MPIR_Request * req) MPL_STATIC_INLINE_SUFFIX; MPL_STATIC_INLINE_PREFIX void MPIDI_NM_am_request_finalize(MPIR_Request * req) MPL_STATIC_INLINE_SUFFIX; @@ -1004,14 +1004,14 @@ MPL_STATIC_INLINE_PREFIX int MPIDI_NM_iscatterv(const void *sendbuf, const int * MPI_Datatype recvtype, int root, MPIR_Comm * comm_ptr, MPI_Request * req) MPL_STATIC_INLINE_SUFFIX; -MPL_STATIC_INLINE_PREFIX void MPIDI_NM_datatype_commit(MPIR_Datatype * +MPL_STATIC_INLINE_PREFIX void MPIDI_NM_type_dup_hook(MPIR_Datatype * old_datatype_p, + MPIR_Datatype * + new_datatype_p) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_type_create_hook(MPIR_Datatype * datatype_p) MPL_STATIC_INLINE_SUFFIX; -MPL_STATIC_INLINE_PREFIX void MPIDI_NM_datatype_dup(MPIR_Datatype * old_datatype_p, - MPIR_Datatype * - new_datatype_p) MPL_STATIC_INLINE_SUFFIX; -MPL_STATIC_INLINE_PREFIX void MPIDI_NM_datatype_destroy(MPIR_Datatype * - datatype_p) MPL_STATIC_INLINE_SUFFIX; -MPL_STATIC_INLINE_PREFIX void MPIDI_NM_op_commit(MPIR_Op * op_p) MPL_STATIC_INLINE_SUFFIX; -MPL_STATIC_INLINE_PREFIX void MPIDI_NM_op_destroy(MPIR_Op * op_p) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_type_free_hook(MPIR_Datatype * + datatype_p) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_op_create_hook(MPIR_Op * op_p) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_op_free_hook(MPIR_Op * op_p) MPL_STATIC_INLINE_SUFFIX; #endif diff --git a/src/mpid/ch4/netmod/include/netmod_impl.h b/src/mpid/ch4/netmod/include/netmod_impl.h index 5c3f221..7b50bfb 100644 --- a/src/mpid/ch4/netmod/include/netmod_impl.h +++ b/src/mpid/ch4/netmod/include/netmod_impl.h @@ -189,14 +189,14 @@ MPL_STATIC_INLINE_PREFIX int MPIDI_NM_create_intercomm_from_lpids(MPIR_Comm * ne return MPIDI_NM_func->create_intercomm_from_lpids(newcomm_ptr, size, lpids); }; -MPL_STATIC_INLINE_PREFIX int MPIDI_NM_comm_create(MPIR_Comm * comm) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_comm_create_hook(MPIR_Comm * comm) { - return MPIDI_NM_func->comm_create(comm); + return MPIDI_NM_func->comm_create_hook(comm); }; -MPL_STATIC_INLINE_PREFIX int MPIDI_NM_comm_destroy(MPIR_Comm * comm) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_comm_free_hook(MPIR_Comm * comm) { - return MPIDI_NM_func->comm_destroy(comm); + return MPIDI_NM_func->comm_free_hook(comm); }; MPL_STATIC_INLINE_PREFIX void MPIDI_NM_am_request_init(MPIR_Request * req) @@ -998,30 +998,30 @@ MPL_STATIC_INLINE_PREFIX int MPIDI_NM_iscatterv(const void *sendbuf, const int * recvcount, recvtype, root, comm_ptr, req); }; -MPL_STATIC_INLINE_PREFIX void MPIDI_NM_datatype_commit(MPIR_Datatype * datatype_p) +MPL_STATIC_INLINE_PREFIX void MPIDI_NM_type_dup_hook(MPIR_Datatype * old_datatype_p, + MPIR_Datatype * new_datatype_p) { - return MPIDI_NM_native_func->datatype_commit(datatype_p); + return MPIDI_NM_native_func->type_dup_hook(old_datatype_p, new_datatype_p); }; -MPL_STATIC_INLINE_PREFIX void MPIDI_NM_datatype_dup(MPIR_Datatype * old_datatype_p, - MPIR_Datatype * new_datatype_p) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_type_create_hook(MPIR_Datatype * datatype_p) { - return MPIDI_NM_native_func->datatype_dup(old_datatype_p, new_datatype_p); + return MPIDI_NM_native_func->type_create_hook(datatype_p); }; -MPL_STATIC_INLINE_PREFIX void MPIDI_NM_datatype_destroy(MPIR_Datatype * datatype_p) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_type_free_hook(MPIR_Datatype * datatype_p) { - return MPIDI_NM_native_func->datatype_destroy(datatype_p); + return MPIDI_NM_native_func->type_free_hook(datatype_p); }; -MPL_STATIC_INLINE_PREFIX void MPIDI_NM_op_commit(MPIR_Op * op_p) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_op_create_hook(MPIR_Op * op_p) { - return MPIDI_NM_native_func->op_commit(op_p); + return MPIDI_NM_native_func->op_create_hook(op_p); }; -MPL_STATIC_INLINE_PREFIX void MPIDI_NM_op_destroy(MPIR_Op * op_p) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_op_free_hook(MPIR_Op * op_p) { - return MPIDI_NM_native_func->op_destroy(op_p); + return MPIDI_NM_native_func->op_free_hook(op_p); }; #endif /* NETMOD_DISABLE_INLINES */ diff --git a/src/mpid/ch4/netmod/ofi/func_table.c b/src/mpid/ch4/netmod/ofi/func_table.c index f8a54a4..dff22ff 100644 --- a/src/mpid/ch4/netmod/ofi/func_table.c +++ b/src/mpid/ch4/netmod/ofi/func_table.c @@ -27,8 +27,8 @@ MPIDI_NM_funcs_t MPIDI_NM_ofi_funcs = { MPIDI_NM_getallincomm, MPIDI_NM_gpid_tolpidarray, MPIDI_NM_create_intercomm_from_lpids, - MPIDI_NM_comm_create, - MPIDI_NM_comm_destroy, + MPIDI_NM_comm_create_hook, + MPIDI_NM_comm_free_hook, MPIDI_NM_am_request_init, MPIDI_NM_am_request_finalize, MPIDI_NM_reg_hdr_handler, @@ -145,10 +145,10 @@ MPIDI_NM_native_funcs_t MPIDI_NM_native_ofi_funcs = { MPIDI_NM_iscan, MPIDI_NM_iscatter, MPIDI_NM_iscatterv, - MPIDI_NM_datatype_commit, - MPIDI_NM_datatype_dup, - MPIDI_NM_datatype_destroy, - MPIDI_NM_op_commit, - MPIDI_NM_op_destroy, + MPIDI_NM_type_create_hook, + MPIDI_NM_type_dup_hook, + MPIDI_NM_type_free_hook, + MPIDI_NM_op_create_hook, + MPIDI_NM_op_free_hook, }; #endif diff --git a/src/mpid/ch4/netmod/ofi/ofi_comm.h b/src/mpid/ch4/netmod/ofi/ofi_comm.h index 9f540c9..bf17dab 100644 --- a/src/mpid/ch4/netmod/ofi/ofi_comm.h +++ b/src/mpid/ch4/netmod/ofi/ofi_comm.h @@ -15,10 +15,10 @@ #include "mpl_utlist.h" #undef FUNCNAME -#define FUNCNAME MPIDI_NM_comm_create +#define FUNCNAME MPIDI_NM_comm_create_hook #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_NM_comm_create(MPIR_Comm * comm) +static inline int MPIDI_NM_comm_create_hook(MPIR_Comm * comm) { int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_COMM_CREATE); @@ -42,10 +42,10 @@ static inline int MPIDI_NM_comm_create(MPIR_Comm * comm) } #undef FUNCNAME -#define FUNCNAME MPIDI_NM_comm_destroy +#define FUNCNAME MPIDI_NM_comm_free_hook #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_NM_comm_destroy(MPIR_Comm * comm) +static inline int MPIDI_NM_comm_free_hook(MPIR_Comm * comm) { int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_COMM_DESTROY); diff --git a/src/mpid/ch4/netmod/ofi/ofi_datatype.h b/src/mpid/ch4/netmod/ofi/ofi_datatype.h index 37ddc69..8e110ca 100644 --- a/src/mpid/ch4/netmod/ofi/ofi_datatype.h +++ b/src/mpid/ch4/netmod/ofi/ofi_datatype.h @@ -13,18 +13,18 @@ #include "ofi_impl.h" -static inline void MPIDI_NM_datatype_destroy(MPIR_Datatype * datatype_p) +static inline int MPIDI_NM_type_free_hook(MPIR_Datatype * datatype_p) { - return; + return 0; } -static inline void MPIDI_NM_datatype_commit(MPIR_Datatype * datatype_p) +static inline int MPIDI_NM_type_create_hook(MPIR_Datatype * datatype_p) { - return; + return 0; } -static inline void MPIDI_NM_datatype_dup(MPIR_Datatype * old_datatype_p, - MPIR_Datatype * new_datatype_p) +static inline void MPIDI_NM_type_dup_hook(MPIR_Datatype * old_datatype_p, + MPIR_Datatype * new_datatype_p) { return; } diff --git a/src/mpid/ch4/netmod/ofi/ofi_op.h b/src/mpid/ch4/netmod/ofi/ofi_op.h index 7666d25..7fb4904 100644 --- a/src/mpid/ch4/netmod/ofi/ofi_op.h +++ b/src/mpid/ch4/netmod/ofi/ofi_op.h @@ -13,14 +13,14 @@ #include "ofi_impl.h" -static inline void MPIDI_NM_op_destroy(MPIR_Op * op_p) +static inline int MPIDI_NM_op_free_hook(MPIR_Op * op_p) { - return; + return 0; } -static inline void MPIDI_NM_op_commit(MPIR_Op * op_p) +static inline int MPIDI_NM_op_create_hook(MPIR_Op * op_p) { - return; + return 0; } diff --git a/src/mpid/ch4/netmod/portals4/func_table.c b/src/mpid/ch4/netmod/portals4/func_table.c index c6e9334..4511394 100644 --- a/src/mpid/ch4/netmod/portals4/func_table.c +++ b/src/mpid/ch4/netmod/portals4/func_table.c @@ -27,8 +27,8 @@ MPIDI_NM_funcs_t MPIDI_NM_portals4_funcs = { MPIDI_NM_getallincomm, MPIDI_NM_gpid_tolpidarray, MPIDI_NM_create_intercomm_from_lpids, - MPIDI_NM_comm_create, - MPIDI_NM_comm_destroy, + MPIDI_NM_comm_create_hook, + MPIDI_NM_comm_free_hook, MPIDI_NM_am_request_init, MPIDI_NM_am_request_finalize, MPIDI_NM_reg_hdr_handler, @@ -145,10 +145,10 @@ MPIDI_NM_native_funcs_t MPIDI_NM_native_portals4_funcs = { MPIDI_NM_iscan, MPIDI_NM_iscatter, MPIDI_NM_iscatterv, - MPIDI_NM_datatype_commit, - MPIDI_NM_datatype_dup, - MPIDI_NM_datatype_destroy, - MPIDI_NM_op_commit, - MPIDI_NM_op_destroy + MPIDI_NM_type_create_hook, + MPIDI_NM_type_dup_hook, + MPIDI_NM_type_free_hook, + MPIDI_NM_op_create_hook, + MPIDI_NM_op_free_hook }; #endif diff --git a/src/mpid/ch4/netmod/portals4/ptl_comm.h b/src/mpid/ch4/netmod/portals4/ptl_comm.h index 651ab1f..4edbcd9 100644 --- a/src/mpid/ch4/netmod/portals4/ptl_comm.h +++ b/src/mpid/ch4/netmod/portals4/ptl_comm.h @@ -14,10 +14,10 @@ #include "ptl_impl.h" #undef FUNCNAME -#define FUNCNAME MPIDI_NM_comm_create +#define FUNCNAME MPIDI_NM_comm_create_hook #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_NM_comm_create(MPIR_Comm * comm) +static inline int MPIDI_NM_comm_create_hook(MPIR_Comm * comm) { int mpi_errno = MPI_SUCCESS; mpi_errno = MPIDI_CH4U_init_comm(comm); @@ -25,10 +25,10 @@ static inline int MPIDI_NM_comm_create(MPIR_Comm * comm) } #undef FUNCNAME -#define FUNCNAME MPIDI_NM_comm_destroy +#define FUNCNAME MPIDI_NM_comm_free_hook #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_NM_comm_destroy(MPIR_Comm * comm) +static inline int MPIDI_NM_comm_free_hook(MPIR_Comm * comm) { int mpi_errno = MPI_SUCCESS; mpi_errno = MPIDI_CH4U_destroy_comm(comm); diff --git a/src/mpid/ch4/netmod/portals4/ptl_datatype.h b/src/mpid/ch4/netmod/portals4/ptl_datatype.h index 3e529d6..679a48b 100644 --- a/src/mpid/ch4/netmod/portals4/ptl_datatype.h +++ b/src/mpid/ch4/netmod/portals4/ptl_datatype.h @@ -9,18 +9,18 @@ #include "ptl_impl.h" -static inline void MPIDI_NM_datatype_destroy(MPIR_Datatype * datatype_p) +static inline int MPIDI_NM_type_free_hook(MPIR_Datatype * datatype_p) { return; } -static inline void MPIDI_NM_datatype_commit(MPIR_Datatype * datatype_p) +static inline int MPIDI_NM_type_create_hook(MPIR_Datatype * datatype_p) { return; } -static inline void MPIDI_NM_datatype_dup(MPIR_Datatype * old_datatype_p, - MPIR_Datatype * new_datatype_p) +static inline void MPIDI_NM_type_dup_hook(MPIR_Datatype * old_datatype_p, + MPIR_Datatype * new_datatype_p) { return; } diff --git a/src/mpid/ch4/netmod/portals4/ptl_op.h b/src/mpid/ch4/netmod/portals4/ptl_op.h index a202468..a79ad6e 100644 --- a/src/mpid/ch4/netmod/portals4/ptl_op.h +++ b/src/mpid/ch4/netmod/portals4/ptl_op.h @@ -9,16 +9,16 @@ #include "ptl_impl.h" -static inline void MPIDI_NM_op_destroy(MPIR_Op * op_p) +static inline int MPIDI_NM_op_free_hook(MPIR_Op * op_p) { MPIR_Assert(0); - return; + return 0; } -static inline void MPIDI_NM_op_commit(MPIR_Op * op_p) +static inline int MPIDI_NM_op_create_hook(MPIR_Op * op_p) { MPIR_Assert(0); - return; + return 0; } #endif /* NETMOD_PTL_OP_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/stubnm/globals.c b/src/mpid/ch4/netmod/stubnm/globals.c index 9642082..b886803 100644 --- a/src/mpid/ch4/netmod/stubnm/globals.c +++ b/src/mpid/ch4/netmod/stubnm/globals.c @@ -27,8 +27,8 @@ MPIDI_NM_funcs_t MPIDI_NM_stubnm_funcs = { MPIDI_NM_getallincomm, MPIDI_NM_gpid_tolpidarray, MPIDI_NM_create_intercomm_from_lpids, - MPIDI_NM_comm_create, - MPIDI_NM_comm_destroy, + MPIDI_NM_comm_create_hook, + MPIDI_NM_comm_free_hook, MPIDI_NM_am_request_init, MPIDI_NM_am_request_finalize, MPIDI_NM_reg_hdr_handler, @@ -145,10 +145,10 @@ MPIDI_NM_native_funcs_t MPIDI_NM_native_stubnm_funcs = { MPIDI_NM_iscan, MPIDI_NM_iscatter, MPIDI_NM_iscatterv, - MPIDI_NM_datatype_commit, - MPIDI_NM_datatype_dup, - MPIDI_NM_datatype_destroy, - MPIDI_NM_op_commit, - MPIDI_NM_op_destroy, + MPIDI_NM_type_create_hook, + MPIDI_NM_type_dup_hook, + MPIDI_NM_type_free_hook, + MPIDI_NM_op_create_hook, + MPIDI_NM_op_free_hook, }; #endif diff --git a/src/mpid/ch4/netmod/stubnm/stubnm_comm.h b/src/mpid/ch4/netmod/stubnm/stubnm_comm.h index 3188e94..da5ef9a 100644 --- a/src/mpid/ch4/netmod/stubnm/stubnm_comm.h +++ b/src/mpid/ch4/netmod/stubnm/stubnm_comm.h @@ -14,10 +14,10 @@ #include "stubnm_impl.h" #undef FUNCNAME -#define FUNCNAME MPIDI_NM_comm_create +#define FUNCNAME MPIDI_NM_comm_create_hook #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_NM_comm_create(MPIR_Comm * comm) +static inline int MPIDI_NM_comm_create_hook(MPIR_Comm * comm) { int mpi_errno = MPI_SUCCESS; MPIR_Assert(0); @@ -25,10 +25,10 @@ static inline int MPIDI_NM_comm_create(MPIR_Comm * comm) } #undef FUNCNAME -#define FUNCNAME MPIDI_NM_comm_destroy +#define FUNCNAME MPIDI_NM_comm_free_hook #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_NM_comm_destroy(MPIR_Comm * comm) +static inline int MPIDI_NM_comm_free_hook(MPIR_Comm * comm) { int mpi_errno = MPI_SUCCESS; MPIR_Assert(0); diff --git a/src/mpid/ch4/netmod/stubnm/stubnm_datatype.h b/src/mpid/ch4/netmod/stubnm/stubnm_datatype.h index 877009b..323aa6d 100644 --- a/src/mpid/ch4/netmod/stubnm/stubnm_datatype.h +++ b/src/mpid/ch4/netmod/stubnm/stubnm_datatype.h @@ -13,20 +13,20 @@ #include "stubnm_impl.h" -static inline void MPIDI_NM_datatype_destroy(MPIR_Datatype * datatype_p) +static inline int MPIDI_NM_type_free_hook(MPIR_Datatype * datatype_p) { MPIR_Assert(0); - return; + return 0; } -static inline void MPIDI_NM_datatype_commit(MPIR_Datatype * datatype_p) +static inline int MPIDI_NM_type_create_hook(MPIR_Datatype * datatype_p) { MPIR_Assert(0); - return; + return 0; } -static inline void MPIDI_NM_datatype_dup(MPIR_Datatype * old_datatype_p, - MPIR_Datatype * new_datatype_p) +static inline void MPIDI_NM_type_dup_hook(MPIR_Datatype * old_datatype_p, + MPIR_Datatype * new_datatype_p) { MPIR_Assert(0); return; diff --git a/src/mpid/ch4/netmod/stubnm/stubnm_op.h b/src/mpid/ch4/netmod/stubnm/stubnm_op.h index ae4eade..3c37b0d 100644 --- a/src/mpid/ch4/netmod/stubnm/stubnm_op.h +++ b/src/mpid/ch4/netmod/stubnm/stubnm_op.h @@ -13,13 +13,13 @@ #include "stubnm_impl.h" -static inline void MPIDI_NM_op_destroy(MPIR_Op * op_p) +static inline int MPIDI_NM_op_free_hook(MPIR_Op * op_p) { MPIR_Assert(0); return; } -static inline void MPIDI_NM_op_commit(MPIR_Op * op_p) +static inline int MPIDI_NM_op_create_hook(MPIR_Op * op_p) { MPIR_Assert(0); return; diff --git a/src/mpid/ch4/netmod/ucx/func_table.c b/src/mpid/ch4/netmod/ucx/func_table.c index 4f60187..8bd03fb 100644 --- a/src/mpid/ch4/netmod/ucx/func_table.c +++ b/src/mpid/ch4/netmod/ucx/func_table.c @@ -25,8 +25,8 @@ MPIDI_NM_funcs_t MPIDI_NM_ucx_funcs = { MPIDI_NM_getallincomm, MPIDI_NM_gpid_tolpidarray, MPIDI_NM_create_intercomm_from_lpids, - MPIDI_NM_comm_create, - MPIDI_NM_comm_destroy, + MPIDI_NM_comm_create_hook, + MPIDI_NM_comm_free_hook, MPIDI_NM_am_request_init, MPIDI_NM_am_request_finalize, MPIDI_NM_reg_hdr_handler, @@ -143,10 +143,10 @@ MPIDI_NM_native_funcs_t MPIDI_NM_native_ucx_funcs = { MPIDI_NM_iscan, MPIDI_NM_iscatter, MPIDI_NM_iscatterv, - MPIDI_NM_datatype_commit, - MPIDI_NM_datatype_dup, - MPIDI_NM_datatype_destroy, - MPIDI_NM_op_commit, - MPIDI_NM_op_destroy + MPIDI_NM_type_create_hook, + MPIDI_NM_type_dup_hook, + MPIDI_NM_type_free_hook, + MPIDI_NM_op_create_hook, + MPIDI_NM_op_free_hook }; #endif diff --git a/src/mpid/ch4/netmod/ucx/ucx_comm.h b/src/mpid/ch4/netmod/ucx/ucx_comm.h index b152c29..5b4368c 100644 --- a/src/mpid/ch4/netmod/ucx/ucx_comm.h +++ b/src/mpid/ch4/netmod/ucx/ucx_comm.h @@ -12,10 +12,10 @@ #include "ucx_impl.h" #undef FUNCNAME -#define FUNCNAME MPIDI_NM_comm_create +#define FUNCNAME MPIDI_NM_comm_create_hook #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_NM_comm_create(MPIR_Comm * comm) +static inline int MPIDI_NM_comm_create_hook(MPIR_Comm * comm) { int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_UCX_COMM_CREATE); @@ -29,10 +29,10 @@ static inline int MPIDI_NM_comm_create(MPIR_Comm * comm) } #undef FUNCNAME -#define FUNCNAME MPIDI_NM_comm_destroy +#define FUNCNAME MPIDI_NM_comm_free_hook #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_NM_comm_destroy(MPIR_Comm * comm) +static inline int MPIDI_NM_comm_free_hook(MPIR_Comm * comm) { int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_UCX_COMM_DESTROY); diff --git a/src/mpid/ch4/netmod/ucx/ucx_datatype.h b/src/mpid/ch4/netmod/ucx/ucx_datatype.h index 32f1df1..eb115f0 100644 --- a/src/mpid/ch4/netmod/ucx/ucx_datatype.h +++ b/src/mpid/ch4/netmod/ucx/ucx_datatype.h @@ -107,7 +107,7 @@ static ucp_generic_dt_ops_t MPIDI_UCX_datatype_ops = { }; -static inline void MPIDI_NM_datatype_destroy(MPIR_Datatype * datatype_p) +static inline int MPIDI_NM_type_free_hook(MPIR_Datatype * datatype_p) { @@ -116,10 +116,10 @@ static inline void MPIDI_NM_datatype_destroy(MPIR_Datatype * datatype_p) datatype_p->dev.netmod.ucx.ucp_datatype = -1; } - return; + return 0; } -static inline void MPIDI_NM_datatype_commit(MPIR_Datatype * datatype_p) +static inline int MPIDI_NM_type_create_hook(MPIR_Datatype * datatype_p) { ucp_datatype_t ucp_datatype; ucs_status_t status; @@ -138,11 +138,11 @@ static inline void MPIDI_NM_datatype_commit(MPIR_Datatype * datatype_p) } - return; + return 0; } -static inline void MPIDI_NM_datatype_dup(MPIR_Datatype * old_datatype_p, - MPIR_Datatype * new_datatype_p) +static inline void MPIDI_NM_type_dup_hook(MPIR_Datatype * old_datatype_p, + MPIR_Datatype * new_datatype_p) { return; } diff --git a/src/mpid/ch4/netmod/ucx/ucx_op.h b/src/mpid/ch4/netmod/ucx/ucx_op.h index ac10797..fb58231 100644 --- a/src/mpid/ch4/netmod/ucx/ucx_op.h +++ b/src/mpid/ch4/netmod/ucx/ucx_op.h @@ -11,14 +11,14 @@ #include "ucx_impl.h" -static inline void MPIDI_NM_op_destroy(MPIR_Op * op_p) +static inline int MPIDI_NM_op_free_hook(MPIR_Op * op_p) { - return; + return 0; } -static inline void MPIDI_NM_op_commit(MPIR_Op * op_p) +static inline int MPIDI_NM_op_create_hook(MPIR_Op * op_p) { - return; + return 0; } #endif /* NETMOD_UCX_OP_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/ucx/ucx_pre.h b/src/mpid/ch4/netmod/ucx/ucx_pre.h index 9779a4d..84e99c8 100644 --- a/src/mpid/ch4/netmod/ucx/ucx_pre.h +++ b/src/mpid/ch4/netmod/ucx/ucx_pre.h @@ -11,8 +11,8 @@ #include <ucp/api/ucp.h> -#define HAVE_MPIDI_NM_datatype_commit_hook -#define HAVE_MPIDI_NM_datatype_destroy_hook +#define HAVE_MPIDI_NM_type_create_hook +#define HAVE_MPIDI_NM_type_free_hook #define MPIDI_UCX_KVSAPPSTRLEN 4096 diff --git a/src/mpid/ch4/shm/include/shm.h b/src/mpid/ch4/shm/include/shm.h index 0ef4704..11acade 100644 --- a/src/mpid/ch4/shm/include/shm.h +++ b/src/mpid/ch4/shm/include/shm.h @@ -93,8 +93,12 @@ typedef int (*MPIDI_SHM_getallincomm_t) (MPIR_Comm * comm_ptr, int local_size, typedef int (*MPIDI_SHM_gpid_tolpidarray_t) (int size, MPIR_Gpid gpid[], int lpid[]); typedef int (*MPIDI_SHM_create_intercomm_from_lpids_t) (MPIR_Comm * newcomm_ptr, int size, const int lpids[]); -typedef int (*MPIDI_SHM_comm_create_t) (MPIR_Comm * comm); -typedef int (*MPIDI_SHM_comm_destroy_t) (MPIR_Comm * comm); +typedef int (*MPIDI_SHM_comm_create_hook_t) (MPIR_Comm * comm); +typedef int (*MPIDI_SHM_comm_free_hook_t) (MPIR_Comm * comm); +typedef int (*MPIDI_SHM_type_create_hook_t) (MPIR_Datatype * type); +typedef int (*MPIDI_SHM_type_free_hook_t) (MPIR_Datatype * type); +typedef int (*MPIDI_SHM_op_create_hook_t) (MPIR_Op * op); +typedef int (*MPIDI_SHM_op_free_hook_t) (MPIR_Op * op); typedef void (*MPIDI_SHM_am_request_init_t) (MPIR_Request * req); typedef void (*MPIDI_SHM_am_request_finalize_t) (MPIR_Request * req); typedef int (*MPIDI_SHM_send_t) (const void *buf, int count, MPI_Datatype datatype, int rank, @@ -401,8 +405,12 @@ typedef struct MPIDI_SHM_funcs { MPIDI_SHM_getallincomm_t getallincomm; MPIDI_SHM_gpid_tolpidarray_t gpid_tolpidarray; MPIDI_SHM_create_intercomm_from_lpids_t create_intercomm_from_lpids; - MPIDI_SHM_comm_create_t comm_create; - MPIDI_SHM_comm_destroy_t comm_destroy; + MPIDI_SHM_comm_create_hook_t comm_create_hook; + MPIDI_SHM_comm_free_hook_t comm_free_hook; + MPIDI_SHM_type_create_hook_t type_create_hook; + MPIDI_SHM_type_free_hook_t type_free_hook; + MPIDI_SHM_op_create_hook_t op_create_hook; + MPIDI_SHM_op_free_hook_t op_free_hook; /* Request allocation routines */ MPIDI_SHM_am_request_init_t am_request_init; MPIDI_SHM_am_request_finalize_t am_request_finalize; @@ -622,8 +630,14 @@ MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_create_intercomm_from_lpids(MPIR_Comm * n int size, const int lpids[]) MPL_STATIC_INLINE_SUFFIX; -MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_create(MPIR_Comm * comm) MPL_STATIC_INLINE_SUFFIX; -MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_destroy(MPIR_Comm * comm) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_create_hook(MPIR_Comm * comm) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_free_hook(MPIR_Comm * comm) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_type_create_hook(MPIR_Datatype * + type) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_type_free_hook(MPIR_Datatype * + type) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_op_create_hook(MPIR_Op * op) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_op_free_hook(MPIR_Op * op) MPL_STATIC_INLINE_SUFFIX; MPL_STATIC_INLINE_PREFIX void MPIDI_SHM_am_request_init(MPIR_Request * req) MPL_STATIC_INLINE_SUFFIX; MPL_STATIC_INLINE_PREFIX void MPIDI_SHM_am_request_finalize(MPIR_Request * diff --git a/src/mpid/ch4/shm/include/shm_impl.h b/src/mpid/ch4/shm/include/shm_impl.h index 34d6038..d1125cf 100644 --- a/src/mpid/ch4/shm/include/shm_impl.h +++ b/src/mpid/ch4/shm/include/shm_impl.h @@ -231,14 +231,34 @@ MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_create_intercomm_from_lpids(MPIR_Comm * n return MPIDI_SHM_func->create_intercomm_from_lpids(newcomm_ptr, size, lpids); }; -MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_create(MPIR_Comm * comm) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_create_hook(MPIR_Comm * comm) { - return MPIDI_SHM_func->comm_create(comm); + return MPIDI_SHM_func->comm_create_hook(comm); }; -MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_destroy(MPIR_Comm * comm) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_free_hook(MPIR_Comm * comm) { - return MPIDI_SHM_func->comm_destroy(comm); + return MPIDI_SHM_func->comm_free_hook(comm); +}; + +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_type_create_hook(MPIR_Datatype * type) +{ + return MPIDI_SHM_func->type_create_hook(type); +}; + +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_type_free_hook(MPIR_Datatype * type) +{ + return MPIDI_SHM_func->type_free_hook(type); +}; + +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_op_create_hook(MPIR_Op * op) +{ + return MPIDI_SHM_func->op_create_hook(op); +}; + +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_op_free_hook(MPIR_Op * op) +{ + return MPIDI_SHM_func->op_free_hook(op); }; MPL_STATIC_INLINE_PREFIX void MPIDI_SHM_am_request_init(MPIR_Request * req) diff --git a/src/mpid/ch4/shm/posix/func_table.c b/src/mpid/ch4/shm/posix/func_table.c index 9b8a049..39f2ea5 100644 --- a/src/mpid/ch4/shm/posix/func_table.c +++ b/src/mpid/ch4/shm/posix/func_table.c @@ -44,8 +44,12 @@ MPIDI_SHM_funcs_t MPIDI_SHM_posix_funcs = { MPIDI_SHM_getallincomm, MPIDI_SHM_gpid_tolpidarray, MPIDI_SHM_create_intercomm_from_lpids, - MPIDI_SHM_comm_create, - MPIDI_SHM_comm_destroy, + MPIDI_SHM_comm_create_hook, + MPIDI_SHM_comm_free_hook, + MPIDI_SHM_type_create_hook, + MPIDI_SHM_type_free_hook, + MPIDI_SHM_op_create_hook, + MPIDI_SHM_op_free_hook, MPIDI_SHM_am_request_init, }; diff --git a/src/mpid/ch4/shm/posix/posix_comm.h b/src/mpid/ch4/shm/posix/posix_comm.h index 336cd74..0aaf6ef 100644 --- a/src/mpid/ch4/shm/posix/posix_comm.h +++ b/src/mpid/ch4/shm/posix/posix_comm.h @@ -14,10 +14,10 @@ #include "mpl_utlist.h" #undef FUNCNAME -#define FUNCNAME MPIDI_SHM_comm_create +#define FUNCNAME MPIDI_SHM_comm_create_hook #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_SHM_comm_create(MPIR_Comm * comm) +static inline int MPIDI_SHM_comm_create_hook(MPIR_Comm * comm) { int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_COMM_CREATE); @@ -28,10 +28,10 @@ static inline int MPIDI_SHM_comm_create(MPIR_Comm * comm) } #undef FUNCNAME -#define FUNCNAME MPIDI_SHM_comm_destroy +#define FUNCNAME MPIDI_SHM_comm_free_hook #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIDI_SHM_comm_destroy(MPIR_Comm * comm) +static inline int MPIDI_SHM_comm_free_hook(MPIR_Comm * comm) { int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_COMM_DESTROY); diff --git a/src/mpid/ch4/shm/posix/posix_init.h b/src/mpid/ch4/shm/posix/posix_init.h index 30723a6..c07cbb9 100644 --- a/src/mpid/ch4/shm/posix/posix_init.h +++ b/src/mpid/ch4/shm/posix/posix_init.h @@ -321,4 +321,60 @@ static inline int MPIDI_SHM_create_intercomm_from_lpids(MPIR_Comm * newcomm_ptr, return MPI_SUCCESS; } +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_type_create_hook +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_type_create_hook(MPIR_Datatype * type) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_TYPE_CREATE_HOOK); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_TYPE_CREATE_HOOK); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_TYPE_CREATE_HOOK); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_type_free_hook +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_type_free_hook(MPIR_Datatype * type) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_TYPE_FREE_HOOK); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_TYPE_FREE_HOOK); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_TYPE_FREE_HOOK); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_op_create_hook +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_op_create_hook(MPIR_Op * op) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_OP_CREATE_HOOK); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_OP_CREATE_HOOK); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_OP_CREATE_HOOK); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_op_free_hook +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_op_free_hook(MPIR_Op * op) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_OP_FREE_HOOK); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_OP_FREE_HOOK); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_OP_FREE_HOOK); + return mpi_errno; +} + #endif /* SHM_POSIX_INIT_H_INCLUDED */ diff --git a/src/mpid/ch4/shm/stubshm/func_table.c b/src/mpid/ch4/shm/stubshm/func_table.c index 1244e17..264b179 100644 --- a/src/mpid/ch4/shm/stubshm/func_table.c +++ b/src/mpid/ch4/shm/stubshm/func_table.c @@ -44,8 +44,12 @@ MPIDI_SHM_funcs_t MPIDI_SHM_stubshm_funcs = { MPIDI_SHM_getallincomm, MPIDI_SHM_gpid_tolpidarray, MPIDI_SHM_create_intercomm_from_lpids, - MPIDI_SHM_comm_create, - MPIDI_SHM_comm_destroy, + MPIDI_SHM_comm_create_hook, + MPIDI_SHM_comm_free_hook, + MPIDI_SHM_type_create_hook, + MPIDI_SHM_type_free_hook, + MPIDI_SHM_op_create_hook, + MPIDI_SHM_op_free_hook, MPIDI_SHM_am_request_init, }; diff --git a/src/mpid/ch4/shm/stubshm/stubshm_comm.h b/src/mpid/ch4/shm/stubshm/stubshm_comm.h index db0b3a6..99ae077 100644 --- a/src/mpid/ch4/shm/stubshm/stubshm_comm.h +++ b/src/mpid/ch4/shm/stubshm/stubshm_comm.h @@ -11,14 +11,14 @@ #define SHM_STUBSHM_COMM_H_INCLUDED #include "stubshm_impl.h" -static inline int MPIDI_SHM_comm_create(MPIR_Comm * comm) +static inline int MPIDI_SHM_comm_create_hook(MPIR_Comm * comm) { int mpi_errno = MPI_SUCCESS; MPIR_Assert(0); return mpi_errno; } -static inline int MPIDI_SHM_comm_destroy(MPIR_Comm * comm) +static inline int MPIDI_SHM_comm_free_hook(MPIR_Comm * comm) { int mpi_errno = MPI_SUCCESS; MPIR_Assert(0); diff --git a/src/mpid/ch4/shm/stubshm/stubshm_init.h b/src/mpid/ch4/shm/stubshm/stubshm_init.h index cfe95fe..f101951 100644 --- a/src/mpid/ch4/shm/stubshm/stubshm_init.h +++ b/src/mpid/ch4/shm/stubshm/stubshm_init.h @@ -82,4 +82,60 @@ static inline int MPIDI_SHM_create_intercomm_from_lpids(MPIR_Comm * newcomm_ptr, return MPI_SUCCESS; } +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_type_create_hook +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_type_create_hook(MPIR_Datatype * type) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_TYPE_CREATE_HOOK); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_TYPE_CREATE_HOOK); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_TYPE_CREATE_HOOK); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_type_free_hook +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_type_free_hook(MPIR_Datatype * type) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_TYPE_FREE_HOOK); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_TYPE_FREE_HOOK); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_TYPE_FREE_HOOK); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_op_create_hook +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_op_create_hook(MPIR_Op * op) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_OP_CREATE_HOOK); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_OP_CREATE_HOOK); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_OP_CREATE_HOOK); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_op_free_hook +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_op_free_hook(MPIR_Op * op) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_OP_FREE_HOOK); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_OP_FREE_HOOK); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_OP_FREE_HOOK); + return mpi_errno; +} + #endif /* SHM_STUBSHM_INIT_H_INCLUDED */ diff --git a/src/mpid/ch4/src/ch4_comm.h b/src/mpid/ch4/src/ch4_comm.h index 695566d..c5a9520 100644 --- a/src/mpid/ch4/src/ch4_comm.h +++ b/src/mpid/ch4/src/ch4_comm.h @@ -92,22 +92,22 @@ __CH4_INLINE__ int MPIDI_Comm_split_type(MPIR_Comm * comm_ptr, } #undef FUNCNAME -#define FUNCNAME MPIDI_Comm_create +#define FUNCNAME MPIDI_Comm_create_hook #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Comm_create(MPIR_Comm * comm) +__CH4_INLINE__ int MPIDI_Comm_create_hook(MPIR_Comm * comm) { int mpi_errno; int i, *uniq_avtids; int max_n_avts; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_COMM_CREATE); MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_COMM_CREATE); - mpi_errno = MPIDI_NM_comm_create(comm); + mpi_errno = MPIDI_NM_comm_create_hook(comm); if (mpi_errno != MPI_SUCCESS) { MPIR_ERR_POP(mpi_errno); } #if defined(MPIDI_BUILD_CH4_SHM) - mpi_errno = MPIDI_SHM_comm_create(comm); + mpi_errno = MPIDI_SHM_comm_create_hook(comm); if (mpi_errno != MPI_SUCCESS) { MPIR_ERR_POP(mpi_errno); } @@ -164,10 +164,10 @@ __CH4_INLINE__ int MPIDI_Comm_create(MPIR_Comm * comm) } #undef FUNCNAME -#define FUNCNAME MPIDI_Comm_destroy +#define FUNCNAME MPIDI_Comm_free_hook #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -__CH4_INLINE__ int MPIDI_Comm_destroy(MPIR_Comm * comm) +__CH4_INLINE__ int MPIDI_Comm_free_hook(MPIR_Comm * comm) { int mpi_errno; int i, *uniq_avtids; @@ -213,12 +213,12 @@ __CH4_INLINE__ int MPIDI_Comm_destroy(MPIR_Comm * comm) MPIDIU_avt_release_ref(MPIDII_COMM(comm, local_map).avtid); } - mpi_errno = MPIDI_NM_comm_destroy(comm); + mpi_errno = MPIDI_NM_comm_free_hook(comm); if (mpi_errno != MPI_SUCCESS) { MPIR_ERR_POP(mpi_errno); } #if defined(MPIDI_BUILD_CH4_SHM) - mpi_errno = MPIDI_SHM_comm_destroy(comm); + mpi_errno = MPIDI_SHM_comm_free_hook(comm); if (mpi_errno != MPI_SUCCESS) { MPIR_ERR_POP(mpi_errno); } diff --git a/src/mpid/ch4/src/ch4_init.h b/src/mpid/ch4/src/ch4_init.h index c26db6b..2365b85 100644 --- a/src/mpid/ch4/src/ch4_init.h +++ b/src/mpid/ch4/src/ch4_init.h @@ -748,4 +748,124 @@ __CH4_INLINE__ MPI_Aint MPIDI_Aint_diff(MPI_Aint addr1, MPI_Aint addr2) return result; } +#undef FUNCNAME +#define FUNCNAME MPIDI_Type_create_hook +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Type_create_hook(MPIR_Datatype * type) +{ + int mpi_errno; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDI_TYPE_CREATE_HOOK); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDI_TYPE_CREATE_HOOK); + + mpi_errno = MPIDI_NM_type_create_hook(type); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + +#if defined(MPIDI_BUILD_CH4_SHM) + mpi_errno = MPIDI_SHM_type_create_hook(type); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } +#endif + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDI_TYPE_CREATE_HOOK); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Type_free_hook +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Type_free_hook(MPIR_Datatype * type) +{ + int mpi_errno; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDI_TYPE_FREE_HOOK); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDI_TYPE_FREE_HOOK); + + mpi_errno = MPIDI_NM_type_free_hook(type); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + +#if defined(MPIDI_BUILD_CH4_SHM) + mpi_errno = MPIDI_SHM_type_free_hook(type); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } +#endif + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDI_TYPE_FREE_HOOK); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Op_create_hook +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Op_create_hook(MPIR_Op * op) +{ + int mpi_errno; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDI_OP_CREATE_HOOK); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDI_OP_CREATE_HOOK); + + mpi_errno = MPIDI_NM_op_create_hook(op); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + +#if defined(MPIDI_BUILD_CH4_SHM) + mpi_errno = MPIDI_SHM_op_create_hook(op); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } +#endif + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDI_OP_CREATE_HOOK); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Op_free_hook +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Op_free_hook(MPIR_Op * op) +{ + int mpi_errno; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDI_OP_FREE_HOOK); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDI_OP_FREE_HOOK); + + mpi_errno = MPIDI_NM_op_free_hook(op); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + +#if defined(MPIDI_BUILD_CH4_SHM) + mpi_errno = MPIDI_SHM_op_free_hook(op); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } +#endif + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDI_OP_FREE_HOOK); + return mpi_errno; + fn_fail: + goto fn_exit; +} + #endif /* CH4_INIT_H_INCLUDED */ diff --git a/src/mpid/common/datatype/mpidu_datatype_free.c b/src/mpid/common/datatype/mpidu_datatype_free.c index 362973d..4527af4 100644 --- a/src/mpid/common/datatype/mpidu_datatype_free.c +++ b/src/mpid/common/datatype/mpidu_datatype_free.c @@ -37,9 +37,9 @@ void MPIDU_Datatype_free(MPIDU_Datatype *ptr) { MPL_DBG_MSG_P(MPIR_DBG_DATATYPE,VERBOSE,"type %x freed.", ptr->handle); -#ifdef MPID_Dev_datatype_destroy_hook - MPID_Dev_datatype_destroy_hook(ptr); -#endif /* MPID_Dev_datatype_destroy_hook */ +#ifdef MPID_Type_free_hook + MPID_Type_free_hook(ptr); +#endif /* MPID_Type_free_hook */ /* before freeing the contents, check whether the pointer is not null because it is null in the case of a datatype shipped to the target diff --git a/src/mpid/common/datatype/mpidu_type_commit.c b/src/mpid/common/datatype/mpidu_type_commit.c index a847ebf..c40f9e7 100644 --- a/src/mpid/common/datatype/mpidu_type_commit.c +++ b/src/mpid/common/datatype/mpidu_type_commit.c @@ -63,9 +63,9 @@ int MPIDU_Type_commit(MPI_Datatype *datatype_p) MPIDI_Dataloop_dot_printf(datatype_ptr->dataloop, 0, 1); #endif -#ifdef MPID_Dev_datatype_commit_hook - MPID_Dev_datatype_commit_hook(datatype_ptr); -#endif /* MPID_Dev_datatype_commit_hook */ +#ifdef MPID_Type_commit_hook + MPID_Type_commit_hook(datatype_ptr); +#endif /* MPID_Type_commit_hook */ } return mpi_errno; diff --git a/src/mpid/common/datatype/mpidu_type_dup.c b/src/mpid/common/datatype/mpidu_type_dup.c index 55e5155..baf7a5a 100644 --- a/src/mpid/common/datatype/mpidu_type_dup.c +++ b/src/mpid/common/datatype/mpidu_type_dup.c @@ -98,9 +98,9 @@ int MPIDU_Type_dup(MPI_Datatype oldtype, &new_dtp->hetero_dloop); } -#ifdef MPID_Dev_datatype_commit_hook - MPID_Dev_datatype_commit_hook(new_dtp); -#endif /* MPID_Dev_datatype_commit_hook */ +#ifdef MPID_Type_commit_hook + MPID_Type_commit_hook(new_dtp); +#endif /* MPID_Type_commit_hook */ } } diff --git a/src/mpid/pamid/include/mpidi_hooks.h b/src/mpid/pamid/include/mpidi_hooks.h index f636731..54e47af 100644 --- a/src/mpid/pamid/include/mpidi_hooks.h +++ b/src/mpid/pamid/include/mpidi_hooks.h @@ -59,8 +59,8 @@ typedef size_t intptr_t; /** \brief This defines the portion of MPIR_Win that is specific to the Device */ #define MPID_DEV_WIN_DECL struct MPIDI_Win mpid; -#define MPID_Dev_comm_create_hook(a) ({ int MPIDI_Comm_create (MPIR_Comm *comm); MPIDI_Comm_create (a); }) -#define MPID_Dev_comm_destroy_hook(a) ({ int MPIDI_Comm_destroy(MPIR_Comm *comm); MPIDI_Comm_destroy(a); }) +#define MPID_Comm_create_hook(a) ({ int MPIDI_Comm_create (MPIR_Comm *comm); MPIDI_Comm_create (a); }) +#define MPID_Comm_free_hook(a) ({ int MPIDI_Comm_destroy(MPIR_Comm *comm); MPIDI_Comm_destroy(a); }) #endif http://git.mpich.org/mpich.git/commitdiff/6ff49c7d01617b757fc4b9450d74496aa… commit 6ff49c7d01617b757fc4b9450d74496aaf774d80 Author: Pavan Balaji <balaji(a)anl.gov> Date: Wed Aug 24 09:03:48 2016 -0500 Move static inline macros to MPL. Signed-off-by: Ken Raffenetti <raffenet(a)mcs.anl.gov> diff --git a/src/mpid/ch4/netmod/include/netmod.h b/src/mpid/ch4/netmod/include/netmod.h index 6b79d06..6a5cbc3 100644 --- a/src/mpid/ch4/netmod/include/netmod.h +++ b/src/mpid/ch4/netmod/include/netmod.h @@ -509,612 +509,509 @@ extern MPIDI_NM_native_funcs_t *MPIDI_NM_native_func; extern int MPIDI_num_netmods; extern char MPIDI_NM_strings[][MPIDI_MAX_NETMOD_STRING_LEN]; -#ifndef MPIDI_NM_STATIC_INLINE_PREFIX -#define MPIDI_NM_STATIC_INLINE_PREFIX __attribute__((always_inline)) static inline -#endif - -#ifndef MPIDI_NM_STATIC_INLINE_SUFFIX -#define MPIDI_NM_STATIC_INLINE_SUFFIX __attribute__((always_inline)) -#endif - -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_init(int rank, int size, int appnum, int *tag_ub, - MPIR_Comm * comm_world, MPIR_Comm * comm_self, - int spawned, int num_contexts, - void **netmod_contexts) - MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_finalize(void) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_progress(void *netmod_context, - int blocking) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_reg_hdr_handler(int handler_id, - MPIDI_NM_am_origin_handler_fn - origin_handler_fn, - MPIDI_NM_am_target_handler_fn - target_handler_fn) - MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_comm_connect(const char *port_name, MPIR_Info * info, - int root, MPIR_Comm * comm, - MPIR_Comm ** - newcomm_ptr) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_comm_disconnect(MPIR_Comm * - comm_ptr) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_open_port(MPIR_Info * info_ptr, - char *port_name) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_close_port(const char *port_name) - MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_comm_accept(const char *port_name, MPIR_Info * info, - int root, MPIR_Comm * comm, - MPIR_Comm ** - newcomm_ptr) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_send_am_hdr(int rank, MPIR_Comm * comm, int handler_id, - const void *am_hdr, size_t am_hdr_sz, - MPIR_Request * sreq, - void *netmod_context) - MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_inject_am_hdr(int rank, MPIR_Comm * comm, int handler_id, - const void *am_hdr, size_t am_hdr_sz, - void *netmod_context) - MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_send_am(int rank, MPIR_Comm * comm, int handler_id, - const void *am_hdr, size_t am_hdr_sz, - const void *data, MPI_Count count, - MPI_Datatype datatype, MPIR_Request * sreq, - void *netmod_context) - MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_send_amv(int rank, MPIR_Comm * comm, int handler_id, - struct iovec *am_hdrs, size_t iov_len, - const void *data, MPI_Count count, - MPI_Datatype datatype, MPIR_Request * sreq, - void *netmod_context) - MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_send_amv_hdr(int rank, MPIR_Comm * comm, int handler_id, - struct iovec *am_hdrs, size_t iov_len, - MPIR_Request * sreq, - void *netmod_context) - MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_send_am_hdr_reply(MPIR_Context_id_t context_id, - int src_rank, int handler_id, - const void *am_hdr, size_t am_hdr_sz, - MPIR_Request * - sreq) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_inject_am_hdr_reply(MPIR_Context_id_t context_id, - int src_rank, int handler_id, - const void *am_hdr, - size_t am_hdr_sz) - MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_send_am_reply(MPIR_Context_id_t context_id, int src_rank, - int handler_id, const void *am_hdr, - size_t am_hdr_sz, const void *data, - MPI_Count count, MPI_Datatype datatype, - MPIR_Request * - sreq) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_send_amv_reply(MPIR_Context_id_t context_id, +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_init(int rank, int size, int appnum, int *tag_ub, + MPIR_Comm * comm_world, MPIR_Comm * comm_self, + int spawned, int num_contexts, + void **netmod_contexts) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_finalize(void) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_progress(void *netmod_context, + int blocking) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_reg_hdr_handler(int handler_id, + MPIDI_NM_am_origin_handler_fn + origin_handler_fn, + MPIDI_NM_am_target_handler_fn + target_handler_fn) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_comm_connect(const char *port_name, MPIR_Info * info, + int root, MPIR_Comm * comm, + MPIR_Comm ** + newcomm_ptr) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_comm_disconnect(MPIR_Comm * + comm_ptr) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_open_port(MPIR_Info * info_ptr, + char *port_name) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_close_port(const char *port_name) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_comm_accept(const char *port_name, MPIR_Info * info, + int root, MPIR_Comm * comm, + MPIR_Comm ** + newcomm_ptr) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_send_am_hdr(int rank, MPIR_Comm * comm, int handler_id, + const void *am_hdr, size_t am_hdr_sz, + MPIR_Request * sreq, + void *netmod_context) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_inject_am_hdr(int rank, MPIR_Comm * comm, int handler_id, + const void *am_hdr, size_t am_hdr_sz, + void *netmod_context) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_send_am(int rank, MPIR_Comm * comm, int handler_id, + const void *am_hdr, size_t am_hdr_sz, + const void *data, MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq, + void *netmod_context) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_send_amv(int rank, MPIR_Comm * comm, int handler_id, + struct iovec *am_hdrs, size_t iov_len, + const void *data, MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq, + void *netmod_context) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_send_amv_hdr(int rank, MPIR_Comm * comm, int handler_id, + struct iovec *am_hdrs, size_t iov_len, + MPIR_Request * sreq, + void *netmod_context) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_send_am_hdr_reply(MPIR_Context_id_t context_id, + int src_rank, int handler_id, + const void *am_hdr, size_t am_hdr_sz, + MPIR_Request * + sreq) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_inject_am_hdr_reply(MPIR_Context_id_t context_id, int src_rank, int handler_id, - struct iovec *am_hdr, size_t iov_len, - const void *data, MPI_Count count, - MPI_Datatype datatype, - MPIR_Request * - sreq) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX size_t MPIDI_NM_am_hdr_max_sz(void) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_am_recv(MPIR_Request * - req) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_comm_get_lpid(MPIR_Comm * comm_ptr, int idx, - int *lpid_ptr, - MPL_bool is_remote) - MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_gpid_get(MPIR_Comm * comm_ptr, int rank, - MPIR_Gpid * gpid) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_getallincomm(MPIR_Comm * comm_ptr, int local_size, - MPIR_Gpid local_gpids[], - int *singleAVT) - MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_gpid_tolpidarray(int size, MPIR_Gpid gpid[], - int lpid[]) - MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_create_intercomm_from_lpids(MPIR_Comm * newcomm_ptr, - int size, - const int lpids[]) - MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_comm_create(MPIR_Comm * - comm) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_comm_destroy(MPIR_Comm * - comm) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX void MPIDI_NM_am_request_init(MPIR_Request * - req) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX void MPIDI_NM_am_request_finalize(MPIR_Request * - req) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_send(const void *buf, int count, MPI_Datatype datatype, + const void *am_hdr, + size_t am_hdr_sz) + MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_send_am_reply(MPIR_Context_id_t context_id, int src_rank, + int handler_id, const void *am_hdr, + size_t am_hdr_sz, const void *data, + MPI_Count count, MPI_Datatype datatype, + MPIR_Request * sreq) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_send_amv_reply(MPIR_Context_id_t context_id, + int src_rank, int handler_id, + struct iovec *am_hdr, size_t iov_len, + const void *data, MPI_Count count, + MPI_Datatype datatype, + MPIR_Request * sreq) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX size_t MPIDI_NM_am_hdr_max_sz(void) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_recv(MPIR_Request * req) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_comm_get_lpid(MPIR_Comm * comm_ptr, int idx, + int *lpid_ptr, + MPL_bool is_remote) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_gpid_get(MPIR_Comm * comm_ptr, int rank, + MPIR_Gpid * gpid) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_getallincomm(MPIR_Comm * comm_ptr, int local_size, + MPIR_Gpid local_gpids[], + int *singleAVT) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_gpid_tolpidarray(int size, MPIR_Gpid gpid[], + int lpid[]) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_create_intercomm_from_lpids(MPIR_Comm * newcomm_ptr, + int size, + const int lpids[]) + MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_comm_create(MPIR_Comm * comm) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_comm_destroy(MPIR_Comm * comm) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX void MPIDI_NM_am_request_init(MPIR_Request * req) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX void MPIDI_NM_am_request_finalize(MPIR_Request * + req) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_send(const void *buf, int count, MPI_Datatype datatype, + int rank, int tag, MPIR_Comm * comm, + int context_offset, + MPIR_Request ** request) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_ssend(const void *buf, int count, MPI_Datatype datatype, + int rank, int tag, MPIR_Comm * comm, + int context_offset, + MPIR_Request ** request) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_startall(int count, + MPIR_Request * requests[]) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_send_init(const void *buf, int count, + MPI_Datatype datatype, int rank, int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_ssend_init(const void *buf, int count, + MPI_Datatype datatype, int rank, int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_rsend_init(const void *buf, int count, + MPI_Datatype datatype, int rank, int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_bsend_init(const void *buf, int count, + MPI_Datatype datatype, int rank, int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_isend(const void *buf, int count, MPI_Datatype datatype, + int rank, int tag, MPIR_Comm * comm, + int context_offset, + MPIR_Request ** request) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_issend(const void *buf, int count, MPI_Datatype datatype, + int rank, int tag, MPIR_Comm * comm, + int context_offset, + MPIR_Request ** request) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_cancel_send(MPIR_Request * sreq) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_recv_init(void *buf, int count, MPI_Datatype datatype, int rank, int tag, MPIR_Comm * comm, int context_offset, - MPIR_Request ** - request) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ssend(const void *buf, int count, MPI_Datatype datatype, - int rank, int tag, MPIR_Comm * comm, - int context_offset, - MPIR_Request ** - request) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_startall(int count, - MPIR_Request * - requests[]) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_send_init(const void *buf, int count, - MPI_Datatype datatype, int rank, int tag, - MPIR_Comm * comm, int context_offset, - MPIR_Request ** - request) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ssend_init(const void *buf, int count, - MPI_Datatype datatype, int rank, int tag, - MPIR_Comm * comm, int context_offset, - MPIR_Request ** - request) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_rsend_init(const void *buf, int count, - MPI_Datatype datatype, int rank, int tag, - MPIR_Comm * comm, int context_offset, - MPIR_Request ** - request) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_bsend_init(const void *buf, int count, - MPI_Datatype datatype, int rank, int tag, - MPIR_Comm * comm, int context_offset, - MPIR_Request ** - request) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_isend(const void *buf, int count, MPI_Datatype datatype, - int rank, int tag, MPIR_Comm * comm, - int context_offset, - MPIR_Request ** - request) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_issend(const void *buf, int count, MPI_Datatype datatype, - int rank, int tag, MPIR_Comm * comm, - int context_offset, - MPIR_Request ** - request) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_cancel_send(MPIR_Request * - sreq) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_recv_init(void *buf, int count, MPI_Datatype datatype, - int rank, int tag, MPIR_Comm * comm, - int context_offset, - MPIR_Request ** - request) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_recv(void *buf, int count, MPI_Datatype datatype, - int rank, int tag, MPIR_Comm * comm, - int context_offset, MPI_Status * status, - MPIR_Request ** - request) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_irecv(void *buf, int count, MPI_Datatype datatype, - int rank, int tag, MPIR_Comm * comm, - int context_offset, - MPIR_Request ** - request) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_imrecv(void *buf, int count, MPI_Datatype datatype, - MPIR_Request * message, - MPIR_Request ** - rreqp) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_cancel_recv(MPIR_Request * - rreq) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX void *MPIDI_NM_alloc_mem(size_t size, - MPIR_Info * - info_ptr) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_free_mem(void *ptr) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_improbe(int source, int tag, MPIR_Comm * comm, - int context_offset, int *flag, - MPIR_Request ** message, - MPI_Status * - status) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_iprobe(int source, int tag, MPIR_Comm * comm, - int context_offset, int *flag, - MPI_Status * - status) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_set_info(MPIR_Win * win, - MPIR_Info * - info) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_shared_query(MPIR_Win * win, int rank, - MPI_Aint * size, int *disp_unit, - void *baseptr) - MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_put(const void *origin_addr, int origin_count, - MPI_Datatype origin_datatype, int target_rank, - MPI_Aint target_disp, int target_count, - MPI_Datatype target_datatype, - MPIR_Win * win) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_start(MPIR_Group * group, int assert, - MPIR_Win * win) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_complete(MPIR_Win * - win) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_post(MPIR_Group * group, int assert, - MPIR_Win * win) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_wait(MPIR_Win * win) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_test(MPIR_Win * win, - int *flag) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_lock(int lock_type, int rank, int assert, - MPIR_Win * win) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_unlock(int rank, - MPIR_Win * win) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_get_info(MPIR_Win * win, - MPIR_Info ** - info_p_p) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_get(void *origin_addr, int origin_count, - MPI_Datatype origin_datatype, int target_rank, - MPI_Aint target_disp, int target_count, - MPI_Datatype target_datatype, - MPIR_Win * win) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_free(MPIR_Win ** - win_ptr) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_fence(int assert, - MPIR_Win * win) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_create(void *base, MPI_Aint length, int disp_unit, - MPIR_Info * info, MPIR_Comm * comm_ptr, - MPIR_Win ** - win_ptr) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_accumulate(const void *origin_addr, int origin_count, - MPI_Datatype origin_datatype, int target_rank, - MPI_Aint target_disp, int target_count, + MPIR_Request ** request) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_recv(void *buf, int count, MPI_Datatype datatype, + int rank, int tag, MPIR_Comm * comm, + int context_offset, MPI_Status * status, + MPIR_Request ** request) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_irecv(void *buf, int count, MPI_Datatype datatype, + int rank, int tag, MPIR_Comm * comm, + int context_offset, + MPIR_Request ** request) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_imrecv(void *buf, int count, MPI_Datatype datatype, + MPIR_Request * message, + MPIR_Request ** rreqp) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_cancel_recv(MPIR_Request * rreq) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX void *MPIDI_NM_alloc_mem(size_t size, + MPIR_Info * info_ptr) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_free_mem(void *ptr) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_improbe(int source, int tag, MPIR_Comm * comm, + int context_offset, int *flag, + MPIR_Request ** message, + MPI_Status * status) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_iprobe(int source, int tag, MPIR_Comm * comm, + int context_offset, int *flag, + MPI_Status * status) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_win_set_info(MPIR_Win * win, + MPIR_Info * info) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_win_shared_query(MPIR_Win * win, int rank, + MPI_Aint * size, int *disp_unit, + void *baseptr) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_put(const void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, int target_rank, + MPI_Aint target_disp, int target_count, + MPI_Datatype target_datatype, + MPIR_Win * win) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_win_start(MPIR_Group * group, int assert, + MPIR_Win * win) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_win_complete(MPIR_Win * win) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_win_post(MPIR_Group * group, int assert, + MPIR_Win * win) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_win_wait(MPIR_Win * win) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_win_test(MPIR_Win * win, int *flag) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_win_lock(int lock_type, int rank, int assert, + MPIR_Win * win) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_win_unlock(int rank, MPIR_Win * win) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_win_get_info(MPIR_Win * win, + MPIR_Info ** info_p_p) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_get(void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, int target_rank, + MPI_Aint target_disp, int target_count, + MPI_Datatype target_datatype, + MPIR_Win * win) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_win_free(MPIR_Win ** win_ptr) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_win_fence(int assert, + MPIR_Win * win) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_win_create(void *base, MPI_Aint length, int disp_unit, + MPIR_Info * info, MPIR_Comm * comm_ptr, + MPIR_Win ** win_ptr) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_accumulate(const void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, int target_rank, + MPI_Aint target_disp, int target_count, + MPI_Datatype target_datatype, MPI_Op op, + MPIR_Win * win) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_win_attach(MPIR_Win * win, void *base, + MPI_Aint size) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_win_allocate_shared(MPI_Aint size, int disp_unit, + MPIR_Info * info_ptr, + MPIR_Comm * comm_ptr, + void **base_ptr, + MPIR_Win ** + win_ptr) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_rput(const void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, int target_rank, + MPI_Aint target_disp, int target_count, + MPI_Datatype target_datatype, MPIR_Win * win, + MPIR_Request ** request) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_win_flush_local(int rank, + MPIR_Win * win) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_win_detach(MPIR_Win * win, + const void *base) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_compare_and_swap(const void *origin_addr, + const void *compare_addr, + void *result_addr, + MPI_Datatype datatype, int target_rank, + MPI_Aint target_disp, + MPIR_Win * win) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_raccumulate(const void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, + int target_rank, MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, MPI_Op op, + MPIR_Win * win, + MPIR_Request ** request) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_rget_accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + void *result_addr, int result_count, + MPI_Datatype result_datatype, + int target_rank, MPI_Aint target_disp, + int target_count, MPI_Datatype target_datatype, MPI_Op op, - MPIR_Win * win) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_attach(MPIR_Win * win, void *base, - MPI_Aint size) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_allocate_shared(MPI_Aint size, int disp_unit, - MPIR_Info * info_ptr, - MPIR_Comm * comm_ptr, - void **base_ptr, - MPIR_Win ** - win_ptr) - MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_rput(const void *origin_addr, int origin_count, - MPI_Datatype origin_datatype, int target_rank, - MPI_Aint target_disp, int target_count, - MPI_Datatype target_datatype, MPIR_Win * win, - MPIR_Request ** - request) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_flush_local(int rank, - MPIR_Win * - win) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_detach(MPIR_Win * win, - const void *base) - MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_compare_and_swap(const void *origin_addr, - const void *compare_addr, - void *result_addr, - MPI_Datatype datatype, int target_rank, - MPI_Aint target_disp, - MPIR_Win * - win) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_raccumulate(const void *origin_addr, int origin_count, - MPI_Datatype origin_datatype, - int target_rank, MPI_Aint target_disp, - int target_count, - MPI_Datatype target_datatype, MPI_Op op, - MPIR_Win * win, - MPIR_Request ** - request) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_rget_accumulate(const void *origin_addr, - int origin_count, - MPI_Datatype origin_datatype, - void *result_addr, int result_count, - MPI_Datatype result_datatype, - int target_rank, MPI_Aint target_disp, - int target_count, - MPI_Datatype target_datatype, MPI_Op op, - MPIR_Win * win, - MPIR_Request ** - request) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_fetch_and_op(const void *origin_addr, void *result_addr, - MPI_Datatype datatype, int target_rank, - MPI_Aint target_disp, MPI_Op op, - MPIR_Win * - win) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_allocate(MPI_Aint size, int disp_unit, - MPIR_Info * info, MPIR_Comm * comm, - void *baseptr, - MPIR_Win ** - win) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_flush(int rank, - MPIR_Win * win) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_flush_local_all(MPIR_Win * - win) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_unlock_all(MPIR_Win * - win) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_create_dynamic(MPIR_Info * info, MPIR_Comm * comm, - MPIR_Win ** - win) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_rget(void *origin_addr, int origin_count, - MPI_Datatype origin_datatype, int target_rank, - MPI_Aint target_disp, int target_count, - MPI_Datatype target_datatype, MPIR_Win * win, - MPIR_Request ** - request) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_sync(MPIR_Win * win) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_flush_all(MPIR_Win * - win) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_get_accumulate(const void *origin_addr, int origin_count, - MPI_Datatype origin_datatype, - void *result_addr, int result_count, - MPI_Datatype result_datatype, - int target_rank, MPI_Aint target_disp, - int target_count, - MPI_Datatype target_datatype, MPI_Op op, - MPIR_Win * - win) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_lock_all(int assert, - MPIR_Win * - win) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_rank_is_local(int target, - MPIR_Comm * - comm) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_barrier(MPIR_Comm * comm, - MPIR_Errflag_t * - errflag) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_bcast(void *buffer, int count, MPI_Datatype datatype, - int root, MPIR_Comm * comm, - MPIR_Errflag_t * - errflag) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_allreduce(const void *sendbuf, void *recvbuf, int count, + MPIR_Win * win, + MPIR_Request ** + request) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_fetch_and_op(const void *origin_addr, void *result_addr, + MPI_Datatype datatype, int target_rank, + MPI_Aint target_disp, MPI_Op op, + MPIR_Win * win) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_win_allocate(MPI_Aint size, int disp_unit, + MPIR_Info * info, MPIR_Comm * comm, + void *baseptr, + MPIR_Win ** win) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_win_flush(int rank, MPIR_Win * win) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_win_flush_local_all(MPIR_Win * win) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_win_unlock_all(MPIR_Win * win) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_win_create_dynamic(MPIR_Info * info, MPIR_Comm * comm, + MPIR_Win ** win) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_rget(void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, int target_rank, + MPI_Aint target_disp, int target_count, + MPI_Datatype target_datatype, MPIR_Win * win, + MPIR_Request ** request) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_win_sync(MPIR_Win * win) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_win_flush_all(MPIR_Win * win) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_get_accumulate(const void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, + void *result_addr, int result_count, + MPI_Datatype result_datatype, + int target_rank, MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, MPI_Op op, + MPIR_Win * win) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_win_lock_all(int assert, + MPIR_Win * win) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_rank_is_local(int target, + MPIR_Comm * comm) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_barrier(MPIR_Comm * comm, + MPIR_Errflag_t * errflag) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_bcast(void *buffer, int count, MPI_Datatype datatype, + int root, MPIR_Comm * comm, + MPIR_Errflag_t * errflag) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_allreduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, + MPIR_Errflag_t * errflag) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_allgather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, + MPIR_Errflag_t * errflag) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_allgatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, const int *displs, + MPI_Datatype recvtype, MPIR_Comm * comm, + MPIR_Errflag_t * errflag) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_scatter(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, int root, + MPIR_Comm * comm, + MPIR_Errflag_t * errflag) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_scatterv(const void *sendbuf, const int *sendcounts, + const int *displs, MPI_Datatype sendtype, + void *recvbuf, int recvcount, + MPI_Datatype recvtype, int root, + MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_gather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, int root, + MPIR_Comm * comm, + MPIR_Errflag_t * errflag) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_gatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, const int *displs, + MPI_Datatype recvtype, int root, + MPIR_Comm * comm, + MPIR_Errflag_t * errflag) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_alltoall(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, + MPIR_Errflag_t * errflag) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_alltoallv(const void *sendbuf, const int *sendcounts, + const int *sdispls, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, + const int *rdispls, MPI_Datatype recvtype, + MPIR_Comm * comm, + MPIR_Errflag_t * errflag) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_alltoallw(const void *sendbuf, const int *sendcounts, + const int *sdispls, + const MPI_Datatype sendtypes[], void *recvbuf, + const int *recvcounts, const int *rdispls, + const MPI_Datatype recvtypes[], + MPIR_Comm * comm, + MPIR_Errflag_t * errflag) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_reduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, int root, + MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_reduce_scatter(const void *sendbuf, void *recvbuf, + const int *recvcounts, MPI_Datatype datatype, MPI_Op op, - MPIR_Comm * comm, - MPIR_Errflag_t * - errflag) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_allgather(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - int recvcount, MPI_Datatype recvtype, - MPIR_Comm * comm, - MPIR_Errflag_t * - errflag) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_allgatherv(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - const int *recvcounts, const int *displs, - MPI_Datatype recvtype, MPIR_Comm * comm, - MPIR_Errflag_t * - errflag) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_scatter(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - int recvcount, MPI_Datatype recvtype, int root, - MPIR_Comm * comm, - MPIR_Errflag_t * - errflag) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_scatterv(const void *sendbuf, const int *sendcounts, - const int *displs, MPI_Datatype sendtype, - void *recvbuf, int recvcount, - MPI_Datatype recvtype, int root, - MPIR_Comm * comm_ptr, - MPIR_Errflag_t * - errflag) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_gather(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - int recvcount, MPI_Datatype recvtype, int root, - MPIR_Comm * comm, - MPIR_Errflag_t * - errflag) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_gatherv(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - const int *recvcounts, const int *displs, - MPI_Datatype recvtype, int root, - MPIR_Comm * comm, - MPIR_Errflag_t * - errflag) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_alltoall(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - int recvcount, MPI_Datatype recvtype, - MPIR_Comm * comm, - MPIR_Errflag_t * - errflag) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_alltoallv(const void *sendbuf, const int *sendcounts, - const int *sdispls, MPI_Datatype sendtype, - void *recvbuf, const int *recvcounts, - const int *rdispls, MPI_Datatype recvtype, - MPIR_Comm * comm, - MPIR_Errflag_t * - errflag) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_alltoallw(const void *sendbuf, const int *sendcounts, - const int *sdispls, - const MPI_Datatype sendtypes[], void *recvbuf, - const int *recvcounts, const int *rdispls, - const MPI_Datatype recvtypes[], - MPIR_Comm * comm, + MPIR_Comm * comm_ptr, MPIR_Errflag_t * - errflag) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_reduce(const void *sendbuf, void *recvbuf, int count, - MPI_Datatype datatype, MPI_Op op, int root, - MPIR_Comm * comm_ptr, - MPIR_Errflag_t * - errflag) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_reduce_scatter(const void *sendbuf, void *recvbuf, + errflag) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_reduce_scatter_block(const void *sendbuf, void *recvbuf, + int recvcount, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm_ptr, + MPIR_Errflag_t * + errflag) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_scan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm, + MPIR_Errflag_t * errflag) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_exscan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, + MPIR_Errflag_t * errflag) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_neighbor_allgather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_neighbor_allgatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, const int *recvcounts, - MPI_Datatype datatype, MPI_Op op, - MPIR_Comm * comm_ptr, - MPIR_Errflag_t * - errflag) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_reduce_scatter_block(const void *sendbuf, void *recvbuf, - int recvcount, - MPI_Datatype datatype, MPI_Op op, - MPIR_Comm * comm_ptr, - MPIR_Errflag_t * - errflag) - MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_scan(const void *sendbuf, void *recvbuf, int count, - MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm, - MPIR_Errflag_t * - errflag) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_exscan(const void *sendbuf, void *recvbuf, int count, - MPI_Datatype datatype, MPI_Op op, - MPIR_Comm * comm, - MPIR_Errflag_t * - errflag) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_neighbor_allgather(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - int recvcount, MPI_Datatype recvtype, - MPIR_Comm * - comm) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_neighbor_allgatherv(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - const int *recvcounts, - const int *displs, - MPI_Datatype recvtype, - MPIR_Comm * - comm) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_neighbor_alltoallv(const void *sendbuf, - const int *sendcounts, - const int *sdispls, - MPI_Datatype sendtype, void *recvbuf, - const int *recvcounts, - const int *rdispls, - MPI_Datatype recvtype, - MPIR_Comm * - comm) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_neighbor_alltoallw(const void *sendbuf, - const int *sendcounts, - const MPI_Aint * sdispls, - const MPI_Datatype * sendtypes, - void *recvbuf, const int *recvcounts, - const MPI_Aint * rdispls, - const MPI_Datatype * recvtypes, - MPIR_Comm * - comm) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_neighbor_alltoall(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - int recvcount, MPI_Datatype recvtype, - MPIR_Comm * - comm) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ineighbor_allgather(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - int recvcount, MPI_Datatype recvtype, - MPIR_Comm * comm, - MPI_Request * - req) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ineighbor_allgatherv(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, - void *recvbuf, - const int *recvcounts, - const int *displs, - MPI_Datatype recvtype, - MPIR_Comm * comm, - MPI_Request * - req) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ineighbor_alltoall(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - int recvcount, MPI_Datatype recvtype, - MPIR_Comm * comm, - MPI_Request * - req) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ineighbor_alltoallv(const void *sendbuf, - const int *sendcounts, - const int *sdispls, - MPI_Datatype sendtype, void *recvbuf, - const int *recvcounts, - const int *rdispls, - MPI_Datatype recvtype, - MPIR_Comm * comm, - MPI_Request * - req) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ineighbor_alltoallw(const void *sendbuf, - const int *sendcounts, - const MPI_Aint * sdispls, - const MPI_Datatype * sendtypes, - void *recvbuf, const int *recvcounts, - const MPI_Aint * rdispls, - const MPI_Datatype * recvtypes, - MPIR_Comm * comm, - MPI_Request * - req) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ibarrier(MPIR_Comm * comm, - MPI_Request * - req) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ibcast(void *buffer, int count, MPI_Datatype datatype, - int root, MPIR_Comm * comm, - MPI_Request * req) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_iallgather(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - int recvcount, MPI_Datatype recvtype, - MPIR_Comm * comm, - MPI_Request * - req) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_iallgatherv(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - const int *recvcounts, const int *displs, - MPI_Datatype recvtype, MPIR_Comm * comm, - MPI_Request * - req) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_iallreduce(const void *sendbuf, void *recvbuf, int count, - MPI_Datatype datatype, MPI_Op op, - MPIR_Comm * comm, - MPI_Request * - req) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ialltoall(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - int recvcount, MPI_Datatype recvtype, - MPIR_Comm * comm, - MPI_Request * - req) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ialltoallv(const void *sendbuf, const int *sendcounts, - const int *sdispls, MPI_Datatype sendtype, - void *recvbuf, const int *recvcounts, - const int *rdispls, MPI_Datatype recvtype, - MPIR_Comm * comm, - MPI_Request * - req) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ialltoallw(const void *sendbuf, const int *sendcounts, - const int *sdispls, - const MPI_Datatype sendtypes[], void *recvbuf, - const int *recvcounts, const int *rdispls, - const MPI_Datatype recvtypes[], - MPIR_Comm * comm, - MPI_Request * - req) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_iexscan(const void *sendbuf, void *recvbuf, int count, - MPI_Datatype datatype, MPI_Op op, - MPIR_Comm * comm, - MPI_Request * req) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_igather(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - int recvcount, MPI_Datatype recvtype, int root, - MPIR_Comm * comm, - MPI_Request * req) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_igatherv(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - const int *recvcounts, const int *displs, - MPI_Datatype recvtype, int root, - MPIR_Comm * comm, - MPI_Request * - req) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ireduce_scatter_block(const void *sendbuf, void *recvbuf, - int recvcount, - MPI_Datatype datatype, MPI_Op op, - MPIR_Comm * comm, - MPI_Request * - req) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ireduce_scatter(const void *sendbuf, void *recvbuf, + const int *displs, + MPI_Datatype recvtype, + MPIR_Comm * + comm) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_neighbor_alltoallv(const void *sendbuf, + const int *sendcounts, + const int *sdispls, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, + const int *rdispls, + MPI_Datatype recvtype, + MPIR_Comm * comm) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_neighbor_alltoallw(const void *sendbuf, + const int *sendcounts, + const MPI_Aint * sdispls, + const MPI_Datatype * sendtypes, + void *recvbuf, const int *recvcounts, + const MPI_Aint * rdispls, + const MPI_Datatype * recvtypes, + MPIR_Comm * comm) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_neighbor_alltoall(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_ineighbor_allgather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, + MPI_Request * + req) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_ineighbor_allgatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, - MPI_Datatype datatype, MPI_Op op, + const int *displs, + MPI_Datatype recvtype, MPIR_Comm * comm, MPI_Request * - req) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ireduce(const void *sendbuf, void *recvbuf, int count, - MPI_Datatype datatype, MPI_Op op, int root, - MPIR_Comm * comm_ptr, - MPI_Request * req) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_iscan(const void *sendbuf, void *recvbuf, int count, - MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm, - MPI_Request * req) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_iscatter(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - int recvcount, MPI_Datatype recvtype, int root, - MPIR_Comm * comm, - MPI_Request * - req) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_iscatterv(const void *sendbuf, const int *sendcounts, - const int *displs, MPI_Datatype sendtype, - void *recvbuf, int recvcount, - MPI_Datatype recvtype, int root, - MPIR_Comm * comm_ptr, - MPI_Request * - req) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX void MPIDI_NM_datatype_commit(MPIR_Datatype * - datatype_p) - MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX void MPIDI_NM_datatype_dup(MPIR_Datatype * old_datatype_p, - MPIR_Datatype * - new_datatype_p) - MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX void MPIDI_NM_datatype_destroy(MPIR_Datatype * - datatype_p) - MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX void MPIDI_NM_op_commit(MPIR_Op * op_p) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX void MPIDI_NM_op_destroy(MPIR_Op * - op_p) MPIDI_NM_STATIC_INLINE_SUFFIX; + req) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_ineighbor_alltoall(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, + MPI_Request * + req) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_ineighbor_alltoallv(const void *sendbuf, + const int *sendcounts, + const int *sdispls, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, + const int *rdispls, + MPI_Datatype recvtype, + MPIR_Comm * comm, + MPI_Request * + req) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_ineighbor_alltoallw(const void *sendbuf, + const int *sendcounts, + const MPI_Aint * sdispls, + const MPI_Datatype * sendtypes, + void *recvbuf, const int *recvcounts, + const MPI_Aint * rdispls, + const MPI_Datatype * recvtypes, + MPIR_Comm * comm, + MPI_Request * + req) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_ibarrier(MPIR_Comm * comm, + MPI_Request * req) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_ibcast(void *buffer, int count, MPI_Datatype datatype, + int root, MPIR_Comm * comm, + MPI_Request * req) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_iallgather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, + MPI_Request * req) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_iallgatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, const int *displs, + MPI_Datatype recvtype, MPIR_Comm * comm, + MPI_Request * req) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_iallreduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, + MPI_Request * req) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_ialltoall(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, + MPI_Request * req) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_ialltoallv(const void *sendbuf, const int *sendcounts, + const int *sdispls, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, + const int *rdispls, MPI_Datatype recvtype, + MPIR_Comm * comm, + MPI_Request * req) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_ialltoallw(const void *sendbuf, const int *sendcounts, + const int *sdispls, + const MPI_Datatype sendtypes[], void *recvbuf, + const int *recvcounts, const int *rdispls, + const MPI_Datatype recvtypes[], + MPIR_Comm * comm, + MPI_Request * req) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_iexscan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, + MPI_Request * req) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_igather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, int root, + MPIR_Comm * comm, + MPI_Request * req) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_igatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, const int *displs, + MPI_Datatype recvtype, int root, + MPIR_Comm * comm, + MPI_Request * req) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_ireduce_scatter_block(const void *sendbuf, void *recvbuf, + int recvcount, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, + MPI_Request * + req) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_ireduce_scatter(const void *sendbuf, void *recvbuf, + const int *recvcounts, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, + MPI_Request * req) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_ireduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, int root, + MPIR_Comm * comm_ptr, + MPI_Request * req) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_iscan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm, + MPI_Request * req) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_iscatter(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, int root, + MPIR_Comm * comm, + MPI_Request * req) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_iscatterv(const void *sendbuf, const int *sendcounts, + const int *displs, MPI_Datatype sendtype, + void *recvbuf, int recvcount, + MPI_Datatype recvtype, int root, + MPIR_Comm * comm_ptr, + MPI_Request * req) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX void MPIDI_NM_datatype_commit(MPIR_Datatype * + datatype_p) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX void MPIDI_NM_datatype_dup(MPIR_Datatype * old_datatype_p, + MPIR_Datatype * + new_datatype_p) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX void MPIDI_NM_datatype_destroy(MPIR_Datatype * + datatype_p) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX void MPIDI_NM_op_commit(MPIR_Op * op_p) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX void MPIDI_NM_op_destroy(MPIR_Op * op_p) MPL_STATIC_INLINE_SUFFIX; #endif diff --git a/src/mpid/ch4/netmod/include/netmod_impl.h b/src/mpid/ch4/netmod/include/netmod_impl.h index 4fba2a4..5c3f221 100644 --- a/src/mpid/ch4/netmod/include/netmod_impl.h +++ b/src/mpid/ch4/netmod/include/netmod_impl.h @@ -16,507 +16,498 @@ #ifndef NETMOD_DIRECT #ifndef NETMOD_DISABLE_INLINES -#ifndef MPIDI_NM_STATIC_INLINE_PREFIX -#define MPIDI_NM_STATIC_INLINE_PREFIX __attribute__((always_inline)) static inline -#endif - -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_init(int rank, int size, int appnum, int *tag_ub, - MPIR_Comm * comm_world, MPIR_Comm * comm_self, - int spawned, int num_contexts, - void **netmod_contexts) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_init(int rank, int size, int appnum, int *tag_ub, + MPIR_Comm * comm_world, MPIR_Comm * comm_self, + int spawned, int num_contexts, void **netmod_contexts) { return MPIDI_NM_func->init(rank, size, appnum, tag_ub, comm_world, comm_self, spawned, num_contexts, netmod_contexts); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_finalize(void) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_finalize(void) { return MPIDI_NM_func->finalize(); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_progress(void *netmod_context, int blocking) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_progress(void *netmod_context, int blocking) { return MPIDI_NM_func->progress(netmod_context, blocking); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_reg_hdr_handler(int handler_id, - MPIDI_NM_am_origin_handler_fn - origin_handler_fn, - MPIDI_NM_am_target_handler_fn - target_handler_fn) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_reg_hdr_handler(int handler_id, + MPIDI_NM_am_origin_handler_fn + origin_handler_fn, + MPIDI_NM_am_target_handler_fn + target_handler_fn) { return MPIDI_NM_func->reg_hdr_handler(handler_id, origin_handler_fn, target_handler_fn); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_comm_connect(const char *port_name, MPIR_Info * info, - int root, MPIR_Comm * comm, - MPIR_Comm ** newcomm_ptr) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_comm_connect(const char *port_name, MPIR_Info * info, + int root, MPIR_Comm * comm, + MPIR_Comm ** newcomm_ptr) { return MPIDI_NM_func->comm_connect(port_name, info, root, comm, newcomm_ptr); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_comm_disconnect(MPIR_Comm * comm_ptr) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_comm_disconnect(MPIR_Comm * comm_ptr) { return MPIDI_NM_func->comm_disconnect(comm_ptr); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_open_port(MPIR_Info * info_ptr, char *port_name) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_open_port(MPIR_Info * info_ptr, char *port_name) { return MPIDI_NM_func->open_port(info_ptr, port_name); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_close_port(const char *port_name) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_close_port(const char *port_name) { return MPIDI_NM_func->close_port(port_name); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_comm_accept(const char *port_name, MPIR_Info * info, - int root, MPIR_Comm * comm, - MPIR_Comm ** newcomm_ptr) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_comm_accept(const char *port_name, MPIR_Info * info, + int root, MPIR_Comm * comm, + MPIR_Comm ** newcomm_ptr) { return MPIDI_NM_func->comm_accept(port_name, info, root, comm, newcomm_ptr); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_send_am_hdr(int rank, MPIR_Comm * comm, int handler_id, - const void *am_hdr, size_t am_hdr_sz, - MPIR_Request * sreq, void *netmod_context) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_send_am_hdr(int rank, MPIR_Comm * comm, int handler_id, + const void *am_hdr, size_t am_hdr_sz, + MPIR_Request * sreq, void *netmod_context) { return MPIDI_NM_func->send_am_hdr(rank, comm, handler_id, am_hdr, am_hdr_sz, sreq, netmod_context); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_inject_am_hdr(int rank, MPIR_Comm * comm, int handler_id, - const void *am_hdr, size_t am_hdr_sz, - void *netmod_context) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_inject_am_hdr(int rank, MPIR_Comm * comm, int handler_id, + const void *am_hdr, size_t am_hdr_sz, + void *netmod_context) { return MPIDI_NM_func->inject_am_hdr(rank, comm, handler_id, am_hdr, am_hdr_sz, netmod_context); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_send_am(int rank, MPIR_Comm * comm, int handler_id, - const void *am_hdr, size_t am_hdr_sz, - const void *data, MPI_Count count, - MPI_Datatype datatype, MPIR_Request * sreq, - void *netmod_context) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_send_am(int rank, MPIR_Comm * comm, int handler_id, + const void *am_hdr, size_t am_hdr_sz, + const void *data, MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq, + void *netmod_context) { return MPIDI_NM_func->send_am(rank, comm, handler_id, am_hdr, am_hdr_sz, data, count, datatype, sreq, netmod_context); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_send_amv(int rank, MPIR_Comm * comm, int handler_id, - struct iovec *am_hdrs, size_t iov_len, - const void *data, MPI_Count count, - MPI_Datatype datatype, MPIR_Request * sreq, - void *netmod_context) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_send_amv(int rank, MPIR_Comm * comm, int handler_id, + struct iovec *am_hdrs, size_t iov_len, + const void *data, MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq, + void *netmod_context) { return MPIDI_NM_func->send_amv(rank, comm, handler_id, am_hdrs, iov_len, data, count, datatype, sreq, netmod_context); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_send_amv_hdr(int rank, MPIR_Comm * comm, int handler_id, - struct iovec *am_hdrs, size_t iov_len, - MPIR_Request * sreq, void *netmod_context) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_send_amv_hdr(int rank, MPIR_Comm * comm, int handler_id, + struct iovec *am_hdrs, size_t iov_len, + MPIR_Request * sreq, void *netmod_context) { return MPIDI_NM_func->send_amv_hdr(rank, comm, handler_id, am_hdrs, iov_len, sreq, netmod_context); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_send_am_hdr_reply(MPIR_Context_id_t context_id, - int src_rank, int handler_id, - const void *am_hdr, size_t am_hdr_sz, - MPIR_Request * sreq) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_send_am_hdr_reply(MPIR_Context_id_t context_id, + int src_rank, int handler_id, + const void *am_hdr, size_t am_hdr_sz, + MPIR_Request * sreq) { return MPIDI_NM_func->send_am_hdr_reply(context_id, src_rank, handler_id, am_hdr, am_hdr_sz, sreq); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_inject_am_hdr_reply(MPIR_Context_id_t context_id, - int src_rank, int handler_id, - const void *am_hdr, size_t am_hdr_sz) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_inject_am_hdr_reply(MPIR_Context_id_t context_id, + int src_rank, int handler_id, + const void *am_hdr, size_t am_hdr_sz) { return MPIDI_NM_func->inject_am_hdr_reply(context_id, src_rank, handler_id, am_hdr, am_hdr_sz); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_send_am_reply(MPIR_Context_id_t context_id, int src_rank, - int handler_id, const void *am_hdr, - size_t am_hdr_sz, const void *data, - MPI_Count count, MPI_Datatype datatype, - MPIR_Request * sreq) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_send_am_reply(MPIR_Context_id_t context_id, int src_rank, + int handler_id, const void *am_hdr, + size_t am_hdr_sz, const void *data, + MPI_Count count, MPI_Datatype datatype, + MPIR_Request * sreq) { return MPIDI_NM_func->send_am_reply(context_id, src_rank, handler_id, am_hdr, am_hdr_sz, data, count, datatype, sreq); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_send_amv_reply(MPIR_Context_id_t context_id, - int src_rank, int handler_id, - struct iovec *am_hdr, size_t iov_len, - const void *data, MPI_Count count, - MPI_Datatype datatype, - MPIR_Request * sreq) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_send_amv_reply(MPIR_Context_id_t context_id, + int src_rank, int handler_id, + struct iovec *am_hdr, size_t iov_len, + const void *data, MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq) { return MPIDI_NM_func->send_amv_reply(context_id, src_rank, handler_id, am_hdr, iov_len, data, count, datatype, sreq); }; -MPIDI_NM_STATIC_INLINE_PREFIX size_t MPIDI_NM_am_hdr_max_sz(void) +MPL_STATIC_INLINE_PREFIX size_t MPIDI_NM_am_hdr_max_sz(void) { return MPIDI_NM_func->am_hdr_max_sz(); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_am_recv(MPIR_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_am_recv(MPIR_Request * req) { return MPIDI_NM_func->am_recv(req); } -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_comm_get_lpid(MPIR_Comm * comm_ptr, int idx, - int *lpid_ptr, MPL_bool is_remote) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_comm_get_lpid(MPIR_Comm * comm_ptr, int idx, + int *lpid_ptr, MPL_bool is_remote) { return MPIDI_NM_func->comm_get_lpid(comm_ptr, idx, lpid_ptr, is_remote); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_gpid_get(MPIR_Comm * comm_ptr, int rank, - MPIR_Gpid * gpid) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_gpid_get(MPIR_Comm * comm_ptr, int rank, MPIR_Gpid * gpid) { return MPIDI_NM_func->gpid_get(comm_ptr, rank, gpid); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_getallincomm(MPIR_Comm * comm_ptr, int local_size, - MPIR_Gpid local_gpid[], int *singleAVT) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_getallincomm(MPIR_Comm * comm_ptr, int local_size, + MPIR_Gpid local_gpid[], int *singleAVT) { return MPIDI_NM_func->getallincomm(comm_ptr, local_size, local_gpid, singleAVT); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_gpid_tolpidarray(int size, MPIR_Gpid gpid[], int lpid[]) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_gpid_tolpidarray(int size, MPIR_Gpid gpid[], int lpid[]) { return MPIDI_NM_func->gpid_tolpidarray(size, gpid, lpid); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_create_intercomm_from_lpids(MPIR_Comm * newcomm_ptr, - int size, const int lpids[]) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_create_intercomm_from_lpids(MPIR_Comm * newcomm_ptr, + int size, const int lpids[]) { return MPIDI_NM_func->create_intercomm_from_lpids(newcomm_ptr, size, lpids); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_comm_create(MPIR_Comm * comm) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_comm_create(MPIR_Comm * comm) { return MPIDI_NM_func->comm_create(comm); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_comm_destroy(MPIR_Comm * comm) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_comm_destroy(MPIR_Comm * comm) { return MPIDI_NM_func->comm_destroy(comm); }; -MPIDI_NM_STATIC_INLINE_PREFIX void MPIDI_NM_am_request_init(MPIR_Request * req) +MPL_STATIC_INLINE_PREFIX void MPIDI_NM_am_request_init(MPIR_Request * req) { return MPIDI_NM_func->am_request_init(req); }; -MPIDI_NM_STATIC_INLINE_PREFIX void MPIDI_NM_am_request_finalize(MPIR_Request * req) +MPL_STATIC_INLINE_PREFIX void MPIDI_NM_am_request_finalize(MPIR_Request * req) { return MPIDI_NM_func->am_request_finalize(req); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_send(const void *buf, int count, MPI_Datatype datatype, - int rank, int tag, MPIR_Comm * comm, - int context_offset, MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_send(const void *buf, int count, MPI_Datatype datatype, + int rank, int tag, MPIR_Comm * comm, + int context_offset, MPIR_Request ** request) { return MPIDI_NM_native_func->send(buf, count, datatype, rank, tag, comm, context_offset, request); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ssend(const void *buf, int count, MPI_Datatype datatype, - int rank, int tag, MPIR_Comm * comm, - int context_offset, MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_ssend(const void *buf, int count, MPI_Datatype datatype, + int rank, int tag, MPIR_Comm * comm, + int context_offset, MPIR_Request ** request) { return MPIDI_NM_native_func->ssend(buf, count, datatype, rank, tag, comm, context_offset, request); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_startall(int count, MPIR_Request * requests[]) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_startall(int count, MPIR_Request * requests[]) { return MPIDI_NM_native_func->startall(count, requests); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_send_init(const void *buf, int count, - MPI_Datatype datatype, int rank, int tag, - MPIR_Comm * comm, int context_offset, - MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_send_init(const void *buf, int count, + MPI_Datatype datatype, int rank, int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) { return MPIDI_NM_native_func->send_init(buf, count, datatype, rank, tag, comm, context_offset, request); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ssend_init(const void *buf, int count, - MPI_Datatype datatype, int rank, int tag, - MPIR_Comm * comm, int context_offset, - MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_ssend_init(const void *buf, int count, + MPI_Datatype datatype, int rank, int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) { return MPIDI_NM_native_func->ssend_init(buf, count, datatype, rank, tag, comm, context_offset, request); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_rsend_init(const void *buf, int count, - MPI_Datatype datatype, int rank, int tag, - MPIR_Comm * comm, int context_offset, - MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_rsend_init(const void *buf, int count, + MPI_Datatype datatype, int rank, int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) { return MPIDI_NM_native_func->rsend_init(buf, count, datatype, rank, tag, comm, context_offset, request); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_bsend_init(const void *buf, int count, - MPI_Datatype datatype, int rank, int tag, - MPIR_Comm * comm, int context_offset, - MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_bsend_init(const void *buf, int count, + MPI_Datatype datatype, int rank, int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) { return MPIDI_NM_native_func->bsend_init(buf, count, datatype, rank, tag, comm, context_offset, request); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_isend(const void *buf, int count, MPI_Datatype datatype, - int rank, int tag, MPIR_Comm * comm, - int context_offset, MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_isend(const void *buf, int count, MPI_Datatype datatype, + int rank, int tag, MPIR_Comm * comm, + int context_offset, MPIR_Request ** request) { return MPIDI_NM_native_func->isend(buf, count, datatype, rank, tag, comm, context_offset, request); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_issend(const void *buf, int count, MPI_Datatype datatype, - int rank, int tag, MPIR_Comm * comm, - int context_offset, MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_issend(const void *buf, int count, MPI_Datatype datatype, + int rank, int tag, MPIR_Comm * comm, + int context_offset, MPIR_Request ** request) { return MPIDI_NM_native_func->issend(buf, count, datatype, rank, tag, comm, context_offset, request); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_cancel_send(MPIR_Request * sreq) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_cancel_send(MPIR_Request * sreq) { return MPIDI_NM_native_func->cancel_send(sreq); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_recv_init(void *buf, int count, MPI_Datatype datatype, - int rank, int tag, MPIR_Comm * comm, - int context_offset, MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_recv_init(void *buf, int count, MPI_Datatype datatype, + int rank, int tag, MPIR_Comm * comm, + int context_offset, MPIR_Request ** request) { return MPIDI_NM_native_func->recv_init(buf, count, datatype, rank, tag, comm, context_offset, request); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_recv(void *buf, int count, MPI_Datatype datatype, - int rank, int tag, MPIR_Comm * comm, - int context_offset, MPI_Status * status, - MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_recv(void *buf, int count, MPI_Datatype datatype, + int rank, int tag, MPIR_Comm * comm, + int context_offset, MPI_Status * status, + MPIR_Request ** request) { return MPIDI_NM_native_func->recv(buf, count, datatype, rank, tag, comm, context_offset, status, request); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_irecv(void *buf, int count, MPI_Datatype datatype, - int rank, int tag, MPIR_Comm * comm, - int context_offset, MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_irecv(void *buf, int count, MPI_Datatype datatype, + int rank, int tag, MPIR_Comm * comm, + int context_offset, MPIR_Request ** request) { return MPIDI_NM_native_func->irecv(buf, count, datatype, rank, tag, comm, context_offset, request); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_imrecv(void *buf, int count, MPI_Datatype datatype, - MPIR_Request * message, MPIR_Request ** rreqp) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_imrecv(void *buf, int count, MPI_Datatype datatype, + MPIR_Request * message, MPIR_Request ** rreqp) { return MPIDI_NM_native_func->imrecv(buf, count, datatype, message, rreqp); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_cancel_recv(MPIR_Request * rreq) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_cancel_recv(MPIR_Request * rreq) { return MPIDI_NM_native_func->cancel_recv(rreq); }; -MPIDI_NM_STATIC_INLINE_PREFIX void *MPIDI_NM_alloc_mem(size_t size, MPIR_Info * info_ptr) +MPL_STATIC_INLINE_PREFIX void *MPIDI_NM_alloc_mem(size_t size, MPIR_Info * info_ptr) { return MPIDI_NM_native_func->alloc_mem(size, info_ptr); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_free_mem(void *ptr) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_free_mem(void *ptr) { return MPIDI_NM_native_func->free_mem(ptr); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_improbe(int source, int tag, MPIR_Comm * comm, - int context_offset, int *flag, - MPIR_Request ** message, MPI_Status * status) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_improbe(int source, int tag, MPIR_Comm * comm, + int context_offset, int *flag, + MPIR_Request ** message, MPI_Status * status) { return MPIDI_NM_native_func->improbe(source, tag, comm, context_offset, flag, message, status); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_iprobe(int source, int tag, MPIR_Comm * comm, - int context_offset, int *flag, - MPI_Status * status) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_iprobe(int source, int tag, MPIR_Comm * comm, + int context_offset, int *flag, MPI_Status * status) { return MPIDI_NM_native_func->iprobe(source, tag, comm, context_offset, flag, status); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_set_info(MPIR_Win * win, MPIR_Info * info) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_win_set_info(MPIR_Win * win, MPIR_Info * info) { return MPIDI_NM_native_func->win_set_info(win, info); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_shared_query(MPIR_Win * win, int rank, - MPI_Aint * size, int *disp_unit, - void *baseptr) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_win_shared_query(MPIR_Win * win, int rank, + MPI_Aint * size, int *disp_unit, + void *baseptr) { return MPIDI_NM_native_func->win_shared_query(win, rank, size, disp_unit, baseptr); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_put(const void *origin_addr, int origin_count, - MPI_Datatype origin_datatype, int target_rank, - MPI_Aint target_disp, int target_count, - MPI_Datatype target_datatype, MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_put(const void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, int target_rank, + MPI_Aint target_disp, int target_count, + MPI_Datatype target_datatype, MPIR_Win * win) { return MPIDI_NM_native_func->put(origin_addr, origin_count, origin_datatype, target_rank, target_disp, target_count, target_datatype, win); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_start(MPIR_Group * group, int assert, MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_win_start(MPIR_Group * group, int assert, MPIR_Win * win) { return MPIDI_NM_native_func->win_start(group, assert, win); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_complete(MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_win_complete(MPIR_Win * win) { return MPIDI_NM_native_func->win_complete(win); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_post(MPIR_Group * group, int assert, MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_win_post(MPIR_Group * group, int assert, MPIR_Win * win) { return MPIDI_NM_native_func->win_post(group, assert, win); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_wait(MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_win_wait(MPIR_Win * win) { return MPIDI_NM_native_func->win_wait(win); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_test(MPIR_Win * win, int *flag) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_win_test(MPIR_Win * win, int *flag) { return MPIDI_NM_native_func->win_test(win, flag); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_lock(int lock_type, int rank, int assert, - MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_win_lock(int lock_type, int rank, int assert, MPIR_Win * win) { return MPIDI_NM_native_func->win_lock(lock_type, rank, assert, win); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_unlock(int rank, MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_win_unlock(int rank, MPIR_Win * win) { return MPIDI_NM_native_func->win_unlock(rank, win); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_get_info(MPIR_Win * win, MPIR_Info ** info_p_p) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_win_get_info(MPIR_Win * win, MPIR_Info ** info_p_p) { return MPIDI_NM_native_func->win_get_info(win, info_p_p); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_get(void *origin_addr, int origin_count, - MPI_Datatype origin_datatype, int target_rank, - MPI_Aint target_disp, int target_count, - MPI_Datatype target_datatype, MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_get(void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, int target_rank, + MPI_Aint target_disp, int target_count, + MPI_Datatype target_datatype, MPIR_Win * win) { return MPIDI_NM_native_func->get(origin_addr, origin_count, origin_datatype, target_rank, target_disp, target_count, target_datatype, win); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_free(MPIR_Win ** win_ptr) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_win_free(MPIR_Win ** win_ptr) { return MPIDI_NM_native_func->win_free(win_ptr); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_fence(int assert, MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_win_fence(int assert, MPIR_Win * win) { return MPIDI_NM_native_func->win_fence(assert, win); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_create(void *base, MPI_Aint length, int disp_unit, - MPIR_Info * info, MPIR_Comm * comm_ptr, - MPIR_Win ** win_ptr) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_win_create(void *base, MPI_Aint length, int disp_unit, + MPIR_Info * info, MPIR_Comm * comm_ptr, + MPIR_Win ** win_ptr) { return MPIDI_NM_native_func->win_create(base, length, disp_unit, info, comm_ptr, win_ptr); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_accumulate(const void *origin_addr, int origin_count, - MPI_Datatype origin_datatype, int target_rank, - MPI_Aint target_disp, int target_count, - MPI_Datatype target_datatype, MPI_Op op, - MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_accumulate(const void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, int target_rank, + MPI_Aint target_disp, int target_count, + MPI_Datatype target_datatype, MPI_Op op, + MPIR_Win * win) { return MPIDI_NM_native_func->accumulate(origin_addr, origin_count, origin_datatype, target_rank, target_disp, target_count, target_datatype, op, win); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_attach(MPIR_Win * win, void *base, MPI_Aint size) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_win_attach(MPIR_Win * win, void *base, MPI_Aint size) { return MPIDI_NM_native_func->win_attach(win, base, size); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_allocate_shared(MPI_Aint size, int disp_unit, - MPIR_Info * info_ptr, - MPIR_Comm * comm_ptr, - void **base_ptr, MPIR_Win ** win_ptr) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_win_allocate_shared(MPI_Aint size, int disp_unit, + MPIR_Info * info_ptr, + MPIR_Comm * comm_ptr, + void **base_ptr, MPIR_Win ** win_ptr) { return MPIDI_NM_native_func->win_allocate_shared(size, disp_unit, info_ptr, comm_ptr, base_ptr, win_ptr); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_rput(const void *origin_addr, int origin_count, - MPI_Datatype origin_datatype, int target_rank, - MPI_Aint target_disp, int target_count, - MPI_Datatype target_datatype, MPIR_Win * win, - MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_rput(const void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, int target_rank, + MPI_Aint target_disp, int target_count, + MPI_Datatype target_datatype, MPIR_Win * win, + MPIR_Request ** request) { return MPIDI_NM_native_func->rput(origin_addr, origin_count, origin_datatype, target_rank, target_disp, target_count, target_datatype, win, request); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_flush_local(int rank, MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_win_flush_local(int rank, MPIR_Win * win) { return MPIDI_NM_native_func->win_flush_local(rank, win); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_detach(MPIR_Win * win, const void *base) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_win_detach(MPIR_Win * win, const void *base) { return MPIDI_NM_native_func->win_detach(win, base); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_compare_and_swap(const void *origin_addr, - const void *compare_addr, - void *result_addr, - MPI_Datatype datatype, int target_rank, - MPI_Aint target_disp, MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_compare_and_swap(const void *origin_addr, + const void *compare_addr, + void *result_addr, + MPI_Datatype datatype, int target_rank, + MPI_Aint target_disp, MPIR_Win * win) { return MPIDI_NM_native_func->compare_and_swap(origin_addr, compare_addr, result_addr, datatype, target_rank, target_disp, win); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_raccumulate(const void *origin_addr, int origin_count, - MPI_Datatype origin_datatype, - int target_rank, MPI_Aint target_disp, - int target_count, - MPI_Datatype target_datatype, MPI_Op op, - MPIR_Win * win, MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_raccumulate(const void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, + int target_rank, MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, MPI_Op op, + MPIR_Win * win, MPIR_Request ** request) { return MPIDI_NM_native_func->raccumulate(origin_addr, origin_count, origin_datatype, target_rank, target_disp, target_count, target_datatype, op, win, request); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_rget_accumulate(const void *origin_addr, - int origin_count, - MPI_Datatype origin_datatype, - void *result_addr, int result_count, - MPI_Datatype result_datatype, - int target_rank, MPI_Aint target_disp, - int target_count, - MPI_Datatype target_datatype, MPI_Op op, - MPIR_Win * win, MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_rget_accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + void *result_addr, int result_count, + MPI_Datatype result_datatype, + int target_rank, MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, MPI_Op op, + MPIR_Win * win, MPIR_Request ** request) { return MPIDI_NM_native_func->rget_accumulate(origin_addr, origin_count, origin_datatype, result_addr, result_count, result_datatype, @@ -524,71 +515,70 @@ MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_rget_accumulate(const void *origin_ad target_datatype, op, win, request); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_fetch_and_op(const void *origin_addr, void *result_addr, - MPI_Datatype datatype, int target_rank, - MPI_Aint target_disp, MPI_Op op, - MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_fetch_and_op(const void *origin_addr, void *result_addr, + MPI_Datatype datatype, int target_rank, + MPI_Aint target_disp, MPI_Op op, MPIR_Win * win) { return MPIDI_NM_native_func->fetch_and_op(origin_addr, result_addr, datatype, target_rank, target_disp, op, win); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_allocate(MPI_Aint size, int disp_unit, - MPIR_Info * info, MPIR_Comm * comm, - void *baseptr, MPIR_Win ** win) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_win_allocate(MPI_Aint size, int disp_unit, + MPIR_Info * info, MPIR_Comm * comm, + void *baseptr, MPIR_Win ** win) { return MPIDI_NM_native_func->win_allocate(size, disp_unit, info, comm, baseptr, win); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_flush(int rank, MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_win_flush(int rank, MPIR_Win * win) { return MPIDI_NM_native_func->win_flush(rank, win); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_flush_local_all(MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_win_flush_local_all(MPIR_Win * win) { return MPIDI_NM_native_func->win_flush_local_all(win); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_unlock_all(MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_win_unlock_all(MPIR_Win * win) { return MPIDI_NM_native_func->win_unlock_all(win); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_create_dynamic(MPIR_Info * info, MPIR_Comm * comm, - MPIR_Win ** win) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_win_create_dynamic(MPIR_Info * info, MPIR_Comm * comm, + MPIR_Win ** win) { return MPIDI_NM_native_func->win_create_dynamic(info, comm, win); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_rget(void *origin_addr, int origin_count, - MPI_Datatype origin_datatype, int target_rank, - MPI_Aint target_disp, int target_count, - MPI_Datatype target_datatype, MPIR_Win * win, - MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_rget(void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, int target_rank, + MPI_Aint target_disp, int target_count, + MPI_Datatype target_datatype, MPIR_Win * win, + MPIR_Request ** request) { return MPIDI_NM_native_func->rget(origin_addr, origin_count, origin_datatype, target_rank, target_disp, target_count, target_datatype, win, request); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_sync(MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_win_sync(MPIR_Win * win) { return MPIDI_NM_native_func->win_sync(win); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_flush_all(MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_win_flush_all(MPIR_Win * win) { return MPIDI_NM_native_func->win_flush_all(win); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_get_accumulate(const void *origin_addr, int origin_count, - MPI_Datatype origin_datatype, - void *result_addr, int result_count, - MPI_Datatype result_datatype, - int target_rank, MPI_Aint target_disp, - int target_count, - MPI_Datatype target_datatype, MPI_Op op, - MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_get_accumulate(const void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, + void *result_addr, int result_count, + MPI_Datatype result_datatype, + int target_rank, MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, MPI_Op op, + MPIR_Win * win) { return MPIDI_NM_native_func->get_accumulate(origin_addr, origin_count, origin_datatype, result_addr, result_count, result_datatype, @@ -596,445 +586,440 @@ MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_get_accumulate(const void *origin_add target_datatype, op, win); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_lock_all(int assert, MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_win_lock_all(int assert, MPIR_Win * win) { return MPIDI_NM_native_func->win_lock_all(assert, win); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_rank_is_local(int target, MPIR_Comm * comm) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_rank_is_local(int target, MPIR_Comm * comm) { return MPIDI_NM_native_func->rank_is_local(target, comm); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_barrier(MPIR_Comm * comm, MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_barrier(MPIR_Comm * comm, MPIR_Errflag_t * errflag) { return MPIDI_NM_native_func->barrier(comm, errflag); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_bcast(void *buffer, int count, MPI_Datatype datatype, - int root, MPIR_Comm * comm, - MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_bcast(void *buffer, int count, MPI_Datatype datatype, + int root, MPIR_Comm * comm, MPIR_Errflag_t * errflag) { return MPIDI_NM_native_func->bcast(buffer, count, datatype, root, comm, errflag); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_allreduce(const void *sendbuf, void *recvbuf, int count, - MPI_Datatype datatype, MPI_Op op, - MPIR_Comm * comm, MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_allreduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, MPIR_Errflag_t * errflag) { return MPIDI_NM_native_func->allreduce(sendbuf, recvbuf, count, datatype, op, comm, errflag); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_allgather(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - int recvcount, MPI_Datatype recvtype, - MPIR_Comm * comm, MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_allgather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, MPIR_Errflag_t * errflag) { return MPIDI_NM_native_func->allgather(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, comm, errflag); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_allgatherv(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - const int *recvcounts, const int *displs, - MPI_Datatype recvtype, MPIR_Comm * comm, - MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_allgatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, const int *displs, + MPI_Datatype recvtype, MPIR_Comm * comm, + MPIR_Errflag_t * errflag) { return MPIDI_NM_native_func->allgatherv(sendbuf, sendcount, sendtype, recvbuf, recvcounts, displs, recvtype, comm, errflag); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_scatter(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - int recvcount, MPI_Datatype recvtype, int root, - MPIR_Comm * comm, MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_scatter(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, int root, + MPIR_Comm * comm, MPIR_Errflag_t * errflag) { return MPIDI_NM_native_func->scatter(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, root, comm, errflag); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_scatterv(const void *sendbuf, const int *sendcounts, - const int *displs, MPI_Datatype sendtype, - void *recvbuf, int recvcount, - MPI_Datatype recvtype, int root, - MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_scatterv(const void *sendbuf, const int *sendcounts, + const int *displs, MPI_Datatype sendtype, + void *recvbuf, int recvcount, + MPI_Datatype recvtype, int root, + MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) { return MPIDI_NM_native_func->scatterv(sendbuf, sendcounts, displs, sendtype, recvbuf, recvcount, recvtype, root, comm_ptr, errflag); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_gather(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - int recvcount, MPI_Datatype recvtype, int root, - MPIR_Comm * comm, MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_gather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, int root, + MPIR_Comm * comm, MPIR_Errflag_t * errflag) { return MPIDI_NM_native_func->gather(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, root, comm, errflag); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_gatherv(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - const int *recvcounts, const int *displs, - MPI_Datatype recvtype, int root, - MPIR_Comm * comm, MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_gatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, const int *displs, + MPI_Datatype recvtype, int root, + MPIR_Comm * comm, MPIR_Errflag_t * errflag) { return MPIDI_NM_native_func->gatherv(sendbuf, sendcount, sendtype, recvbuf, recvcounts, displs, recvtype, root, comm, errflag); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_alltoall(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - int recvcount, MPI_Datatype recvtype, - MPIR_Comm * comm, MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_alltoall(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, MPIR_Errflag_t * errflag) { return MPIDI_NM_native_func->alltoall(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, comm, errflag); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_alltoallv(const void *sendbuf, const int *sendcounts, - const int *sdispls, MPI_Datatype sendtype, - void *recvbuf, const int *recvcounts, - const int *rdispls, MPI_Datatype recvtype, - MPIR_Comm * comm, MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_alltoallv(const void *sendbuf, const int *sendcounts, + const int *sdispls, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, + const int *rdispls, MPI_Datatype recvtype, + MPIR_Comm * comm, MPIR_Errflag_t * errflag) { return MPIDI_NM_native_func->alltoallv(sendbuf, sendcounts, sdispls, sendtype, recvbuf, recvcounts, rdispls, recvtype, comm, errflag); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_alltoallw(const void *sendbuf, const int *sendcounts, - const int *sdispls, - const MPI_Datatype sendtypes[], void *recvbuf, - const int *recvcounts, const int *rdispls, - const MPI_Datatype recvtypes[], - MPIR_Comm * comm, MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_alltoallw(const void *sendbuf, const int *sendcounts, + const int *sdispls, + const MPI_Datatype sendtypes[], void *recvbuf, + const int *recvcounts, const int *rdispls, + const MPI_Datatype recvtypes[], + MPIR_Comm * comm, MPIR_Errflag_t * errflag) { return MPIDI_NM_native_func->alltoallw(sendbuf, sendcounts, sdispls, sendtypes, recvbuf, recvcounts, rdispls, recvtypes, comm, errflag); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_reduce(const void *sendbuf, void *recvbuf, int count, - MPI_Datatype datatype, MPI_Op op, int root, - MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_reduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, int root, + MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) { return MPIDI_NM_native_func->reduce(sendbuf, recvbuf, count, datatype, op, root, comm_ptr, errflag); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_reduce_scatter(const void *sendbuf, void *recvbuf, - const int *recvcounts, - MPI_Datatype datatype, MPI_Op op, - MPIR_Comm * comm_ptr, - MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_reduce_scatter(const void *sendbuf, void *recvbuf, + const int *recvcounts, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) { return MPIDI_NM_native_func->reduce_scatter(sendbuf, recvbuf, recvcounts, datatype, op, comm_ptr, errflag); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_reduce_scatter_block(const void *sendbuf, void *recvbuf, - int recvcount, - MPI_Datatype datatype, MPI_Op op, - MPIR_Comm * comm_ptr, - MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_reduce_scatter_block(const void *sendbuf, void *recvbuf, + int recvcount, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) { return MPIDI_NM_native_func->reduce_scatter_block(sendbuf, recvbuf, recvcount, datatype, op, comm_ptr, errflag); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_scan(const void *sendbuf, void *recvbuf, int count, - MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm, - MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_scan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm, + MPIR_Errflag_t * errflag) { return MPIDI_NM_native_func->scan(sendbuf, recvbuf, count, datatype, op, comm, errflag); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_exscan(const void *sendbuf, void *recvbuf, int count, - MPI_Datatype datatype, MPI_Op op, - MPIR_Comm * comm, MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_exscan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, MPIR_Errflag_t * errflag) { return MPIDI_NM_native_func->exscan(sendbuf, recvbuf, count, datatype, op, comm, errflag); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_neighbor_allgather(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - int recvcount, MPI_Datatype recvtype, - MPIR_Comm * comm) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_neighbor_allgather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm) { return MPIDI_NM_native_func->neighbor_allgather(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, comm); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_neighbor_allgatherv(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - const int *recvcounts, - const int *displs, - MPI_Datatype recvtype, - MPIR_Comm * comm) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_neighbor_allgatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, + const int *displs, + MPI_Datatype recvtype, MPIR_Comm * comm) { return MPIDI_NM_native_func->neighbor_allgatherv(sendbuf, sendcount, sendtype, recvbuf, recvcounts, displs, recvtype, comm); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_neighbor_alltoallv(const void *sendbuf, - const int *sendcounts, - const int *sdispls, - MPI_Datatype sendtype, void *recvbuf, - const int *recvcounts, - const int *rdispls, - MPI_Datatype recvtype, - MPIR_Comm * comm) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_neighbor_alltoallv(const void *sendbuf, + const int *sendcounts, + const int *sdispls, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, + const int *rdispls, + MPI_Datatype recvtype, MPIR_Comm * comm) { return MPIDI_NM_native_func->neighbor_alltoallv(sendbuf, sendcounts, sdispls, sendtype, recvbuf, recvcounts, rdispls, recvtype, comm); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_neighbor_alltoallw(const void *sendbuf, - const int *sendcounts, - const MPI_Aint * sdispls, - const MPI_Datatype * sendtypes, - void *recvbuf, const int *recvcounts, - const MPI_Aint * rdispls, - const MPI_Datatype * recvtypes, - MPIR_Comm * comm) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_neighbor_alltoallw(const void *sendbuf, + const int *sendcounts, + const MPI_Aint * sdispls, + const MPI_Datatype * sendtypes, + void *recvbuf, const int *recvcounts, + const MPI_Aint * rdispls, + const MPI_Datatype * recvtypes, + MPIR_Comm * comm) { return MPIDI_NM_native_func->neighbor_alltoallw(sendbuf, sendcounts, sdispls, sendtypes, recvbuf, recvcounts, rdispls, recvtypes, comm); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_neighbor_alltoall(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - int recvcount, MPI_Datatype recvtype, - MPIR_Comm * comm) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_neighbor_alltoall(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm) { return MPIDI_NM_native_func->neighbor_alltoall(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, comm); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ineighbor_allgather(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - int recvcount, MPI_Datatype recvtype, - MPIR_Comm * comm, MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_ineighbor_allgather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, MPI_Request * req) { return MPIDI_NM_native_func->ineighbor_allgather(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, comm, req); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ineighbor_allgatherv(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, - void *recvbuf, - const int *recvcounts, - const int *displs, - MPI_Datatype recvtype, - MPIR_Comm * comm, MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_ineighbor_allgatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, + void *recvbuf, + const int *recvcounts, + const int *displs, + MPI_Datatype recvtype, + MPIR_Comm * comm, MPI_Request * req) { return MPIDI_NM_native_func->ineighbor_allgatherv(sendbuf, sendcount, sendtype, recvbuf, recvcounts, displs, recvtype, comm, req); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ineighbor_alltoall(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - int recvcount, MPI_Datatype recvtype, - MPIR_Comm * comm, MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_ineighbor_alltoall(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, MPI_Request * req) { return MPIDI_NM_native_func->ineighbor_alltoall(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, comm, req); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ineighbor_alltoallv(const void *sendbuf, - const int *sendcounts, - const int *sdispls, - MPI_Datatype sendtype, void *recvbuf, - const int *recvcounts, - const int *rdispls, - MPI_Datatype recvtype, - MPIR_Comm * comm, MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_ineighbor_alltoallv(const void *sendbuf, + const int *sendcounts, + const int *sdispls, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, + const int *rdispls, + MPI_Datatype recvtype, + MPIR_Comm * comm, MPI_Request * req) { return MPIDI_NM_native_func->ineighbor_alltoallv(sendbuf, sendcounts, sdispls, sendtype, recvbuf, recvcounts, rdispls, recvtype, comm, req); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ineighbor_alltoallw(const void *sendbuf, - const int *sendcounts, - const MPI_Aint * sdispls, - const MPI_Datatype * sendtypes, - void *recvbuf, const int *recvcounts, - const MPI_Aint * rdispls, - const MPI_Datatype * recvtypes, - MPIR_Comm * comm, MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_ineighbor_alltoallw(const void *sendbuf, + const int *sendcounts, + const MPI_Aint * sdispls, + const MPI_Datatype * sendtypes, + void *recvbuf, const int *recvcounts, + const MPI_Aint * rdispls, + const MPI_Datatype * recvtypes, + MPIR_Comm * comm, MPI_Request * req) { return MPIDI_NM_native_func->ineighbor_alltoallw(sendbuf, sendcounts, sdispls, sendtypes, recvbuf, recvcounts, rdispls, recvtypes, comm, req); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ibarrier(MPIR_Comm * comm, MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_ibarrier(MPIR_Comm * comm, MPI_Request * req) { return MPIDI_NM_native_func->ibarrier(comm, req); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ibcast(void *buffer, int count, MPI_Datatype datatype, - int root, MPIR_Comm * comm, MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_ibcast(void *buffer, int count, MPI_Datatype datatype, + int root, MPIR_Comm * comm, MPI_Request * req) { return MPIDI_NM_native_func->ibcast(buffer, count, datatype, root, comm, req); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_iallgather(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - int recvcount, MPI_Datatype recvtype, - MPIR_Comm * comm, MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_iallgather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, MPI_Request * req) { return MPIDI_NM_native_func->iallgather(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, comm, req); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_iallgatherv(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - const int *recvcounts, const int *displs, - MPI_Datatype recvtype, MPIR_Comm * comm, - MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_iallgatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, const int *displs, + MPI_Datatype recvtype, MPIR_Comm * comm, + MPI_Request * req) { return MPIDI_NM_native_func->iallgatherv(sendbuf, sendcount, sendtype, recvbuf, recvcounts, displs, recvtype, comm, req); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_iallreduce(const void *sendbuf, void *recvbuf, int count, - MPI_Datatype datatype, MPI_Op op, - MPIR_Comm * comm, MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_iallreduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, MPI_Request * req) { return MPIDI_NM_native_func->iallreduce(sendbuf, recvbuf, count, datatype, op, comm, req); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ialltoall(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - int recvcount, MPI_Datatype recvtype, - MPIR_Comm * comm, MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_ialltoall(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, MPI_Request * req) { return MPIDI_NM_native_func->ialltoall(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, comm, req); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ialltoallv(const void *sendbuf, const int *sendcounts, - const int *sdispls, MPI_Datatype sendtype, - void *recvbuf, const int *recvcounts, - const int *rdispls, MPI_Datatype recvtype, - MPIR_Comm * comm, MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_ialltoallv(const void *sendbuf, const int *sendcounts, + const int *sdispls, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, + const int *rdispls, MPI_Datatype recvtype, + MPIR_Comm * comm, MPI_Request * req) { return MPIDI_NM_native_func->ialltoallv(sendbuf, sendcounts, sdispls, sendtype, recvbuf, recvcounts, rdispls, recvtype, comm, req); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ialltoallw(const void *sendbuf, const int *sendcounts, - const int *sdispls, - const MPI_Datatype sendtypes[], void *recvbuf, - const int *recvcounts, const int *rdispls, - const MPI_Datatype recvtypes[], - MPIR_Comm * comm, MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_ialltoallw(const void *sendbuf, const int *sendcounts, + const int *sdispls, + const MPI_Datatype sendtypes[], void *recvbuf, + const int *recvcounts, const int *rdispls, + const MPI_Datatype recvtypes[], + MPIR_Comm * comm, MPI_Request * req) { return MPIDI_NM_native_func->ialltoallw(sendbuf, sendcounts, sdispls, sendtypes, recvbuf, recvcounts, rdispls, recvtypes, comm, req); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_iexscan(const void *sendbuf, void *recvbuf, int count, - MPI_Datatype datatype, MPI_Op op, - MPIR_Comm * comm, MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_iexscan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, MPI_Request * req) { return MPIDI_NM_native_func->iexscan(sendbuf, recvbuf, count, datatype, op, comm, req); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_igather(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - int recvcount, MPI_Datatype recvtype, int root, - MPIR_Comm * comm, MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_igather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, int root, + MPIR_Comm * comm, MPI_Request * req) { return MPIDI_NM_native_func->igather(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, root, comm, req); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_igatherv(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - const int *recvcounts, const int *displs, - MPI_Datatype recvtype, int root, - MPIR_Comm * comm, MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_igatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, const int *displs, + MPI_Datatype recvtype, int root, + MPIR_Comm * comm, MPI_Request * req) { return MPIDI_NM_native_func->igatherv(sendbuf, sendcount, sendtype, recvbuf, recvcounts, displs, recvtype, root, comm, req); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ireduce_scatter_block(const void *sendbuf, void *recvbuf, - int recvcount, - MPI_Datatype datatype, MPI_Op op, - MPIR_Comm * comm, - MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_ireduce_scatter_block(const void *sendbuf, void *recvbuf, + int recvcount, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, MPI_Request * req) { return MPIDI_NM_native_func->ireduce_scatter_block(sendbuf, recvbuf, recvcount, datatype, op, comm, req); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ireduce_scatter(const void *sendbuf, void *recvbuf, - const int *recvcounts, - MPI_Datatype datatype, MPI_Op op, - MPIR_Comm * comm, MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_ireduce_scatter(const void *sendbuf, void *recvbuf, + const int *recvcounts, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, MPI_Request * req) { return MPIDI_NM_native_func->ireduce_scatter(sendbuf, recvbuf, recvcounts, datatype, op, comm, req); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ireduce(const void *sendbuf, void *recvbuf, int count, - MPI_Datatype datatype, MPI_Op op, int root, - MPIR_Comm * comm_ptr, MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_ireduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, int root, + MPIR_Comm * comm_ptr, MPI_Request * req) { return MPIDI_NM_native_func->ireduce(sendbuf, recvbuf, count, datatype, op, root, comm_ptr, req); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_iscan(const void *sendbuf, void *recvbuf, int count, - MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm, - MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_iscan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm, + MPI_Request * req) { return MPIDI_NM_native_func->iscan(sendbuf, recvbuf, count, datatype, op, comm, req); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_iscatter(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - int recvcount, MPI_Datatype recvtype, int root, - MPIR_Comm * comm, MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_iscatter(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, int root, + MPIR_Comm * comm, MPI_Request * req) { return MPIDI_NM_native_func->iscatter(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, root, comm, req); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_iscatterv(const void *sendbuf, const int *sendcounts, - const int *displs, MPI_Datatype sendtype, - void *recvbuf, int recvcount, - MPI_Datatype recvtype, int root, - MPIR_Comm * comm_ptr, MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_NM_iscatterv(const void *sendbuf, const int *sendcounts, + const int *displs, MPI_Datatype sendtype, + void *recvbuf, int recvcount, + MPI_Datatype recvtype, int root, + MPIR_Comm * comm_ptr, MPI_Request * req) { return MPIDI_NM_native_func->iscatterv(sendbuf, sendcounts, displs, sendtype, recvbuf, recvcount, recvtype, root, comm_ptr, req); }; -MPIDI_NM_STATIC_INLINE_PREFIX void MPIDI_NM_datatype_commit(MPIR_Datatype * datatype_p) +MPL_STATIC_INLINE_PREFIX void MPIDI_NM_datatype_commit(MPIR_Datatype * datatype_p) { return MPIDI_NM_native_func->datatype_commit(datatype_p); }; -MPIDI_NM_STATIC_INLINE_PREFIX void MPIDI_NM_datatype_dup(MPIR_Datatype * old_datatype_p, - MPIR_Datatype * new_datatype_p) +MPL_STATIC_INLINE_PREFIX void MPIDI_NM_datatype_dup(MPIR_Datatype * old_datatype_p, + MPIR_Datatype * new_datatype_p) { return MPIDI_NM_native_func->datatype_dup(old_datatype_p, new_datatype_p); }; -MPIDI_NM_STATIC_INLINE_PREFIX void MPIDI_NM_datatype_destroy(MPIR_Datatype * datatype_p) +MPL_STATIC_INLINE_PREFIX void MPIDI_NM_datatype_destroy(MPIR_Datatype * datatype_p) { return MPIDI_NM_native_func->datatype_destroy(datatype_p); }; -MPIDI_NM_STATIC_INLINE_PREFIX void MPIDI_NM_op_commit(MPIR_Op * op_p) +MPL_STATIC_INLINE_PREFIX void MPIDI_NM_op_commit(MPIR_Op * op_p) { return MPIDI_NM_native_func->op_commit(op_p); }; -MPIDI_NM_STATIC_INLINE_PREFIX void MPIDI_NM_op_destroy(MPIR_Op * op_p) +MPL_STATIC_INLINE_PREFIX void MPIDI_NM_op_destroy(MPIR_Op * op_p) { return MPIDI_NM_native_func->op_destroy(op_p); }; diff --git a/src/mpid/ch4/shm/include/shm.h b/src/mpid/ch4/shm/include/shm.h index 1fa4138..0ef4704 100644 --- a/src/mpid/ch4/shm/include/shm.h +++ b/src/mpid/ch4/shm/include/shm.h @@ -517,676 +517,549 @@ extern MPIDI_SHM_native_funcs_t *MPIDI_SHM_native_func; extern int MPIDI_num_shms; extern char MPIDI_SHM_strings[][MPIDI_MAX_SHM_STRING_LEN]; -#ifndef MPIDI_SHM_STATIC_INLINE_PREFIX -#define MPIDI_SHM_STATIC_INLINE_PREFIX __attribute__((always_inline)) static inline -#endif - -#ifndef MPIDI_SHM_STATIC_INLINE_SUFFIX -#define MPIDI_SHM_STATIC_INLINE_SUFFIX __attribute__((always_inline)) -#endif - -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_init(int rank, - int size) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_finalize(void) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_progress(int blocking) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_reg_hdr_handler(int handler_id, - MPIDI_SHM_am_origin_handler_fn - origin_handler_fn, - MPIDI_SHM_am_target_handler_fn - target_handler_fn) - MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_connect(const char *port_name, MPIR_Info * info, - int root, MPIR_Comm * comm, - MPIR_Comm ** - newcomm_ptr) - MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_disconnect(MPIR_Comm * - comm_ptr) - MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_open_port(MPIR_Info * info_ptr, - char *port_name) - MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_close_port(const char *port_name) - MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_accept(const char *port_name, MPIR_Info * info, - int root, MPIR_Comm * comm, - MPIR_Comm ** - newcomm_ptr) - MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_send_am_hdr(int rank, MPIR_Comm * comm, int handler_id, +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_init(int rank, int size) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_finalize(void) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_progress(int blocking) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_reg_hdr_handler(int handler_id, + MPIDI_SHM_am_origin_handler_fn + origin_handler_fn, + MPIDI_SHM_am_target_handler_fn + target_handler_fn) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_connect(const char *port_name, MPIR_Info * info, + int root, MPIR_Comm * comm, + MPIR_Comm ** + newcomm_ptr) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_disconnect(MPIR_Comm * + comm_ptr) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_open_port(MPIR_Info * info_ptr, + char *port_name) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_close_port(const char *port_name) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_accept(const char *port_name, MPIR_Info * info, + int root, MPIR_Comm * comm, + MPIR_Comm ** + newcomm_ptr) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_send_am_hdr(int rank, MPIR_Comm * comm, int handler_id, + const void *am_hdr, size_t am_hdr_sz, + MPIR_Request * sreq, + void *shm_context) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_am_hdr(int rank, MPIR_Comm * comm, + int handler_id, const void *am_hdr, + size_t am_hdr_sz, + void *shm_context) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_send_am(int rank, MPIR_Comm * comm, int handler_id, + const void *am_hdr, size_t am_hdr_sz, + const void *data, MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq, + void *shm_context) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_am(int rank, MPIR_Comm * comm, int handler_id, + const void *am_hdr, size_t am_hdr_sz, + const void *data, MPI_Count count, + MPI_Datatype datatype, + void *shm_context) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_send_amv(int rank, MPIR_Comm * comm, int handler_id, + struct iovec *am_hdrs, size_t iov_len, + const void *data, MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq, + void *shm_context) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_amv(int rank, MPIR_Comm * comm, int handler_id, + struct iovec *am_hdrs, size_t iov_len, + const void *data, MPI_Count count, + MPI_Datatype datatype, + void *shm_context) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_send_am_hdr_reply(MPIR_Context_id_t context_id, + int src_rank, int handler_id, const void *am_hdr, size_t am_hdr_sz, - MPIR_Request * sreq, - void *shm_context) - MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_am_hdr(int rank, MPIR_Comm * comm, - int handler_id, const void *am_hdr, - size_t am_hdr_sz, - void *shm_context) - MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_send_am(int rank, MPIR_Comm * comm, int handler_id, + MPIR_Request * + sreq) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_am_hdr_reply(MPIR_Context_id_t context_id, + int src_rank, int handler_id, + const void *am_hdr, + size_t am_hdr_sz) + MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_send_am_reply(MPIR_Context_id_t context_id, + int src_rank, int handler_id, const void *am_hdr, size_t am_hdr_sz, const void *data, MPI_Count count, - MPI_Datatype datatype, MPIR_Request * sreq, - void *shm_context) - MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_am(int rank, MPIR_Comm * comm, int handler_id, + MPI_Datatype datatype, + MPIR_Request * sreq) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_am_reply(MPIR_Context_id_t context_id, + int src_rank, int handler_id, const void *am_hdr, size_t am_hdr_sz, const void *data, MPI_Count count, - MPI_Datatype datatype, - void *shm_context) - MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_send_amv(int rank, MPIR_Comm * comm, int handler_id, - struct iovec *am_hdrs, size_t iov_len, + MPI_Datatype datatype) + MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_send_amv_reply(MPIR_Context_id_t context_id, + int src_rank, int handler_id, + struct iovec *am_hdr, size_t iov_len, const void *data, MPI_Count count, - MPI_Datatype datatype, MPIR_Request * sreq, - void *shm_context) - MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_amv(int rank, MPIR_Comm * comm, int handler_id, + MPI_Datatype datatype, + MPIR_Request * sreq) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_amv_reply(MPIR_Context_id_t context_id, + int src_rank, int handler_id, struct iovec *am_hdrs, size_t iov_len, const void *data, MPI_Count count, + MPI_Datatype datatype) + MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX size_t MPIDI_SHM_am_hdr_max_sz(void) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX size_t MPIDI_SHM_am_inject_max_sz(void) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_am_recv(MPIR_Request * req) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_get_lpid(MPIR_Comm * comm_ptr, int idx, + int *lpid_ptr, + MPL_bool is_remote) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_gpid_get(MPIR_Comm * comm_ptr, int rank, + MPIR_Gpid * gpid) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_get_node_id(MPIR_Comm * comm, int rank, + MPID_Node_id_t * id_p) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_get_max_node_id(MPIR_Comm * comm, + MPID_Node_id_t * + max_id_p) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_getallincomm(MPIR_Comm * comm_ptr, int local_size, + MPIR_Gpid local_gpid[], + int *singleAVT) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_gpid_tolpidarray(int size, MPIR_Gpid gpid[], + int lpid[]) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_create_intercomm_from_lpids(MPIR_Comm * newcomm_ptr, + int size, + const int lpids[]) + MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_create(MPIR_Comm * comm) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_destroy(MPIR_Comm * comm) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX void MPIDI_SHM_am_request_init(MPIR_Request * + req) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX void MPIDI_SHM_am_request_finalize(MPIR_Request * + req) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_send(const void *buf, int count, MPI_Datatype datatype, + int rank, int tag, MPIR_Comm * comm, + int context_offset, + MPIR_Request ** request) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_ssend(const void *buf, int count, + MPI_Datatype datatype, int rank, int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_startall(int count, + MPIR_Request * requests[]) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_send_init(const void *buf, int count, + MPI_Datatype datatype, int rank, int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_ssend_init(const void *buf, int count, + MPI_Datatype datatype, int rank, int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_rsend_init(const void *buf, int count, + MPI_Datatype datatype, int rank, int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_bsend_init(const void *buf, int count, + MPI_Datatype datatype, int rank, int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_isend(const void *buf, int count, + MPI_Datatype datatype, int rank, int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_issend(const void *buf, int count, + MPI_Datatype datatype, int rank, int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_cancel_send(MPIR_Request * sreq) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_recv_init(void *buf, int count, MPI_Datatype datatype, + int rank, int tag, MPIR_Comm * comm, + int context_offset, + MPIR_Request ** request) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_recv(void *buf, int count, MPI_Datatype datatype, + int rank, int tag, MPIR_Comm * comm, + int context_offset, MPI_Status * status, + MPIR_Request ** request) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_irecv(void *buf, int count, MPI_Datatype datatype, + int rank, int tag, MPIR_Comm * comm, + int context_offset, + MPIR_Request ** request) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_imrecv(void *buf, int count, MPI_Datatype datatype, + MPIR_Request * message, + MPIR_Request ** rreqp) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_cancel_recv(MPIR_Request * rreq) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX void *MPIDI_SHM_alloc_mem(size_t size, + MPIR_Info * info_ptr) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_free_mem(void *ptr) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_improbe(int source, int tag, MPIR_Comm * comm, + int context_offset, int *flag, + MPIR_Request ** message, + MPI_Status * status) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_iprobe(int source, int tag, MPIR_Comm * comm, + int context_offset, int *flag, + MPI_Status * status) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_win_set_info(MPIR_Win * win, + MPIR_Info * info) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_win_shared_query(MPIR_Win * win, int rank, + MPI_Aint * size, int *disp_unit, + void *baseptr) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_put(const void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, int target_rank, + MPI_Aint target_disp, int target_count, + MPI_Datatype target_datatype, + MPIR_Win * win) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_win_start(MPIR_Group * group, int assert, + MPIR_Win * win) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_win_complete(MPIR_Win * win) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_win_post(MPIR_Group * group, int assert, + MPIR_Win * win) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_win_wait(MPIR_Win * win) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_win_test(MPIR_Win * win, int *flag) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_win_lock(int lock_type, int rank, int assert, + MPIR_Win * win) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_win_unlock(int rank, + MPIR_Win * win) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_win_get_info(MPIR_Win * win, + MPIR_Info ** info_p_p) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_get(void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, int target_rank, + MPI_Aint target_disp, int target_count, + MPI_Datatype target_datatype, + MPIR_Win * win) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_win_free(MPIR_Win ** win_ptr) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_win_fence(int assert, + MPIR_Win * win) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_win_create(void *base, MPI_Aint length, int disp_unit, + MPIR_Info * info, MPIR_Comm * comm_ptr, + MPIR_Win ** win_ptr) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_accumulate(const void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, + int target_rank, MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, MPI_Op op, + MPIR_Win * win) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_win_attach(MPIR_Win * win, void *base, + MPI_Aint size) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_win_allocate_shared(MPI_Aint size, int disp_unit, + MPIR_Info * info_ptr, + MPIR_Comm * comm_ptr, + void **base_ptr, + MPIR_Win ** + win_ptr) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_rput(const void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, int target_rank, + MPI_Aint target_disp, int target_count, + MPI_Datatype target_datatype, MPIR_Win * win, + MPIR_Request ** request) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_win_flush_local(int rank, + MPIR_Win * win) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_win_detach(MPIR_Win * win, + const void *base) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_compare_and_swap(const void *origin_addr, + const void *compare_addr, + void *result_addr, MPI_Datatype datatype, - void *shm_context) - MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_send_am_hdr_reply(MPIR_Context_id_t context_id, - int src_rank, int handler_id, - const void *am_hdr, size_t am_hdr_sz, - MPIR_Request * - sreq) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_am_hdr_reply(MPIR_Context_id_t context_id, - int src_rank, int handler_id, - const void *am_hdr, - size_t am_hdr_sz) - MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_send_am_reply(MPIR_Context_id_t context_id, - int src_rank, int handler_id, - const void *am_hdr, size_t am_hdr_sz, - const void *data, MPI_Count count, - MPI_Datatype datatype, - MPIR_Request * - sreq) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_am_reply(MPIR_Context_id_t context_id, - int src_rank, int handler_id, - const void *am_hdr, size_t am_hdr_sz, - const void *data, MPI_Count count, - MPI_Datatype datatype) - MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_send_amv_reply(MPIR_Context_id_t context_id, - int src_rank, int handler_id, - struct iovec *am_hdr, size_t iov_len, - const void *data, MPI_Count count, - MPI_Datatype datatype, - MPIR_Request * - sreq) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_amv_reply(MPIR_Context_id_t context_id, - int src_rank, int handler_id, - struct iovec *am_hdrs, size_t iov_len, - const void *data, MPI_Count count, - MPI_Datatype datatype) - MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX size_t MPIDI_SHM_am_hdr_max_sz(void) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX size_t MPIDI_SHM_am_inject_max_sz(void) - MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_am_recv(MPIR_Request * - req) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_get_lpid(MPIR_Comm * comm_ptr, int idx, - int *lpid_ptr, - MPL_bool is_remote) - MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_gpid_get(MPIR_Comm * comm_ptr, int rank, - MPIR_Gpid * - gpid) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_get_node_id(MPIR_Comm * comm, int rank, - MPID_Node_id_t * - id_p) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_get_max_node_id(MPIR_Comm * comm, - MPID_Node_id_t * - max_id_p) - MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_getallincomm(MPIR_Comm * comm_ptr, int local_size, - MPIR_Gpid local_gpid[], - int *singleAVT) - MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_gpid_tolpidarray(int size, MPIR_Gpid gpid[], - int lpid[]) - MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_create_intercomm_from_lpids(MPIR_Comm * newcomm_ptr, - int size, - const int lpids[]) - MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_create(MPIR_Comm * - comm) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_destroy(MPIR_Comm * - comm) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX void MPIDI_SHM_am_request_init(MPIR_Request * - req) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX void MPIDI_SHM_am_request_finalize(MPIR_Request * - req) - MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_send(const void *buf, int count, MPI_Datatype datatype, - int rank, int tag, MPIR_Comm * comm, - int context_offset, - MPIR_Request ** - request) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ssend(const void *buf, int count, - MPI_Datatype datatype, int rank, int tag, - MPIR_Comm * comm, int context_offset, - MPIR_Request ** - request) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_startall(int count, - MPIR_Request * - requests[]) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_send_init(const void *buf, int count, - MPI_Datatype datatype, int rank, int tag, - MPIR_Comm * comm, int context_offset, - MPIR_Request ** - request) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ssend_init(const void *buf, int count, - MPI_Datatype datatype, int rank, int tag, - MPIR_Comm * comm, int context_offset, - MPIR_Request ** - request) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_rsend_init(const void *buf, int count, - MPI_Datatype datatype, int rank, int tag, - MPIR_Comm * comm, int context_offset, - MPIR_Request ** - request) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_bsend_init(const void *buf, int count, - MPI_Datatype datatype, int rank, int tag, - MPIR_Comm * comm, int context_offset, - MPIR_Request ** - request) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_isend(const void *buf, int count, - MPI_Datatype datatype, int rank, int tag, - MPIR_Comm * comm, int context_offset, + int target_rank, MPI_Aint target_disp, + MPIR_Win * win) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_raccumulate(const void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, + int target_rank, MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, MPI_Op op, + MPIR_Win * win, MPIR_Request ** - request) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_issend(const void *buf, int count, - MPI_Datatype datatype, int rank, int tag, - MPIR_Comm * comm, int context_offset, - MPIR_Request ** - request) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_cancel_send(MPIR_Request * - sreq) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_recv_init(void *buf, int count, MPI_Datatype datatype, - int rank, int tag, MPIR_Comm * comm, - int context_offset, + request) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_rget_accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + void *result_addr, int result_count, + MPI_Datatype result_datatype, + int target_rank, MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPI_Op op, MPIR_Win * win, MPIR_Request ** - request) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_recv(void *buf, int count, MPI_Datatype datatype, - int rank, int tag, MPIR_Comm * comm, - int context_offset, MPI_Status * status, - MPIR_Request ** - request) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_irecv(void *buf, int count, MPI_Datatype datatype, - int rank, int tag, MPIR_Comm * comm, - int context_offset, - MPIR_Request ** - request) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_imrecv(void *buf, int count, MPI_Datatype datatype, - MPIR_Request * message, - MPIR_Request ** - rreqp) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_cancel_recv(MPIR_Request * - rreq) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX void *MPIDI_SHM_alloc_mem(size_t size, - MPIR_Info * - info_ptr) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_free_mem(void *ptr) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_improbe(int source, int tag, MPIR_Comm * comm, - int context_offset, int *flag, - MPIR_Request ** message, - MPI_Status * - status) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_iprobe(int source, int tag, MPIR_Comm * comm, - int context_offset, int *flag, - MPI_Status * - status) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_set_info(MPIR_Win * win, - MPIR_Info * - info) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_shared_query(MPIR_Win * win, int rank, - MPI_Aint * size, int *disp_unit, - void *baseptr) - MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_put(const void *origin_addr, int origin_count, - MPI_Datatype origin_datatype, int target_rank, - MPI_Aint target_disp, int target_count, - MPI_Datatype target_datatype, - MPIR_Win * win) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_start(MPIR_Group * group, int assert, - MPIR_Win * - win) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_complete(MPIR_Win * - win) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_post(MPIR_Group * group, int assert, - MPIR_Win * - win) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_wait(MPIR_Win * - win) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_test(MPIR_Win * win, - int *flag) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_lock(int lock_type, int rank, int assert, - MPIR_Win * - win) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_unlock(int rank, - MPIR_Win * - win) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_get_info(MPIR_Win * win, - MPIR_Info ** - info_p_p) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_get(void *origin_addr, int origin_count, - MPI_Datatype origin_datatype, int target_rank, - MPI_Aint target_disp, int target_count, - MPI_Datatype target_datatype, - MPIR_Win * win) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_free(MPIR_Win ** - win_ptr) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_fence(int assert, - MPIR_Win * - win) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_create(void *base, MPI_Aint length, int disp_unit, - MPIR_Info * info, MPIR_Comm * comm_ptr, - MPIR_Win ** - win_ptr) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_accumulate(const void *origin_addr, int origin_count, - MPI_Datatype origin_datatype, - int target_rank, MPI_Aint target_disp, - int target_count, - MPI_Datatype target_datatype, MPI_Op op, - MPIR_Win * - win) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_attach(MPIR_Win * win, void *base, - MPI_Aint size) - MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_allocate_shared(MPI_Aint size, int disp_unit, - MPIR_Info * info_ptr, - MPIR_Comm * comm_ptr, - void **base_ptr, - MPIR_Win ** - win_ptr) - MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_rput(const void *origin_addr, int origin_count, - MPI_Datatype origin_datatype, int target_rank, - MPI_Aint target_disp, int target_count, - MPI_Datatype target_datatype, MPIR_Win * win, - MPIR_Request ** - request) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_flush_local(int rank, - MPIR_Win * - win) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_detach(MPIR_Win * win, - const void *base) - MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_compare_and_swap(const void *origin_addr, - const void *compare_addr, - void *result_addr, - MPI_Datatype datatype, - int target_rank, MPI_Aint target_disp, - MPIR_Win * - win) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_raccumulate(const void *origin_addr, int origin_count, - MPI_Datatype origin_datatype, - int target_rank, MPI_Aint target_disp, - int target_count, - MPI_Datatype target_datatype, MPI_Op op, - MPIR_Win * win, - MPIR_Request ** - request) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_rget_accumulate(const void *origin_addr, - int origin_count, - MPI_Datatype origin_datatype, - void *result_addr, int result_count, - MPI_Datatype result_datatype, - int target_rank, MPI_Aint target_disp, - int target_count, - MPI_Datatype target_datatype, - MPI_Op op, MPIR_Win * win, - MPIR_Request ** - request) - MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_fetch_and_op(const void *origin_addr, - void *result_addr, MPI_Datatype datatype, - int target_rank, MPI_Aint target_disp, - MPI_Op op, - MPIR_Win * - win) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_allocate(MPI_Aint size, int disp_unit, - MPIR_Info * info, MPIR_Comm * comm, - void *baseptr, - MPIR_Win ** - win) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_flush(int rank, - MPIR_Win * - win) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_flush_local_all(MPIR_Win * - win) - MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_unlock_all(MPIR_Win * - win) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_create_dynamic(MPIR_Info * info, MPIR_Comm * comm, - MPIR_Win ** - win) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_rget(void *origin_addr, int origin_count, - MPI_Datatype origin_datatype, int target_rank, - MPI_Aint target_disp, int target_count, - MPI_Datatype target_datatype, MPIR_Win * win, - MPIR_Request ** - request) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_sync(MPIR_Win * - win) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_flush_all(MPIR_Win * - win) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_get_accumulate(const void *origin_addr, - int origin_count, - MPI_Datatype origin_datatype, - void *result_addr, int result_count, - MPI_Datatype result_datatype, - int target_rank, MPI_Aint target_disp, - int target_count, - MPI_Datatype target_datatype, MPI_Op op, - MPIR_Win * - win) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_lock_all(int assert, - MPIR_Win * - win) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_barrier(MPIR_Comm * comm, - MPIR_Errflag_t * - errflag) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_bcast(void *buffer, int count, MPI_Datatype datatype, - int root, MPIR_Comm * comm, - MPIR_Errflag_t * - errflag) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_allreduce(const void *sendbuf, void *recvbuf, - int count, MPI_Datatype datatype, MPI_Op op, - MPIR_Comm * comm, - MPIR_Errflag_t * - errflag) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_allgather(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - int recvcount, MPI_Datatype recvtype, - MPIR_Comm * comm, - MPIR_Errflag_t * - errflag) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_allgatherv(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - const int *recvcounts, const int *displs, - MPI_Datatype recvtype, MPIR_Comm * comm, - MPIR_Errflag_t * - errflag) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_scatter(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - int recvcount, MPI_Datatype recvtype, int root, - MPIR_Comm * comm, - MPIR_Errflag_t * - errflag) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_scatterv(const void *sendbuf, const int *sendcounts, - const int *displs, MPI_Datatype sendtype, - void *recvbuf, int recvcount, - MPI_Datatype recvtype, int root, + request) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_fetch_and_op(const void *origin_addr, + void *result_addr, MPI_Datatype datatype, + int target_rank, MPI_Aint target_disp, + MPI_Op op, + MPIR_Win * win) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_win_allocate(MPI_Aint size, int disp_unit, + MPIR_Info * info, MPIR_Comm * comm, + void *baseptr, + MPIR_Win ** win) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_win_flush(int rank, MPIR_Win * win) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_win_flush_local_all(MPIR_Win * win) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_win_unlock_all(MPIR_Win * win) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_win_create_dynamic(MPIR_Info * info, MPIR_Comm * comm, + MPIR_Win ** win) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_rget(void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, int target_rank, + MPI_Aint target_disp, int target_count, + MPI_Datatype target_datatype, MPIR_Win * win, + MPIR_Request ** request) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_win_sync(MPIR_Win * win) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_win_flush_all(MPIR_Win * win) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_get_accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + void *result_addr, int result_count, + MPI_Datatype result_datatype, + int target_rank, MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, MPI_Op op, + MPIR_Win * win) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_win_lock_all(int assert, + MPIR_Win * win) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_barrier(MPIR_Comm * comm, + MPIR_Errflag_t * errflag) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_bcast(void *buffer, int count, MPI_Datatype datatype, + int root, MPIR_Comm * comm, + MPIR_Errflag_t * errflag) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_allreduce(const void *sendbuf, void *recvbuf, + int count, MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, + MPIR_Errflag_t * errflag) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_allgather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, + MPIR_Errflag_t * errflag) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_allgatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, const int *displs, + MPI_Datatype recvtype, MPIR_Comm * comm, + MPIR_Errflag_t * + errflag) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_scatter(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, int root, + MPIR_Comm * comm, + MPIR_Errflag_t * errflag) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_scatterv(const void *sendbuf, const int *sendcounts, + const int *displs, MPI_Datatype sendtype, + void *recvbuf, int recvcount, + MPI_Datatype recvtype, int root, + MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_gather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, int root, + MPIR_Comm * comm, + MPIR_Errflag_t * errflag) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_gatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, const int *displs, + MPI_Datatype recvtype, int root, + MPIR_Comm * comm, + MPIR_Errflag_t * errflag) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_alltoall(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, + MPIR_Errflag_t * errflag) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_alltoallv(const void *sendbuf, const int *sendcounts, + const int *sdispls, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, + const int *rdispls, MPI_Datatype recvtype, + MPIR_Comm * comm, + MPIR_Errflag_t * errflag) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_alltoallw(const void *sendbuf, const int *sendcounts, + const int *sdispls, + const MPI_Datatype sendtypes[], + void *recvbuf, const int *recvcounts, + const int *rdispls, + const MPI_Datatype recvtypes[], + MPIR_Comm * comm, + MPIR_Errflag_t * errflag) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_reduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, int root, + MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_reduce_scatter(const void *sendbuf, void *recvbuf, + const int *recvcounts, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm_ptr, MPIR_Errflag_t * - errflag) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_gather(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - int recvcount, MPI_Datatype recvtype, int root, - MPIR_Comm * comm, - MPIR_Errflag_t * - errflag) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_gatherv(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - const int *recvcounts, const int *displs, - MPI_Datatype recvtype, int root, - MPIR_Comm * comm, - MPIR_Errflag_t * - errflag) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_alltoall(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - int recvcount, MPI_Datatype recvtype, - MPIR_Comm * comm, - MPIR_Errflag_t * - errflag) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_alltoallv(const void *sendbuf, const int *sendcounts, - const int *sdispls, MPI_Datatype sendtype, - void *recvbuf, const int *recvcounts, - const int *rdispls, MPI_Datatype recvtype, - MPIR_Comm * comm, - MPIR_Errflag_t * - errflag) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_alltoallw(const void *sendbuf, const int *sendcounts, - const int *sdispls, - const MPI_Datatype sendtypes[], - void *recvbuf, const int *recvcounts, - const int *rdispls, - const MPI_Datatype recvtypes[], - MPIR_Comm * comm, - MPIR_Errflag_t * - errflag) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_reduce(const void *sendbuf, void *recvbuf, int count, - MPI_Datatype datatype, MPI_Op op, int root, - MPIR_Comm * comm_ptr, - MPIR_Errflag_t * - errflag) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_reduce_scatter(const void *sendbuf, void *recvbuf, - const int *recvcounts, + errflag) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_reduce_scatter_block(const void *sendbuf, + void *recvbuf, int recvcount, MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm_ptr, MPIR_Errflag_t * - errflag) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_reduce_scatter_block(const void *sendbuf, - void *recvbuf, int recvcount, - MPI_Datatype datatype, MPI_Op op, - MPIR_Comm * comm_ptr, - MPIR_Errflag_t * - errflag) - MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_scan(const void *sendbuf, void *recvbuf, int count, - MPI_Datatype datatype, MPI_Op op, - MPIR_Comm * comm, - MPIR_Errflag_t * - errflag) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_exscan(const void *sendbuf, void *recvbuf, int count, - MPI_Datatype datatype, MPI_Op op, - MPIR_Comm * comm, - MPIR_Errflag_t * - errflag) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_neighbor_allgather(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, - void *recvbuf, int recvcount, - MPI_Datatype recvtype, - MPIR_Comm * comm, - MPIR_Errflag_t * - errflag) - MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_neighbor_allgatherv(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, - void *recvbuf, - const int *recvcounts, - const int *displs, - MPI_Datatype recvtype, - MPIR_Comm * comm, - MPIR_Errflag_t * - errflag) - MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_neighbor_alltoallv(const void *sendbuf, - const int *sendcounts, - const int *sdispls, - MPI_Datatype sendtype, - void *recvbuf, - const int *recvcounts, - const int *rdispls, - MPI_Datatype recvtype, - MPIR_Comm * comm, - MPIR_Errflag_t * - errflag) - MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_neighbor_alltoallw(const void *sendbuf, - const int *sendcounts, - const MPI_Aint * sdispls, - const MPI_Datatype * sendtypes, - void *recvbuf, - const int *recvcounts, - const MPI_Aint * rdispls, - const MPI_Datatype * recvtypes, - MPIR_Comm * comm, - MPIR_Errflag_t * - errflag) - MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_neighbor_alltoall(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - int recvcount, MPI_Datatype recvtype, - MPIR_Comm * comm, - MPIR_Errflag_t * - errflag) - MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ineighbor_allgather(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, - void *recvbuf, int recvcount, - MPI_Datatype recvtype, - MPIR_Comm * comm, - MPI_Request * - req) - MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ineighbor_allgatherv(const void *sendbuf, - int sendcount, - MPI_Datatype sendtype, - void *recvbuf, - const int *recvcounts, - const int *displs, - MPI_Datatype recvtype, - MPIR_Comm * comm, - MPI_Request * - req) - MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ineighbor_alltoall(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, - void *recvbuf, int recvcount, - MPI_Datatype recvtype, - MPIR_Comm * comm, - MPI_Request * - req) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ineighbor_alltoallv(const void *sendbuf, - const int *sendcounts, - const int *sdispls, - MPI_Datatype sendtype, - void *recvbuf, - const int *recvcounts, - const int *rdispls, - MPI_Datatype recvtype, - MPIR_Comm * comm, - MPI_Request * - req) - MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ineighbor_alltoallw(const void *sendbuf, - const int *sendcounts, - const MPI_Aint * sdispls, - const MPI_Datatype * sendtypes, - void *recvbuf, - const int *recvcounts, - const MPI_Aint * rdispls, - const MPI_Datatype * recvtypes, - MPIR_Comm * comm, - MPI_Request * - req) - MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ibarrier(MPIR_Comm * comm, - MPI_Request * - req) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ibcast(void *buffer, int count, MPI_Datatype datatype, - int root, MPIR_Comm * comm, - MPI_Request * - req) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_iallgather(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - int recvcount, MPI_Datatype recvtype, - MPIR_Comm * comm, - MPI_Request * - req) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_iallgatherv(const void *sendbuf, int sendcount, + errflag) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_scan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, + MPIR_Errflag_t * errflag) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_exscan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, + MPIR_Errflag_t * errflag) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_neighbor_allgather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, + void *recvbuf, int recvcount, + MPI_Datatype recvtype, + MPIR_Comm * comm, + MPIR_Errflag_t * + errflag) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_neighbor_allgatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, + void *recvbuf, + const int *recvcounts, + const int *displs, + MPI_Datatype recvtype, + MPIR_Comm * comm, + MPIR_Errflag_t * + errflag) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_neighbor_alltoallv(const void *sendbuf, + const int *sendcounts, + const int *sdispls, + MPI_Datatype sendtype, + void *recvbuf, + const int *recvcounts, + const int *rdispls, + MPI_Datatype recvtype, + MPIR_Comm * comm, + MPIR_Errflag_t * + errflag) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_neighbor_alltoallw(const void *sendbuf, + const int *sendcounts, + const MPI_Aint * sdispls, + const MPI_Datatype * sendtypes, + void *recvbuf, + const int *recvcounts, + const MPI_Aint * rdispls, + const MPI_Datatype * recvtypes, + MPIR_Comm * comm, + MPIR_Errflag_t * + errflag) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_neighbor_alltoall(const void *sendbuf, int sendcount, MPI_Datatype sendtype, void *recvbuf, - const int *recvcounts, const int *displs, - MPI_Datatype recvtype, MPIR_Comm * comm, - MPI_Request * - req) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_iallreduce(const void *sendbuf, void *recvbuf, - int count, MPI_Datatype datatype, MPI_Op op, - MPIR_Comm * comm, - MPI_Request * - req) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ialltoall(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - int recvcount, MPI_Datatype recvtype, - MPIR_Comm * comm, - MPI_Request * - req) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ialltoallv(const void *sendbuf, const int *sendcounts, - const int *sdispls, MPI_Datatype sendtype, - void *recvbuf, const int *recvcounts, - const int *rdispls, MPI_Datatype recvtype, - MPIR_Comm * comm, - MPI_Request * - req) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ialltoallw(const void *sendbuf, const int *sendcounts, - const int *sdispls, - const MPI_Datatype sendtypes[], - void *recvbuf, const int *recvcounts, - const int *rdispls, - const MPI_Datatype recvtypes[], - MPIR_Comm * comm, - MPI_Request * - req) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_iexscan(const void *sendbuf, void *recvbuf, int count, - MPI_Datatype datatype, MPI_Op op, - MPIR_Comm * comm, - MPI_Request * - req) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_igather(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - int recvcount, MPI_Datatype recvtype, int root, - MPIR_Comm * comm, - MPI_Request * - req) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_igatherv(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - const int *recvcounts, const int *displs, - MPI_Datatype recvtype, int root, - MPIR_Comm * comm, - MPI_Request * - req) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ireduce_scatter_block(const void *sendbuf, - void *recvbuf, int recvcount, - MPI_Datatype datatype, MPI_Op op, - MPIR_Comm * comm, - MPI_Request * - req) - MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ireduce_scatter(const void *sendbuf, void *recvbuf, - const int *recvcounts, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, + MPIR_Errflag_t * + errflag) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_ineighbor_allgather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, + void *recvbuf, int recvcount, + MPI_Datatype recvtype, + MPIR_Comm * comm, + MPI_Request * + req) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_ineighbor_allgatherv(const void *sendbuf, + int sendcount, + MPI_Datatype sendtype, + void *recvbuf, + const int *recvcounts, + const int *displs, + MPI_Datatype recvtype, + MPIR_Comm * comm, + MPI_Request * + req) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_ineighbor_alltoall(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, + void *recvbuf, int recvcount, + MPI_Datatype recvtype, + MPIR_Comm * comm, + MPI_Request * + req) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_ineighbor_alltoallv(const void *sendbuf, + const int *sendcounts, + const int *sdispls, + MPI_Datatype sendtype, + void *recvbuf, + const int *recvcounts, + const int *rdispls, + MPI_Datatype recvtype, + MPIR_Comm * comm, + MPI_Request * + req) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_ineighbor_alltoallw(const void *sendbuf, + const int *sendcounts, + const MPI_Aint * sdispls, + const MPI_Datatype * sendtypes, + void *recvbuf, + const int *recvcounts, + const MPI_Aint * rdispls, + const MPI_Datatype * recvtypes, + MPIR_Comm * comm, + MPI_Request * + req) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_ibarrier(MPIR_Comm * comm, + MPI_Request * req) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_ibcast(void *buffer, int count, MPI_Datatype datatype, + int root, MPIR_Comm * comm, + MPI_Request * req) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_iallgather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, + MPI_Request * req) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_iallgatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, const int *displs, + MPI_Datatype recvtype, MPIR_Comm * comm, + MPI_Request * req) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_iallreduce(const void *sendbuf, void *recvbuf, + int count, MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, + MPI_Request * req) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_ialltoall(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, + MPI_Request * req) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_ialltoallv(const void *sendbuf, const int *sendcounts, + const int *sdispls, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, + const int *rdispls, MPI_Datatype recvtype, + MPIR_Comm * comm, + MPI_Request * req) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_ialltoallw(const void *sendbuf, const int *sendcounts, + const int *sdispls, + const MPI_Datatype sendtypes[], + void *recvbuf, const int *recvcounts, + const int *rdispls, + const MPI_Datatype recvtypes[], + MPIR_Comm * comm, + MPI_Request * req) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_iexscan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, + MPI_Request * req) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_igather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, int root, + MPIR_Comm * comm, + MPI_Request * req) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_igatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, const int *displs, + MPI_Datatype recvtype, int root, + MPIR_Comm * comm, + MPI_Request * req) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_ireduce_scatter_block(const void *sendbuf, + void *recvbuf, int recvcount, MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm, MPI_Request * - req) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ireduce(const void *sendbuf, void *recvbuf, int count, - MPI_Datatype datatype, MPI_Op op, int root, - MPIR_Comm * comm_ptr, - MPI_Request * - req) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_iscan(const void *sendbuf, void *recvbuf, int count, - MPI_Datatype datatype, MPI_Op op, - MPIR_Comm * comm, - MPI_Request * - req) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_iscatter(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - int recvcount, MPI_Datatype recvtype, - int root, MPIR_Comm * comm, - MPI_Request * - req) MPIDI_SHM_STATIC_INLINE_SUFFIX; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_iscatterv(const void *sendbuf, const int *sendcounts, - const int *displs, MPI_Datatype sendtype, - void *recvbuf, int recvcount, - MPI_Datatype recvtype, int root, - MPIR_Comm * comm_ptr, - MPI_Request * - req) MPIDI_SHM_STATIC_INLINE_SUFFIX; + req) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_ireduce_scatter(const void *sendbuf, void *recvbuf, + const int *recvcounts, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, + MPI_Request * req) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_ireduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, int root, + MPIR_Comm * comm_ptr, + MPI_Request * req) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_iscan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, + MPI_Request * req) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_iscatter(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + int root, MPIR_Comm * comm, + MPI_Request * req) MPL_STATIC_INLINE_SUFFIX; +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_iscatterv(const void *sendbuf, const int *sendcounts, + const int *displs, MPI_Datatype sendtype, + void *recvbuf, int recvcount, + MPI_Datatype recvtype, int root, + MPIR_Comm * comm_ptr, + MPI_Request * req) MPL_STATIC_INLINE_SUFFIX; #endif /* SHM_PROTOTYPES_H_INCLUDED */ diff --git a/src/mpid/ch4/shm/include/shm_impl.h b/src/mpid/ch4/shm/include/shm_impl.h index a6d0baa..34d6038 100644 --- a/src/mpid/ch4/shm/include/shm_impl.h +++ b/src/mpid/ch4/shm/include/shm_impl.h @@ -15,564 +15,548 @@ #ifndef SHM_DIRECT #ifndef SHM_DISABLE_INLINES -#ifndef MPIDI_SHM_STATIC_INLINE_PREFIX -#define MPIDI_SHM_STATIC_INLINE_PREFIX __attribute__((always_inline)) static inline -#endif - -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_init(int rank, int size) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_init(int rank, int size) { return MPIDI_SHM_func->init(rank, size); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_finalize(void) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_finalize(void) { return MPIDI_SHM_func->finalize(); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_progress(int blocking) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_progress(int blocking) { return MPIDI_SHM_func->progress(blocking); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_reg_hdr_handler(int handler_id, - MPIDI_SHM_am_origin_handler_fn - origin_handler_fn, - MPIDI_SHM_am_target_handler_fn - target_handler_fn) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_reg_hdr_handler(int handler_id, + MPIDI_SHM_am_origin_handler_fn + origin_handler_fn, + MPIDI_SHM_am_target_handler_fn + target_handler_fn) { return MPIDI_SHM_func->reg_hdr_handler(handler_id, origin_handler_fn, target_handler_fn); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_connect(const char *port_name, MPIR_Info * info, - int root, MPIR_Comm * comm, - MPIR_Comm ** newcomm_ptr) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_connect(const char *port_name, MPIR_Info * info, + int root, MPIR_Comm * comm, + MPIR_Comm ** newcomm_ptr) { return MPIDI_SHM_func->comm_connect(port_name, info, root, comm, newcomm_ptr); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_disconnect(MPIR_Comm * comm_ptr) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_disconnect(MPIR_Comm * comm_ptr) { return MPIDI_SHM_func->comm_disconnect(comm_ptr); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_open_port(MPIR_Info * info_ptr, char *port_name) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_open_port(MPIR_Info * info_ptr, char *port_name) { return MPIDI_SHM_func->open_port(info_ptr, port_name); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_close_port(const char *port_name) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_close_port(const char *port_name) { return MPIDI_SHM_func->close_port(port_name); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_accept(const char *port_name, MPIR_Info * info, - int root, MPIR_Comm * comm, - MPIR_Comm ** newcomm_ptr) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_accept(const char *port_name, MPIR_Info * info, + int root, MPIR_Comm * comm, + MPIR_Comm ** newcomm_ptr) { return MPIDI_SHM_func->comm_accept(port_name, info, root, comm, newcomm_ptr); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_send_am_hdr(int rank, MPIR_Comm * comm, int handler_id, - const void *am_hdr, size_t am_hdr_sz, - MPIR_Request * sreq, void *shm_context) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_send_am_hdr(int rank, MPIR_Comm * comm, int handler_id, + const void *am_hdr, size_t am_hdr_sz, + MPIR_Request * sreq, void *shm_context) { return MPIDI_SHM_func->send_am_hdr(rank, comm, handler_id, am_hdr, am_hdr_sz, sreq, shm_context); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_am_hdr(int rank, MPIR_Comm * comm, - int handler_id, const void *am_hdr, - size_t am_hdr_sz, void *shm_context) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_am_hdr(int rank, MPIR_Comm * comm, + int handler_id, const void *am_hdr, + size_t am_hdr_sz, void *shm_context) { return MPIDI_SHM_func->inject_am_hdr(rank, comm, handler_id, am_hdr, am_hdr_sz, shm_context); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_send_am(int rank, MPIR_Comm * comm, int handler_id, - const void *am_hdr, size_t am_hdr_sz, - const void *data, MPI_Count count, - MPI_Datatype datatype, MPIR_Request * sreq, - void *shm_context) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_send_am(int rank, MPIR_Comm * comm, int handler_id, + const void *am_hdr, size_t am_hdr_sz, + const void *data, MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq, + void *shm_context) { return MPIDI_SHM_func->send_am(rank, comm, handler_id, am_hdr, am_hdr_sz, data, count, datatype, sreq, shm_context); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_am(int rank, MPIR_Comm * comm, int handler_id, - const void *am_hdr, size_t am_hdr_sz, - const void *data, MPI_Count count, - MPI_Datatype datatype, void *shm_context) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_am(int rank, MPIR_Comm * comm, int handler_id, + const void *am_hdr, size_t am_hdr_sz, + const void *data, MPI_Count count, + MPI_Datatype datatype, void *shm_context) { return MPIDI_SHM_func->inject_am(rank, comm, handler_id, am_hdr, am_hdr_sz, data, count, datatype, shm_context); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_send_amv(int rank, MPIR_Comm * comm, int handler_id, - struct iovec *am_hdrs, size_t iov_len, - const void *data, MPI_Count count, - MPI_Datatype datatype, MPIR_Request * sreq, - void *shm_context) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_send_amv(int rank, MPIR_Comm * comm, int handler_id, + struct iovec *am_hdrs, size_t iov_len, + const void *data, MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq, + void *shm_context) { return MPIDI_SHM_func->send_amv(rank, comm, handler_id, am_hdrs, iov_len, data, count, datatype, sreq, shm_context); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_amv(int rank, MPIR_Comm * comm, int handler_id, - struct iovec *am_hdrs, size_t iov_len, - const void *data, MPI_Count count, - MPI_Datatype datatype, void *shm_context) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_amv(int rank, MPIR_Comm * comm, int handler_id, + struct iovec *am_hdrs, size_t iov_len, + const void *data, MPI_Count count, + MPI_Datatype datatype, void *shm_context) { return MPIDI_SHM_func->inject_amv(rank, comm, handler_id, am_hdrs, iov_len, data, count, datatype, shm_context); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_send_am_hdr_reply(MPIR_Context_id_t context_id, - int src_rank, int handler_id, - const void *am_hdr, size_t am_hdr_sz, - MPIR_Request * sreq) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_send_am_hdr_reply(MPIR_Context_id_t context_id, + int src_rank, int handler_id, + const void *am_hdr, size_t am_hdr_sz, + MPIR_Request * sreq) { return MPIDI_SHM_func->send_am_hdr_reply(context_id, src_rank, handler_id, am_hdr, am_hdr_sz, sreq); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_am_hdr_reply(MPIR_Context_id_t context_id, - int src_rank, int handler_id, - const void *am_hdr, - size_t am_hdr_sz) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_am_hdr_reply(MPIR_Context_id_t context_id, + int src_rank, int handler_id, + const void *am_hdr, size_t am_hdr_sz) { return MPIDI_SHM_func->inject_am_hdr_reply(context_id, src_rank, handler_id, am_hdr, am_hdr_sz); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_send_am_reply(MPIR_Context_id_t context_id, - int src_rank, int handler_id, - const void *am_hdr, size_t am_hdr_sz, - const void *data, MPI_Count count, - MPI_Datatype datatype, - MPIR_Request * sreq) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_send_am_reply(MPIR_Context_id_t context_id, + int src_rank, int handler_id, + const void *am_hdr, size_t am_hdr_sz, + const void *data, MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq) { return MPIDI_SHM_func->send_am_reply(context_id, src_rank, handler_id, am_hdr, am_hdr_sz, data, count, datatype, sreq); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_am_reply(MPIR_Context_id_t context_id, - int src_rank, int handler_id, - const void *am_hdr, size_t am_hdr_sz, - const void *data, MPI_Count count, - MPI_Datatype datatype) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_am_reply(MPIR_Context_id_t context_id, + int src_rank, int handler_id, + const void *am_hdr, size_t am_hdr_sz, + const void *data, MPI_Count count, + MPI_Datatype datatype) { return MPIDI_SHM_func->inject_am_reply(context_id, src_rank, handler_id, am_hdr, am_hdr_sz, data, count, datatype); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_send_amv_reply(MPIR_Context_id_t context_id, - int src_rank, int handler_id, - struct iovec *am_hdr, size_t iov_len, - const void *data, MPI_Count count, - MPI_Datatype datatype, - MPIR_Request * sreq) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_send_amv_reply(MPIR_Context_id_t context_id, + int src_rank, int handler_id, + struct iovec *am_hdr, size_t iov_len, + const void *data, MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq) { return MPIDI_SHM_func->send_amv_reply(context_id, src_rank, handler_id, am_hdr, iov_len, data, count, datatype, sreq); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_amv_reply(MPIR_Context_id_t context_id, - int src_rank, int handler_id, - struct iovec *am_hdrs, size_t iov_len, - const void *data, MPI_Count count, - MPI_Datatype datatype) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_amv_reply(MPIR_Context_id_t context_id, + int src_rank, int handler_id, + struct iovec *am_hdrs, size_t iov_len, + const void *data, MPI_Count count, + MPI_Datatype datatype) { return MPIDI_SHM_func->inject_amv_reply(context_id, src_rank, handler_id, am_hdrs, iov_len, data, count, datatype); }; -MPIDI_SHM_STATIC_INLINE_PREFIX size_t MPIDI_SHM_am_hdr_max_sz(void) +MPL_STATIC_INLINE_PREFIX size_t MPIDI_SHM_am_hdr_max_sz(void) { return MPIDI_SHM_func->am_hdr_max_sz(); }; -MPIDI_SHM_STATIC_INLINE_PREFIX size_t MPIDI_SHM_am_inject_max_sz(void) +MPL_STATIC_INLINE_PREFIX size_t MPIDI_SHM_am_inject_max_sz(void) { return MPIDI_SHM_func->am_inject_max_sz(); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_am_recv(MPIR_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_am_recv(MPIR_Request * req) { return MPIDI_SHM_func->am_recv(); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_get_lpid(MPIR_Comm * comm_ptr, int idx, - int *lpid_ptr, MPL_bool is_remote) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_get_lpid(MPIR_Comm * comm_ptr, int idx, + int *lpid_ptr, MPL_bool is_remote) { return MPIDI_SHM_func->comm_get_lpid(comm_ptr, idx, lpid_ptr, is_remote); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_gpid_get(MPIR_Comm * comm_ptr, int rank, - MPIR_Gpid * gpid) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_gpid_get(MPIR_Comm * comm_ptr, int rank, MPIR_Gpid * gpid) { return MPIDI_SHM_func->gpid_get(comm_ptr, rank, gpid); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_get_node_id(MPIR_Comm * comm, int rank, - MPID_Node_id_t * id_p) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_get_node_id(MPIR_Comm * comm, int rank, + MPID_Node_id_t * id_p) { return MPIDI_SHM_func->get_node_id(comm, rank, id_p); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_get_max_node_id(MPIR_Comm * comm, - MPID_Node_id_t * max_id_p) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_get_max_node_id(MPIR_Comm * comm, MPID_Node_id_t * max_id_p) { return MPIDI_SHM_func->get_max_node_id(comm, max_id_p); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_getallincomm(MPIR_Comm * comm_ptr, int local_size, - MPIR_Gpid local_gpid[], int *singleAVT) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_getallincomm(MPIR_Comm * comm_ptr, int local_size, + MPIR_Gpid local_gpid[], int *singleAVT) { return MPIDI_SHM_func->getallincomm(comm_ptr, local_size, local_gpid, singleAVT); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_gpid_tolpidarray(int size, MPIR_Gpid gpid[], - int lpid[]) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_gpid_tolpidarray(int size, MPIR_Gpid gpid[], int lpid[]) { return MPIDI_SHM_func->gpid_tolpidarray(size, gpid, lpid); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_create_intercomm_from_lpids(MPIR_Comm * newcomm_ptr, - int size, - const int lpids[]) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_create_intercomm_from_lpids(MPIR_Comm * newcomm_ptr, + int size, const int lpids[]) { return MPIDI_SHM_func->create_intercomm_from_lpids(newcomm_ptr, size, lpids); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_create(MPIR_Comm * comm) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_create(MPIR_Comm * comm) { return MPIDI_SHM_func->comm_create(comm); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_destroy(MPIR_Comm * comm) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_destroy(MPIR_Comm * comm) { return MPIDI_SHM_func->comm_destroy(comm); }; -MPIDI_SHM_STATIC_INLINE_PREFIX void MPIDI_SHM_am_request_init(MPIR_Request * req) +MPL_STATIC_INLINE_PREFIX void MPIDI_SHM_am_request_init(MPIR_Request * req) { return MPIDI_SHM_func->am_request_init(req); }; -MPIDI_SHM_STATIC_INLINE_PREFIX void MPIDI_SHM_am_request_finalize(MPIR_Request * req) +MPL_STATIC_INLINE_PREFIX void MPIDI_SHM_am_request_finalize(MPIR_Request * req) { return MPIDI_SHM_func->am_request_finalize(req); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_send(const void *buf, int count, MPI_Datatype datatype, - int rank, int tag, MPIR_Comm * comm, - int context_offset, MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_send(const void *buf, int count, MPI_Datatype datatype, + int rank, int tag, MPIR_Comm * comm, + int context_offset, MPIR_Request ** request) { return MPIDI_SHM_native_func->send(buf, count, datatype, rank, tag, comm, context_offset, request); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ssend(const void *buf, int count, - MPI_Datatype datatype, int rank, int tag, - MPIR_Comm * comm, int context_offset, - MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_ssend(const void *buf, int count, + MPI_Datatype datatype, int rank, int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) { return MPIDI_SHM_native_func->ssend(buf, count, datatype, rank, tag, comm, context_offset, request); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_startall(int count, MPIR_Request * requests[]) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_startall(int count, MPIR_Request * requests[]) { return MPIDI_SHM_native_func->startall(count, requests); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_send_init(const void *buf, int count, - MPI_Datatype datatype, int rank, int tag, - MPIR_Comm * comm, int context_offset, - MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_send_init(const void *buf, int count, + MPI_Datatype datatype, int rank, int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) { return MPIDI_SHM_native_func->send_init(buf, count, datatype, rank, tag, comm, context_offset, request); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ssend_init(const void *buf, int count, - MPI_Datatype datatype, int rank, int tag, - MPIR_Comm * comm, int context_offset, - MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_ssend_init(const void *buf, int count, + MPI_Datatype datatype, int rank, int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) { return MPIDI_SHM_native_func->ssend_init(buf, count, datatype, rank, tag, comm, context_offset, request); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_rsend_init(const void *buf, int count, - MPI_Datatype datatype, int rank, int tag, - MPIR_Comm * comm, int context_offset, - MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_rsend_init(const void *buf, int count, + MPI_Datatype datatype, int rank, int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) { return MPIDI_SHM_native_func->rsend_init(buf, count, datatype, rank, tag, comm, context_offset, request); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_bsend_init(const void *buf, int count, - MPI_Datatype datatype, int rank, int tag, - MPIR_Comm * comm, int context_offset, - MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_bsend_init(const void *buf, int count, + MPI_Datatype datatype, int rank, int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) { return MPIDI_SHM_native_func->bsend_init(buf, count, datatype, rank, tag, comm, context_offset, request); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_isend(const void *buf, int count, - MPI_Datatype datatype, int rank, int tag, - MPIR_Comm * comm, int context_offset, - MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_isend(const void *buf, int count, + MPI_Datatype datatype, int rank, int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) { return MPIDI_SHM_native_func->isend(buf, count, datatype, rank, tag, comm, context_offset, request); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_issend(const void *buf, int count, - MPI_Datatype datatype, int rank, int tag, - MPIR_Comm * comm, int context_offset, - MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_issend(const void *buf, int count, + MPI_Datatype datatype, int rank, int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) { return MPIDI_SHM_native_func->issend(buf, count, datatype, rank, tag, comm, context_offset, request); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_cancel_send(MPIR_Request * sreq) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_cancel_send(MPIR_Request * sreq) { return MPIDI_SHM_native_func->cancel_send(sreq); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_recv_init(void *buf, int count, MPI_Datatype datatype, - int rank, int tag, MPIR_Comm * comm, - int context_offset, MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_recv_init(void *buf, int count, MPI_Datatype datatype, + int rank, int tag, MPIR_Comm * comm, + int context_offset, MPIR_Request ** request) { return MPIDI_SHM_native_func->recv_init(buf, count, datatype, rank, tag, comm, context_offset, request); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_recv(void *buf, int count, MPI_Datatype datatype, - int rank, int tag, MPIR_Comm * comm, - int context_offset, MPI_Status * status, - MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_recv(void *buf, int count, MPI_Datatype datatype, + int rank, int tag, MPIR_Comm * comm, + int context_offset, MPI_Status * status, + MPIR_Request ** request) { return MPIDI_SHM_native_func->recv(buf, count, datatype, rank, tag, comm, context_offset, status, request); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_irecv(void *buf, int count, MPI_Datatype datatype, - int rank, int tag, MPIR_Comm * comm, - int context_offset, MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_irecv(void *buf, int count, MPI_Datatype datatype, + int rank, int tag, MPIR_Comm * comm, + int context_offset, MPIR_Request ** request) { return MPIDI_SHM_native_func->irecv(buf, count, datatype, rank, tag, comm, context_offset, request); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_imrecv(void *buf, int count, MPI_Datatype datatype, - MPIR_Request * message, MPIR_Request ** rreqp) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_imrecv(void *buf, int count, MPI_Datatype datatype, + MPIR_Request * message, MPIR_Request ** rreqp) { return MPIDI_SHM_native_func->imrecv(buf, count, datatype, message, rreqp); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_cancel_recv(MPIR_Request * rreq) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_cancel_recv(MPIR_Request * rreq) { return MPIDI_SHM_native_func->cancel_recv(rreq); }; -MPIDI_SHM_STATIC_INLINE_PREFIX void *MPIDI_SHM_alloc_mem(size_t size, MPIR_Info * info_ptr) +MPL_STATIC_INLINE_PREFIX void *MPIDI_SHM_alloc_mem(size_t size, MPIR_Info * info_ptr) { return MPIDI_SHM_native_func->alloc_mem(size, info_ptr); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_free_mem(void *ptr) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_free_mem(void *ptr) { return MPIDI_SHM_native_func->free_mem(ptr); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_improbe(int source, int tag, MPIR_Comm * comm, - int context_offset, int *flag, - MPIR_Request ** message, MPI_Status * status) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_improbe(int source, int tag, MPIR_Comm * comm, + int context_offset, int *flag, + MPIR_Request ** message, MPI_Status * status) { return MPIDI_SHM_native_func->improbe(source, tag, comm, context_offset, flag, message, status); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_iprobe(int source, int tag, MPIR_Comm * comm, - int context_offset, int *flag, - MPI_Status * status) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_iprobe(int source, int tag, MPIR_Comm * comm, + int context_offset, int *flag, MPI_Status * status) { return MPIDI_SHM_native_func->iprobe(source, tag, comm, context_offset, flag, status); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_set_info(MPIR_Win * win, MPIR_Info * info) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_win_set_info(MPIR_Win * win, MPIR_Info * info) { return MPIDI_SHM_native_func->win_set_info(win, info); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_shared_query(MPIR_Win * win, int rank, - MPI_Aint * size, int *disp_unit, - void *baseptr) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_win_shared_query(MPIR_Win * win, int rank, + MPI_Aint * size, int *disp_unit, + void *baseptr) { return MPIDI_SHM_native_func->win_shared_query(win, rank, size, disp_unit, baseptr); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_put(const void *origin_addr, int origin_count, - MPI_Datatype origin_datatype, int target_rank, - MPI_Aint target_disp, int target_count, - MPI_Datatype target_datatype, MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_put(const void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, int target_rank, + MPI_Aint target_disp, int target_count, + MPI_Datatype target_datatype, MPIR_Win * win) { return MPIDI_SHM_native_func->put(origin_addr, origin_count, origin_datatype, target_rank, target_disp, target_count, target_datatype, win); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_start(MPIR_Group * group, int assert, - MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_win_start(MPIR_Group * group, int assert, MPIR_Win * win) { return MPIDI_SHM_native_func->win_start(group, assert, win); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_complete(MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_win_complete(MPIR_Win * win) { return MPIDI_SHM_native_func->win_complete(win); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_post(MPIR_Group * group, int assert, - MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_win_post(MPIR_Group * group, int assert, MPIR_Win * win) { return MPIDI_SHM_native_func->win_post(group, assert, win); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_wait(MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_win_wait(MPIR_Win * win) { return MPIDI_SHM_native_func->win_wait(win); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_test(MPIR_Win * win, int *flag) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_win_test(MPIR_Win * win, int *flag) { return MPIDI_SHM_native_func->win_test(win, flag); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_lock(int lock_type, int rank, int assert, - MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_win_lock(int lock_type, int rank, int assert, MPIR_Win * win) { return MPIDI_SHM_native_func->win_lock(lock_type, rank, assert, win); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_unlock(int rank, MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_win_unlock(int rank, MPIR_Win * win) { return MPIDI_SHM_native_func->win_unlock(rank, win); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_get_info(MPIR_Win * win, MPIR_Info ** info_p_p) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_win_get_info(MPIR_Win * win, MPIR_Info ** info_p_p) { return MPIDI_SHM_native_func->win_get_info(win, info_p_p); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_get(void *origin_addr, int origin_count, - MPI_Datatype origin_datatype, int target_rank, - MPI_Aint target_disp, int target_count, - MPI_Datatype target_datatype, MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_get(void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, int target_rank, + MPI_Aint target_disp, int target_count, + MPI_Datatype target_datatype, MPIR_Win * win) { return MPIDI_SHM_native_func->get(origin_addr, origin_count, origin_datatype, target_rank, target_disp, target_count, target_datatype, win); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_free(MPIR_Win ** win_ptr) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_win_free(MPIR_Win ** win_ptr) { return MPIDI_SHM_native_func->win_free(win_ptr); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_fence(int assert, MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_win_fence(int assert, MPIR_Win * win) { return MPIDI_SHM_native_func->win_fence(assert, win); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_create(void *base, MPI_Aint length, int disp_unit, - MPIR_Info * info, MPIR_Comm * comm_ptr, - MPIR_Win ** win_ptr) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_win_create(void *base, MPI_Aint length, int disp_unit, + MPIR_Info * info, MPIR_Comm * comm_ptr, + MPIR_Win ** win_ptr) { return MPIDI_SHM_native_func->win_create(base, length, disp_unit, info, comm_ptr, win_ptr); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_accumulate(const void *origin_addr, int origin_count, - MPI_Datatype origin_datatype, - int target_rank, MPI_Aint target_disp, - int target_count, - MPI_Datatype target_datatype, MPI_Op op, - MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_accumulate(const void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, + int target_rank, MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, MPI_Op op, + MPIR_Win * win) { return MPIDI_SHM_native_func->accumulate(origin_addr, origin_count, origin_datatype, target_rank, target_disp, target_count, target_datatype, op, win); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_attach(MPIR_Win * win, void *base, MPI_Aint size) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_win_attach(MPIR_Win * win, void *base, MPI_Aint size) { return MPIDI_SHM_native_func->win_attach(win, base, size); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_allocate_shared(MPI_Aint size, int disp_unit, - MPIR_Info * info_ptr, - MPIR_Comm * comm_ptr, - void **base_ptr, - MPIR_Win ** win_ptr) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_win_allocate_shared(MPI_Aint size, int disp_unit, + MPIR_Info * info_ptr, + MPIR_Comm * comm_ptr, + void **base_ptr, MPIR_Win ** win_ptr) { return MPIDI_SHM_native_func->win_allocate_shared(size, disp_unit, info_ptr, comm_ptr, base_ptr, win_ptr); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_rput(const void *origin_addr, int origin_count, - MPI_Datatype origin_datatype, int target_rank, - MPI_Aint target_disp, int target_count, - MPI_Datatype target_datatype, MPIR_Win * win, - MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_rput(const void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, int target_rank, + MPI_Aint target_disp, int target_count, + MPI_Datatype target_datatype, MPIR_Win * win, + MPIR_Request ** request) { return MPIDI_SHM_native_func->rput(origin_addr, origin_count, origin_datatype, target_rank, target_disp, target_count, target_datatype, win, request); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_flush_local(int rank, MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_win_flush_local(int rank, MPIR_Win * win) { return MPIDI_SHM_native_func->win_flush_local(rank, win); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_detach(MPIR_Win * win, const void *base) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_win_detach(MPIR_Win * win, const void *base) { return MPIDI_SHM_native_func->win_detach(win, base); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_compare_and_swap(const void *origin_addr, - const void *compare_addr, - void *result_addr, - MPI_Datatype datatype, - int target_rank, MPI_Aint target_disp, - MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_compare_and_swap(const void *origin_addr, + const void *compare_addr, + void *result_addr, + MPI_Datatype datatype, + int target_rank, MPI_Aint target_disp, + MPIR_Win * win) { return MPIDI_SHM_native_func->compare_and_swap(origin_addr, compare_addr, result_addr, datatype, target_rank, target_disp, win); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_raccumulate(const void *origin_addr, int origin_count, - MPI_Datatype origin_datatype, - int target_rank, MPI_Aint target_disp, - int target_count, - MPI_Datatype target_datatype, MPI_Op op, - MPIR_Win * win, MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_raccumulate(const void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, + int target_rank, MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, MPI_Op op, + MPIR_Win * win, MPIR_Request ** request) { return MPIDI_SHM_native_func->raccumulate(origin_addr, origin_count, origin_datatype, target_rank, target_disp, target_count, target_datatype, op, win, request); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_rget_accumulate(const void *origin_addr, - int origin_count, - MPI_Datatype origin_datatype, - void *result_addr, int result_count, - MPI_Datatype result_datatype, - int target_rank, MPI_Aint target_disp, - int target_count, - MPI_Datatype target_datatype, - MPI_Op op, MPIR_Win * win, - MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_rget_accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + void *result_addr, int result_count, + MPI_Datatype result_datatype, + int target_rank, MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPI_Op op, MPIR_Win * win, + MPIR_Request ** request) { return MPIDI_SHM_native_func->rget_accumulate(origin_addr, origin_count, origin_datatype, result_addr, result_count, result_datatype, @@ -580,72 +564,72 @@ MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_rget_accumulate(const void *origin_ target_datatype, op, win, request); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_fetch_and_op(const void *origin_addr, - void *result_addr, MPI_Datatype datatype, - int target_rank, MPI_Aint target_disp, - MPI_Op op, MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_fetch_and_op(const void *origin_addr, + void *result_addr, MPI_Datatype datatype, + int target_rank, MPI_Aint target_disp, + MPI_Op op, MPIR_Win * win) { return MPIDI_SHM_native_func->fetch_and_op(origin_addr, result_addr, datatype, target_rank, target_disp, op, win); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_allocate(MPI_Aint size, int disp_unit, - MPIR_Info * info, MPIR_Comm * comm, - void *baseptr, MPIR_Win ** win) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_win_allocate(MPI_Aint size, int disp_unit, + MPIR_Info * info, MPIR_Comm * comm, + void *baseptr, MPIR_Win ** win) { return MPIDI_SHM_native_func->win_allocate(size, disp_unit, info, comm, baseptr, win); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_flush(int rank, MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_win_flush(int rank, MPIR_Win * win) { return MPIDI_SHM_native_func->win_flush(rank, win); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_flush_local_all(MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_win_flush_local_all(MPIR_Win * win) { return MPIDI_SHM_native_func->win_flush_local_all(win); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_unlock_all(MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_win_unlock_all(MPIR_Win * win) { return MPIDI_SHM_native_func->win_unlock_all(win); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_create_dynamic(MPIR_Info * info, MPIR_Comm * comm, - MPIR_Win ** win) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_win_create_dynamic(MPIR_Info * info, MPIR_Comm * comm, + MPIR_Win ** win) { return MPIDI_SHM_native_func->win_create_dynamic(info, comm, win); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_rget(void *origin_addr, int origin_count, - MPI_Datatype origin_datatype, int target_rank, - MPI_Aint target_disp, int target_count, - MPI_Datatype target_datatype, MPIR_Win * win, - MPIR_Request ** request) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_rget(void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, int target_rank, + MPI_Aint target_disp, int target_count, + MPI_Datatype target_datatype, MPIR_Win * win, + MPIR_Request ** request) { return MPIDI_SHM_native_func->rget(origin_addr, origin_count, origin_datatype, target_rank, target_disp, target_count, target_datatype, win, request); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_sync(MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_win_sync(MPIR_Win * win) { return MPIDI_SHM_native_func->win_sync(win); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_flush_all(MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_win_flush_all(MPIR_Win * win) { return MPIDI_SHM_native_func->win_flush_all(win); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_get_accumulate(const void *origin_addr, - int origin_count, - MPI_Datatype origin_datatype, - void *result_addr, int result_count, - MPI_Datatype result_datatype, - int target_rank, MPI_Aint target_disp, - int target_count, - MPI_Datatype target_datatype, MPI_Op op, - MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_get_accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + void *result_addr, int result_count, + MPI_Datatype result_datatype, + int target_rank, MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, MPI_Op op, + MPIR_Win * win) { return MPIDI_SHM_native_func->get_accumulate(origin_addr, origin_count, origin_datatype, result_addr, result_count, result_datatype, @@ -653,436 +637,428 @@ MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_get_accumulate(const void *origin_a target_datatype, op, win); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_lock_all(int assert, MPIR_Win * win) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_win_lock_all(int assert, MPIR_Win * win) { return MPIDI_SHM_native_func->win_lock_all(assert, win); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_barrier(MPIR_Comm * comm, MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_barrier(MPIR_Comm * comm, MPIR_Errflag_t * errflag) { return MPIDI_SHM_native_func->barrier(comm, errflag); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_bcast(void *buffer, int count, MPI_Datatype datatype, - int root, MPIR_Comm * comm, - MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_bcast(void *buffer, int count, MPI_Datatype datatype, + int root, MPIR_Comm * comm, MPIR_Errflag_t * errflag) { return MPIDI_SHM_native_func->bcast(buffer, count, datatype, root, comm, errflag); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_allreduce(const void *sendbuf, void *recvbuf, - int count, MPI_Datatype datatype, MPI_Op op, - MPIR_Comm * comm, MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_allreduce(const void *sendbuf, void *recvbuf, + int count, MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, MPIR_Errflag_t * errflag) { return MPIDI_SHM_native_func->allreduce(sendbuf, recvbuf, count, datatype, op, comm, errflag); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_allgather(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - int recvcount, MPI_Datatype recvtype, - MPIR_Comm * comm, MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_allgather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, MPIR_Errflag_t * errflag) { return MPIDI_SHM_native_func->allgather(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, comm, errflag); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_allgatherv(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - const int *recvcounts, const int *displs, - MPI_Datatype recvtype, MPIR_Comm * comm, - MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_allgatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, const int *displs, + MPI_Datatype recvtype, MPIR_Comm * comm, + MPIR_Errflag_t * errflag) { return MPIDI_SHM_native_func->allgatherv(sendbuf, sendcount, sendtype, recvbuf, recvcounts, displs, recvtype, comm, errflag); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_scatter(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - int recvcount, MPI_Datatype recvtype, int root, - MPIR_Comm * comm, MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_scatter(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, int root, + MPIR_Comm * comm, MPIR_Errflag_t * errflag) { return MPIDI_SHM_native_func->scatter(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, root, comm, errflag); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_scatterv(const void *sendbuf, const int *sendcounts, - const int *displs, MPI_Datatype sendtype, - void *recvbuf, int recvcount, - MPI_Datatype recvtype, int root, - MPIR_Comm * comm_ptr, - MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_scatterv(const void *sendbuf, const int *sendcounts, + const int *displs, MPI_Datatype sendtype, + void *recvbuf, int recvcount, + MPI_Datatype recvtype, int root, + MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) { return MPIDI_SHM_native_func->scatterv(sendbuf, sendcounts, displs, sendtype, recvbuf, recvcount, recvtype, root, comm_ptr, errflag); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_gather(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - int recvcount, MPI_Datatype recvtype, int root, - MPIR_Comm * comm, MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_gather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, int root, + MPIR_Comm * comm, MPIR_Errflag_t * errflag) { return MPIDI_SHM_native_func->gather(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, root, comm, errflag); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_gatherv(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - const int *recvcounts, const int *displs, - MPI_Datatype recvtype, int root, - MPIR_Comm * comm, MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_gatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, const int *displs, + MPI_Datatype recvtype, int root, + MPIR_Comm * comm, MPIR_Errflag_t * errflag) { return MPIDI_SHM_native_func->gatherv(sendbuf, sendcount, sendtype, recvbuf, recvcounts, displs, recvtype, root, comm, errflag); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_alltoall(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - int recvcount, MPI_Datatype recvtype, - MPIR_Comm * comm, MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_alltoall(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, MPIR_Errflag_t * errflag) { return MPIDI_SHM_native_func->alltoall(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, comm, errflag); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_alltoallv(const void *sendbuf, const int *sendcounts, - const int *sdispls, MPI_Datatype sendtype, - void *recvbuf, const int *recvcounts, - const int *rdispls, MPI_Datatype recvtype, - MPIR_Comm * comm, MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_alltoallv(const void *sendbuf, const int *sendcounts, + const int *sdispls, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, + const int *rdispls, MPI_Datatype recvtype, + MPIR_Comm * comm, MPIR_Errflag_t * errflag) { return MPIDI_SHM_native_func->alltoallv(sendbuf, sendcounts, sdispls, sendtype, recvbuf, recvcounts, rdispls, recvtype, comm, errflag); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_alltoallw(const void *sendbuf, const int *sendcounts, - const int *sdispls, - const MPI_Datatype sendtypes[], - void *recvbuf, const int *recvcounts, - const int *rdispls, - const MPI_Datatype recvtypes[], - MPIR_Comm * comm, MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_alltoallw(const void *sendbuf, const int *sendcounts, + const int *sdispls, + const MPI_Datatype sendtypes[], + void *recvbuf, const int *recvcounts, + const int *rdispls, + const MPI_Datatype recvtypes[], + MPIR_Comm * comm, MPIR_Errflag_t * errflag) { return MPIDI_SHM_native_func->alltoallw(sendbuf, sendcounts, sdispls, sendtypes, recvbuf, recvcounts, rdispls, recvtypes, comm, errflag); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_reduce(const void *sendbuf, void *recvbuf, int count, - MPI_Datatype datatype, MPI_Op op, int root, - MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_reduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, int root, + MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) { return MPIDI_SHM_native_func->reduce(sendbuf, recvbuf, count, datatype, op, root, comm_ptr, errflag); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_reduce_scatter(const void *sendbuf, void *recvbuf, - const int *recvcounts, - MPI_Datatype datatype, MPI_Op op, - MPIR_Comm * comm_ptr, - MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_reduce_scatter(const void *sendbuf, void *recvbuf, + const int *recvcounts, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) { return MPIDI_SHM_native_func->reduce_scatter(sendbuf, recvbuf, recvcounts, datatype, op, comm_ptr, errflag); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_reduce_scatter_block(const void *sendbuf, - void *recvbuf, int recvcount, - MPI_Datatype datatype, MPI_Op op, - MPIR_Comm * comm_ptr, - MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_reduce_scatter_block(const void *sendbuf, + void *recvbuf, int recvcount, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) { return MPIDI_SHM_native_func->reduce_scatter_block(sendbuf, recvbuf, recvcount, datatype, op, comm_ptr, errflag); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_scan(const void *sendbuf, void *recvbuf, int count, - MPI_Datatype datatype, MPI_Op op, - MPIR_Comm * comm, MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_scan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, MPIR_Errflag_t * errflag) { return MPIDI_SHM_native_func->scan(sendbuf, recvbuf, count, datatype, op, comm, errflag); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_exscan(const void *sendbuf, void *recvbuf, int count, - MPI_Datatype datatype, MPI_Op op, - MPIR_Comm * comm, MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_exscan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, MPIR_Errflag_t * errflag) { return MPIDI_SHM_native_func->exscan(sendbuf, recvbuf, count, datatype, op, comm, errflag); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_neighbor_allgather(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, - void *recvbuf, int recvcount, - MPI_Datatype recvtype, - MPIR_Comm * comm, - MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_neighbor_allgather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, + void *recvbuf, int recvcount, + MPI_Datatype recvtype, + MPIR_Comm * comm, + MPIR_Errflag_t * errflag) { return MPIDI_SHM_native_func->neighbor_allgather(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, comm, errflag); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_neighbor_allgatherv(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, - void *recvbuf, - const int *recvcounts, - const int *displs, - MPI_Datatype recvtype, - MPIR_Comm * comm, - MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_neighbor_allgatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, + void *recvbuf, + const int *recvcounts, + const int *displs, + MPI_Datatype recvtype, + MPIR_Comm * comm, + MPIR_Errflag_t * errflag) { return MPIDI_SHM_native_func->neighbor_allgatherv(sendbuf, sendcount, sendtype, recvbuf, recvcounts, displs, recvtype, comm, errflag); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_neighbor_alltoallv(const void *sendbuf, - const int *sendcounts, - const int *sdispls, - MPI_Datatype sendtype, - void *recvbuf, - const int *recvcounts, - const int *rdispls, - MPI_Datatype recvtype, - MPIR_Comm * comm, - MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_neighbor_alltoallv(const void *sendbuf, + const int *sendcounts, + const int *sdispls, + MPI_Datatype sendtype, + void *recvbuf, + const int *recvcounts, + const int *rdispls, + MPI_Datatype recvtype, + MPIR_Comm * comm, + MPIR_Errflag_t * errflag) { return MPIDI_SHM_native_func->neighbor_alltoallv(sendbuf, sendcounts, sdispls, sendtype, recvbuf, recvcounts, rdispls, recvtype, comm, errflag); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_neighbor_alltoallw(const void *sendbuf, - const int *sendcounts, - const MPI_Aint * sdispls, - const MPI_Datatype * sendtypes, - void *recvbuf, - const int *recvcounts, - const MPI_Aint * rdispls, - const MPI_Datatype * recvtypes, - MPIR_Comm * comm, - MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_neighbor_alltoallw(const void *sendbuf, + const int *sendcounts, + const MPI_Aint * sdispls, + const MPI_Datatype * sendtypes, + void *recvbuf, + const int *recvcounts, + const MPI_Aint * rdispls, + const MPI_Datatype * recvtypes, + MPIR_Comm * comm, + MPIR_Errflag_t * errflag) { return MPIDI_SHM_native_func->neighbor_alltoallw(sendbuf, sendcounts, sdispls, sendtypes, recvbuf, recvcounts, rdispls, recvtypes, comm, errflag); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_neighbor_alltoall(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - int recvcount, MPI_Datatype recvtype, - MPIR_Comm * comm, - MPIR_Errflag_t * errflag) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_neighbor_alltoall(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, MPIR_Errflag_t * errflag) { return MPIDI_SHM_native_func->neighbor_alltoall(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, comm, errflag); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ineighbor_allgather(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, - void *recvbuf, int recvcount, - MPI_Datatype recvtype, - MPIR_Comm * comm, - MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_ineighbor_allgather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, + void *recvbuf, int recvcount, + MPI_Datatype recvtype, + MPIR_Comm * comm, MPI_Request * req) { return MPIDI_SHM_native_func->ineighbor_allgather(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, comm, req); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ineighbor_allgatherv(const void *sendbuf, - int sendcount, - MPI_Datatype sendtype, - void *recvbuf, - const int *recvcounts, - const int *displs, - MPI_Datatype recvtype, - MPIR_Comm * comm, - MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_ineighbor_allgatherv(const void *sendbuf, + int sendcount, + MPI_Datatype sendtype, + void *recvbuf, + const int *recvcounts, + const int *displs, + MPI_Datatype recvtype, + MPIR_Comm * comm, MPI_Request * req) { return MPIDI_SHM_native_func->ineighbor_allgatherv(sendbuf, sendcount, sendtype, recvbuf, recvcounts, displs, recvtype, comm, req); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ineighbor_alltoall(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, - void *recvbuf, int recvcount, - MPI_Datatype recvtype, - MPIR_Comm * comm, MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_ineighbor_alltoall(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, + void *recvbuf, int recvcount, + MPI_Datatype recvtype, + MPIR_Comm * comm, MPI_Request * req) { return MPIDI_SHM_native_func->ineighbor_alltoall(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, comm, req); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ineighbor_alltoallv(const void *sendbuf, - const int *sendcounts, - const int *sdispls, - MPI_Datatype sendtype, - void *recvbuf, - const int *recvcounts, - const int *rdispls, - MPI_Datatype recvtype, - MPIR_Comm * comm, - MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_ineighbor_alltoallv(const void *sendbuf, + const int *sendcounts, + const int *sdispls, + MPI_Datatype sendtype, + void *recvbuf, + const int *recvcounts, + const int *rdispls, + MPI_Datatype recvtype, + MPIR_Comm * comm, MPI_Request * req) { return MPIDI_SHM_native_func->ineighbor_alltoallv(sendbuf, sendcounts, sdispls, sendtype, recvbuf, recvcounts, rdispls, recvtype, comm, req); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ineighbor_alltoallw(const void *sendbuf, - const int *sendcounts, - const MPI_Aint * sdispls, - const MPI_Datatype * sendtypes, - void *recvbuf, - const int *recvcounts, - const MPI_Aint * rdispls, - const MPI_Datatype * recvtypes, - MPIR_Comm * comm, - MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_ineighbor_alltoallw(const void *sendbuf, + const int *sendcounts, + const MPI_Aint * sdispls, + const MPI_Datatype * sendtypes, + void *recvbuf, + const int *recvcounts, + const MPI_Aint * rdispls, + const MPI_Datatype * recvtypes, + MPIR_Comm * comm, MPI_Request * req) { return MPIDI_SHM_native_func->ineighbor_alltoallw(sendbuf, sendcounts, sdispls, sendtypes, recvbuf, recvcounts, rdispls, recvtypes, comm, req); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ibarrier(MPIR_Comm * comm, MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_ibarrier(MPIR_Comm * comm, MPI_Request * req) { return MPIDI_SHM_native_func->ibarrier(comm, req); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ibcast(void *buffer, int count, MPI_Datatype datatype, - int root, MPIR_Comm * comm, MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_ibcast(void *buffer, int count, MPI_Datatype datatype, + int root, MPIR_Comm * comm, MPI_Request * req) { return MPIDI_SHM_native_func->ibcast(buffer, count, datatype, root, comm, req); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_iallgather(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - int recvcount, MPI_Datatype recvtype, - MPIR_Comm * comm, MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_iallgather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, MPI_Request * req) { return MPIDI_SHM_native_func->iallgather(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, comm, req); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_iallgatherv(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - const int *recvcounts, const int *displs, - MPI_Datatype recvtype, MPIR_Comm * comm, - MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_iallgatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, const int *displs, + MPI_Datatype recvtype, MPIR_Comm * comm, + MPI_Request * req) { return MPIDI_SHM_native_func->iallgatherv(sendbuf, sendcount, sendtype, recvbuf, recvcounts, displs, recvtype, comm, req); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_iallreduce(const void *sendbuf, void *recvbuf, - int count, MPI_Datatype datatype, MPI_Op op, - MPIR_Comm * comm, MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_iallreduce(const void *sendbuf, void *recvbuf, + int count, MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, MPI_Request * req) { return MPIDI_SHM_native_func->iallreduce(sendbuf, recvbuf, count, datatype, op, comm, req); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ialltoall(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - int recvcount, MPI_Datatype recvtype, - MPIR_Comm * comm, MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_ialltoall(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, MPI_Request * req) { return MPIDI_SHM_native_func->ialltoall(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, comm, req); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ialltoallv(const void *sendbuf, const int *sendcounts, - const int *sdispls, MPI_Datatype sendtype, - void *recvbuf, const int *recvcounts, - const int *rdispls, MPI_Datatype recvtype, - MPIR_Comm * comm, MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_ialltoallv(const void *sendbuf, const int *sendcounts, + const int *sdispls, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, + const int *rdispls, MPI_Datatype recvtype, + MPIR_Comm * comm, MPI_Request * req) { return MPIDI_SHM_native_func->ialltoallv(sendbuf, sendcounts, sdispls, sendtype, recvbuf, recvcounts, rdispls, recvtype, comm, req); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ialltoallw(const void *sendbuf, const int *sendcounts, - const int *sdispls, - const MPI_Datatype sendtypes[], - void *recvbuf, const int *recvcounts, - const int *rdispls, - const MPI_Datatype recvtypes[], - MPIR_Comm * comm, MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_ialltoallw(const void *sendbuf, const int *sendcounts, + const int *sdispls, + const MPI_Datatype sendtypes[], + void *recvbuf, const int *recvcounts, + const int *rdispls, + const MPI_Datatype recvtypes[], + MPIR_Comm * comm, MPI_Request * req) { return MPIDI_SHM_native_func->ialltoallw(sendbuf, sendcounts, sdispls, sendtypes, recvbuf, recvcounts, rdispls, recvtypes, comm, req); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_iexscan(const void *sendbuf, void *recvbuf, int count, - MPI_Datatype datatype, MPI_Op op, - MPIR_Comm * comm, MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_iexscan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, MPI_Request * req) { return MPIDI_SHM_native_func->iexscan(sendbuf, recvbuf, count, datatype, op, comm, req); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_igather(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - int recvcount, MPI_Datatype recvtype, int root, - MPIR_Comm * comm, MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_igather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, int root, + MPIR_Comm * comm, MPI_Request * req) { return MPIDI_SHM_native_func->igather(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, root, comm, req); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_igatherv(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - const int *recvcounts, const int *displs, - MPI_Datatype recvtype, int root, - MPIR_Comm * comm, MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_igatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, const int *displs, + MPI_Datatype recvtype, int root, + MPIR_Comm * comm, MPI_Request * req) { return MPIDI_SHM_native_func->igatherv(sendbuf, sendcount, sendtype, recvbuf, recvcounts, displs, recvtype, root, comm, req); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ireduce_scatter_block(const void *sendbuf, - void *recvbuf, int recvcount, - MPI_Datatype datatype, MPI_Op op, - MPIR_Comm * comm, - MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_ireduce_scatter_block(const void *sendbuf, + void *recvbuf, int recvcount, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, MPI_Request * req) { return MPIDI_SHM_native_func->ireduce_scatter_block(sendbuf, recvbuf, recvcount, datatype, op, comm, req); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ireduce_scatter(const void *sendbuf, void *recvbuf, - const int *recvcounts, - MPI_Datatype datatype, MPI_Op op, - MPIR_Comm * comm, MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_ireduce_scatter(const void *sendbuf, void *recvbuf, + const int *recvcounts, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, MPI_Request * req) { return MPIDI_SHM_native_func->ireduce_scatter(sendbuf, recvbuf, recvcounts, datatype, op, comm, req); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ireduce(const void *sendbuf, void *recvbuf, int count, - MPI_Datatype datatype, MPI_Op op, int root, - MPIR_Comm * comm_ptr, MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_ireduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, int root, + MPIR_Comm * comm_ptr, MPI_Request * req) { return MPIDI_SHM_native_func->ireduce(sendbuf, recvbuf, count, datatype, op, root, comm_ptr, req); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_iscan(const void *sendbuf, void *recvbuf, int count, - MPI_Datatype datatype, MPI_Op op, - MPIR_Comm * comm, MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_iscan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, MPI_Request * req) { return MPIDI_SHM_native_func->iscan(sendbuf, recvbuf, count, datatype, op, comm, req); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_iscatter(const void *sendbuf, int sendcount, - MPI_Datatype sendtype, void *recvbuf, - int recvcount, MPI_Datatype recvtype, - int root, MPIR_Comm * comm, MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_iscatter(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + int root, MPIR_Comm * comm, MPI_Request * req) { return MPIDI_SHM_native_func->iscatter(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, root, comm, req); }; -MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_iscatterv(const void *sendbuf, const int *sendcounts, - const int *displs, MPI_Datatype sendtype, - void *recvbuf, int recvcount, - MPI_Datatype recvtype, int root, - MPIR_Comm * comm_ptr, MPI_Request * req) +MPL_STATIC_INLINE_PREFIX int MPIDI_SHM_iscatterv(const void *sendbuf, const int *sendcounts, + const int *displs, MPI_Datatype sendtype, + void *recvbuf, int recvcount, + MPI_Datatype recvtype, int root, + MPIR_Comm * comm_ptr, MPI_Request * req) { return MPIDI_SHM_native_func->iscatterv(sendbuf, sendcounts, displs, sendtype, recvbuf, recvcount, recvtype, root, comm_ptr, req); diff --git a/src/mpl/include/mpl_base.h b/src/mpl/include/mpl_base.h index 7718502..9194a10 100644 --- a/src/mpl/include/mpl_base.h +++ b/src/mpl/include/mpl_base.h @@ -52,6 +52,9 @@ # endif /* MPL_HAVE_GCC_ATTRIBUTE */ #endif /* ATTRIBUTE */ +#define MPL_STATIC_INLINE_PREFIX ATTRIBUTE((always_inline)) static inline +#define MPL_STATIC_INLINE_SUFFIX ATTRIBUTE((always_inline)) + /* These likely/unlikely macros provide static branch prediction hints to the * compiler, if such hints are available. Simply wrap the relevant expression in * the macro, like this: http://git.mpich.org/mpich.git/commitdiff/5d2ded70d81461d9db6661e2fb32c8a1b… commit 5d2ded70d81461d9db6661e2fb32c8a1b4e97db9 Author: Pavan Balaji <balaji(a)anl.gov> Date: Wed Aug 24 09:01:51 2016 -0500 CH4: Delete unused functions MPIDI_NM_am_inject_max_sz, MPIDI_NM_get_max_node_id, MPIDI_NM_get_node_id. Signed-off-by: Ken Raffenetti <raffenet(a)mcs.anl.gov> diff --git a/src/mpid/ch4/netmod/include/netmod.h b/src/mpid/ch4/netmod/include/netmod.h index bd33b0c..6b79d06 100644 --- a/src/mpid/ch4/netmod/include/netmod.h +++ b/src/mpid/ch4/netmod/include/netmod.h @@ -73,13 +73,10 @@ typedef int (*MPIDI_NM_send_amv_reply_t) (MPIR_Context_id_t context_id, int src_ const void *data, MPI_Count count, MPI_Datatype datatype, MPIR_Request * sreq); typedef size_t(*MPIDI_NM_am_hdr_max_sz_t) (void); -typedef size_t(*MPIDI_NM_am_inject_max_sz_t) (void); typedef int (*MPIDI_NM_am_recv_t) (MPIR_Request * req); typedef int (*MPIDI_NM_comm_get_lpid_t) (MPIR_Comm * comm_ptr, int idx, int *lpid_ptr, MPL_bool is_remote); typedef int (*MPIDI_NM_gpid_get_t) (MPIR_Comm * comm_ptr, int rank, MPIR_Gpid * gpid); -typedef int (*MPIDI_NM_get_node_id_t) (MPIR_Comm * comm, int rank, MPID_Node_id_t * id_p); -typedef int (*MPIDI_NM_get_max_node_id_t) (MPIR_Comm * comm, MPID_Node_id_t * max_id_p); typedef int (*MPIDI_NM_getallincomm_t) (MPIR_Comm * comm_ptr, int local_size, MPIR_Gpid local_gpids[], int *singleAVT); typedef int (*MPIDI_NM_gpid_tolpidarray_t) (int size, MPIR_Gpid gpid[], int lpid[]); @@ -372,8 +369,6 @@ typedef struct MPIDI_NM_funcs { /* Routines that handle addressing */ MPIDI_NM_comm_get_lpid_t comm_get_lpid; MPIDI_NM_gpid_get_t gpid_get; - MPIDI_NM_get_node_id_t get_node_id; - MPIDI_NM_get_max_node_id_t get_max_node_id; MPIDI_NM_getallincomm_t getallincomm; MPIDI_NM_gpid_tolpidarray_t gpid_tolpidarray; MPIDI_NM_create_intercomm_from_lpids_t create_intercomm_from_lpids; @@ -394,7 +389,6 @@ typedef struct MPIDI_NM_funcs { MPIDI_NM_send_am_reply_t send_am_reply; MPIDI_NM_send_amv_reply_t send_amv_reply; MPIDI_NM_am_hdr_max_sz_t am_hdr_max_sz; - MPIDI_NM_am_inject_max_sz_t am_inject_max_sz; MPIDI_NM_am_recv_t am_recv; } MPIDI_NM_funcs_t; @@ -601,7 +595,6 @@ MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_send_amv_reply(MPIR_Context_id_t cont MPIR_Request * sreq) MPIDI_NM_STATIC_INLINE_SUFFIX; MPIDI_NM_STATIC_INLINE_PREFIX size_t MPIDI_NM_am_hdr_max_sz(void) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX size_t MPIDI_NM_am_inject_max_sz(void) MPIDI_NM_STATIC_INLINE_SUFFIX; MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_am_recv(MPIR_Request * req) MPIDI_NM_STATIC_INLINE_SUFFIX; MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_comm_get_lpid(MPIR_Comm * comm_ptr, int idx, @@ -610,12 +603,6 @@ MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_comm_get_lpid(MPIR_Comm * comm_ptr, i MPIDI_NM_STATIC_INLINE_SUFFIX; MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_gpid_get(MPIR_Comm * comm_ptr, int rank, MPIR_Gpid * gpid) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_get_node_id(MPIR_Comm * comm, int rank, - MPID_Node_id_t * - id_p) MPIDI_NM_STATIC_INLINE_SUFFIX; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_get_max_node_id(MPIR_Comm * comm, - MPID_Node_id_t * - max_id_p) MPIDI_NM_STATIC_INLINE_SUFFIX; MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_getallincomm(MPIR_Comm * comm_ptr, int local_size, MPIR_Gpid local_gpids[], int *singleAVT) diff --git a/src/mpid/ch4/netmod/include/netmod_impl.h b/src/mpid/ch4/netmod/include/netmod_impl.h index 96cb922..4fba2a4 100644 --- a/src/mpid/ch4/netmod/include/netmod_impl.h +++ b/src/mpid/ch4/netmod/include/netmod_impl.h @@ -162,11 +162,6 @@ MPIDI_NM_STATIC_INLINE_PREFIX size_t MPIDI_NM_am_hdr_max_sz(void) return MPIDI_NM_func->am_hdr_max_sz(); }; -MPIDI_NM_STATIC_INLINE_PREFIX size_t MPIDI_NM_am_inject_max_sz(void) -{ - return MPIDI_NM_func->am_inject_max_sz(); -}; - MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_am_recv(MPIR_Request * req) { return MPIDI_NM_func->am_recv(req); @@ -184,18 +179,6 @@ MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_gpid_get(MPIR_Comm * comm_ptr, int ra return MPIDI_NM_func->gpid_get(comm_ptr, rank, gpid); }; -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_get_node_id(MPIR_Comm * comm, int rank, - MPID_Node_id_t * id_p) -{ - return MPIDI_NM_func->get_node_id(comm, rank, id_p); -}; - -MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_get_max_node_id(MPIR_Comm * comm, - MPID_Node_id_t * max_id_p) -{ - return MPIDI_NM_func->get_max_node_id(comm, max_id_p); -}; - MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_getallincomm(MPIR_Comm * comm_ptr, int local_size, MPIR_Gpid local_gpid[], int *singleAVT) { diff --git a/src/mpid/ch4/netmod/ofi/func_table.c b/src/mpid/ch4/netmod/ofi/func_table.c index 777c73e..f8a54a4 100644 --- a/src/mpid/ch4/netmod/ofi/func_table.c +++ b/src/mpid/ch4/netmod/ofi/func_table.c @@ -24,8 +24,6 @@ MPIDI_NM_funcs_t MPIDI_NM_ofi_funcs = { MPIDI_NM_comm_accept, MPIDI_NM_comm_get_lpid, MPIDI_NM_gpid_get, - MPIDI_NM_get_node_id, - MPIDI_NM_get_max_node_id, MPIDI_NM_getallincomm, MPIDI_NM_gpid_tolpidarray, MPIDI_NM_create_intercomm_from_lpids, @@ -44,7 +42,6 @@ MPIDI_NM_funcs_t MPIDI_NM_ofi_funcs = { MPIDI_NM_send_am_reply, MPIDI_NM_send_amv_reply, MPIDI_NM_am_hdr_max_sz, - MPIDI_NM_am_inject_max_sz, MPIDI_NM_am_recv }; diff --git a/src/mpid/ch4/netmod/ofi/ofi_am.h b/src/mpid/ch4/netmod/ofi/ofi_am.h index f3425dd..e3b6287 100644 --- a/src/mpid/ch4/netmod/ofi/ofi_am.h +++ b/src/mpid/ch4/netmod/ofi/ofi_am.h @@ -349,13 +349,6 @@ static inline int MPIDI_NM_inject_am_hdr_reply(MPIR_Context_id_t context_id, goto fn_exit; } -static inline size_t MPIDI_NM_am_inject_max_sz(void) -{ - if (unlikely(MPIDI_Global.max_buffered_send < sizeof(MPIDI_OFI_am_header_t))) - return 0; - return MPIDI_Global.max_buffered_send - sizeof(MPIDI_OFI_am_header_t); -} - static inline int MPIDI_NM_am_recv(MPIR_Request * req) { int mpi_errno = MPI_SUCCESS; diff --git a/src/mpid/ch4/netmod/ofi/ofi_init.h b/src/mpid/ch4/netmod/ofi/ofi_init.h index 09e34d2..7b12704 100644 --- a/src/mpid/ch4/netmod/ofi/ofi_init.h +++ b/src/mpid/ch4/netmod/ofi/ofi_init.h @@ -643,18 +643,6 @@ static inline int MPIDI_NM_gpid_get(MPIR_Comm * comm_ptr, int rank, MPIR_Gpid * goto fn_exit; } -static inline int MPIDI_NM_get_node_id(MPIR_Comm * comm, int rank, MPID_Node_id_t * id_p) -{ - MPIDI_CH4U_get_node_id(comm, rank, id_p); - return MPI_SUCCESS; -} - -static inline int MPIDI_NM_get_max_node_id(MPIR_Comm * comm, MPID_Node_id_t * max_id_p) -{ - MPIDI_CH4U_get_max_node_id(comm, max_id_p); - return MPI_SUCCESS; -} - static inline int MPIDI_NM_getallincomm(MPIR_Comm * comm_ptr, int local_size, MPIR_Gpid local_gpids[], int *singleAVT) { diff --git a/src/mpid/ch4/netmod/portals4/func_table.c b/src/mpid/ch4/netmod/portals4/func_table.c index 1c123eb..c6e9334 100644 --- a/src/mpid/ch4/netmod/portals4/func_table.c +++ b/src/mpid/ch4/netmod/portals4/func_table.c @@ -24,8 +24,6 @@ MPIDI_NM_funcs_t MPIDI_NM_portals4_funcs = { MPIDI_NM_comm_accept, MPIDI_NM_comm_get_lpid, MPIDI_NM_gpid_get, - MPIDI_NM_get_node_id, - MPIDI_NM_get_max_node_id, MPIDI_NM_getallincomm, MPIDI_NM_gpid_tolpidarray, MPIDI_NM_create_intercomm_from_lpids, @@ -44,7 +42,6 @@ MPIDI_NM_funcs_t MPIDI_NM_portals4_funcs = { MPIDI_NM_send_am_reply, MPIDI_NM_send_amv_reply, MPIDI_NM_am_hdr_max_sz, - MPIDI_NM_am_inject_max_sz, MPIDI_NM_am_recv, }; diff --git a/src/mpid/ch4/netmod/portals4/ptl_am.h b/src/mpid/ch4/netmod/portals4/ptl_am.h index 6b12443..de0e607 100644 --- a/src/mpid/ch4/netmod/portals4/ptl_am.h +++ b/src/mpid/ch4/netmod/portals4/ptl_am.h @@ -355,12 +355,6 @@ static inline int MPIDI_NM_inject_am_hdr_reply(MPIR_Context_id_t context_id, goto fn_exit; } -static inline size_t MPIDI_NM_am_inject_max_sz(void) -{ - MPIR_Assert(0); - return 0; -} - static inline int MPIDI_NM_am_recv(MPIR_Request * req) { MPIR_Assert(0); diff --git a/src/mpid/ch4/netmod/portals4/ptl_init.h b/src/mpid/ch4/netmod/portals4/ptl_init.h index da36d4e..4f2d592 100644 --- a/src/mpid/ch4/netmod/portals4/ptl_init.h +++ b/src/mpid/ch4/netmod/portals4/ptl_init.h @@ -225,18 +225,6 @@ static inline int MPIDI_NM_gpid_get(MPIR_Comm * comm_ptr, int rank, MPIR_Gpid * return MPI_SUCCESS; } -static inline int MPIDI_NM_get_node_id(MPIR_Comm * comm, int rank, MPID_Node_id_t * id_p) -{ - *id_p = MPIDI_PTL_global.node_map[rank]; - return MPI_SUCCESS; -} - -static inline int MPIDI_NM_get_max_node_id(MPIR_Comm * comm, MPID_Node_id_t * max_id_p) -{ - *max_id_p = MPIDI_PTL_global.max_node_id; - return MPI_SUCCESS; -} - static inline int MPIDI_NM_getallincomm(MPIR_Comm * comm_ptr, int local_size, MPIR_Gpid local_gpids[], int *singleAVT) { diff --git a/src/mpid/ch4/netmod/stubnm/globals.c b/src/mpid/ch4/netmod/stubnm/globals.c index b830d4b..9642082 100644 --- a/src/mpid/ch4/netmod/stubnm/globals.c +++ b/src/mpid/ch4/netmod/stubnm/globals.c @@ -24,8 +24,6 @@ MPIDI_NM_funcs_t MPIDI_NM_stubnm_funcs = { MPIDI_NM_comm_accept, MPIDI_NM_comm_get_lpid, MPIDI_NM_gpid_get, - MPIDI_NM_get_node_id, - MPIDI_NM_get_max_node_id, MPIDI_NM_getallincomm, MPIDI_NM_gpid_tolpidarray, MPIDI_NM_create_intercomm_from_lpids, @@ -44,7 +42,6 @@ MPIDI_NM_funcs_t MPIDI_NM_stubnm_funcs = { MPIDI_NM_send_am_reply, MPIDI_NM_send_amv_reply, MPIDI_NM_am_hdr_max_sz, - MPIDI_NM_am_inject_max_sz, MPIDI_NM_am_recv, }; diff --git a/src/mpid/ch4/netmod/stubnm/stubnm_am.h b/src/mpid/ch4/netmod/stubnm/stubnm_am.h index 2d7d9f2..e3bcda3 100644 --- a/src/mpid/ch4/netmod/stubnm/stubnm_am.h +++ b/src/mpid/ch4/netmod/stubnm/stubnm_am.h @@ -123,12 +123,6 @@ static inline int MPIDI_NM_inject_am_hdr_reply(MPIR_Context_id_t context_id, int return MPI_SUCCESS; } -static inline size_t MPIDI_NM_am_inject_max_sz(void) -{ - MPIR_Assert(0); - return 0; -} - static inline int MPIDI_NM_am_recv(MPIR_Request * req) { MPIR_Assert(0); diff --git a/src/mpid/ch4/netmod/stubnm/stubnm_init.h b/src/mpid/ch4/netmod/stubnm/stubnm_init.h index 5f458a6..8b1a78d 100644 --- a/src/mpid/ch4/netmod/stubnm/stubnm_init.h +++ b/src/mpid/ch4/netmod/stubnm/stubnm_init.h @@ -49,18 +49,6 @@ static inline int MPIDI_NM_gpid_get(MPIR_Comm * comm_ptr, int rank, MPIR_Gpid * return MPI_SUCCESS; } -static inline int MPIDI_NM_get_node_id(MPIR_Comm * comm, int rank, MPID_Node_id_t * id_p) -{ - MPIR_Assert(0); - return MPI_SUCCESS; -} - -static inline int MPIDI_NM_get_max_node_id(MPIR_Comm * comm, MPID_Node_id_t * max_id_p) -{ - MPIR_Assert(0); - return MPI_SUCCESS; -} - static inline int MPIDI_NM_getallincomm(MPIR_Comm * comm_ptr, int local_size, MPIR_Gpid local_gpids[], int *singleAVT) { diff --git a/src/mpid/ch4/netmod/ucx/func_table.c b/src/mpid/ch4/netmod/ucx/func_table.c index 7e99f6d..4f60187 100644 --- a/src/mpid/ch4/netmod/ucx/func_table.c +++ b/src/mpid/ch4/netmod/ucx/func_table.c @@ -22,8 +22,6 @@ MPIDI_NM_funcs_t MPIDI_NM_ucx_funcs = { MPIDI_NM_comm_accept, MPIDI_NM_comm_get_lpid, MPIDI_NM_gpid_get, - MPIDI_NM_get_node_id, - MPIDI_NM_get_max_node_id, MPIDI_NM_getallincomm, MPIDI_NM_gpid_tolpidarray, MPIDI_NM_create_intercomm_from_lpids, @@ -42,7 +40,6 @@ MPIDI_NM_funcs_t MPIDI_NM_ucx_funcs = { MPIDI_NM_send_am_reply, MPIDI_NM_send_amv_reply, MPIDI_NM_am_hdr_max_sz, - MPIDI_NM_am_inject_max_sz, MPIDI_NM_am_recv }; diff --git a/src/mpid/ch4/netmod/ucx/ucx_am.h b/src/mpid/ch4/netmod/ucx/ucx_am.h index 992729d..1a4c595 100644 --- a/src/mpid/ch4/netmod/ucx/ucx_am.h +++ b/src/mpid/ch4/netmod/ucx/ucx_am.h @@ -582,11 +582,6 @@ static inline int MPIDI_NM_inject_am_hdr_reply(MPIR_Context_id_t context_id, goto fn_exit; } -static inline size_t MPIDI_NM_am_inject_max_sz(void) -{ - return MPIDI_NM_am_hdr_max_sz(); -} - static inline int MPIDI_NM_am_recv(MPIR_Request * req) { int mpi_errno = MPI_SUCCESS; diff --git a/src/mpid/ch4/netmod/ucx/ucx_init.h b/src/mpid/ch4/netmod/ucx/ucx_init.h index 06b9e7e..52374b0 100644 --- a/src/mpid/ch4/netmod/ucx/ucx_init.h +++ b/src/mpid/ch4/netmod/ucx/ucx_init.h @@ -233,18 +233,6 @@ static inline int MPIDI_NM_gpid_get(MPIR_Comm * comm_ptr, int rank, MPIR_Gpid * goto fn_exit; } -static inline int MPIDI_NM_get_node_id(MPIR_Comm * comm, int rank, MPID_Node_id_t * id_p) -{ - MPIDI_CH4U_get_node_id(comm, rank, id_p); - return MPI_SUCCESS; -} - -static inline int MPIDI_NM_get_max_node_id(MPIR_Comm * comm, MPID_Node_id_t * max_id_p) -{ - MPIDI_CH4U_get_max_node_id(comm, max_id_p); - return MPI_SUCCESS; -} - static inline int MPIDI_NM_getallincomm(MPIR_Comm * comm_ptr, int local_size, MPIR_Gpid local_gpids[], int *singleAVT) { http://git.mpich.org/mpich.git/commitdiff/7040a6289af4de7ace66ad71248c1ef01… commit 7040a6289af4de7ace66ad71248c1ef01f411855 Author: Pavan Balaji <balaji(a)anl.gov> Date: Thu Aug 18 11:36:13 2016 -0500 CH4/UCX: Fix function name Signed-off-by: Ken Raffenetti <raffenet(a)mcs.anl.gov> diff --git a/src/mpid/ch4/netmod/ucx/ucx_request.h b/src/mpid/ch4/netmod/ucx/ucx_request.h index 35779da..b0a2783 100644 --- a/src/mpid/ch4/netmod/ucx/ucx_request.h +++ b/src/mpid/ch4/netmod/ucx/ucx_request.h @@ -15,7 +15,7 @@ #include "mpidch4r.h" #undef FUNCNAME -#define FUNCNAME MPIDI_NM_request_release +#define FUNCNAME MPIDI_NM_am_request_init #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) static inline void MPIDI_NM_am_request_init(MPIR_Request * req) http://git.mpich.org/mpich.git/commitdiff/21dd483c6c5a075cbe86b0a979af481df… commit 21dd483c6c5a075cbe86b0a979af481dfddf9329 Author: Pavan Balaji <balaji(a)anl.gov> Date: Wed Aug 24 09:15:05 2016 -0500 CH4: Manually manage indentation for some code indent does not understand configure-filled strings, so we turn off indenting for those pieces of code. Signed-off-by: Ken Raffenetti <raffenet(a)mcs.anl.gov> diff --git a/src/mpid/ch4/include/netmodpre.h.in b/src/mpid/ch4/include/netmodpre.h.in index 086714e..9a4ffdf 100644 --- a/src/mpid/ch4/include/netmodpre.h.in +++ b/src/mpid/ch4/include/netmodpre.h.in @@ -11,7 +11,9 @@ #ifndef NETMODPRE_H_INCLUDED #define NETMODPRE_H_INCLUDED +/* *INDENT-OFF* */ @ch4_netmod_pre_include@ +/* *INDENT-ON* */ #define MPIDI_NM_REQUEST_AM_DECL @ch4_netmod_amrequest_decl@ #define MPIDI_NM_REQUEST_DECL @ch4_netmod_request_decl@ diff --git a/src/mpid/ch4/include/shmpre.h.in b/src/mpid/ch4/include/shmpre.h.in index c6bcbdb..069de00 100644 --- a/src/mpid/ch4/include/shmpre.h.in +++ b/src/mpid/ch4/include/shmpre.h.in @@ -11,9 +11,10 @@ #ifndef SHMPRE_H_INCLUDED #define SHMPRE_H_INCLUDED +/* *INDENT-OFF* */ @ch4_shm_pre_include@ +/* *INDENT-ON* */ #define MPIDI_SHM_REQUEST_DECL @ch4_shm_request_decl@ #define MPIDI_SHM_COMM_DECL @ch4_shm_comm_decl@ - #endif /* SHMPRE_H_INCLUDED */ diff --git a/src/mpid/ch4/src/mpid_ch4_net_array.c.in b/src/mpid/ch4/src/mpid_ch4_net_array.c.in index 8765f39..c07b7a6 100644 --- a/src/mpid/ch4/src/mpid_ch4_net_array.c.in +++ b/src/mpid/ch4/src/mpid_ch4_net_array.c.in @@ -11,6 +11,7 @@ #include <mpidimpl.h> +/* *INDENT-OFF* */ /* forward declaration of funcs structs defined in network modules */ extern MPIDI_NM_funcs_t @ch4_nets_func_decl@; extern MPIDI_NM_native_funcs_t @ch4_nets_native_func_decl@; @@ -26,3 +27,4 @@ MPIDI_NM_native_funcs_t *MPIDI_NM_native_funcs[@ch4_nets_array_sz@] = { 0 }; int MPIDI_num_netmods = @ch4_nets_array_sz@; char MPIDI_NM_strings[@ch4_nets_array_sz@][MPIDI_MAX_NETMOD_STRING_LEN] = { @ch4_nets_strings@ }; +/* *INDENT-ON* */ diff --git a/src/mpid/ch4/src/mpid_ch4_shm_array.c.in b/src/mpid/ch4/src/mpid_ch4_shm_array.c.in index 6bce704..3649d45 100644 --- a/src/mpid/ch4/src/mpid_ch4_shm_array.c.in +++ b/src/mpid/ch4/src/mpid_ch4_shm_array.c.in @@ -13,6 +13,7 @@ #ifdef MPIDI_BUILD_CH4_SHM +/* *INDENT-OFF* */ /* forward declaration of funcs structs defined in network modules */ extern MPIDI_SHM_funcs_t @ch4_shm_func_decl@; extern MPIDI_SHM_native_funcs_t @ch4_shm_native_func_decl@; @@ -28,5 +29,6 @@ MPIDI_SHM_native_funcs_t *MPIDI_SHM_native_funcs[@ch4_shm_array_sz@] = { 0 }; int MPIDI_num_shms = @ch4_shm_array_sz@; char MPIDI_SHM_strings[@ch4_shm_array_sz@][MPIDI_MAX_SHM_STRING_LEN] = { @ch4_shm_strings@ }; +/* *INDENT-ON* */ #endif http://git.mpich.org/mpich.git/commitdiff/3dd1f56608e155242d8118cb2391d87f6… commit 3dd1f56608e155242d8118cb2391d87f6bb33091 Author: Pavan Balaji <balaji(a)anl.gov> Date: Wed Aug 24 09:16:47 2016 -0500 CH4: White-space cleanup Signed-off-by: Ken Raffenetti <raffenet(a)mcs.anl.gov> diff --git a/src/mpid/ch4/netmod/ofi/ofi_iovec_util.h b/src/mpid/ch4/netmod/ofi/ofi_iovec_util.h index 36c039f..e40f7f1 100644 --- a/src/mpid/ch4/netmod/ofi/ofi_iovec_util.h +++ b/src/mpid/ch4/netmod/ofi/ofi_iovec_util.h @@ -203,9 +203,8 @@ static inline uintptr_t * target_addr_next, size_t * buf_len) { if ((iov_state->origin_size != 0) && (iov_state->target_size != 0)) { - uintptr_t buf_size = - MPL_MIN(MPL_MIN(iov_state->target_size, iov_state->origin_size), - iov_state->buf_limit_left); + uintptr_t buf_size = MPL_MIN(MPL_MIN(iov_state->target_size, iov_state->origin_size), + iov_state->buf_limit_left); *buf_len = buf_size; MPIDI_OFI_NEXT_IOV_STATE(target); MPIDI_OFI_NEXT_IOV_STATE(origin); diff --git a/src/mpid/ch4/src/ch4r_win.h b/src/mpid/ch4/src/ch4r_win.h index 840f890..833066c 100644 --- a/src/mpid/ch4/src/ch4r_win.h +++ b/src/mpid/ch4/src/ch4r_win.h @@ -53,20 +53,20 @@ static inline int MPIDI_CH4R_win_set_info(MPIR_Win * win, MPIR_Info * info) while (token) { if (!memcmp(token, "rar", 3)) MPIDI_CH4U_WIN(win, info_args).accumulate_ordering = - (MPIDI_CH4U_WIN(win, info_args). - accumulate_ordering | MPIDI_CH4I_ACCU_ORDER_RAR); + (MPIDI_CH4U_WIN(win, info_args).accumulate_ordering | + MPIDI_CH4I_ACCU_ORDER_RAR); else if (!memcmp(token, "raw", 3)) MPIDI_CH4U_WIN(win, info_args).accumulate_ordering = - (MPIDI_CH4U_WIN(win, info_args). - accumulate_ordering | MPIDI_CH4I_ACCU_ORDER_RAW); + (MPIDI_CH4U_WIN(win, info_args).accumulate_ordering | + MPIDI_CH4I_ACCU_ORDER_RAW); else if (!memcmp(token, "war", 3)) MPIDI_CH4U_WIN(win, info_args).accumulate_ordering = - (MPIDI_CH4U_WIN(win, info_args). - accumulate_ordering | MPIDI_CH4I_ACCU_ORDER_WAR); + (MPIDI_CH4U_WIN(win, info_args).accumulate_ordering | + MPIDI_CH4I_ACCU_ORDER_WAR); else if (!memcmp(token, "waw", 3)) MPIDI_CH4U_WIN(win, info_args).accumulate_ordering = - (MPIDI_CH4U_WIN(win, info_args). - accumulate_ordering | MPIDI_CH4I_ACCU_ORDER_WAW); + (MPIDI_CH4U_WIN(win, info_args).accumulate_ordering | + MPIDI_CH4I_ACCU_ORDER_WAW); else MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_ARG, goto fn_fail, "**info"); http://git.mpich.org/mpich.git/commitdiff/daa80957db99eea196c920bf465a67972… commit daa80957db99eea196c920bf465a67972ad5af9f Author: Pavan Balaji <balaji(a)anl.gov> Date: Thu Aug 18 12:23:14 2016 -0500 maint: fixup mpich-replace Signed-off-by: Ken Raffenetti <raffenet(a)mcs.anl.gov> diff --git a/maint/mpich-replace.sh b/maint/mpich-replace.sh index baba105..55e4e30 100755 --- a/maint/mpich-replace.sh +++ b/maint/mpich-replace.sh @@ -1,3 +1,3 @@ #! /bin/bash -git grep $1 * | cut -f1 -d':' | uniq | xargs sed -i "s/$1/$2/g" +git grep $1 * | cut -f1 -d':' | uniq | xargs sed -i "s/\b$1\b/$2/g" http://git.mpich.org/mpich.git/commitdiff/e22e938f22f4af2f432ca3011074c3e63… commit e22e938f22f4af2f432ca3011074c3e630922f2c Author: Pavan Balaji <balaji(a)anl.gov> Date: Wed Aug 24 09:17:21 2016 -0500 maint: fixup the code-cleanup script. Run the indent script twice. The indent distributed on linux seems to be somewhat buggy in that it toggles the indentation each time you run it, thus flip-flopping between two indentations. Running it twice will discard such issues. Signed-off-by: Ken Raffenetti <raffenet(a)mcs.anl.gov> diff --git a/maint/code-cleanup.sh b/maint/code-cleanup.sh index 01c12ac..413c5b4 100755 --- a/maint/code-cleanup.sh +++ b/maint/code-cleanup.sh @@ -73,12 +73,15 @@ if [ "$recursive" = "1" ]; then for i in `find . \! -type d | egrep '(\.c$|\.h$|\.c\.in$|\.h\.in$|\.cpp$|\.cpp.in$)' | \ egrep -v "($ignore_list)"` ; do ${debug} indent_code $i + ${debug} indent_code $i done elif [ "$all" = "1" ]; then for i in `find . -maxdepth 1 \! -type d | egrep '(\.c$|\.h$|\.c\.in$|\.h\.in$|\.cpp$|\.cpp.in$)' | \ egrep -v "($ignore_list)"` ; do ${debug} indent_code $i + ${debug} indent_code $i done else ${debug} indent_code $@ + ${debug} indent_code $@ fi ----------------------------------------------------------------------- Summary of changes: maint/code-cleanup.sh | 3 + maint/mpich-replace.sh | 2 +- src/include/mpir_attr_generic.h | 2 +- src/mpi/attr/comm_set_attr.c | 2 +- src/mpi/attr/type_set_attr.c | 2 +- src/mpi/attr/win_set_attr.c | 2 +- src/mpi/coll/op_create.c | 4 +- src/mpi/coll/op_free.c | 4 +- src/mpi/comm/commutil.c | 8 +- src/mpid/ch3/include/mpidpre.h | 4 +- src/mpid/ch3/src/ch3u_port.c | 2 +- src/mpid/ch4/include/mpidch4.h | 13 +- src/mpid/ch4/include/mpidpre.h | 12 +- src/mpid/ch4/include/netmodpre.h.in | 2 + src/mpid/ch4/include/shmpre.h.in | 3 +- src/mpid/ch4/netmod/include/netmod.h | 1181 ++++++++++------------- src/mpid/ch4/netmod/include/netmod_impl.h | 875 ++++++++--------- src/mpid/ch4/netmod/ofi/func_table.c | 33 +- src/mpid/ch4/netmod/ofi/ofi_am.h | 256 ++---- src/mpid/ch4/netmod/ofi/ofi_am_events.h | 8 +- src/mpid/ch4/netmod/ofi/ofi_am_impl.h | 78 +- src/mpid/ch4/netmod/ofi/ofi_comm.h | 8 +- src/mpid/ch4/netmod/ofi/ofi_datatype.h | 12 +- src/mpid/ch4/netmod/ofi/ofi_events.h | 8 +- src/mpid/ch4/netmod/ofi/ofi_impl.h | 2 +- src/mpid/ch4/netmod/ofi/ofi_init.h | 16 +- src/mpid/ch4/netmod/ofi/ofi_iovec_util.h | 5 +- src/mpid/ch4/netmod/ofi/ofi_op.h | 8 +- src/mpid/ch4/netmod/ofi/ofi_recv.h | 5 + src/mpid/ch4/netmod/ofi/ofi_types.h | 4 +- src/mpid/ch4/netmod/ofi/util.c | 2 +- src/mpid/ch4/netmod/portals4/func_table.c | 33 +- src/mpid/ch4/netmod/portals4/ptl_am.h | 161 +-- src/mpid/ch4/netmod/portals4/ptl_comm.h | 8 +- src/mpid/ch4/netmod/portals4/ptl_datatype.h | 8 +- src/mpid/ch4/netmod/portals4/ptl_init.h | 12 - src/mpid/ch4/netmod/portals4/ptl_op.h | 8 +- src/mpid/ch4/netmod/stubnm/globals.c | 33 +- src/mpid/ch4/netmod/stubnm/stubnm_am.h | 98 +-- src/mpid/ch4/netmod/stubnm/stubnm_comm.h | 8 +- src/mpid/ch4/netmod/stubnm/stubnm_datatype.h | 12 +- src/mpid/ch4/netmod/stubnm/stubnm_init.h | 12 - src/mpid/ch4/netmod/stubnm/stubnm_op.h | 4 +- src/mpid/ch4/netmod/ucx/func_table.c | 33 +- src/mpid/ch4/netmod/ucx/ucx_am.h | 258 +---- src/mpid/ch4/netmod/ucx/ucx_comm.h | 8 +- src/mpid/ch4/netmod/ucx/ucx_datatype.h | 12 +- src/mpid/ch4/netmod/ucx/ucx_init.h | 12 - src/mpid/ch4/netmod/ucx/ucx_op.h | 8 +- src/mpid/ch4/netmod/ucx/ucx_pre.h | 4 +- src/mpid/ch4/netmod/ucx/ucx_progress.h | 2 +- src/mpid/ch4/netmod/ucx/ucx_request.h | 2 +- src/mpid/ch4/shm/include/shm.h | 1251 +++++++++++------------- src/mpid/ch4/shm/include/shm_impl.h | 951 +++++++++---------- src/mpid/ch4/shm/posix/func_table.c | 23 +- src/mpid/ch4/shm/posix/posix_am.h | 78 +- src/mpid/ch4/shm/posix/posix_comm.h | 8 +- src/mpid/ch4/shm/posix/posix_init.h | 56 ++ src/mpid/ch4/shm/stubshm/func_table.c | 23 +- src/mpid/ch4/shm/stubshm/stubshm_am.h | 78 +- src/mpid/ch4/shm/stubshm/stubshm_comm.h | 4 +- src/mpid/ch4/shm/stubshm/stubshm_init.h | 56 ++ src/mpid/ch4/src/ch4_coll.h | 328 ++++--- src/mpid/ch4/src/ch4_comm.h | 46 +- src/mpid/ch4/src/ch4_impl.h | 4 +- src/mpid/ch4/src/ch4_init.h | 168 +++- src/mpid/ch4/src/ch4_probe.h | 32 +- src/mpid/ch4/src/ch4_proc.h | 2 +- src/mpid/ch4/src/ch4_progress.h | 18 +- src/mpid/ch4/src/ch4_recv.h | 63 +- src/mpid/ch4/src/ch4_request.h | 12 +- src/mpid/ch4/src/ch4_rma.h | 152 ++-- src/mpid/ch4/src/ch4_send.h | 134 ++-- src/mpid/ch4/src/ch4_spawn.h | 35 +- src/mpid/ch4/src/ch4_types.h | 11 +- src/mpid/ch4/src/ch4_win.h | 75 +- src/mpid/ch4/src/ch4i_comm.h | 2 +- src/mpid/ch4/src/ch4r_callbacks.h | 240 +++--- src/mpid/ch4/src/ch4r_init.h | 176 +++-- src/mpid/ch4/src/ch4r_probe.h | 36 +- src/mpid/ch4/src/ch4r_recv.h | 60 +- src/mpid/ch4/src/ch4r_recvq.h | 54 +- src/mpid/ch4/src/ch4r_request.h | 2 +- src/mpid/ch4/src/ch4r_rma.h | 232 +++--- src/mpid/ch4/src/ch4r_send.h | 148 ++-- src/mpid/ch4/src/ch4r_win.h | 46 +- src/mpid/ch4/src/mpid_ch4_net_array.c.in | 2 + src/mpid/ch4/src/mpid_ch4_shm_array.c.in | 2 + src/mpid/common/datatype/mpidu_datatype_free.c | 6 +- src/mpid/common/datatype/mpidu_type_commit.c | 6 +- src/mpid/common/datatype/mpidu_type_dup.c | 6 +- src/mpid/pamid/include/mpidi_hooks.h | 4 +- src/mpl/include/mpl_base.h | 3 + 93 files changed, 3685 insertions(+), 4250 deletions(-) hooks/post-receive -- MPICH primary repository

1 0

Argonne Spam Quarantine Notification
by Argonne Anti Spam Quarantine 25 Aug '16

25 Aug '16

++ Note: This message has been sent by a notification only system. Please do not reply ++ IronPort Spam Quarantine Notification If a spam message was missed, or if a message was incorrectly flagged as spam, you can add the sender to a personal safelist or blocklist via the quarantine. Senders on the safelist will not be flagged as spam, and senders on the blocklist will be automatically quarantined. For more information, visit: https://wiki.inside.anl.gov/inside/Ironport_Anti-Spam_Appliances/Documentat… The message(s) below have been quarantined by the Argonne National Laboratory spam filtering system. There are 1 new messages in your Email Quarantine since you received your last Spam Quarantine Notification. If the messages below are spam, you do not need to take any action. Messages will be automatically deleted from the quarantine after 28 day(s). If any of the messages below are not spam, click the 'Release' link to have them sent to your Inbox. Messages that have been correctly flagged as spam do not need to be reported. ----------- New Quarantine Messages --------------- Message 1 From: European Company Network <register(a)eucompanynetwork.com> Subject: *****SPAM*****European Company Network 2016/2017 Date: 24 Aug 2016 Release: http://mailgateway.anl.gov:8080/Message?action=Release&mid=128083556&h=ca84… --------------------------------------------------- To manage your quarantine please visit the URL below: http://mailgateway.anl.gov:8080/Search?h=fc87c670d8c3e2454112880d759ad6c4&e… If you have any questions regarding the Laboratory spam filtering product, please contact the CIS helpdesk at help(a)anl.gov or x-9999 option 2.

1 0

[mpich] MPICH primary repository branch, master, updated. v3.2-385-gcd846d2
by noreply＠mpich.org 23 Aug '16

23 Aug '16

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "MPICH primary repository". The branch, master has been updated via cd846d2c0f3349032f981b297a28eef8af9a5995 (commit) from 1494e0f44808516588ab7f57d0754cf27556f43c (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- http://git.mpich.org/mpich.git/commitdiff/cd846d2c0f3349032f981b297a28eef8a… commit cd846d2c0f3349032f981b297a28eef8af9a5995 Author: Ken Raffenetti <raffenet(a)mcs.anl.gov> Date: Mon Aug 22 14:10:26 2016 -0500 mpid: Initial commit of CH4 device CH4 is a new device layer implementation designed for low software overheads. Network modules to be supported at first are OFI, UCX, and Portals 4. POSIX shared memory is supported for intranode communication. CH4 design was a collaborative effort - contributions from (in alphabetical order): Argonne National Laboratory Intel Corporation Mellanox Technologies RIKEN AICS diff --git a/.gitignore b/.gitignore index 6ad5751..6a297db 100644 --- a/.gitignore +++ b/.gitignore @@ -599,3 +599,6 @@ Makefile.am-stamp # /test/util/timer/ /test/util/timer/timertest +# /src/mpid/ch4/include +/src/mpid/ch4/include/netmodpre.h +/src/mpid/ch4/include/shmpre.h diff --git a/src/mpid/Makefile.mk b/src/mpid/Makefile.mk index b1a06ae..84122ef 100644 --- a/src/mpid/Makefile.mk +++ b/src/mpid/Makefile.mk @@ -12,5 +12,6 @@ noinst_HEADERS += \ include $(top_srcdir)/src/mpid/ch3/Makefile.mk +include $(top_srcdir)/src/mpid/ch4/Makefile.mk include $(top_srcdir)/src/mpid/pamid/Makefile.mk include $(top_srcdir)/src/mpid/common/Makefile.mk diff --git a/src/mpid/ch4/.gitignore b/src/mpid/ch4/.gitignore new file mode 100644 index 0000000..f0558fd --- /dev/null +++ b/src/mpid/ch4/.gitignore @@ -0,0 +1,3 @@ +# ch4 ignore files +src/mpid_ch4_net_array.c +src/mpid_ch4_shm_array.c diff --git a/src/mpid/ch4/Makefile.mk b/src/mpid/ch4/Makefile.mk new file mode 100644 index 0000000..8fceac6 --- /dev/null +++ b/src/mpid/ch4/Makefile.mk @@ -0,0 +1,20 @@ +## -*- Mode: Makefile; -*- +## vim: set ft=automake : +## +## (C) 2016 by Argonne National Laboratory. +## See COPYRIGHT in top-level directory. +## +## Portions of this code were written by Intel Corporation. +## Copyright (C) 2011-2016 Intel Corporation. Intel provides this material +## to Argonne National Laboratory subject to Software Grant and Corporate +## Contributor License Agreement dated February 8, 2012. +## + +if BUILD_CH4 + +include $(top_srcdir)/src/mpid/ch4/include/Makefile.mk +include $(top_srcdir)/src/mpid/ch4/src/Makefile.mk +include $(top_srcdir)/src/mpid/ch4/netmod/Makefile.mk +include $(top_srcdir)/src/mpid/ch4/shm/Makefile.mk + +endif BUILD_CH4 diff --git a/src/mpid/ch4/cross/gcc-linux-x86-8 b/src/mpid/ch4/cross/gcc-linux-x86-8 new file mode 100644 index 0000000..33e0896 --- /dev/null +++ b/src/mpid/ch4/cross/gcc-linux-x86-8 @@ -0,0 +1,13 @@ +CROSS_F77_SIZEOF_INTEGER="4" +CROSS_F77_SIZEOF_REAL="4" +CROSS_F77_SIZEOF_DOUBLE_PRECISION="8" +CROSS_F77_TRUE_VALUE="1" +CROSS_F77_FALSE_VALUE="0" +CROSS_F90_ADDRESS_KIND="8" +CROSS_F90_OFFSET_KIND="8" +CROSS_F90_INTEGER_KIND="4" +CROSS_F90_REAL_MODEL=" 6 , 37" +CROSS_F90_DOUBLE_MODEL=" 15 , 307" +CROSS_F90_INTEGER_MODEL=" 9" +CROSS_F90_ALL_INTEGER_MODELS=" 2 , 1, 4 , 2, 9 , 4, 18 , 8," +CROSS_F90_INTEGER_MODEL_MAP=" { 2 , 1 , 1 }, { 4 , 2 , 2 }, { 9 , 4 , 4 }, { 18 , 8 , 8 }," diff --git a/src/mpid/ch4/cross/icc-linux-x86-8 b/src/mpid/ch4/cross/icc-linux-x86-8 new file mode 100644 index 0000000..c3800d1 --- /dev/null +++ b/src/mpid/ch4/cross/icc-linux-x86-8 @@ -0,0 +1,13 @@ +CROSS_F77_SIZEOF_INTEGER="4" +CROSS_F77_SIZEOF_REAL="4" +CROSS_F77_SIZEOF_DOUBLE_PRECISION="8" +CROSS_F77_TRUE_VALUE="-1" +CROSS_F77_FALSE_VALUE="0" +CROSS_F90_ADDRESS_KIND="8" +CROSS_F90_OFFSET_KIND="8" +CROSS_F90_INTEGER_KIND="4" +CROSS_F90_REAL_MODEL=" 6 , 37" +CROSS_F90_DOUBLE_MODEL=" 15 , 307" +CROSS_F90_INTEGER_MODEL=" 9" +CROSS_F90_ALL_INTEGER_MODELS=" 2 , 1, 4 , 2, 9 , 4, 18 , 8," +CROSS_F90_INTEGER_MODEL_MAP=" { 2 , 1 , 1 }, { 4 , 2 , 2 }, { 9 , 4 , 4 }, { 18 , 8 , 8 }," diff --git a/src/mpid/ch4/errnames.txt b/src/mpid/ch4/errnames.txt new file mode 100644 index 0000000..55c9135 --- /dev/null +++ b/src/mpid/ch4/errnames.txt @@ -0,0 +1,47 @@ +# +# CH4 errors +# +**ch4|pktarraytoosmall: Size of the array of packet handlers is too small +**ch4|badmsgtype %d:request contained an invalid message type (%d) +**ch4|badmsgtype:request contained an invalid message type +**ch4|badreqtype %d:request contained an invalid request type (%d) +**ch4|badreqtype:request contained an invalid request type +**ch4|cancelreq:failure occurred while sending remote cancellation request packet +**ch4|cancelresp:failure occurred while attempting to send cancel response packet +**ch4|cancelrndv:failure occurred while performing local cancellation of a rendezvous message +**ch4|ch4_init:channel initialization failed +**ch4|ctspkt:failure occurred while attempting to send CTS packet +**ch4|eagermsg:failure occurred while attempting to send an eager message +**ch4|loadrecviov:failure occurred while loading the receive I/O vector +**ch4|loadsendiov:failure occurred while loading the send I/O vector +**ch4|nopktcontainermem:failed to allocate memory for a packet reorder container +**ch4|ooocancelreq:UNIMPLEMENTED: unable to process out-of-order cancellation requests +**ch4|pktordered:failure occurred while processing a reordered packet +**ch4|postrecv %s:failure occurred while posting a receive for message data (%s) +**ch4|postrecv:failure occurred while posting a receive for message data +**ch4|rmamsg:failure occurred while attempting to send an RMA message +**ch4|rtspkt:failure occurred while attempting to send RTS packet +**ch4|senddata:failure occurred while attempting to send message data +**ch4|syncack:failure occurred while attempting to send eager synchronization packet +**ch4|get_parent_port:spawn process group was unable to obtain parent port name from the channel +**ch4|conn_parent:spawned process group was unable to connect back to the parent +**ch4|conn_parent %s:spawned process group was unable to connect back to the parent on port <%s> +**ch4|unhandled_connection_state:encountered an unexpected connection state +**ch4|unhandled_connection_state %p %d:encountered an unexpected connection state (vc=%p, state=%d) +**ch4|send_close_ack:an error occurred when the device attempted to acknowledge the closing of a connection +**ch4|close_progress:an error occurred while the device was waiting for all open connections to close +**ch4|pmi_finalize:PMI_Finalize failed +**ch4|pmi_finalize %d:PMI_Finalize failed, error %d +**ch4|invalid_shm:Invalid shm specified +**ch4|invalid_shm %s:Invalid shm specified (%s) +**ch4|invalid_locality:Invalid locality usage +**ch4|invalid_locality %s:Invalid locality usage (%s) +# +# RMA errors +# +**ch4|sync_arg:Invalid RMA synchronization argument +**ch4|sync_arg %d:Invalid RMA synchronization argument (%d) +**ch4|rma_flags:Invalid combination of RMA packet flags +**ch4|nocheck_invalid:MPI_MODE_NOCHECK was specified, but the lock was not available at the target +**ch4|rma_msg:RMA message operation failed +**ch4|win_shared_comm:Cannot map shared memory using the given communicator diff --git a/src/mpid/ch4/include/Makefile.mk b/src/mpid/ch4/include/Makefile.mk new file mode 100644 index 0000000..bd58048 --- /dev/null +++ b/src/mpid/ch4/include/Makefile.mk @@ -0,0 +1,23 @@ +## -*- Mode: Makefile; -*- +## vim: set ft=automake : +## +## (C) 2016 by Argonne National Laboratory. +## See COPYRIGHT in top-level directory. +## +## Portions of this code were written by Intel Corporation. +## Copyright (C) 2011-2016 Intel Corporation. Intel provides this material +## to Argonne National Laboratory subject to Software Grant and Corporate +## Contributor License Agreement dated February 8, 2012. +## + +AM_CPPFLAGS += -I$(top_srcdir)/src/mpid/ch4/include +AM_CPPFLAGS += -I$(top_builddir)/src/mpid/ch4/include + +noinst_HEADERS += src/mpid/ch4/include/netmodpre.h \ + src/mpid/ch4/include/shmpre.h \ + src/mpid/ch4/include/mpidch4.h \ + src/mpid/ch4/include/mpidch4r.h \ + src/mpid/ch4/include/mpidimpl.h \ + src/mpid/ch4/include/mpidpre.h \ + src/mpid/ch4/include/mpid_sched.h \ + src/mpid/ch4/include/mpid_thread.h diff --git a/src/mpid/ch4/include/mpid_sched.h b/src/mpid/ch4/include/mpid_sched.h new file mode 100644 index 0000000..818a1cf --- /dev/null +++ b/src/mpid/ch4/include/mpid_sched.h @@ -0,0 +1,30 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2015 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef MPID_SCHED_H_INCLUDED +#define MPID_SCHED_H_INCLUDED +#include "mpidu_sched.h" + +#define MPIR_Sched_cb MPIDU_Sched_cb +#define MPIR_Sched_cb2 MPIDU_Sched_cb2 +#define MPIR_Sched_next_tag MPIDU_Sched_next_tag +#define MPIR_Sched_create MPIDU_Sched_create +#define MPIR_Sched_clone MPIDU_Sched_clone +#define MPIR_Sched_start MPIDU_Sched_start +#define MPIR_Sched_send MPIDU_Sched_send +#define MPIR_Sched_send_defer MPIDU_Sched_send_defer +#define MPIR_Sched_recv MPIDU_Sched_recv +#define MPIR_Sched_recv_status MPIDU_Sched_recv_status +#define MPIR_Sched_ssend MPIDU_Sched_ssend +#define MPIR_Sched_reduce MPIDU_Sched_reduce +#define MPIR_Sched_copy MPIDU_Sched_copy +#define MPIR_Sched_barrier MPIDU_Sched_barrier + +#endif /* MPID_SCHED_H_INCLUDED */ diff --git a/src/mpid/ch4/include/mpid_thread.h b/src/mpid/ch4/include/mpid_thread.h new file mode 100644 index 0000000..2122d3d --- /dev/null +++ b/src/mpid/ch4/include/mpid_thread.h @@ -0,0 +1,68 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2001 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef MPID_THREAD_H_INCLUDED +#define MPID_THREAD_H_INCLUDED + +#include "mpidu_thread_fallback.h" + +/* We simply use the fallback timer functionality and do not define + * our own */ + +typedef MPIDU_Thread_cond_t MPID_Thread_cond_t; +typedef MPIDU_Thread_id_t MPID_Thread_id_t; +typedef MPIDU_Thread_tls_t MPID_Thread_tls_t; +typedef MPIDU_Thread_func_t MPID_Thread_func_t; + +#ifdef MPIDI_CH4_USE_TICKET_LOCK +#include "mpid_ticketlock.h" +typedef MPIDI_CH4_Ticket_lock MPID_Thread_mutex_t; +#define MPID_THREAD_CS_ENTER MPIDI_CH4I_THREAD_CS_ENTER +#define MPID_THREAD_CS_EXIT MPIDI_CH4I_THREAD_CS_EXIT +#define MPID_THREAD_CS_YIELD MPIDI_CH4I_THREAD_CS_YIELD +#define MPID_Thread_mutex_create MPIDI_CH4I_Thread_mutex_create +#define MPID_Thread_mutex_destroy MPIDI_CH4I_Thread_mutex_destroy +#define MPID_Thread_mutex_lock MPIDI_CH4I_Thread_mutex_lock +#define MPID_Thread_mutex_unlock MPIDI_CH4I_Thread_mutex_unlock +#define MPID_Thread_cond_wait MPIDI_CH4I_Thread_cond_wait +#else +typedef MPIDU_Thread_mutex_t MPID_Thread_mutex_t; +#define MPID_THREAD_CS_ENTER MPIDU_THREAD_CS_ENTER +#define MPID_THREAD_CS_EXIT MPIDU_THREAD_CS_EXIT +#define MPID_THREAD_CS_YIELD MPIDU_THREAD_CS_YIELD +#define MPID_Thread_mutex_create MPIDU_Thread_mutex_create +#define MPID_Thread_mutex_destroy MPIDU_Thread_mutex_destroy +#define MPID_Thread_mutex_lock MPIDU_Thread_mutex_lock +#define MPID_Thread_mutex_unlock MPIDU_Thread_mutex_unlock +#define MPID_Thread_cond_wait MPIDU_Thread_cond_wait +#endif /* MPIDI_CH4_USE_TICKET_LOCK */ + +#define MPID_Thread_create MPIDU_Thread_create +#define MPID_Thread_exit MPIDU_Thread_exit +#define MPID_Thread_self MPIDU_Thread_self +#define MPID_Thread_same MPIDU_Thread_same +#define MPID_Thread_same MPIDU_Thread_same + +#define MPID_Thread_cond_create MPIDU_Thread_cond_create +#define MPID_Thread_cond_destroy MPIDU_Thread_cond_destroy +#define MPID_Thread_cond_broadcast MPIDU_Thread_cond_broadcast +#define MPID_Thread_cond_signal MPIDU_Thread_cond_signal + +#define MPID_Thread_tls_create MPIDU_Thread_tls_create +#define MPID_Thread_tls_destroy MPIDU_Thread_tls_destroy +#define MPID_Thread_tls_set MPIDU_Thread_tls_set +#define MPID_Thread_tls_get MPIDU_Thread_tls_get + +#define MPID_THREADPRIV_KEY_CREATE MPIDU_THREADPRIV_KEY_CREATE +#define MPID_THREADPRIV_KEY_GET_ADDR MPIDU_THREADPRIV_KEY_GET_ADDR +#define MPID_THREADPRIV_KEY_DESTROY MPIDU_THREADPRIV_KEY_DESTROY + + +#endif /* MPID_THREAD_H_INCLUDED */ diff --git a/src/mpid/ch4/include/mpid_ticketlock.h b/src/mpid/ch4/include/mpid_ticketlock.h new file mode 100644 index 0000000..0dc5567 --- /dev/null +++ b/src/mpid/ch4/include/mpid_ticketlock.h @@ -0,0 +1,159 @@ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef MPID_TICKETLOCK_H_INCLUDED +#define MPID_TICKETLOCK_H_INCLUDED + +#define __MUTEX_INLINE__ __attribute__((always_inline))static inline + +#define MPIDI_CH4_CACHELINE_SIZE 64 + +typedef union MPIDI_CH4_Ticket_lock { + unsigned u; + char cacheline[MPIDI_CH4_CACHELINE_SIZE]; + struct { + unsigned short ticket; + unsigned short clients; + } s; +} MPIDI_CH4_Ticket_lock __attribute__ ((aligned(MPIDI_CH4_CACHELINE_SIZE))); + +__MUTEX_INLINE__ void MPIDI_CH4I_Thread_mutex_acquire(MPIDI_CH4_Ticket_lock * m) +{ + uint16_t u = __sync_fetch_and_add(&m->s.clients, 1); + while (m->s.ticket != u) + asm volatile ("pause\n":::"memory"); +} + +__MUTEX_INLINE__ void MPIDI_CH4I_Thread_mutex_release(MPIDI_CH4_Ticket_lock * m) +{ + asm volatile ("":::"memory"); + m->s.ticket++; +} + +__MUTEX_INLINE__ int MPIDI_CH4I_Thread_mutex_try_acquire(MPIDI_CH4_Ticket_lock * m) +{ + uint16_t u = m->s.clients; + uint16_t u2 = u + 1; + uint32_t val = ((uint32_t) u << 16) + u; + uint32_t val2 = ((uint32_t) u2 << 16) + u; + + if (__sync_val_compare_and_swap(&m->u, val, val2) == val) + return 0; + + return EBUSY; +} + +__MUTEX_INLINE__ void MPIDI_CH4I_Thread_mutex_lock(MPIDI_CH4_Ticket_lock * m, int *mpi_error) +{ + MPIDI_CH4I_Thread_mutex_acquire(m); + *mpi_error = 0; +} + +__MUTEX_INLINE__ void MPIDI_CH4I_Thread_mutex_unlock(MPIDI_CH4_Ticket_lock * m, int *mpi_error) +{ + MPIDI_CH4I_Thread_mutex_release(m); + *mpi_error = 0; +} + +__MUTEX_INLINE__ void MPIDI_CH4I_Thread_mutex_create(MPIDI_CH4_Ticket_lock * m, int *mpi_error) +{ + m->u = 0; + *mpi_error = 0; +} + +__MUTEX_INLINE__ void MPIDI_CH4I_Thread_mutex_destroy(MPIDI_CH4_Ticket_lock * m, int *mpi_error) +{ + m->u = 0; + *mpi_error = 0; +} + +/* For this implementation we have two options */ +/* 1) Split the typedef for condition variable mutexes and call the utility routines */ +/* 2) Implement it from scratch */ +/* Currently only async.c is using condition variables, so we should figure out what */ +/* we really want from the cv implementations */ +__MUTEX_INLINE__ void +MPIDI_CH4I_Thread_cond_wait(MPIDU_Thread_cond_t * cond, MPIDI_CH4_Ticket_lock * m, int *mpi_error) +{ + MPIR_Assert(0); +} + + +#if MPICH_THREAD_GRANULARITY == MPICH_THREAD_GRANULARITY__GLOBAL + +#define MPIDI_CH4I_THREAD_CS_ENTER_POBJ(mutex) +#define MPIDI_CH4I_THREAD_CS_EXIT_POBJ(mutex) +#define MPIDI_CH4I_THREAD_CS_TRY_POBJ(mutex) +#define MPIDI_CH4I_THREAD_CS_YIELD_POBJ(mutex) + +#define MPIDI_CH4I_THREAD_CS_ENTER_GLOBAL(m) do { if (MPIR_ThreadInfo.isThreaded) { MPIDI_CH4I_Thread_mutex_acquire(&m); }} while (0) +#define MPIDI_CH4I_THREAD_CS_EXIT_GLOBAL(m) do { if (MPIR_ThreadInfo.isThreaded) { MPIDI_CH4I_Thread_mutex_release(&m); }} while (0) +#define MPIDI_CH4I_THREAD_CS_TRY_GLOBAL(m) do { (0==MPIDI_CH4I_Thread_mutex_try_acquire(&m));}} while (0) +#define MPIDI_CH4I_THREAD_CS_YIELD_GLOBAL(m) do { if (MPIR_ThreadInfo.isThreaded) { MPIDI_CH4I_Thread_mutex_release(&m); sched_yield(); MPIDI_CH4I_Thread_mutex_acquire(&m); }} while (0) + +#define MPIDI_CH4I_THREAD_CS_ENTER_ALLGRAN(mutex) MPIDI_CH4I_THREAD_CS_ENTER_GLOBAL(m) +#define MPIDI_CH4I_THREAD_CS_EXIT_ALLGRAN(mutex) MPIDI_CH4I_THREAD_CS_EXIT_GLOBAL(m) +#define MPIDI_CH4I_THREAD_CS_TRY_ALLGRAN(mutex) MPIDI_CH4I_THREAD_CS_TRY_GLOBAL(m) +#define MPIDI_CH4I_THREAD_CS_YIELD_ALLGRAN(mutex) MPIDI_CH4I_THREAD_CS_YIELD_GLOBAL(m) + +#elif MPICH_THREAD_GRANULARITY == MPICH_THREAD_GRANULARITY__POBJ + +#define MPIDI_CH4I_THREAD_CS_ENTER_POBJ(m) \ + do { \ + if (likely(MPIR_ThreadInfo.isThreaded)) { \ + MPIDI_CH4I_Thread_mutex_acquire(&m); \ + } \ + } while (0) + +#define MPIDI_CH4I_THREAD_CS_EXIT_POBJ(m) \ + do { \ + if (likely(MPIR_ThreadInfo.isThreaded)) { \ + MPIDI_CH4I_Thread_mutex_release(&m); \ + } \ + } while (0) + +#define MPIDI_CH4I_THREAD_CS_TRY_POBJ(m) \ + do { \ + if (likely(MPIR_ThreadInfo.isThreaded)) { \ + MPIDI_CH4I_Thread_mutex_try_acquire(&m); \ + } \ + } while (0) + +#define MPIDI_CH4I_THREAD_CS_YIELD_POBJ(m) \ + do { \ + if (likely(MPIR_ThreadInfo.isThreaded)) { \ + MPIDI_CH4I_Thread_mutex_release(&m); \ + sched_yield(); \ + MPIDI_CH4I_Thread_mutex_acquire(&m); \ + } \ + } while (0) + +#define MPIDI_CH4I_THREAD_CS_ENTER_ALLGRAN MPIDI_CH4I_THREAD_CS_ENTER_POBJ +#define MPIDI_CH4I_THREAD_CS_EXIT_ALLGRAN MPIDI_CH4I_THREAD_CS_EXIT_POBJ +#define MPIDI_CH4I_THREAD_CS_TRY_ALLGRAN MPIDI_CH4I_THREAD_CS_TRY_POBJ +#define MPIDI_CH4I_THREAD_CS_YIELD_ALLGRAN MPIDI_CH4I_THREAD_CS_YIELD_POBJ + +/* GLOBAL locks are all NO-OPs */ +#define MPIDI_CH4I_THREAD_CS_ENTER_GLOBAL(mutex) +#define MPIDI_CH4I_THREAD_CS_EXIT_GLOBAL(mutex) +#define MPIDI_CH4I_THREAD_CS_TRY_GLOBAL(mutex) +#define MPIDI_CH4I_THREAD_CS_YIELD_GLOBAL(mutex) + +#else + +#error "Ticket locks are only supported in Global or Per-Object Granularity" + +#endif /* MPICH_THREAD_GRANULARITY == MPICH_THREAD_GRANULARITY__GLOBAL */ + +#define MPIDI_CH4I_THREAD_CS_ENTER(name, mutex) MPIDI_CH4I_THREAD_CS_ENTER_##name(mutex) +#define MPIDI_CH4I_THREAD_CS_EXIT(name, mutex) MPIDI_CH4I_THREAD_CS_EXIT_##name(mutex) +#define MPIDI_CH4I_THREAD_CS_TRY(name, mutex) MPIDI_CH4I_THREAD_CS_TRY_##name(mutex) +#define MPIDI_CH4I_THREAD_CS_YIELD(name, mutex) MPIDI_CH4I_THREAD_CS_YIELD_##name(mutex) + +#endif /* MPID_TICKETLOCK_H_INCLUDED */ diff --git a/src/mpid/ch4/include/mpidch4.h b/src/mpid/ch4/include/mpidch4.h new file mode 100644 index 0000000..8110d81 --- /dev/null +++ b/src/mpid/ch4/include/mpidch4.h @@ -0,0 +1,435 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef MPIDCH4_H_INCLUDED +#define MPIDCH4_H_INCLUDED + +#define __CH4_INLINE__ __attribute__((always_inline)) static inline + +/* We need to define the static inlines right away to avoid + * any implicit prototype generation and subsequent warnings + * This allows us to make ADI up calls from within a direct + * netmod. + */ +#define MPIDI_CH4I_API(rc,fcnname,...) \ + __CH4_INLINE__ rc MPIDI_##fcnname(__VA_ARGS__) \ + __attribute__((always_inline)) + +MPIDI_CH4I_API(int, Init, int *, char ***, int, int *, int *, int *); +MPIDI_CH4I_API(int, InitCompleted, void); +MPIDI_CH4I_API(int, Abort, MPIR_Comm *, int, int, const char *); +MPIDI_CH4I_API(int, Cancel_recv, MPIR_Request *); +MPIDI_CH4I_API(int, Cancel_send, MPIR_Request *); +MPIDI_CH4I_API(int, Comm_disconnect, MPIR_Comm *); +MPIDI_CH4I_API(int, Comm_spawn_multiple, int, char *[], char **[], const int[], MPIR_Info *[], int, + MPIR_Comm *, MPIR_Comm **, int[]); +MPIDI_CH4I_API(int, Comm_failure_get_acked, MPIR_Comm *, MPIR_Group **); +MPIDI_CH4I_API(int, Comm_get_all_failed_procs, MPIR_Comm *, MPIR_Group **, int); +MPIDI_CH4I_API(int, Comm_revoke, MPIR_Comm *, int); +MPIDI_CH4I_API(int, Comm_failure_ack, MPIR_Comm *); +MPIDI_CH4I_API(int, Comm_AS_enabled, MPIR_Comm *); +MPIDI_CH4I_API(int, Comm_get_lpid, MPIR_Comm *, int, int *, MPL_bool); +MPIDI_CH4I_API(int, Finalize, void); +MPIDI_CH4I_API(int, Get_universe_size, int *); +MPIDI_CH4I_API(int, Get_processor_name, char *, int, int *); +MPIDI_CH4I_API(int, Iprobe, int, int, MPIR_Comm *, int, int *, MPI_Status *); +MPIDI_CH4I_API(int, Irecv, void *, int, MPI_Datatype, int, int, MPIR_Comm *, int, MPIR_Request **); +MPIDI_CH4I_API(int, Isend, const void *, int, MPI_Datatype, int, int, MPIR_Comm *, int, + MPIR_Request **); +MPIDI_CH4I_API(int, Issend, const void *, int, MPI_Datatype, int, int, MPIR_Comm *, int, + MPIR_Request **); +MPIDI_CH4I_API(int, Mrecv, void *, int, MPI_Datatype, MPIR_Request *, MPI_Status *); +MPIDI_CH4I_API(int, Imrecv, void *, int, MPI_Datatype, MPIR_Request *, MPIR_Request **); +MPIDI_CH4I_API(int, Open_port, MPIR_Info *, char *); +MPIDI_CH4I_API(int, Close_port, const char *); +MPIDI_CH4I_API(int, Comm_accept, const char *, MPIR_Info *, int, MPIR_Comm *, MPIR_Comm **); +MPIDI_CH4I_API(int, Comm_connect, const char *, MPIR_Info *, int, MPIR_Comm *, MPIR_Comm **); +MPIDI_CH4I_API(int, Probe, int, int, MPIR_Comm *, int, MPI_Status *); +MPIDI_CH4I_API(int, Mprobe, int, int, MPIR_Comm *, int, MPIR_Request **, MPI_Status *); +MPIDI_CH4I_API(int, Improbe, int, int, MPIR_Comm *, int, int *, MPIR_Request **, MPI_Status *); +MPIDI_CH4I_API(int, Progress_test, void); +MPIDI_CH4I_API(int, Progress_poke, void); +MPIDI_CH4I_API(void, Progress_start, MPID_Progress_state *); +MPIDI_CH4I_API(void, Progress_end, MPID_Progress_state *); +MPIDI_CH4I_API(int, Progress_wait, MPID_Progress_state *); +MPIDI_CH4I_API(int, Progress_register, int (*progress_fn) (int *), int *id); +MPIDI_CH4I_API(int, Progress_deregister, int id); +MPIDI_CH4I_API(int, Progress_activate, int id); +MPIDI_CH4I_API(int, Progress_deactivate, int id); +MPIDI_CH4I_API(int, Recv, void *, int, MPI_Datatype, int, int, MPIR_Comm *, int, MPI_Status *, + MPIR_Request **); +MPIDI_CH4I_API(int, Recv_init, void *, int, MPI_Datatype, int, int, MPIR_Comm *, int, + MPIR_Request **); +MPIDI_CH4I_API(void, Request_set_completed, MPIR_Request *); +MPIDI_CH4I_API(int, Request_complete, MPIR_Request *); +MPIDI_CH4I_API(int, Request_is_anysource, MPIR_Request *); +MPIDI_CH4I_API(int, Send, const void *, int, MPI_Datatype, int, int, MPIR_Comm *, int, + MPIR_Request **); +MPIDI_CH4I_API(int, Ssend, const void *, int, MPI_Datatype, int, int, MPIR_Comm *, int, + MPIR_Request **); +MPIDI_CH4I_API(int, Rsend, const void *, int, MPI_Datatype, int, int, MPIR_Comm *, int, + MPIR_Request **); +MPIDI_CH4I_API(int, Irsend, const void *, int, MPI_Datatype, int, int, MPIR_Comm *, int, + MPIR_Request **); +MPIDI_CH4I_API(int, Send_init, const void *, int, MPI_Datatype, int, int, MPIR_Comm *, int, + MPIR_Request **); +MPIDI_CH4I_API(int, Ssend_init, const void *, int, MPI_Datatype, int, int, MPIR_Comm *, int, + MPIR_Request **); +MPIDI_CH4I_API(int, Bsend_init, const void *, int, MPI_Datatype, int, int, MPIR_Comm *, int, + MPIR_Request **); +MPIDI_CH4I_API(int, Rsend_init, const void *, int, MPI_Datatype, int, int, MPIR_Comm *, int, + MPIR_Request **); +MPIDI_CH4I_API(int, Startall, int, MPIR_Request *[]); +MPIDI_CH4I_API(int, GPID_Get, MPIR_Comm *, int, MPIR_Gpid *); +MPIDI_CH4I_API(int, Accumulate, const void *, int, MPI_Datatype, int, MPI_Aint, int, MPI_Datatype, + MPI_Op, MPIR_Win *); +MPIDI_CH4I_API(int, Win_create, void *, MPI_Aint, int, MPIR_Info *, MPIR_Comm *, MPIR_Win **); +MPIDI_CH4I_API(int, Win_fence, int, MPIR_Win *); +MPIDI_CH4I_API(int, Win_free, MPIR_Win **); +MPIDI_CH4I_API(int, Get, void *, int, MPI_Datatype, int, MPI_Aint, int, MPI_Datatype, MPIR_Win *); +MPIDI_CH4I_API(int, Win_get_info, MPIR_Win *, MPIR_Info **); +MPIDI_CH4I_API(int, Win_lock, int, int, int, MPIR_Win *); +MPIDI_CH4I_API(int, Win_unlock, int, MPIR_Win *); +MPIDI_CH4I_API(int, Win_start, MPIR_Group *, int, MPIR_Win *); +MPIDI_CH4I_API(int, Win_complete, MPIR_Win *); +MPIDI_CH4I_API(int, Win_post, MPIR_Group *, int, MPIR_Win *); +MPIDI_CH4I_API(int, Win_wait, MPIR_Win *); +MPIDI_CH4I_API(int, Win_test, MPIR_Win *, int *); +MPIDI_CH4I_API(int, Put, const void *, int, MPI_Datatype, int, MPI_Aint, int, MPI_Datatype, + MPIR_Win *); +MPIDI_CH4I_API(int, Win_set_info, MPIR_Win *, MPIR_Info *); +MPIDI_CH4I_API(int, Comm_reenable_anysource, MPIR_Comm *, MPIR_Group **); +MPIDI_CH4I_API(int, Comm_remote_group_failed, MPIR_Comm *, MPIR_Group **); +MPIDI_CH4I_API(int, Comm_group_failed, MPIR_Comm *, MPIR_Group **); +MPIDI_CH4I_API(int, Win_attach, MPIR_Win *, void *, MPI_Aint); +MPIDI_CH4I_API(int, Win_allocate_shared, MPI_Aint, int, MPIR_Info *, MPIR_Comm *, void **, + MPIR_Win **); +MPIDI_CH4I_API(int, Rput, const void *, int, MPI_Datatype, int, MPI_Aint, int, MPI_Datatype, + MPIR_Win *, MPIR_Request **); +MPIDI_CH4I_API(int, Win_flush_local, int, MPIR_Win *); +MPIDI_CH4I_API(int, Win_detach, MPIR_Win *, const void *); +MPIDI_CH4I_API(int, Compare_and_swap, const void *, const void *, void *, MPI_Datatype, int, + MPI_Aint, MPIR_Win *); +MPIDI_CH4I_API(int, Raccumulate, const void *, int, MPI_Datatype, int, MPI_Aint, int, MPI_Datatype, + MPI_Op, MPIR_Win *, MPIR_Request **); +MPIDI_CH4I_API(int, Rget_accumulate, const void *, int, MPI_Datatype, void *, int, MPI_Datatype, + int, MPI_Aint, int, MPI_Datatype, MPI_Op, MPIR_Win *, MPIR_Request **); +MPIDI_CH4I_API(int, Fetch_and_op, const void *, void *, MPI_Datatype, int, MPI_Aint, MPI_Op, + MPIR_Win *); +MPIDI_CH4I_API(int, Win_shared_query, MPIR_Win *, int, MPI_Aint *, int *, void *); +MPIDI_CH4I_API(int, Win_allocate, MPI_Aint, int, MPIR_Info *, MPIR_Comm *, void *, MPIR_Win **); +MPIDI_CH4I_API(int, Win_flush, int, MPIR_Win *); +MPIDI_CH4I_API(int, Win_flush_local_all, MPIR_Win *); +MPIDI_CH4I_API(int, Win_unlock_all, MPIR_Win *); +MPIDI_CH4I_API(int, Win_create_dynamic, MPIR_Info *, MPIR_Comm *, MPIR_Win **); +MPIDI_CH4I_API(int, Rget, void *, int, MPI_Datatype, int, MPI_Aint, int, MPI_Datatype, MPIR_Win *, + MPIR_Request **); +MPIDI_CH4I_API(int, Win_sync, MPIR_Win *); +MPIDI_CH4I_API(int, Win_flush_all, MPIR_Win *); +MPIDI_CH4I_API(int, Get_accumulate, const void *, int, MPI_Datatype, void *, int, MPI_Datatype, int, + MPI_Aint, int, MPI_Datatype, MPI_Op, MPIR_Win *); +MPIDI_CH4I_API(int, Win_lock_all, int, MPIR_Win *); +MPIDI_CH4I_API(void *, Alloc_mem, size_t, MPIR_Info *); +MPIDI_CH4I_API(int, Free_mem, void *); +MPIDI_CH4I_API(int, Get_node_id, MPIR_Comm *, int rank, MPID_Node_id_t *); +MPIDI_CH4I_API(int, Get_max_node_id, MPIR_Comm *, MPID_Node_id_t *); +MPIDI_CH4I_API(int, Request_is_pending_failure, MPIR_Request *); +MPIDI_CH4I_API(MPI_Aint, Aint_add, MPI_Aint, MPI_Aint); +MPIDI_CH4I_API(MPI_Aint, Aint_diff, MPI_Aint, MPI_Aint); +MPIDI_CH4I_API(int, GPID_GetAllInComm, MPIR_Comm *, int, MPIR_Gpid[], int *); +MPIDI_CH4I_API(int, GPID_ToLpidArray, int, MPIR_Gpid[], int[]); +MPIDI_CH4I_API(int, Create_intercomm_from_lpids, MPIR_Comm *, int, const int[]); +MPIDI_CH4I_API(int, Comm_create, MPIR_Comm *); +MPIDI_CH4I_API(int, Comm_destroy, MPIR_Comm *); +MPIDI_CH4I_API(int, Barrier, MPIR_Comm *, MPIR_Errflag_t *); +MPIDI_CH4I_API(int, Bcast, void *, int, MPI_Datatype, int, MPIR_Comm *, MPIR_Errflag_t *); +MPIDI_CH4I_API(int, Allreduce, const void *, void *, int, MPI_Datatype, MPI_Op, MPIR_Comm *, + MPIR_Errflag_t *); +MPIDI_CH4I_API(int, Allgather, const void *, int, MPI_Datatype, void *, int, MPI_Datatype, + MPIR_Comm *, MPIR_Errflag_t *); +MPIDI_CH4I_API(int, Allgatherv, const void *, int, MPI_Datatype, void *, const int *, const int *, + MPI_Datatype, MPIR_Comm *, MPIR_Errflag_t *); +MPIDI_CH4I_API(int, Scatter, const void *, int, MPI_Datatype, void *, int, MPI_Datatype, int, + MPIR_Comm *, MPIR_Errflag_t *); +MPIDI_CH4I_API(int, Scatterv, const void *, const int *, const int *, MPI_Datatype, void *, int, + MPI_Datatype, int, MPIR_Comm *, MPIR_Errflag_t *); +MPIDI_CH4I_API(int, Gather, const void *, int, MPI_Datatype, void *, int, MPI_Datatype, int, + MPIR_Comm *, MPIR_Errflag_t *); +MPIDI_CH4I_API(int, Gatherv, const void *, int, MPI_Datatype, void *, const int *, const int *, + MPI_Datatype, int, MPIR_Comm *, MPIR_Errflag_t *); +MPIDI_CH4I_API(int, Alltoall, const void *, int, MPI_Datatype, void *, int, MPI_Datatype, + MPIR_Comm *, MPIR_Errflag_t *); +MPIDI_CH4I_API(int, Alltoallv, const void *, const int *, const int *, MPI_Datatype, void *, + const int *, const int *, MPI_Datatype, MPIR_Comm *, MPIR_Errflag_t *); +MPIDI_CH4I_API(int, Alltoallw, const void *, const int[], const int[], const MPI_Datatype[], void *, + const int[], const int[], const MPI_Datatype[], MPIR_Comm *, MPIR_Errflag_t *); +MPIDI_CH4I_API(int, Reduce, const void *, void *, int, MPI_Datatype, MPI_Op, int, MPIR_Comm *, + MPIR_Errflag_t *); +MPIDI_CH4I_API(int, Reduce_scatter, const void *, void *, const int[], MPI_Datatype, MPI_Op, + MPIR_Comm *, MPIR_Errflag_t *); +MPIDI_CH4I_API(int, Reduce_scatter_block, const void *, void *, int, MPI_Datatype, MPI_Op, + MPIR_Comm *, MPIR_Errflag_t *); +MPIDI_CH4I_API(int, Scan, const void *, void *, int, MPI_Datatype, MPI_Op, MPIR_Comm *, + MPIR_Errflag_t *); +MPIDI_CH4I_API(int, Exscan, const void *, void *, int, MPI_Datatype, MPI_Op, MPIR_Comm *, + MPIR_Errflag_t *); +MPIDI_CH4I_API(int, Neighbor_allgather, const void *, int, MPI_Datatype, void *, int, MPI_Datatype, + MPIR_Comm *); +MPIDI_CH4I_API(int, Neighbor_allgatherv, const void *, int, MPI_Datatype, void *, const int[], + const int[], MPI_Datatype, MPIR_Comm *); +MPIDI_CH4I_API(int, Neighbor_alltoallv, const void *, const int[], const int[], MPI_Datatype, + void *, const int[], const int[], MPI_Datatype, MPIR_Comm *); +MPIDI_CH4I_API(int, Neighbor_alltoallw, const void *, const int[], const MPI_Aint[], + const MPI_Datatype[], void *, const int[], const MPI_Aint[], const MPI_Datatype[], + MPIR_Comm *); +MPIDI_CH4I_API(int, Neighbor_alltoall, const void *, int, MPI_Datatype, void *, int, MPI_Datatype, + MPIR_Comm *); +MPIDI_CH4I_API(int, Ineighbor_allgather, const void *, int, MPI_Datatype, void *, int, MPI_Datatype, + MPIR_Comm *, MPI_Request *); +MPIDI_CH4I_API(int, Ineighbor_allgatherv, const void *, int, MPI_Datatype, void *, const int[], + const int[], MPI_Datatype, MPIR_Comm *, MPI_Request *); +MPIDI_CH4I_API(int, Ineighbor_alltoall, const void *, int, MPI_Datatype, void *, int, MPI_Datatype, + MPIR_Comm *, MPI_Request *); +MPIDI_CH4I_API(int, Ineighbor_alltoallv, const void *, const int[], const int[], MPI_Datatype, + void *, const int[], const int[], MPI_Datatype, MPIR_Comm *, MPI_Request *); +MPIDI_CH4I_API(int, Ineighbor_alltoallw, const void *, const int[], const MPI_Aint[], + const MPI_Datatype[], void *, const int[], const MPI_Aint[], const MPI_Datatype[], + MPIR_Comm *, MPI_Request *); +MPIDI_CH4I_API(int, Ibarrier, MPIR_Comm *, MPI_Request *); +MPIDI_CH4I_API(int, Ibcast, void *, int, MPI_Datatype, int, MPIR_Comm *, MPI_Request *); +MPIDI_CH4I_API(int, Iallgather, const void *, int, MPI_Datatype, void *, int, MPI_Datatype, + MPIR_Comm *, MPI_Request *); +MPIDI_CH4I_API(int, Iallgatherv, const void *, int, MPI_Datatype, void *, const int *, const int *, + MPI_Datatype, MPIR_Comm *, MPI_Request *); +MPIDI_CH4I_API(int, Iallreduce, const void *, void *, int, MPI_Datatype, MPI_Op, MPIR_Comm *, + MPI_Request *); +MPIDI_CH4I_API(int, Ialltoall, const void *, int, MPI_Datatype, void *, int, MPI_Datatype, + MPIR_Comm *, MPI_Request *); +MPIDI_CH4I_API(int, Ialltoallv, const void *, const int[], const int[], MPI_Datatype, void *, + const int[], const int[], MPI_Datatype, MPIR_Comm *, MPI_Request *); +MPIDI_CH4I_API(int, Ialltoallw, const void *, const int[], const int[], const MPI_Datatype[], + void *, const int[], const int[], const MPI_Datatype[], MPIR_Comm *, MPI_Request *); +MPIDI_CH4I_API(int, Iexscan, const void *, void *, int, MPI_Datatype, MPI_Op, MPIR_Comm *, + MPI_Request *); +MPIDI_CH4I_API(int, Igather, const void *, int, MPI_Datatype, void *, int, MPI_Datatype, int, + MPIR_Comm *, MPI_Request *); +MPIDI_CH4I_API(int, Igatherv, const void *, int, MPI_Datatype, void *, const int *, const int *, + MPI_Datatype, int, MPIR_Comm *, MPI_Request *); +MPIDI_CH4I_API(int, Ireduce_scatter_block, const void *, void *, int, MPI_Datatype, MPI_Op, + MPIR_Comm *, MPI_Request *); +MPIDI_CH4I_API(int, Ireduce_scatter, const void *, void *, const int[], MPI_Datatype, MPI_Op, + MPIR_Comm *, MPI_Request *); +MPIDI_CH4I_API(int, Ireduce, const void *, void *, int, MPI_Datatype, MPI_Op, int, MPIR_Comm *, + MPI_Request *); +MPIDI_CH4I_API(int, Iscan, const void *, void *, int, MPI_Datatype, MPI_Op, MPIR_Comm *, + MPI_Request *); +MPIDI_CH4I_API(int, Iscatter, const void *, int, MPI_Datatype, void *, int, MPI_Datatype, int, + MPIR_Comm *, MPI_Request *); +MPIDI_CH4I_API(int, Iscatterv, const void *, const int *, const int *, MPI_Datatype, void *, int, + MPI_Datatype, int, MPIR_Comm *, MPI_Request *); + +/* This function is not exposed to the upper layers but functions in a way + * similar to the functions above. Other CH4-level functions should call this + * function to query locality. This function will determine whether to call the + * netmod or CH4U locality functions. */ +__CH4_INLINE__ int MPIDI_CH4_rank_is_local(int rank, MPIR_Comm * comm); + +/* Include netmod prototypes */ +#include <netmod.h> +#ifdef MPIDI_BUILD_CH4_SHM +#include "shm.h" +#endif + +/* Declare request functions here so netmods can refer to + them in the NETMOD_DIRECT mode */ +#include "ch4_request.h" + +/* Include netmod and shm implementations */ +/* Prototypes are split from impl to avoid */ +/* circular dependencies */ +#include <netmod_impl.h> +#ifdef MPIDI_BUILD_CH4_SHM +#include "shm_impl.h" +#endif + +#include "ch4_init.h" +#include "ch4_probe.h" +#include "ch4_send.h" +#include "ch4_recv.h" +#include "ch4_comm.h" +#include "ch4_win.h" +#include "ch4_rma.h" +#include "ch4_progress.h" +#include "ch4_spawn.h" +#include "ch4_proc.h" +#include "ch4_coll.h" + +#define MPID_Abort MPIDI_Abort +#define MPID_Accumulate MPIDI_Accumulate +#define MPID_Alloc_mem MPIDI_Alloc_mem +#define MPID_Bsend_init MPIDI_Bsend_init +#define MPID_Cancel_recv MPIDI_Cancel_recv +#define MPID_Cancel_send MPIDI_Cancel_send +#define MPID_Close_port MPIDI_Close_port +#define MPID_Comm_accept MPIDI_Comm_accept +#define MPID_Comm_connect MPIDI_Comm_connect +#define MPID_Comm_disconnect MPIDI_Comm_disconnect +#define MPID_Comm_group_failed MPIDI_Comm_group_failed +#define MPID_Comm_reenable_anysource MPIDI_Comm_reenable_anysource +#define MPID_Comm_remote_group_failed MPIDI_Comm_remote_group_failed +#define MPID_Comm_spawn_multiple MPIDI_Comm_spawn_multiple +#define MPID_Comm_failure_get_acked MPIDI_Comm_failure_get_acked +#define MPID_Comm_get_all_failed_procs MPIDI_Comm_get_all_failed_procs +#define MPID_Comm_revoke MPIDI_Comm_revoke +#define MPID_Comm_failure_ack MPIDI_Comm_failure_ack +#define MPID_Comm_AS_enabled MPIDI_Comm_AS_enabled +#define MPID_Comm_get_lpid MPIDI_Comm_get_lpid +#define MPID_Compare_and_swap MPIDI_Compare_and_swap +#define MPID_Fetch_and_op MPIDI_Fetch_and_op +#define MPID_Finalize MPIDI_Finalize +#define MPID_Free_mem MPIDI_Free_mem +#define MPID_GPID_Get MPIDI_GPID_Get +#define MPID_Get MPIDI_Get +#define MPID_Get_accumulate MPIDI_Get_accumulate +#define MPID_Get_processor_name MPIDI_Get_processor_name +#define MPID_Get_universe_size MPIDI_Get_universe_size +#define MPID_Improbe MPIDI_Improbe +#define MPID_Imrecv MPIDI_Imrecv +#define MPID_Init MPIDI_Init +#define MPID_InitCompleted MPIDI_InitCompleted +#define MPID_Iprobe MPIDI_Iprobe +#define MPID_Irecv MPIDI_Irecv +#define MPID_Irsend MPIDI_Irsend +#define MPID_Isend MPIDI_Isend +#define MPID_Issend MPIDI_Issend +#define MPID_Mprobe MPIDI_Mprobe +#define MPID_Mrecv MPIDI_Mrecv +#define MPID_Open_port MPIDI_Open_port +#define MPID_Probe MPIDI_Probe +#define MPID_Progress_end MPIDI_Progress_end +#define MPID_Progress_poke MPIDI_Progress_poke +#define MPID_Progress_start MPIDI_Progress_start +#define MPID_Progress_test MPIDI_Progress_test +#define MPID_Progress_wait MPIDI_Progress_wait +#define MPID_Progress_register MPIDI_Progress_register +#define MPID_Progress_deregister MPIDI_Progress_deregister +#define MPID_Progress_activate MPIDI_Progress_activate +#define MPID_Progress_deactivate MPIDI_Progress_deactivate +#define MPID_Put MPIDI_Put +#define MPID_Raccumulate MPIDI_Raccumulate +#define MPID_Recv MPIDI_Recv +#define MPID_Recv_init MPIDI_Recv_init +#define MPID_Request_release MPIDI_Request_release +#define MPID_Request_complete MPIDI_Request_complete +#define MPID_Request_is_anysource MPIDI_Request_is_anysource +#define MPID_Request_set_completed MPIDI_Request_set_completed +#define MPID_Rget MPIDI_Rget +#define MPID_Rget_accumulate MPIDI_Rget_accumulate +#define MPID_Rput MPIDI_Rput +#define MPID_Rsend MPIDI_Rsend +#define MPID_Rsend_init MPIDI_Rsend_init +#define MPID_Send MPIDI_Send +#define MPID_Send_init MPIDI_Send_init +#define MPID_Ssend MPIDI_Ssend +#define MPID_Ssend_init MPIDI_Ssend_init +#define MPID_Startall MPIDI_Startall +#define MPID_Win_allocate MPIDI_Win_allocate +#define MPID_Win_allocate_shared MPIDI_Win_allocate_shared +#define MPID_Win_attach MPIDI_Win_attach +#define MPID_Win_complete MPIDI_Win_complete +#define MPID_Win_create MPIDI_Win_create +#define MPID_Win_create_dynamic MPIDI_Win_create_dynamic +#define MPID_Win_detach MPIDI_Win_detach +#define MPID_Win_fence MPIDI_Win_fence +#define MPID_Win_flush MPIDI_Win_flush +#define MPID_Win_flush_all MPIDI_Win_flush_all +#define MPID_Win_flush_local MPIDI_Win_flush_local +#define MPID_Win_flush_local_all MPIDI_Win_flush_local_all +#define MPID_Win_free MPIDI_Win_free +#define MPID_Win_get_info MPIDI_Win_get_info +#define MPID_Win_lock MPIDI_Win_lock +#define MPID_Win_lock_all MPIDI_Win_lock_all +#define MPID_Win_post MPIDI_Win_post +#define MPID_Win_set_info MPIDI_Win_set_info +#define MPID_Win_shared_query MPIDI_Win_shared_query +#define MPID_Win_start MPIDI_Win_start +#define MPID_Win_sync MPIDI_Win_sync +#define MPID_Win_test MPIDI_Win_test +#define MPID_Win_unlock MPIDI_Win_unlock +#define MPID_Win_unlock_all MPIDI_Win_unlock_all +#define MPID_Win_wait MPIDI_Win_wait +#define MPID_Get_node_id MPIDI_Get_node_id +#define MPID_Get_max_node_id MPIDI_Get_max_node_id +#define MPID_Request_is_pending_failure MPIDI_Request_is_pending_failure +#define MPID_Aint_add MPIDI_Aint_add +#define MPID_Aint_diff MPIDI_Aint_diff +#define MPID_GPID_GetAllInComm MPIDI_GPID_GetAllInComm +#define MPID_GPID_ToLpidArray MPIDI_GPID_ToLpidArray +#define MPID_Create_intercomm_from_lpids MPIDI_Create_intercomm_from_lpids +/* Variables */ +#define MPID_Comm_create MPIDI_Comm_create +#define MPID_Comm_destroy MPIDI_Comm_destroy +#define MPID_Barrier MPIDI_Barrier +#define MPID_Bcast MPIDI_Bcast +#define MPID_Allreduce MPIDI_Allreduce +#define MPID_Allgather MPIDI_Allgather +#define MPID_Allgatherv MPIDI_Allgatherv +#define MPID_Scatter MPIDI_Scatter +#define MPID_Scatterv MPIDI_Scatterv +#define MPID_Gather MPIDI_Gather +#define MPID_Gatherv MPIDI_Gatherv +#define MPID_Alltoall MPIDI_Alltoall +#define MPID_Alltoallv MPIDI_Alltoallv +#define MPID_Alltoallw MPIDI_Alltoallw +#define MPID_Reduce MPIDI_Reduce +#define MPID_Reduce_scatter MPIDI_Reduce_scatter +#define MPID_Reduce_scatter_block MPIDI_Reduce_scatter_block +#define MPID_Scan MPIDI_Scan +#define MPID_Exscan MPIDI_Exscan +#define MPID_Neighbor_allgather MPIDI_Neighbor_allgather +#define MPID_Neighbor_allgatherv MPIDI_Neighbor_allgatherv +#define MPID_Neighbor_alltoallv MPIDI_Neighbor_alltoallv +#define MPID_Neighbor_alltoallw MPIDI_Neighbor_alltoallw +#define MPID_Neighbor_alltoall MPIDI_Neighbor_alltoall +#define MPID_Ineighbor_allgather MPIDI_Ineighbor_allgather +#define MPID_Ineighbor_allgatherv MPIDI_Ineighbor_allgatherv +#define MPID_Ineighbor_alltoall MPIDI_Ineighbor_alltoall +#define MPID_Ineighbor_alltoallv MPIDI_Ineighbor_alltoallv +#define MPID_Ineighbor_alltoallw MPIDI_Ineighbor_alltoallw +#define MPID_Ibarrier MPIDI_Ibarrier +#define MPID_Ibcast MPIDI_Ibcast +#define MPID_Iallgather MPIDI_Iallgather +#define MPID_Iallgatherv MPIDI_Iallgatherv +#define MPID_Iallreduce MPIDI_Iallreduce +#define MPID_Ialltoall MPIDI_Ialltoall +#define MPID_Ialltoallv MPIDI_Ialltoallv +#define MPID_Ialltoallw MPIDI_Ialltoallw +#define MPID_Iexscan MPIDI_Iexscan +#define MPID_Igather MPIDI_Igather +#define MPID_Igatherv MPIDI_Igatherv +#define MPID_Ireduce_scatter_block MPIDI_Ireduce_scatter_block +#define MPID_Ireduce_scatter MPIDI_Ireduce_scatter +#define MPID_Ireduce MPIDI_Ireduce +#define MPID_Iscan MPIDI_Iscan +#define MPID_Iscatter MPIDI_Iscatter +#define MPID_Iscatterv MPIDI_Iscatterv + +#define MPIDI_MAX_NETMOD_STRING_LEN 64 +extern int MPIDI_num_netmods; +#if defined(MPL_USE_DBG_LOGGING) +extern MPL_dbg_class MPIDI_CH4_DBG_GENERAL; +extern MPL_dbg_class MPIDI_CH4_DBG_MAP; +extern MPL_dbg_class MPIDI_CH4_DBG_MEMORY; +#endif /* MPL_USE_DBG_LOGGING */ + + + + +#endif /* MPIDCH4_H_INCLUDED */ diff --git a/src/mpid/ch4/include/mpidch4r.h b/src/mpid/ch4/include/mpidch4r.h new file mode 100644 index 0000000..e70f11e --- /dev/null +++ b/src/mpid/ch4/include/mpidch4r.h @@ -0,0 +1,26 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef MPIDCH4R_H_INCLUDED +#define MPIDCH4R_H_INCLUDED + +#include "ch4r_symheap.h" +#include "ch4r_recvq.h" +#include "ch4r_proc.h" +#include "ch4r_init.h" +#include "ch4r_probe.h" +#include "ch4r_recv.h" +#include "ch4r_rma.h" +#include "ch4r_send.h" +#include "ch4r_win.h" +#include "ch4r_buf.h" +#include "ch4r_request.h" + +#endif /* MPIDCH4R_H_INCLUDED */ diff --git a/src/mpid/ch4/include/mpidimpl.h b/src/mpid/ch4/include/mpidimpl.h new file mode 100644 index 0000000..e32d90e --- /dev/null +++ b/src/mpid/ch4/include/mpidimpl.h @@ -0,0 +1,30 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef MPIDIMPL_H_INCLUDED +#define MPIDIMPL_H_INCLUDED + +#include "mpichconf.h" +#include <stdio.h> + +#if defined(HAVE_ASSERT_H) +#include <assert.h> +#endif + +#define MPICH_SKIP_MPICXX +#include "mpiimpl.h" + +#if !defined(MPICH_MPIDPRE_H_INCLUDED) +#include "mpidpre.h" +#endif + +#include "mpidch4.h" + +#endif /* MPIDIMPL_H_INCLUDED */ diff --git a/src/mpid/ch4/include/mpidpost.h b/src/mpid/ch4/include/mpidpost.h new file mode 100644 index 0000000..40edeb7 --- /dev/null +++ b/src/mpid/ch4/include/mpidpost.h @@ -0,0 +1,30 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef MPIDPOST_H_INCLUDED +#define MPIDPOST_H_INCLUDED + +#include "mpidu_datatype.h" +#include "mpidch4.h" + +__ALWAYS_INLINE__ void MPID_Request_init(MPIR_Request * req) +{ + MPIDI_CH4U_REQUEST(req, req) = NULL; +#ifdef MPIDI_BUILD_CH4_SHM + MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(req) = NULL; +#endif +} + +__ALWAYS_INLINE__ void MPID_Request_finalize(MPIR_Request * req) +{ + return; +} + +#endif /* MPIDPOST_H_INCLUDED */ diff --git a/src/mpid/ch4/include/mpidpre.h b/src/mpid/ch4/include/mpidpre.h new file mode 100644 index 0000000..6fdd3f8 --- /dev/null +++ b/src/mpid/ch4/include/mpidpre.h @@ -0,0 +1,465 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef MPIDPRE_H_INCLUDED +#define MPIDPRE_H_INCLUDED + +#if defined(HAVE_SYS_TYPES_H) +#include <sys/types.h> +#endif + +#include "mpidu_dataloop.h" +#include "mpid_thread.h" +#include "mpid_sched.h" +#include "mpid_timers_fallback.h" +#include "netmodpre.h" +#include "shmpre.h" +#include "mpl_uthash.h" + +typedef struct { + union { + MPIDI_NM_DT_DECL} netmod; +} MPIDI_Devdt_t; +#define MPID_DEV_DATATYPE_DECL MPIDI_Devdt_t dev; +#include "mpid_datatype_fallback.h" + +typedef int MPID_Progress_state; +#define HAVE_GPID_ROUTINES + +#define __ALWAYS_INLINE__ __attribute__((always_inline)) static inline + +#define CH4_COMPILE_TIME_ASSERT(expr_) \ + do { switch(0) { case 0: case (expr_): default: break; } } while (0) + +/* Forward declaration of MPIR_Win so that we can refer to it in this file */ +struct MPIR_Win; +typedef struct MPIR_Win MPIR_Win; + +typedef enum { + MPIDI_PTYPE_RECV, + MPIDI_PTYPE_SEND, + MPIDI_PTYPE_BSEND, + MPIDI_PTYPE_SSEND +} MPIDI_ptype; + +#define MPIDI_CH4U_REQ_BUSY (0x1) +#define MPIDI_CH4U_REQ_PEER_SSEND (0x1 << 1) +#define MPIDI_CH4U_REQ_UNEXPECTED (0x1 << 2) +#define MPIDI_CH4U_REQ_UNEXP_DQUED (0x1 << 3) +#define MPIDI_CH4U_REQ_UNEXP_CLAIMED (0x1 << 4) +#define MPIDI_CH4U_REQ_RCV_NON_CONTIG (0x1 << 5) +#define MPIDI_CH4U_REQ_MATCHED (0x1 << 6) +#define MPIDI_CH4U_REQ_LONG_RTS (0x1 << 7) + +#define MPIDI_PARENT_PORT_KVSKEY "PARENT_ROOT_PORT_NAME" +#define MPIDI_MAX_KVS_VALUE_LEN 4096 + +typedef struct MPIDI_CH4U_sreq_t { + /* persistent send fields */ +} MPIDI_CH4U_sreq_t; + +typedef struct MPIDI_CH4U_lreq_t { + /* Long send fields */ + const void *src_buf; + MPI_Count count; + MPI_Datatype datatype; + uint64_t msg_tag; +} MPIDI_CH4U_lreq_t; + +typedef struct MPIDI_CH4U_rreq_t { + /* mrecv fields */ + void *mrcv_buffer; + uint64_t mrcv_count; + MPI_Datatype mrcv_datatype; + + uint64_t ignore; + uint64_t peer_req_ptr; + uint64_t match_req; + uint64_t request; + + struct MPIDI_CH4U_rreq_t *prev, *next; +} MPIDI_CH4U_rreq_t; + +typedef struct MPIDI_CH4U_put_req_t { + MPIR_Win *win_ptr; + uint64_t preq_ptr; + void *dt_iov; + void *origin_addr; + int origin_count; + MPI_Datatype origin_datatype; + int n_iov; +} MPIDI_CH4U_put_req_t; + +typedef struct MPIDI_CH4U_get_req_t { + MPIR_Win *win_ptr; + uint64_t greq_ptr; + uint64_t addr; + MPI_Datatype datatype; + int count; + int n_iov; + void *dt_iov; +} MPIDI_CH4U_get_req_t; + +typedef struct MPIDI_CH4U_cswap_req_t { + MPIR_Win *win_ptr; + uint64_t creq_ptr; + uint64_t addr; + MPI_Datatype datatype; + void *data; + void *result_addr; +} MPIDI_CH4U_cswap_req_t; + +typedef struct MPIDI_CH4U_acc_req_t { + MPIR_Win *win_ptr; + uint64_t req_ptr; + MPI_Datatype origin_datatype; + MPI_Datatype target_datatype; + int origin_count; + int target_count; + int n_iov; + void *target_addr; + void *dt_iov; + void *data; + size_t data_sz; + MPI_Op op; + void *result_addr; + int result_count; + int do_get; + void *origin_addr; + MPI_Datatype result_datatype; +} MPIDI_CH4U_acc_req_t; + +typedef struct MPIDI_CH4U_req_ext_t { + union { + MPIDI_CH4U_sreq_t sreq; + MPIDI_CH4U_lreq_t lreq; + MPIDI_CH4U_rreq_t rreq; + MPIDI_CH4U_put_req_t preq; + MPIDI_CH4U_get_req_t greq; + MPIDI_CH4U_cswap_req_t creq; + MPIDI_CH4U_acc_req_t areq; + }; + + struct iovec *iov; + void *cmpl_handler_fn; + uint64_t seq_no; + uint64_t request; + uint64_t status; + struct MPIDI_CH4U_req_ext_t *next, *prev; + +} MPIDI_CH4U_req_ext_t; + +typedef struct MPIDI_CH4U_req_t { + union { + MPIDI_NM_REQUEST_AM_DECL} netmod_am; + MPIDI_CH4U_req_ext_t *req; + MPIDI_ptype p_type; + void *buffer; + uint64_t count; + uint64_t tag; + int src_rank; + MPI_Datatype datatype; +} MPIDI_CH4U_req_t; + +typedef struct { +#ifdef MPIDI_CH4_EXCLUSIVE_SHM + int is_local; +#endif + /* Anysource handling. Netmod and shm specific requests are cross + * referenced. This must be present all of the time to avoid lots of extra + * ifdefs in the code. */ +#ifdef MPIDI_BUILD_CH4_SHM + struct MPIR_Request *anysource_partner_request; +#endif + + union { + /* The first fields are used by the CH4U apis */ + MPIDI_CH4U_req_t ch4u; + + /* Used by the netmod direct apis */ + union { + MPIDI_NM_REQUEST_DECL} netmod; + + union { + MPIDI_SHM_REQUEST_DECL} shm; + } ch4; +} MPIDI_Devreq_t; +#define MPIDI_REQUEST_HDR_SIZE offsetof(struct MPIR_Request, dev.ch4.netmod) +#define MPIDI_REQUEST_CH4U_HDR_SIZE offsetof(struct MPIR_Request, dev.ch4.netmod_am) +#define MPIDI_CH4I_REQUEST(req,field) (((req)->dev).field) +#define MPIDI_CH4U_REQUEST(req,field) (((req)->dev.ch4.ch4u).field) + +#ifdef MPIDI_BUILD_CH4_SHM +#define MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(req) (((req)->dev).anysource_partner_request) +#else +#define MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(req) NULL +#endif + +__ALWAYS_INLINE__ void MPID_Request_init(struct MPIR_Request *req); + +__ALWAYS_INLINE__ void MPID_Request_finalize(struct MPIR_Request *req); + +typedef struct MPIDI_CH4U_win_shared_info { + uint32_t disp_unit; + size_t size; +} __attribute__ ((packed)) MPIDI_CH4U_win_shared_info_t; + +#define MPIDI_CH4I_ACCU_ORDER_RAR (1) +#define MPIDI_CH4I_ACCU_ORDER_RAW (1 << 1) +#define MPIDI_CH4I_ACCU_ORDER_WAR (1 << 2) +#define MPIDI_CH4I_ACCU_ORDER_WAW (1 << 3) + +typedef enum { + MPIDI_CH4I_ACCU_SAME_OP, + MPIDI_CH4I_ACCU_SAME_OP_NO_OP +} MPIDI_CH4U_win_info_accumulate_ops; + +typedef struct MPIDI_CH4U_win_info_args_t { + int no_locks; + int same_size; + int accumulate_ordering; + int alloc_shared_noncontig; + MPIDI_CH4U_win_info_accumulate_ops accumulate_ops; +} MPIDI_CH4U_win_info_args_t; + +struct MPIDI_CH4U_win_lock { + struct MPIDI_CH4U_win_lock *next; + int rank; + uint16_t mtype; + uint16_t type; +}; + +struct MPIDI_CH4U_win_queue { + struct MPIDI_CH4U_win_lock *head; + struct MPIDI_CH4U_win_lock *tail; +}; + +typedef struct MPIDI_CH4U_win_lock_info { + unsigned peer; + int lock_type; + struct MPIR_Win *win; + volatile unsigned done; +} MPIDI_CH4U_win_lock_info; + +typedef struct MPIDI_CH4U_win_sync_lock { + struct { + volatile unsigned locked; + volatile unsigned allLocked; + } remote; + struct { + struct MPIDI_CH4U_win_queue requested; + int type; + unsigned count; + } local; +} MPIDI_CH4U_win_sync_lock; + +typedef struct MPIDI_CH4U_win_sync_pscw { + struct MPIR_Group *group; + volatile unsigned count; +} MPIDI_CH4U_win_sync_pscw; + +typedef struct MPIDI_CH4U_win_sync_t { + volatile int origin_epoch_type; + volatile int target_epoch_type; + MPIDI_CH4U_win_sync_pscw sc, pw; + MPIDI_CH4U_win_sync_lock lock; +} MPIDI_CH4U_win_sync_t; + +typedef struct MPIDI_CH4U_win_t { + uint64_t win_id; + void *mmap_addr; + int64_t mmap_sz; + OPA_int_t outstanding_ops; + MPI_Aint *sizes; + MPIDI_CH4U_win_lock_info *lockQ; + MPIDI_CH4U_win_sync_t sync; + MPIDI_CH4U_win_info_args_t info_args; + MPIDI_CH4U_win_shared_info_t *shared_table; + MPL_UT_hash_handle hash_handle; +} MPIDI_CH4U_win_t; + +typedef struct { + MPIDI_CH4U_win_t ch4u; + union { + MPIDI_NM_WIN_DECL} netmod; +} MPIDI_Devwin_t; + +#define MPIDI_CH4U_WIN(win,field) (((win)->dev.ch4u).field) +#define MPIDI_CH4U_WINFO(win,rank) (MPIDI_CH4U_win_info_t*) &(MPIDI_CH4U_WIN(win, info_table)[rank]) + +typedef unsigned MPIDII_locality_t; + +typedef struct MPIDI_CH4U_comm_t { + MPIDI_CH4U_rreq_t *posted_list; + MPIDI_CH4U_rreq_t *unexp_list; + uint32_t window_instance; +} MPIDI_CH4U_comm_t; + +#define MPIDII_CALC_STRIDE(rank, stride, blocksize, offset) \ + ((rank) / (blocksize) * ((stride) - (blocksize)) + (rank) + (offset)) + +#define MPIDII_CALC_STRIDE_SIMPLE(rank, stride, offset) \ + ((rank) * (stride) + (offset)) + +typedef enum { + MPIDII_RANK_MAP_DIRECT, + MPIDII_RANK_MAP_DIRECT_INTRA, + MPIDII_RANK_MAP_OFFSET, + MPIDII_RANK_MAP_OFFSET_INTRA, + MPIDII_RANK_MAP_STRIDE, + MPIDII_RANK_MAP_STRIDE_INTRA, + MPIDII_RANK_MAP_STRIDE_BLOCK, + MPIDII_RANK_MAP_STRIDE_BLOCK_INTRA, + MPIDII_RANK_MAP_LUT, + MPIDII_RANK_MAP_LUT_INTRA, + MPIDII_RANK_MAP_MLUT, + MPIDII_RANK_MAP_NONE +} MPIDII_rank_map_mode; + +typedef int MPIDII_lpid_t; +typedef struct { + int avtid; + int lpid; +} MPIDII_gpid_t; + +typedef struct { + MPIR_OBJECT_HEADER; + MPIDII_lpid_t lpid[0]; +} MPIDII_rank_map_lut_t; + +typedef struct { + MPIR_OBJECT_HEADER; + MPIDII_gpid_t gpid[0]; +} MPIDII_rank_map_mlut_t; + +typedef struct { + MPIDII_rank_map_mode mode; + int avtid; + int size; + + union { + int offset; + struct { + int offset; + int stride; + int blocksize; + } stride; + } reg; + + union { + struct { + MPIDII_rank_map_lut_t *t; + MPIDII_lpid_t *lpid; + } lut; + struct { + MPIDII_rank_map_mlut_t *t; + MPIDII_gpid_t *gpid; + } mlut; + } irreg; +} MPIDII_rank_map_t; + +typedef struct MPIDI_Devcomm_t { + struct { + /* The first fields are used by the CH4U apis */ + MPIDI_CH4U_comm_t ch4u; + + /* Used by the netmod direct apis */ + union { + MPIDI_NM_COMM_DECL} netmod; + + union { + MPIDI_SHM_COMM_DECL} shm; + + MPIDII_rank_map_t map; + MPIDII_rank_map_t local_map; + } ch4; +} MPIDI_Devcomm_t; +#define MPIDI_CH4U_COMM(comm,field) ((comm)->dev.ch4.ch4u).field +#define MPIDII_COMM(comm,field) ((comm)->dev.ch4).field + + +#define MPID_USE_NODE_IDS +typedef uint16_t MPID_Node_id_t; + +typedef struct { + union { + MPIDI_NM_OP_DECL} netmod; +} MPIDI_Devop_t; + +typedef struct { + union { + MPIDI_NM_GPID_DECL} netmod; + MPID_Node_id_t node; +} MPIDI_Devgpid_t; + +#define MPID_DEV_REQUEST_DECL MPIDI_Devreq_t dev; +#define MPID_DEV_WIN_DECL MPIDI_Devwin_t dev; +#define MPID_DEV_COMM_DECL MPIDI_Devcomm_t dev; +#define MPID_DEV_OP_DECL MPIDI_Devop_t dev; +#define MPID_DEV_GPID_DECL MPIDI_Devgpid_t dev; + +#define MPIDII_GPID(gpid) (gpid)->dev + +typedef struct { + union { + MPIDI_NM_ADDR_DECL} netmod; +#ifdef MPIDI_BUILD_CH4_LOCALITY_INFO + MPIDII_locality_t is_local; +#endif +} MPIDII_av_entry_t; + +typedef struct { + MPIR_OBJECT_HEADER; + int size; + MPIDII_av_entry_t table[0]; +} MPIDII_av_table_t; + +extern MPIDII_av_table_t **MPIDII_av_table; +extern MPIDII_av_table_t *MPIDII_av_table0; + +#define MPIDIU_get_av_table(avtid) (MPIDII_av_table[(avtid)]) +#define MPIDIU_get_av(avtid, lpid) (MPIDII_av_table[(avtid)]->table[(lpid)]) + +#define MPIDIU_get_node_map(avtid) (MPIDI_CH4_Global.node_map[(avtid)]) + +#define MPID_Progress_register_hook(fn_, id_) MPID_Progress_register(fn_, id_) +#define MPID_Progress_deregister_hook(id_) MPID_Progress_deregister(id_) +#define MPID_Progress_activate_hook(id_) MPID_Progress_activate(id_) +#define MPID_Progress_deactivate_hook(id_) MPID_Progress_deactivate(id_) + +#define HAVE_DEV_COMM_HOOK +#define MPID_Dev_comm_create_hook(a) (MPID_Comm_create(a)) +#define MPID_Dev_comm_destroy_hook(a) (MPID_Comm_destroy(a)) + +#define MPID_Dev_datatype_commit_hook MPIDI_NM_datatype_commit +#define MPID_Dev_datatype_destroy_hook MPIDI_NM_datatype_destroy + +#define MPID_Dev_op_commit_hook MPIDI_NM_op_commit +#define MPID_Dev_op_destroy_hook MPIDI_NM_op_destroy + +/* operation for (avtid, lpid) to/from "lpid64" */ +/* hard code limit on number of live comm worlds. This should be fixed by future + * LUPID patch */ +#define MPIDIU_AVTID_BITS (8) +#define MPIDIU_LPID_BITS (24) +#define MPIDIU_LPID_MASK (0x00FFFFFFU) +#define MPIDIU_AVTID_MASK (0xFF000000U) +#define MPIDIU_NEW_AVT_MARK (0x80000000U) +#define MPIDIU_LPID_CREATE(avtid, lpid) (((avtid) << MPIDIU_LPID_BITS) | (lpid)) +#define MPIDIU_LPID_GET_AVTID(lpid) ((((lpid) & MPIDIU_AVTID_MASK) >> MPIDIU_LPID_BITS)) +#define MPIDIU_LPID_GET_LPID(lpid) (((lpid) & MPIDIU_LPID_MASK)) +#define MPIDIU_LPID_SET_NEW_AVT_MARK(lpid) ((lpid) |= MPIDIU_NEW_AVT_MARK) +#define MPIDIU_LPID_CLEAR_NEW_AVT_MARK(lpid) ((lpid) &= (~MPIDIU_NEW_AVT_MARK)) +#define MPIDIU_LPID_IS_NEW_AVT(lpid) ((lpid) & MPIDIU_NEW_AVT_MARK) + + +#include "mpidu_pre.h" + +#endif /* MPIDPRE_H_INCLUDED */ diff --git a/src/mpid/ch4/include/netmodpre.h.in b/src/mpid/ch4/include/netmodpre.h.in new file mode 100644 index 0000000..086714e --- /dev/null +++ b/src/mpid/ch4/include/netmodpre.h.in @@ -0,0 +1,26 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMODPRE_H_INCLUDED +#define NETMODPRE_H_INCLUDED + +@ch4_netmod_pre_include@ + +#define MPIDI_NM_REQUEST_AM_DECL @ch4_netmod_amrequest_decl@ +#define MPIDI_NM_REQUEST_DECL @ch4_netmod_request_decl@ + +#define MPIDI_NM_COMM_DECL @ch4_netmod_comm_decl@ +#define MPIDI_NM_DT_DECL @ch4_netmod_dt_decl@ +#define MPIDI_NM_WIN_DECL @ch4_netmod_win_decl@ +#define MPIDI_NM_GPID_DECL @ch4_netmod_gpid_decl@ +#define MPIDI_NM_ADDR_DECL @ch4_netmod_addr_decl@ +#define MPIDI_NM_OP_DECL @ch4_netmod_op_decl@ + +#endif /* NETMODPRE_H_INCLUDED */ diff --git a/src/mpid/ch4/include/shmpre.h.in b/src/mpid/ch4/include/shmpre.h.in new file mode 100644 index 0000000..c6bcbdb --- /dev/null +++ b/src/mpid/ch4/include/shmpre.h.in @@ -0,0 +1,19 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef SHMPRE_H_INCLUDED +#define SHMPRE_H_INCLUDED + +@ch4_shm_pre_include@ + +#define MPIDI_SHM_REQUEST_DECL @ch4_shm_request_decl@ +#define MPIDI_SHM_COMM_DECL @ch4_shm_comm_decl@ + +#endif /* SHMPRE_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/Makefile.mk b/src/mpid/ch4/netmod/Makefile.mk new file mode 100644 index 0000000..2b875a6 --- /dev/null +++ b/src/mpid/ch4/netmod/Makefile.mk @@ -0,0 +1,22 @@ +## -*- Mode: Makefile; -*- +## vim: set ft=automake : +## +## (C) 2016 by Argonne National Laboratory. +## (C) 2014 by Mellanox Technologies, Inc. +## See COPYRIGHT in top-level directory. +## +## Portions of this code were written by Intel Corporation. +## Copyright (C) 2011-2016 Intel Corporation. Intel provides this material +## to Argonne National Laboratory subject to Software Grant and Corporate +## Contributor License Agreement dated February 8, 2012. +## + +AM_CPPFLAGS += -I$(top_srcdir)/src/mpid/ch4/netmod/include + +noinst_HEADERS += src/mpid/ch4/netmod/include/netmod.h +noinst_HEADERS += src/mpid/ch4/netmod/include/netmod_impl.h + +include $(top_srcdir)/src/mpid/ch4/netmod/ofi/Makefile.mk +include $(top_srcdir)/src/mpid/ch4/netmod/ucx/Makefile.mk +include $(top_srcdir)/src/mpid/ch4/netmod/portals4/Makefile.mk +include $(top_srcdir)/src/mpid/ch4/netmod/stubnm/Makefile.mk diff --git a/src/mpid/ch4/netmod/include/netmod.h b/src/mpid/ch4/netmod/include/netmod.h new file mode 100644 index 0000000..bd33b0c --- /dev/null +++ b/src/mpid/ch4/netmod/include/netmod.h @@ -0,0 +1,1133 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +/* ch4 netmod functions */ +#ifndef NETMOD_PROTOTYPES_H_INCLUDED +#define NETMOD_PROTOTYPES_H_INCLUDED + +#include <mpidimpl.h> + +#define MPIDI_MAX_NETMOD_STRING_LEN 64 + +typedef int (*MPIDI_NM_am_completion_handler_fn) (MPIR_Request * req); +typedef int (*MPIDI_NM_am_origin_handler_fn) (MPIR_Request * req); + +/* Callback function setup by handler register function */ +/* for short cases, output arguments are NULL */ +typedef int (*MPIDI_NM_am_target_handler_fn) + (void *am_hdr, void **data, /* data should be iovs if *is_contig is false */ + size_t * data_sz, int *is_contig, MPIDI_NM_am_completion_handler_fn * cmpl_handler_fn, /* completion handler */ + MPIR_Request ** req); /* if allocated, need pointer to completion function */ + +typedef int (*MPIDI_NM_init_t) (int rank, int size, int appnum, int *tag_ub, MPIR_Comm * comm_world, + MPIR_Comm * comm_self, int spawned, int num_contexts, + void **netmod_contexts); +typedef int (*MPIDI_NM_finalize_t) (void); +typedef int (*MPIDI_NM_progress_t) (void *netmod_context, int blocking); +typedef int (*MPIDI_NM_reg_hdr_handler_t) (int handler_id, + MPIDI_NM_am_origin_handler_fn origin_handler_fn, + MPIDI_NM_am_target_handler_fn target_handler_fn); +typedef int (*MPIDI_NM_comm_connect_t) (const char *port_name, MPIR_Info * info, int root, + MPIR_Comm * comm, MPIR_Comm ** newcomm_ptr); +typedef int (*MPIDI_NM_comm_disconnect_t) (MPIR_Comm * comm_ptr); +typedef int (*MPIDI_NM_open_port_t) (MPIR_Info * info_ptr, char *port_name); +typedef int (*MPIDI_NM_close_port_t) (const char *port_name); +typedef int (*MPIDI_NM_comm_accept_t) (const char *port_name, MPIR_Info * info, int root, + MPIR_Comm * comm, MPIR_Comm ** newcomm_ptr); +typedef int (*MPIDI_NM_send_am_hdr_t) (int rank, MPIR_Comm * comm, int handler_id, + const void *am_hdr, size_t am_hdr_sz, MPIR_Request * sreq, + void *netmod_context); +typedef int (*MPIDI_NM_inject_am_hdr_t) (int rank, MPIR_Comm * comm, int handler_id, + const void *am_hdr, size_t am_hdr_sz, + void *netmod_context); +typedef int (*MPIDI_NM_send_am_t) (int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, + size_t am_hdr_sz, const void *data, MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq, + void *netmod_context); +typedef int (*MPIDI_NM_send_amv_t) (int rank, MPIR_Comm * comm, int handler_id, + struct iovec * am_hdrs, size_t iov_len, const void *data, + MPI_Count count, MPI_Datatype datatype, MPIR_Request * sreq, + void *netmod_context); +typedef int (*MPIDI_NM_send_amv_hdr_t) (int rank, MPIR_Comm * comm, int handler_id, + struct iovec * am_hdrs, size_t iov_len, MPIR_Request * sreq, + void *netmod_context); +typedef int (*MPIDI_NM_send_am_hdr_reply_t) (MPIR_Context_id_t context_id, int src_rank, + int handler_id, const void *am_hdr, size_t am_hdr_sz, + MPIR_Request * sreq); +typedef int (*MPIDI_NM_inject_am_hdr_reply_t) (MPIR_Context_id_t context_id, int src_rank, + int handler_id, const void *am_hdr, + size_t am_hdr_sz); +typedef int (*MPIDI_NM_send_am_reply_t) (MPIR_Context_id_t context_id, int src_rank, int handler_id, + const void *am_hdr, size_t am_hdr_sz, const void *data, + MPI_Count count, MPI_Datatype datatype, + MPIR_Request * sreq); +typedef int (*MPIDI_NM_send_amv_reply_t) (MPIR_Context_id_t context_id, int src_rank, + int handler_id, struct iovec * am_hdr, size_t iov_len, + const void *data, MPI_Count count, MPI_Datatype datatype, + MPIR_Request * sreq); +typedef size_t(*MPIDI_NM_am_hdr_max_sz_t) (void); +typedef size_t(*MPIDI_NM_am_inject_max_sz_t) (void); +typedef int (*MPIDI_NM_am_recv_t) (MPIR_Request * req); +typedef int (*MPIDI_NM_comm_get_lpid_t) (MPIR_Comm * comm_ptr, int idx, int *lpid_ptr, + MPL_bool is_remote); +typedef int (*MPIDI_NM_gpid_get_t) (MPIR_Comm * comm_ptr, int rank, MPIR_Gpid * gpid); +typedef int (*MPIDI_NM_get_node_id_t) (MPIR_Comm * comm, int rank, MPID_Node_id_t * id_p); +typedef int (*MPIDI_NM_get_max_node_id_t) (MPIR_Comm * comm, MPID_Node_id_t * max_id_p); +typedef int (*MPIDI_NM_getallincomm_t) (MPIR_Comm * comm_ptr, int local_size, + MPIR_Gpid local_gpids[], int *singleAVT); +typedef int (*MPIDI_NM_gpid_tolpidarray_t) (int size, MPIR_Gpid gpid[], int lpid[]); +typedef int (*MPIDI_NM_create_intercomm_from_lpids_t) (MPIR_Comm * newcomm_ptr, int size, + const int lpids[]); +typedef int (*MPIDI_NM_comm_create_t) (MPIR_Comm * comm); +typedef int (*MPIDI_NM_comm_destroy_t) (MPIR_Comm * comm); +typedef void (*MPIDI_NM_am_request_init_t) (MPIR_Request * req); +typedef void (*MPIDI_NM_am_request_finalize_t) (MPIR_Request * req); +typedef int (*MPIDI_NM_send_t) (const void *buf, int count, MPI_Datatype datatype, int rank, + int tag, MPIR_Comm * comm, int context_offset, + MPIR_Request ** request); +typedef int (*MPIDI_NM_ssend_t) (const void *buf, int count, MPI_Datatype datatype, int rank, + int tag, MPIR_Comm * comm, int context_offset, + MPIR_Request ** request); +typedef int (*MPIDI_NM_startall_t) (int count, MPIR_Request * requests[]); +typedef int (*MPIDI_NM_send_init_t) (const void *buf, int count, MPI_Datatype datatype, int rank, + int tag, MPIR_Comm * comm, int context_offset, + MPIR_Request ** request); +typedef int (*MPIDI_NM_ssend_init_t) (const void *buf, int count, MPI_Datatype datatype, int rank, + int tag, MPIR_Comm * comm, int context_offset, + MPIR_Request ** request); +typedef int (*MPIDI_NM_rsend_init_t) (const void *buf, int count, MPI_Datatype datatype, int rank, + int tag, MPIR_Comm * comm, int context_offset, + MPIR_Request ** request); +typedef int (*MPIDI_NM_bsend_init_t) (const void *buf, int count, MPI_Datatype datatype, int rank, + int tag, MPIR_Comm * comm, int context_offset, + MPIR_Request ** request); +typedef int (*MPIDI_NM_isend_t) (const void *buf, int count, MPI_Datatype datatype, int rank, + int tag, MPIR_Comm * comm, int context_offset, + MPIR_Request ** request); +typedef int (*MPIDI_NM_issend_t) (const void *buf, int count, MPI_Datatype datatype, int rank, + int tag, MPIR_Comm * comm, int context_offset, + MPIR_Request ** request); +typedef int (*MPIDI_NM_cancel_send_t) (MPIR_Request * sreq); +typedef int (*MPIDI_NM_recv_init_t) (void *buf, int count, MPI_Datatype datatype, int rank, int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request); +typedef int (*MPIDI_NM_recv_t) (void *buf, int count, MPI_Datatype datatype, int rank, int tag, + MPIR_Comm * comm, int context_offset, MPI_Status * status, + MPIR_Request ** request); +typedef int (*MPIDI_NM_irecv_t) (void *buf, int count, MPI_Datatype datatype, int rank, int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request); +typedef int (*MPIDI_NM_imrecv_t) (void *buf, int count, MPI_Datatype datatype, + MPIR_Request * message, MPIR_Request ** rreqp); +typedef int (*MPIDI_NM_cancel_recv_t) (MPIR_Request * rreq); +typedef void *(*MPIDI_NM_alloc_mem_t) (size_t size, MPIR_Info * info_ptr); +typedef int (*MPIDI_NM_free_mem_t) (void *ptr); +typedef int (*MPIDI_NM_improbe_t) (int source, int tag, MPIR_Comm * comm, int context_offset, + int *flag, MPIR_Request ** message, MPI_Status * status); +typedef int (*MPIDI_NM_iprobe_t) (int source, int tag, MPIR_Comm * comm, int context_offset, + int *flag, MPI_Status * status); +typedef int (*MPIDI_NM_win_set_info_t) (MPIR_Win * win, MPIR_Info * info); +typedef int (*MPIDI_NM_win_shared_query_t) (MPIR_Win * win, int rank, MPI_Aint * size, + int *disp_unit, void *baseptr); +typedef int (*MPIDI_NM_put_t) (const void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, int target_rank, MPI_Aint target_disp, + int target_count, MPI_Datatype target_datatype, MPIR_Win * win); +typedef int (*MPIDI_NM_win_start_t) (MPIR_Group * group, int assert, MPIR_Win * win); +typedef int (*MPIDI_NM_win_complete_t) (MPIR_Win * win); +typedef int (*MPIDI_NM_win_post_t) (MPIR_Group * group, int assert, MPIR_Win * win); +typedef int (*MPIDI_NM_win_wait_t) (MPIR_Win * win); +typedef int (*MPIDI_NM_win_test_t) (MPIR_Win * win, int *flag); +typedef int (*MPIDI_NM_win_lock_t) (int lock_type, int rank, int assert, MPIR_Win * win); +typedef int (*MPIDI_NM_win_unlock_t) (int rank, MPIR_Win * win); +typedef int (*MPIDI_NM_win_get_info_t) (MPIR_Win * win, MPIR_Info ** info_p_p); +typedef int (*MPIDI_NM_get_t) (void *origin_addr, int origin_count, MPI_Datatype origin_datatype, + int target_rank, MPI_Aint target_disp, int target_count, + MPI_Datatype target_datatype, MPIR_Win * win); +typedef int (*MPIDI_NM_win_free_t) (MPIR_Win ** win_ptr); +typedef int (*MPIDI_NM_win_fence_t) (int assert, MPIR_Win * win); +typedef int (*MPIDI_NM_win_create_t) (void *base, MPI_Aint length, int disp_unit, MPIR_Info * info, + MPIR_Comm * comm_ptr, MPIR_Win ** win_ptr); +typedef int (*MPIDI_NM_accumulate_t) (const void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, int target_rank, + MPI_Aint target_disp, int target_count, + MPI_Datatype target_datatype, MPI_Op op, MPIR_Win * win); +typedef int (*MPIDI_NM_win_attach_t) (MPIR_Win * win, void *base, MPI_Aint size); +typedef int (*MPIDI_NM_win_allocate_shared_t) (MPI_Aint size, int disp_unit, MPIR_Info * info_ptr, + MPIR_Comm * comm_ptr, void **base_ptr, + MPIR_Win ** win_ptr); +typedef int (*MPIDI_NM_rput_t) (const void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, int target_rank, MPI_Aint target_disp, + int target_count, MPI_Datatype target_datatype, MPIR_Win * win, + MPIR_Request ** request); +typedef int (*MPIDI_NM_win_flush_local_t) (int rank, MPIR_Win * win); +typedef int (*MPIDI_NM_win_detach_t) (MPIR_Win * win, const void *base); +typedef int (*MPIDI_NM_compare_and_swap_t) (const void *origin_addr, const void *compare_addr, + void *result_addr, MPI_Datatype datatype, + int target_rank, MPI_Aint target_disp, MPIR_Win * win); +typedef int (*MPIDI_NM_raccumulate_t) (const void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, int target_rank, + MPI_Aint target_disp, int target_count, + MPI_Datatype target_datatype, MPI_Op op, MPIR_Win * win, + MPIR_Request ** request); +typedef int (*MPIDI_NM_rget_accumulate_t) (const void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, void *result_addr, + int result_count, MPI_Datatype result_datatype, + int target_rank, MPI_Aint target_disp, int target_count, + MPI_Datatype target_datatype, MPI_Op op, MPIR_Win * win, + MPIR_Request ** request); +typedef int (*MPIDI_NM_fetch_and_op_t) (const void *origin_addr, void *result_addr, + MPI_Datatype datatype, int target_rank, + MPI_Aint target_disp, MPI_Op op, MPIR_Win * win); +typedef int (*MPIDI_NM_win_allocate_t) (MPI_Aint size, int disp_unit, MPIR_Info * info, + MPIR_Comm * comm, void *baseptr, MPIR_Win ** win); +typedef int (*MPIDI_NM_win_flush_t) (int rank, MPIR_Win * win); +typedef int (*MPIDI_NM_win_flush_local_all_t) (MPIR_Win * win); +typedef int (*MPIDI_NM_win_unlock_all_t) (MPIR_Win * win); +typedef int (*MPIDI_NM_win_create_dynamic_t) (MPIR_Info * info, MPIR_Comm * comm, MPIR_Win ** win); +typedef int (*MPIDI_NM_rget_t) (void *origin_addr, int origin_count, MPI_Datatype origin_datatype, + int target_rank, MPI_Aint target_disp, int target_count, + MPI_Datatype target_datatype, MPIR_Win * win, + MPIR_Request ** request); +typedef int (*MPIDI_NM_win_sync_t) (MPIR_Win * win); +typedef int (*MPIDI_NM_win_flush_all_t) (MPIR_Win * win); +typedef int (*MPIDI_NM_get_accumulate_t) (const void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, void *result_addr, + int result_count, MPI_Datatype result_datatype, + int target_rank, MPI_Aint target_disp, int target_count, + MPI_Datatype target_datatype, MPI_Op op, MPIR_Win * win); +typedef int (*MPIDI_NM_win_lock_all_t) (int assert, MPIR_Win * win); +typedef int (*MPIDI_NM_rank_is_local_t) (int target, MPIR_Comm * comm); +typedef int (*MPIDI_NM_barrier_t) (MPIR_Comm * comm, MPIR_Errflag_t * errflag); +typedef int (*MPIDI_NM_bcast_t) (void *buffer, int count, MPI_Datatype datatype, int root, + MPIR_Comm * comm, MPIR_Errflag_t * errflag); +typedef int (*MPIDI_NM_allreduce_t) (const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm, + MPIR_Errflag_t * errflag); +typedef int (*MPIDI_NM_allgather_t) (const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, MPIR_Errflag_t * errflag); +typedef int (*MPIDI_NM_allgatherv_t) (const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, const int *displs, + MPI_Datatype recvtype, MPIR_Comm * comm, + MPIR_Errflag_t * errflag); +typedef int (*MPIDI_NM_scatter_t) (const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, int root, + MPIR_Comm * comm, MPIR_Errflag_t * errflag); +typedef int (*MPIDI_NM_scatterv_t) (const void *sendbuf, const int *sendcounts, const int *displs, + MPI_Datatype sendtype, void *recvbuf, int recvcount, + MPI_Datatype recvtype, int root, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag); +typedef int (*MPIDI_NM_gather_t) (const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, int root, + MPIR_Comm * comm, MPIR_Errflag_t * errflag); +typedef int (*MPIDI_NM_gatherv_t) (const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, const int *displs, + MPI_Datatype recvtype, int root, MPIR_Comm * comm, + MPIR_Errflag_t * errflag); +typedef int (*MPIDI_NM_alltoall_t) (const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, MPIR_Errflag_t * errflag); +typedef int (*MPIDI_NM_alltoallv_t) (const void *sendbuf, const int *sendcounts, const int *sdispls, + MPI_Datatype sendtype, void *recvbuf, const int *recvcounts, + const int *rdispls, MPI_Datatype recvtype, MPIR_Comm * comm, + MPIR_Errflag_t * errflag); +typedef int (*MPIDI_NM_alltoallw_t) (const void *sendbuf, const int *sendcounts, const int *sdispls, + const MPI_Datatype sendtypes[], void *recvbuf, + const int *recvcounts, const int *rdispls, + const MPI_Datatype recvtypes[], MPIR_Comm * comm, + MPIR_Errflag_t * errflag); +typedef int (*MPIDI_NM_reduce_t) (const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, int root, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag); +typedef int (*MPIDI_NM_reduce_scatter_t) (const void *sendbuf, void *recvbuf, const int *recvcounts, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag); +typedef int (*MPIDI_NM_reduce_scatter_block_t) (const void *sendbuf, void *recvbuf, int recvcount, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag); +typedef int (*MPIDI_NM_scan_t) (const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm, + MPIR_Errflag_t * errflag); +typedef int (*MPIDI_NM_exscan_t) (const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm, + MPIR_Errflag_t * errflag); +typedef int (*MPIDI_NM_neighbor_allgather_t) (const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, int recvcount, + MPI_Datatype recvtype, MPIR_Comm * comm); +typedef int (*MPIDI_NM_neighbor_allgatherv_t) (const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, const int *displs, + MPI_Datatype recvtype, MPIR_Comm * comm); +typedef int (*MPIDI_NM_neighbor_alltoallv_t) (const void *sendbuf, const int *sendcounts, + const int *sdispls, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, + const int *rdispls, MPI_Datatype recvtype, + MPIR_Comm * comm); +typedef int (*MPIDI_NM_neighbor_alltoallw_t) (const void *sendbuf, const int *sendcounts, + const MPI_Aint * sdispls, + const MPI_Datatype * sendtypes, void *recvbuf, + const int *recvcounts, const MPI_Aint * rdispls, + const MPI_Datatype * recvtypes, MPIR_Comm * comm); +typedef int (*MPIDI_NM_neighbor_alltoall_t) (const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, int recvcount, + MPI_Datatype recvtype, MPIR_Comm * comm); +typedef int (*MPIDI_NM_ineighbor_allgather_t) (const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, int recvcount, + MPI_Datatype recvtype, MPIR_Comm * comm, + MPI_Request * req); +typedef int (*MPIDI_NM_ineighbor_allgatherv_t) (const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, const int *displs, + MPI_Datatype recvtype, MPIR_Comm * comm, + MPI_Request * req); +typedef int (*MPIDI_NM_ineighbor_alltoall_t) (const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, int recvcount, + MPI_Datatype recvtype, MPIR_Comm * comm, + MPI_Request * req); +typedef int (*MPIDI_NM_ineighbor_alltoallv_t) (const void *sendbuf, const int *sendcounts, + const int *sdispls, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, + const int *rdispls, MPI_Datatype recvtype, + MPIR_Comm * comm, MPI_Request * req); +typedef int (*MPIDI_NM_ineighbor_alltoallw_t) (const void *sendbuf, const int *sendcounts, + const MPI_Aint * sdispls, + const MPI_Datatype * sendtypes, void *recvbuf, + const int *recvcounts, const MPI_Aint * rdispls, + const MPI_Datatype * recvtypes, MPIR_Comm * comm, + MPI_Request * req); +typedef int (*MPIDI_NM_ibarrier_t) (MPIR_Comm * comm, MPI_Request * req); +typedef int (*MPIDI_NM_ibcast_t) (void *buffer, int count, MPI_Datatype datatype, int root, + MPIR_Comm * comm, MPI_Request * req); +typedef int (*MPIDI_NM_iallgather_t) (const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, MPI_Request * req); +typedef int (*MPIDI_NM_iallgatherv_t) (const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, const int *displs, + MPI_Datatype recvtype, MPIR_Comm * comm, MPI_Request * req); +typedef int (*MPIDI_NM_iallreduce_t) (const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm, + MPI_Request * req); +typedef int (*MPIDI_NM_ialltoall_t) (const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, MPI_Request * req); +typedef int (*MPIDI_NM_ialltoallv_t) (const void *sendbuf, const int *sendcounts, + const int *sdispls, MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, const int *rdispls, + MPI_Datatype recvtype, MPIR_Comm * comm, MPI_Request * req); +typedef int (*MPIDI_NM_ialltoallw_t) (const void *sendbuf, const int *sendcounts, + const int *sdispls, const MPI_Datatype sendtypes[], + void *recvbuf, const int *recvcounts, const int *rdispls, + const MPI_Datatype recvtypes[], MPIR_Comm * comm, + MPI_Request * req); +typedef int (*MPIDI_NM_iexscan_t) (const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm, + MPI_Request * req); +typedef int (*MPIDI_NM_igather_t) (const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, int root, + MPIR_Comm * comm, MPI_Request * req); +typedef int (*MPIDI_NM_igatherv_t) (const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, const int *displs, + MPI_Datatype recvtype, int root, MPIR_Comm * comm, + MPI_Request * req); +typedef int (*MPIDI_NM_ireduce_scatter_block_t) (const void *sendbuf, void *recvbuf, int recvcount, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm, + MPI_Request * req); +typedef int (*MPIDI_NM_ireduce_scatter_t) (const void *sendbuf, void *recvbuf, + const int *recvcounts, MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, MPI_Request * req); +typedef int (*MPIDI_NM_ireduce_t) (const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, int root, MPIR_Comm * comm_ptr, + MPI_Request * req); +typedef int (*MPIDI_NM_iscan_t) (const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm, + MPI_Request * req); +typedef int (*MPIDI_NM_iscatter_t) (const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, int root, + MPIR_Comm * comm, MPI_Request * req); +typedef int (*MPIDI_NM_iscatterv_t) (const void *sendbuf, const int *sendcounts, const int *displs, + MPI_Datatype sendtype, void *recvbuf, int recvcount, + MPI_Datatype recvtype, int root, MPIR_Comm * comm_ptr, + MPI_Request * req); +typedef void (*MPIDI_NM_datatype_commit_t) (MPIR_Datatype * datatype_p); +typedef void (*MPIDI_NM_datatype_dup_t) (MPIR_Datatype * old_datatype_p, + MPIR_Datatype * new_datatype_p); +typedef void (*MPIDI_NM_datatype_destroy_t) (MPIR_Datatype * datatype_p); +typedef void (*MPIDI_NM_op_commit_t) (MPIR_Op * op_p); +typedef void (*MPIDI_NM_op_destroy_t) (MPIR_Op * op_p); + +typedef struct MPIDI_NM_funcs { + MPIDI_NM_init_t init; + MPIDI_NM_finalize_t finalize; + MPIDI_NM_progress_t progress; + MPIDI_NM_comm_connect_t comm_connect; + MPIDI_NM_comm_disconnect_t comm_disconnect; + MPIDI_NM_open_port_t open_port; + MPIDI_NM_close_port_t close_port; + MPIDI_NM_comm_accept_t comm_accept; + /* Routines that handle addressing */ + MPIDI_NM_comm_get_lpid_t comm_get_lpid; + MPIDI_NM_gpid_get_t gpid_get; + MPIDI_NM_get_node_id_t get_node_id; + MPIDI_NM_get_max_node_id_t get_max_node_id; + MPIDI_NM_getallincomm_t getallincomm; + MPIDI_NM_gpid_tolpidarray_t gpid_tolpidarray; + MPIDI_NM_create_intercomm_from_lpids_t create_intercomm_from_lpids; + MPIDI_NM_comm_create_t comm_create; + MPIDI_NM_comm_destroy_t comm_destroy; + /* Request allocation routines */ + MPIDI_NM_am_request_init_t am_request_init; + MPIDI_NM_am_request_finalize_t am_request_finalize; + /* Active Message Routines */ + MPIDI_NM_reg_hdr_handler_t reg_hdr_handler; + MPIDI_NM_send_am_hdr_t send_am_hdr; + MPIDI_NM_inject_am_hdr_t inject_am_hdr; + MPIDI_NM_send_am_t send_am; + MPIDI_NM_send_amv_t send_amv; + MPIDI_NM_send_amv_hdr_t send_amv_hdr; + MPIDI_NM_send_am_hdr_reply_t send_am_hdr_reply; + MPIDI_NM_inject_am_hdr_reply_t inject_am_hdr_reply; + MPIDI_NM_send_am_reply_t send_am_reply; + MPIDI_NM_send_amv_reply_t send_amv_reply; + MPIDI_NM_am_hdr_max_sz_t am_hdr_max_sz; + MPIDI_NM_am_inject_max_sz_t am_inject_max_sz; + MPIDI_NM_am_recv_t am_recv; +} MPIDI_NM_funcs_t; + +typedef struct MPIDI_NM_native_funcs { + MPIDI_NM_send_t send; + MPIDI_NM_ssend_t ssend; + MPIDI_NM_startall_t startall; + MPIDI_NM_send_init_t send_init; + MPIDI_NM_ssend_init_t ssend_init; + MPIDI_NM_rsend_init_t rsend_init; + MPIDI_NM_bsend_init_t bsend_init; + MPIDI_NM_isend_t isend; + MPIDI_NM_issend_t issend; + MPIDI_NM_cancel_send_t cancel_send; + MPIDI_NM_recv_init_t recv_init; + MPIDI_NM_recv_t recv; + MPIDI_NM_irecv_t irecv; + MPIDI_NM_imrecv_t imrecv; + MPIDI_NM_cancel_recv_t cancel_recv; + MPIDI_NM_alloc_mem_t alloc_mem; + MPIDI_NM_free_mem_t free_mem; + MPIDI_NM_improbe_t improbe; + MPIDI_NM_iprobe_t iprobe; + MPIDI_NM_win_set_info_t win_set_info; + MPIDI_NM_win_shared_query_t win_shared_query; + MPIDI_NM_put_t put; + MPIDI_NM_win_start_t win_start; + MPIDI_NM_win_complete_t win_complete; + MPIDI_NM_win_post_t win_post; + MPIDI_NM_win_wait_t win_wait; + MPIDI_NM_win_test_t win_test; + MPIDI_NM_win_lock_t win_lock; + MPIDI_NM_win_unlock_t win_unlock; + MPIDI_NM_win_get_info_t win_get_info; + MPIDI_NM_get_t get; + MPIDI_NM_win_free_t win_free; + MPIDI_NM_win_fence_t win_fence; + MPIDI_NM_win_create_t win_create; + MPIDI_NM_accumulate_t accumulate; + MPIDI_NM_win_attach_t win_attach; + MPIDI_NM_win_allocate_shared_t win_allocate_shared; + MPIDI_NM_rput_t rput; + MPIDI_NM_win_flush_local_t win_flush_local; + MPIDI_NM_win_detach_t win_detach; + MPIDI_NM_compare_and_swap_t compare_and_swap; + MPIDI_NM_raccumulate_t raccumulate; + MPIDI_NM_rget_accumulate_t rget_accumulate; + MPIDI_NM_fetch_and_op_t fetch_and_op; + MPIDI_NM_win_allocate_t win_allocate; + MPIDI_NM_win_flush_t win_flush; + MPIDI_NM_win_flush_local_all_t win_flush_local_all; + MPIDI_NM_win_unlock_all_t win_unlock_all; + MPIDI_NM_win_create_dynamic_t win_create_dynamic; + MPIDI_NM_rget_t rget; + MPIDI_NM_win_sync_t win_sync; + MPIDI_NM_win_flush_all_t win_flush_all; + MPIDI_NM_get_accumulate_t get_accumulate; + MPIDI_NM_win_lock_all_t win_lock_all; + MPIDI_NM_rank_is_local_t rank_is_local; + /* Collectives */ + MPIDI_NM_barrier_t barrier; + MPIDI_NM_bcast_t bcast; + MPIDI_NM_allreduce_t allreduce; + MPIDI_NM_allgather_t allgather; + MPIDI_NM_allgatherv_t allgatherv; + MPIDI_NM_scatter_t scatter; + MPIDI_NM_scatterv_t scatterv; + MPIDI_NM_gather_t gather; + MPIDI_NM_gatherv_t gatherv; + MPIDI_NM_alltoall_t alltoall; + MPIDI_NM_alltoallv_t alltoallv; + MPIDI_NM_alltoallw_t alltoallw; + MPIDI_NM_reduce_t reduce; + MPIDI_NM_reduce_scatter_t reduce_scatter; + MPIDI_NM_reduce_scatter_block_t reduce_scatter_block; + MPIDI_NM_scan_t scan; + MPIDI_NM_exscan_t exscan; + MPIDI_NM_neighbor_allgather_t neighbor_allgather; + MPIDI_NM_neighbor_allgatherv_t neighbor_allgatherv; + MPIDI_NM_neighbor_alltoall_t neighbor_alltoall; + MPIDI_NM_neighbor_alltoallv_t neighbor_alltoallv; + MPIDI_NM_neighbor_alltoallw_t neighbor_alltoallw; + MPIDI_NM_ineighbor_allgather_t ineighbor_allgather; + MPIDI_NM_ineighbor_allgatherv_t ineighbor_allgatherv; + MPIDI_NM_ineighbor_alltoall_t ineighbor_alltoall; + MPIDI_NM_ineighbor_alltoallv_t ineighbor_alltoallv; + MPIDI_NM_ineighbor_alltoallw_t ineighbor_alltoallw; + MPIDI_NM_ibarrier_t ibarrier; + MPIDI_NM_ibcast_t ibcast; + MPIDI_NM_iallgather_t iallgather; + MPIDI_NM_iallgatherv_t iallgatherv; + MPIDI_NM_iallreduce_t iallreduce; + MPIDI_NM_ialltoall_t ialltoall; + MPIDI_NM_ialltoallv_t ialltoallv; + MPIDI_NM_ialltoallw_t ialltoallw; + MPIDI_NM_iexscan_t iexscan; + MPIDI_NM_igather_t igather; + MPIDI_NM_igatherv_t igatherv; + MPIDI_NM_ireduce_scatter_block_t ireduce_scatter_block; + MPIDI_NM_ireduce_scatter_t ireduce_scatter; + MPIDI_NM_ireduce_t ireduce; + MPIDI_NM_iscan_t iscan; + MPIDI_NM_iscatter_t iscatter; + MPIDI_NM_iscatterv_t iscatterv; + /* Datatype hooks */ + MPIDI_NM_datatype_commit_t datatype_commit; + MPIDI_NM_datatype_dup_t datatype_dup; + MPIDI_NM_datatype_destroy_t datatype_destroy; + /* Op hooks */ + MPIDI_NM_op_commit_t op_commit; + MPIDI_NM_op_destroy_t op_destroy; +} MPIDI_NM_native_funcs_t; + +extern MPIDI_NM_funcs_t *MPIDI_NM_funcs[]; +extern MPIDI_NM_funcs_t *MPIDI_NM_func; +extern MPIDI_NM_native_funcs_t *MPIDI_NM_native_funcs[]; +extern MPIDI_NM_native_funcs_t *MPIDI_NM_native_func; +extern int MPIDI_num_netmods; +extern char MPIDI_NM_strings[][MPIDI_MAX_NETMOD_STRING_LEN]; + +#ifndef MPIDI_NM_STATIC_INLINE_PREFIX +#define MPIDI_NM_STATIC_INLINE_PREFIX __attribute__((always_inline)) static inline +#endif + +#ifndef MPIDI_NM_STATIC_INLINE_SUFFIX +#define MPIDI_NM_STATIC_INLINE_SUFFIX __attribute__((always_inline)) +#endif + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_init(int rank, int size, int appnum, int *tag_ub, + MPIR_Comm * comm_world, MPIR_Comm * comm_self, + int spawned, int num_contexts, + void **netmod_contexts) + MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_finalize(void) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_progress(void *netmod_context, + int blocking) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_reg_hdr_handler(int handler_id, + MPIDI_NM_am_origin_handler_fn + origin_handler_fn, + MPIDI_NM_am_target_handler_fn + target_handler_fn) + MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_comm_connect(const char *port_name, MPIR_Info * info, + int root, MPIR_Comm * comm, + MPIR_Comm ** + newcomm_ptr) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_comm_disconnect(MPIR_Comm * + comm_ptr) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_open_port(MPIR_Info * info_ptr, + char *port_name) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_close_port(const char *port_name) + MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_comm_accept(const char *port_name, MPIR_Info * info, + int root, MPIR_Comm * comm, + MPIR_Comm ** + newcomm_ptr) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_send_am_hdr(int rank, MPIR_Comm * comm, int handler_id, + const void *am_hdr, size_t am_hdr_sz, + MPIR_Request * sreq, + void *netmod_context) + MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_inject_am_hdr(int rank, MPIR_Comm * comm, int handler_id, + const void *am_hdr, size_t am_hdr_sz, + void *netmod_context) + MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_send_am(int rank, MPIR_Comm * comm, int handler_id, + const void *am_hdr, size_t am_hdr_sz, + const void *data, MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq, + void *netmod_context) + MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_send_amv(int rank, MPIR_Comm * comm, int handler_id, + struct iovec *am_hdrs, size_t iov_len, + const void *data, MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq, + void *netmod_context) + MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_send_amv_hdr(int rank, MPIR_Comm * comm, int handler_id, + struct iovec *am_hdrs, size_t iov_len, + MPIR_Request * sreq, + void *netmod_context) + MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_send_am_hdr_reply(MPIR_Context_id_t context_id, + int src_rank, int handler_id, + const void *am_hdr, size_t am_hdr_sz, + MPIR_Request * + sreq) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_inject_am_hdr_reply(MPIR_Context_id_t context_id, + int src_rank, int handler_id, + const void *am_hdr, + size_t am_hdr_sz) + MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_send_am_reply(MPIR_Context_id_t context_id, int src_rank, + int handler_id, const void *am_hdr, + size_t am_hdr_sz, const void *data, + MPI_Count count, MPI_Datatype datatype, + MPIR_Request * + sreq) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_send_amv_reply(MPIR_Context_id_t context_id, + int src_rank, int handler_id, + struct iovec *am_hdr, size_t iov_len, + const void *data, MPI_Count count, + MPI_Datatype datatype, + MPIR_Request * + sreq) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX size_t MPIDI_NM_am_hdr_max_sz(void) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX size_t MPIDI_NM_am_inject_max_sz(void) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_am_recv(MPIR_Request * + req) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_comm_get_lpid(MPIR_Comm * comm_ptr, int idx, + int *lpid_ptr, + MPL_bool is_remote) + MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_gpid_get(MPIR_Comm * comm_ptr, int rank, + MPIR_Gpid * gpid) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_get_node_id(MPIR_Comm * comm, int rank, + MPID_Node_id_t * + id_p) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_get_max_node_id(MPIR_Comm * comm, + MPID_Node_id_t * + max_id_p) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_getallincomm(MPIR_Comm * comm_ptr, int local_size, + MPIR_Gpid local_gpids[], + int *singleAVT) + MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_gpid_tolpidarray(int size, MPIR_Gpid gpid[], + int lpid[]) + MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_create_intercomm_from_lpids(MPIR_Comm * newcomm_ptr, + int size, + const int lpids[]) + MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_comm_create(MPIR_Comm * + comm) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_comm_destroy(MPIR_Comm * + comm) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX void MPIDI_NM_am_request_init(MPIR_Request * + req) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX void MPIDI_NM_am_request_finalize(MPIR_Request * + req) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_send(const void *buf, int count, MPI_Datatype datatype, + int rank, int tag, MPIR_Comm * comm, + int context_offset, + MPIR_Request ** + request) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ssend(const void *buf, int count, MPI_Datatype datatype, + int rank, int tag, MPIR_Comm * comm, + int context_offset, + MPIR_Request ** + request) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_startall(int count, + MPIR_Request * + requests[]) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_send_init(const void *buf, int count, + MPI_Datatype datatype, int rank, int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** + request) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ssend_init(const void *buf, int count, + MPI_Datatype datatype, int rank, int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** + request) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_rsend_init(const void *buf, int count, + MPI_Datatype datatype, int rank, int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** + request) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_bsend_init(const void *buf, int count, + MPI_Datatype datatype, int rank, int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** + request) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_isend(const void *buf, int count, MPI_Datatype datatype, + int rank, int tag, MPIR_Comm * comm, + int context_offset, + MPIR_Request ** + request) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_issend(const void *buf, int count, MPI_Datatype datatype, + int rank, int tag, MPIR_Comm * comm, + int context_offset, + MPIR_Request ** + request) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_cancel_send(MPIR_Request * + sreq) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_recv_init(void *buf, int count, MPI_Datatype datatype, + int rank, int tag, MPIR_Comm * comm, + int context_offset, + MPIR_Request ** + request) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_recv(void *buf, int count, MPI_Datatype datatype, + int rank, int tag, MPIR_Comm * comm, + int context_offset, MPI_Status * status, + MPIR_Request ** + request) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_irecv(void *buf, int count, MPI_Datatype datatype, + int rank, int tag, MPIR_Comm * comm, + int context_offset, + MPIR_Request ** + request) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_imrecv(void *buf, int count, MPI_Datatype datatype, + MPIR_Request * message, + MPIR_Request ** + rreqp) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_cancel_recv(MPIR_Request * + rreq) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX void *MPIDI_NM_alloc_mem(size_t size, + MPIR_Info * + info_ptr) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_free_mem(void *ptr) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_improbe(int source, int tag, MPIR_Comm * comm, + int context_offset, int *flag, + MPIR_Request ** message, + MPI_Status * + status) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_iprobe(int source, int tag, MPIR_Comm * comm, + int context_offset, int *flag, + MPI_Status * + status) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_set_info(MPIR_Win * win, + MPIR_Info * + info) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_shared_query(MPIR_Win * win, int rank, + MPI_Aint * size, int *disp_unit, + void *baseptr) + MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_put(const void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, int target_rank, + MPI_Aint target_disp, int target_count, + MPI_Datatype target_datatype, + MPIR_Win * win) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_start(MPIR_Group * group, int assert, + MPIR_Win * win) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_complete(MPIR_Win * + win) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_post(MPIR_Group * group, int assert, + MPIR_Win * win) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_wait(MPIR_Win * win) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_test(MPIR_Win * win, + int *flag) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_lock(int lock_type, int rank, int assert, + MPIR_Win * win) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_unlock(int rank, + MPIR_Win * win) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_get_info(MPIR_Win * win, + MPIR_Info ** + info_p_p) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_get(void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, int target_rank, + MPI_Aint target_disp, int target_count, + MPI_Datatype target_datatype, + MPIR_Win * win) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_free(MPIR_Win ** + win_ptr) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_fence(int assert, + MPIR_Win * win) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_create(void *base, MPI_Aint length, int disp_unit, + MPIR_Info * info, MPIR_Comm * comm_ptr, + MPIR_Win ** + win_ptr) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_accumulate(const void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, int target_rank, + MPI_Aint target_disp, int target_count, + MPI_Datatype target_datatype, MPI_Op op, + MPIR_Win * win) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_attach(MPIR_Win * win, void *base, + MPI_Aint size) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_allocate_shared(MPI_Aint size, int disp_unit, + MPIR_Info * info_ptr, + MPIR_Comm * comm_ptr, + void **base_ptr, + MPIR_Win ** + win_ptr) + MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_rput(const void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, int target_rank, + MPI_Aint target_disp, int target_count, + MPI_Datatype target_datatype, MPIR_Win * win, + MPIR_Request ** + request) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_flush_local(int rank, + MPIR_Win * + win) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_detach(MPIR_Win * win, + const void *base) + MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_compare_and_swap(const void *origin_addr, + const void *compare_addr, + void *result_addr, + MPI_Datatype datatype, int target_rank, + MPI_Aint target_disp, + MPIR_Win * + win) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_raccumulate(const void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, + int target_rank, MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, MPI_Op op, + MPIR_Win * win, + MPIR_Request ** + request) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_rget_accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + void *result_addr, int result_count, + MPI_Datatype result_datatype, + int target_rank, MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, MPI_Op op, + MPIR_Win * win, + MPIR_Request ** + request) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_fetch_and_op(const void *origin_addr, void *result_addr, + MPI_Datatype datatype, int target_rank, + MPI_Aint target_disp, MPI_Op op, + MPIR_Win * + win) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_allocate(MPI_Aint size, int disp_unit, + MPIR_Info * info, MPIR_Comm * comm, + void *baseptr, + MPIR_Win ** + win) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_flush(int rank, + MPIR_Win * win) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_flush_local_all(MPIR_Win * + win) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_unlock_all(MPIR_Win * + win) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_create_dynamic(MPIR_Info * info, MPIR_Comm * comm, + MPIR_Win ** + win) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_rget(void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, int target_rank, + MPI_Aint target_disp, int target_count, + MPI_Datatype target_datatype, MPIR_Win * win, + MPIR_Request ** + request) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_sync(MPIR_Win * win) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_flush_all(MPIR_Win * + win) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_get_accumulate(const void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, + void *result_addr, int result_count, + MPI_Datatype result_datatype, + int target_rank, MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, MPI_Op op, + MPIR_Win * + win) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_lock_all(int assert, + MPIR_Win * + win) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_rank_is_local(int target, + MPIR_Comm * + comm) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_barrier(MPIR_Comm * comm, + MPIR_Errflag_t * + errflag) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_bcast(void *buffer, int count, MPI_Datatype datatype, + int root, MPIR_Comm * comm, + MPIR_Errflag_t * + errflag) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_allreduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, + MPIR_Errflag_t * + errflag) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_allgather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, + MPIR_Errflag_t * + errflag) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_allgatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, const int *displs, + MPI_Datatype recvtype, MPIR_Comm * comm, + MPIR_Errflag_t * + errflag) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_scatter(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, int root, + MPIR_Comm * comm, + MPIR_Errflag_t * + errflag) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_scatterv(const void *sendbuf, const int *sendcounts, + const int *displs, MPI_Datatype sendtype, + void *recvbuf, int recvcount, + MPI_Datatype recvtype, int root, + MPIR_Comm * comm_ptr, + MPIR_Errflag_t * + errflag) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_gather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, int root, + MPIR_Comm * comm, + MPIR_Errflag_t * + errflag) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_gatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, const int *displs, + MPI_Datatype recvtype, int root, + MPIR_Comm * comm, + MPIR_Errflag_t * + errflag) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_alltoall(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, + MPIR_Errflag_t * + errflag) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_alltoallv(const void *sendbuf, const int *sendcounts, + const int *sdispls, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, + const int *rdispls, MPI_Datatype recvtype, + MPIR_Comm * comm, + MPIR_Errflag_t * + errflag) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_alltoallw(const void *sendbuf, const int *sendcounts, + const int *sdispls, + const MPI_Datatype sendtypes[], void *recvbuf, + const int *recvcounts, const int *rdispls, + const MPI_Datatype recvtypes[], + MPIR_Comm * comm, + MPIR_Errflag_t * + errflag) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_reduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, int root, + MPIR_Comm * comm_ptr, + MPIR_Errflag_t * + errflag) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_reduce_scatter(const void *sendbuf, void *recvbuf, + const int *recvcounts, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm_ptr, + MPIR_Errflag_t * + errflag) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_reduce_scatter_block(const void *sendbuf, void *recvbuf, + int recvcount, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm_ptr, + MPIR_Errflag_t * + errflag) + MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_scan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm, + MPIR_Errflag_t * + errflag) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_exscan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, + MPIR_Errflag_t * + errflag) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_neighbor_allgather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * + comm) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_neighbor_allgatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, + const int *displs, + MPI_Datatype recvtype, + MPIR_Comm * + comm) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_neighbor_alltoallv(const void *sendbuf, + const int *sendcounts, + const int *sdispls, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, + const int *rdispls, + MPI_Datatype recvtype, + MPIR_Comm * + comm) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_neighbor_alltoallw(const void *sendbuf, + const int *sendcounts, + const MPI_Aint * sdispls, + const MPI_Datatype * sendtypes, + void *recvbuf, const int *recvcounts, + const MPI_Aint * rdispls, + const MPI_Datatype * recvtypes, + MPIR_Comm * + comm) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_neighbor_alltoall(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * + comm) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ineighbor_allgather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, + MPI_Request * + req) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ineighbor_allgatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, + void *recvbuf, + const int *recvcounts, + const int *displs, + MPI_Datatype recvtype, + MPIR_Comm * comm, + MPI_Request * + req) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ineighbor_alltoall(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, + MPI_Request * + req) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ineighbor_alltoallv(const void *sendbuf, + const int *sendcounts, + const int *sdispls, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, + const int *rdispls, + MPI_Datatype recvtype, + MPIR_Comm * comm, + MPI_Request * + req) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ineighbor_alltoallw(const void *sendbuf, + const int *sendcounts, + const MPI_Aint * sdispls, + const MPI_Datatype * sendtypes, + void *recvbuf, const int *recvcounts, + const MPI_Aint * rdispls, + const MPI_Datatype * recvtypes, + MPIR_Comm * comm, + MPI_Request * + req) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ibarrier(MPIR_Comm * comm, + MPI_Request * + req) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ibcast(void *buffer, int count, MPI_Datatype datatype, + int root, MPIR_Comm * comm, + MPI_Request * req) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_iallgather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, + MPI_Request * + req) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_iallgatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, const int *displs, + MPI_Datatype recvtype, MPIR_Comm * comm, + MPI_Request * + req) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_iallreduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, + MPI_Request * + req) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ialltoall(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, + MPI_Request * + req) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ialltoallv(const void *sendbuf, const int *sendcounts, + const int *sdispls, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, + const int *rdispls, MPI_Datatype recvtype, + MPIR_Comm * comm, + MPI_Request * + req) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ialltoallw(const void *sendbuf, const int *sendcounts, + const int *sdispls, + const MPI_Datatype sendtypes[], void *recvbuf, + const int *recvcounts, const int *rdispls, + const MPI_Datatype recvtypes[], + MPIR_Comm * comm, + MPI_Request * + req) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_iexscan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, + MPI_Request * req) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_igather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, int root, + MPIR_Comm * comm, + MPI_Request * req) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_igatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, const int *displs, + MPI_Datatype recvtype, int root, + MPIR_Comm * comm, + MPI_Request * + req) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ireduce_scatter_block(const void *sendbuf, void *recvbuf, + int recvcount, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, + MPI_Request * + req) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ireduce_scatter(const void *sendbuf, void *recvbuf, + const int *recvcounts, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, + MPI_Request * + req) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ireduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, int root, + MPIR_Comm * comm_ptr, + MPI_Request * req) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_iscan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm, + MPI_Request * req) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_iscatter(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, int root, + MPIR_Comm * comm, + MPI_Request * + req) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_iscatterv(const void *sendbuf, const int *sendcounts, + const int *displs, MPI_Datatype sendtype, + void *recvbuf, int recvcount, + MPI_Datatype recvtype, int root, + MPIR_Comm * comm_ptr, + MPI_Request * + req) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX void MPIDI_NM_datatype_commit(MPIR_Datatype * + datatype_p) + MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX void MPIDI_NM_datatype_dup(MPIR_Datatype * old_datatype_p, + MPIR_Datatype * + new_datatype_p) + MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX void MPIDI_NM_datatype_destroy(MPIR_Datatype * + datatype_p) + MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX void MPIDI_NM_op_commit(MPIR_Op * op_p) MPIDI_NM_STATIC_INLINE_SUFFIX; +MPIDI_NM_STATIC_INLINE_PREFIX void MPIDI_NM_op_destroy(MPIR_Op * + op_p) MPIDI_NM_STATIC_INLINE_SUFFIX; + +#endif diff --git a/src/mpid/ch4/netmod/include/netmod_impl.h b/src/mpid/ch4/netmod/include/netmod_impl.h new file mode 100644 index 0000000..96cb922 --- /dev/null +++ b/src/mpid/ch4/netmod/include/netmod_impl.h @@ -0,0 +1,1083 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +/* ch4 netmod functions */ +#ifndef NETMOD_IMPL_PROTOTYPES_H_INCLUDED +#define NETMOD_IMPL_PROTOTYPES_H_INCLUDED + + +#ifndef NETMOD_DIRECT +#ifndef NETMOD_DISABLE_INLINES + +#ifndef MPIDI_NM_STATIC_INLINE_PREFIX +#define MPIDI_NM_STATIC_INLINE_PREFIX __attribute__((always_inline)) static inline +#endif + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_init(int rank, int size, int appnum, int *tag_ub, + MPIR_Comm * comm_world, MPIR_Comm * comm_self, + int spawned, int num_contexts, + void **netmod_contexts) +{ + return MPIDI_NM_func->init(rank, size, appnum, tag_ub, comm_world, comm_self, spawned, + num_contexts, netmod_contexts); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_finalize(void) +{ + return MPIDI_NM_func->finalize(); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_progress(void *netmod_context, int blocking) +{ + return MPIDI_NM_func->progress(netmod_context, blocking); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_reg_hdr_handler(int handler_id, + MPIDI_NM_am_origin_handler_fn + origin_handler_fn, + MPIDI_NM_am_target_handler_fn + target_handler_fn) +{ + return MPIDI_NM_func->reg_hdr_handler(handler_id, origin_handler_fn, target_handler_fn); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_comm_connect(const char *port_name, MPIR_Info * info, + int root, MPIR_Comm * comm, + MPIR_Comm ** newcomm_ptr) +{ + return MPIDI_NM_func->comm_connect(port_name, info, root, comm, newcomm_ptr); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_comm_disconnect(MPIR_Comm * comm_ptr) +{ + return MPIDI_NM_func->comm_disconnect(comm_ptr); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_open_port(MPIR_Info * info_ptr, char *port_name) +{ + return MPIDI_NM_func->open_port(info_ptr, port_name); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_close_port(const char *port_name) +{ + return MPIDI_NM_func->close_port(port_name); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_comm_accept(const char *port_name, MPIR_Info * info, + int root, MPIR_Comm * comm, + MPIR_Comm ** newcomm_ptr) +{ + return MPIDI_NM_func->comm_accept(port_name, info, root, comm, newcomm_ptr); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_send_am_hdr(int rank, MPIR_Comm * comm, int handler_id, + const void *am_hdr, size_t am_hdr_sz, + MPIR_Request * sreq, void *netmod_context) +{ + return MPIDI_NM_func->send_am_hdr(rank, comm, handler_id, am_hdr, am_hdr_sz, sreq, + netmod_context); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_inject_am_hdr(int rank, MPIR_Comm * comm, int handler_id, + const void *am_hdr, size_t am_hdr_sz, + void *netmod_context) +{ + return MPIDI_NM_func->inject_am_hdr(rank, comm, handler_id, am_hdr, am_hdr_sz, netmod_context); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_send_am(int rank, MPIR_Comm * comm, int handler_id, + const void *am_hdr, size_t am_hdr_sz, + const void *data, MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq, + void *netmod_context) +{ + return MPIDI_NM_func->send_am(rank, comm, handler_id, am_hdr, am_hdr_sz, data, count, datatype, + sreq, netmod_context); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_send_amv(int rank, MPIR_Comm * comm, int handler_id, + struct iovec *am_hdrs, size_t iov_len, + const void *data, MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq, + void *netmod_context) +{ + return MPIDI_NM_func->send_amv(rank, comm, handler_id, am_hdrs, iov_len, data, count, datatype, + sreq, netmod_context); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_send_amv_hdr(int rank, MPIR_Comm * comm, int handler_id, + struct iovec *am_hdrs, size_t iov_len, + MPIR_Request * sreq, void *netmod_context) +{ + return MPIDI_NM_func->send_amv_hdr(rank, comm, handler_id, am_hdrs, iov_len, sreq, + netmod_context); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_send_am_hdr_reply(MPIR_Context_id_t context_id, + int src_rank, int handler_id, + const void *am_hdr, size_t am_hdr_sz, + MPIR_Request * sreq) +{ + return MPIDI_NM_func->send_am_hdr_reply(context_id, src_rank, handler_id, am_hdr, am_hdr_sz, + sreq); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_inject_am_hdr_reply(MPIR_Context_id_t context_id, + int src_rank, int handler_id, + const void *am_hdr, size_t am_hdr_sz) +{ + return MPIDI_NM_func->inject_am_hdr_reply(context_id, src_rank, handler_id, am_hdr, am_hdr_sz); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_send_am_reply(MPIR_Context_id_t context_id, int src_rank, + int handler_id, const void *am_hdr, + size_t am_hdr_sz, const void *data, + MPI_Count count, MPI_Datatype datatype, + MPIR_Request * sreq) +{ + return MPIDI_NM_func->send_am_reply(context_id, src_rank, handler_id, am_hdr, am_hdr_sz, data, + count, datatype, sreq); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_send_amv_reply(MPIR_Context_id_t context_id, + int src_rank, int handler_id, + struct iovec *am_hdr, size_t iov_len, + const void *data, MPI_Count count, + MPI_Datatype datatype, + MPIR_Request * sreq) +{ + return MPIDI_NM_func->send_amv_reply(context_id, src_rank, handler_id, am_hdr, iov_len, data, + count, datatype, sreq); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX size_t MPIDI_NM_am_hdr_max_sz(void) +{ + return MPIDI_NM_func->am_hdr_max_sz(); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX size_t MPIDI_NM_am_inject_max_sz(void) +{ + return MPIDI_NM_func->am_inject_max_sz(); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_am_recv(MPIR_Request * req) +{ + return MPIDI_NM_func->am_recv(req); +} + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_comm_get_lpid(MPIR_Comm * comm_ptr, int idx, + int *lpid_ptr, MPL_bool is_remote) +{ + return MPIDI_NM_func->comm_get_lpid(comm_ptr, idx, lpid_ptr, is_remote); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_gpid_get(MPIR_Comm * comm_ptr, int rank, + MPIR_Gpid * gpid) +{ + return MPIDI_NM_func->gpid_get(comm_ptr, rank, gpid); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_get_node_id(MPIR_Comm * comm, int rank, + MPID_Node_id_t * id_p) +{ + return MPIDI_NM_func->get_node_id(comm, rank, id_p); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_get_max_node_id(MPIR_Comm * comm, + MPID_Node_id_t * max_id_p) +{ + return MPIDI_NM_func->get_max_node_id(comm, max_id_p); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_getallincomm(MPIR_Comm * comm_ptr, int local_size, + MPIR_Gpid local_gpid[], int *singleAVT) +{ + return MPIDI_NM_func->getallincomm(comm_ptr, local_size, local_gpid, singleAVT); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_gpid_tolpidarray(int size, MPIR_Gpid gpid[], int lpid[]) +{ + return MPIDI_NM_func->gpid_tolpidarray(size, gpid, lpid); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_create_intercomm_from_lpids(MPIR_Comm * newcomm_ptr, + int size, const int lpids[]) +{ + return MPIDI_NM_func->create_intercomm_from_lpids(newcomm_ptr, size, lpids); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_comm_create(MPIR_Comm * comm) +{ + return MPIDI_NM_func->comm_create(comm); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_comm_destroy(MPIR_Comm * comm) +{ + return MPIDI_NM_func->comm_destroy(comm); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX void MPIDI_NM_am_request_init(MPIR_Request * req) +{ + return MPIDI_NM_func->am_request_init(req); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX void MPIDI_NM_am_request_finalize(MPIR_Request * req) +{ + return MPIDI_NM_func->am_request_finalize(req); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_send(const void *buf, int count, MPI_Datatype datatype, + int rank, int tag, MPIR_Comm * comm, + int context_offset, MPIR_Request ** request) +{ + return MPIDI_NM_native_func->send(buf, count, datatype, rank, tag, comm, context_offset, + request); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ssend(const void *buf, int count, MPI_Datatype datatype, + int rank, int tag, MPIR_Comm * comm, + int context_offset, MPIR_Request ** request) +{ + return MPIDI_NM_native_func->ssend(buf, count, datatype, rank, tag, comm, context_offset, + request); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_startall(int count, MPIR_Request * requests[]) +{ + return MPIDI_NM_native_func->startall(count, requests); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_send_init(const void *buf, int count, + MPI_Datatype datatype, int rank, int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) +{ + return MPIDI_NM_native_func->send_init(buf, count, datatype, rank, tag, comm, context_offset, + request); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ssend_init(const void *buf, int count, + MPI_Datatype datatype, int rank, int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) +{ + return MPIDI_NM_native_func->ssend_init(buf, count, datatype, rank, tag, comm, context_offset, + request); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_rsend_init(const void *buf, int count, + MPI_Datatype datatype, int rank, int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) +{ + return MPIDI_NM_native_func->rsend_init(buf, count, datatype, rank, tag, comm, context_offset, + request); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_bsend_init(const void *buf, int count, + MPI_Datatype datatype, int rank, int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) +{ + return MPIDI_NM_native_func->bsend_init(buf, count, datatype, rank, tag, comm, context_offset, + request); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_isend(const void *buf, int count, MPI_Datatype datatype, + int rank, int tag, MPIR_Comm * comm, + int context_offset, MPIR_Request ** request) +{ + return MPIDI_NM_native_func->isend(buf, count, datatype, rank, tag, comm, context_offset, + request); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_issend(const void *buf, int count, MPI_Datatype datatype, + int rank, int tag, MPIR_Comm * comm, + int context_offset, MPIR_Request ** request) +{ + return MPIDI_NM_native_func->issend(buf, count, datatype, rank, tag, comm, context_offset, + request); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_cancel_send(MPIR_Request * sreq) +{ + return MPIDI_NM_native_func->cancel_send(sreq); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_recv_init(void *buf, int count, MPI_Datatype datatype, + int rank, int tag, MPIR_Comm * comm, + int context_offset, MPIR_Request ** request) +{ + return MPIDI_NM_native_func->recv_init(buf, count, datatype, rank, tag, comm, context_offset, + request); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_recv(void *buf, int count, MPI_Datatype datatype, + int rank, int tag, MPIR_Comm * comm, + int context_offset, MPI_Status * status, + MPIR_Request ** request) +{ + return MPIDI_NM_native_func->recv(buf, count, datatype, rank, tag, comm, context_offset, status, + request); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_irecv(void *buf, int count, MPI_Datatype datatype, + int rank, int tag, MPIR_Comm * comm, + int context_offset, MPIR_Request ** request) +{ + return MPIDI_NM_native_func->irecv(buf, count, datatype, rank, tag, comm, context_offset, + request); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_imrecv(void *buf, int count, MPI_Datatype datatype, + MPIR_Request * message, MPIR_Request ** rreqp) +{ + return MPIDI_NM_native_func->imrecv(buf, count, datatype, message, rreqp); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_cancel_recv(MPIR_Request * rreq) +{ + return MPIDI_NM_native_func->cancel_recv(rreq); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX void *MPIDI_NM_alloc_mem(size_t size, MPIR_Info * info_ptr) +{ + return MPIDI_NM_native_func->alloc_mem(size, info_ptr); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_free_mem(void *ptr) +{ + return MPIDI_NM_native_func->free_mem(ptr); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_improbe(int source, int tag, MPIR_Comm * comm, + int context_offset, int *flag, + MPIR_Request ** message, MPI_Status * status) +{ + return MPIDI_NM_native_func->improbe(source, tag, comm, context_offset, flag, message, status); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_iprobe(int source, int tag, MPIR_Comm * comm, + int context_offset, int *flag, + MPI_Status * status) +{ + return MPIDI_NM_native_func->iprobe(source, tag, comm, context_offset, flag, status); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_set_info(MPIR_Win * win, MPIR_Info * info) +{ + return MPIDI_NM_native_func->win_set_info(win, info); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_shared_query(MPIR_Win * win, int rank, + MPI_Aint * size, int *disp_unit, + void *baseptr) +{ + return MPIDI_NM_native_func->win_shared_query(win, rank, size, disp_unit, baseptr); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_put(const void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, int target_rank, + MPI_Aint target_disp, int target_count, + MPI_Datatype target_datatype, MPIR_Win * win) +{ + return MPIDI_NM_native_func->put(origin_addr, origin_count, origin_datatype, target_rank, + target_disp, target_count, target_datatype, win); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_start(MPIR_Group * group, int assert, MPIR_Win * win) +{ + return MPIDI_NM_native_func->win_start(group, assert, win); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_complete(MPIR_Win * win) +{ + return MPIDI_NM_native_func->win_complete(win); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_post(MPIR_Group * group, int assert, MPIR_Win * win) +{ + return MPIDI_NM_native_func->win_post(group, assert, win); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_wait(MPIR_Win * win) +{ + return MPIDI_NM_native_func->win_wait(win); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_test(MPIR_Win * win, int *flag) +{ + return MPIDI_NM_native_func->win_test(win, flag); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_lock(int lock_type, int rank, int assert, + MPIR_Win * win) +{ + return MPIDI_NM_native_func->win_lock(lock_type, rank, assert, win); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_unlock(int rank, MPIR_Win * win) +{ + return MPIDI_NM_native_func->win_unlock(rank, win); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_get_info(MPIR_Win * win, MPIR_Info ** info_p_p) +{ + return MPIDI_NM_native_func->win_get_info(win, info_p_p); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_get(void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, int target_rank, + MPI_Aint target_disp, int target_count, + MPI_Datatype target_datatype, MPIR_Win * win) +{ + return MPIDI_NM_native_func->get(origin_addr, origin_count, origin_datatype, target_rank, + target_disp, target_count, target_datatype, win); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_free(MPIR_Win ** win_ptr) +{ + return MPIDI_NM_native_func->win_free(win_ptr); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_fence(int assert, MPIR_Win * win) +{ + return MPIDI_NM_native_func->win_fence(assert, win); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_create(void *base, MPI_Aint length, int disp_unit, + MPIR_Info * info, MPIR_Comm * comm_ptr, + MPIR_Win ** win_ptr) +{ + return MPIDI_NM_native_func->win_create(base, length, disp_unit, info, comm_ptr, win_ptr); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_accumulate(const void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, int target_rank, + MPI_Aint target_disp, int target_count, + MPI_Datatype target_datatype, MPI_Op op, + MPIR_Win * win) +{ + return MPIDI_NM_native_func->accumulate(origin_addr, origin_count, origin_datatype, target_rank, + target_disp, target_count, target_datatype, op, win); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_attach(MPIR_Win * win, void *base, MPI_Aint size) +{ + return MPIDI_NM_native_func->win_attach(win, base, size); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_allocate_shared(MPI_Aint size, int disp_unit, + MPIR_Info * info_ptr, + MPIR_Comm * comm_ptr, + void **base_ptr, MPIR_Win ** win_ptr) +{ + return MPIDI_NM_native_func->win_allocate_shared(size, disp_unit, info_ptr, comm_ptr, base_ptr, + win_ptr); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_rput(const void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, int target_rank, + MPI_Aint target_disp, int target_count, + MPI_Datatype target_datatype, MPIR_Win * win, + MPIR_Request ** request) +{ + return MPIDI_NM_native_func->rput(origin_addr, origin_count, origin_datatype, target_rank, + target_disp, target_count, target_datatype, win, request); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_flush_local(int rank, MPIR_Win * win) +{ + return MPIDI_NM_native_func->win_flush_local(rank, win); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_detach(MPIR_Win * win, const void *base) +{ + return MPIDI_NM_native_func->win_detach(win, base); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_compare_and_swap(const void *origin_addr, + const void *compare_addr, + void *result_addr, + MPI_Datatype datatype, int target_rank, + MPI_Aint target_disp, MPIR_Win * win) +{ + return MPIDI_NM_native_func->compare_and_swap(origin_addr, compare_addr, result_addr, datatype, + target_rank, target_disp, win); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_raccumulate(const void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, + int target_rank, MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, MPI_Op op, + MPIR_Win * win, MPIR_Request ** request) +{ + return MPIDI_NM_native_func->raccumulate(origin_addr, origin_count, origin_datatype, + target_rank, target_disp, target_count, + target_datatype, op, win, request); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_rget_accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + void *result_addr, int result_count, + MPI_Datatype result_datatype, + int target_rank, MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, MPI_Op op, + MPIR_Win * win, MPIR_Request ** request) +{ + return MPIDI_NM_native_func->rget_accumulate(origin_addr, origin_count, origin_datatype, + result_addr, result_count, result_datatype, + target_rank, target_disp, target_count, + target_datatype, op, win, request); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_fetch_and_op(const void *origin_addr, void *result_addr, + MPI_Datatype datatype, int target_rank, + MPI_Aint target_disp, MPI_Op op, + MPIR_Win * win) +{ + return MPIDI_NM_native_func->fetch_and_op(origin_addr, result_addr, datatype, target_rank, + target_disp, op, win); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_allocate(MPI_Aint size, int disp_unit, + MPIR_Info * info, MPIR_Comm * comm, + void *baseptr, MPIR_Win ** win) +{ + return MPIDI_NM_native_func->win_allocate(size, disp_unit, info, comm, baseptr, win); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_flush(int rank, MPIR_Win * win) +{ + return MPIDI_NM_native_func->win_flush(rank, win); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_flush_local_all(MPIR_Win * win) +{ + return MPIDI_NM_native_func->win_flush_local_all(win); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_unlock_all(MPIR_Win * win) +{ + return MPIDI_NM_native_func->win_unlock_all(win); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_create_dynamic(MPIR_Info * info, MPIR_Comm * comm, + MPIR_Win ** win) +{ + return MPIDI_NM_native_func->win_create_dynamic(info, comm, win); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_rget(void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, int target_rank, + MPI_Aint target_disp, int target_count, + MPI_Datatype target_datatype, MPIR_Win * win, + MPIR_Request ** request) +{ + return MPIDI_NM_native_func->rget(origin_addr, origin_count, origin_datatype, target_rank, + target_disp, target_count, target_datatype, win, request); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_sync(MPIR_Win * win) +{ + return MPIDI_NM_native_func->win_sync(win); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_flush_all(MPIR_Win * win) +{ + return MPIDI_NM_native_func->win_flush_all(win); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_get_accumulate(const void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, + void *result_addr, int result_count, + MPI_Datatype result_datatype, + int target_rank, MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, MPI_Op op, + MPIR_Win * win) +{ + return MPIDI_NM_native_func->get_accumulate(origin_addr, origin_count, origin_datatype, + result_addr, result_count, result_datatype, + target_rank, target_disp, target_count, + target_datatype, op, win); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_win_lock_all(int assert, MPIR_Win * win) +{ + return MPIDI_NM_native_func->win_lock_all(assert, win); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_rank_is_local(int target, MPIR_Comm * comm) +{ + return MPIDI_NM_native_func->rank_is_local(target, comm); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_barrier(MPIR_Comm * comm, MPIR_Errflag_t * errflag) +{ + return MPIDI_NM_native_func->barrier(comm, errflag); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_bcast(void *buffer, int count, MPI_Datatype datatype, + int root, MPIR_Comm * comm, + MPIR_Errflag_t * errflag) +{ + return MPIDI_NM_native_func->bcast(buffer, count, datatype, root, comm, errflag); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_allreduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, MPIR_Errflag_t * errflag) +{ + return MPIDI_NM_native_func->allreduce(sendbuf, recvbuf, count, datatype, op, comm, errflag); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_allgather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, MPIR_Errflag_t * errflag) +{ + return MPIDI_NM_native_func->allgather(sendbuf, sendcount, sendtype, recvbuf, recvcount, + recvtype, comm, errflag); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_allgatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, const int *displs, + MPI_Datatype recvtype, MPIR_Comm * comm, + MPIR_Errflag_t * errflag) +{ + return MPIDI_NM_native_func->allgatherv(sendbuf, sendcount, sendtype, recvbuf, recvcounts, + displs, recvtype, comm, errflag); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_scatter(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, int root, + MPIR_Comm * comm, MPIR_Errflag_t * errflag) +{ + return MPIDI_NM_native_func->scatter(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, + root, comm, errflag); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_scatterv(const void *sendbuf, const int *sendcounts, + const int *displs, MPI_Datatype sendtype, + void *recvbuf, int recvcount, + MPI_Datatype recvtype, int root, + MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + return MPIDI_NM_native_func->scatterv(sendbuf, sendcounts, displs, sendtype, recvbuf, recvcount, + recvtype, root, comm_ptr, errflag); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_gather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, int root, + MPIR_Comm * comm, MPIR_Errflag_t * errflag) +{ + return MPIDI_NM_native_func->gather(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, + root, comm, errflag); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_gatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, const int *displs, + MPI_Datatype recvtype, int root, + MPIR_Comm * comm, MPIR_Errflag_t * errflag) +{ + return MPIDI_NM_native_func->gatherv(sendbuf, sendcount, sendtype, recvbuf, recvcounts, displs, + recvtype, root, comm, errflag); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_alltoall(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, MPIR_Errflag_t * errflag) +{ + return MPIDI_NM_native_func->alltoall(sendbuf, sendcount, sendtype, recvbuf, recvcount, + recvtype, comm, errflag); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_alltoallv(const void *sendbuf, const int *sendcounts, + const int *sdispls, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, + const int *rdispls, MPI_Datatype recvtype, + MPIR_Comm * comm, MPIR_Errflag_t * errflag) +{ + return MPIDI_NM_native_func->alltoallv(sendbuf, sendcounts, sdispls, sendtype, recvbuf, + recvcounts, rdispls, recvtype, comm, errflag); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_alltoallw(const void *sendbuf, const int *sendcounts, + const int *sdispls, + const MPI_Datatype sendtypes[], void *recvbuf, + const int *recvcounts, const int *rdispls, + const MPI_Datatype recvtypes[], + MPIR_Comm * comm, MPIR_Errflag_t * errflag) +{ + return MPIDI_NM_native_func->alltoallw(sendbuf, sendcounts, sdispls, sendtypes, recvbuf, + recvcounts, rdispls, recvtypes, comm, errflag); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_reduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, int root, + MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + return MPIDI_NM_native_func->reduce(sendbuf, recvbuf, count, datatype, op, root, comm_ptr, + errflag); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_reduce_scatter(const void *sendbuf, void *recvbuf, + const int *recvcounts, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) +{ + return MPIDI_NM_native_func->reduce_scatter(sendbuf, recvbuf, recvcounts, datatype, op, + comm_ptr, errflag); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_reduce_scatter_block(const void *sendbuf, void *recvbuf, + int recvcount, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) +{ + return MPIDI_NM_native_func->reduce_scatter_block(sendbuf, recvbuf, recvcount, datatype, op, + comm_ptr, errflag); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_scan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm, + MPIR_Errflag_t * errflag) +{ + return MPIDI_NM_native_func->scan(sendbuf, recvbuf, count, datatype, op, comm, errflag); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_exscan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, MPIR_Errflag_t * errflag) +{ + return MPIDI_NM_native_func->exscan(sendbuf, recvbuf, count, datatype, op, comm, errflag); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_neighbor_allgather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm) +{ + return MPIDI_NM_native_func->neighbor_allgather(sendbuf, sendcount, sendtype, recvbuf, + recvcount, recvtype, comm); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_neighbor_allgatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, + const int *displs, + MPI_Datatype recvtype, + MPIR_Comm * comm) +{ + return MPIDI_NM_native_func->neighbor_allgatherv(sendbuf, sendcount, sendtype, recvbuf, + recvcounts, displs, recvtype, comm); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_neighbor_alltoallv(const void *sendbuf, + const int *sendcounts, + const int *sdispls, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, + const int *rdispls, + MPI_Datatype recvtype, + MPIR_Comm * comm) +{ + return MPIDI_NM_native_func->neighbor_alltoallv(sendbuf, sendcounts, sdispls, sendtype, recvbuf, + recvcounts, rdispls, recvtype, comm); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_neighbor_alltoallw(const void *sendbuf, + const int *sendcounts, + const MPI_Aint * sdispls, + const MPI_Datatype * sendtypes, + void *recvbuf, const int *recvcounts, + const MPI_Aint * rdispls, + const MPI_Datatype * recvtypes, + MPIR_Comm * comm) +{ + return MPIDI_NM_native_func->neighbor_alltoallw(sendbuf, sendcounts, sdispls, sendtypes, + recvbuf, recvcounts, rdispls, recvtypes, comm); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_neighbor_alltoall(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm) +{ + return MPIDI_NM_native_func->neighbor_alltoall(sendbuf, sendcount, sendtype, recvbuf, recvcount, + recvtype, comm); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ineighbor_allgather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, MPI_Request * req) +{ + return MPIDI_NM_native_func->ineighbor_allgather(sendbuf, sendcount, sendtype, recvbuf, + recvcount, recvtype, comm, req); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ineighbor_allgatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, + void *recvbuf, + const int *recvcounts, + const int *displs, + MPI_Datatype recvtype, + MPIR_Comm * comm, MPI_Request * req) +{ + return MPIDI_NM_native_func->ineighbor_allgatherv(sendbuf, sendcount, sendtype, recvbuf, + recvcounts, displs, recvtype, comm, req); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ineighbor_alltoall(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, MPI_Request * req) +{ + return MPIDI_NM_native_func->ineighbor_alltoall(sendbuf, sendcount, sendtype, recvbuf, + recvcount, recvtype, comm, req); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ineighbor_alltoallv(const void *sendbuf, + const int *sendcounts, + const int *sdispls, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, + const int *rdispls, + MPI_Datatype recvtype, + MPIR_Comm * comm, MPI_Request * req) +{ + return MPIDI_NM_native_func->ineighbor_alltoallv(sendbuf, sendcounts, sdispls, sendtype, + recvbuf, recvcounts, rdispls, recvtype, comm, + req); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ineighbor_alltoallw(const void *sendbuf, + const int *sendcounts, + const MPI_Aint * sdispls, + const MPI_Datatype * sendtypes, + void *recvbuf, const int *recvcounts, + const MPI_Aint * rdispls, + const MPI_Datatype * recvtypes, + MPIR_Comm * comm, MPI_Request * req) +{ + return MPIDI_NM_native_func->ineighbor_alltoallw(sendbuf, sendcounts, sdispls, sendtypes, + recvbuf, recvcounts, rdispls, recvtypes, comm, + req); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ibarrier(MPIR_Comm * comm, MPI_Request * req) +{ + return MPIDI_NM_native_func->ibarrier(comm, req); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ibcast(void *buffer, int count, MPI_Datatype datatype, + int root, MPIR_Comm * comm, MPI_Request * req) +{ + return MPIDI_NM_native_func->ibcast(buffer, count, datatype, root, comm, req); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_iallgather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, MPI_Request * req) +{ + return MPIDI_NM_native_func->iallgather(sendbuf, sendcount, sendtype, recvbuf, recvcount, + recvtype, comm, req); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_iallgatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, const int *displs, + MPI_Datatype recvtype, MPIR_Comm * comm, + MPI_Request * req) +{ + return MPIDI_NM_native_func->iallgatherv(sendbuf, sendcount, sendtype, recvbuf, recvcounts, + displs, recvtype, comm, req); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_iallreduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, MPI_Request * req) +{ + return MPIDI_NM_native_func->iallreduce(sendbuf, recvbuf, count, datatype, op, comm, req); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ialltoall(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, MPI_Request * req) +{ + return MPIDI_NM_native_func->ialltoall(sendbuf, sendcount, sendtype, recvbuf, recvcount, + recvtype, comm, req); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ialltoallv(const void *sendbuf, const int *sendcounts, + const int *sdispls, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, + const int *rdispls, MPI_Datatype recvtype, + MPIR_Comm * comm, MPI_Request * req) +{ + return MPIDI_NM_native_func->ialltoallv(sendbuf, sendcounts, sdispls, sendtype, recvbuf, + recvcounts, rdispls, recvtype, comm, req); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ialltoallw(const void *sendbuf, const int *sendcounts, + const int *sdispls, + const MPI_Datatype sendtypes[], void *recvbuf, + const int *recvcounts, const int *rdispls, + const MPI_Datatype recvtypes[], + MPIR_Comm * comm, MPI_Request * req) +{ + return MPIDI_NM_native_func->ialltoallw(sendbuf, sendcounts, sdispls, sendtypes, recvbuf, + recvcounts, rdispls, recvtypes, comm, req); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_iexscan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, MPI_Request * req) +{ + return MPIDI_NM_native_func->iexscan(sendbuf, recvbuf, count, datatype, op, comm, req); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_igather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, int root, + MPIR_Comm * comm, MPI_Request * req) +{ + return MPIDI_NM_native_func->igather(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, + root, comm, req); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_igatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, const int *displs, + MPI_Datatype recvtype, int root, + MPIR_Comm * comm, MPI_Request * req) +{ + return MPIDI_NM_native_func->igatherv(sendbuf, sendcount, sendtype, recvbuf, recvcounts, displs, + recvtype, root, comm, req); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ireduce_scatter_block(const void *sendbuf, void *recvbuf, + int recvcount, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, + MPI_Request * req) +{ + return MPIDI_NM_native_func->ireduce_scatter_block(sendbuf, recvbuf, recvcount, datatype, op, + comm, req); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ireduce_scatter(const void *sendbuf, void *recvbuf, + const int *recvcounts, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, MPI_Request * req) +{ + return MPIDI_NM_native_func->ireduce_scatter(sendbuf, recvbuf, recvcounts, datatype, op, comm, + req); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_ireduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, int root, + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + return MPIDI_NM_native_func->ireduce(sendbuf, recvbuf, count, datatype, op, root, comm_ptr, + req); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_iscan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm, + MPI_Request * req) +{ + return MPIDI_NM_native_func->iscan(sendbuf, recvbuf, count, datatype, op, comm, req); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_iscatter(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, int root, + MPIR_Comm * comm, MPI_Request * req) +{ + return MPIDI_NM_native_func->iscatter(sendbuf, sendcount, sendtype, recvbuf, recvcount, + recvtype, root, comm, req); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX int MPIDI_NM_iscatterv(const void *sendbuf, const int *sendcounts, + const int *displs, MPI_Datatype sendtype, + void *recvbuf, int recvcount, + MPI_Datatype recvtype, int root, + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + return MPIDI_NM_native_func->iscatterv(sendbuf, sendcounts, displs, sendtype, recvbuf, + recvcount, recvtype, root, comm_ptr, req); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX void MPIDI_NM_datatype_commit(MPIR_Datatype * datatype_p) +{ + return MPIDI_NM_native_func->datatype_commit(datatype_p); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX void MPIDI_NM_datatype_dup(MPIR_Datatype * old_datatype_p, + MPIR_Datatype * new_datatype_p) +{ + return MPIDI_NM_native_func->datatype_dup(old_datatype_p, new_datatype_p); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX void MPIDI_NM_datatype_destroy(MPIR_Datatype * datatype_p) +{ + return MPIDI_NM_native_func->datatype_destroy(datatype_p); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX void MPIDI_NM_op_commit(MPIR_Op * op_p) +{ + return MPIDI_NM_native_func->op_commit(op_p); +}; + +MPIDI_NM_STATIC_INLINE_PREFIX void MPIDI_NM_op_destroy(MPIR_Op * op_p) +{ + return MPIDI_NM_native_func->op_destroy(op_p); +}; + +#endif /* NETMOD_DISABLE_INLINES */ + +#else +#define __netmod_direct_stubnm__ 0 +#define __netmod_direct_ofi__ 1 +#define __netmod_direct_shm__ 2 +#define __netmod_direct_ucx__ 3 +#define __netmod_direct_portals4__ 4 + +#if NETMOD_DIRECT==__netmod_direct_stubnm__ +#include "../stubnm/netmod_direct.h" +#elif NETMOD_DIRECT==__netmod_direct_ofi__ +#include "../ofi/netmod_direct.h" +#elif NETMOD_DIRECT==__netmod_direct_shm__ +#include "../shm/netmod_direct.h" +#elif NETMOD_DIRECT==__netmod_direct_ucx__ +#include "../ucx/netmod_direct.h" +#elif NETMOD_DIRECT==__netmod_direct_portals4__ +#include "../portals4/netmod_direct.h" +#else +#error "No direct netmod included" +#endif +#endif /* NETMOD_DIRECT */ + +#endif diff --git a/src/mpid/ch4/netmod/ofi/Makefile.mk b/src/mpid/ch4/netmod/ofi/Makefile.mk new file mode 100644 index 0000000..df2a564 --- /dev/null +++ b/src/mpid/ch4/netmod/ofi/Makefile.mk @@ -0,0 +1,23 @@ +## -*- Mode: Makefile; -*- +## vim: set ft=automake : +## +## (C) 2016 by Argonne National Laboratory. +## See COPYRIGHT in top-level directory. +## +## Portions of this code were written by Intel Corporation. +## Copyright (C) 2011-2016 Intel Corporation. Intel provides this material +## to Argonne National Laboratory subject to Software Grant and Corporate +## Contributor License Agreement dated February 8, 2012. +## +if BUILD_CH4_NETMOD_OFI + +noinst_HEADERS += +mpi_core_sources += src/mpid/ch4/netmod/ofi/func_table.c \ + src/mpid/ch4/netmod/ofi/globals.c \ + src/mpid/ch4/netmod/ofi/util.c +errnames_txt_files += src/mpid/ch4/netmod/ofi/errnames.txt +external_ldflags += -ldl -lpthread +external_subdirs += @ofisrcdir@ +pmpi_convenience_libs += @ofilib@ + +endif diff --git a/src/mpid/ch4/netmod/ofi/catalog.c b/src/mpid/ch4/netmod/ofi/catalog.c new file mode 100644 index 0000000..0905a25 --- /dev/null +++ b/src/mpid/ch4/netmod/ofi/catalog.c @@ -0,0 +1,61 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +static inline void MPIDI_OFI_unused_gen_catalog() +{ +#if 0 + char *a; + int b, e; + MPIR_ERR_SET2(e, MPI_ERR_OTHER, "**ofid_pmi", "**ofid_pmi %s %d %s %s", a, b, a, a); + MPIR_ERR_SET2(e, MPI_ERR_OTHER, "**ofid_addrinfo", "**ofid_addrinfo %s %d %s %s", a, b, a, a); + MPIR_ERR_SET2(e, MPI_ERR_OTHER, "**ofid_opendomain", "**ofid_opendomain %s %d %s %s", a, b, a, + a); + MPIR_ERR_SET2(e, MPI_ERR_OTHER, "**ofid_bind", "**ofid_bind %s %d %s %s", a, b, a, a); + MPIR_ERR_SET2(e, MPI_ERR_OTHER, "**ofid_fabric", "**ofid_fabric %s %d %s %s", a, b, a, a); + MPIR_ERR_SET2(e, MPI_ERR_OTHER, "**ofid_opencq", "**ofid_opencq %s %d %s %s", a, b, a, a); + MPIR_ERR_SET2(e, MPI_ERR_OTHER, "**ofid_openct", "**ofid_openct %s %d %s %s", a, b, a, a); + MPIR_ERR_SET2(e, MPI_ERR_OTHER, "**ofid_bind", "**ofid_bind %s %d %s %s", a, b, a, a); + MPIR_ERR_SET2(e, MPI_ERR_OTHER, "**ofid_ep_enable", "**ofid_ep_enable %s %d %s %s", a, b, a, a); + MPIR_ERR_SET2(e, MPI_ERR_OTHER, "**ofid_ep", "**ofid_ep %s %d %s %s", a, b, a, a); + MPIR_ERR_SET2(e, MPI_ERR_OTHER, "**ofid_avopen", "**ofid_avopen %s %d %s %s", a, b, a, a); + MPIR_ERR_SET2(e, MPI_ERR_OTHER, "**ofid_getname", "**ofid_getname %s %d %s %s", a, b, a, a); + MPIR_ERR_SET2(e, MPI_ERR_OTHER, "**ofid_avmap", "**ofid_avmap %s %d %s %s", a, b, a, a); + MPIR_ERR_SET2(e, MPI_ERR_OTHER, "**ofid_avlookup", "**ofid_avlookup %s %d %s %s", a, b, a, a); + MPIR_ERR_SET2(e, MPI_ERR_OTHER, "**ofid_avsync", "**ofid_avsync %s %d %s %s", a, b, a, a); + MPIR_ERR_SET2(e, MPI_ERR_OTHER, "**ofid_epclose", "**ofid_epclose %s %d %s %s", a, b, a, a); + MPIR_ERR_SET2(e, MPI_ERR_OTHER, "**ofid_cqclose", "**ofid_cqclose %s %d %s %s", a, b, a, a); + MPIR_ERR_SET2(e, MPI_ERR_OTHER, "**ofid_epsync", "**ofid_epsync %s %d %s %s", a, b, a, a); + MPIR_ERR_SET2(e, MPI_ERR_OTHER, "**ofid_alias", "**ofid_alias %s %d %s %s", a, b, a, a); + MPIR_ERR_SET2(e, MPI_ERR_OTHER, "**ofid_getopt", "**ofid_getopt %s %d %s %s", a, b, a, a); + MPIR_ERR_SET2(e, MPI_ERR_OTHER, "**ofid_setopt", "**ofid_setopt %s %d %s %s", a, b, a, a); + MPIR_ERR_SET2(e, MPI_ERR_OTHER, "**ofid_domainclose", "**ofid_domainclose %s %d %s %s", a, b, a, + a); + MPIR_ERR_SET2(e, MPI_ERR_OTHER, "**ofid_avclose", "**ofid_avclose %s %d %s %s", a, b, a, a); + MPIR_ERR_SET2(e, MPI_ERR_OTHER, "**ofid_tsend", "**ofid_tsend %s %d %s %s", a, b, a, a); + MPIR_ERR_SET2(e, MPI_ERR_OTHER, "**ofid_tinject", "**ofid_tinject %s %d %s %s", a, b, a, a); + MPIR_ERR_SET2(e, MPI_ERR_OTHER, "**ofid_tsendsync", "**ofid_tsendsync %s %d %s %s", a, b, a, a); + MPIR_ERR_SET2(e, MPI_ERR_OTHER, "**ofid_trecv", "**ofid_trecv %s %d %s %s", a, b, a, a); + MPIR_ERR_SET2(e, MPI_ERR_OTHER, "**ofid_trecvsync", "**ofid_trecvsync %s %d %s %s", a, b, a, a); + MPIR_ERR_SET2(e, MPI_ERR_OTHER, "**ofid_poll", "**ofid_poll %s %d %s %s", a, b, a, a); + MPIR_ERR_SET2(e, MPI_ERR_OTHER, "**ofid_peek", "**ofid_peek %s %d %s %s", a, b, a, a); + MPIR_ERR_SET2(e, MPI_ERR_OTHER, "**ofid_send", "**ofid_send %s %d %s %s", a, b, a, a); + MPIR_ERR_SET2(e, MPI_ERR_OTHER, "**ofid_inject", "**ofid_inject %s %d %s %s", a, b, a, a); + MPIR_ERR_SET2(e, MPI_ERR_OTHER, "**ofid_rdma_write", "**ofid_rdma_write %s %d %s %s", a, b, a, + a); + MPIR_ERR_SET2(e, MPI_ERR_OTHER, "**ofid_rdma_inject_write", + "**ofid_rdma_inject_write %s %d %s %s", a, b, a, a); + MPIR_ERR_SET2(e, MPI_ERR_OTHER, "**ofid_rdma_atomicto", "**ofid_rdma_atomicto %s %d %s %s", a, + b, a, a); + MPIR_ERR_SET2(e, MPI_ERR_OTHER, "**ofid_rdma_cswap", "**ofid_rdma_cswap %s %d %s %s", a, b, a, + a); + MPIR_ERR_SET2(e, MPI_ERR_OTHER, "**ofid_rdma_readfrom", "**ofid_rdma_readfrom %s %d %s %s", a, + b, a, a); + MPIR_ERR_SET2(e, MPI_ERR_OTHER, "**ofid_rdma_readfrom", "**ofid_rdma_readfrom %s %d %s %s", a, + b, a, a); + MPIR_ERR_SET2(e, MPI_ERR_OTHER, "**ofid_mr_reg", "**ofid_mr_reg %s %d %s %s", a, b, a, a); + MPIR_ERR_SET2(e, MPI_ERR_OTHER, "**ofid_mr_unreg", "**ofid_mr_unreg %s %d %s %s", a, b, a, a); + MPIR_ERR_SET2(e, MPI_ERR_OTHER, "**ofid_prepost", "**ofid_prepost %s %d %s %s", a, b, a, a); + MPIR_ERR_SET2(e, MPI_ERR_OTHER, "**ofid_ctrlcancel", "**ofid_ctrlcancel %s %d %s %s", a, b, a, + a); + MPIR_ERR_SET2(e, MPI_ERR_OTHER, "**ofid_cntr_wait", "**ofid_cntr_wait %s %d %s %s", a, b, a, a); + +#endif +} diff --git a/src/mpid/ch4/netmod/ofi/errnames.txt b/src/mpid/ch4/netmod/ofi/errnames.txt new file mode 100644 index 0000000..2648e10 --- /dev/null +++ b/src/mpid/ch4/netmod/ofi/errnames.txt @@ -0,0 +1,97 @@ +**ofid_pmi:PMI_Init() failure +**ofid_pmi %s %d %s %s:pmi failed (%s:%d:%s:%s) +**ofid_addrinfo:OFI addrinfo() failure +**ofid_addrinfo %s %d %s %s:OFI addrinfo() failed (%s:%d:%s:%s) +**ofid_addrinfo %s:Addrinfo failure ('%s') +**ofid_opendomain:OFI fi_open domain failure +**ofid_opendomain %s %d %s %s:OFI fi_open domain failed (%s:%d:%s:%s) +**ofid_fabric:OFI fi_fabric failure +**ofid_fabric %s %d %s %s:OFI fi_fabric failed (%s:%d:%s:%s) +**ofid_opencq:OFI event queue create failure +**ofid_opencq %s %d %s %s:OFI event queue create failed (%s:%d:%s:%s) +**ofid_openct:OFI event counter create failure +**ofid_openct %s %d %s %s:OFI event counter create failed (%s:%d:%s:%s) +**ofid_bind:OFI resource bind failure +**ofid_bind %s %d %s %s:OFI resource bind failed (%s:%d:%s:%s) +**ofid_ep_enable:OFI EP enable failure +**ofid_ep_enable %s %d %s %s:OFI EP enable failed (%s:%d:%s:%s) +**ofid_avopen:OFI address vector open failed +**ofid_avopen %s %d %s %s:OFI address vector open failed (%s:%d:%s:%s) +**ofid_ep:OFI endpoint open failed +**ofid_ep %s %d %s %s:OFI endpoint open failed (%s:%d:%s:%s) +**ofid_getname:OFI get endpoint name failed +**ofid_getname %s %d %s %s:OFI get endpoint name failed (%s:%d:%s:%s) +**ofid_avmap:OFI get address vector map failed +**ofid_avmap %s %d %s %s:OFI address vector map failed (%s:%d:%s:%s) +**ofid_avlookup:OFI get address vector lookup failed +**ofid_avlookup %s %d %s %s:OFI address vector lookup failed (%s:%d:%s:%s) +**ofid_avsync:OFI get address vector sync failed +**ofid_avsync %s %d %s %s:OFI address vector sync failed (%s:%d:%s:%s) +**ofid_epclose:OFI endpoint close failed +**ofid_epclose %s %d %s %s:OFI endpoint close failed (%s:%d:%s:%s) +**ofid_cqclose:OFI cq close failed +**ofid_cqclose %s %d %s %s:OFI cq close failed (%s:%d:%s:%s) +**ofid_epsync:OFI synchronization failed +**ofid_epsync %s %d %s %s:OFI endpoint synchronization failed (%s:%d:%s:%s) +**ofid_alias:OFI cq alias failed +**ofid_alias %s %d %s %s:OFI cq alias failed (%s:%d:%s:%s) +**ofid_getopt:OFI getopt failed +**ofid_getopt %s %d %s %s:OFI getopt failed (%s:%d:%s:%s) +**ofid_setopt:OFI setopt failed +**ofid_setopt %s %d %s %s:OFI setopt failed (%s:%d:%s:%s) +**ofid_domainclose:OFI domain close failed +**ofid_domainclose %s %d %s %s:OFI domain close failed (%s:%d:%s:%s) +**ofid_avclose:OFI av close failed +**ofid_avclose %s %d %s %s:OFI av close failed (%s:%d:%s:%s) +**ofid_tsend:OFI tagged send failed +**ofid_tsend %s %d %s %s:OFI tagged send failed (%s:%d:%s:%s) +**ofid_tsenddata:OFI tagged senddata failed +**ofid_tsenddata %s %d %s %s:OFI tagged senddata failed (%s:%d:%s:%s) +**ofid_tsendmsg:OFI tagged sendmsg failed +**ofid_tsendmsg %s %d %s %s:OFI tagged sendmsg failed (%s:%d:%s:%s) +**ofid_tinject:OFI tagged inject failed +**ofid_tinject %s %d %s %s:OFI tagged inject failed (%s:%d:%s:%s) +**ofid_tinjectdata:OFI tagged injectdata failed +**ofid_tinjectdata %s %d %s %s:OFI tagged injectdata failed (%s:%d:%s:%s) +**ofid_tsendsync:OFI tagged send sync failed +**ofid_tsendsync %s %d %s %s:OFI tagged send sync failed (%s:%d:%s:%s) +**ofid_trecv:OFI tagged recv failed +**ofid_trecv %s %d %s %s:OFI tagged recv failed (%s:%d:%s:%s) +**ofid_trecvmsg:OFI tagged recvmsg failed +**ofid_trecvmsg %s %d %s %s:OFI tagged recvmsg failed (%s:%d:%s:%s) +**ofid_trecvsync:OFI tagged recv sync failed +**ofid_trecvsync %s %d %s %s:OFI tagged recv sync failed (%s:%d:%s:%s) +**ofid_poll:OFI poll failed +**ofid_poll %s %d %s %s:OFI poll failed (%s:%d:%s:%s) +**ofid_peek:OFI peek failed +**ofid_peek %s %d %s %s:OFI peek failed (%s:%d:%s:%s) +**ofid_send:OFI send failed +**ofid_send %s %d %s %s:OFI send failed (%s:%d:%s:%s) +**ofid_inject:OFI inject failed +**ofid_inject %s %d %s %s:OFI inject failed (%s:%d:%s:%s) +**ofid_rdma_write:OFI rdma write failed +**ofid_rdma_write %s %d %s %s:OFI rdma write failed (%s:%d:%s:%s) +**ofid_rdma_inject_write:OFI rdma write immediatefailed +**ofid_rdma_inject_write %s %d %s %s:OFI rdma write immediate failed (%s:%d:%s:%s) +**ofid_rdma_atomicto:OFI rdma atomicto failed +**ofid_rdma_atomicto %s %d %s %s:OFI rdma atomicto failed (%s:%d:%s:%s) +**ofid_rdma_cswap:OFI rdma cswap failed +**ofid_rdma_cswap %s %d %s %s:OFI rdma cswap failed (%s:%d:%s:%s) +**ofid_rdma_readfrom:OFI rdma read failed +**ofid_rdma_readfrom %s %d %s %s:OFI read failed (%s:%d:%s:%s) +**ofid_mr_reg:OFI memory registration failed +**ofid_mr_reg %s %d %s %s:OFI memory registration failed (%s:%d:%s:%s) +**ofid_mr_unreg:OFI memory deregistration failed +**ofid_mr_unreg %s %d %s %s:OFI memory deregistration failed (%s:%d:%s:%s) +**ofid_prepost:OFI preposting receives failed +**ofid_prepost %s %d %s %s:OFI preposting receives failed (%s:%d:%s:%s) +**ofid_ctrlcancel:OFI Control cancel failed +**ofid_ctrlcancel %s %d %s %s:OFI control cancel failed (%s:%d:%s:%s) +**ofid_cntr_wait:OFI Counter wait failed +**ofid_cntr_wait %s %d %s %s:OFI OFI Counter wait failed (%s:%d:%s:%s) +**ofid_rma_init:OFI RMA Initialization failed +**ofid_rma_init %s %d %s %s:OFI RMA Initialization failed (%s:%d:%s:%s) +**ofid_stx_ctx:OFI fi_stx_context failed +**ofid_stx_ctx %s %d %s %s:OFI fi_stx_context failed (%s:%d:%s:%s) +**ofid_stx_ctx_close:OFI stx context close failed +**ofid_stx_ctx_close %s %d %s %s:OFI stx context close failed (%s:%d:%s:%s) diff --git a/src/mpid/ch4/netmod/ofi/fi_list.h b/src/mpid/ch4/netmod/ofi/fi_list.h new file mode 100644 index 0000000..488aed8 --- /dev/null +++ b/src/mpid/ch4/netmod/ofi/fi_list.h @@ -0,0 +1,196 @@ +/* + * Copyright (c) 2011-2016 Intel Corporation. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + * + */ + +#if !defined(LIST_H) +#define LIST_H + +#include <unistd.h> +#include <errno.h> +#include <fcntl.h> + +/* + * Double-linked list + */ +struct dlist_entry { + struct dlist_entry *next; + struct dlist_entry *prev; +}; + +#define DLIST_INIT(addr) { addr, addr } +#define DEFINE_LIST(name) struct dlist_entry name = DLIST_INIT(&name) + +static inline void dlist_init(struct dlist_entry *head) +{ + head->next = head; + head->prev = head; +} + +static inline int dlist_empty(struct dlist_entry *head) +{ + return head->next == head; +} + +static inline void dlist_insert_after(struct dlist_entry *item, struct dlist_entry *head) +{ + item->next = head->next; + item->prev = head; + head->next->prev = item; + head->next = item; +} + +static inline void dlist_insert_before(struct dlist_entry *item, struct dlist_entry *head) +{ + dlist_insert_after(item, head->prev); +} + +#define dlist_insert_head dlist_insert_after +#define dlist_insert_tail dlist_insert_before + +static inline void dlist_remove(struct dlist_entry *item) +{ + item->prev->next = item->next; + item->next->prev = item->prev; +} + +#define dlist_foreach(head, item) \ + for (item = (head)->next; item != head; item = item->next) + +typedef int dlist_func_t(struct dlist_entry *item, const void *arg); + +static inline struct dlist_entry *dlist_find_first_match(struct dlist_entry *head, + dlist_func_t * match, const void *arg) +{ + struct dlist_entry *item; + + dlist_foreach(head, item) { + if (match(item, arg)) + return item; + } + + return NULL; +} + +static inline struct dlist_entry *dlist_remove_first_match(struct dlist_entry *head, + dlist_func_t * match, const void *arg) +{ + struct dlist_entry *item; + + item = dlist_find_first_match(head, match, arg); + if (item) + dlist_remove(item); + + return item; +} + +/* + * Single-linked list + */ +struct slist_entry { + struct slist_entry *next; +}; + +struct slist { + struct slist_entry *head; + struct slist_entry *tail; +}; + +static inline void slist_init(struct slist *list) +{ + list->head = list->tail = NULL; +} + +static inline int slist_empty(struct slist *list) +{ + return !list->head; +} + +static inline void slist_insert_head(struct slist_entry *item, struct slist *list) +{ + if (slist_empty(list)) + list->tail = item; + else + item->next = list->head; + + list->head = item; +} + +static inline void slist_insert_tail(struct slist_entry *item, struct slist *list) +{ + if (slist_empty(list)) + list->head = item; + else + list->tail->next = item; + + list->tail = item; +} + +static inline struct slist_entry *slist_remove_head(struct slist *list) +{ + struct slist_entry *item; + + item = list->head; + if (list->head == list->tail) + slist_init(list); + else + list->head = item->next; + return item; +} + +#define slist_foreach(list, item, prev) \ + for (prev = NULL, item = list->head; item; prev = item, item = item->next) + +typedef int slist_func_t(struct slist_entry *item, const void *arg); + +static inline struct slist_entry *slist_remove_first_match(struct slist *list, slist_func_t * match, + const void *arg) +{ + struct slist_entry *item, *prev; + + slist_foreach(list, item, prev) { + if (match(item, arg)) { + if (prev) + prev->next = item->next; + else + list->head = item->next; + + if (!item->next) + list->tail = prev; + + return item; + } + } + + return NULL; +} + +#endif /* LIST_H */ diff --git a/src/mpid/ch4/netmod/ofi/func_table.c b/src/mpid/ch4/netmod/ofi/func_table.c new file mode 100644 index 0000000..777c73e --- /dev/null +++ b/src/mpid/ch4/netmod/ofi/func_table.c @@ -0,0 +1,157 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ + +#ifndef NETMOD_DIRECT +#define NETMOD_DISABLE_INLINES +#include <mpidimpl.h> +#include "netmod_direct.h" +MPIDI_NM_funcs_t MPIDI_NM_ofi_funcs = { + MPIDI_NM_init, + MPIDI_NM_finalize, + MPIDI_NM_progress, + MPIDI_NM_comm_connect, + MPIDI_NM_comm_disconnect, + MPIDI_NM_open_port, + MPIDI_NM_close_port, + MPIDI_NM_comm_accept, + MPIDI_NM_comm_get_lpid, + MPIDI_NM_gpid_get, + MPIDI_NM_get_node_id, + MPIDI_NM_get_max_node_id, + MPIDI_NM_getallincomm, + MPIDI_NM_gpid_tolpidarray, + MPIDI_NM_create_intercomm_from_lpids, + MPIDI_NM_comm_create, + MPIDI_NM_comm_destroy, + MPIDI_NM_am_request_init, + MPIDI_NM_am_request_finalize, + MPIDI_NM_reg_hdr_handler, + MPIDI_NM_send_am_hdr, + MPIDI_NM_inject_am_hdr, + MPIDI_NM_send_am, + MPIDI_NM_send_amv, + MPIDI_NM_send_amv_hdr, + MPIDI_NM_send_am_hdr_reply, + MPIDI_NM_inject_am_hdr_reply, + MPIDI_NM_send_am_reply, + MPIDI_NM_send_amv_reply, + MPIDI_NM_am_hdr_max_sz, + MPIDI_NM_am_inject_max_sz, + MPIDI_NM_am_recv +}; + +MPIDI_NM_native_funcs_t MPIDI_NM_native_ofi_funcs = { + MPIDI_NM_send, + MPIDI_NM_ssend, + MPIDI_NM_startall, + MPIDI_NM_send_init, + MPIDI_NM_ssend_init, + MPIDI_NM_rsend_init, + MPIDI_NM_bsend_init, + MPIDI_NM_isend, + MPIDI_NM_issend, + MPIDI_NM_cancel_send, + MPIDI_NM_recv_init, + MPIDI_NM_recv, + MPIDI_NM_irecv, + MPIDI_NM_imrecv, + MPIDI_NM_cancel_recv, + MPIDI_NM_alloc_mem, + MPIDI_NM_free_mem, + MPIDI_NM_improbe, + MPIDI_NM_iprobe, + MPIDI_NM_win_set_info, + MPIDI_NM_win_shared_query, + MPIDI_NM_put, + MPIDI_NM_win_start, + MPIDI_NM_win_complete, + MPIDI_NM_win_post, + MPIDI_NM_win_wait, + MPIDI_NM_win_test, + MPIDI_NM_win_lock, + MPIDI_NM_win_unlock, + MPIDI_NM_win_get_info, + MPIDI_NM_get, + MPIDI_NM_win_free, + MPIDI_NM_win_fence, + MPIDI_NM_win_create, + MPIDI_NM_accumulate, + MPIDI_NM_win_attach, + MPIDI_NM_win_allocate_shared, + MPIDI_NM_rput, + MPIDI_NM_win_flush_local, + MPIDI_NM_win_detach, + MPIDI_NM_compare_and_swap, + MPIDI_NM_raccumulate, + MPIDI_NM_rget_accumulate, + MPIDI_NM_fetch_and_op, + MPIDI_NM_win_allocate, + MPIDI_NM_win_flush, + MPIDI_NM_win_flush_local_all, + MPIDI_NM_win_unlock_all, + MPIDI_NM_win_create_dynamic, + MPIDI_NM_rget, + MPIDI_NM_win_sync, + MPIDI_NM_win_flush_all, + MPIDI_NM_get_accumulate, + MPIDI_NM_win_lock_all, + MPIDI_NM_rank_is_local, + MPIDI_NM_barrier, + MPIDI_NM_bcast, + MPIDI_NM_allreduce, + MPIDI_NM_allgather, + MPIDI_NM_allgatherv, + MPIDI_NM_scatter, + MPIDI_NM_scatterv, + MPIDI_NM_gather, + MPIDI_NM_gatherv, + MPIDI_NM_alltoall, + MPIDI_NM_alltoallv, + MPIDI_NM_alltoallw, + MPIDI_NM_reduce, + MPIDI_NM_reduce_scatter, + MPIDI_NM_reduce_scatter_block, + MPIDI_NM_scan, + MPIDI_NM_exscan, + MPIDI_NM_neighbor_allgather, + MPIDI_NM_neighbor_allgatherv, + MPIDI_NM_neighbor_alltoall, + MPIDI_NM_neighbor_alltoallv, + MPIDI_NM_neighbor_alltoallw, + MPIDI_NM_ineighbor_allgather, + MPIDI_NM_ineighbor_allgatherv, + MPIDI_NM_ineighbor_alltoall, + MPIDI_NM_ineighbor_alltoallv, + MPIDI_NM_ineighbor_alltoallw, + MPIDI_NM_ibarrier, + MPIDI_NM_ibcast, + MPIDI_NM_iallgather, + MPIDI_NM_iallgatherv, + MPIDI_NM_iallreduce, + MPIDI_NM_ialltoall, + MPIDI_NM_ialltoallv, + MPIDI_NM_ialltoallw, + MPIDI_NM_iexscan, + MPIDI_NM_igather, + MPIDI_NM_igatherv, + MPIDI_NM_ireduce_scatter_block, + MPIDI_NM_ireduce_scatter, + MPIDI_NM_ireduce, + MPIDI_NM_iscan, + MPIDI_NM_iscatter, + MPIDI_NM_iscatterv, + MPIDI_NM_datatype_commit, + MPIDI_NM_datatype_dup, + MPIDI_NM_datatype_destroy, + MPIDI_NM_op_commit, + MPIDI_NM_op_destroy, +}; +#endif diff --git a/src/mpid/ch4/netmod/ofi/globals.c b/src/mpid/ch4/netmod/ofi/globals.c new file mode 100644 index 0000000..cf74fd9 --- /dev/null +++ b/src/mpid/ch4/netmod/ofi/globals.c @@ -0,0 +1,13 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#include <mpidimpl.h> +#include "ofi_impl.h" +MPIDI_OFI_global_t MPIDI_Global = { 0 }; diff --git a/src/mpid/ch4/netmod/ofi/netmod_direct.h b/src/mpid/ch4/netmod/ofi/netmod_direct.h new file mode 100644 index 0000000..0d77db2 --- /dev/null +++ b/src/mpid/ch4/netmod/ofi/netmod_direct.h @@ -0,0 +1,41 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_DIRECT_H_INCLUDED +#define NETMOD_DIRECT_H_INCLUDED + +#include "ofi_am.h" +#include "ofi_events.h" +#include "ofi_comm.h" +#include "ofi_proc.h" +#include "ofi_progress.h" +#include "ofi_unimpl.h" +#include "ofi_init.h" +#include "ofi_coll.h" +#include "ofi_datatype.h" +#include "ofi_op.h" + +#ifdef USE_OFI_TAGGED +#include "ofi_probe.h" +#include "ofi_recv.h" +#include "ofi_send.h" +#include "ofi_win.h" +#include "ofi_rma.h" +#include "ofi_spawn.h" +#else +#include "ofi_am_probe.h" +#include "ofi_am_recv.h" +#include "ofi_am_send.h" +#include "ofi_am_win.h" +#include "ofi_am_rma.h" +#include "ofi_am_spawn.h" +#endif /* USE_OFI_TAGGED */ + +#endif /* NETMOD_DIRECT_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/ofi/ofi_am.h b/src/mpid/ch4/netmod/ofi/ofi_am.h new file mode 100644 index 0000000..f3425dd --- /dev/null +++ b/src/mpid/ch4/netmod/ofi/ofi_am.h @@ -0,0 +1,383 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_OFI_AM_H_INCLUDED +#define NETMOD_OFI_AM_H_INCLUDED +#include "ofi_impl.h" +#include "ofi_am_impl.h" +#include "ofi_am_events.h" + +static inline int MPIDI_OFI_progress_do_queue(void *netmod_context); + +static inline void MPIDI_NM_am_request_init(MPIR_Request * req) +{ + MPIDI_OFI_AMREQUEST(req, req_hdr) = NULL; +} + +static inline void MPIDI_NM_am_request_finalize(MPIR_Request * req) +{ + MPIDI_OFI_am_clear_request(req); +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_reg_hdr_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_reg_hdr_handler(int handler_id, + MPIDI_NM_am_origin_handler_fn origin_handler_fn, + MPIDI_NM_am_target_handler_fn target_handler_fn) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_REG_HDR_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_REG_HDR_HANDLER); + + if (handler_id > MPIDI_OFI_MAX_AM_HANDLERS) { + mpi_errno = MPI_ERR_OTHER; + goto fn_fail; + } + + MPIDI_Global.am_handlers[handler_id] = target_handler_fn; + MPIDI_Global.am_send_cmpl_handlers[handler_id] = origin_handler_fn; + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_REG_HDR_HANDLER); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_send_am_hdr +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_send_am_hdr(int rank, + MPIR_Comm * comm, + int handler_id, + const void *am_hdr, + size_t am_hdr_sz, MPIR_Request * sreq, void *netmod_context) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_SEND_AM_HDR); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_SEND_AM_HDR); + + mpi_errno = MPIDI_OFI_do_send_am_header(rank, comm, handler_id, am_hdr, am_hdr_sz, sreq, FALSE); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_SEND_AM_HDR); + return mpi_errno; +} + + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_send_am +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_send_am(int rank, + MPIR_Comm * comm, + int handler_id, + const void *am_hdr, + size_t am_hdr_sz, + const void *data, + MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq, void *netmod_context) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_SEND_AM); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_SEND_AM); + mpi_errno = MPIDI_OFI_do_send_am(rank, comm, handler_id, + am_hdr, am_hdr_sz, data, count, datatype, sreq, FALSE); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_SEND_AM); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_send_amv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_send_amv(int rank, + MPIR_Comm * comm, + int handler_id, + struct iovec *am_hdr, + size_t iov_len, + const void *data, + MPI_Count count, + MPI_Datatype datatype, + MPIR_Request * sreq, void *netmod_context) +{ + int mpi_errno = MPI_SUCCESS, is_allocated; + size_t am_hdr_sz = 0, i; + char *am_hdr_buf; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_SEND_AMV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_SEND_AMV); + + for (i = 0; i < iov_len; i++) { + am_hdr_sz += am_hdr[i].iov_len; + } + + if (am_hdr_sz > MPIDI_OFI_BUF_POOL_SIZE) { + am_hdr_buf = (char *) MPL_malloc(am_hdr_sz); + is_allocated = 1; + } + else { + am_hdr_buf = (char *) MPIDI_CH4R_get_buf(MPIDI_Global.am_buf_pool); + is_allocated = 0; + } + + MPIR_Assert(am_hdr_buf); + am_hdr_sz = 0; + + for (i = 0; i < iov_len; i++) { + MPIR_Memcpy(am_hdr_buf + am_hdr_sz, am_hdr[i].iov_base, am_hdr[i].iov_len); + am_hdr_sz += am_hdr[i].iov_len; + } + + mpi_errno = MPIDI_NM_send_am(rank, comm, handler_id, am_hdr_buf, am_hdr_sz, + data, count, datatype, sreq, netmod_context); + + if (is_allocated) + MPL_free(am_hdr_buf); + else + MPIDI_CH4R_release_buf(am_hdr_buf); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_SEND_AMV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_send_amv_hdr +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_send_amv_hdr(int rank, + MPIR_Comm * comm, + int handler_id, + struct iovec *am_hdr, + size_t iov_len, MPIR_Request * sreq, void *netmod_context) +{ + int mpi_errno = MPI_SUCCESS, is_allocated; + size_t am_hdr_sz = 0, i; + char *am_hdr_buf; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_SEND_AMV_HDR); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_SEND_AMV_HDR); + + for (i = 0; i < iov_len; i++) { + am_hdr_sz += am_hdr[i].iov_len; + } + + /* TODO: avoid the malloc here, use the am_hdr directly */ + if (am_hdr_sz > MPIDI_OFI_BUF_POOL_SIZE) { + am_hdr_buf = (char *) MPL_malloc(am_hdr_sz); + is_allocated = 1; + } + else { + am_hdr_buf = (char *) MPIDI_CH4R_get_buf(MPIDI_Global.am_buf_pool); + is_allocated = 0; + } + + MPIR_Assert(am_hdr_buf); + am_hdr_sz = 0; + + for (i = 0; i < iov_len; i++) { + MPIR_Memcpy(am_hdr_buf + am_hdr_sz, am_hdr[i].iov_base, am_hdr[i].iov_len); + am_hdr_sz += am_hdr[i].iov_len; + } + + mpi_errno = MPIDI_NM_send_am_hdr(rank, comm, handler_id, am_hdr_buf, am_hdr_sz, + sreq, netmod_context); + + if (is_allocated) + MPL_free(am_hdr_buf); + else + MPIDI_CH4R_release_buf(am_hdr_buf); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_SEND_AMV_HDR); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_send_am_hdr_reply +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_send_am_hdr_reply(MPIR_Context_id_t context_id, + int src_rank, + int handler_id, + const void *am_hdr, + size_t am_hdr_sz, MPIR_Request * sreq) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_SEND_AM_HDR_REPLY); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_SEND_AM_HDR_REPLY); + mpi_errno = MPIDI_OFI_do_send_am_header(src_rank, + MPIDI_CH4U_context_id_to_comm(context_id), + handler_id, am_hdr, am_hdr_sz, sreq, TRUE); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_SEND_AM_HDR_REPLY); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_send_am_reply +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_send_am_reply(MPIR_Context_id_t context_id, + int src_rank, + int handler_id, + const void *am_hdr, + size_t am_hdr_sz, + const void *data, + MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_SEND_AM_REPLY); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_SEND_AM_REPLY); + mpi_errno = MPIDI_OFI_do_send_am(src_rank, + MPIDI_CH4U_context_id_to_comm(context_id), + handler_id, + am_hdr, am_hdr_sz, data, count, datatype, sreq, TRUE); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_SEND_AM_REPLY); + return mpi_errno; +} + +static inline int MPIDI_NM_send_amv_reply(MPIR_Context_id_t context_id, + int src_rank, + int handler_id, + struct iovec *am_hdr, + size_t iov_len, + const void *data, + MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq) +{ + int mpi_errno = MPI_SUCCESS, is_allocated; + size_t am_hdr_sz = 0, i; + char *am_hdr_buf; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_SEND_AMV_REPLY); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_SEND_AMV_REPLY); + + for (i = 0; i < iov_len; i++) { + am_hdr_sz += am_hdr[i].iov_len; + } + + /* TODO: avoid the malloc here, use the am_hdr directly */ + if (am_hdr_sz > MPIDI_OFI_BUF_POOL_SIZE) { + am_hdr_buf = (char *) MPL_malloc(am_hdr_sz); + is_allocated = 1; + } + else { + am_hdr_buf = (char *) MPIDI_CH4R_get_buf(MPIDI_Global.am_buf_pool); + is_allocated = 0; + } + + MPIR_Assert(am_hdr_buf); + am_hdr_sz = 0; + + for (i = 0; i < iov_len; i++) { + MPIR_Memcpy(am_hdr_buf + am_hdr_sz, am_hdr[i].iov_base, am_hdr[i].iov_len); + am_hdr_sz += am_hdr[i].iov_len; + } + + mpi_errno = MPIDI_NM_send_am_reply(context_id, src_rank, handler_id, am_hdr_buf, am_hdr_sz, + data, count, datatype, sreq); + + if (is_allocated) + MPL_free(am_hdr_buf); + else + MPIDI_CH4R_release_buf(am_hdr_buf); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_SEND_AMV_REPLY); + return mpi_errno; +} + +static inline size_t MPIDI_NM_am_hdr_max_sz(void) +{ + /* Maximum size that fits in short send */ + size_t max_shortsend = MPIDI_OFI_DEFAULT_SHORT_SEND_SIZE - + (sizeof(MPIDI_OFI_am_header_t) + sizeof(MPIDI_OFI_lmt_msg_payload_t)); + /* Maximum payload size representable by MPIDI_OFI_am_header_t::am_hdr_sz field */ + size_t max_representable = (1 << MPIDI_OFI_AM_HDR_SZ_BITS) - 1; + + return MPL_MIN(max_shortsend, max_representable); +} + +static inline int MPIDI_NM_inject_am_hdr(int rank, + MPIR_Comm * comm, + int handler_id, + const void *am_hdr, size_t am_hdr_sz, void *netmod_context) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_INJECT_AM_HDR); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_INJECT_AM_HDR); + mpi_errno = MPIDI_OFI_do_inject(rank, comm, + handler_id, am_hdr, am_hdr_sz, + netmod_context, FALSE, TRUE, TRUE); + + if (mpi_errno != MPI_SUCCESS) + MPIR_ERR_POP(mpi_errno); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_INJECT_AM_HDR); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +static inline int MPIDI_NM_inject_am_hdr_reply(MPIR_Context_id_t context_id, + int src_rank, + int handler_id, const void *am_hdr, size_t am_hdr_sz) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_INJECT_AM_HDR_REPLY); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_INJECT_AM_HDR_REPLY); + + mpi_errno = MPIDI_OFI_do_inject(src_rank, MPIDI_CH4U_context_id_to_comm(context_id), + handler_id, am_hdr, am_hdr_sz, NULL, TRUE, TRUE, FALSE); + + if (mpi_errno != MPI_SUCCESS) + MPIR_ERR_POP(mpi_errno); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_INJECT_AM_HDR); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +static inline size_t MPIDI_NM_am_inject_max_sz(void) +{ + if (unlikely(MPIDI_Global.max_buffered_send < sizeof(MPIDI_OFI_am_header_t))) + return 0; + return MPIDI_Global.max_buffered_send - sizeof(MPIDI_OFI_am_header_t); +} + +static inline int MPIDI_NM_am_recv(MPIR_Request * req) +{ + int mpi_errno = MPI_SUCCESS; + MPIDI_CH4U_send_long_ack_msg_t msg; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_AM_MATCHED); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_AM_MATCHED); + + msg.sreq_ptr = (MPIDI_CH4U_REQUEST(req, req->rreq.peer_req_ptr)); + msg.rreq_ptr = (uint64_t) req; + MPIR_Assert((void *) msg.sreq_ptr != NULL); + mpi_errno = MPIDI_NM_inject_am_hdr_reply(MPIDI_CH4U_get_context(MPIDI_CH4U_REQUEST(req, tag)), + MPIDI_CH4U_REQUEST(req, src_rank), + MPIDI_CH4U_SEND_LONG_ACK, &msg, sizeof(msg)); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_AM_MATCHED); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#endif /* NETMOD_OFI_AM_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/ofi/ofi_am_events.h b/src/mpid/ch4/netmod/ofi/ofi_am_events.h new file mode 100644 index 0000000..fdc920c --- /dev/null +++ b/src/mpid/ch4/netmod/ofi/ofi_am_events.h @@ -0,0 +1,414 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_AM_OFI_EVENTS_H_INCLUDED +#define NETMOD_AM_OFI_EVENTS_H_INCLUDED + +#include "ofi_am_impl.h" + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_handle_short_am +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_OFI_handle_short_am(MPIDI_OFI_am_header_t * msg_hdr) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Request *rreq; + void *p_data; + void *in_data; + + size_t data_sz, in_data_sz; + MPIDI_NM_am_completion_handler_fn cmpl_handler_fn; + struct iovec *iov; + int i, is_contig, iov_len; + size_t done, curr_len, rem; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_HANDLE_SHORT_AM); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_HANDLE_SHORT_AM); + + p_data = in_data = (char *) msg_hdr->payload + msg_hdr->am_hdr_sz; + in_data_sz = data_sz = msg_hdr->data_sz; + + MPIDI_Global.am_handlers[msg_hdr->handler_id] (msg_hdr->payload, + &p_data, &data_sz, + &is_contig, &cmpl_handler_fn, &rreq); + + if (!rreq) + goto fn_exit; + + if ((!p_data || !data_sz) && cmpl_handler_fn) { + MPIR_STATUS_SET_COUNT(rreq->status, data_sz); + cmpl_handler_fn(rreq); + goto fn_exit; + } + + if (is_contig) { + if (in_data_sz > data_sz) { + rreq->status.MPI_ERROR = MPI_ERR_TRUNCATE; + } + else { + rreq->status.MPI_ERROR = MPI_SUCCESS; + } + + data_sz = MPL_MIN(data_sz, in_data_sz); + MPIR_Memcpy(p_data, in_data, data_sz); + MPIR_STATUS_SET_COUNT(rreq->status, data_sz); + } + else { + done = 0; + rem = in_data_sz; + iov = (struct iovec *) p_data; + iov_len = data_sz; + + for (i = 0; i < iov_len && rem > 0; i++) { + curr_len = MPL_MIN(rem, iov[i].iov_len); + MPIR_Memcpy(iov[i].iov_base, (char *) in_data + done, curr_len); + rem -= curr_len; + done += curr_len; + } + + if (rem) { + rreq->status.MPI_ERROR = MPI_ERR_TRUNCATE; + } + else { + rreq->status.MPI_ERROR = MPI_SUCCESS; + } + + MPIR_STATUS_SET_COUNT(rreq->status, done); + } + + if (cmpl_handler_fn) { + cmpl_handler_fn(rreq); + } + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_HANDLE_SHORT_AM); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_handle_short_am_hdr +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_OFI_handle_short_am_hdr(MPIDI_OFI_am_header_t * msg_hdr, void *am_hdr) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Request *rreq = NULL; + MPIDI_NM_am_completion_handler_fn cmpl_handler_fn = NULL; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_HANDLE_SHORT_AM_HDR); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_HANDLE_SHORT_AM_HDR); + + MPIDI_Global.am_handlers[msg_hdr->handler_id] (am_hdr, + NULL, NULL, NULL, &cmpl_handler_fn, &rreq); + + if (!rreq) + goto fn_exit; + + if (cmpl_handler_fn) { + cmpl_handler_fn(rreq); + } + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_HANDLE_SHORT_AM_HDR); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_do_rdma_read +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_OFI_do_rdma_read(void *dst, + uint64_t src, + size_t data_sz, + MPIR_Context_id_t context_id, + int src_rank, MPIR_Request * rreq) +{ + int mpi_errno = MPI_SUCCESS; + size_t done = 0, curr_len, rem = 0; + MPIDI_OFI_am_request_t *am_req; + MPIR_Comm *comm; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_DO_RDMA_READ); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_DO_RDMA_READ); + + rem = data_sz; + + while (done != data_sz) { + curr_len = MPL_MIN(rem, MPIDI_Global.max_send); + + MPIR_Assert(sizeof(MPIDI_OFI_am_request_t) <= MPIDI_OFI_BUF_POOL_SIZE); + am_req = (MPIDI_OFI_am_request_t *) MPIDI_CH4R_get_buf(MPIDI_Global.am_buf_pool); + MPIR_Assert(am_req); + + am_req->req_hdr = MPIDI_OFI_AMREQUEST(rreq, req_hdr); + am_req->event_id = MPIDI_OFI_EVENT_AM_READ; + comm = MPIDI_CH4U_context_id_to_comm(context_id); + MPIR_Assert(comm); + MPIDI_OFI_conditional_cntr_incr(); + MPIDI_OFI_CALL_RETRY_AM(fi_read(MPIDI_OFI_EP_TX_RMA(0), + (char *) dst + done, + curr_len, NULL, + MPIDI_OFI_comm_to_phys(comm, src_rank, MPIDI_OFI_API_TAG), + src + done, + MPIDI_OFI_AMREQUEST_HDR(rreq, lmt_info).rma_key, + &am_req->context), FALSE /* no lock */ , read); + done += curr_len; + rem -= curr_len; + } + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_DO_RDMA_READ); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_handle_long_am_hdr +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_OFI_handle_long_am_hdr(MPIDI_OFI_am_header_t * msg_hdr) +{ + int mpi_errno = MPI_SUCCESS; + MPIDI_OFI_lmt_msg_payload_t *lmt_msg; + MPIR_Request *rreq; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_HANDLE_LONG_AM_HDR); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_HANDLE_LONG_AM_HDR); + + rreq = MPIR_Request_create(MPIR_REQUEST_KIND__RECV); + + mpi_errno = MPIDI_OFI_am_init_request(NULL, 0, rreq); + + if (mpi_errno != MPI_SUCCESS) + MPIR_ERR_POP(mpi_errno); + + lmt_msg = (MPIDI_OFI_lmt_msg_payload_t *) msg_hdr->payload; + MPIDI_OFI_AMREQUEST_HDR(rreq, lmt_info) = *lmt_msg; + MPIDI_OFI_AMREQUEST_HDR(rreq, msg_hdr) = *msg_hdr; + MPIDI_OFI_AMREQUEST_HDR(rreq, rreq_ptr) = (void *) rreq; + MPIDI_OFI_AMREQUEST_HDR(rreq, am_hdr) = MPL_malloc(msg_hdr->am_hdr_sz); + MPIDI_OFI_AMREQUEST_HDR(rreq, lmt_cntr) = + ((msg_hdr->am_hdr_sz - 1) / MPIDI_Global.max_send) + 1; + MPIDI_OFI_do_rdma_read(MPIDI_OFI_AMREQUEST_HDR(rreq, am_hdr), lmt_msg->am_hdr_src, + msg_hdr->am_hdr_sz, lmt_msg->context_id, lmt_msg->src_rank, rreq); + + fn_exit: + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_HANDLE_LONG_AM_HDR); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_do_handle_long_am +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_OFI_do_handle_long_am(MPIDI_OFI_am_header_t * msg_hdr, + MPIDI_OFI_lmt_msg_payload_t * lmt_msg, void *am_hdr) +{ + int num_reads, i, iov_len, c, mpi_errno = MPI_SUCCESS, is_contig = 0; + MPIR_Request *rreq; + void *p_data; + size_t data_sz, rem, done, curr_len, in_data_sz; + MPIDI_NM_am_completion_handler_fn cmpl_handler_fn; + struct iovec *iov; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_DO_HANDLE_LONG_AM); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_DO_HANDLE_LONG_AM); + + in_data_sz = data_sz = msg_hdr->data_sz; + MPIDI_Global.am_handlers[msg_hdr->handler_id] (am_hdr, + &p_data, &data_sz, &is_contig, + &cmpl_handler_fn, &rreq); + + if (!rreq) + goto fn_exit; + + MPIDI_OFI_AMREQUEST(rreq, req_hdr) = NULL; + mpi_errno = MPIDI_OFI_am_init_request(NULL, 0, rreq); + + if (mpi_errno != MPI_SUCCESS) + MPIR_ERR_POP(mpi_errno); + + MPIR_cc_incr(rreq->cc_ptr, &c); + + MPIDI_OFI_AMREQUEST_HDR(rreq, cmpl_handler_fn) = cmpl_handler_fn; + + if ((!p_data || !data_sz) && cmpl_handler_fn) { + cmpl_handler_fn(rreq); + MPIDI_OFI_am_request_complete(rreq); + goto fn_exit; + } + + MPIDI_OFI_AMREQUEST_HDR(rreq, msg_hdr) = *msg_hdr; + MPIDI_OFI_AMREQUEST_HDR(rreq, lmt_info) = *lmt_msg; + MPIDI_OFI_AMREQUEST_HDR(rreq, rreq_ptr) = (void *) rreq; + + if (is_contig) { + if (in_data_sz > data_sz) { + rreq->status.MPI_ERROR = MPI_ERR_TRUNCATE; + } + else { + rreq->status.MPI_ERROR = MPI_SUCCESS; + } + + data_sz = MPL_MIN(data_sz, in_data_sz); + MPIDI_OFI_AMREQUEST_HDR(rreq, lmt_cntr) = ((data_sz - 1) / MPIDI_Global.max_send) + 1; + MPIDI_OFI_do_rdma_read(p_data, + lmt_msg->src_offset, + data_sz, lmt_msg->context_id, lmt_msg->src_rank, rreq); + MPIR_STATUS_SET_COUNT(rreq->status, data_sz); + } + else { + done = 0; + rem = in_data_sz; + iov = (struct iovec *) p_data; + iov_len = data_sz; + + /* FIXME: optimize iov processing part */ + + /* set lmt counter */ + MPIDI_OFI_AMREQUEST_HDR(rreq, lmt_cntr) = 0; + + for (i = 0; i < iov_len && rem > 0; i++) { + curr_len = MPL_MIN(rem, iov[i].iov_len); + num_reads = ((curr_len - 1) / MPIDI_Global.max_send) + 1; + MPIDI_OFI_AMREQUEST_HDR(rreq, lmt_cntr) += num_reads; + rem -= curr_len; + } + + done = 0; + rem = in_data_sz; + + for (i = 0; i < iov_len && rem > 0; i++) { + curr_len = MPL_MIN(rem, iov[i].iov_len); + MPIDI_OFI_do_rdma_read(iov[i].iov_base, lmt_msg->src_offset + done, + curr_len, lmt_msg->context_id, lmt_msg->src_rank, rreq); + rem -= curr_len; + done += curr_len; + } + + if (rem) { + rreq->status.MPI_ERROR = MPI_ERR_TRUNCATE; + } + else { + rreq->status.MPI_ERROR = MPI_SUCCESS; + } + + MPIR_STATUS_SET_COUNT(rreq->status, done); + } + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_DO_HANDLE_LONG_AM); + return mpi_errno; + + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_handle_long_am +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_OFI_handle_long_am(MPIDI_OFI_am_header_t * msg_hdr) +{ + int mpi_errno = MPI_SUCCESS; + MPIDI_OFI_lmt_msg_payload_t *lmt_msg; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_HANDLE_LONG_AM); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_HANDLE_LONG_AM); + + lmt_msg = (MPIDI_OFI_lmt_msg_payload_t *) ((char *) msg_hdr->payload + msg_hdr->am_hdr_sz); + mpi_errno = MPIDI_OFI_do_handle_long_am(msg_hdr, lmt_msg, msg_hdr->payload); + + if (mpi_errno != MPI_SUCCESS) + MPIR_ERR_POP(mpi_errno); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_HANDLE_LONG_AM); + return mpi_errno; + + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_handle_lmt_ack +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_OFI_handle_lmt_ack(MPIDI_OFI_am_header_t * msg_hdr) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Request *sreq; + MPIDI_OFI_ack_msg_payload_t *ack_msg; + int handler_id; + uint64_t index; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_HANDLE_LMT_ACK); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_HANDLE_LMT_ACK); + + ack_msg = (MPIDI_OFI_ack_msg_payload_t *) msg_hdr->payload; + sreq = (MPIR_Request *) ack_msg->sreq_ptr; + + index = fi_mr_key(MPIDI_OFI_AMREQUEST_HDR(sreq, lmt_mr)) >> MPIDI_Global.huge_rma_shift; + MPIDI_OFI_index_allocator_free(MPIDI_OFI_COMM(MPIR_Process.comm_world).rma_id_allocator, index); + MPIDI_OFI_CALL_NOLOCK(fi_close(&MPIDI_OFI_AMREQUEST_HDR(sreq, lmt_mr)->fid), mr_unreg); + OPA_decr_int(&MPIDI_Global.am_inflight_rma_send_mrs); + + if (MPIDI_OFI_AMREQUEST_HDR(sreq, pack_buffer)) { + MPL_free(MPIDI_OFI_AMREQUEST_HDR(sreq, pack_buffer)); + } + + handler_id = MPIDI_OFI_AMREQUEST_HDR(sreq, msg_hdr).handler_id; + MPIDI_OFI_am_request_complete(sreq); + mpi_errno = MPIDI_Global.am_send_cmpl_handlers[handler_id] (sreq); + + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_HANDLE_LMT_ACK); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_dispatch_ack +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_OFI_dispatch_ack(int rank, + int context_id, + uint64_t sreq_ptr, int am_type, void *netmod_context) +{ + int mpi_errno = MPI_SUCCESS; + MPIDI_OFI_ack_msg_t msg; + MPIR_Comm *comm; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_DISPATCH_ACK); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_DISPATCH_ACK); + + comm = MPIDI_CH4U_context_id_to_comm(context_id); + + msg.hdr.am_hdr_sz = sizeof(msg.pyld); + msg.hdr.data_sz = 0; + msg.hdr.am_type = am_type; + msg.pyld.sreq_ptr = sreq_ptr; + MPIDI_OFI_CALL_RETRY_AM(fi_inject(MPIDI_OFI_EP_TX_MSG(0), &msg, sizeof(msg), + MPIDI_OFI_comm_to_phys(comm, rank, MPIDI_OFI_API_TAG)), + FALSE /* no lock */ , inject); + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_DISPATCH_ACK); + return mpi_errno; + fn_fail: + goto fn_exit; +} + + +#endif /* NETMOD_AM_OFI_EVENTS_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/ofi/ofi_am_impl.h b/src/mpid/ch4/netmod/ofi/ofi_am_impl.h new file mode 100644 index 0000000..1a9cba6 --- /dev/null +++ b/src/mpid/ch4/netmod/ofi/ofi_am_impl.h @@ -0,0 +1,565 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_OFI_AM_IMPL_H_INCLUDED +#define NETMOD_OFI_AM_IMPL_H_INCLUDED + +#include "ofi_impl.h" + +static inline int MPIDI_OFI_progress_do_queue(void *netmod_context); + +/* + Per-object lock for OFI + + * When calling OFI function MPIDI_OFI_THREAD_FI_MUTEX must be held. + * When being called from the MPI layer (app), we must grab the lock. + This is the case for regular (non-reply) functions such as send_am. + * When being called from callback function or progress engine, we must + not grab the lock because the progress engine is already holding the lock. + This is the case for reply functions such as send_am_reply. +*/ +#define MPIDI_OFI_CALL_RETRY_AM(FUNC,LOCK,STR) \ + do { \ + ssize_t _ret; \ + do { \ + if (LOCK) MPID_THREAD_CS_ENTER(POBJ,MPIDI_OFI_THREAD_FI_MUTEX); \ + _ret = FUNC; \ + if (LOCK) MPID_THREAD_CS_EXIT(POBJ,MPIDI_OFI_THREAD_FI_MUTEX); \ + if (likely(_ret==0)) break; \ + MPIR_ERR_##CHKANDJUMP4(_ret != -FI_EAGAIN, \ + mpi_errno, \ + MPI_ERR_OTHER, \ + "**ofi_"#STR, \ + "**ofi_"#STR" %s %d %s %s", \ + __SHORT_FILE__, \ + __LINE__, \ + FCNAME, \ + fi_strerror(-_ret)); \ + if (LOCK) MPID_THREAD_CS_ENTER(POBJ,MPIDI_OFI_THREAD_FI_MUTEX); \ + mpi_errno = MPIDI_OFI_progress_do_queue(NULL); \ + if (LOCK) MPID_THREAD_CS_EXIT(POBJ,MPIDI_OFI_THREAD_FI_MUTEX); \ + if (mpi_errno != MPI_SUCCESS) \ + MPIR_ERR_POP(mpi_errno); \ + } while (_ret == -FI_EAGAIN); \ + } while (0) + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_am_clear_request +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline void MPIDI_OFI_am_clear_request(MPIR_Request * sreq) +{ + MPIDI_OFI_am_request_header_t *req_hdr; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_AM_OFI_CLEAR_REQ); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_AM_OFI_CLEAR_REQ); + + req_hdr = MPIDI_OFI_AMREQUEST(sreq, req_hdr); + + if (!req_hdr) + return; + + if (req_hdr->am_hdr != &req_hdr->am_hdr_buf[0]) { + MPL_free(req_hdr->am_hdr); + } + + MPIDI_CH4R_release_buf(req_hdr); + MPIDI_OFI_AMREQUEST(sreq, req_hdr) = NULL; + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_AM_OFI_CLEAR_REQ); + return; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_am_init_request +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_OFI_am_init_request(const void *am_hdr, + size_t am_hdr_sz, MPIR_Request * sreq) +{ + int mpi_errno = MPI_SUCCESS; + MPIDI_OFI_am_request_header_t *req_hdr; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_AM_OFI_INIT_REQ); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_AM_OFI_INIT_REQ); + + if (MPIDI_OFI_AMREQUEST(sreq, req_hdr) == NULL) { + req_hdr = (MPIDI_OFI_am_request_header_t *) + MPIDI_CH4R_get_buf(MPIDI_Global.am_buf_pool); + MPIR_Assert(req_hdr); + MPIDI_OFI_AMREQUEST(sreq, req_hdr) = req_hdr; + + req_hdr->am_hdr = (void *) &req_hdr->am_hdr_buf[0]; + req_hdr->am_hdr_sz = MPIDI_OFI_MAX_AM_HDR_SIZE; + } + else { + req_hdr = MPIDI_OFI_AMREQUEST(sreq, req_hdr); + } + + if (am_hdr_sz > req_hdr->am_hdr_sz) { + if (req_hdr->am_hdr != &req_hdr->am_hdr_buf[0]) + MPL_free(req_hdr->am_hdr); + + req_hdr->am_hdr = MPL_malloc(am_hdr_sz); + MPIR_Assert(req_hdr->am_hdr); + req_hdr->am_hdr_sz = am_hdr_sz; + } + + if (am_hdr) { + MPIR_Memcpy(req_hdr->am_hdr, am_hdr, am_hdr_sz); + } + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_AM_OFI_INIT_REQ); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_repost_buffer +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_OFI_repost_buffer(void *buf, MPIR_Request * req) +{ + int mpi_errno = MPI_SUCCESS; + MPIDI_OFI_am_repost_request_t *am = (MPIDI_OFI_am_repost_request_t *) req; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_REPOST_BUFFER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_REPOST_BUFFER); + MPIDI_OFI_CALL_RETRY_AM(fi_recvmsg(MPIDI_OFI_EP_RX_MSG(0), + &MPIDI_Global.am_msg[am->index], + FI_MULTI_RECV | FI_COMPLETION), FALSE /* lock */ , repost); + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_REPOST_BUFFER); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_progress_do_queue +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_OFI_progress_do_queue(void *netmod_context) +{ + int mpi_errno = MPI_SUCCESS, ret; + struct fi_cq_tagged_entry cq_entry; + + /* Caller must hold MPIDI_OFI_THREAD_FI_MUTEX */ + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_PROGRESS_DO_QUEUE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_PROGRESS_DO_QUEUE); + + ret = fi_cq_read(MPIDI_Global.p2p_cq, &cq_entry, 1); + + if (unlikely(ret == -FI_EAGAIN)) + goto fn_exit; + + if (ret < 0) { + mpi_errno = MPIDI_OFI_handle_cq_error_util(ret); + goto fn_fail; + } + + if (((MPIDI_Global.cq_buff_head + 1) % + MPIDI_OFI_NUM_CQ_BUFFERED == MPIDI_Global.cq_buff_tail) || + !slist_empty(&MPIDI_Global.cq_buff_list)) { + MPIDI_OFI_cq_list_t *list_entry = + (MPIDI_OFI_cq_list_t *) MPL_malloc(sizeof(MPIDI_OFI_cq_list_t)); + MPIR_Assert(list_entry); + list_entry->cq_entry = cq_entry; + slist_insert_tail(&list_entry->entry, &MPIDI_Global.cq_buff_list); + } + else { + MPIDI_Global.cq_buffered[MPIDI_Global.cq_buff_head].cq_entry = cq_entry; + MPIDI_Global.cq_buff_head = (MPIDI_Global.cq_buff_head + 1) % MPIDI_OFI_NUM_CQ_BUFFERED; + } + + if ((cq_entry.flags & FI_RECV) && (cq_entry.flags & FI_MULTI_RECV)) { + mpi_errno = MPIDI_OFI_repost_buffer(cq_entry.op_context, + MPIDI_OFI_context_to_request(cq_entry.op_context)); + + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + } + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_PROGRESS_DO_QUEUE); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_do_send_am_header +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_OFI_do_send_am_header(int rank, + MPIR_Comm * comm, + int handler_id, + const void *am_hdr, + size_t am_hdr_sz, MPIR_Request * sreq, int is_reply) +{ + struct iovec iov[2]; + MPIDI_OFI_am_header_t *msg_hdr; + int mpi_errno = MPI_SUCCESS, c; + int need_lock = !is_reply; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_DO_SEND_AM_HDR); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_DO_SEND_AM_HDR); + + MPIDI_OFI_AMREQUEST(sreq, req_hdr) = NULL; + mpi_errno = MPIDI_OFI_am_init_request(am_hdr, am_hdr_sz, sreq); + + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + MPIR_Assert(handler_id < (1 << MPIDI_OFI_AM_HANDLER_ID_BITS)); + MPIR_Assert(am_hdr_sz < (1ULL << MPIDI_OFI_AM_HDR_SZ_BITS)); + msg_hdr = &MPIDI_OFI_AMREQUEST_HDR(sreq, msg_hdr); + msg_hdr->handler_id = handler_id; + msg_hdr->am_hdr_sz = am_hdr_sz; + msg_hdr->data_sz = 0; + msg_hdr->am_type = MPIDI_AMTYPE_SHORT_HDR; + + MPIR_Assert((uint64_t) comm->rank < (1ULL << MPIDI_OFI_AM_RANK_BITS)); + + MPIDI_OFI_AMREQUEST_HDR(sreq, pack_buffer) = NULL; + MPIR_cc_incr(sreq->cc_ptr, &c); + + iov[0].iov_base = msg_hdr; + iov[0].iov_len = sizeof(*msg_hdr); + + MPIR_Assert((sizeof(*msg_hdr) + am_hdr_sz) <= MPIDI_OFI_DEFAULT_SHORT_SEND_SIZE); + iov[1].iov_base = MPIDI_OFI_AMREQUEST_HDR(sreq, am_hdr); + iov[1].iov_len = am_hdr_sz; + MPIDI_OFI_AMREQUEST(sreq, event_id) = MPIDI_OFI_EVENT_AM_SEND; + MPIDI_OFI_CALL_RETRY_AM(fi_sendv(MPIDI_OFI_EP_TX_MSG(0), iov, NULL, 2, + MPIDI_OFI_comm_to_phys(comm, rank, MPIDI_OFI_API_TAG), + &MPIDI_OFI_AMREQUEST(sreq, context)), need_lock, sendv); + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_DO_SEND_AM_HDR); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_send_am_long +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_OFI_send_am_long(int rank, + MPIR_Comm * comm, + int handler_id, + const void *am_hdr, + size_t am_hdr_sz, + const void *data, + size_t data_sz, MPIR_Request * sreq, int need_lock) +{ + int mpi_errno = MPI_SUCCESS, c; + MPIDI_OFI_am_header_t *msg_hdr; + MPIDI_OFI_lmt_msg_payload_t *lmt_info; + struct iovec iov[3]; + uint64_t index; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_SEND_AM_LONG); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_SEND_AM_LONG); + + MPIR_Assert(handler_id < (1 << MPIDI_OFI_AM_HANDLER_ID_BITS)); + MPIR_Assert(am_hdr_sz < (1ULL << MPIDI_OFI_AM_HDR_SZ_BITS)); + MPIR_Assert(data_sz < (1ULL << MPIDI_OFI_AM_DATA_SZ_BITS)); + MPIR_Assert((uint64_t) comm->rank < (1ULL << MPIDI_OFI_AM_RANK_BITS)); + + msg_hdr = &MPIDI_OFI_AMREQUEST_HDR(sreq, msg_hdr); + msg_hdr->handler_id = handler_id; + msg_hdr->am_hdr_sz = am_hdr_sz; + msg_hdr->data_sz = data_sz; + msg_hdr->am_type = MPIDI_AMTYPE_LMT_REQ; + + lmt_info = &MPIDI_OFI_AMREQUEST_HDR(sreq, lmt_info); + lmt_info->context_id = comm->context_id; + lmt_info->src_rank = comm->rank; + lmt_info->src_offset = MPIDI_OFI_ENABLE_MR_SCALABLE ? (uint64_t) 0 /* MR_SCALABLE */ : (uint64_t) data; /* MR_BASIC */ + lmt_info->sreq_ptr = (uint64_t) sreq; + /* Always allocates RMA ID from COMM_WORLD as the actual associated communicator + * is not available here */ + index = + MPIDI_OFI_index_allocator_alloc(MPIDI_OFI_COMM(MPIR_Process.comm_world).rma_id_allocator); + MPIR_Assert((int) index < MPIDI_Global.max_huge_rmas); + lmt_info->rma_key = MPIDI_OFI_ENABLE_MR_SCALABLE ? index << MPIDI_Global.huge_rma_shift : 0; + + MPIR_cc_incr(sreq->cc_ptr, &c); /* send completion */ + MPIR_cc_incr(sreq->cc_ptr, &c); /* lmt ack handler */ + MPIR_Assert((sizeof(*msg_hdr) + sizeof(*lmt_info) + am_hdr_sz) <= + MPIDI_OFI_DEFAULT_SHORT_SEND_SIZE); + if (need_lock) + MPIDI_OFI_CALL(fi_mr_reg(MPIDI_Global.domain, + data, + data_sz, + FI_REMOTE_READ, + 0ULL, + lmt_info->rma_key, + 0ULL, &MPIDI_OFI_AMREQUEST_HDR(sreq, lmt_mr), NULL), mr_reg); + else + MPIDI_OFI_CALL_NOLOCK(fi_mr_reg(MPIDI_Global.domain, + data, + data_sz, + FI_REMOTE_READ, + 0ULL, + lmt_info->rma_key, + 0ULL, + &MPIDI_OFI_AMREQUEST_HDR(sreq, lmt_mr), NULL), mr_reg); + OPA_incr_int(&MPIDI_Global.am_inflight_rma_send_mrs); + + if (!MPIDI_OFI_ENABLE_MR_SCALABLE) { + /* MR_BASIC */ + lmt_info->rma_key = fi_mr_key(MPIDI_OFI_AMREQUEST_HDR(sreq, lmt_mr)); + } + + iov[0].iov_base = msg_hdr; + iov[0].iov_len = sizeof(*msg_hdr); + + iov[1].iov_base = MPIDI_OFI_AMREQUEST_HDR(sreq, am_hdr); + iov[1].iov_len = am_hdr_sz; + + iov[2].iov_base = lmt_info; + iov[2].iov_len = sizeof(*lmt_info); + MPIDI_OFI_AMREQUEST(sreq, event_id) = MPIDI_OFI_EVENT_AM_SEND; + MPIDI_OFI_CALL_RETRY_AM(fi_sendv(MPIDI_OFI_EP_TX_MSG(0), iov, NULL, 3, + MPIDI_OFI_comm_to_phys(comm, rank, MPIDI_OFI_API_TAG), + &MPIDI_OFI_AMREQUEST(sreq, context)), need_lock, sendv); + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_SEND_AM_LONG); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_send_am_short +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_OFI_send_am_short(int rank, + MPIR_Comm * comm, + int handler_id, + const void *am_hdr, + size_t am_hdr_sz, + const void *data, + MPI_Count count, MPIR_Request * sreq, int need_lock) +{ + int mpi_errno = MPI_SUCCESS, c; + MPIDI_OFI_am_header_t *msg_hdr; + struct iovec iov[3]; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_SEND_AM_SHORT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_SEND_AM_SHORT); + + MPIR_Assert(handler_id < (1 << MPIDI_OFI_AM_HANDLER_ID_BITS)); + MPIR_Assert(am_hdr_sz < (1ULL << MPIDI_OFI_AM_HDR_SZ_BITS)); + MPIR_Assert((uint64_t) count < (1ULL << MPIDI_OFI_AM_DATA_SZ_BITS)); + MPIR_Assert((uint64_t) comm->rank < (1ULL << MPIDI_OFI_AM_RANK_BITS)); + + msg_hdr = &MPIDI_OFI_AMREQUEST_HDR(sreq, msg_hdr); + msg_hdr->handler_id = handler_id; + msg_hdr->am_hdr_sz = am_hdr_sz; + msg_hdr->data_sz = count; + msg_hdr->am_type = MPIDI_AMTYPE_SHORT; + + iov[0].iov_base = msg_hdr; + iov[0].iov_len = sizeof(*msg_hdr); + + iov[1].iov_base = MPIDI_OFI_AMREQUEST_HDR(sreq, am_hdr); + iov[1].iov_len = am_hdr_sz; + + iov[2].iov_base = (void *) data; + iov[2].iov_len = count; + + MPIR_cc_incr(sreq->cc_ptr, &c); + MPIDI_OFI_AMREQUEST(sreq, event_id) = MPIDI_OFI_EVENT_AM_SEND; + MPIDI_OFI_CALL_RETRY_AM(fi_sendv(MPIDI_OFI_EP_TX_MSG(0), iov, NULL, 3, + MPIDI_OFI_comm_to_phys(comm, rank, MPIDI_OFI_API_TAG), + &MPIDI_OFI_AMREQUEST(sreq, context)), need_lock, sendv); + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_SEND_AM_SHORT); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_do_send_am +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_OFI_do_send_am(int rank, + MPIR_Comm * comm, + int handler_id, + const void *am_hdr, + size_t am_hdr_sz, + const void *buf, + size_t count, + MPI_Datatype datatype, MPIR_Request * sreq, int is_reply) +{ + int dt_contig, mpi_errno = MPI_SUCCESS; + char *send_buf; + size_t data_sz; + MPI_Aint dt_true_lb, last; + MPIR_Datatype *dt_ptr; + int need_lock = !is_reply; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_DO_SEND_AM); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_DO_SEND_AM); + + MPIDI_Datatype_get_info(count, datatype, dt_contig, data_sz, dt_ptr, dt_true_lb); + send_buf = (char *) buf + dt_true_lb; + + if (handler_id == MPIDI_CH4U_SEND && + am_hdr_sz + data_sz + sizeof(MPIDI_OFI_am_header_t) > MPIDI_OFI_DEFAULT_SHORT_SEND_SIZE) { + MPIDI_CH4U_send_long_req_msg_t lreq_hdr; + + MPIR_Memcpy(&lreq_hdr.hdr, am_hdr, am_hdr_sz); + lreq_hdr.data_sz = data_sz; + lreq_hdr.sreq_ptr = (uint64_t) sreq; + MPIDI_CH4U_REQUEST(sreq, req->lreq).src_buf = buf; + MPIDI_CH4U_REQUEST(sreq, req->lreq).count = count; + dtype_add_ref_if_not_builtin(datatype); + MPIDI_CH4U_REQUEST(sreq, req->lreq).datatype = datatype; + MPIDI_CH4U_REQUEST(sreq, req->lreq).msg_tag = lreq_hdr.hdr.msg_tag; + MPIDI_CH4U_REQUEST(sreq, src_rank) = rank; + mpi_errno = MPIDI_NM_inject_am_hdr(rank, comm, MPIDI_CH4U_SEND_LONG_REQ, + &lreq_hdr, sizeof(lreq_hdr), NULL); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + goto fn_exit; + } + + MPIDI_OFI_AMREQUEST(sreq, req_hdr) = NULL; + mpi_errno = MPIDI_OFI_am_init_request(am_hdr, am_hdr_sz, sreq); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + if (!dt_contig) { + size_t segment_first; + struct MPIDU_Segment *segment_ptr; + segment_ptr = MPIDU_Segment_alloc(); + MPIR_ERR_CHKANDJUMP1(segment_ptr == NULL, mpi_errno, + MPI_ERR_OTHER, "**nomem", "**nomem %s", "Send MPIDU_Segment_alloc"); + MPIDU_Segment_init(buf, count, datatype, segment_ptr, 0); + segment_first = 0; + last = data_sz; + MPIDI_OFI_AMREQUEST_HDR(sreq, pack_buffer) = (char *) MPL_malloc(data_sz); + MPIR_ERR_CHKANDJUMP1(MPIDI_OFI_AMREQUEST_HDR(sreq, pack_buffer) == NULL, mpi_errno, + MPI_ERR_OTHER, "**nomem", "**nomem %s", "Send Pack buffer alloc"); + MPIDU_Segment_pack(segment_ptr, segment_first, &last, + MPIDI_OFI_AMREQUEST_HDR(sreq, pack_buffer)); + MPIDU_Segment_free(segment_ptr); + send_buf = (char *) MPIDI_OFI_AMREQUEST_HDR(sreq, pack_buffer); + } + else { + MPIDI_OFI_AMREQUEST_HDR(sreq, pack_buffer) = NULL; + } + + if (am_hdr_sz + data_sz + sizeof(MPIDI_OFI_am_header_t) <= MPIDI_OFI_DEFAULT_SHORT_SEND_SIZE) { + mpi_errno = + MPIDI_OFI_send_am_short(rank, comm, handler_id, MPIDI_OFI_AMREQUEST_HDR(sreq, am_hdr), + am_hdr_sz, send_buf, data_sz, sreq, need_lock); + } + else { + mpi_errno = + MPIDI_OFI_send_am_long(rank, comm, handler_id, MPIDI_OFI_AMREQUEST_HDR(sreq, am_hdr), + am_hdr_sz, send_buf, data_sz, sreq, need_lock); + } + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_DO_SEND_AM); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +static inline int MPIDI_OFI_do_inject(int rank, + MPIR_Comm * comm, + int handler_id, + const void *am_hdr, + size_t am_hdr_sz, + void *netmod_context, + int is_reply, int use_comm_table, int need_lock) +{ + int mpi_errno = MPI_SUCCESS; + MPIDI_OFI_am_header_t msg_hdr; + struct fi_msg msg; + struct iovec msg_iov[2]; + uint64_t send_flag = FI_INJECT; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_DO_INJECT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_DO_INJECT); + + MPIR_Assert(handler_id < (1 << MPIDI_OFI_AM_HANDLER_ID_BITS)); + MPIR_Assert(am_hdr_sz < (1ULL << MPIDI_OFI_AM_HDR_SZ_BITS)); + + msg_hdr.handler_id = handler_id; + msg_hdr.am_hdr_sz = am_hdr_sz; + msg_hdr.data_sz = 0; + msg_hdr.am_type = MPIDI_AMTYPE_SHORT_HDR; + + MPIR_Assert((uint64_t) comm->rank < (1ULL << MPIDI_OFI_AM_RANK_BITS)); + + msg_iov[0].iov_base = (void *) &msg_hdr; + msg_iov[0].iov_len = sizeof(msg_hdr); + + msg_iov[1].iov_base = (void *) am_hdr; + msg_iov[1].iov_len = am_hdr_sz; + + msg.msg_iov = &msg_iov[0]; + msg.desc = NULL; + msg.iov_count = 2; + msg.context = NULL; + msg.addr = use_comm_table ? + MPIDI_OFI_comm_to_phys(comm, rank, MPIDI_OFI_API_MSG) : + MPIDI_OFI_to_phys(rank, MPIDI_OFI_API_MSG); + + if (unlikely(am_hdr_sz + sizeof(msg_hdr) > MPIDI_Global.max_buffered_send)) { + MPIR_Request *sreq; + char *ibuf; + + sreq = MPIR_Request_create(MPIR_REQUEST_KIND__SEND); + MPIR_Assert(sreq); + ibuf = (char *) MPL_malloc(am_hdr_sz + sizeof(msg_hdr)); + MPIR_Assert(ibuf); + memcpy(ibuf, &msg_hdr, sizeof(msg_hdr)); + memcpy(ibuf + sizeof(msg_hdr), am_hdr, am_hdr_sz); + msg_iov[0].iov_base = ibuf; + msg_iov[0].iov_len = am_hdr_sz + sizeof(msg_hdr); + msg.iov_count = 1; + + MPIDI_OFI_REQUEST(sreq, event_id) = MPIDI_OFI_EVENT_INJECT_EMU; + MPIDI_OFI_REQUEST(sreq, util.inject_buf) = ibuf; + /* Cancel FI_INJECT and ask for completion event */ + send_flag = FI_COMPLETION; + msg.context = (void *) &(MPIDI_OFI_REQUEST(sreq, context)); + OPA_incr_int(&MPIDI_Global.am_inflight_inject_emus); + } + + MPIDI_OFI_CALL_RETRY_AM(fi_sendmsg(MPIDI_OFI_EP_TX_MSG(0), &msg, send_flag), need_lock, send); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_DO_INJECT); + return mpi_errno; + fn_fail: + goto fn_exit; +} + + +static inline void MPIDI_OFI_am_request_complete(MPIR_Request * req) +{ + int incomplete; + MPIR_cc_decr(req->cc_ptr, &incomplete); + + if (!incomplete) { + MPIDI_CH4U_request_release(req); + } +} + +#endif /*NETMOD_OFI_AM_IMPL_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/ofi/ofi_am_probe.h b/src/mpid/ch4/netmod/ofi/ofi_am_probe.h new file mode 100644 index 0000000..e8575ec --- /dev/null +++ b/src/mpid/ch4/netmod/ofi/ofi_am_probe.h @@ -0,0 +1,39 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_AM_OFI_PROBE_H_INCLUDED +#define NETMOD_AM_OFI_PROBE_H_INCLUDED + +#include "ofi_impl.h" + +static inline int MPIDI_NM_probe(int source, + int tag, MPIR_Comm * comm, int context_offset, MPI_Status * status) +{ + return MPIDI_CH4U_probe(source, tag, comm, context_offset, status); +} + +static inline int MPIDI_NM_improbe(int source, + int tag, + MPIR_Comm * comm, + int context_offset, + int *flag, MPIR_Request ** message, MPI_Status * status) +{ + return MPIDI_CH4U_improbe(source, tag, comm, context_offset, flag, message, status); +} + +static inline int MPIDI_NM_iprobe(int source, + int tag, + MPIR_Comm * comm, + int context_offset, int *flag, MPI_Status * status) +{ + return MPIDI_CH4U_iprobe(source, tag, comm, context_offset, flag, status); +} + +#endif /* NETMOD_AM_OFI_PROBE_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/ofi/ofi_am_recv.h b/src/mpid/ch4/netmod/ofi/ofi_am_recv.h new file mode 100644 index 0000000..0e844db --- /dev/null +++ b/src/mpid/ch4/netmod/ofi/ofi_am_recv.h @@ -0,0 +1,61 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_AM_OFI_RECV_H_INCLUDED +#define NETMOD_AM_OFI_RECV_H_INCLUDED + +#include "ofi_impl.h" + +static inline int MPIDI_NM_recv(void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, + int context_offset, MPI_Status * status, MPIR_Request ** request) +{ + return MPIDI_CH4U_recv(buf, count, datatype, rank, tag, comm, context_offset, status, request); +} + +static inline int MPIDI_NM_recv_init(void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + return MPIDI_CH4U_recv_init(buf, count, datatype, rank, tag, comm, context_offset, request); +} + +static inline int MPIDI_NM_imrecv(void *buf, + int count, + MPI_Datatype datatype, + MPIR_Request * message, MPIR_Request ** rreqp) +{ + return MPIDI_CH4U_imrecv(buf, count, datatype, message, rreqp); +} + +static inline int MPIDI_NM_irecv(void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + return MPIDI_CH4U_irecv(buf, count, datatype, rank, tag, comm, context_offset, request); + +} + +static inline int MPIDI_NM_cancel_recv(MPIR_Request * rreq) +{ + return MPIDI_CH4U_cancel_recv(rreq); +} + +#endif /* NETMOD_AM_OFI_RECV_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/ofi/ofi_am_rma.h b/src/mpid/ch4/netmod/ofi/ofi_am_rma.h new file mode 100644 index 0000000..e93b892 --- /dev/null +++ b/src/mpid/ch4/netmod/ofi/ofi_am_rma.h @@ -0,0 +1,148 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_AM_OFI_RMA_H_INCLUDED +#define NETMOD_AM_OFI_RMA_H_INCLUDED + +#include "ofi_impl.h" + +static inline int MPIDI_NM_put(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, MPI_Datatype target_datatype, MPIR_Win * win) +{ + return MPIDI_CH4U_put(origin_addr, origin_count, origin_datatype, + target_rank, target_disp, target_count, target_datatype, win); +} + +static inline int MPIDI_NM_get(void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, MPI_Datatype target_datatype, MPIR_Win * win) +{ + return MPIDI_CH4U_get(origin_addr, origin_count, origin_datatype, + target_rank, target_disp, target_count, target_datatype, win); +} + +static inline int MPIDI_NM_rput(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPIR_Win * win, MPIR_Request ** request) +{ + return MPIDI_CH4U_rput(origin_addr, origin_count, origin_datatype, + target_rank, target_disp, target_count, target_datatype, win, request); +} + + +static inline int MPIDI_NM_compare_and_swap(const void *origin_addr, + const void *compare_addr, + void *result_addr, + MPI_Datatype datatype, + int target_rank, MPI_Aint target_disp, MPIR_Win * win) +{ + return MPIDI_CH4U_compare_and_swap(origin_addr, compare_addr, result_addr, + datatype, target_rank, target_disp, win); +} + +static inline int MPIDI_NM_raccumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPI_Op op, MPIR_Win * win, MPIR_Request ** request) +{ + return MPIDI_CH4U_raccumulate(origin_addr, origin_count, origin_datatype, + target_rank, target_disp, target_count, + target_datatype, op, win, request); +} + +static inline int MPIDI_NM_rget_accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + void *result_addr, + int result_count, + MPI_Datatype result_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPI_Op op, MPIR_Win * win, MPIR_Request ** request) +{ + return MPIDI_CH4U_rget_accumulate(origin_addr, origin_count, origin_datatype, + result_addr, result_count, result_datatype, + target_rank, target_disp, target_count, + target_datatype, op, win, request); +} + +static inline int MPIDI_NM_fetch_and_op(const void *origin_addr, + void *result_addr, + MPI_Datatype datatype, + int target_rank, + MPI_Aint target_disp, MPI_Op op, MPIR_Win * win) +{ + return MPIDI_CH4U_fetch_and_op(origin_addr, result_addr, datatype, + target_rank, target_disp, op, win); +} + + +static inline int MPIDI_NM_rget(void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPIR_Win * win, MPIR_Request ** request) +{ + return MPIDI_CH4U_rget(origin_addr, origin_count, origin_datatype, + target_rank, target_disp, target_count, target_datatype, win, request); +} + + +static inline int MPIDI_NM_get_accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + void *result_addr, + int result_count, + MPI_Datatype result_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, MPI_Op op, MPIR_Win * win) +{ + return MPIDI_CH4U_get_accumulate(origin_addr, origin_count, origin_datatype, + result_addr, result_count, result_datatype, + target_rank, target_disp, target_count, + target_datatype, op, win); +} + +static inline int MPIDI_NM_accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, MPI_Op op, MPIR_Win * win) +{ + return MPIDI_CH4U_accumulate(origin_addr, origin_count, origin_datatype, + target_rank, target_disp, target_count, target_datatype, op, win); +} + +#endif /* NETMOD_AM_OFI_RMA_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/ofi/ofi_am_send.h b/src/mpid/ch4/netmod/ofi/ofi_am_send.h new file mode 100644 index 0000000..777377a --- /dev/null +++ b/src/mpid/ch4/netmod/ofi/ofi_am_send.h @@ -0,0 +1,128 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_AM_OFI_SEND_H_INCLUDED +#define NETMOD_AM_OFI_SEND_H_INCLUDED + +#include "ofi_impl.h" + +static inline int MPIDI_NM_send(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + return MPIDI_CH4U_send(buf, count, datatype, rank, tag, comm, context_offset, request); +} + +static inline int MPIDI_NM_rsend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + return MPIDI_CH4U_rsend(buf, count, datatype, rank, tag, comm, context_offset, request); +} + + + +static inline int MPIDI_NM_irsend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + return MPIDI_CH4U_irsend(buf, count, datatype, rank, tag, comm, context_offset, request); +} + +static inline int MPIDI_NM_ssend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + return MPIDI_CH4U_ssend(buf, count, datatype, rank, tag, comm, context_offset, request); +} + +static inline int MPIDI_NM_startall(int count, MPIR_Request * requests[]) +{ + return MPIDI_CH4U_startall(count, requests); +} + +static inline int MPIDI_NM_send_init(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + return MPIDI_CH4U_send_init(buf, count, datatype, rank, tag, comm, context_offset, request); +} + +static inline int MPIDI_NM_ssend_init(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + return MPIDI_CH4U_ssend_init(buf, count, datatype, rank, tag, comm, context_offset, request); +} + +static inline int MPIDI_NM_bsend_init(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + return MPIDI_CH4U_bsend_init(buf, count, datatype, rank, tag, comm, context_offset, request); +} + +static inline int MPIDI_NM_rsend_init(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + return MPIDI_CH4U_rsend_init(buf, count, datatype, rank, tag, comm, context_offset, request); +} + +static inline int MPIDI_NM_isend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + return MPIDI_CH4U_isend(buf, count, datatype, rank, tag, comm, context_offset, request); +} + +static inline int MPIDI_NM_issend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + return MPIDI_CH4U_issend(buf, count, datatype, rank, tag, comm, context_offset, request); +} + +static inline int MPIDI_NM_cancel_send(MPIR_Request * sreq) +{ + return MPIDI_CH4U_cancel_send(sreq); +} + +#endif /* NETMOD_AM_OFI_SEND_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/ofi/ofi_am_spawn.h b/src/mpid/ch4/netmod/ofi/ofi_am_spawn.h new file mode 100644 index 0000000..75a5ef5 --- /dev/null +++ b/src/mpid/ch4/netmod/ofi/ofi_am_spawn.h @@ -0,0 +1,50 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_AM_OFI_DYNPROC_H_INCLUDED +#define NETMOD_AM_OFI_DYNPROC_H_INCLUDED + +#include "ofi_impl.h" + +static inline int MPIDI_NM_comm_connect(const char *port_name, + MPIR_Info * info, + int root, MPIR_Comm * comm, MPIR_Comm ** newcomm_ptr) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_comm_disconnect(MPIR_Comm * comm_ptr) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_open_port(MPIR_Info * info_ptr, char *port_name) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_close_port(const char *port_name) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_comm_accept(const char *port_name, + MPIR_Info * info, + int root, MPIR_Comm * comm, MPIR_Comm ** newcomm_ptr) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +#endif /* NETMOD_AM_OFI_WIN_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/ofi/ofi_am_win.h b/src/mpid/ch4/netmod/ofi/ofi_am_win.h new file mode 100644 index 0000000..89e5081 --- /dev/null +++ b/src/mpid/ch4/netmod/ofi/ofi_am_win.h @@ -0,0 +1,160 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_AM_OFI_WIN_H_INCLUDED +#define NETMOD_AM_OFI_WIN_H_INCLUDED + +#include "ofi_impl.h" + +static inline int MPIDI_NM_win_set_info(MPIR_Win * win, MPIR_Info * info) +{ + return MPIDI_CH4R_win_set_info(win, info); +} + + +static inline int MPIDI_NM_win_start(MPIR_Group * group, int assert, MPIR_Win * win) +{ + return MPIDI_CH4R_win_start(group, assert, win); +} + + +static inline int MPIDI_NM_win_complete(MPIR_Win * win) +{ + return MPIDI_CH4R_win_complete(win); +} + +static inline int MPIDI_NM_win_post(MPIR_Group * group, int assert, MPIR_Win * win) +{ + return MPIDI_CH4R_win_post(group, assert, win); +} + + +static inline int MPIDI_NM_win_wait(MPIR_Win * win) +{ + return MPIDI_CH4R_win_wait(win); +} + + +static inline int MPIDI_NM_win_test(MPIR_Win * win, int *flag) +{ + return MPIDI_CH4R_win_test(win, flag); +} + +static inline int MPIDI_NM_win_lock(int lock_type, int rank, int assert, MPIR_Win * win) +{ + return MPIDI_CH4R_win_lock(lock_type, rank, assert, win); +} + + +static inline int MPIDI_NM_win_unlock(int rank, MPIR_Win * win) +{ + return MPIDI_CH4R_win_unlock(rank, win); +} + +static inline int MPIDI_NM_win_get_info(MPIR_Win * win, MPIR_Info ** info_p_p) +{ + return MPIDI_CH4R_win_get_info(win, info_p_p); +} + + +static inline int MPIDI_NM_win_free(MPIR_Win ** win_ptr) +{ + return MPIDI_CH4R_win_free(win_ptr); +} + +static inline int MPIDI_NM_win_fence(int assert, MPIR_Win * win) +{ + return MPIDI_CH4R_win_fence(assert, win); +} + +static inline int MPIDI_NM_win_create(void *base, + MPI_Aint length, + int disp_unit, + MPIR_Info * info, MPIR_Comm * comm_ptr, MPIR_Win ** win_ptr) +{ + return MPIDI_CH4R_win_create(base, length, disp_unit, info, comm_ptr, win_ptr); +} + +static inline int MPIDI_NM_win_attach(MPIR_Win * win, void *base, MPI_Aint size) +{ + return MPIDI_CH4R_win_attach(win, base, size); +} + +static inline int MPIDI_NM_win_allocate_shared(MPI_Aint size, + int disp_unit, + MPIR_Info * info_ptr, + MPIR_Comm * comm_ptr, + void **base_ptr, MPIR_Win ** win_ptr) +{ + return MPIDI_CH4R_win_allocate_shared(size, disp_unit, info_ptr, comm_ptr, base_ptr, win_ptr); +} + +static inline int MPIDI_NM_win_detach(MPIR_Win * win, const void *base) +{ + return MPIDI_CH4R_win_detach(win, base); +} + +static inline int MPIDI_NM_win_shared_query(MPIR_Win * win, + int rank, + MPI_Aint * size, int *disp_unit, void *baseptr) +{ + return MPIDI_CH4R_win_shared_query(win, rank, size, disp_unit, baseptr); +} + +static inline int MPIDI_NM_win_allocate(MPI_Aint size, + int disp_unit, + MPIR_Info * info, + MPIR_Comm * comm, void *baseptr, MPIR_Win ** win) +{ + return MPIDI_CH4R_win_allocate(size, disp_unit, info, comm, baseptr, win); +} + +static inline int MPIDI_NM_win_flush(int rank, MPIR_Win * win) +{ + return MPIDI_CH4R_win_flush(rank, win); +} + +static inline int MPIDI_NM_win_flush_local_all(MPIR_Win * win) +{ + return MPIDI_CH4R_win_flush_local_all(win); +} + +static inline int MPIDI_NM_win_unlock_all(MPIR_Win * win) +{ + return MPIDI_CH4R_win_unlock_all(win); +} + +static inline int MPIDI_NM_win_create_dynamic(MPIR_Info * info, MPIR_Comm * comm, MPIR_Win ** win) +{ + return MPIDI_CH4R_win_create_dynamic(info, comm, win); +} + +static inline int MPIDI_NM_win_flush_local(int rank, MPIR_Win * win) +{ + return MPIDI_CH4R_win_flush_local(rank, win); +} + +static inline int MPIDI_NM_win_sync(MPIR_Win * win) +{ + return MPIDI_CH4R_win_sync(win); +} + +static inline int MPIDI_NM_win_flush_all(MPIR_Win * win) +{ + return MPIDI_CH4R_win_flush_all(win); +} + +static inline int MPIDI_NM_win_lock_all(int assert, MPIR_Win * win) +{ + return MPIDI_CH4R_win_lock_all(assert, win); +} + + +#endif /* NETMOD_AM_OFI_WIN_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/ofi/ofi_coll.h b/src/mpid/ch4/netmod/ofi/ofi_coll.h new file mode 100644 index 0000000..1f87481 --- /dev/null +++ b/src/mpid/ch4/netmod/ofi/ofi_coll.h @@ -0,0 +1,869 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_OFI_COLL_H_INCLUDED +#define NETMOD_OFI_COLL_H_INCLUDED + +#include "ofi_impl.h" + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_barrier +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_barrier(MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_BARRIER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_BARRIER); + + mpi_errno = MPIR_Barrier(comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_BARRIER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_bcast +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_bcast(void *buffer, int count, MPI_Datatype datatype, + int root, MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_BCAST); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_BCAST); + + mpi_errno = MPIR_Bcast(buffer, count, datatype, root, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_BCAST); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_allreduce +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_allreduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_ALLREDUCE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_ALLREDUCE); + + mpi_errno = MPIR_Allreduce(sendbuf, recvbuf, count, datatype, op, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_ALLREDUCE); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_allgather +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_allgather(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_ALLGATHER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_ALLGATHER); + + mpi_errno = MPIR_Allgather(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, + comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_ALLGATHER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_allgatherv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_allgatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, const int *displs, + MPI_Datatype recvtype, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_ALLGATHERV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_ALLGATHERV); + + mpi_errno = MPIR_Allgatherv(sendbuf, sendcount, sendtype, + recvbuf, recvcounts, displs, recvtype, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_ALLGATHERV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_gather +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_gather(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + int root, MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_GATHER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_GATHER); + + mpi_errno = MPIR_Gather(sendbuf, sendcount, sendtype, recvbuf, recvcount, + recvtype, root, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_GATHER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_gatherv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_gatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, const int *displs, + MPI_Datatype recvtype, int root, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_GATHERV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_GATHERV); + + mpi_errno = MPIR_Gatherv(sendbuf, sendcount, sendtype, + recvbuf, recvcounts, displs, recvtype, root, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_GATHERV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_scatter +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_scatter(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + int root, MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_SCATTER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_SCATTER); + + mpi_errno = MPIR_Scatter(sendbuf, sendcount, sendtype, + recvbuf, recvcount, recvtype, root, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_SCATTER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_scatterv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_scatterv(const void *sendbuf, const int *sendcounts, + const int *displs, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + int root, MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_SCATTERV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_SCATTERV); + + mpi_errno = MPIR_Scatterv(sendbuf, sendcounts, displs, + sendtype, recvbuf, recvcount, recvtype, root, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_SCATTERV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_alltoall +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_alltoall(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_ALLTOALL); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_ALLTOALL); + + mpi_errno = MPIR_Alltoall(sendbuf, sendcount, sendtype, recvbuf, recvcount, + recvtype, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_ALLTOALL); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_alltoallv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_alltoallv(const void *sendbuf, const int *sendcounts, + const int *sdispls, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, + const int *rdispls, MPI_Datatype recvtype, + MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_ALLTOALLV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_ALLTOALLV); + + mpi_errno = MPIR_Alltoallv(sendbuf, sendcounts, sdispls, + sendtype, recvbuf, recvcounts, rdispls, recvtype, comm_ptr, errflag); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_ALLTOALLV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_alltoallw +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_alltoallw(const void *sendbuf, const int sendcounts[], + const int sdispls[], const MPI_Datatype sendtypes[], + void *recvbuf, const int recvcounts[], + const int rdispls[], const MPI_Datatype recvtypes[], + MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_ALLTOALLW); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_ALLTOALLW); + + mpi_errno = MPIR_Alltoallw(sendbuf, sendcounts, sdispls, + sendtypes, recvbuf, recvcounts, + rdispls, recvtypes, comm_ptr, errflag); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_ALLTOALLW); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_reduce +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_reduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, int root, + MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_REDUCE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_REDUCE); + + mpi_errno = MPIR_Reduce(sendbuf, recvbuf, count, datatype, op, root, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_REDUCE); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_reduce_scatter +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_reduce_scatter(const void *sendbuf, void *recvbuf, + const int recvcounts[], MPI_Datatype datatype, + MPI_Op op, MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_REDUCE_SCATTER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_REDUCE_SCATTER); + + mpi_errno = MPIR_Reduce_scatter(sendbuf, recvbuf, recvcounts, datatype, op, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_REDUCE_SCATTER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_reduce_scatter_block +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_reduce_scatter_block(const void *sendbuf, void *recvbuf, + int recvcount, MPI_Datatype datatype, + MPI_Op op, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_REDUCE_SCATTER_BLOCK); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_REDUCE_SCATTER_BLOCK); + + mpi_errno = MPIR_Reduce_scatter_block(sendbuf, recvbuf, recvcount, + datatype, op, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_REDUCE_SCATTER_BLOCK); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_scan +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_scan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_SCAN); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_SCAN); + + mpi_errno = MPIR_Scan(sendbuf, recvbuf, count, datatype, op, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_SCAN); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_exscan +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_exscan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_EXSCAN); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_EXSCAN); + + mpi_errno = MPIR_Exscan(sendbuf, recvbuf, count, datatype, op, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_EXSCAN); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_neighbor_allgather +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_neighbor_allgather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, int recvcount, + MPI_Datatype recvtype, MPIR_Comm * comm_ptr) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_NEIGHBOR_ALLGATHER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_NEIGHBOR_ALLGATHER); + + mpi_errno = + MPIR_Neighbor_allgather_impl(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, + comm_ptr); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_NEIGHBOR_ALLGATHER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_neighbor_allgatherv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_neighbor_allgatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int recvcounts[], const int displs[], + MPI_Datatype recvtype, MPIR_Comm * comm_ptr) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_NEIGHBOR_ALLGATHERV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_NEIGHBOR_ALLGATHERV); + + mpi_errno = MPIR_Neighbor_allgatherv_impl(sendbuf, sendcount, sendtype, + recvbuf, recvcounts, displs, recvtype, comm_ptr); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_NEIGHBOR_ALLGATHERV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_neighbor_alltoall +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_neighbor_alltoall(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, int recvcount, + MPI_Datatype recvtype, MPIR_Comm * comm_ptr) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_NEIGHBOR_ALLTOALL); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_NEIGHBOR_ALLTOALL); + + mpi_errno = MPIR_Neighbor_alltoall_impl(sendbuf, sendcount, sendtype, + recvbuf, recvcount, recvtype, comm_ptr); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_NEIGHBOR_ALLTOALL); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_neighbor_alltoallv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_neighbor_alltoallv(const void *sendbuf, const int sendcounts[], + const int sdispls[], MPI_Datatype sendtype, + void *recvbuf, const int recvcounts[], + const int rdispls[], MPI_Datatype recvtype, + MPIR_Comm * comm_ptr) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_NEIGHBOR_ALLTOALLV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_NEIGHBOR_ALLTOALLV); + + mpi_errno = MPIR_Neighbor_alltoallv_impl(sendbuf, sendcounts, sdispls, sendtype, + recvbuf, recvcounts, rdispls, recvtype, comm_ptr); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_NEIGHBOR_ALLTOALLV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_neighbor_alltoallw +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_neighbor_alltoallw(const void *sendbuf, const int sendcounts[], + const MPI_Aint sdispls[], + const MPI_Datatype sendtypes[], void *recvbuf, + const int recvcounts[], const MPI_Aint rdispls[], + const MPI_Datatype recvtypes[], MPIR_Comm * comm_ptr) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_NEIGHBOR_ALLTOALLW); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_NEIGHBOR_ALLTOALLW); + + mpi_errno = MPIR_Neighbor_alltoallw_impl(sendbuf, sendcounts, sdispls, sendtypes, + recvbuf, recvcounts, rdispls, recvtypes, comm_ptr); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_NEIGHBOR_ALLTOALLW); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ineighbor_allgather +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ineighbor_allgather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, int recvcount, + MPI_Datatype recvtype, MPIR_Comm * comm_ptr, + MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_INEIGHBOR_ALLGATHER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_INEIGHBOR_ALLGATHER); + + mpi_errno = MPIR_Ineighbor_allgather_impl(sendbuf, sendcount, sendtype, + recvbuf, recvcount, recvtype, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_INEIGHBOR_ALLGATHER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ineighbor_allgatherv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ineighbor_allgatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int recvcounts[], const int displs[], + MPI_Datatype recvtype, MPIR_Comm * comm_ptr, + MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_INEIGHBOR_ALLGATHERV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_INEIGHBOR_ALLGATHERV); + + mpi_errno = MPIR_Ineighbor_allgatherv_impl(sendbuf, sendcount, sendtype, + recvbuf, recvcounts, displs, recvtype, + comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_INEIGHBOR_ALLGATHERV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ineighbor_alltoall +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ineighbor_alltoall(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_INEIGHBOR_ALLTOALL); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_INEIGHBOR_ALLTOALL); + + mpi_errno = MPIR_Ineighbor_alltoall_impl(sendbuf, sendcount, sendtype, + recvbuf, recvcount, recvtype, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_INEIGHBOR_ALLTOALL); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ineighbor_alltoallv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ineighbor_alltoallv(const void *sendbuf, const int sendcounts[], + const int sdispls[], MPI_Datatype sendtype, + void *recvbuf, const int recvcounts[], + const int rdispls[], MPI_Datatype recvtype, + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_INEIGHBOR_ALLTOALLV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_INEIGHBOR_ALLTOALLV); + + mpi_errno = MPIR_Ineighbor_alltoallv_impl(sendbuf, sendcounts, sdispls, sendtype, + recvbuf, recvcounts, rdispls, recvtype, + comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_INEIGHBOR_ALLTOALLV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ineighbor_alltoallw +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ineighbor_alltoallw(const void *sendbuf, const int sendcounts[], + const MPI_Aint sdispls[], + const MPI_Datatype sendtypes[], void *recvbuf, + const int recvcounts[], const MPI_Aint rdispls[], + const MPI_Datatype recvtypes[], MPIR_Comm * comm_ptr, + MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_INEIGHBOR_ALLTOALLW); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_INEIGHBOR_ALLTOALLW); + + mpi_errno = MPIR_Ineighbor_alltoallw_impl(sendbuf, sendcounts, sdispls, sendtypes, + recvbuf, recvcounts, rdispls, recvtypes, + comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_INEIGHBOR_ALLTOALLW); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ibarrier +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ibarrier(MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IBARRIER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IBARRIER); + + mpi_errno = MPIR_Ibarrier_impl(comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IBARRIER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ibcast +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ibcast(void *buffer, int count, MPI_Datatype datatype, + int root, MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IBCAST); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IBCAST); + + mpi_errno = MPIR_Ibcast_impl(buffer, count, datatype, root, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IBCAST); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_iallgather +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_iallgather(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IALLGATHER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IALLGATHER); + + mpi_errno = MPIR_Iallgather_impl(sendbuf, sendcount, sendtype, recvbuf, + recvcount, recvtype, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IALLGATHER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_iallgatherv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_iallgatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, const int *displs, + MPI_Datatype recvtype, MPIR_Comm * comm_ptr, + MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IALLGATHERV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IALLGATHERV); + + mpi_errno = MPIR_Iallgatherv_impl(sendbuf, sendcount, sendtype, + recvbuf, recvcounts, displs, recvtype, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IALLGATHERV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_iallreduce +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_iallreduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm, + MPI_Request * request) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IALLREDUCE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IALLREDUCE); + + mpi_errno = MPIR_Iallreduce_impl(sendbuf, recvbuf, count, datatype, op, comm, request); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IALLREDUCE); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ialltoall +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ialltoall(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IALLTOALL); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IALLTOALL); + + mpi_errno = MPIR_Ialltoall_impl(sendbuf, sendcount, sendtype, recvbuf, + recvcount, recvtype, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IALLTOALL); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ialltoallv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ialltoallv(const void *sendbuf, const int *sendcounts, + const int *sdispls, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, + const int *rdispls, MPI_Datatype recvtype, + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IALLTOALLV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IALLTOALLV); + + mpi_errno = MPIR_Ialltoallv_impl(sendbuf, sendcounts, sdispls, + sendtype, recvbuf, recvcounts, + rdispls, recvtype, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IALLTOALLV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ialltoallw +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ialltoallw(const void *sendbuf, const int *sendcounts, + const int *sdispls, const MPI_Datatype sendtypes[], + void *recvbuf, const int *recvcounts, + const int *rdispls, const MPI_Datatype recvtypes[], + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IALLTOALLW); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IALLTOALLW); + + mpi_errno = MPIR_Ialltoallw_impl(sendbuf, sendcounts, sdispls, + sendtypes, recvbuf, recvcounts, + rdispls, recvtypes, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IALLTOALLW); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_iexscan +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_iexscan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm_ptr, + MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IEXSCAN); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IEXSCAN); + + mpi_errno = MPIR_Iexscan_impl(sendbuf, recvbuf, count, datatype, op, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IEXSCAN); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_igather +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_igather(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + int root, MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IGATHER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IGATHER); + + mpi_errno = MPIR_Igather_impl(sendbuf, sendcount, sendtype, recvbuf, + recvcount, recvtype, root, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IGATHER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_igatherv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_igatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, const int *displs, + MPI_Datatype recvtype, int root, MPIR_Comm * comm_ptr, + MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IGATHERV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IGATHERV); + + mpi_errno = MPIR_Igatherv_impl(sendbuf, sendcount, sendtype, + recvbuf, recvcounts, displs, recvtype, root, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IGATHERV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ireduce_scatter_block +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ireduce_scatter_block(const void *sendbuf, void *recvbuf, + int recvcount, MPI_Datatype datatype, + MPI_Op op, MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IREDUCE_SCATTER_BLOCK); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IREDUCE_SCATTER_BLOCK); + + mpi_errno = MPIR_Ireduce_scatter_block_impl(sendbuf, recvbuf, recvcount, + datatype, op, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IREDUCE_SCATTER_BLOCK); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ireduce_scatter +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ireduce_scatter(const void *sendbuf, void *recvbuf, + const int recvcounts[], MPI_Datatype datatype, + MPI_Op op, MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IREDUCE_SCATTER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IREDUCE_SCATTER); + + mpi_errno = MPIR_Ireduce_scatter_impl(sendbuf, recvbuf, recvcounts, datatype, op, + comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IREDUCE_SCATTER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ireduce +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ireduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, int root, + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IREDUCE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IREDUCE); + + mpi_errno = MPIR_Ireduce_impl(sendbuf, recvbuf, count, datatype, op, root, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IREDUCE); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_iscan +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_iscan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm_ptr, + MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_ISCAN); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_ISCAN); + + mpi_errno = MPIR_Iscan_impl(sendbuf, recvbuf, count, datatype, op, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_ISCAN); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_iscatter +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_iscatter(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + int root, MPIR_Comm * comm, MPI_Request * request) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_ISCATTER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_ISCATTER); + + mpi_errno = MPIR_Iscatter_impl(sendbuf, sendcount, sendtype, recvbuf, + recvcount, recvtype, root, comm, request); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_ISCATTER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_iscatterv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_iscatterv(const void *sendbuf, const int *sendcounts, + const int *displs, MPI_Datatype sendtype, + void *recvbuf, int recvcount, + MPI_Datatype recvtype, int root, + MPIR_Comm * comm, MPI_Request * request) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_ISCATTERV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_ISCATTERV); + + mpi_errno = MPIR_Iscatterv_impl(sendbuf, sendcounts, displs, sendtype, + recvbuf, recvcount, recvtype, root, comm, request); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_ISCATTERV); + return mpi_errno; +} + +#endif diff --git a/src/mpid/ch4/netmod/ofi/ofi_comm.h b/src/mpid/ch4/netmod/ofi/ofi_comm.h new file mode 100644 index 0000000..9f540c9 --- /dev/null +++ b/src/mpid/ch4/netmod/ofi/ofi_comm.h @@ -0,0 +1,65 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_OFI_COMM_H_INCLUDED +#define NETMOD_OFI_COMM_H_INCLUDED + +#include "ofi_impl.h" +#include "mpl_utlist.h" + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_comm_create +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_comm_create(MPIR_Comm * comm) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_COMM_CREATE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_COMM_CREATE); + + MPIDI_OFI_map_create(&MPIDI_OFI_COMM(comm).huge_send_counters); + MPIDI_OFI_map_create(&MPIDI_OFI_COMM(comm).huge_recv_counters); + MPIDI_OFI_index_allocator_create(&MPIDI_OFI_COMM(comm).win_id_allocator, 0); + MPIDI_OFI_index_allocator_create(&MPIDI_OFI_COMM(comm).rma_id_allocator, 1); + + mpi_errno = MPIDI_CH4U_init_comm(comm); + + /* Do not handle intercomms */ + if (comm->comm_kind == MPIR_COMM_KIND__INTERCOMM) + goto fn_exit; + + MPIR_Assert(comm->coll_fns != NULL); + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_COMM_CREATE); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_comm_destroy +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_comm_destroy(MPIR_Comm * comm) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_COMM_DESTROY); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_COMM_DESTROY); + + mpi_errno = MPIDI_CH4U_destroy_comm(comm); + MPIDI_OFI_map_destroy(MPIDI_OFI_COMM(comm).huge_send_counters); + MPIDI_OFI_map_destroy(MPIDI_OFI_COMM(comm).huge_recv_counters); + MPIDI_OFI_index_allocator_destroy(MPIDI_OFI_COMM(comm).win_id_allocator); + MPIDI_OFI_index_allocator_destroy(MPIDI_OFI_COMM(comm).rma_id_allocator); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_COMM_DESTROY); + return mpi_errno; +} + + +#endif /* NETMOD_OFI_COMM_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/ofi/ofi_control.h b/src/mpid/ch4/netmod/ofi/ofi_control.h new file mode 100644 index 0000000..ff9dab7 --- /dev/null +++ b/src/mpid/ch4/netmod/ofi/ofi_control.h @@ -0,0 +1,72 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_AM_OFI_CONTROL_H_INCLUDED +#define NETMOD_AM_OFI_CONTROL_H_INCLUDED + +#include "ofi_am_impl.h" + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_do_control_win +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_OFI_do_control_win(MPIDI_OFI_win_control_t * control, + int rank, MPIR_Win * win, int use_comm, int use_lock) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_OFI_DO_CONTROL_WIN); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_OFI_DO_CONTROL_WIN); + + control->win_id = MPIDI_OFI_WIN(win).win_id; + control->origin_rank = win->comm_ptr->rank; + + mpi_errno = MPIDI_OFI_do_inject(rank, + win->comm_ptr, + MPIDI_OFI_INTERNAL_HANDLER_CONTROL, + (void *) control, + sizeof(*control), NULL, FALSE, use_comm, use_lock); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_OFI_DO_CONTROL_WIN); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_do_control_send +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_OFI_do_control_send(MPIDI_OFI_send_control_t * control, + char *send_buf, + size_t msgsize, + int rank, + MPIR_Comm * comm_ptr, + MPIR_Request * ackreq, int need_lock) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_OFI_DO_CONTROL_SEND); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_OFI_DO_CONTROL_SEND); + + control->origin_rank = comm_ptr->rank; + control->send_buf = (uintptr_t) send_buf; + control->msgsize = msgsize; + control->comm_id = comm_ptr->context_id; + control->endpoint_id = MPIDI_OFI_COMM_TO_EP(comm_ptr, comm_ptr->rank); + control->ackreq = ackreq; + + mpi_errno = MPIDI_OFI_do_inject(rank, comm_ptr, + MPIDI_OFI_INTERNAL_HANDLER_CONTROL, + (void *) control, + sizeof(*control), NULL, FALSE, TRUE, need_lock); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_OFI_DO_CONTROL_SEND); + return mpi_errno; +} + + +#endif /* NETMOD_AM_OFI_CONTROL_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/ofi/ofi_datatype.h b/src/mpid/ch4/netmod/ofi/ofi_datatype.h new file mode 100644 index 0000000..37ddc69 --- /dev/null +++ b/src/mpid/ch4/netmod/ofi/ofi_datatype.h @@ -0,0 +1,31 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2012 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_OFI_DATATYPE_H_INCLUDED +#define NETMOD_OFI_DATATYPE_H_INCLUDED + +#include "ofi_impl.h" + +static inline void MPIDI_NM_datatype_destroy(MPIR_Datatype * datatype_p) +{ + return; +} + +static inline void MPIDI_NM_datatype_commit(MPIR_Datatype * datatype_p) +{ + return; +} + +static inline void MPIDI_NM_datatype_dup(MPIR_Datatype * old_datatype_p, + MPIR_Datatype * new_datatype_p) +{ + return; +} +#endif diff --git a/src/mpid/ch4/netmod/ofi/ofi_events.h b/src/mpid/ch4/netmod/ofi/ofi_events.h new file mode 100644 index 0000000..3ddb664 --- /dev/null +++ b/src/mpid/ch4/netmod/ofi/ofi_events.h @@ -0,0 +1,804 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_OFI_EVENTS_H_INCLUDED +#define NETMOD_OFI_EVENTS_H_INCLUDED + +#include "ofi_impl.h" +#include "ofi_am_impl.h" +#include "ofi_am_events.h" +#include "ofi_control.h" + +__ALWAYS_INLINE__ int MPIDI_OFI_get_huge_event(struct fi_cq_tagged_entry *wc, MPIR_Request * req); + +__ALWAYS_INLINE__ int MPIDI_OFI_cqe_get_source(struct fi_cq_tagged_entry *wc, int do_data) +{ + if (do_data) + return wc->data; + else + return MPIDI_OFI_init_get_source(wc->tag); +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_peek_event +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__ALWAYS_INLINE__ int MPIDI_OFI_peek_event(struct fi_cq_tagged_entry *wc, MPIR_Request * rreq) +{ + size_t count; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_NETMOD_PEEK_EVENT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_NETMOD_PEEK_EVENT); + MPIDI_OFI_REQUEST(rreq, util_id) = MPIDI_OFI_PEEK_FOUND; + rreq->status.MPI_SOURCE = MPIDI_OFI_cqe_get_source(wc, MPIDI_OFI_ENABLE_DATA); + rreq->status.MPI_TAG = MPIDI_OFI_init_get_tag(wc->tag); + count = wc->len; + rreq->status.MPI_ERROR = MPI_SUCCESS; + MPIR_STATUS_SET_COUNT(rreq->status, count); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_NETMOD_PEEK_EVENT); + return MPI_SUCCESS; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_peek_empty_event +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__ALWAYS_INLINE__ int MPIDI_OFI_peek_empty_event(struct fi_cq_tagged_entry *wc, MPIR_Request * rreq) +{ + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_NETMOD_PEEK_EMPTY_EVENT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_NETMOD_PEEK_EMPTY_EVENT); + MPIDI_OFI_dynamic_process_request_t *ctrl; + + switch (MPIDI_OFI_REQUEST(rreq, event_id)) { + case MPIDI_OFI_EVENT_PEEK: + MPIDI_OFI_REQUEST(rreq, util_id) = MPIDI_OFI_PEEK_NOT_FOUND; + rreq->status.MPI_ERROR = MPI_SUCCESS; + break; + + case MPIDI_OFI_EVENT_ACCEPT_PROBE: + ctrl = (MPIDI_OFI_dynamic_process_request_t *) rreq; + ctrl->done = MPIDI_OFI_PEEK_NOT_FOUND; + break; + + default: + MPIR_Assert(0); + break; + } + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_NETMOD_PEEK_EMPTY_EVENT); + return MPI_SUCCESS; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_recv_event +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__ALWAYS_INLINE__ int MPIDI_OFI_recv_event(struct fi_cq_tagged_entry *wc, MPIR_Request * rreq) +{ + int mpi_errno = MPI_SUCCESS; + MPI_Aint last; + size_t count; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_RECV_EVENT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_RECV_EVENT); + + rreq->status.MPI_ERROR = MPI_SUCCESS; + rreq->status.MPI_SOURCE = MPIDI_OFI_cqe_get_source(wc, MPIDI_OFI_ENABLE_DATA); + rreq->status.MPI_TAG = MPIDI_OFI_init_get_tag(wc->tag); + count = wc->len; + MPIR_STATUS_SET_COUNT(rreq->status, count); + +#ifdef MPIDI_BUILD_CH4_SHM + + if (MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(rreq)) { + int continue_matching = 1; + + MPIDI_CH4R_anysource_matched(MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(rreq), MPIDI_CH4R_NETMOD, + &continue_matching); + + /* It is always possible to cancel a request on shm side w/o an aux thread */ + + /* Decouple requests */ + if (unlikely(MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(rreq))) { + MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(rreq)) = NULL; + MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(rreq) = NULL; + } + + if (!continue_matching) + goto fn_exit; + } + +#endif + + if (MPIDI_OFI_REQUEST(rreq, noncontig)) { + last = count; + MPID_Segment_unpack(&MPIDI_OFI_REQUEST(rreq, noncontig->segment), 0, &last, + MPIDI_OFI_REQUEST(rreq, noncontig->pack_buffer)); + MPL_free(MPIDI_OFI_REQUEST(rreq, noncontig)); + if (last != (MPI_Aint) count) { + rreq->status.MPI_ERROR = + MPIR_Err_create_code(MPI_SUCCESS, + MPIR_ERR_RECOVERABLE, + __FUNCTION__, __LINE__, MPI_ERR_TYPE, "**dtypemismatch", 0); + } + } + + dtype_release_if_not_builtin(MPIDI_OFI_REQUEST(rreq, datatype)); + + /* If syncronous, ack and complete when the ack is done */ + if (unlikely(MPIDI_OFI_is_tag_sync(wc->tag))) { + uint64_t ss_bits = MPIDI_OFI_init_sendtag(MPIDI_OFI_REQUEST(rreq, util_id), + MPIDI_OFI_REQUEST(rreq, util_comm->rank), + rreq->status.MPI_TAG, + MPIDI_OFI_SYNC_SEND_ACK, MPIDI_OFI_ENABLE_DATA); + MPIR_Comm *c = MPIDI_OFI_REQUEST(rreq, util_comm); + int r = rreq->status.MPI_SOURCE; + mpi_errno = MPIDI_OFI_send_handler(MPIDI_OFI_EP_TX_TAG(0), NULL, 0, NULL, + MPIDI_OFI_REQUEST(rreq, util_comm->rank), + MPIDI_OFI_comm_to_phys(c, r, MPIDI_OFI_API_TAG), + ss_bits, NULL, MPIDI_OFI_DO_INJECT, + MPIDI_OFI_ENABLE_DATA, MPIDI_OFI_CALL_NO_LOCK); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + } + + MPIDI_CH4U_request_complete(rreq); + + /* Polling loop will check for truncation */ + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_RECV_EVENT); + return mpi_errno; + fn_fail: + rreq->status.MPI_ERROR = mpi_errno; + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_recv_huge_event +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__ALWAYS_INLINE__ int MPIDI_OFI_recv_huge_event(struct fi_cq_tagged_entry *wc, MPIR_Request * rreq) +{ + MPIDI_OFI_huge_recv_t *recv; + MPIR_Comm *comm_ptr; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_RECV_HUGE_EVENT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_RECV_HUGE_EVENT); + + /* Look up the receive sequence number and chunk queue */ + comm_ptr = MPIDI_OFI_REQUEST(rreq, util_comm); + recv = + (MPIDI_OFI_huge_recv_t *) MPIDI_OFI_map_lookup(MPIDI_OFI_COMM(comm_ptr).huge_recv_counters, + MPIDI_OFI_cqe_get_source(wc, + MPIDI_OFI_ENABLE_DATA)); + if (recv == MPIDI_OFI_MAP_NOT_FOUND) { + recv = (MPIDI_OFI_huge_recv_t *) MPL_calloc(sizeof(*recv), 1); + MPIDI_OFI_map_set(MPIDI_OFI_COMM(comm_ptr).huge_recv_counters, + MPIDI_OFI_cqe_get_source(wc, MPIDI_OFI_ENABLE_DATA), recv); + } + + recv->event_id = MPIDI_OFI_EVENT_GET_HUGE; + recv->localreq = rreq; + recv->done_fn = MPIDI_OFI_recv_event; + recv->wc = *wc; + MPIDI_OFI_get_huge_event(NULL, (MPIR_Request *) recv); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_RECV_HUGE_EVENT); + return MPI_SUCCESS; +} + + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_send_event +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__ALWAYS_INLINE__ int MPIDI_OFI_send_event(struct fi_cq_tagged_entry *wc, MPIR_Request * sreq) +{ + int c; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_SEND_EVENT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_SEND_EVENT); + + MPIR_cc_decr(sreq->cc_ptr, &c); + + if (c == 0) { + if (MPIDI_OFI_REQUEST(sreq, noncontig)) + MPL_free(MPIDI_OFI_REQUEST(sreq, noncontig)); + + dtype_release_if_not_builtin(MPIDI_OFI_REQUEST(sreq, datatype)); + MPIDI_CH4U_request_release(sreq); + } /* c != 0, ssend */ + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_SEND_EVENT); + return MPI_SUCCESS; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_send_huge_event +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__ALWAYS_INLINE__ int MPIDI_OFI_send_huge_event(struct fi_cq_tagged_entry *wc, MPIR_Request * sreq) +{ + int mpi_errno = MPI_SUCCESS; + int c; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_SEND_EVENT_HUGE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_SEND_EVENT_HUGE); + + MPIR_cc_decr(sreq->cc_ptr, &c); + + if (c == 0) { + MPIR_Comm *comm; + void *ptr; + MPIDI_OFI_huge_counter_t *cntr; + comm = MPIDI_OFI_REQUEST(sreq, util_comm); + ptr = + MPIDI_OFI_map_lookup(MPIDI_OFI_COMM(comm).huge_send_counters, + MPIDI_OFI_REQUEST(sreq, util_id)); + MPIR_Assert(ptr != MPIDI_OFI_MAP_NOT_FOUND); + cntr = (MPIDI_OFI_huge_counter_t *) ptr; + cntr->outstanding--; + if (cntr->outstanding == 0) { + MPIDI_OFI_send_control_t ctrl; + uint64_t key; + int key_back; + MPIDI_OFI_map_erase(MPIDI_OFI_COMM(comm).huge_send_counters, + MPIDI_OFI_REQUEST(sreq, util_id)); + key = fi_mr_key(cntr->mr); + key_back = (key >> MPIDI_Global.huge_rma_shift); + MPIDI_OFI_index_allocator_free(MPIDI_OFI_COMM(comm).rma_id_allocator, key_back); + MPIDI_OFI_CALL_NOLOCK(fi_close(&cntr->mr->fid), mr_unreg); + MPL_free(ptr); + ctrl.type = MPIDI_OFI_CTRL_HUGE_CLEANUP; + MPIDI_OFI_MPI_CALL_POP(MPIDI_OFI_do_control_send + (&ctrl, NULL, 0, MPIDI_OFI_REQUEST(sreq, util_id), comm, NULL, + FALSE /* no lock */)); + } + + if (MPIDI_OFI_REQUEST(sreq, noncontig)) + MPL_free(MPIDI_OFI_REQUEST(sreq, noncontig)); + + dtype_release_if_not_builtin(MPIDI_OFI_REQUEST(sreq, datatype)); + MPIDI_CH4U_request_release(sreq); + } /* c != 0, ssend */ + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_SEND_EVENT_HUGE); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_ssend_ack_event +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__ALWAYS_INLINE__ int MPIDI_OFI_ssend_ack_event(struct fi_cq_tagged_entry *wc, MPIR_Request * sreq) +{ + int mpi_errno; + MPIDI_OFI_ssendack_request_t *req = (MPIDI_OFI_ssendack_request_t *) sreq; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_SSEND_ACK_EVENT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_SSEND_ACK_EVENT); + mpi_errno = MPIDI_OFI_send_event(NULL, req->signal_req); + MPIDI_OFI_ssendack_request_t_tls_free(req); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_SSEND_ACK_EVENT); + return mpi_errno; +} + +__ALWAYS_INLINE__ uintptr_t MPIDI_OFI_recv_rbase(MPIDI_OFI_huge_recv_t * recv) +{ +#ifdef USE_OFI_MR_SCALABLE + return 0; +#else + return recv->remote_info.send_buf; +#endif +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_get_huge_event +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__ALWAYS_INLINE__ int MPIDI_OFI_get_huge_event(struct fi_cq_tagged_entry *wc, MPIR_Request * req) +{ + int mpi_errno = MPI_SUCCESS; + MPIDI_OFI_huge_recv_t *recv = (MPIDI_OFI_huge_recv_t *) req; + uint64_t remote_key; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_GETHUGE_EVENT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_GETHUGE_EVENT); + + if (recv->localreq && recv->cur_offset != 0) { + size_t bytesSent = recv->cur_offset - MPIDI_Global.max_send; + size_t bytesLeft = recv->remote_info.msgsize - bytesSent - MPIDI_Global.max_send; + size_t bytesToGet = + (bytesLeft <= MPIDI_Global.max_send) ? bytesLeft : MPIDI_Global.max_send; + + if (bytesToGet == 0ULL) { + MPIDI_OFI_send_control_t ctrl; + recv->wc.len = recv->cur_offset; + recv->done_fn(&recv->wc, recv->localreq); + ctrl.type = MPIDI_OFI_CTRL_HUGEACK; + MPIDI_OFI_MPI_CALL_POP(MPIDI_OFI_do_control_send + (&ctrl, NULL, 0, recv->remote_info.origin_rank, recv->comm_ptr, + recv->remote_info.ackreq, FALSE)); + /* "recv" and maps will be freed in MPIDI_OFI_get_huge_cleanup */ + goto fn_exit; + } + + if (MPIDI_OFI_ENABLE_MR_SCALABLE) + remote_key = recv->remote_info.rma_key << MPIDI_Global.huge_rma_shift; + else + remote_key = recv->remote_info.rma_key; + + MPIDI_OFI_conditional_cntr_incr(); + MPIDI_OFI_CALL_RETRY(fi_read(MPIDI_OFI_EP_TX_RMA(0), /* endpoint */ + (void *) ((uintptr_t) recv->wc.buf + recv->cur_offset), /* local buffer */ + bytesToGet, /* bytes */ + NULL, /* descriptor */ + MPIDI_OFI_comm_to_phys(recv->comm_ptr, recv->remote_info.origin_rank, MPIDI_OFI_API_MSG), /* Destination */ + MPIDI_OFI_recv_rbase(recv) + recv->cur_offset, /* remote maddr */ + remote_key, /* Key */ + (void *) &recv->context), rdma_readfrom, /* Context */ + MPIDI_OFI_CALL_NO_LOCK); + recv->cur_offset += bytesToGet; + } + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_GETHUGE_EVENT); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_chunk_done_event +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__ALWAYS_INLINE__ int MPIDI_OFI_chunk_done_event(struct fi_cq_tagged_entry *wc, MPIR_Request * req) +{ + int c; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_CHUNK_DONE_EVENT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_CHUNK_DONE_EVENT); + + MPIDI_OFI_chunk_request *creq = (MPIDI_OFI_chunk_request *) req; + MPIR_cc_decr(creq->parent->cc_ptr, &c); + + if (c == 0) + MPIDI_CH4U_request_release(creq->parent); + + MPL_free(creq); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_CHUNK_DONE_EVENT); + return MPI_SUCCESS; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_inject_emu_event +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__ALWAYS_INLINE__ int MPIDI_OFI_inject_emu_event(struct fi_cq_tagged_entry *wc, MPIR_Request * req) +{ + int incomplete; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_INJECT_EMU_EVENT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_INJECT_EMU_EVENT); + + MPIR_cc_decr(req->cc_ptr, &incomplete); + + if (!incomplete) { + MPL_free(MPIDI_OFI_REQUEST(req, util.inject_buf)); + MPIDI_CH4U_request_release(req); + OPA_decr_int(&MPIDI_Global.am_inflight_inject_emus); + } + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_INJECT_EMU_EVENT); + return MPI_SUCCESS; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_rma_done_event +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__ALWAYS_INLINE__ int MPIDI_OFI_rma_done_event(struct fi_cq_tagged_entry *wc, MPIR_Request * in_req) +{ + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_OFI_RMA_DONE_EVENT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_OFI_RMA_DONE_EVENT); + + MPIDI_OFI_win_request_t *req = (MPIDI_OFI_win_request_t *) in_req; + MPIDI_OFI_win_request_complete(req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_OFI_RMA_DONE_EVENT); + return MPI_SUCCESS; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_accept_probe_event +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__ALWAYS_INLINE__ int MPIDI_OFI_accept_probe_event(struct fi_cq_tagged_entry *wc, + MPIR_Request * rreq) +{ + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_OFI_ACCEPT_PROBE_EVENT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_OFI_ACCEPT_PROBE_EVENT); + MPIDI_OFI_dynamic_process_request_t *ctrl = (MPIDI_OFI_dynamic_process_request_t *) rreq; + ctrl->source = MPIDI_OFI_cqe_get_source(wc, MPIDI_OFI_ENABLE_DATA); + ctrl->tag = MPIDI_OFI_init_get_tag(wc->tag); + ctrl->msglen = wc->len; + ctrl->done = MPIDI_OFI_PEEK_FOUND; + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_OFI_ACCEPT_PROBE_EVENT); + return MPI_SUCCESS; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_dynproc_done_event +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__ALWAYS_INLINE__ int MPIDI_OFI_dynproc_done_event(struct fi_cq_tagged_entry *wc, + MPIR_Request * rreq) +{ + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_OFI_DYNPROC_DONE_EVENT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_OFI_DYNPROC_DONE_EVENT); + MPIDI_OFI_dynamic_process_request_t *ctrl = (MPIDI_OFI_dynamic_process_request_t *) rreq; + ctrl->done++; + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_OFI_DYNPROC_DONE_EVENT); + return MPI_SUCCESS; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_am_send_event +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__ALWAYS_INLINE__ int MPIDI_OFI_am_send_event(struct fi_cq_tagged_entry *wc, MPIR_Request * sreq) +{ + int mpi_errno = MPI_SUCCESS; + MPIDI_OFI_am_header_t *msg_hdr; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_HANDLE_SEND_COMPLETION); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_HANDLE_SEND_COMPLETION); + + msg_hdr = &MPIDI_OFI_AMREQUEST_HDR(sreq, msg_hdr); + MPIDI_OFI_am_request_complete(sreq); + + switch (msg_hdr->am_type) { + case MPIDI_AMTYPE_LMT_ACK: + case MPIDI_AMTYPE_LMT_REQ: + goto fn_exit; + + default: + break; + } + + if (MPIDI_OFI_AMREQUEST_HDR(sreq, pack_buffer)) { + MPL_free(MPIDI_OFI_AMREQUEST_HDR(sreq, pack_buffer)); + MPIDI_OFI_AMREQUEST_HDR(sreq, pack_buffer) = NULL; + } + + mpi_errno = MPIDI_Global.am_send_cmpl_handlers[msg_hdr->handler_id] (sreq); + + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_HANDLE_SEND_COMPLETION); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_am_recv_event +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__ALWAYS_INLINE__ int MPIDI_OFI_am_recv_event(struct fi_cq_tagged_entry *wc, MPIR_Request * rreq) +{ + int mpi_errno = MPI_SUCCESS; + MPIDI_OFI_am_header_t *am_hdr; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_HANDLE_RECV_COMPLETION); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_HANDLE_RECV_COMPLETION); + + am_hdr = (MPIDI_OFI_am_header_t *) wc->buf; + + switch (am_hdr->am_type) { + case MPIDI_AMTYPE_SHORT_HDR: + mpi_errno = MPIDI_OFI_handle_short_am_hdr(am_hdr, am_hdr->payload); + + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + break; + + case MPIDI_AMTYPE_SHORT: + mpi_errno = MPIDI_OFI_handle_short_am(am_hdr); + + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + break; + + case MPIDI_AMTYPE_LMT_REQ: + mpi_errno = MPIDI_OFI_handle_long_am(am_hdr); + + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + break; + + case MPIDI_AMTYPE_LMT_ACK: + mpi_errno = MPIDI_OFI_handle_lmt_ack(am_hdr); + + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + break; + + default: + MPIR_Assert(0); + } + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_HANDLE_RECV_COMPLETION); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_am_read_event +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__ALWAYS_INLINE__ int MPIDI_OFI_am_read_event(struct fi_cq_tagged_entry *wc, + MPIR_Request * dont_use_me) +{ + int mpi_errno = MPI_SUCCESS; + void *netmod_context = NULL; + MPIR_Request *rreq; + MPIDI_OFI_am_request_t *ofi_req; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_HANDLE_READ_COMPLETION); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_HANDLE_READ_COMPLETION); + + ofi_req = container_of(wc->op_context, MPIDI_OFI_am_request_t, context); + ofi_req->req_hdr->lmt_cntr--; + + if (ofi_req->req_hdr->lmt_cntr) + goto fn_exit; + + rreq = (MPIR_Request *) ofi_req->req_hdr->rreq_ptr; + mpi_errno = MPIDI_OFI_dispatch_ack(MPIDI_OFI_AMREQUEST_HDR(rreq, lmt_info).src_rank, + MPIDI_OFI_AMREQUEST_HDR(rreq, lmt_info).context_id, + MPIDI_OFI_AMREQUEST_HDR(rreq, lmt_info).sreq_ptr, + MPIDI_AMTYPE_LMT_ACK, netmod_context); + + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + MPIDI_OFI_am_request_complete(rreq); + ofi_req->req_hdr->cmpl_handler_fn(rreq); + fn_exit: + MPIDI_CH4R_release_buf((void *) ofi_req); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_HANDLE_READ_COMPLETION); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_am_repost_event +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__ALWAYS_INLINE__ int MPIDI_OFI_am_repost_event(struct fi_cq_tagged_entry *wc, MPIR_Request * rreq) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_REPOST_BUFFER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_REPOST_BUFFER); + + mpi_errno = MPIDI_OFI_repost_buffer(wc->op_context, rreq); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_REPOST_BUFFER); + return mpi_errno; +} + +__ALWAYS_INLINE__ int MPIDI_OFI_dispatch_function(struct fi_cq_tagged_entry *wc, + MPIR_Request * req, int buffered) +{ + int mpi_errno; + + if (likely(MPIDI_OFI_REQUEST(req, event_id) == MPIDI_OFI_EVENT_SEND)) { + mpi_errno = MPIDI_OFI_send_event(wc, req); + goto fn_exit; + } + else if (likely(MPIDI_OFI_REQUEST(req, event_id) == MPIDI_OFI_EVENT_RECV)) { + mpi_errno = MPIDI_OFI_recv_event(wc, req); + goto fn_exit; + } + else if (likely(MPIDI_OFI_REQUEST(req, event_id) == MPIDI_OFI_EVENT_RMA_DONE)) { + mpi_errno = MPIDI_OFI_rma_done_event(wc, req); + goto fn_exit; + } + else if (likely(MPIDI_OFI_REQUEST(req, event_id) == MPIDI_OFI_EVENT_AM_SEND)) { + mpi_errno = MPIDI_OFI_am_send_event(wc, req); + goto fn_exit; + } + else if (likely(MPIDI_OFI_REQUEST(req, event_id) == MPIDI_OFI_EVENT_AM_RECV)) { + mpi_errno = MPIDI_OFI_am_recv_event(wc, req); + + if (unlikely((wc->flags & FI_MULTI_RECV) && !buffered)) + MPIDI_OFI_am_repost_event(wc, req); + + goto fn_exit; + } + else if (likely(MPIDI_OFI_REQUEST(req, event_id) == MPIDI_OFI_EVENT_AM_READ)) { + mpi_errno = MPIDI_OFI_am_read_event(wc, req); + goto fn_exit; + } + else if (unlikely(1)) { + switch (MPIDI_OFI_REQUEST(req, event_id)) { + case MPIDI_OFI_EVENT_AM_MULTI: + mpi_errno = MPIDI_OFI_am_repost_event(wc, req); + break; + + case MPIDI_OFI_EVENT_PEEK: + mpi_errno = MPIDI_OFI_peek_event(wc, req); + break; + + case MPIDI_OFI_EVENT_RECV_HUGE: + mpi_errno = MPIDI_OFI_recv_huge_event(wc, req); + break; + + case MPIDI_OFI_EVENT_SEND_HUGE: + mpi_errno = MPIDI_OFI_send_huge_event(wc, req); + break; + + case MPIDI_OFI_EVENT_SSEND_ACK: + mpi_errno = MPIDI_OFI_ssend_ack_event(wc, req); + break; + + case MPIDI_OFI_EVENT_GET_HUGE: + mpi_errno = MPIDI_OFI_get_huge_event(wc, req); + break; + + case MPIDI_OFI_EVENT_CHUNK_DONE: + mpi_errno = MPIDI_OFI_chunk_done_event(wc, req); + break; + + case MPIDI_OFI_EVENT_INJECT_EMU: + mpi_errno = MPIDI_OFI_inject_emu_event(wc, req); + break; + + case MPIDI_OFI_EVENT_DYNPROC_DONE: + mpi_errno = MPIDI_OFI_dynproc_done_event(wc, req); + break; + + case MPIDI_OFI_EVENT_ACCEPT_PROBE: + mpi_errno = MPIDI_OFI_accept_probe_event(wc, req); + break; + + case MPIDI_OFI_EVENT_ABORT: + default: + mpi_errno = MPI_SUCCESS; + MPIR_Assert(0); + break; + } + } + + fn_exit: + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_get_buffered +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__ALWAYS_INLINE__ int MPIDI_OFI_get_buffered(struct fi_cq_tagged_entry *wc, ssize_t num) +{ + int rc = 0; + + if ((MPIDI_Global.cq_buff_head != MPIDI_Global.cq_buff_tail) || + !slist_empty(&MPIDI_Global.cq_buff_list)) { + if (MPIDI_Global.cq_buff_head != MPIDI_Global.cq_buff_tail) { + wc[0] = MPIDI_Global.cq_buffered[MPIDI_Global.cq_buff_tail].cq_entry; + MPIDI_Global.cq_buff_tail = (MPIDI_Global.cq_buff_tail + 1) % MPIDI_OFI_NUM_CQ_BUFFERED; + } + else { + MPIDI_OFI_cq_list_t *MPIDI_OFI_cq_list_entry; + struct slist_entry *entry = slist_remove_head(&MPIDI_Global.cq_buff_list); + MPIDI_OFI_cq_list_entry = container_of(entry, MPIDI_OFI_cq_list_t, entry); + wc[0] = MPIDI_OFI_cq_list_entry->cq_entry; + MPL_free((void *) MPIDI_OFI_cq_list_entry); + } + + rc = 1; + } + + return rc; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_handle_cq_entries +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__ALWAYS_INLINE__ int MPIDI_OFI_handle_cq_entries(struct fi_cq_tagged_entry *wc, + ssize_t num, int buffered) +{ + int i, mpi_errno = MPI_SUCCESS; + MPIR_Request *req; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_HANDLE_CQ_ENTRIES); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_HANDLE_CQ_ENTRIES); + + for (i = 0; i < num; i++) { + req = MPIDI_OFI_context_to_request(wc[i].op_context); + MPIDI_OFI_MPI_CALL_POP(MPIDI_OFI_dispatch_function(&wc[i], req, buffered)); + } + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_HANDLE_CQ_ENTRIES); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_handle_cq_error +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__ALWAYS_INLINE__ int MPIDI_OFI_handle_cq_error(ssize_t ret) +{ + int mpi_errno = MPI_SUCCESS; + struct fi_cq_err_entry e; + MPIR_Request *req; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_HANDLE_CQ_ERROR); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_HANDLE_CQ_ERROR); + + switch (ret) { + case -FI_EAVAIL: + fi_cq_readerr(MPIDI_Global.p2p_cq, &e, 0); + + switch (e.err) { + case FI_ETRUNC: + req = MPIDI_OFI_context_to_request(e.op_context); + + switch (req->kind) { + case MPIR_REQUEST_KIND__SEND: + mpi_errno = MPIDI_OFI_dispatch_function(NULL, req, 0); + break; + + case MPIR_REQUEST_KIND__RECV: + mpi_errno = MPIDI_OFI_dispatch_function((struct fi_cq_tagged_entry *) &e, req, 0); + req->status.MPI_ERROR = MPI_ERR_TRUNCATE; + break; + + default: + MPIR_ERR_SETFATALANDJUMP4(mpi_errno, MPI_ERR_OTHER, "**ofid_poll", + "**ofid_poll %s %d %s %s", __SHORT_FILE__, + __LINE__, FCNAME, fi_strerror(e.err)); + } + + break; + + case FI_ECANCELED: + req = MPIDI_OFI_context_to_request(e.op_context); + MPIR_STATUS_SET_CANCEL_BIT(req->status, TRUE); + break; + + case FI_ENOMSG: + req = MPIDI_OFI_context_to_request(e.op_context); + MPIDI_OFI_peek_empty_event(NULL, req); + break; + } + + break; + + default: + MPIR_ERR_SETFATALANDJUMP4(mpi_errno, MPI_ERR_OTHER, "**ofid_poll", + "**ofid_poll %s %d %s %s", __SHORT_FILE__, __LINE__, + FCNAME, fi_strerror(errno)); + break; + } + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_HANDLE_CQ_ERROR); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#endif /* NETMOD_OFI_EVENTS_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/ofi/ofi_impl.h b/src/mpid/ch4/netmod/ofi/ofi_impl.h new file mode 100644 index 0000000..9e6ff5c --- /dev/null +++ b/src/mpid/ch4/netmod/ofi/ofi_impl.h @@ -0,0 +1,484 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_OFI_IMPL_H_INCLUDED +#define NETMOD_OFI_IMPL_H_INCLUDED + +#include <mpidimpl.h> +#include "ofi_types.h" +#include "mpidch4r.h" +#include "ch4_impl.h" +#include "ofi_iovec_util.h" + +/* Tag the prototypes with always_inline to force object allocation */ +/* routines to inline This allows the library, compiled without */ +/* ipo/pgo enabled to inline MPI layer functions */ +__ALWAYS_INLINE__ MPIR_Request *MPIR_Request_create(MPIR_Request_kind_t kind); +__ALWAYS_INLINE__ void *MPIR_Handle_obj_alloc(MPIR_Object_alloc_t *); +__ALWAYS_INLINE__ void *MPIR_Handle_obj_alloc_unsafe(MPIR_Object_alloc_t *); +__ALWAYS_INLINE__ void MPIR_Handle_obj_free(MPIR_Object_alloc_t *, void *); +__ALWAYS_INLINE__ void *MPIR_Handle_get_ptr_indirect(int, MPIR_Object_alloc_t *); +__ALWAYS_INLINE__ MPIDII_av_entry_t *MPIDIU_comm_rank_to_av(MPIR_Comm * comm, int rank); + +#define MPIDI_OFI_DT(dt) ((dt)->dev.netmod.ofi) +#define MPIDI_OFI_OP(op) ((op)->dev.netmod.ofi) +#define MPIDI_OFI_COMM(comm) ((comm)->dev.ch4.netmod.ofi) +#define MPIDI_OFI_COMM_TO_INDEX(comm,rank) \ + MPIDIU_comm_rank_to_pid(comm, rank, NULL, NULL) +#ifdef MPIDI_OFI_CONFIG_USE_AV_TABLE +#define MPIDI_OFI_COMM_TO_PHYS(comm,rank) \ + ((fi_addr_t)MPIDI_OFI_COMM_TO_INDEX(comm,rank)) +#define MPIDI_OFI_TO_PHYS(avtid, rank) ((fi_addr_t)rank) +#else +#define MPIDI_OFI_COMM_TO_PHYS(comm,rank) \ + MPIDI_OFI_AV(MPIDIU_comm_rank_to_av((comm), (rank))).dest +#define MPIDI_OFI_TO_PHYS(avtid, lpid) \ + MPIDI_OFI_AV(&MPIDIU_get_av((avtid), (lpid))).dest +#endif + +#define MPIDI_OFI_WIN(win) ((win)->dev.netmod.ofi) +/* + * Helper routines and macros for request completion + */ +#define MPIDI_OFI_ssendack_request_t_tls_alloc(req) \ + do { \ + (req) = (MPIDI_OFI_ssendack_request_t*) \ + MPIR_Request_create(MPIR_REQUEST_KIND__SEND); \ + if (req == NULL) \ + MPID_Abort(NULL, MPI_ERR_NO_SPACE, -1, \ + "Cannot allocate Ssendack Request"); \ + } while (0) + +#define MPIDI_OFI_ssendack_request_t_tls_free(req) \ + MPIR_Handle_obj_free(&MPIR_Request_mem, (req)) + +#define MPIDI_OFI_ssendack_request_t_alloc_and_init(req) \ + do { \ + MPIDI_OFI_ssendack_request_t_tls_alloc(req); \ + MPIR_Assert(req != NULL); \ + MPIR_Assert(HANDLE_GET_MPI_KIND(req->handle) \ + == MPID_SSENDACK_REQUEST); \ + } while (0) + +#define MPIDI_OFI_request_create_null_rreq(rreq_, mpi_errno_, FAIL_) \ + do { \ + (rreq_) = MPIR_Request_create(MPIR_REQUEST_KIND__RECV); \ + if ((rreq_) != NULL) { \ + MPIR_cc_set(&(rreq_)->cc, 0); \ + (rreq_)->kind = MPIR_REQUEST_KIND__RECV; \ + MPIR_Status_set_procnull(&(rreq_)->status); \ + } \ + else { \ + MPIR_ERR_SETANDJUMP(mpi_errno_,MPI_ERR_OTHER,"**nomemreq"); \ + } \ + } while (0) + + +#define MPIDI_OFI_PROGRESS() \ + do { \ + mpi_errno = MPIDI_Progress_test(); \ + if (mpi_errno!=MPI_SUCCESS) MPIR_ERR_POP(mpi_errno); \ + } while (0) + +#define MPIDI_OFI_PROGRESS_NONINLINE() \ + do { \ + mpi_errno = MPIDI_OFI_progress_test_no_inline(); \ + if (mpi_errno!=MPI_SUCCESS) MPIR_ERR_POP(mpi_errno); \ + } while (0) + +#define MPIDI_OFI_PROGRESS_WHILE(cond) \ + while (cond) MPIDI_OFI_PROGRESS() + +#define MPIDI_OFI_ERR MPIR_ERR_CHKANDJUMP4 +#define MPIDI_OFI_CALL(FUNC,STR) \ + do { \ + MPID_THREAD_CS_ENTER(POBJ,MPIDI_OFI_THREAD_FI_MUTEX); \ + ssize_t _ret = FUNC; \ + MPID_THREAD_CS_EXIT(POBJ,MPIDI_OFI_THREAD_FI_MUTEX); \ + MPIDI_OFI_ERR(_ret<0, \ + mpi_errno, \ + MPI_ERR_OTHER, \ + "**ofid_"#STR, \ + "**ofid_"#STR" %s %d %s %s", \ + __SHORT_FILE__, \ + __LINE__, \ + FCNAME, \ + fi_strerror(-_ret)); \ + } while (0) + +#define MPIDI_OFI_CALL_NOLOCK(FUNC,STR) \ + do { \ + ssize_t _ret = FUNC; \ + MPIDI_OFI_ERR(_ret<0, \ + mpi_errno, \ + MPI_ERR_OTHER, \ + "**ofid_"#STR, \ + "**ofid_"#STR" %s %d %s %s", \ + __SHORT_FILE__, \ + __LINE__, \ + FCNAME, \ + fi_strerror(-_ret)); \ + } while (0) + +#define MPIDI_OFI_CALL_LOCK 1 +#define MPIDI_OFI_CALL_NO_LOCK 0 +#define MPIDI_OFI_CALL_RETRY(FUNC,STR,LOCK) \ + do { \ + ssize_t _ret; \ + do { \ + if (LOCK == MPIDI_OFI_CALL_LOCK) \ + MPID_THREAD_CS_ENTER(POBJ,MPIDI_OFI_THREAD_FI_MUTEX); \ + _ret = FUNC; \ + if (LOCK == MPIDI_OFI_CALL_LOCK) \ + MPID_THREAD_CS_EXIT(POBJ,MPIDI_OFI_THREAD_FI_MUTEX); \ + if (likely(_ret==0)) break; \ + MPIDI_OFI_ERR(_ret!=-FI_EAGAIN, \ + mpi_errno, \ + MPI_ERR_OTHER, \ + "**ofid_"#STR, \ + "**ofid_"#STR" %s %d %s %s", \ + __SHORT_FILE__, \ + __LINE__, \ + FCNAME, \ + fi_strerror(-_ret)); \ + if (LOCK == MPIDI_OFI_CALL_NO_LOCK) \ + MPID_THREAD_CS_EXIT(POBJ,MPIDI_OFI_THREAD_FI_MUTEX); \ + MPIDI_OFI_PROGRESS_NONINLINE(); \ + if (LOCK == MPIDI_OFI_CALL_NO_LOCK) \ + MPID_THREAD_CS_ENTER(POBJ,MPIDI_OFI_THREAD_FI_MUTEX); \ + } while (_ret == -FI_EAGAIN); \ + } while (0) + +#define MPIDI_OFI_CALL_RETRY2(FUNC1,FUNC2,STR) \ + do { \ + ssize_t _ret; \ + MPID_THREAD_CS_ENTER(POBJ,MPIDI_OFI_THREAD_FI_MUTEX); \ + FUNC1; \ + do { \ + _ret = FUNC2; \ + MPID_THREAD_CS_EXIT(POBJ,MPIDI_OFI_THREAD_FI_MUTEX); \ + if (likely(_ret==0)) break; \ + MPIDI_OFI_ERR(_ret!=-FI_EAGAIN, \ + mpi_errno, \ + MPI_ERR_OTHER, \ + "**ofid_"#STR, \ + "**ofid_"#STR" %s %d %s %s", \ + __SHORT_FILE__, \ + __LINE__, \ + FCNAME, \ + fi_strerror(-_ret)); \ + MPIDI_OFI_PROGRESS_NONINLINE(); \ + MPID_THREAD_CS_ENTER(POBJ,MPIDI_OFI_THREAD_FI_MUTEX); \ + } while (_ret == -FI_EAGAIN); \ + } while (0) + +#define MPIDI_OFI_CALL_RETURN(FUNC, _ret) \ + do { \ + MPID_THREAD_CS_ENTER(POBJ,MPIDI_OFI_THREAD_FI_MUTEX); \ + (_ret) = FUNC; \ + MPID_THREAD_CS_EXIT(POBJ,MPIDI_OFI_THREAD_FI_MUTEX); \ + } while (0) + +#define MPIDI_OFI_PMI_CALL_POP(FUNC,STR) \ + do \ + { \ + pmi_errno = FUNC; \ + MPIDI_OFI_ERR(pmi_errno!=PMI_SUCCESS, \ + mpi_errno, \ + MPI_ERR_OTHER, \ + "**ofid_"#STR, \ + "**ofid_"#STR" %s %d %s %s", \ + __SHORT_FILE__, \ + __LINE__, \ + FCNAME, \ + #STR); \ + } while (0) + +#define MPIDI_OFI_MPI_CALL_POP(FUNC) \ + do \ + { \ + mpi_errno = FUNC; \ + if (unlikely(mpi_errno!=MPI_SUCCESS)) MPIR_ERR_POP(mpi_errno); \ + } while (0) + +#define MPIDI_OFI_STR_CALL(FUNC,STR) \ + do \ + { \ + str_errno = FUNC; \ + MPIDI_OFI_ERR(str_errno!=MPL_STR_SUCCESS, \ + mpi_errno, \ + MPI_ERR_OTHER, \ + "**"#STR, \ + "**"#STR" %s %d %s %s", \ + __SHORT_FILE__, \ + __LINE__, \ + FCNAME, \ + #STR); \ + } while (0) + +#define MPIDI_OFI_REQUEST_CREATE(req, kind) \ + do { \ + (req) = MPIR_Request_create(kind); \ + MPIR_Request_add_ref((req)); \ + } while (0) + +#define MPIDI_OFI_SEND_REQUEST_CREATE_LW(req) \ + do { \ + (req) = MPIR_Request_create(MPIR_REQUEST_KIND__SEND); \ + MPIR_cc_set(&(req)->cc, 0); \ + } while (0) + +#define MPIDI_OFI_SSEND_ACKREQUEST_CREATE(req) \ + do { \ + MPIDI_OFI_ssendack_request_t_tls_alloc(req); \ + } while (0) + +#define WINFO(w,rank) MPIDI_CH4U_WINFO(w,rank) + +__ALWAYS_INLINE__ uintptr_t MPIDI_OFI_winfo_base(MPIR_Win * w, int rank) +{ +#if MPIDI_OFI_ENABLE_MR_SCALABLE + return 0; +#else + return MPIDI_OFI_WIN(w).winfo[rank].base; +#endif +} + +__ALWAYS_INLINE__ uint64_t MPIDI_OFI_winfo_mr_key(MPIR_Win * w, int rank) +{ +#if MPIDI_OFI_ENABLE_MR_SCALABLE + return MPIDI_OFI_WIN(w).mr_key; +#else + return MPIDI_OFI_WIN(w).winfo[rank].mr_key; +#endif +} + +#ifdef MPIDI_OFI_CONFIG_USE_SCALABLE_ENDPOINTS +__ALWAYS_INLINE__ void MPIDI_OFI_win_conditional_cntr_incr(MPIR_Win * win) +{ +} + +__ALWAYS_INLINE__ void MPIDI_OFI_win_cntr_incr(MPIR_Win * win) +{ + (*MPIDI_OFI_WIN(win).issued_cntr)++; +} + +__ALWAYS_INLINE__ void MPIDI_OFI_conditional_cntr_incr() +{ +} + +__ALWAYS_INLINE__ void MPIDI_OFI_cntr_incr() +{ + MPIDI_Global.rma_issued_cntr++; +} +#else +__ALWAYS_INLINE__ void MPIDI_OFI_win_conditional_cntr_incr(MPIR_Win * win) +{ + (*MPIDI_OFI_WIN(win).issued_cntr)++; +} + +__ALWAYS_INLINE__ void MPIDI_OFI_win_cntr_incr(MPIR_Win * win) +{ + (*MPIDI_OFI_WIN(win).issued_cntr)++; +} + +__ALWAYS_INLINE__ void MPIDI_OFI_conditional_cntr_incr() +{ + MPIDI_Global.rma_issued_cntr++; +} + +__ALWAYS_INLINE__ void MPIDI_OFI_cntr_incr() +{ + MPIDI_Global.rma_issued_cntr++; +} +#endif + +/* Externs: see util.c for definition */ +extern int MPIDI_OFI_handle_cq_error_util(ssize_t ret); +extern int MPIDI_OFI_progress_test_no_inline(); +extern int MPIDI_OFI_control_handler(void *am_hdr, + void **data, size_t * data_sz, int *is_contig, + MPIDI_NM_am_completion_handler_fn * cmpl_handler_fn, + MPIR_Request ** req); +extern void MPIDI_OFI_map_create(void **map); +extern void MPIDI_OFI_map_destroy(void *map); +extern void MPIDI_OFI_map_set(void *_map, uint64_t id, void *val); +extern void MPIDI_OFI_map_erase(void *_map, uint64_t id); +extern void *MPIDI_OFI_map_lookup(void *_map, uint64_t id); +extern int MPIDI_OFI_control_dispatch(void *buf); +extern void MPIDI_OFI_index_datatypes(); +extern void MPIDI_OFI_index_allocator_create(void **_indexmap, int start); +extern int MPIDI_OFI_index_allocator_alloc(void *_indexmap); +extern void MPIDI_OFI_index_allocator_free(void *_indexmap, int index); +extern void MPIDI_OFI_index_allocator_destroy(void *_indexmap); + +/* Common Utility functions used by the + * C and C++ components + */ +__ALWAYS_INLINE__ MPIDI_OFI_win_request_t *MPIDI_OFI_win_request_alloc_and_init(int extra) +{ + MPIDI_OFI_win_request_t *req; + req = (MPIDI_OFI_win_request_t *) MPIR_Request_create(MPIR_REQUEST_KIND__RMA); + memset((char *) req + MPIDI_REQUEST_HDR_SIZE, 0, + sizeof(MPIDI_OFI_win_request_t) - MPIDI_REQUEST_HDR_SIZE); + req->noncontig = + (MPIDI_OFI_win_noncontig_t *) MPL_calloc(1, (extra) + sizeof(*(req->noncontig))); + return req; +} + +__ALWAYS_INLINE__ void MPIDI_OFI_win_datatype_unmap(MPIDI_OFI_win_datatype_t * dt) +{ + if (dt->map != &dt->__map) + MPL_free(dt->map); +} + +__ALWAYS_INLINE__ void MPIDI_OFI_win_request_complete(MPIDI_OFI_win_request_t * req) +{ + int count; + MPIR_Assert(HANDLE_GET_MPI_KIND(req->handle) == MPIR_REQUEST); + MPIR_Object_release_ref(req, &count); + MPIR_Assert(count >= 0); + if (count == 0) { + MPIDI_OFI_win_datatype_unmap(&req->noncontig->target_dt); + MPIDI_OFI_win_datatype_unmap(&req->noncontig->origin_dt); + MPIDI_OFI_win_datatype_unmap(&req->noncontig->result_dt); + MPL_free(req->noncontig); + MPIR_Handle_obj_free(&MPIR_Request_mem, (req)); + } +} + +__ALWAYS_INLINE__ fi_addr_t MPIDI_OFI_comm_to_phys(MPIR_Comm * comm, int rank, int ep_family) +{ +#ifdef MPIDI_OFI_CONFIG_USE_SCALABLE_ENDPOINTS + int ep_num = MPIDI_OFI_COMM_TO_EP(comm, rank); + int offset = MPIDI_Global.ctx[ep_num].ctx_offset; + int rx_idx = offset + ep_family; + return fi_rx_addr(MPIDI_OFI_COMM_TO_PHYS(comm, rank), rx_idx, MPIDI_OFI_MAX_ENDPOINTS_BITS); +#else + return MPIDI_OFI_COMM_TO_PHYS(comm, rank); +#endif +} + +__ALWAYS_INLINE__ fi_addr_t MPIDI_OFI_to_phys(int rank, int ep_family) +{ +#ifdef MPIDI_OFI_CONFIG_USE_SCALABLE_ENDPOINTS + int ep_num = 0; + int offset = MPIDI_Global.ctx[ep_num].ctx_offset; + int rx_idx = offset + ep_family; + return fi_rx_addr(MPIDI_OFI_TO_PHYS(0, rank), rx_idx, MPIDI_OFI_MAX_ENDPOINTS_BITS); +#else + return MPIDI_OFI_TO_PHYS(0, rank); +#endif +} + +__ALWAYS_INLINE__ bool MPIDI_OFI_is_tag_sync(uint64_t match_bits) +{ + return (0 != (MPIDI_OFI_SYNC_SEND & match_bits)); +} + +__ALWAYS_INLINE__ uint64_t MPIDI_OFI_init_sendtag(MPIR_Context_id_t contextid, + int source, int tag, uint64_t type, int do_data) +{ + uint64_t match_bits; + match_bits = contextid; + + if (!do_data) { + match_bits = (match_bits << MPIDI_OFI_SOURCE_SHIFT); + match_bits |= source; + } + + match_bits = (match_bits << MPIDI_OFI_TAG_SHIFT); + match_bits |= (MPIDI_OFI_TAG_MASK & tag) | type; + return match_bits; +} + +/* receive posting */ +__ALWAYS_INLINE__ uint64_t MPIDI_OFI_init_recvtag(uint64_t * mask_bits, + MPIR_Context_id_t contextid, + int source, int tag, int do_data) +{ + uint64_t match_bits = 0; + *mask_bits = MPIDI_OFI_PROTOCOL_MASK; + match_bits = contextid; + + if (!do_data) { + match_bits = (match_bits << MPIDI_OFI_SOURCE_SHIFT); + + if (MPI_ANY_SOURCE == source) { + match_bits = (match_bits << MPIDI_OFI_TAG_SHIFT); + *mask_bits |= MPIDI_OFI_SOURCE_MASK; + } + else { + match_bits |= source; + match_bits = (match_bits << MPIDI_OFI_TAG_SHIFT); + } + } + else { + match_bits = (match_bits << MPIDI_OFI_TAG_SHIFT); + } + + if (MPI_ANY_TAG == tag) + *mask_bits |= MPIDI_OFI_TAG_MASK; + else + match_bits |= (MPIDI_OFI_TAG_MASK & tag); + + return match_bits; +} + +__ALWAYS_INLINE__ int MPIDI_OFI_init_get_tag(uint64_t match_bits) +{ + return ((int) (match_bits & MPIDI_OFI_TAG_MASK)); +} + +__ALWAYS_INLINE__ int MPIDI_OFI_init_get_source(uint64_t match_bits) +{ + return ((int) ((match_bits & MPIDI_OFI_SOURCE_MASK) >> MPIDI_OFI_TAG_SHIFT)); +} + +__ALWAYS_INLINE__ MPIR_Request *MPIDI_OFI_context_to_request(void *context) +{ + char *base = (char *) context; + return (MPIR_Request *) container_of(base, MPIR_Request, dev.ch4.netmod); +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_send_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__ALWAYS_INLINE__ int MPIDI_OFI_send_handler(struct fid_ep *ep, const void *buf, size_t len, + void *desc, uint64_t dest, fi_addr_t dest_addr, + uint64_t tag, void *context, int is_inject, + int do_data, int do_lock) +{ + int mpi_errno = MPI_SUCCESS; + + if (is_inject) { + if (do_data) + MPIDI_OFI_CALL_RETRY(fi_tinjectdata(ep, buf, len, dest, dest_addr, tag), tinjectdata, + do_lock); + else + MPIDI_OFI_CALL_RETRY(fi_tinject(ep, buf, len, dest_addr, tag), tinject, do_lock); + } + else { + if (do_data) + MPIDI_OFI_CALL_RETRY(fi_tsenddata(ep, buf, len, desc, dest, dest_addr, tag, context), + tsenddata, do_lock); + else + MPIDI_OFI_CALL_RETRY(fi_tsend(ep, buf, len, desc, dest_addr, tag, context), tsend, + do_lock); + } + + fn_exit: + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#endif diff --git a/src/mpid/ch4/netmod/ofi/ofi_init.h b/src/mpid/ch4/netmod/ofi/ofi_init.h new file mode 100644 index 0000000..09e34d2 --- /dev/null +++ b/src/mpid/ch4/netmod/ofi/ofi_init.h @@ -0,0 +1,896 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_OFI_INIT_H_INCLUDED +#define NETMOD_OFI_INIT_H_INCLUDED + +#include "ofi_impl.h" +#include "mpir_cvars.h" +#include "pmi.h" + +static inline int MPIDI_OFI_choose_provider(struct fi_info *prov, struct fi_info **prov_use); +static inline int MPIDI_OFI_create_endpoint(struct fi_info *prov_use, + struct fid_domain *domain, + struct fid_cq *p2p_cq, + struct fid_cntr *rma_ctr, + struct fid_av *av, + struct fid_ep **ep, int index, int do_scalable_ep); + +#define MPIDI_OFI_CHOOSE_PROVIDER(prov, prov_use,errstr) \ + do { \ + struct fi_info *p = prov; \ + MPIR_ERR_CHKANDJUMP4(p==NULL, mpi_errno,MPI_ERR_OTHER,"**ofid_addrinfo", \ + "**ofid_addrinfo %s %d %s %s",__SHORT_FILE__, \ + __LINE__,FCNAME, errstr); \ + MPIDI_OFI_choose_provider(prov,prov_use); \ + } while (0); + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_init_generic +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_OFI_init_generic(int rank, + int size, + int appnum, + int *tag_ub, + MPIR_Comm * comm_world, + MPIR_Comm * comm_self, + int spawned, + int num_contexts, + void **netmod_contexts, + int do_av_table, + int do_scalable_ep, + int do_am, + int do_tagged, + int do_data, int do_stx_rma, int do_mr_scalable) +{ + int mpi_errno = MPI_SUCCESS, pmi_errno, i, fi_version; + int thr_err = 0, str_errno, maxlen; + char *table = NULL, *provname = NULL; + struct fi_info *hints, *prov, *prov_use; + struct fi_cq_attr cq_attr; + struct fi_cntr_attr cntr_attr; + fi_addr_t *mapped_table; + struct fi_av_attr av_attr; + char valS[MPIDI_KVSAPPSTRLEN], *val; + char keyS[MPIDI_KVSAPPSTRLEN]; + size_t optlen; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_INIT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_INIT); + + CH4_COMPILE_TIME_ASSERT(offsetof(struct MPIR_Request, dev.ch4.netmod) == + offsetof(MPIDI_OFI_chunk_request, context)); + CH4_COMPILE_TIME_ASSERT(offsetof(struct MPIR_Request, dev.ch4.netmod) == + offsetof(MPIDI_OFI_huge_recv_t, context)); + CH4_COMPILE_TIME_ASSERT(offsetof(struct MPIR_Request, dev.ch4.netmod) == + offsetof(MPIDI_OFI_am_repost_request_t, context)); + CH4_COMPILE_TIME_ASSERT(offsetof(struct MPIR_Request, dev.ch4.netmod) == + offsetof(MPIDI_OFI_ssendack_request_t, context)); + CH4_COMPILE_TIME_ASSERT(offsetof(struct MPIR_Request, dev.ch4.netmod) == + offsetof(MPIDI_OFI_dynamic_process_request_t, context)); + CH4_COMPILE_TIME_ASSERT(offsetof(struct MPIR_Request, dev.ch4.netmod) == + offsetof(MPIDI_OFI_win_request_t, context)); + CH4_COMPILE_TIME_ASSERT(offsetof(struct MPIR_Request, dev.ch4.ch4u.netmod_am.ofi.context) == + offsetof(struct MPIR_Request, dev.ch4.netmod.ofi.context)); + CH4_COMPILE_TIME_ASSERT(sizeof(MPIDI_Devreq_t) >= sizeof(MPIDI_OFI_request_t)); + CH4_COMPILE_TIME_ASSERT(sizeof(MPIR_Request) >= sizeof(MPIDI_OFI_win_request_t)); + CH4_COMPILE_TIME_ASSERT(sizeof(MPIDI_Devgpid_t) >= sizeof(MPIDI_OFI_gpid_t)); + CH4_COMPILE_TIME_ASSERT(sizeof(MPIR_Context_id_t) * 8 >= MPIDI_OFI_AM_CONTEXT_ID_BITS); + + *tag_ub = (1ULL << MPIDI_OFI_TAG_SHIFT) - 1; + + MPID_Thread_mutex_create(&MPIDI_OFI_THREAD_UTIL_MUTEX, &thr_err); + MPID_Thread_mutex_create(&MPIDI_OFI_THREAD_PROGRESS_MUTEX, &thr_err); + MPID_Thread_mutex_create(&MPIDI_OFI_THREAD_FI_MUTEX, &thr_err); + MPID_Thread_mutex_create(&MPIDI_OFI_THREAD_SPAWN_MUTEX, &thr_err); + + /* ------------------------------------------------------------------------ */ + /* Hints to filter providers */ + /* See man fi_getinfo for a list */ + /* of all filters */ + /* mode: Select capabilities that this netmod will support */ + /* FI_CONTEXT: This netmod will pass in context into communication */ + /* to optimize storage locality between MPI requests and OFI opaque */ + /* data structures. */ + /* FI_ASYNC_IOV: MPICH will provide storage for iovecs on */ + /* communication calls, avoiding the OFI provider needing to require */ + /* a copy. */ + /* FI_LOCAL_MR unset: Note that we do not set FI_LOCAL_MR, */ + /* which means this netmod does not support exchange of memory */ + /* regions on communication calls. */ + /* caps: Capabilities required from the provider. The bits specified */ + /* with buffered receive, cancel, and remote complete implements */ + /* MPI semantics. */ + /* Tagged: used to support tag matching, 2-sided */ + /* RMA|Atomics: supports MPI 1-sided */ + /* MSG|MULTI_RECV: Supports synchronization protocol for 1-sided */ + /* FI_DIRECTED_RECV: Support not putting the source in the match */ + /* bits */ + /* We expect to register all memory up front for use with this */ + /* endpoint, so the netmod requires dynamic memory regions */ + /* ------------------------------------------------------------------------ */ + + /* ------------------------------------------------------------------------ */ + /* fi_allocinfo: allocate and zero an fi_info structure and all related */ + /* substructures */ + /* ------------------------------------------------------------------------ */ + hints = fi_allocinfo(); + MPIR_Assert(hints != NULL); + + hints->mode = FI_CONTEXT | FI_ASYNC_IOV; /* We can handle contexts */ + hints->caps = 0ULL; /* Tag matching interface */ + hints->caps |= FI_RMA; /* RMA(read/write) */ + hints->caps |= FI_ATOMICS; /* Atomics capabilities */ + + if (do_tagged) { + hints->caps |= FI_TAGGED; /* Tag matching interface */ + } + + if (do_am) { + hints->caps |= FI_MSG; /* Message Queue apis */ + hints->caps |= FI_MULTI_RECV; /* Shared receive buffer */ + } + + if (do_data) { + hints->caps |= FI_DIRECTED_RECV; /* Match source address */ + } + + /* ------------------------------------------------------------------------ */ + /* FI_VERSION provides binary backward and forward compatibility support */ + /* Specify the version of OFI is coded to, the provider will select struct */ + /* layouts that are compatible with this version. */ + /* ------------------------------------------------------------------------ */ + fi_version = FI_VERSION(MPIDI_OFI_MAJOR_VERSION, MPIDI_OFI_MINOR_VERSION); + + /* ------------------------------------------------------------------------ */ + /* Set object options to be filtered by getinfo */ + /* domain_attr: domain attribute requirements */ + /* op_flags: persistent flag settings for an endpoint */ + /* endpoint type: see FI_EP_RDM */ + /* Filters applied (for this netmod, we need providers that can support): */ + /* THREAD_DOMAIN: Progress serialization is handled by netmod (locking) */ + /* PROGRESS_AUTO: request providers that make progress without requiring */ + /* the ADI to dedicate a thread to advance the state */ + /* FI_DELIVERY_COMPLETE: RMA operations are visible in remote memory */ + /* FI_COMPLETION: Selective completions of RMA ops */ + /* FI_EP_RDM: Reliable datagram */ + /* ------------------------------------------------------------------------ */ + hints->addr_format = FI_FORMAT_UNSPEC; + hints->domain_attr->threading = FI_THREAD_DOMAIN; + hints->domain_attr->control_progress = FI_PROGRESS_MANUAL; + hints->domain_attr->data_progress = FI_PROGRESS_MANUAL; + hints->domain_attr->resource_mgmt = FI_RM_ENABLED; + hints->domain_attr->av_type = do_av_table ? FI_AV_TABLE : FI_AV_MAP; + hints->domain_attr->mr_mode = do_mr_scalable ? FI_MR_SCALABLE : FI_MR_BASIC; + hints->tx_attr->op_flags = FI_DELIVERY_COMPLETE | FI_COMPLETION; + hints->tx_attr->msg_order = FI_ORDER_SAS; + hints->tx_attr->comp_order = FI_ORDER_NONE; + hints->rx_attr->op_flags = FI_COMPLETION; + hints->rx_attr->total_buffered_recv = 0; /* FI_RM_ENABLED ensures buffering of unexpected messages */ + hints->ep_attr->type = FI_EP_RDM; + + /* ------------------------------------------------------------------------ */ + /* fi_getinfo: returns information about fabric services for reaching a */ + /* remote node or service. this does not necessarily allocate resources. */ + /* Pass NULL for name/service because we want a list of providers supported */ + /* ------------------------------------------------------------------------ */ + provname = MPIR_CVAR_OFI_USE_PROVIDER ? (char *) MPL_strdup(MPIR_CVAR_OFI_USE_PROVIDER) : NULL; + hints->fabric_attr->prov_name = provname; + MPIDI_OFI_CALL(fi_getinfo(fi_version, NULL, NULL, 0ULL, hints, &prov), addrinfo); + MPIDI_OFI_CHOOSE_PROVIDER(prov, &prov_use, "No suitable provider provider found"); + + MPIDI_Global.prov_use = fi_dupinfo(prov_use); + MPIR_Assert(MPIDI_Global.prov_use); + + /* ------------------------------------------------------------------------ */ + /* Set global attributes attributes based on the provider choice */ + /* ------------------------------------------------------------------------ */ + MPIDI_Global.max_buffered_send = prov_use->tx_attr->inject_size; + MPIDI_Global.max_buffered_write = prov_use->tx_attr->inject_size; + MPIDI_Global.max_send = prov_use->ep_attr->max_msg_size; + MPIDI_Global.max_write = prov_use->ep_attr->max_msg_size; + MPIDI_Global.iov_limit = MIN(prov_use->tx_attr->iov_limit, MPIDI_OFI_IOV_MAX); + MPIDI_Global.rma_iov_limit = MIN(prov_use->tx_attr->rma_iov_limit, MPIDI_OFI_IOV_MAX); + MPIDI_Global.max_mr_key_size = prov_use->domain_attr->mr_key_size; + + if (MPIDI_Global.max_mr_key_size >= 8) { + MPIDI_Global.max_windows_bits = MPIDI_OFI_MAX_WINDOWS_BITS_64; + MPIDI_Global.max_huge_rma_bits = MPIDI_OFI_MAX_HUGE_RMA_BITS_64; + MPIDI_Global.max_huge_rmas = MPIDI_OFI_MAX_HUGE_RMAS_64; + MPIDI_Global.huge_rma_shift = MPIDI_OFI_HUGE_RMA_SHIFT_64; + MPIDI_Global.context_shift = MPIDI_OFI_CONTEXT_SHIFT_64; + } + else if (MPIDI_Global.max_mr_key_size >= 4) { + MPIDI_Global.max_windows_bits = MPIDI_OFI_MAX_WINDOWS_BITS_32; + MPIDI_Global.max_huge_rma_bits = MPIDI_OFI_MAX_HUGE_RMA_BITS_32; + MPIDI_Global.max_huge_rmas = MPIDI_OFI_MAX_HUGE_RMAS_32; + MPIDI_Global.huge_rma_shift = MPIDI_OFI_HUGE_RMA_SHIFT_32; + MPIDI_Global.context_shift = MPIDI_OFI_CONTEXT_SHIFT_32; + } + else if (MPIDI_Global.max_mr_key_size >= 2) { + MPIDI_Global.max_windows_bits = MPIDI_OFI_MAX_WINDOWS_BITS_16; + MPIDI_Global.max_huge_rma_bits = MPIDI_OFI_MAX_HUGE_RMA_BITS_16; + MPIDI_Global.max_huge_rmas = MPIDI_OFI_MAX_HUGE_RMAS_16; + MPIDI_Global.huge_rma_shift = MPIDI_OFI_HUGE_RMA_SHIFT_16; + MPIDI_Global.context_shift = MPIDI_OFI_CONTEXT_SHIFT_16; + } + else { + MPIR_ERR_SETFATALANDJUMP4(mpi_errno, + MPI_ERR_OTHER, + "**ofid_rma_init", + "**ofid_rma_init %s %d %s %s", + __SHORT_FILE__, __LINE__, FCNAME, "Key space too small"); + } + + /* ------------------------------------------------------------------------ */ + /* Open fabric */ + /* The getinfo struct returns a fabric attribute struct that can be used to */ + /* instantiate the virtual or physical network. This opens a "fabric */ + /* provider". We choose the first available fabric, but getinfo */ + /* returns a list. */ + /* ------------------------------------------------------------------------ */ + MPIDI_OFI_CALL(fi_fabric(prov_use->fabric_attr, &MPIDI_Global.fabric, NULL), fabric); + + /* ------------------------------------------------------------------------ */ + /* Create the access domain, which is the physical or virtual network or */ + /* hardware port/collection of ports. Returns a domain object that can be */ + /* used to create endpoints. */ + /* ------------------------------------------------------------------------ */ + MPIDI_OFI_CALL(fi_domain(MPIDI_Global.fabric, prov_use, &MPIDI_Global.domain, NULL), + opendomain); + + /* ------------------------------------------------------------------------ */ + /* Create the objects that will be bound to the endpoint. */ + /* The objects include: */ + /* * dynamic memory-spanning memory region */ + /* * completion queues for events */ + /* * counters for rma operations */ + /* * address vector of other endpoint addresses */ + /* ------------------------------------------------------------------------ */ + + /* ------------------------------------------------------------------------ */ + /* Construct: Completion Queues */ + /* ------------------------------------------------------------------------ */ + memset(&cq_attr, 0, sizeof(cq_attr)); + cq_attr.format = FI_CQ_FORMAT_TAGGED; + MPIDI_OFI_CALL(fi_cq_open(MPIDI_Global.domain, /* In: Domain Object */ + &cq_attr, /* In: Configuration object */ + &MPIDI_Global.p2p_cq, /* Out: CQ Object */ + NULL), opencq); /* In: Context for cq events */ + + /* ------------------------------------------------------------------------ */ + /* Construct: Counters */ + /* ------------------------------------------------------------------------ */ + memset(&cntr_attr, 0, sizeof(cntr_attr)); + cntr_attr.events = FI_CNTR_EVENTS_COMP; + MPIDI_OFI_CALL(fi_cntr_open(MPIDI_Global.domain, /* In: Domain Object */ + &cntr_attr, /* In: Configuration object */ + &MPIDI_Global.rma_cmpl_cntr, /* Out: Counter Object */ + NULL), openct); /* Context: counter events */ + + /* ------------------------------------------------------------------------ */ + /* Construct: Address Vector */ + /* ------------------------------------------------------------------------ */ + + memset(&av_attr, 0, sizeof(av_attr)); + + + if (do_av_table) { + av_attr.type = FI_AV_TABLE; + mapped_table = NULL; + } + else { + av_attr.type = FI_AV_MAP; + mapped_table = (fi_addr_t *) MPL_malloc(size * sizeof(fi_addr_t)); + } + + av_attr.rx_ctx_bits = MPIDI_OFI_MAX_ENDPOINTS_BITS; + + MPIDI_OFI_CALL(fi_av_open(MPIDI_Global.domain, /* In: Domain Object */ + &av_attr, /* In: Configuration object */ + &MPIDI_Global.av, /* Out: AV Object */ + NULL), avopen); /* Context: AV events */ + + /* ------------------------------------------------------------------------ */ + /* Construct: Shared TX Context for RMA */ + /* ------------------------------------------------------------------------ */ + if (do_stx_rma) { + int ret; + struct fi_tx_attr tx_attr; + memset(&tx_attr, 0, sizeof(tx_attr)); + MPIDI_OFI_CALL_RETURN(fi_stx_context(MPIDI_Global.domain, + &tx_attr, + &MPIDI_Global.stx_ctx, NULL /* context */), ret); + if (ret < 0) { + MPL_DBG_MSG(MPIDI_CH4_DBG_GENERAL, VERBOSE, + "Failed to create shared TX context for RMA, " + "falling back to global EP/counter scheme"); + MPIDI_Global.stx_ctx = NULL; + } + } + + /* ------------------------------------------------------------------------ */ + /* Create a transport level communication endpoint. To use the endpoint, */ + /* it must be bound to completion counters or event queues and enabled, */ + /* and the resources consumed by it, such as address vectors, counters, */ + /* completion queues, etc. */ + /* ------------------------------------------------------------------------ */ + MPIDI_OFI_MPI_CALL_POP(MPIDI_OFI_create_endpoint(prov_use, + MPIDI_Global.domain, + MPIDI_Global.p2p_cq, + MPIDI_Global.rma_cmpl_cntr, + MPIDI_Global.av, + &MPIDI_Global.ep, 0, do_scalable_ep)); + + /* ---------------------------------- */ + /* Get our endpoint name and publish */ + /* the socket to the KVS */ + /* ---------------------------------- */ + MPIDI_Global.addrnamelen = FI_NAME_MAX; + MPIDI_OFI_CALL(fi_getname((fid_t) MPIDI_Global.ep, MPIDI_Global.addrname, + &MPIDI_Global.addrnamelen), getname); + MPIR_Assert(MPIDI_Global.addrnamelen <= FI_NAME_MAX); + + val = valS; + str_errno = MPL_STR_SUCCESS; + maxlen = MPIDI_KVSAPPSTRLEN; + memset(val, 0, maxlen); + MPIDI_OFI_STR_CALL(MPL_str_add_binary_arg(&val, &maxlen, "OFI", (char *) &MPIDI_Global.addrname, + MPIDI_Global.addrnamelen), buscard_len); + MPIDI_OFI_PMI_CALL_POP(PMI_KVS_Get_my_name(MPIDI_Global.kvsname, MPIDI_KVSAPPSTRLEN), pmi); + + val = valS; + sprintf(keyS, "OFI-%d", rank); + MPIDI_OFI_PMI_CALL_POP(PMI_KVS_Put(MPIDI_Global.kvsname, keyS, val), pmi); + MPIDI_OFI_PMI_CALL_POP(PMI_KVS_Commit(MPIDI_Global.kvsname), pmi); + MPIDI_OFI_PMI_CALL_POP(PMI_Barrier(), pmi); + + /* -------------------------------- */ + /* Create our address table from */ + /* encoded KVS values */ + /* -------------------------------- */ + table = (char *) MPL_malloc(size * MPIDI_Global.addrnamelen); + maxlen = MPIDI_KVSAPPSTRLEN; + + for (i = 0; i < size; i++) { + sprintf(keyS, "OFI-%d", i); + MPIDI_OFI_PMI_CALL_POP(PMI_KVS_Get(MPIDI_Global.kvsname, keyS, valS, MPIDI_KVSAPPSTRLEN), + pmi); + MPIDI_OFI_STR_CALL(MPL_str_get_binary_arg + (valS, "OFI", (char *) &table[i * MPIDI_Global.addrnamelen], + MPIDI_Global.addrnamelen, &maxlen), buscard_len); + } + + /* -------------------------------- */ + /* Table is constructed. Map it */ + /* -------------------------------- */ + MPIDI_OFI_CALL(fi_av_insert(MPIDI_Global.av, table, size, mapped_table, 0ULL, NULL), avmap); + if (!do_av_table) { /* AV_MAP */ + for (i = 0; i < size; i++) { + MPIDI_OFI_AV(&MPIDIU_get_av(0, i)).dest = mapped_table[i]; + } + MPL_free(mapped_table); + } + + /* -------------------------------- */ + /* Create the id to object maps */ + /* -------------------------------- */ + MPIDI_OFI_map_create(&MPIDI_Global.win_map); + + /* ---------------------------------- */ + /* Initialize Active Message */ + /* ---------------------------------- */ + if (do_am) { + /* Maximum possible message size for short message send (=eager send) + * See MPIDI_OFI_do_send_am for short/long switching logic */ + MPIR_Assert(MPIDI_OFI_DEFAULT_SHORT_SEND_SIZE <= MPIDI_Global.max_send); + MPIDI_Global.am_buf_pool = + MPIDI_CH4U_create_buf_pool(MPIDI_OFI_BUF_POOL_NUM, MPIDI_OFI_BUF_POOL_SIZE); + mpi_errno = MPIDI_CH4U_init(comm_world, comm_self, num_contexts, netmod_contexts); + + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + slist_init(&MPIDI_Global.cq_buff_list); + MPIDI_Global.cq_buff_head = MPIDI_Global.cq_buff_tail = 0; + optlen = MPIDI_OFI_DEFAULT_SHORT_SEND_SIZE; + + MPIDI_OFI_CALL(fi_setopt(&(MPIDI_OFI_EP_RX_MSG(0)->fid), + FI_OPT_ENDPOINT, + FI_OPT_MIN_MULTI_RECV, &optlen, sizeof(optlen)), setopt); + + for (i = 0; i < MPIDI_OFI_NUM_AM_BUFFERS; i++) { + MPIDI_Global.am_bufs[i] = MPL_malloc(MPIDI_OFI_AM_BUFF_SZ); + MPIDI_Global.am_reqs[i].event_id = MPIDI_OFI_EVENT_AM_RECV; + MPIDI_Global.am_reqs[i].index = i; + MPIR_Assert(MPIDI_Global.am_bufs[i]); + MPIDI_Global.am_iov[i].iov_base = MPIDI_Global.am_bufs[i]; + MPIDI_Global.am_iov[i].iov_len = MPIDI_OFI_AM_BUFF_SZ; + MPIDI_Global.am_msg[i].msg_iov = &MPIDI_Global.am_iov[i]; + MPIDI_Global.am_msg[i].desc = NULL; + MPIDI_Global.am_msg[i].addr = FI_ADDR_UNSPEC; + MPIDI_Global.am_msg[i].context = &MPIDI_Global.am_reqs[i].context; + MPIDI_Global.am_msg[i].iov_count = 1; + MPIDI_OFI_CALL_RETRY(fi_recvmsg(MPIDI_OFI_EP_RX_MSG(0), + &MPIDI_Global.am_msg[i], + FI_MULTI_RECV | FI_COMPLETION), prepost, + MPIDI_OFI_CALL_LOCK); + } + + /* Grow the header handlers down */ + MPIDI_Global.am_handlers[MPIDI_OFI_INTERNAL_HANDLER_CONTROL] = MPIDI_OFI_control_handler; + MPIDI_Global.am_send_cmpl_handlers[MPIDI_OFI_INTERNAL_HANDLER_CONTROL] = NULL; + } + OPA_store_int(&MPIDI_Global.am_inflight_inject_emus, 0); + OPA_store_int(&MPIDI_Global.am_inflight_rma_send_mrs, 0); + + /* max_inject_size is temporarily set to 1 inorder to avoid deadlock in + * shm initialization since PMI_Barrier does not call progress and flush its injects */ + MPIDI_Global.max_buffered_send = 1; + MPIDI_Global.max_buffered_write = 1; + + MPIDI_Global.max_buffered_send = prov_use->tx_attr->inject_size; + MPIDI_Global.max_buffered_write = prov_use->tx_attr->inject_size; + + MPIR_Datatype_init_names(); + MPIDI_OFI_index_datatypes(); + + /* -------------------------------- */ + /* Initialize Dynamic Tasking */ + /* -------------------------------- */ + if (spawned) { + char parent_port[MPIDI_MAX_KVS_VALUE_LEN]; + MPIDI_OFI_PMI_CALL_POP(PMI_KVS_Get(MPIDI_Global.kvsname, + MPIDI_PARENT_PORT_KVSKEY, + parent_port, MPIDI_MAX_KVS_VALUE_LEN), pmi); + MPIDI_OFI_MPI_CALL_POP(MPIDI_Comm_connect + (parent_port, NULL, 0, comm_world, &MPIR_Process.comm_parent)); + MPIR_Assert(MPIR_Process.comm_parent != NULL); + MPL_strncpy(MPIR_Process.comm_parent->name, "MPI_COMM_PARENT", MPI_MAX_OBJECT_NAME); + } + + fn_exit: + + /* -------------------------------- */ + /* Free temporary resources */ + /* -------------------------------- */ + if (provname) { + MPL_free(provname); + hints->fabric_attr->prov_name = NULL; + } + + if (prov) + fi_freeinfo(prov); + + fi_freeinfo(hints); + + if (table) + MPL_free(table); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_INIT); + return mpi_errno; + fn_fail: + goto fn_exit; +} + + + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_init +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_init(int rank, + int size, + int appnum, + int *tag_ub, + MPIR_Comm * comm_world, + MPIR_Comm * comm_self, + int spawned, int num_contexts, void **netmod_contexts) +{ + int mpi_errno; + mpi_errno = MPIDI_OFI_init_generic(rank, size, appnum, tag_ub, comm_world, + comm_self, spawned, num_contexts, + netmod_contexts, + MPIDI_OFI_ENABLE_AV_TABLE, + MPIDI_OFI_ENABLE_SCALABLE_ENDPOINTS, + MPIDI_OFI_ENABLE_AM, + MPIDI_OFI_ENABLE_TAGGED, + MPIDI_OFI_ENABLE_DATA, + MPIDI_OFI_ENABLE_STX_RMA, MPIDI_OFI_ENABLE_MR_SCALABLE); + return mpi_errno; +} + + + + +static inline int MPIDI_OFI_finalize_generic(int do_scalable_ep, int do_am, int do_stx_rma) +{ + int thr_err = 0, mpi_errno = MPI_SUCCESS; + int i = 0; + int barrier[2] = { 0 }; + MPIR_Errflag_t errflag = MPIR_ERR_NONE; + MPIR_Comm *comm; + + /* Progress until we drain all inflight RMA send long buffers */ + while (OPA_load_int(&MPIDI_Global.am_inflight_rma_send_mrs) > 0) + MPIDI_OFI_PROGRESS(); + + /* Barrier over allreduce, but force non-immediate send */ + MPIDI_Global.max_buffered_send = 0; + MPIDI_OFI_MPI_CALL_POP(MPIR_Allreduce_impl(&barrier[0], &barrier[1], 1, MPI_INT, + MPI_SUM, MPIR_Process.comm_world, &errflag)); + + /* Progress until we drain all inflight injection emulation requests */ + while (OPA_load_int(&MPIDI_Global.am_inflight_inject_emus) > 0) + MPIDI_OFI_PROGRESS(); + MPIR_Assert(OPA_load_int(&MPIDI_Global.am_inflight_inject_emus) == 0); + + if (do_scalable_ep) { + MPIDI_OFI_CALL(fi_close((fid_t) MPIDI_OFI_EP_TX_TAG(0)), epclose); + MPIDI_OFI_CALL(fi_close((fid_t) MPIDI_OFI_EP_TX_RMA(0)), epclose); + MPIDI_OFI_CALL(fi_close((fid_t) MPIDI_OFI_EP_TX_MSG(0)), epclose); + MPIDI_OFI_CALL(fi_close((fid_t) MPIDI_OFI_EP_TX_CTR(0)), epclose); + + MPIDI_OFI_CALL(fi_close((fid_t) MPIDI_OFI_EP_RX_TAG(0)), epclose); + MPIDI_OFI_CALL(fi_close((fid_t) MPIDI_OFI_EP_RX_RMA(0)), epclose); + MPIDI_OFI_CALL(fi_close((fid_t) MPIDI_OFI_EP_RX_MSG(0)), epclose); + MPIDI_OFI_CALL(fi_close((fid_t) MPIDI_OFI_EP_RX_CTR(0)), epclose); + } + + if (do_stx_rma && MPIDI_Global.stx_ctx != NULL) + MPIDI_OFI_CALL(fi_close(&MPIDI_Global.stx_ctx->fid), stx_ctx_close); + MPIDI_OFI_CALL(fi_close(&MPIDI_Global.ep->fid), epclose); + MPIDI_OFI_CALL(fi_close(&MPIDI_Global.av->fid), avclose); + MPIDI_OFI_CALL(fi_close(&MPIDI_Global.p2p_cq->fid), cqclose); + MPIDI_OFI_CALL(fi_close(&MPIDI_Global.rma_cmpl_cntr->fid), cqclose); + MPIDI_OFI_CALL(fi_close(&MPIDI_Global.domain->fid), domainclose); + + fi_freeinfo(MPIDI_Global.prov_use); + + /* --------------------------------------- */ + /* Free comm world addr table */ + /* --------------------------------------- */ + comm = MPIR_Process.comm_world; + MPIR_Comm_release_always(comm); + + comm = MPIR_Process.comm_self; + MPIR_Comm_release_always(comm); + + MPIDI_CH4U_finalize(); + + MPIDI_OFI_map_destroy(MPIDI_Global.win_map); + + if (do_am) { + for (i = 0; i < MPIDI_OFI_NUM_AM_BUFFERS; i++) + MPL_free(MPIDI_Global.am_bufs[i]); + + MPIDI_CH4R_destroy_buf_pool(MPIDI_Global.am_buf_pool); + + MPIR_Assert(MPIDI_Global.cq_buff_head == MPIDI_Global.cq_buff_tail); + MPIR_Assert(slist_empty(&MPIDI_Global.cq_buff_list)); + } + + PMI_Finalize(); + + MPID_Thread_mutex_destroy(&MPIDI_OFI_THREAD_UTIL_MUTEX, &thr_err); + MPID_Thread_mutex_destroy(&MPIDI_OFI_THREAD_PROGRESS_MUTEX, &thr_err); + MPID_Thread_mutex_destroy(&MPIDI_OFI_THREAD_FI_MUTEX, &thr_err); + MPID_Thread_mutex_destroy(&MPIDI_OFI_THREAD_SPAWN_MUTEX, &thr_err); + + fn_exit: + return mpi_errno; + fn_fail: + goto fn_exit; +} + +static inline int MPIDI_NM_finalize(void) +{ + return MPIDI_OFI_finalize_generic(MPIDI_OFI_ENABLE_SCALABLE_ENDPOINTS, + MPIDI_OFI_ENABLE_AM, MPIDI_OFI_ENABLE_STX_RMA); +} + +static inline void *MPIDI_NM_alloc_mem(size_t size, MPIR_Info * info_ptr) +{ + + void *ap; + ap = MPL_malloc(size); + return ap; +} + +static inline int MPIDI_NM_free_mem(void *ptr) +{ + int mpi_errno = MPI_SUCCESS; + MPL_free(ptr); + + return mpi_errno; +} + +static inline int MPIDI_NM_comm_get_lpid(MPIR_Comm * comm_ptr, + int idx, int *lpid_ptr, MPL_bool is_remote) +{ + int avtid = 0, lpid = 0; + if (comm_ptr->comm_kind == MPIR_COMM_KIND__INTRACOMM) + MPIDIU_comm_rank_to_pid(comm_ptr, idx, &lpid, &avtid); + else if (is_remote) + MPIDIU_comm_rank_to_pid(comm_ptr, idx, &lpid, &avtid); + else { + MPIDIU_comm_rank_to_pid_local(comm_ptr, idx, &lpid, &avtid); + } + + *lpid_ptr = MPIDIU_LPID_CREATE(avtid, lpid); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_gpid_get(MPIR_Comm * comm_ptr, int rank, MPIR_Gpid * gpid) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Assert(rank < comm_ptr->local_size); + size_t sz = sizeof(MPIDI_OFI_GPID(gpid).addr); + MPIDI_OFI_CALL(fi_av_lookup(MPIDI_Global.av, MPIDI_OFI_COMM_TO_PHYS(comm_ptr, rank), + &MPIDI_OFI_GPID(gpid).addr, &sz), avlookup); + MPIR_Assert(sz <= sizeof(MPIDI_OFI_GPID(gpid).addr)); + fn_exit: + return mpi_errno; + fn_fail: + goto fn_exit; +} + +static inline int MPIDI_NM_get_node_id(MPIR_Comm * comm, int rank, MPID_Node_id_t * id_p) +{ + MPIDI_CH4U_get_node_id(comm, rank, id_p); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_get_max_node_id(MPIR_Comm * comm, MPID_Node_id_t * max_id_p) +{ + MPIDI_CH4U_get_max_node_id(comm, max_id_p); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_getallincomm(MPIR_Comm * comm_ptr, + int local_size, MPIR_Gpid local_gpids[], int *singleAVT) +{ + int i; + + for (i = 0; i < comm_ptr->local_size; i++) + MPIDI_GPID_Get(comm_ptr, i, &local_gpids[i]); + + return 0; +} + +static inline int MPIDI_NM_gpid_tolpidarray_generic(int size, + MPIR_Gpid gpid[], int lpid[], int use_av_table) +{ + int i, mpi_errno = MPI_SUCCESS; + int *new_avt_procs; + int n_new_procs = 0; + int max_n_avts; + new_avt_procs = (int *) MPL_malloc(size * sizeof(int)); + max_n_avts = MPIDIU_get_max_n_avts(); + + for (i = 0; i < size; i++) { + int j, k; + char tbladdr[FI_NAME_MAX]; + int found = 0; + + for (k = 0; k < max_n_avts; k++) { + if (MPIDIU_get_av_table(k) == NULL) { + continue; + } + for (j = 0; j < MPIDIU_get_av_table(k)->size; j++) { + size_t sz = sizeof(MPIDI_OFI_GPID(&gpid[i]).addr); + MPIDI_OFI_CALL(fi_av_lookup + (MPIDI_Global.av, MPIDI_OFI_TO_PHYS(k, j), &tbladdr, &sz), avlookup); + MPIR_Assert(sz <= sizeof(MPIDI_OFI_GPID(&gpid[i]).addr)); + + if (!memcmp(tbladdr, MPIDI_OFI_GPID(&gpid[i]).addr, sz)) { + lpid[i] = MPIDIU_LPID_CREATE(k, j); + found = 1; + break; + } + } + } + + if (!found) { + new_avt_procs[n_new_procs] = i; + n_new_procs++; + } + } + + /* create new av_table, insert processes */ + if (n_new_procs > 0) { + int avtid; + MPIDIU_new_avt(n_new_procs, &avtid); + + for (i = 0; i < n_new_procs; i++) { + if (use_av_table) { /* logical addressing */ + MPIDI_OFI_CALL(fi_av_insert + (MPIDI_Global.av, &MPIDI_OFI_GPID(&gpid[new_avt_procs[i]]).addr, 1, + NULL, 0ULL, NULL), avmap); + /* FIXME: get logical address */ + } + else { + MPIDI_OFI_CALL(fi_av_insert + (MPIDI_Global.av, &MPIDI_OFI_GPID(&gpid[new_avt_procs[i]]).addr, 1, + (fi_addr_t *) & MPIDI_OFI_AV(&MPIDIU_get_av(avtid, i)).dest, 0ULL, + NULL), avmap); + } + /* highest bit is marked as 1 to indicate this is a new process */ + lpid[i] = MPIDIU_LPID_CREATE(avtid, i); + MPIDIU_LPID_SET_NEW_AVT_MARK(lpid[i]); + } + } + + + fn_exit: + MPL_free(new_avt_procs); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +static inline int MPIDI_NM_gpid_tolpidarray(int size, MPIR_Gpid gpid[], int lpid[]) +{ + return MPIDI_NM_gpid_tolpidarray_generic(size, gpid, lpid, MPIDI_OFI_ENABLE_AV_TABLE); +} + +static inline int MPIDI_NM_create_intercomm_from_lpids(MPIR_Comm * newcomm_ptr, + int size, const int lpids[]) +{ + return 0; +} + + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_create_endpoint +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_OFI_create_endpoint(struct fi_info *prov_use, + struct fid_domain *domain, + struct fid_cq *p2p_cq, + struct fid_cntr *rma_ctr, + struct fid_av *av, + struct fid_ep **ep, int index, int do_scalable_ep) +{ + int mpi_errno = MPI_SUCCESS; + struct fi_tx_attr tx_attr; + struct fi_rx_attr rx_attr; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_CREATE_ENDPOINT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_CREATE_ENDPOINT); + + if (do_scalable_ep) { + MPIDI_OFI_CALL(fi_scalable_ep(domain, prov_use, ep, NULL), ep); + MPIDI_OFI_CALL(fi_scalable_ep_bind(*ep, &av->fid, 0), bind); + + tx_attr = *prov_use->tx_attr; + tx_attr.caps = FI_TAGGED; + tx_attr.caps |= FI_DELIVERY_COMPLETE; + tx_attr.op_flags = FI_DELIVERY_COMPLETE; + MPIDI_OFI_CALL(fi_tx_context(*ep, index, &tx_attr, &MPIDI_OFI_EP_TX_TAG(index), NULL), ep); + MPIDI_OFI_CALL(fi_ep_bind(MPIDI_OFI_EP_TX_TAG(index), &p2p_cq->fid, FI_SEND), bind); + + tx_attr = *prov_use->tx_attr; + tx_attr.caps = FI_RMA; + tx_attr.caps |= FI_ATOMICS; + tx_attr.caps |= FI_DELIVERY_COMPLETE; + tx_attr.op_flags = FI_DELIVERY_COMPLETE; + MPIDI_OFI_CALL(fi_tx_context(*ep, index + 1, &tx_attr, &MPIDI_OFI_EP_TX_RMA(index), NULL), + ep); + MPIDI_OFI_CALL(fi_ep_bind(MPIDI_OFI_EP_TX_RMA(index), &p2p_cq->fid, FI_SEND), bind); + + tx_attr = *prov_use->tx_attr; + tx_attr.caps = FI_MSG; + tx_attr.op_flags = 0; + MPIDI_OFI_CALL(fi_tx_context(*ep, index + 2, &tx_attr, &MPIDI_OFI_EP_TX_MSG(index), NULL), + ep); + MPIDI_OFI_CALL(fi_ep_bind(MPIDI_OFI_EP_TX_MSG(index), &p2p_cq->fid, FI_SEND), bind); + + tx_attr = *prov_use->tx_attr; + tx_attr.caps = FI_RMA; + tx_attr.caps |= FI_ATOMICS; + tx_attr.caps |= FI_DELIVERY_COMPLETE; + tx_attr.op_flags = FI_DELIVERY_COMPLETE; + MPIDI_OFI_CALL(fi_tx_context(*ep, index + 3, &tx_attr, &MPIDI_OFI_EP_TX_CTR(index), NULL), + ep); + MPIDI_OFI_CALL(fi_ep_bind(MPIDI_OFI_EP_TX_CTR(index), &rma_ctr->fid, FI_WRITE | FI_READ), + bind); + + rx_attr = *prov_use->rx_attr; + rx_attr.caps = FI_TAGGED; + rx_attr.caps |= FI_DELIVERY_COMPLETE; + rx_attr.op_flags = 0; + MPIDI_OFI_CALL(fi_rx_context(*ep, index, &rx_attr, &MPIDI_OFI_EP_RX_TAG(index), NULL), ep); + MPIDI_OFI_CALL(fi_ep_bind(MPIDI_OFI_EP_RX_TAG(index), &p2p_cq->fid, FI_RECV), bind); + + rx_attr = *prov_use->rx_attr; + rx_attr.caps = FI_RMA; + rx_attr.caps |= FI_ATOMICS; + rx_attr.op_flags = 0; + MPIDI_OFI_CALL(fi_rx_context(*ep, index + 1, &rx_attr, &MPIDI_OFI_EP_RX_RMA(index), NULL), + ep); + + /* Note: This bind should cause the "passive target" rx context to never generate an event + * We need this bind for manual progress to ensure that progress is made on the + * rx_ctr or rma operations during completion queue reads */ + if (prov_use->domain_attr->data_progress == FI_PROGRESS_MANUAL) + MPIDI_OFI_CALL(fi_ep_bind(MPIDI_OFI_EP_RX_RMA(index), &p2p_cq->fid, + FI_SEND | FI_RECV | FI_SELECTIVE_COMPLETION), bind); + + rx_attr = *prov_use->rx_attr; + rx_attr.caps = FI_MSG; + rx_attr.caps |= FI_MULTI_RECV; + rx_attr.op_flags = FI_MULTI_RECV; + MPIDI_OFI_CALL(fi_rx_context(*ep, index + 2, &rx_attr, &MPIDI_OFI_EP_RX_MSG(index), NULL), + ep); + MPIDI_OFI_CALL(fi_ep_bind(MPIDI_OFI_EP_RX_MSG(index), &p2p_cq->fid, FI_RECV), bind); + + rx_attr = *prov_use->rx_attr; + rx_attr.caps = FI_RMA; + rx_attr.caps |= FI_ATOMICS; + rx_attr.op_flags = 0; + MPIDI_OFI_CALL(fi_rx_context(*ep, index + 3, &rx_attr, &MPIDI_OFI_EP_RX_CTR(index), NULL), + ep); + + /* See note above */ + if (prov_use->domain_attr->data_progress == FI_PROGRESS_MANUAL) + MPIDI_OFI_CALL(fi_ep_bind(MPIDI_OFI_EP_RX_CTR(index), &p2p_cq->fid, + FI_SEND | FI_RECV | FI_SELECTIVE_COMPLETION), bind); + + MPIDI_OFI_CALL(fi_enable(MPIDI_OFI_EP_TX_TAG(index)), ep_enable); + MPIDI_OFI_CALL(fi_enable(MPIDI_OFI_EP_TX_RMA(index)), ep_enable); + MPIDI_OFI_CALL(fi_enable(MPIDI_OFI_EP_TX_MSG(index)), ep_enable); + MPIDI_OFI_CALL(fi_enable(MPIDI_OFI_EP_TX_CTR(index)), ep_enable); + + MPIDI_OFI_CALL(fi_enable(MPIDI_OFI_EP_RX_TAG(index)), ep_enable); + MPIDI_OFI_CALL(fi_enable(MPIDI_OFI_EP_RX_RMA(index)), ep_enable); + MPIDI_OFI_CALL(fi_enable(MPIDI_OFI_EP_RX_MSG(index)), ep_enable); + MPIDI_OFI_CALL(fi_enable(MPIDI_OFI_EP_RX_CTR(index)), ep_enable); + } + else { + /* ---------------------------------------------------------- */ + /* Bind the CQs, counters, and AV to the endpoint object */ + /* ---------------------------------------------------------- */ + /* "Normal Endpoint */ + MPIDI_OFI_CALL(fi_endpoint(domain, prov_use, ep, NULL), ep); + MPIDI_OFI_CALL(fi_ep_bind(*ep, &p2p_cq->fid, FI_SEND | FI_RECV | FI_SELECTIVE_COMPLETION), + bind); + MPIDI_OFI_CALL(fi_ep_bind(*ep, &rma_ctr->fid, FI_READ | FI_WRITE), bind); + MPIDI_OFI_CALL(fi_ep_bind(*ep, &av->fid, 0), bind); + MPIDI_OFI_CALL(fi_enable(*ep), ep_enable); + } + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_CREATE_ENDPOINT); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +static inline int MPIDI_OFI_choose_provider(struct fi_info *prov, struct fi_info **prov_use) +{ + struct fi_info *p = prov; + int i = 0; + *prov_use = prov; + + if (MPIR_CVAR_OFI_DUMP_PROVIDERS) { + fprintf(stdout, "Dumping Providers(first=%p):\n", prov); + + while (p) { + fprintf(stdout, "%s", fi_tostr(p, FI_TYPE_INFO)); + p = p->next; + } + } + + return i; +} + +#endif /* NETMOD_OFI_INIT_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/ofi/ofi_iovec_util.h b/src/mpid/ch4/netmod/ofi/ofi_iovec_util.h new file mode 100644 index 0000000..36c039f --- /dev/null +++ b/src/mpid/ch4/netmod/ofi/ofi_iovec_util.h @@ -0,0 +1,391 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_OFI_IOVEC_UTIL_H_INCLUDED +#define NETMOD_OFI_IOVEC_UTIL_H_INCLUDED + +#ifdef HAVE_STDINT_H +#include <stdint.h> +#endif /* HAVE_STDINT_H */ +#ifdef HAVE_SYS_UIO_H +#include <sys/uio.h> +#endif /* HAVE_SYS_UIO_H */ +#include <assert.h> +#include "ofi_types.h" + +#define MPIDI_OFI_IOV_DONE 0 +#define MPIDI_OFI_IOV_SUCCESS 0 +#define MPIDI_OFI_IOV_EAGAIN 1 +#define MPIDI_OFI_IOV_ERROR -1 + +#define MPIDI_OFI_INIT_IOV_STATE(var) \ + do { \ + iov_state->var## _base_addr = var; \ + iov_state->var## _count = var## _count; \ + iov_state->var## _iov = var## _iov; \ + iov_state->var## _idx = 0; \ + iov_state->var## _addr = (uintptr_t)iov_state->var## _iov[iov_state->var## _idx].iov_base + iov_state->var## _base_addr; \ + iov_state->var## _size = (uintptr_t)iov_state->var## _iov[iov_state->var## _idx].iov_len; \ + while (iov_state->var## _size == 0) { \ + iov_state->var## _idx++; \ + if (iov_state->var## _idx < iov_state->var## _count) { \ + iov_state->var## _addr = (uintptr_t)iov_state->var## _iov[iov_state->var## _idx].iov_base + iov_state->var## _base_addr; \ + iov_state->var## _size = (uintptr_t)iov_state->var## _iov[iov_state->var## _idx].iov_len; \ + } else { \ + break; \ + } \ + } \ + } while (0) + +#define MPIDI_OFI_NEXT_IOV_STATE(var) \ + do { \ + *var## _addr_next = iov_state->var## _addr; \ + iov_state->var## _addr += buf_size; \ + iov_state->var## _size -= buf_size; \ + while (iov_state->var## _size == 0) { \ + iov_state->var## _idx++; \ + if (iov_state->var## _idx < iov_state->var## _count) { \ + iov_state->var## _addr = (uintptr_t)iov_state->var## _iov[iov_state->var## _idx].iov_base + iov_state->var## _base_addr; \ + iov_state->var## _size = (uintptr_t)iov_state->var## _iov[iov_state->var## _idx].iov_len; \ + } else { \ + break; \ + } \ + } \ + } while (0) + +#define MPIDI_OFI_INIT_IOV(var) \ + do { \ + ((struct iovec*)(&var## _iov[0]))->iov_len = last_len; \ + ((struct iovec*)(&var## _iov[0]))->iov_base = (void*)var## _last_addr; \ + *var## _iovs_nout = 1; \ + } while (0) + +#define MPIDI_OFI_UPDATE_IOV(var) \ + do { \ + var## _idx++; \ + (*var## _iovs_nout)++; \ + ((struct iovec*)(&var## _iov[var## _idx]))->iov_base = (void *)var## _addr; \ + ((struct iovec*)(&var## _iov[var## _idx]))->iov_len = len; \ + } while (0) + +#define MPIDI_OFI_UPDATE_IOV_STATE1(var1,var2) \ + do { \ + if (*var2## _iovs_nout>=var2## _max_iovs) return MPIDI_OFI_IOV_EAGAIN; \ + ((struct iovec*)(&var1## _iov[var1## _idx]))->iov_len += len; \ + var2## _idx++; \ + (*var2## _iovs_nout)++; \ + ((struct iovec*)(&var2## _iov[var2## _idx]))->iov_base = (void *)var2## _addr; \ + ((struct iovec*)(&var2## _iov[var2## _idx]))->iov_len = len; \ + MPIDI_OFI_next_iovec_state(iov_state,&origin_addr, &target_addr, &len); \ + } while (0) + +#define MPIDI_OFI_UPDATE_IOV_STATE2(var1,var2,var3) \ + do { \ + if (*var2## _iovs_nout>=var2## _max_iovs) return MPIDI_OFI_IOV_EAGAIN; \ + if (*var3## _iovs_nout>=var3## _max_iovs) return MPIDI_OFI_IOV_EAGAIN; \ + ((struct iovec*)(&var1## _iov[var1## _idx]))->iov_len += len; \ + var2## _idx++; \ + (*var2## _iovs_nout)++; \ + ((struct iovec*)(&var2## _iov[var2## _idx]))->iov_base = (void *)var2## _addr; \ + ((struct iovec*)(&var2## _iov[var2## _idx]))->iov_len = len; \ + var3## _idx++; \ + (*var3## _iovs_nout)++; \ + ((struct iovec*)(&var3## _iov[var3## _idx]))->iov_base = (void *)var3## _addr; \ + ((struct iovec*)(&var3## _iov[var3## _idx]))->iov_len = len; \ + MPIDI_OFI_next_iovec_state2(iov_state,&origin_addr, &result_addr, &target_addr,&len); \ + } while (0) + +static inline + int MPIDI_OFI_init_iovec_state(MPIDI_OFI_iovec_state_t * iov_state, + uintptr_t origin, + uintptr_t target, + size_t origin_count, + size_t target_count, + size_t buf_limit, + struct iovec *origin_iov, struct iovec *target_iov) +{ + iov_state->buf_limit = buf_limit; + iov_state->buf_limit_left = buf_limit; + + if ((origin_count > 0) && (target_count > 0)) { + MPIDI_OFI_INIT_IOV_STATE(target); + MPIDI_OFI_INIT_IOV_STATE(origin); + } + else + return MPIDI_OFI_IOV_ERROR; + + return MPIDI_OFI_IOV_SUCCESS; +} + +static inline + int MPIDI_OFI_init_iovec_state2(MPIDI_OFI_iovec_state_t * iov_state, + uintptr_t origin, + uintptr_t result, + uintptr_t target, + size_t origin_count, + size_t result_count, + size_t target_count, + size_t buf_limit, + struct iovec *origin_iov, + struct iovec *result_iov, struct iovec *target_iov) +{ + iov_state->buf_limit = buf_limit; + iov_state->buf_limit_left = buf_limit; + + if ((origin_count > 0) && (target_count > 0) && (result_count > 0)) { + MPIDI_OFI_INIT_IOV_STATE(target); + MPIDI_OFI_INIT_IOV_STATE(origin); + MPIDI_OFI_INIT_IOV_STATE(result); + } + else + return MPIDI_OFI_IOV_ERROR; + + return MPIDI_OFI_IOV_SUCCESS; +} + + +static inline + int MPIDI_OFI_peek_iovec_state(MPIDI_OFI_iovec_state_t * iov_state, + uintptr_t * next_origin_addr, + uintptr_t * next_target_addr, size_t * buf_len) +{ + if ((iov_state->origin_size != 0) && (iov_state->target_size != 0)) { + *next_origin_addr = iov_state->origin_addr; + *next_target_addr = iov_state->target_addr; + *buf_len = + MPL_MIN(MPL_MIN(iov_state->target_size, iov_state->origin_size), + iov_state->buf_limit_left); + return MPIDI_OFI_IOV_EAGAIN; + } + else { + if (((iov_state->origin_size != 0) || (iov_state->target_size != 0))) + return MPIDI_OFI_IOV_ERROR; + + return MPIDI_OFI_IOV_DONE; + } +} + +static inline + int MPIDI_OFI_peek_iovec_state2(MPIDI_OFI_iovec_state_t * iov_state, + uintptr_t * next_origin_addr, + uintptr_t * next_result_addr, + uintptr_t * next_target_addr, size_t * buf_len) +{ + if ((iov_state->origin_size != 0) && (iov_state->target_size != 0) && + (iov_state->result_size != 0)) { + *next_origin_addr = iov_state->origin_addr; + *next_result_addr = iov_state->result_addr; + *next_target_addr = iov_state->target_addr; + *buf_len = MPL_MIN(MPL_MIN(MPL_MIN(iov_state->target_size, iov_state->origin_size), + iov_state->result_size), iov_state->buf_limit_left); + return MPIDI_OFI_IOV_EAGAIN; + } + else { + if (((iov_state->origin_size != 0) || (iov_state->target_size != 0) || + (iov_state->result_size != 0))) + return MPIDI_OFI_IOV_ERROR; + + return MPIDI_OFI_IOV_DONE; + } +} + + +static inline + int MPIDI_OFI_next_iovec_state(MPIDI_OFI_iovec_state_t * iov_state, + uintptr_t * origin_addr_next, + uintptr_t * target_addr_next, size_t * buf_len) +{ + if ((iov_state->origin_size != 0) && (iov_state->target_size != 0)) { + uintptr_t buf_size = + MPL_MIN(MPL_MIN(iov_state->target_size, iov_state->origin_size), + iov_state->buf_limit_left); + *buf_len = buf_size; + MPIDI_OFI_NEXT_IOV_STATE(target); + MPIDI_OFI_NEXT_IOV_STATE(origin); + return MPIDI_OFI_IOV_EAGAIN; + } + else { + if (((iov_state->origin_size != 0) || (iov_state->target_size != 0))) + return MPIDI_OFI_IOV_ERROR; + + return MPIDI_OFI_IOV_DONE; + } +} + +static inline + int MPIDI_OFI_next_iovec_state2(MPIDI_OFI_iovec_state_t * iov_state, + uintptr_t * origin_addr_next, + uintptr_t * result_addr_next, + uintptr_t * target_addr_next, size_t * buf_len) +{ + if ((iov_state->origin_size != 0) && (iov_state->target_size != 0) && + (iov_state->result_size != 0)) { + uintptr_t buf_size = + MPL_MIN(MPL_MIN(MPL_MIN(iov_state->target_size, iov_state->origin_size), + iov_state->result_size), iov_state->buf_limit_left); + *buf_len = buf_size; + MPIDI_OFI_NEXT_IOV_STATE(target); + MPIDI_OFI_NEXT_IOV_STATE(origin); + MPIDI_OFI_NEXT_IOV_STATE(result); + return MPIDI_OFI_IOV_EAGAIN; + } + else { + if (((iov_state->origin_size != 0) || (iov_state->target_size != 0) || + (iov_state->result_size != 0))) + return MPIDI_OFI_IOV_ERROR; + + return MPIDI_OFI_IOV_DONE; + } +} + +static inline + int MPIDI_OFI_merge_iov_list(MPIDI_OFI_iovec_state_t * iov_state, + struct iovec *origin_iov, + size_t origin_max_iovs, + struct fi_rma_iov *target_iov, + size_t target_max_iovs, + size_t * origin_iovs_nout, size_t * target_iovs_nout) +{ + int rc; + uintptr_t origin_addr = (uintptr_t) NULL, target_addr = (uintptr_t) NULL; + uintptr_t origin_last_addr = 0, target_last_addr = 0; + int origin_idx = 0, target_idx = 0; + size_t len = 0, last_len = 0; + + CH4_COMPILE_TIME_ASSERT(offsetof(struct iovec, iov_base) == offsetof(struct fi_rma_iov, addr)); + CH4_COMPILE_TIME_ASSERT(offsetof(struct iovec, iov_len) == offsetof(struct fi_rma_iov, len)); + + rc = MPIDI_OFI_next_iovec_state(iov_state, &origin_last_addr, &target_last_addr, &last_len); + assert(rc != MPIDI_OFI_IOV_ERROR); + MPIDI_OFI_INIT_IOV(target); + MPIDI_OFI_INIT_IOV(origin); + iov_state->buf_limit_left -= last_len; + + while (rc > 0) { + rc = MPIDI_OFI_peek_iovec_state(iov_state, &origin_addr, &target_addr, &len); + assert(rc != MPIDI_OFI_IOV_ERROR); + + if (rc == MPIDI_OFI_IOV_DONE) { + iov_state->buf_limit_left = iov_state->buf_limit; + return MPIDI_OFI_IOV_EAGAIN; + } + + if (target_last_addr + last_len == target_addr) { + MPIDI_OFI_UPDATE_IOV_STATE1(target, origin); + } + else if (origin_last_addr + last_len == origin_addr) { + MPIDI_OFI_UPDATE_IOV_STATE1(origin, target); + } + else { + if ((*origin_iovs_nout >= origin_max_iovs) || (*target_iovs_nout >= target_max_iovs)) { + iov_state->buf_limit_left = iov_state->buf_limit; + return MPIDI_OFI_IOV_EAGAIN; + } + + MPIDI_OFI_UPDATE_IOV(target); + MPIDI_OFI_UPDATE_IOV(origin); + MPIDI_OFI_next_iovec_state(iov_state, &origin_addr, &target_addr, &len); + } + + origin_last_addr = origin_addr; + target_last_addr = target_addr; + last_len = len; + iov_state->buf_limit_left -= len; + if (iov_state->buf_limit_left == 0) { + iov_state->buf_limit_left = iov_state->buf_limit; + return MPIDI_OFI_IOV_EAGAIN; + } + } + + if (rc == MPIDI_OFI_IOV_DONE) + return MPIDI_OFI_IOV_DONE; + else { + iov_state->buf_limit_left = iov_state->buf_limit; + return MPIDI_OFI_IOV_EAGAIN; + } +} + +static inline + int MPIDI_OFI_merge_iov_list2(MPIDI_OFI_iovec_state_t * iov_state, + struct iovec *origin_iov, + size_t origin_max_iovs, + struct iovec *result_iov, + size_t result_max_iovs, + struct fi_rma_iov *target_iov, + size_t target_max_iovs, + size_t * origin_iovs_nout, + size_t * result_iovs_nout, size_t * target_iovs_nout) +{ + int rc; + uintptr_t origin_addr = (uintptr_t) NULL, result_addr = (uintptr_t) NULL, target_addr = + (uintptr_t) NULL; + uintptr_t origin_last_addr = 0, result_last_addr = 0, target_last_addr = 0; + int origin_idx = 0, result_idx = 0, target_idx = 0; + size_t len = 0, last_len = 0; + + rc = MPIDI_OFI_next_iovec_state2(iov_state, &origin_last_addr, &result_last_addr, + &target_last_addr, &last_len); + assert(rc != MPIDI_OFI_IOV_ERROR); + MPIDI_OFI_INIT_IOV(target); + MPIDI_OFI_INIT_IOV(origin); + MPIDI_OFI_INIT_IOV(result); + iov_state->buf_limit_left -= last_len; + + while (rc > 0) { + rc = MPIDI_OFI_peek_iovec_state2(iov_state, &origin_addr, &result_addr, &target_addr, &len); + assert(rc != MPIDI_OFI_IOV_ERROR); + + if (rc == MPIDI_OFI_IOV_DONE) { + iov_state->buf_limit_left = iov_state->buf_limit; + return MPIDI_OFI_IOV_EAGAIN; + } + + if (target_last_addr + last_len == target_addr) { + MPIDI_OFI_UPDATE_IOV_STATE2(target, origin, result); + } + else if (origin_last_addr + last_len == origin_addr) { + MPIDI_OFI_UPDATE_IOV_STATE2(origin, target, result); + } + else if (result_last_addr + last_len == result_addr) { + MPIDI_OFI_UPDATE_IOV_STATE2(result, target, origin); + } + else { + if ((*origin_iovs_nout >= origin_max_iovs) || (*target_iovs_nout >= target_max_iovs) || + (*result_iovs_nout >= result_max_iovs)) { + iov_state->buf_limit_left = iov_state->buf_limit; + return MPIDI_OFI_IOV_EAGAIN; + } + + MPIDI_OFI_UPDATE_IOV(target); + MPIDI_OFI_UPDATE_IOV(origin); + MPIDI_OFI_UPDATE_IOV(result); + MPIDI_OFI_next_iovec_state2(iov_state, &origin_addr, &result_addr, &target_addr, &len); + } + + origin_last_addr = origin_addr; + result_last_addr = result_addr; + target_last_addr = target_addr; + last_len = len; + iov_state->buf_limit_left -= len; + if (iov_state->buf_limit_left == 0) { + iov_state->buf_limit_left = iov_state->buf_limit; + return MPIDI_OFI_IOV_EAGAIN; + } + + } + + if (rc == MPIDI_OFI_IOV_DONE) + return MPIDI_OFI_IOV_DONE; + else { + iov_state->buf_limit_left = iov_state->buf_limit; + return MPIDI_OFI_IOV_EAGAIN; + } +} +#endif /* __mpid_iovec_util__h__ */ diff --git a/src/mpid/ch4/netmod/ofi/ofi_op.h b/src/mpid/ch4/netmod/ofi/ofi_op.h new file mode 100644 index 0000000..7666d25 --- /dev/null +++ b/src/mpid/ch4/netmod/ofi/ofi_op.h @@ -0,0 +1,27 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2012 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_OFI_OP_H_INCLUDED +#define NETMOD_OFI_OP_H_INCLUDED + +#include "ofi_impl.h" + +static inline void MPIDI_NM_op_destroy(MPIR_Op * op_p) +{ + return; +} + +static inline void MPIDI_NM_op_commit(MPIR_Op * op_p) +{ + return; +} + + +#endif diff --git a/src/mpid/ch4/netmod/ofi/ofi_pre.h b/src/mpid/ch4/netmod/ofi/ofi_pre.h new file mode 100644 index 0000000..5e2e966 --- /dev/null +++ b/src/mpid/ch4/netmod/ofi/ofi_pre.h @@ -0,0 +1,190 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ + +#ifndef NETMOD_OFI_PRE_H_INCLUDED +#define NETMOD_OFI_PRE_H_INCLUDED + +#include <mpi.h> +#include <rdma/fabric.h> +#include <rdma/fi_endpoint.h> +#include <rdma/fi_domain.h> +#include <rdma/fi_tagged.h> +#include <rdma/fi_rma.h> +#include <rdma/fi_atomic.h> +#include <rdma/fi_cm.h> +#include <rdma/fi_errno.h> + +/* Defines */ + +#define MPIDI_OFI_MAX_AM_HDR_SIZE 128 +#define MPIDI_OFI_AM_HANDLER_ID_BITS 8 +#define MPIDI_OFI_AM_TYPE_BITS 8 +#define MPIDI_OFI_AM_HDR_SZ_BITS 8 +#define MPIDI_OFI_AM_DATA_SZ_BITS 48 +#define MPIDI_OFI_AM_CONTEXT_ID_BITS 16 +#define MPIDI_OFI_AM_RANK_BITS 32 +#define MPIDI_OFI_AM_MSG_HEADER_SIZE (sizeof(MPIDI_OFI_am_header_t)) + +#ifdef MPIDI_OFI_CONFIG_USE_SCALABLE_ENDPOINTS +#define MPIDI_OFI_MAX_ENDPOINTS 256 +#define MPIDI_OFI_MAX_ENDPOINTS_BITS 8 +#else +#define MPIDI_OFI_MAX_ENDPOINTS 1 +#define MPIDI_OFI_MAX_ENDPOINTS_BITS 0 +#endif + +/* Typedefs */ + +struct MPIR_Comm; +struct MPIR_Request; + +typedef struct { + void *huge_send_counters; + void *huge_recv_counters; + void *win_id_allocator; + void *rma_id_allocator; +} MPIDI_OFI_comm_t; +enum { + MPIDI_AMTYPE_SHORT_HDR = 0, + MPIDI_AMTYPE_SHORT, + MPIDI_AMTYPE_LMT_REQ, + MPIDI_AMTYPE_LMT_ACK +}; + +typedef struct { + /* context id and src rank so the target side can + * issue RDMA read operation */ + MPIR_Context_id_t context_id; + int src_rank; + + uint64_t src_offset; + uint64_t sreq_ptr; + uint64_t am_hdr_src; + uint64_t rma_key; +} MPIDI_OFI_lmt_msg_payload_t; + +typedef struct { + uint64_t sreq_ptr; +} MPIDI_OFI_ack_msg_payload_t; + +typedef struct MPIDI_OFI_am_header_t { + uint64_t handler_id:MPIDI_OFI_AM_HANDLER_ID_BITS; + uint64_t am_type:MPIDI_OFI_AM_TYPE_BITS; + uint64_t am_hdr_sz:MPIDI_OFI_AM_HDR_SZ_BITS; + uint64_t data_sz:MPIDI_OFI_AM_DATA_SZ_BITS; + uint64_t payload[0]; +} MPIDI_OFI_am_header_t; + +typedef struct { + MPIDI_OFI_am_header_t hdr; + MPIDI_OFI_ack_msg_payload_t pyld; +} MPIDI_OFI_ack_msg_t; + +typedef struct { + MPIDI_OFI_am_header_t hdr; + MPIDI_OFI_lmt_msg_payload_t pyld; +} MPIDI_OFI_lmt_msg_t; + +typedef struct { + MPIDI_OFI_lmt_msg_payload_t lmt_info; + uint64_t lmt_cntr; + struct fid_mr *lmt_mr; + void *pack_buffer; + void *rreq_ptr; + void *am_hdr; + int (*cmpl_handler_fn) (struct MPIR_Request * req); + uint16_t am_hdr_sz; + uint8_t pad[6]; + MPIDI_OFI_am_header_t msg_hdr; + uint8_t am_hdr_buf[MPIDI_OFI_MAX_AM_HDR_SIZE]; +} MPIDI_OFI_am_request_header_t; + +typedef struct { + struct fi_context context; /* fixed field, do not move */ + int event_id; /* fixed field, do not move */ + MPIDI_OFI_am_request_header_t *req_hdr; +} MPIDI_OFI_am_request_t; + + +typedef struct MPIDI_OFI_noncontig_t { + struct MPIDU_Segment segment; + char pack_buffer[0]; +} MPIDI_OFI_noncontig_t; + +typedef struct { + struct fi_context context; /* fixed field, do not move */ + int event_id; /* fixed field, do not move */ + int util_id; + struct MPIR_Comm *util_comm; + MPI_Datatype datatype; + MPIDI_OFI_noncontig_t *noncontig; + /* persistent send fields */ + union { + struct { + int type; + int rank; + int tag; + int count; + void *buf; + } persist; + struct iovec iov; + void *inject_buf; /* Internal buffer for inject emulation */ + } util; +} MPIDI_OFI_request_t; + +typedef struct { + int index; +} MPIDI_OFI_dt_t; + +typedef struct { + int dummy; +} MPIDI_OFI_op_t; + +struct MPIDI_OFI_win_request; + +/* Stores per-rank information for RMA */ +typedef struct { + int32_t disp_unit; +#ifndef USE_OFI_MR_SCALABLE + /* For MR_BASIC mode we need to store an MR key and a base address of the target window */ + uint64_t mr_key; + uintptr_t base; +#endif +} MPIDI_OFI_win_targetinfo_t; + +typedef struct { + struct fid_mr *mr; + uint64_t mr_key; + struct fid_ep *ep; /* EP with counter & completion */ + struct fid_ep *ep_nocmpl; /* EP with counter only (no completion) */ + uint64_t *issued_cntr; + uint64_t issued_cntr_v; /* main body of an issued counter, + * if we are to use per-window counter */ + struct fid_cntr *cmpl_cntr; + uint64_t win_id; + struct MPIDI_OFI_win_request *syncQ; + MPIDI_OFI_win_targetinfo_t *winfo; +} MPIDI_OFI_win_t; + +typedef struct { + char addr[62]; +} MPIDI_OFI_gpid_t; + +typedef struct { +#ifndef MPIDI_OFI_CONFIG_USE_AV_TABLE + fi_addr_t dest; +#endif +#ifdef MPIDI_OFI_CONFIG_USE_SCALABLE_ENDPOINTS + unsigned ep_idx:MPIDI_OFI_MAX_ENDPOINTS_BITS; +#endif +} MPIDI_OFI_addr_t; + +#endif diff --git a/src/mpid/ch4/netmod/ofi/ofi_probe.h b/src/mpid/ch4/netmod/ofi/ofi_probe.h new file mode 100644 index 0000000..b2eaff4 --- /dev/null +++ b/src/mpid/ch4/netmod/ofi/ofi_probe.h @@ -0,0 +1,168 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_OFI_PROBE_H_INCLUDED +#define NETMOD_OFI_PROBE_H_INCLUDED + +#include "ofi_impl.h" + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_do_iprobe +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_OFI_do_iprobe(int source, + int tag, + MPIR_Comm * comm, + int context_offset, + int *flag, + MPI_Status * status, + MPIR_Request ** message, uint64_t peek_flags) +{ + int mpi_errno = MPI_SUCCESS; + fi_addr_t remote_proc; + uint64_t match_bits, mask_bits; + MPIR_Request r, *rreq; /* don't need to init request, output only */ + struct fi_msg_tagged msg; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_NETMOD_DO_PROBE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_NETMOD_DO_PROBE); + + if (unlikely(MPI_ANY_SOURCE == source)) + remote_proc = FI_ADDR_UNSPEC; + else + remote_proc = MPIDI_OFI_comm_to_phys(comm, source, MPIDI_OFI_API_TAG); + + if (message) + MPIDI_OFI_REQUEST_CREATE(rreq, MPIR_REQUEST_KIND__MPROBE); + else + rreq = &r; + + match_bits = + MPIDI_OFI_init_recvtag(&mask_bits, comm->context_id + context_offset, source, tag, + MPIDI_OFI_ENABLE_DATA); + + MPIDI_OFI_REQUEST(rreq, event_id) = MPIDI_OFI_EVENT_PEEK; + MPIDI_OFI_REQUEST(rreq, util_id) = MPIDI_OFI_PEEK_START; + + msg.msg_iov = NULL; + msg.desc = NULL; + msg.iov_count = 0; + msg.addr = remote_proc; + msg.tag = match_bits; + msg.ignore = mask_bits; + msg.context = (void *) &(MPIDI_OFI_REQUEST(rreq, context)); + msg.data = 0; + + MPIDI_OFI_CALL(fi_trecvmsg + (MPIDI_OFI_EP_RX_TAG(0), &msg, + peek_flags | FI_PEEK | FI_COMPLETION | MPIDI_OFI_ENABLE_DATA), trecvmsg); + MPIDI_OFI_PROGRESS_WHILE(MPIDI_OFI_REQUEST(rreq, util_id) == MPIDI_OFI_PEEK_START); + + switch (MPIDI_OFI_REQUEST(rreq, util_id)) { + case MPIDI_OFI_PEEK_NOT_FOUND: + *flag = 0; + + if (message) + MPIR_Handle_obj_free(&MPIR_Request_mem, rreq); + + goto fn_exit; + break; + + case MPIDI_OFI_PEEK_FOUND: + MPIR_Request_extract_status(rreq, status); + *flag = 1; + + if (message) + *message = rreq; + + break; + + default: + MPIR_Assert(0); + } + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_NETMOD_DO_PROBE); + return mpi_errno; + fn_fail: + goto fn_exit; +} + + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_probe +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_probe(int source, + int tag, MPIR_Comm * comm, int context_offset, MPI_Status * status) +{ + int mpi_errno = MPI_SUCCESS, flag = 0; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_NETMOD_PROBE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_NETMOD_PROBE); + + while (!flag) { + mpi_errno = MPIDI_Iprobe(source, tag, comm, context_offset, &flag, status); + + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + MPIDI_OFI_PROGRESS(); + } + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_NETMOD_PROBE); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_improbe +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_improbe(int source, + int tag, + MPIR_Comm * comm, + int context_offset, + int *flag, MPIR_Request ** message, MPI_Status * status) +{ + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_NETMOD_IMPROBE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_NETMOD_IMPROBE); + /* Set flags for mprobe peek, when ready */ + int mpi_errno = MPIDI_OFI_do_iprobe(source, tag, comm, context_offset, + flag, status, message, FI_CLAIM | FI_COMPLETION); + + if (*flag && *message) { + (*message)->comm = comm; + MPIR_Object_add_ref(comm); + } + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_NETMOD_IMPROBE); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_iprobe +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_iprobe(int source, + int tag, + MPIR_Comm * comm, + int context_offset, int *flag, MPI_Status * status) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_NETMOD_IPROBE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_NETMOD_IPROBE); + mpi_errno = MPIDI_OFI_do_iprobe(source, tag, comm, context_offset, flag, status, NULL, 0ULL); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_NETMOD_IPROBE); + return mpi_errno; +} + +#endif /* NETMOD_OFI_PROBE_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/ofi/ofi_proc.h b/src/mpid/ch4/netmod/ofi/ofi_proc.h new file mode 100644 index 0000000..3f20c0f --- /dev/null +++ b/src/mpid/ch4/netmod/ofi/ofi_proc.h @@ -0,0 +1,29 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_OFI_PROC_H_INCLUDED +#define NETMOD_OFI_PROC_H_INCLUDED + +#include "ofi_impl.h" + +static inline int MPIDI_NM_rank_is_local(int rank, MPIR_Comm * comm) +{ + int ret; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPIDI_NETMOD_RANK_IS_LOCAL); + MPIR_FUNC_VERBOSE_ENTER(MPIDI_NETMOD_RANK_IS_LOCAL); + + ret = MPIDI_CH4U_rank_is_local(rank, comm); + + MPIR_FUNC_VERBOSE_EXIT(MPIDI_NETMOD_RANK_IS_LOCAL); + return ret; +} + +#endif /*NETMOD_OFI_PROC_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/ofi/ofi_progress.h b/src/mpid/ch4/netmod/ofi/ofi_progress.h new file mode 100644 index 0000000..e5da34b --- /dev/null +++ b/src/mpid/ch4/netmod/ofi/ofi_progress.h @@ -0,0 +1,63 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_OFI_PROGRESS_H_INCLUDED +#define NETMOD_OFI_PROGRESS_H_INCLUDED + +#include "ofi_impl.h" +#include "ofi_events.h" +#include "ofi_am_events.h" + +__ALWAYS_INLINE__ + int MPIDI_OFI_progress_generic(void *netmod_context, int blocking, int do_am, int do_tagged) +{ + int mpi_errno; + struct fi_cq_tagged_entry wc[MPIDI_OFI_NUM_CQ_ENTRIES]; + ssize_t ret; + + MPID_THREAD_CS_ENTER(POBJ, MPIDI_OFI_THREAD_FI_MUTEX); + + if (unlikely(MPIDI_OFI_get_buffered(wc, 1))) + mpi_errno = MPIDI_OFI_handle_cq_entries(wc, 1, 1); + else if (likely(1)) { + ret = fi_cq_read(MPIDI_Global.p2p_cq, (void *) wc, MPIDI_OFI_NUM_CQ_ENTRIES); + + if (likely(ret > 0)) + mpi_errno = MPIDI_OFI_handle_cq_entries(wc, ret, 0); + else if (ret == -FI_EAGAIN) + mpi_errno = MPI_SUCCESS; + else + mpi_errno = MPIDI_OFI_handle_cq_error(ret); + } + + MPID_THREAD_CS_EXIT(POBJ, MPIDI_OFI_THREAD_FI_MUTEX); + + MPID_THREAD_CS_EXIT(GLOBAL, MPIR_THREAD_GLOBAL_ALLFUNC_MUTEX); + MPID_THREAD_CS_ENTER(GLOBAL, MPIR_THREAD_GLOBAL_ALLFUNC_MUTEX); + + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_progress +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_progress(void *netmod_context, int blocking) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_PROGRESS); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_PROGRESS); + mpi_errno = MPIDI_OFI_progress_generic(netmod_context, + blocking, MPIDI_OFI_ENABLE_AM, MPIDI_OFI_ENABLE_TAGGED); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_PROGRESS); + return mpi_errno; +} + +#endif /* NETMOD_OFI_PROGRESS_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/ofi/ofi_recv.h b/src/mpid/ch4/netmod/ofi/ofi_recv.h new file mode 100644 index 0000000..ecc93b5 --- /dev/null +++ b/src/mpid/ch4/netmod/ofi/ofi_recv.h @@ -0,0 +1,280 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_OFI_RECV_H_INCLUDED +#define NETMOD_OFI_RECV_H_INCLUDED + +#include "ofi_impl.h" + +#define MPIDI_OFI_ON_HEAP 0 +#define MPIDI_OFI_USE_EXISTING 1 + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_do_irecv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__ALWAYS_INLINE__ int MPIDI_OFI_do_irecv(void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, + int context_offset, + MPIR_Request ** request, int mode, uint64_t flags) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Request *rreq = NULL; + uint64_t match_bits, mask_bits; + MPIR_Context_id_t context_id = comm->recvcontext_id + context_offset; + size_t data_sz; + int dt_contig; + MPI_Aint dt_true_lb; + MPIR_Datatype *dt_ptr; + struct fi_msg_tagged msg; + char *recv_buf; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_DO_IRECV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_DO_IRECV); + + if (mode == MPIDI_OFI_ON_HEAP) /* Branch should compile out */ + MPIDI_OFI_REQUEST_CREATE(rreq, MPIR_REQUEST_KIND__RECV); + else if (mode == MPIDI_OFI_USE_EXISTING) { + rreq = *request; + rreq->kind = MPIR_REQUEST_KIND__RECV; + } + + *request = rreq; + + match_bits = MPIDI_OFI_init_recvtag(&mask_bits, context_id, rank, tag, MPIDI_OFI_ENABLE_DATA); + + MPIDI_Datatype_get_info(count, datatype, dt_contig, data_sz, dt_ptr, dt_true_lb); + MPIDI_OFI_REQUEST(rreq, datatype) = datatype; + dtype_add_ref_if_not_builtin(datatype); + + recv_buf = (char *) buf + dt_true_lb; + + if (!dt_contig) { + MPIDI_OFI_REQUEST(rreq, noncontig) = + (MPIDI_OFI_noncontig_t *) MPL_malloc(data_sz + sizeof(MPID_Segment)); + MPIR_ERR_CHKANDJUMP1(MPIDI_OFI_REQUEST(rreq, noncontig->pack_buffer) == NULL, mpi_errno, + MPI_ERR_OTHER, "**nomem", "**nomem %s", "Recv Pack Buffer alloc"); + recv_buf = MPIDI_OFI_REQUEST(rreq, noncontig->pack_buffer); + MPID_Segment_init(buf, count, datatype, &MPIDI_OFI_REQUEST(rreq, noncontig->segment), 0); + } + else + MPIDI_OFI_REQUEST(rreq, noncontig) = NULL; + + MPIDI_OFI_REQUEST(rreq, util_comm) = comm; + MPIDI_OFI_REQUEST(rreq, util_id) = context_id; + + if (unlikely(data_sz > MPIDI_Global.max_send)) { + MPIDI_OFI_REQUEST(rreq, event_id) = MPIDI_OFI_EVENT_RECV_HUGE; + data_sz = MPIDI_Global.max_send; + } + else + MPIDI_OFI_REQUEST(rreq, event_id) = MPIDI_OFI_EVENT_RECV; + + if (!flags) /* Branch should compile out */ + MPIDI_OFI_CALL_RETRY(fi_trecv(MPIDI_OFI_EP_RX_TAG(0), + recv_buf, + data_sz, + NULL, + (MPI_ANY_SOURCE == + rank) ? FI_ADDR_UNSPEC : MPIDI_OFI_comm_to_phys(comm, rank, + MPIDI_OFI_API_TAG), + match_bits, mask_bits, + (void *) &(MPIDI_OFI_REQUEST(rreq, context))), trecv, + MPIDI_OFI_CALL_LOCK); + else { + MPIDI_OFI_REQUEST(rreq, util.iov).iov_base = recv_buf; + MPIDI_OFI_REQUEST(rreq, util.iov).iov_len = data_sz; + + msg.msg_iov = &MPIDI_OFI_REQUEST(rreq, util.iov); + msg.desc = NULL; + msg.iov_count = 1; + msg.tag = match_bits; + msg.ignore = mask_bits; + msg.context = (void *) &(MPIDI_OFI_REQUEST(rreq, context)); + msg.data = 0; + msg.addr = FI_ADDR_UNSPEC; + + MPIDI_OFI_CALL_RETRY(fi_trecvmsg(MPIDI_OFI_EP_RX_TAG(0), &msg, flags), trecv, + MPIDI_OFI_CALL_LOCK); + } + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_DO_IRECV); + return mpi_errno; + fn_fail: + goto fn_exit; +} + + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_recv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__ALWAYS_INLINE__ int MPIDI_NM_recv(void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, + int context_offset, + MPI_Status * status, MPIR_Request ** request) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_RECV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_RECV); + mpi_errno = MPIDI_OFI_do_irecv(buf, count, datatype, rank, tag, comm, + context_offset, request, MPIDI_OFI_ON_HEAP, 0ULL); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_RECV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_recv_init +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__ALWAYS_INLINE__ int MPIDI_NM_recv_init(void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, + int context_offset, MPIR_Request ** request) +{ + MPIR_Request *rreq; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_RECV_INIT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_RECV_INIT); + + MPIDI_OFI_REQUEST_CREATE((rreq), MPIR_REQUEST_KIND__PREQUEST_RECV); + + *request = rreq; + rreq->comm = comm; + MPIR_Comm_add_ref(comm); + + MPIDI_OFI_REQUEST(rreq, util.persist.buf) = (void *) buf; + MPIDI_OFI_REQUEST(rreq, util.persist.count) = count; + MPIDI_OFI_REQUEST(rreq, datatype) = datatype; + MPIDI_OFI_REQUEST(rreq, util.persist.rank) = rank; + MPIDI_OFI_REQUEST(rreq, util.persist.tag) = tag; + MPIDI_OFI_REQUEST(rreq, util_comm) = comm; + MPIDI_OFI_REQUEST(rreq, util_id) = comm->context_id + context_offset; + rreq->u.persist.real_request = NULL; + + MPIDI_CH4U_request_complete(rreq); + + MPIDI_OFI_REQUEST(rreq, util.persist.type) = MPIDI_PTYPE_RECV; + + if (HANDLE_GET_KIND(datatype) != HANDLE_KIND_BUILTIN) { + MPIR_Datatype *dt_ptr; + MPID_Datatype_get_ptr(datatype, dt_ptr); + MPID_Datatype_add_ref(dt_ptr); + } + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_RECV_INIT); + return MPI_SUCCESS; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_imrecv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__ALWAYS_INLINE__ int MPIDI_NM_imrecv(void *buf, + int count, + MPI_Datatype datatype, + MPIR_Request * message, MPIR_Request ** rreqp) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Request *rreq; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_IMRECV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_IMRECV); + + if (message == NULL) { + MPIDI_OFI_request_create_null_rreq(rreq, mpi_errno, fn_fail); + *rreqp = rreq; + goto fn_exit; + } + + MPIR_Assert(message != NULL); + MPIR_Assert(message->kind == MPIR_REQUEST_KIND__MPROBE); + + *rreqp = rreq = message; + + mpi_errno = MPIDI_OFI_do_irecv(buf, count, datatype, message->status.MPI_SOURCE, + message->status.MPI_TAG, rreq->comm, 0, + &rreq, MPIDI_OFI_USE_EXISTING, FI_CLAIM | FI_COMPLETION); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_IMRECV); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_irecv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__ALWAYS_INLINE__ int MPIDI_NM_irecv(void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_IRECV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_IRECV); + mpi_errno = MPIDI_OFI_do_irecv(buf, count, datatype, rank, tag, comm, + context_offset, request, MPIDI_OFI_ON_HEAP, 0ULL); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_IRECV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_cancel_recv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__ALWAYS_INLINE__ int MPIDI_NM_cancel_recv(MPIR_Request * rreq) +{ + + int mpi_errno = MPI_SUCCESS; + ssize_t ret; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_CANCEL_RECV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_CANCEL_RECV); + +#ifndef MPIDI_BUILD_CH4_SHM + MPIDI_OFI_PROGRESS(); +#endif /* MPIDI_BUILD_CH4_SHM */ + MPID_THREAD_CS_ENTER(POBJ, MPIDI_OFI_THREAD_FI_MUTEX); + ret = fi_cancel((fid_t) MPIDI_OFI_EP_RX_TAG(0), &(MPIDI_OFI_REQUEST(rreq, context))); + MPID_THREAD_CS_EXIT(POBJ, MPIDI_OFI_THREAD_FI_MUTEX); + + if (ret == 0) { + while ((!MPIR_STATUS_GET_CANCEL_BIT(rreq->status)) && (!MPIR_cc_is_complete(&rreq->cc))) { + if ((mpi_errno = + MPIDI_NM_progress(&MPIDI_OFI_REQUEST(rreq, context), 0)) != MPI_SUCCESS) + goto fn_exit; + } + + if (MPIR_STATUS_GET_CANCEL_BIT(rreq->status)) { + MPIR_STATUS_SET_CANCEL_BIT(rreq->status, TRUE); + MPIR_STATUS_SET_COUNT(rreq->status, 0); + MPIDI_CH4U_request_complete(rreq); + } + } + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_CANCEL_RECV); + return mpi_errno; +} + +#endif /* NETMOD_OFI_RECV_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/ofi/ofi_rma.h b/src/mpid/ch4/netmod/ofi/ofi_rma.h new file mode 100644 index 0000000..fda3960 --- /dev/null +++ b/src/mpid/ch4/netmod/ofi/ofi_rma.h @@ -0,0 +1,1298 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_OFI_RMA_H_INCLUDED +#define NETMOD_OFI_RMA_H_INCLUDED + +#include "ofi_impl.h" + +#define MPIDI_OFI_QUERY_ATOMIC_COUNT 0 +#define MPIDI_OFI_QUERY_FETCH_ATOMIC_COUNT 1 +#define MPIDI_OFI_QUERY_COMPARE_ATOMIC_COUNT 2 + +#define MPIDI_OFI_INIT_CHUNK_CONTEXT(win,sigreq) \ + do { \ + if (sigreq) { \ + int tmp; \ + MPIDI_OFI_chunk_request *creq; \ + MPIR_cc_incr((*sigreq)->cc_ptr, &tmp); \ + creq=(MPIDI_OFI_chunk_request*)MPL_malloc(sizeof(*creq)); \ + creq->event_id = MPIDI_OFI_EVENT_CHUNK_DONE; \ + creq->parent = *sigreq; \ + msg.context = &creq->context; \ + MPIDI_OFI_win_conditional_cntr_incr(win); \ + } \ + else MPIDI_OFI_win_cntr_incr(win); \ + } while (0) + +#define MPIDI_OFI_INIT_SIGNAL_REQUEST(win,sigreq,flags,ep) \ + do { \ + if (sigreq) \ + { \ + MPIDI_OFI_REQUEST_CREATE((*(sigreq)), MPIR_REQUEST_KIND__RMA); \ + MPIR_cc_set((*(sigreq))->cc_ptr, 0); \ + *(flags) = FI_COMPLETION; \ + *(ep) = MPIDI_OFI_WIN(win).ep; \ + } \ + else { \ + *(ep) = MPIDI_OFI_WIN(win).ep_nocmpl; \ + *(flags) = 0ULL; \ + } \ + } while (0) + +#define MPIDI_OFI_GET_BASIC_TYPE(a,b) \ + do { \ + if (MPIR_DATATYPE_IS_PREDEFINED(a)) \ + b = a; \ + else { \ + MPIR_Datatype *dt_ptr; \ + MPID_Datatype_get_ptr(a,dt_ptr); \ + b = dt_ptr->basic_type; \ + } \ + } while (0) + +static inline uint32_t MPIDI_OFI_winfo_disp_unit(MPIR_Win * win, int rank) +{ + if (MPIDI_OFI_WIN(win).winfo) + return MPIDI_OFI_WIN(win).winfo[rank].disp_unit; + else + return win->disp_unit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_query_datatype +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_OFI_query_datatype(MPI_Datatype dt, + enum fi_datatype *fi_dt, + MPI_Op op, + enum fi_op *fi_op, size_t * count, size_t * dtsize) +{ + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_QUERY_DT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_QUERY_DT); + MPIR_Datatype *dt_ptr; + int op_index, dt_index, rc; + + MPID_Datatype_get_ptr(dt, dt_ptr); + + /* OP_NULL is the oddball */ + /* todo...change configure to table this correctly */ + dt_index = MPIDI_OFI_DATATYPE(dt_ptr).index; + + if (op == MPI_OP_NULL) + op_index = 14; + else + op_index = (0x000000FFU & op) - 1; + + *fi_dt = (enum fi_datatype) MPIDI_Global.win_op_table[dt_index][op_index].dt; + *fi_op = (enum fi_op) MPIDI_Global.win_op_table[dt_index][op_index].op; + *dtsize = MPIDI_Global.win_op_table[dt_index][op_index].dtsize; + + if (*count == MPIDI_OFI_QUERY_ATOMIC_COUNT) + *count = MPIDI_Global.win_op_table[dt_index][op_index].max_atomic_count; + + if (*count == MPIDI_OFI_QUERY_FETCH_ATOMIC_COUNT) + *count = MPIDI_Global.win_op_table[dt_index][op_index].max_fetch_atomic_count; + + if (*count == MPIDI_OFI_QUERY_COMPARE_ATOMIC_COUNT) + *count = MPIDI_Global.win_op_table[dt_index][op_index].max_compare_atomic_count; + + if (((int) *fi_dt) == -1 || ((int) *fi_op) == -1) + rc = -1; + else + rc = MPI_SUCCESS; + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_QUERY_DT); + return rc; +} + + +static inline void MPIDI_OFI_win_datatype_basic(int count, + MPI_Datatype datatype, + MPIDI_OFI_win_datatype_t * dt) +{ + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_WIN_DATATYPE_BASIC); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_WIN_DATATYPE_BASIC); + + if (datatype != MPI_DATATYPE_NULL) + MPIDI_Datatype_get_info(dt->count = count, + dt->type = datatype, + dt->contig, dt->size, dt->pointer, dt->true_lb); + else + memset(dt, 0, sizeof(*dt)); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_WIN_DATATYPE_BASIC); +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_win_datatype_map +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline void MPIDI_OFI_win_datatype_map(MPIDI_OFI_win_datatype_t * dt) +{ + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_WIN_DATATYPE_MAP); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_WIN_DATATYPE_MAP); + + if (dt->contig) { + dt->num_contig = 1; + dt->map = &dt->__map; + dt->map[0].DLOOP_VECTOR_BUF = (void *) (size_t) dt->true_lb; + dt->map[0].DLOOP_VECTOR_LEN = dt->size; + } + else { + unsigned map_size = dt->pointer->max_contig_blocks * dt->count + 1; + dt->num_contig = map_size; + dt->map = (DLOOP_VECTOR *) MPL_malloc(map_size * sizeof(DLOOP_VECTOR)); + MPIR_Assert(dt->map != NULL); + + MPID_Segment seg; + DLOOP_Offset last = dt->pointer->size * dt->count; + MPIDU_Segment_init(NULL, dt->count, dt->type, &seg, 0); + MPIDU_Segment_pack_vector(&seg, 0, &last, dt->map, &dt->num_contig); + MPIR_Assert((unsigned) dt->num_contig <= map_size); + } + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_WIN_DATATYPE_MAP); +} + +__ALWAYS_INLINE__ int MPIDI_OFI_allocate_win_request_put_get(MPIR_Win * win, + int origin_count, + int target_count, + int target_rank, + MPI_Datatype origin_datatype, + MPI_Datatype target_datatype, + MPIDI_OFI_win_request_t ** winreq, + uint64_t * flags, + struct fid_ep **ep, + MPIR_Request ** sigreq) +{ + int mpi_errno = MPI_SUCCESS; + size_t o_size, t_size; + MPIDI_OFI_win_request_t *req; + + o_size = sizeof(struct iovec); + t_size = sizeof(struct fi_rma_iov); + req = MPIDI_OFI_win_request_alloc_and_init(MPIDI_Global.iov_limit * (o_size + t_size)); + *winreq = req; + + req->noncontig->buf.iov.put_get.originv = (struct iovec *) &req->noncontig->buf.iov_store[0]; + req->noncontig->buf.iov.put_get.targetv = + (struct fi_rma_iov *) &req->noncontig->buf.iov_store[o_size * MPIDI_Global.iov_limit]; + MPIDI_OFI_INIT_SIGNAL_REQUEST(win, sigreq, flags, ep); + MPIDI_OFI_win_datatype_basic(origin_count, origin_datatype, &req->noncontig->origin_dt); + MPIDI_OFI_win_datatype_basic(target_count, target_datatype, &req->noncontig->target_dt); + MPIR_ERR_CHKANDJUMP((req->noncontig->origin_dt.size != req->noncontig->target_dt.size), + mpi_errno, MPI_ERR_SIZE, "**rmasize"); + + MPIDI_OFI_win_datatype_map(&req->noncontig->origin_dt); + MPIDI_OFI_win_datatype_map(&req->noncontig->target_dt); + req->target_rank = target_rank; + fn_exit: + return mpi_errno; + fn_fail: + goto fn_exit; +} + +__ALWAYS_INLINE__ int MPIDI_OFI_allocate_win_request_accumulate(MPIR_Win * win, + int origin_count, + int target_count, + int target_rank, + MPI_Datatype origin_datatype, + MPI_Datatype target_datatype, + MPIDI_OFI_win_request_t ** winreq, + uint64_t * flags, + struct fid_ep **ep, + MPIR_Request ** sigreq) +{ + int mpi_errno = MPI_SUCCESS; + size_t o_size, t_size; + MPIDI_OFI_win_request_t *req; + + o_size = sizeof(struct fi_ioc); + t_size = sizeof(struct fi_rma_ioc); + req = MPIDI_OFI_win_request_alloc_and_init(MPIDI_Global.iov_limit * (o_size + t_size)); + *winreq = req; + + req->noncontig->buf.iov.accumulate.originv = + (struct fi_ioc *) &req->noncontig->buf.iov_store[0]; + req->noncontig->buf.iov.accumulate.targetv = + (struct fi_rma_ioc *) &req->noncontig->buf.iov_store[o_size * MPIDI_Global.iov_limit]; + MPIDI_OFI_INIT_SIGNAL_REQUEST(win, sigreq, flags, ep); + MPIDI_OFI_win_datatype_basic(origin_count, origin_datatype, &req->noncontig->origin_dt); + MPIDI_OFI_win_datatype_basic(target_count, target_datatype, &req->noncontig->target_dt); + MPIR_ERR_CHKANDJUMP((req->noncontig->origin_dt.size != req->noncontig->target_dt.size), + mpi_errno, MPI_ERR_SIZE, "**rmasize"); + + MPIDI_OFI_win_datatype_map(&req->noncontig->origin_dt); + MPIDI_OFI_win_datatype_map(&req->noncontig->target_dt); + req->target_rank = target_rank; + fn_exit: + return mpi_errno; + fn_fail: + goto fn_exit; +} + +__ALWAYS_INLINE__ int MPIDI_OFI_allocate_win_request_get_accumulate(MPIR_Win * win, + int origin_count, + int target_count, + int result_count, + int target_rank, + MPI_Op op, + MPI_Datatype origin_datatype, + MPI_Datatype target_datatype, + MPI_Datatype result_datatype, + MPIDI_OFI_win_request_t ** + winreq, uint64_t * flags, + struct fid_ep **ep, + MPIR_Request ** sigreq) +{ + int mpi_errno = MPI_SUCCESS; + size_t o_size, t_size, r_size; + MPIDI_OFI_win_request_t *req; + + o_size = sizeof(struct fi_ioc); + t_size = sizeof(struct fi_rma_ioc); + r_size = sizeof(struct fi_ioc); + req = MPIDI_OFI_win_request_alloc_and_init(MPIDI_Global.iov_limit * (o_size + t_size + r_size)); + *winreq = req; + + req->noncontig->buf.iov.get_accumulate.originv = + (struct fi_ioc *) &req->noncontig->buf.iov_store[0]; + req->noncontig->buf.iov.get_accumulate.targetv = + (struct fi_rma_ioc *) &req->noncontig->buf.iov_store[o_size * MPIDI_Global.iov_limit]; + req->noncontig->buf.iov.get_accumulate.resultv = + (struct fi_ioc *) &req->noncontig->buf.iov_store[o_size * MPIDI_Global.iov_limit + + t_size * MPIDI_Global.rma_iov_limit]; + MPIDI_OFI_INIT_SIGNAL_REQUEST(win, sigreq, flags, ep); + MPIDI_OFI_win_datatype_basic(origin_count, origin_datatype, &req->noncontig->origin_dt); + MPIDI_OFI_win_datatype_basic(target_count, target_datatype, &req->noncontig->target_dt); + MPIDI_OFI_win_datatype_basic(result_count, result_datatype, &req->noncontig->result_dt); + + MPIR_ERR_CHKANDJUMP((req->noncontig->origin_dt.size != req->noncontig->target_dt.size && + op != MPI_NO_OP), mpi_errno, MPI_ERR_SIZE, "**rmasize"); + MPIR_ERR_CHKANDJUMP((req->noncontig->result_dt.size != req->noncontig->target_dt.size), + mpi_errno, MPI_ERR_SIZE, "**rmasize"); + + if (op != MPI_NO_OP) + MPIDI_OFI_win_datatype_map(&req->noncontig->origin_dt); + + MPIDI_OFI_win_datatype_map(&req->noncontig->target_dt); + MPIDI_OFI_win_datatype_map(&req->noncontig->result_dt); + + req->target_rank = target_rank; + fn_exit: + return mpi_errno; + fn_fail: + goto fn_exit; +} + + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_do_put +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_OFI_do_put(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPIR_Win * win, MPIR_Request ** sigreq) +{ + int rc, mpi_errno = MPI_SUCCESS; + MPIDI_OFI_win_request_t *req; + size_t offset, omax, tmax, tout, oout; + uint64_t flags; + struct fid_ep *ep; + struct fi_msg_rma msg; + unsigned i; + struct iovec *originv; + struct fi_rma_iov *targetv; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_DO_PUT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_DO_PUT); + + MPIDI_OFI_MPI_CALL_POP(MPIDI_OFI_allocate_win_request_put_get(win, + origin_count, + target_count, + target_rank, + origin_datatype, + target_datatype, + &req, &flags, &ep, sigreq)); + + offset = target_disp * MPIDI_OFI_winfo_disp_unit(win, target_rank); + + req->event_id = MPIDI_OFI_EVENT_ABORT; + msg.desc = NULL; + msg.addr = MPIDI_OFI_comm_to_phys(win->comm_ptr, req->target_rank, MPIDI_OFI_API_CTR); + msg.context = NULL; + msg.data = 0; + req->next = MPIDI_OFI_WIN(win).syncQ; + MPIDI_OFI_WIN(win).syncQ = req; + MPIDI_OFI_init_iovec_state(&req->noncontig->iovs, + (uintptr_t) origin_addr, + (uintptr_t) MPIDI_OFI_winfo_base(win, req->target_rank) + offset, + req->noncontig->origin_dt.num_contig, + req->noncontig->target_dt.num_contig, + INT64_MAX, + req->noncontig->origin_dt.map, req->noncontig->target_dt.map); + rc = MPIDI_OFI_IOV_EAGAIN; + + while (rc == MPIDI_OFI_IOV_EAGAIN) { + originv = req->noncontig->buf.iov.put_get.originv; + targetv = req->noncontig->buf.iov.put_get.targetv; + omax = MPIDI_Global.iov_limit; + tmax = MPIDI_Global.rma_iov_limit; + rc = MPIDI_OFI_merge_iov_list(&req->noncontig->iovs, originv, + omax, targetv, tmax, &oout, &tout); + + if (rc == MPIDI_OFI_IOV_DONE) + break; + + for (i = 0; i < tout; i++) + targetv[i].key = MPIDI_OFI_winfo_mr_key(win, target_rank); + MPIR_Assert(rc != MPIDI_OFI_IOV_ERROR); + msg.msg_iov = originv; + msg.iov_count = oout; + msg.rma_iov = targetv; + msg.rma_iov_count = tout; + MPIDI_OFI_CALL_RETRY2(MPIDI_OFI_INIT_CHUNK_CONTEXT(win, sigreq), + fi_writemsg(ep, &msg, flags), rdma_write); + } + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_DO_PUT); + return mpi_errno; + fn_fail: + goto fn_exit; +} + + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_put +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_put(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, MPI_Datatype target_datatype, MPIR_Win * win) +{ + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_PUT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_PUT); + int target_contig, origin_contig, mpi_errno = MPI_SUCCESS; + size_t target_bytes, origin_bytes; + MPI_Aint origin_true_lb, target_true_lb; + size_t offset; + + MPIDI_CH4U_EPOCH_CHECK_SYNC(win, mpi_errno, goto fn_fail); + MPIDI_CH4U_EPOCH_START_CHECK(win, mpi_errno, goto fn_fail); + + MPIDI_Datatype_check_contig_size_lb(target_datatype, target_count, + target_contig, target_bytes, target_true_lb); + MPIDI_Datatype_check_contig_size_lb(origin_datatype, origin_count, + origin_contig, origin_bytes, origin_true_lb); + + MPIR_ERR_CHKANDJUMP((origin_bytes != target_bytes), mpi_errno, MPI_ERR_SIZE, "**rmasize"); + + if (unlikely((origin_bytes == 0) || (target_rank == MPI_PROC_NULL))) + goto fn_exit; + + if (target_rank == win->comm_ptr->rank) { + offset = target_disp * MPIDI_OFI_winfo_disp_unit(win, target_rank); + mpi_errno = MPIR_Localcopy(origin_addr, + origin_count, + origin_datatype, + (char *) win->base + offset, target_count, target_datatype); + goto fn_exit; + } + + if (origin_contig && target_contig && origin_bytes <= MPIDI_Global.max_buffered_write) { + MPIDI_OFI_CALL_RETRY2(MPIDI_OFI_win_cntr_incr(win), + fi_inject_write(MPIDI_OFI_WIN(win).ep_nocmpl, + (char *) origin_addr + origin_true_lb, target_bytes, + MPIDI_OFI_comm_to_phys(win->comm_ptr, target_rank, + MPIDI_OFI_API_CTR), + (uint64_t) (char *) MPIDI_OFI_winfo_base(win, + target_rank) + + target_disp * MPIDI_OFI_winfo_disp_unit(win, + target_rank) + + target_true_lb, MPIDI_OFI_winfo_mr_key(win, + target_rank)), + rdma_inject_write); + } + else { + mpi_errno = MPIDI_OFI_do_put(origin_addr, + origin_count, + origin_datatype, + target_rank, + target_disp, target_count, target_datatype, win, NULL); + } + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_PUT); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_do_get +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_OFI_do_get(void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPIR_Win * win, MPIR_Request ** sigreq) +{ + int rc, mpi_errno = MPI_SUCCESS; + MPIDI_OFI_win_request_t *req; + size_t offset, omax, tmax, tout, oout; + uint64_t flags; + struct fid_ep *ep; + struct fi_msg_rma msg; + struct iovec *originv; + struct fi_rma_iov *targetv; + unsigned i; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_DO_GET); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_DO_GET); + + MPIDI_OFI_MPI_CALL_POP(MPIDI_OFI_allocate_win_request_put_get(win, origin_count, target_count, + target_rank, + origin_datatype, target_datatype, + &req, &flags, &ep, sigreq)); + + offset = target_disp * MPIDI_OFI_winfo_disp_unit(win, target_rank); + req->event_id = MPIDI_OFI_EVENT_ABORT; + msg.desc = NULL; + msg.addr = MPIDI_OFI_comm_to_phys(win->comm_ptr, req->target_rank, MPIDI_OFI_API_CTR); + msg.context = NULL; + msg.data = 0; + req->next = MPIDI_OFI_WIN(win).syncQ; + MPIDI_OFI_WIN(win).syncQ = req; + MPIDI_OFI_init_iovec_state(&req->noncontig->iovs, + (uintptr_t) origin_addr, + (uintptr_t) MPIDI_OFI_winfo_base(win, req->target_rank) + offset, + req->noncontig->origin_dt.num_contig, + req->noncontig->target_dt.num_contig, + INT64_MAX, + req->noncontig->origin_dt.map, req->noncontig->target_dt.map); + rc = MPIDI_OFI_IOV_EAGAIN; + + while (rc == MPIDI_OFI_IOV_EAGAIN) { + originv = req->noncontig->buf.iov.put_get.originv; + targetv = req->noncontig->buf.iov.put_get.targetv; + omax = MPIDI_Global.iov_limit; + tmax = MPIDI_Global.rma_iov_limit; + rc = MPIDI_OFI_merge_iov_list(&req->noncontig->iovs, originv, + omax, targetv, tmax, &oout, &tout); + + if (rc == MPIDI_OFI_IOV_DONE) + break; + + MPIR_Assert(rc != MPIDI_OFI_IOV_ERROR); + + for (i = 0; i < tout; i++) + targetv[i].key = MPIDI_OFI_winfo_mr_key(win, target_rank); + + msg.msg_iov = originv; + msg.iov_count = oout; + msg.rma_iov = targetv; + msg.rma_iov_count = tout; + MPIDI_OFI_CALL_RETRY2(MPIDI_OFI_INIT_CHUNK_CONTEXT(win, sigreq), + fi_readmsg(ep, &msg, flags), rdma_write); + } + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_DO_GET); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_get +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_get(void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, MPI_Datatype target_datatype, MPIR_Win * win) +{ + int origin_contig, target_contig, mpi_errno = MPI_SUCCESS; + MPIDI_OFI_win_datatype_t origin_dt, target_dt; + size_t origin_bytes; + size_t offset; + struct fi_rma_iov riov; + struct iovec iov; + struct fi_msg_rma msg; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_GET); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_GET); + + MPIDI_CH4U_EPOCH_CHECK_SYNC(win, mpi_errno, goto fn_fail); + MPIDI_CH4U_EPOCH_START_CHECK(win, mpi_errno, goto fn_fail); + + MPIDI_Datatype_check_contig_size(origin_datatype, origin_count, origin_contig, origin_bytes); + + if (unlikely((origin_bytes == 0) || (target_rank == MPI_PROC_NULL))) + goto fn_exit; + + if (target_rank == win->comm_ptr->rank) { + offset = target_disp * MPIDI_OFI_winfo_disp_unit(win, target_rank); + mpi_errno = MPIR_Localcopy((char *) win->base + offset, + target_count, + target_datatype, origin_addr, origin_count, origin_datatype); + } + + MPIDI_Datatype_check_contig(origin_datatype, origin_contig); + MPIDI_Datatype_check_contig(target_datatype, target_contig); + + if (origin_contig && target_contig) { + offset = target_disp * MPIDI_OFI_winfo_disp_unit(win, target_rank); + MPIDI_OFI_win_datatype_basic(origin_count, origin_datatype, &origin_dt); + MPIDI_OFI_win_datatype_basic(target_count, target_datatype, &target_dt); + MPIR_ERR_CHKANDJUMP((origin_dt.size != target_dt.size), + mpi_errno, MPI_ERR_SIZE, "**rmasize"); + + msg.desc = NULL; + msg.msg_iov = &iov; + msg.iov_count = 1; + msg.addr = MPIDI_OFI_comm_to_phys(win->comm_ptr, target_rank, MPIDI_OFI_API_CTR); + msg.rma_iov = &riov; + msg.rma_iov_count = 1; + msg.context = NULL; + msg.data = 0; + iov.iov_base = (char *) origin_addr + origin_dt.true_lb; + iov.iov_len = target_dt.size; + riov.addr = + (uint64_t) ((char *) MPIDI_OFI_winfo_base(win, target_rank) + offset + + target_dt.true_lb); + riov.len = target_dt.size; + riov.key = MPIDI_OFI_winfo_mr_key(win, target_rank); + MPIDI_OFI_CALL_RETRY2(MPIDI_OFI_win_cntr_incr(win), + fi_readmsg(MPIDI_OFI_WIN(win).ep_nocmpl, &msg, 0), rdma_write); + } + else { + mpi_errno = MPIDI_OFI_do_get(origin_addr, + origin_count, + origin_datatype, + target_rank, + target_disp, target_count, target_datatype, win, NULL); + } + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_GET); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_rput +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_rput(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPIR_Win * win, MPIR_Request ** request) +{ + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_RPUT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_RPUT); + int mpi_errno; + size_t origin_bytes; + size_t offset; + MPIR_Request *rreq; + + MPIDI_Datatype_check_size(origin_datatype, origin_count, origin_bytes); + + if (unlikely((origin_bytes == 0) || (target_rank == MPI_PROC_NULL))) { + mpi_errno = MPI_SUCCESS; + rreq = MPIR_Request_create(MPIR_REQUEST_KIND__RMA); + MPIR_Request_add_ref(rreq); + MPIDI_CH4U_request_complete(rreq); + goto fn_exit; + } + + if (target_rank == win->comm_ptr->rank) { + rreq = MPIR_Request_create(MPIR_REQUEST_KIND__RMA); + MPIR_Request_add_ref(rreq); + offset = target_disp * MPIDI_OFI_winfo_disp_unit(win, target_rank); + mpi_errno = MPIR_Localcopy(origin_addr, + origin_count, + origin_datatype, + (char *) win->base + offset, target_count, target_datatype); + MPIDI_CH4U_request_complete(rreq); + goto fn_exit; + } + + mpi_errno = MPIDI_OFI_do_put((void *) origin_addr, + origin_count, + origin_datatype, + target_rank, + target_disp, target_count, target_datatype, win, &rreq); + + fn_exit: + *request = rreq; + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_RPUT); + return mpi_errno; +} + + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_compare_and_swap +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_compare_and_swap(const void *origin_addr, + const void *compare_addr, + void *result_addr, + MPI_Datatype datatype, + int target_rank, MPI_Aint target_disp, MPIR_Win * win) +{ + int mpi_errno = MPI_SUCCESS; + enum fi_op fi_op; + enum fi_datatype fi_dt; + MPIDI_OFI_win_datatype_t origin_dt, target_dt, result_dt; + size_t offset, max_size, dt_size; + void *buffer, *tbuffer, *rbuffer; + struct fi_ioc originv, resultv, comparev; + struct fi_rma_ioc targetv; + struct fi_msg_atomic msg; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_COMPARE_AND_SWAP); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_COMPARE_AND_SWAP); + + MPIDI_CH4U_EPOCH_CHECK_SYNC(win, mpi_errno, goto fn_fail); + + offset = target_disp * MPIDI_OFI_winfo_disp_unit(win, target_rank); + + MPIDI_OFI_win_datatype_basic(1, datatype, &origin_dt); + MPIDI_OFI_win_datatype_basic(1, datatype, &result_dt); + MPIDI_OFI_win_datatype_basic(1, datatype, &target_dt); + + if ((origin_dt.size == 0) || (target_rank == MPI_PROC_NULL)) + goto fn_exit; + + buffer = (char *) origin_addr + origin_dt.true_lb; + rbuffer = (char *) result_addr + result_dt.true_lb; + tbuffer = (char *) MPIDI_OFI_winfo_base(win, target_rank) + offset; + + MPIDI_CH4U_EPOCH_START_CHECK(win, mpi_errno, goto fn_fail); + + max_size = MPIDI_OFI_QUERY_COMPARE_ATOMIC_COUNT; + MPIDI_OFI_query_datatype(datatype, &fi_dt, MPI_OP_NULL, &fi_op, &max_size, &dt_size); + + originv.addr = (void *) buffer; + originv.count = 1; + resultv.addr = (void *) rbuffer; + resultv.count = 1; + comparev.addr = (void *) compare_addr; + comparev.count = 1; + targetv.addr = (uint64_t) tbuffer; + targetv.count = 1; + targetv.key = MPIDI_OFI_winfo_mr_key(win, target_rank);; + + msg.msg_iov = &originv; + msg.desc = NULL; + msg.iov_count = 1; + msg.addr = MPIDI_OFI_comm_to_phys(win->comm_ptr, target_rank, MPIDI_OFI_API_CTR); + msg.rma_iov = &targetv; + msg.rma_iov_count = 1; + msg.datatype = fi_dt; + msg.op = fi_op; + msg.context = NULL; + msg.data = 0; + MPIDI_OFI_CALL_RETRY2(MPIDI_OFI_win_cntr_incr(win), + fi_compare_atomicmsg(MPIDI_OFI_WIN(win).ep_nocmpl, &msg, + &comparev, NULL, 1, &resultv, NULL, 1, 0), atomicto); + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_COMPARE_AND_SWAP); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_do_accumulate +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_OFI_do_accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPI_Op op, MPIR_Win * win, MPIR_Request ** sigreq) +{ + int rc, acccheck = 0, mpi_errno = MPI_SUCCESS; + uint64_t flags; + MPIDI_OFI_win_request_t *req; + size_t offset, max_size, dt_size, omax, tmax, tout, oout; + struct fid_ep *ep; + MPI_Datatype basic_type; + enum fi_op fi_op; + enum fi_datatype fi_dt; + struct fi_msg_atomic msg; + struct fi_ioc *originv; + struct fi_rma_ioc *targetv; + unsigned i; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_DO_ACCUMULATE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_DO_ACCUMULATE); + + MPIDI_CH4U_EPOCH_CHECK_SYNC(win, mpi_errno, goto fn_fail); + + MPIDI_OFI_MPI_CALL_POP(MPIDI_OFI_allocate_win_request_accumulate(win, + origin_count, + target_count, + target_rank, + origin_datatype, + target_datatype, + &req, &flags, &ep, sigreq)); + + if ((req->noncontig->origin_dt.size == 0) || (target_rank == MPI_PROC_NULL)) { + MPIDI_OFI_win_request_complete(req); + + if (sigreq) + MPIDI_CH4U_request_release(*sigreq); + + return MPI_SUCCESS; + } + + offset = target_disp * MPIDI_OFI_winfo_disp_unit(win, target_rank); + + MPIDI_CH4U_EPOCH_START_CHECK(win, mpi_errno, goto fn_fail); + + MPIDI_OFI_GET_BASIC_TYPE(target_datatype, basic_type); + + switch (basic_type) { + /* 8 byte types */ + case MPI_FLOAT_INT: + case MPI_2INT: + case MPI_LONG_INT: +#ifdef HAVE_FORTRAN_BINDING + case MPI_2REAL: + case MPI_2INTEGER: +#endif + { + basic_type = MPI_LONG_LONG; + acccheck = 1; + break; + } + + /* 16-byte types */ +#ifdef HAVE_FORTRAN_BINDING + + case MPI_2DOUBLE_PRECISION: +#endif +#ifdef MPICH_DEFINE_2COMPLEX + case MPI_2COMPLEX: +#endif + { + basic_type = MPI_DOUBLE_COMPLEX; + acccheck = 1; + break; + } + + /* Types with pads or too large to handle */ + case MPI_DATATYPE_NULL: + case MPI_SHORT_INT: + case MPI_DOUBLE_INT: + case MPI_LONG_DOUBLE_INT: +#ifdef MPICH_DEFINE_2COMPLEX + case MPI_2DOUBLE_COMPLEX: +#endif + goto am_fallback; + } + + if (acccheck && op != MPI_REPLACE) + goto am_fallback; + + max_size = MPIDI_OFI_QUERY_ATOMIC_COUNT; + + MPIDI_OFI_query_datatype(basic_type, &fi_dt, op, &fi_op, &max_size, &dt_size); + if (max_size == 0) + goto am_fallback; + + req->event_id = MPIDI_OFI_EVENT_ABORT; + req->next = MPIDI_OFI_WIN(win).syncQ; + MPIDI_OFI_WIN(win).syncQ = req; + max_size = max_size * dt_size; + + MPIDI_OFI_init_iovec_state(&req->noncontig->iovs, + (uintptr_t) origin_addr, + (uintptr_t) MPIDI_OFI_winfo_base(win, req->target_rank) + offset, + req->noncontig->origin_dt.num_contig, + req->noncontig->target_dt.num_contig, + max_size, + req->noncontig->origin_dt.map, req->noncontig->target_dt.map); + + msg.desc = NULL; + msg.addr = MPIDI_OFI_comm_to_phys(win->comm_ptr, req->target_rank, MPIDI_OFI_API_CTR); + msg.context = NULL; + msg.data = 0; + msg.datatype = fi_dt; + msg.op = fi_op; + rc = MPIDI_OFI_IOV_EAGAIN; + + while (rc == MPIDI_OFI_IOV_EAGAIN) { + originv = req->noncontig->buf.iov.accumulate.originv; + targetv = req->noncontig->buf.iov.accumulate.targetv; + omax = MPIDI_Global.iov_limit; + tmax = MPIDI_Global.rma_iov_limit; + rc = MPIDI_OFI_merge_iov_list(&req->noncontig->iovs, (struct iovec *) originv, omax, + (struct fi_rma_iov *) targetv, tmax, &oout, &tout); + + if (rc == MPIDI_OFI_IOV_DONE) + break; + + MPIR_Assert(rc != MPIDI_OFI_IOV_ERROR); + + for (i = 0; i < tout; i++) + targetv[i].key = MPIDI_OFI_winfo_mr_key(win, target_rank); + + for (i = 0; i < oout; i++) + originv[i].count /= dt_size; + + for (i = 0; i < tout; i++) + targetv[i].count /= dt_size; + + msg.msg_iov = originv; + msg.iov_count = oout; + msg.rma_iov = targetv; + msg.rma_iov_count = tout; + MPIDI_OFI_CALL_RETRY2(MPIDI_OFI_INIT_CHUNK_CONTEXT(win, sigreq), + fi_atomicmsg(ep, &msg, flags), rdma_atomicto); + } + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_DO_ACCUMULATE); + return mpi_errno; + fn_fail: + goto fn_exit; + am_fallback: + /* Fall back to active message */ + MPIDI_OFI_win_request_complete(req); + return MPIDI_CH4U_accumulate(origin_addr, origin_count, origin_datatype, + target_rank, target_disp, target_count, target_datatype, op, win); +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_get_accumulate +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_OFI_do_get_accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + void *result_addr, + int result_count, + MPI_Datatype result_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPI_Op op, MPIR_Win * win, MPIR_Request ** sigreq) +{ + int rc, acccheck = 0, mpi_errno = MPI_SUCCESS; + uint64_t flags; + MPIDI_OFI_win_request_t *req; + size_t offset, max_size, dt_size, omax, rmax, tmax, tout, rout, oout; + struct fid_ep *ep; + MPI_Datatype rt, basic_type, basic_type_res; + enum fi_op fi_op; + enum fi_datatype fi_dt; + struct fi_msg_atomic msg; + struct fi_ioc *originv, *resultv; + struct fi_rma_ioc *targetv; + unsigned i; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_GET_ACCUMULATE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_GET_ACCUMULATE); + + MPIDI_CH4U_EPOCH_CHECK_SYNC(win, mpi_errno, goto fn_fail); + + MPIDI_OFI_MPI_CALL_POP(MPIDI_OFI_allocate_win_request_get_accumulate(win, + origin_count, + target_count, + result_count, + target_rank, + op, + origin_datatype, + target_datatype, + result_datatype, + &req, &flags, &ep, + sigreq)); + + if ((req->noncontig->result_dt.size == 0) || (target_rank == MPI_PROC_NULL)) { + MPIDI_OFI_win_request_complete(req); + + if (sigreq) + MPIDI_CH4U_request_release(*sigreq); + + goto fn_exit; + } + + offset = target_disp * MPIDI_OFI_winfo_disp_unit(win, target_rank); + + MPIDI_CH4U_EPOCH_START_CHECK(win, mpi_errno, goto fn_fail); + + MPIDI_OFI_GET_BASIC_TYPE(target_datatype, basic_type); + rt = result_datatype; + + switch (basic_type) { + /* 8 byte types */ + case MPI_FLOAT_INT: + case MPI_2INT: + case MPI_LONG_INT: +#ifdef HAVE_FORTRAN_BINDING + case MPI_2REAL: + case MPI_2INTEGER: +#endif + { + basic_type = rt = MPI_LONG_LONG; + acccheck = 1; + break; + } + + /* 16-byte types */ +#ifdef HAVE_FORTRAN_BINDING + + case MPI_2DOUBLE_PRECISION: +#endif +#ifdef MPICH_DEFINE_2COMPLEX + case MPI_2COMPLEX: +#endif + { + basic_type = rt = MPI_DOUBLE_COMPLEX; + acccheck = 1; + break; + } + + /* Types with pads or too large to handle */ + case MPI_DATATYPE_NULL: + case MPI_SHORT_INT: + case MPI_DOUBLE_INT: + case MPI_LONG_DOUBLE_INT: +#ifdef MPICH_DEFINE_2COMPLEX + case MPI_2DOUBLE_COMPLEX: +#endif + goto am_fallback; + break; + } + + if (acccheck && op != MPI_REPLACE && op != MPI_NO_OP) + goto am_fallback; + + MPIDI_OFI_GET_BASIC_TYPE(rt, basic_type_res); + MPIR_Assert(basic_type_res != MPI_DATATYPE_NULL); + + max_size = MPIDI_OFI_QUERY_FETCH_ATOMIC_COUNT; + MPIDI_OFI_query_datatype(basic_type_res, &fi_dt, op, &fi_op, &max_size, &dt_size); + max_size = max_size * dt_size; + if (max_size == 0) + goto am_fallback; + + req->event_id = MPIDI_OFI_EVENT_RMA_DONE; + req->next = MPIDI_OFI_WIN(win).syncQ; + MPIDI_OFI_WIN(win).syncQ = req; + + if (op != MPI_NO_OP) + MPIDI_OFI_init_iovec_state2(&req->noncontig->iovs, + (uintptr_t) origin_addr, + (uintptr_t) result_addr, + (uintptr_t) MPIDI_OFI_winfo_base(win, + req->target_rank) + offset, + req->noncontig->origin_dt.num_contig, + req->noncontig->result_dt.num_contig, + req->noncontig->target_dt.num_contig, max_size, + req->noncontig->origin_dt.map, req->noncontig->result_dt.map, + req->noncontig->target_dt.map); + else + MPIDI_OFI_init_iovec_state(&req->noncontig->iovs, + (uintptr_t) result_addr, + (uintptr_t) MPIDI_OFI_winfo_base(win, req->target_rank) + offset, + req->noncontig->result_dt.num_contig, + req->noncontig->target_dt.num_contig, + max_size, + req->noncontig->result_dt.map, req->noncontig->target_dt.map); + + msg.desc = NULL; + msg.addr = MPIDI_OFI_comm_to_phys(win->comm_ptr, req->target_rank, MPIDI_OFI_API_CTR); + msg.context = NULL; + msg.data = 0; + msg.datatype = fi_dt; + msg.op = fi_op; + rc = MPIDI_OFI_IOV_EAGAIN; + + while (rc == MPIDI_OFI_IOV_EAGAIN) { + originv = req->noncontig->buf.iov.get_accumulate.originv; + targetv = req->noncontig->buf.iov.get_accumulate.targetv; + resultv = req->noncontig->buf.iov.get_accumulate.resultv; + omax = rmax = MPIDI_Global.iov_limit; + tmax = MPIDI_Global.rma_iov_limit; + + if (op != MPI_NO_OP) + rc = MPIDI_OFI_merge_iov_list2(&req->noncontig->iovs, (struct iovec *) originv, + omax, (struct iovec *) resultv, rmax, + (struct fi_rma_iov *) targetv, tmax, &oout, &rout, + &tout); + else { + oout = 0; + rc = MPIDI_OFI_merge_iov_list(&req->noncontig->iovs, (struct iovec *) resultv, + rmax, (struct fi_rma_iov *) targetv, tmax, &rout, &tout); + } + + if (rc == MPIDI_OFI_IOV_DONE) + break; + + MPIR_Assert(rc != MPIDI_OFI_IOV_ERROR); + + for (i = 0; i < oout; i++) + originv[i].count /= dt_size; + + for (i = 0; i < rout; i++) + resultv[i].count /= dt_size; + + for (i = 0; i < tout; i++) { + targetv[i].count /= dt_size; + targetv[i].key = MPIDI_OFI_winfo_mr_key(win, target_rank); + } + + msg.msg_iov = originv; + msg.iov_count = oout; + msg.rma_iov = targetv; + msg.rma_iov_count = tout; + MPIDI_OFI_CALL_RETRY2(MPIDI_OFI_INIT_CHUNK_CONTEXT(win, sigreq), + fi_fetch_atomicmsg(ep, &msg, resultv, + NULL, rout, flags), rdma_readfrom); + } + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_GET_ACCUMULATE); + return mpi_errno; + fn_fail: + goto fn_exit; + am_fallback: + MPIDI_OFI_win_request_complete(req); + return MPIDI_CH4U_get_accumulate(origin_addr, origin_count, origin_datatype, + result_addr, result_count, result_datatype, + target_rank, target_disp, target_count, + target_datatype, op, win); +} + + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_raccumulate +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_raccumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPI_Op op, MPIR_Win * win, MPIR_Request ** request) +{ + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_RACCUMULATE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_RACCUMULATE); + MPIR_Request *rreq; + int mpi_errno = MPIDI_OFI_do_accumulate((void *) origin_addr, + origin_count, + origin_datatype, + target_rank, + target_disp, + target_count, + target_datatype, + op, + win, + &rreq); + *request = rreq; + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_RACCUMULATE); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_rget_accumulate +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_rget_accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + void *result_addr, + int result_count, + MPI_Datatype result_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPI_Op op, MPIR_Win * win, MPIR_Request ** request) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Request *rreq; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_RGET_ACCUMULATE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_RGET_ACCUMULATE); + + mpi_errno = MPIDI_OFI_do_get_accumulate(origin_addr, origin_count, origin_datatype, + result_addr, result_count, result_datatype, + target_rank, target_disp, target_count, + target_datatype, op, win, &rreq); + *request = rreq; + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_RGET_ACCUMULATE); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_fetch_and_op +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_fetch_and_op(const void *origin_addr, + void *result_addr, + MPI_Datatype datatype, + int target_rank, + MPI_Aint target_disp, MPI_Op op, MPIR_Win * win) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_FETCH_AND_OP); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_FETCH_AND_OP); + + /* This can be optimized by directly calling the fi directly + * and avoiding all the datatype processing of the full + * MPIDI_Get_accumulate + */ + mpi_errno = MPIDI_OFI_do_get_accumulate(origin_addr, 1, datatype, + result_addr, 1, datatype, + target_rank, target_disp, 1, datatype, op, win, NULL); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_FETCH_AND_OP); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_rget +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_rget(void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPIR_Win * win, MPIR_Request ** request) +{ + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_RGET); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_RGET); + int mpi_errno; + size_t origin_bytes; + size_t offset; + MPIR_Request *rreq; + + MPIDI_Datatype_check_size(origin_datatype, origin_count, origin_bytes); + + if (unlikely((origin_bytes == 0) || (target_rank == MPI_PROC_NULL))) { + mpi_errno = MPI_SUCCESS; + rreq = MPIR_Request_create(MPIR_REQUEST_KIND__RMA); + MPIR_Request_add_ref(rreq); + MPIDI_CH4U_request_complete(rreq); + goto fn_exit; + } + + if (target_rank == win->comm_ptr->rank) { + rreq = MPIR_Request_create(MPIR_REQUEST_KIND__RMA); + MPIR_Request_add_ref(rreq); + offset = win->disp_unit * target_disp; + mpi_errno = MPIR_Localcopy((char *) win->base + offset, + target_count, + target_datatype, origin_addr, origin_count, origin_datatype); + MPIDI_CH4U_request_complete(rreq); + goto fn_exit; + } + + mpi_errno = MPIDI_OFI_do_get(origin_addr, + origin_count, + origin_datatype, + target_rank, + target_disp, target_count, target_datatype, win, &rreq); + fn_exit: + *request = rreq; + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_RGET); + return mpi_errno; +} + + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_get_accumulate +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_get_accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + void *result_addr, + int result_count, + MPI_Datatype result_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, MPI_Op op, MPIR_Win * win) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_GET_ACCUMULATE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_GET_ACCUMULATE); + mpi_errno = MPIDI_OFI_do_get_accumulate(origin_addr, origin_count, origin_datatype, + result_addr, result_count, result_datatype, + target_rank, target_disp, target_count, + target_datatype, op, win, NULL); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_GET_ACCUMULATE); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_accumulate +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, MPI_Op op, MPIR_Win * win) +{ + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_ACCUMULATE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_ACCUMULATE); + int mpi_errno = MPIDI_OFI_do_accumulate(origin_addr, + origin_count, + origin_datatype, + target_rank, + target_disp, + target_count, + target_datatype, + op, + win, + NULL); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_ACCUMULATE); + return mpi_errno; +} + +#endif /* NETMOD_OFI_RMA_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/ofi/ofi_send.h b/src/mpid/ch4/netmod/ofi/ofi_send.h new file mode 100644 index 0000000..8cc943d --- /dev/null +++ b/src/mpid/ch4/netmod/ofi/ofi_send.h @@ -0,0 +1,566 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_OFI_SEND_H_INCLUDED +#define NETMOD_OFI_SEND_H_INCLUDED + +#include "ofi_impl.h" +#include <../mpi/pt2pt/bsendutil.h> + +#define MPIDI_OFI_SENDPARAMS const void *buf,int count,MPI_Datatype datatype, \ + int rank,int tag,MPIR_Comm *comm, \ + int context_offset,MPIR_Request **request + +#define MPIDI_OFI_SENDARGS buf,count,datatype,rank,tag, \ + comm,context_offset,request + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_send_lightweight +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__ALWAYS_INLINE__ int MPIDI_OFI_send_lightweight(const void *buf, + size_t data_sz, + int rank, + int tag, MPIR_Comm * comm, int context_offset) +{ + int mpi_errno = MPI_SUCCESS; + uint64_t match_bits; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_SEND_LIGHTWEIGHT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_SEND_LIGHTWEIGHT); + match_bits = + MPIDI_OFI_init_sendtag(comm->context_id + context_offset, comm->rank, tag, 0, + MPIDI_OFI_ENABLE_DATA); + mpi_errno = + MPIDI_OFI_send_handler(MPIDI_OFI_EP_TX_TAG(0), buf, data_sz, NULL, comm->rank, + MPIDI_OFI_comm_to_phys(comm, rank, MPIDI_OFI_API_TAG), match_bits, + NULL, MPIDI_OFI_DO_INJECT, MPIDI_OFI_ENABLE_DATA, + MPIDI_OFI_CALL_LOCK); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_SEND_LIGHTWEIGHT); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_send_lightweight_request +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__ALWAYS_INLINE__ int MPIDI_OFI_send_lightweight_request(const void *buf, + size_t data_sz, + int rank, + int tag, + MPIR_Comm * comm, + int context_offset, + MPIR_Request ** request) +{ + int mpi_errno = MPI_SUCCESS; + uint64_t match_bits; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_SEND_LIGHTWEIGHT_REQUEST); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_SEND_LIGHTWEIGHT_REQUEST); + MPIR_Request *r; + MPIDI_OFI_SEND_REQUEST_CREATE_LW(r); + *request = r; + match_bits = + MPIDI_OFI_init_sendtag(comm->context_id + context_offset, comm->rank, tag, 0, + MPIDI_OFI_ENABLE_DATA); + mpi_errno = + MPIDI_OFI_send_handler(MPIDI_OFI_EP_TX_TAG(0), buf, data_sz, NULL, comm->rank, + MPIDI_OFI_comm_to_phys(comm, rank, MPIDI_OFI_API_TAG), match_bits, + NULL, MPIDI_OFI_DO_INJECT, MPIDI_OFI_ENABLE_DATA, + MPIDI_OFI_CALL_LOCK); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_SEND_LIGHTWEIGHT_REQUEST); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_send_normal +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__ALWAYS_INLINE__ int MPIDI_OFI_send_normal(MPIDI_OFI_SENDPARAMS, + int dt_contig, + size_t data_sz, + MPIR_Datatype * dt_ptr, + MPI_Aint dt_true_lb, uint64_t type) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Request *sreq = NULL; + MPI_Aint last; + char *send_buf; + uint64_t match_bits; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_SEND_NORMAL); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_SEND_NORMAL); + + MPIDI_OFI_REQUEST_CREATE(sreq, MPIR_REQUEST_KIND__SEND); + *request = sreq; + match_bits = + MPIDI_OFI_init_sendtag(comm->context_id + context_offset, comm->rank, tag, type, + MPIDI_OFI_ENABLE_DATA); + MPIDI_OFI_REQUEST(sreq, event_id) = MPIDI_OFI_EVENT_SEND; + MPIDI_OFI_REQUEST(sreq, datatype) = datatype; + dtype_add_ref_if_not_builtin(datatype); + + if (type == MPIDI_OFI_SYNC_SEND) { /* Branch should compile out */ + int c = 1; + uint64_t ssend_match, ssend_mask; + MPIDI_OFI_ssendack_request_t *ackreq; + MPIDI_OFI_SSEND_ACKREQUEST_CREATE(ackreq); + ackreq->event_id = MPIDI_OFI_EVENT_SSEND_ACK; + ackreq->signal_req = sreq; + MPIR_cc_incr(sreq->cc_ptr, &c); + ssend_match = + MPIDI_OFI_init_recvtag(&ssend_mask, comm->context_id + context_offset, rank, tag, + MPIDI_OFI_ENABLE_DATA); + ssend_match |= MPIDI_OFI_SYNC_SEND_ACK; + MPIDI_OFI_CALL_RETRY(fi_trecv(MPIDI_OFI_EP_RX_TAG(0), /* endpoint */ + NULL, /* recvbuf */ + 0, /* data sz */ + NULL, /* memregion descr */ + MPIDI_OFI_comm_to_phys(comm, rank, MPIDI_OFI_API_TAG), /* remote proc */ + ssend_match, /* match bits */ + 0ULL, /* mask bits */ + (void *) &(ackreq->context)), trecvsync, MPIDI_OFI_CALL_LOCK); + } + + send_buf = (char *) buf + dt_true_lb; + + if (!dt_contig) { + size_t segment_first; + segment_first = 0; + last = data_sz; + MPIDI_OFI_REQUEST(sreq, noncontig) = + (MPIDI_OFI_noncontig_t *) MPL_malloc(data_sz + sizeof(MPID_Segment)); + MPIR_ERR_CHKANDJUMP1(MPIDI_OFI_REQUEST(sreq, noncontig) == NULL, mpi_errno, MPI_ERR_OTHER, + "**nomem", "**nomem %s", "Send Pack buffer alloc"); + MPID_Segment_init(buf, count, datatype, &MPIDI_OFI_REQUEST(sreq, noncontig->segment), 0); + MPID_Segment_pack(&MPIDI_OFI_REQUEST(sreq, noncontig->segment), segment_first, &last, + MPIDI_OFI_REQUEST(sreq, noncontig->pack_buffer)); + send_buf = MPIDI_OFI_REQUEST(sreq, noncontig->pack_buffer); + } + else + MPIDI_OFI_REQUEST(sreq, noncontig) = NULL; + + if (data_sz <= MPIDI_Global.max_buffered_send) { + mpi_errno = + MPIDI_OFI_send_handler(MPIDI_OFI_EP_TX_TAG(0), send_buf, data_sz, NULL, comm->rank, + MPIDI_OFI_comm_to_phys(comm, rank, MPIDI_OFI_API_TAG), + match_bits, NULL, MPIDI_OFI_DO_INJECT, MPIDI_OFI_ENABLE_DATA, + MPIDI_OFI_CALL_LOCK); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + MPIDI_OFI_send_event(NULL, sreq); + } + else if (data_sz <= MPIDI_Global.max_send) { + mpi_errno = + MPIDI_OFI_send_handler(MPIDI_OFI_EP_TX_TAG(0), send_buf, data_sz, NULL, comm->rank, + MPIDI_OFI_comm_to_phys(comm, rank, MPIDI_OFI_API_TAG), + match_bits, (void *) &(MPIDI_OFI_REQUEST(sreq, context)), + MPIDI_OFI_DO_SEND, MPIDI_OFI_ENABLE_DATA, MPIDI_OFI_CALL_LOCK); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + } + else if (unlikely(1)) { + MPIDI_OFI_send_control_t ctrl; + int c; + uint64_t rma_key = 0; + MPIDI_OFI_huge_counter_t *cntr; + void *ptr; + c = 1; + MPIDI_OFI_REQUEST(sreq, event_id) = MPIDI_OFI_EVENT_SEND_HUGE; + MPIR_cc_incr(sreq->cc_ptr, &c); + ptr = MPIDI_OFI_map_lookup(MPIDI_OFI_COMM(comm).huge_send_counters, rank); + + MPID_THREAD_CS_ENTER(POBJ, MPIDI_OFI_THREAD_FI_MUTEX); + + if (ptr == MPIDI_OFI_MAP_NOT_FOUND) { + ptr = MPL_malloc(sizeof(MPIDI_OFI_huge_counter_t)); + cntr = (MPIDI_OFI_huge_counter_t *) ptr; + cntr->outstanding = 0; + cntr->counter = 0; + MPIDI_OFI_map_set(MPIDI_OFI_COMM(comm).huge_send_counters, rank, ptr); + } + + cntr = (MPIDI_OFI_huge_counter_t *) ptr; + + ctrl.rma_key = MPIDI_OFI_index_allocator_alloc(MPIDI_OFI_COMM(comm).rma_id_allocator); + MPIR_Assert(ctrl.rma_key < MPIDI_Global.max_huge_rmas); + if (MPIDI_OFI_ENABLE_MR_SCALABLE) + rma_key = ctrl.rma_key << MPIDI_Global.huge_rma_shift; + MPIDI_OFI_CALL_NOLOCK(fi_mr_reg(MPIDI_Global.domain, /* In: Domain Object */ + send_buf, /* In: Lower memory address */ + data_sz, /* In: Length */ + FI_REMOTE_READ, /* In: Expose MR for read */ + 0ULL, /* In: offset(not used) */ + rma_key, /* In: requested key */ + 0ULL, /* In: flags */ + &cntr->mr, /* Out: memregion object */ + NULL), mr_reg); /* In: context */ + + if (!MPIDI_OFI_ENABLE_MR_SCALABLE) { + /* MR_BASIC */ + ctrl.rma_key = fi_mr_key(cntr->mr); + } + + cntr->outstanding++; + cntr->counter++; + MPIR_Assert(cntr->outstanding != USHRT_MAX); + MPIR_Assert(cntr->counter != USHRT_MAX); + MPIDI_OFI_REQUEST(sreq, util_comm) = comm; + MPIDI_OFI_REQUEST(sreq, util_id) = rank; + mpi_errno = MPIDI_OFI_send_handler(MPIDI_OFI_EP_TX_TAG(0), send_buf, + MPIDI_Global.max_send, + NULL, + comm->rank, + MPIDI_OFI_comm_to_phys(comm, rank, MPIDI_OFI_API_TAG), + match_bits, + (void *) &(MPIDI_OFI_REQUEST(sreq, context)), + MPIDI_OFI_DO_SEND, MPIDI_OFI_ENABLE_DATA, + MPIDI_OFI_CALL_NO_LOCK); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + ctrl.type = MPIDI_OFI_CTRL_HUGE; + ctrl.seqno = cntr->counter - 1; + MPIDI_OFI_MPI_CALL_POP(MPIDI_OFI_do_control_send + (&ctrl, send_buf, data_sz, rank, comm, sreq, FALSE)); + MPID_THREAD_CS_EXIT(POBJ, MPIDI_OFI_THREAD_FI_MUTEX); + } + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_SEND_NORMAL); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_send +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__ALWAYS_INLINE__ int MPIDI_OFI_send(MPIDI_OFI_SENDPARAMS, int noreq, uint64_t syncflag) +{ + int dt_contig, mpi_errno; + size_t data_sz; + MPI_Aint dt_true_lb; + MPIR_Datatype *dt_ptr; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_NM_SEND); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_NM_SEND); + + MPIDI_Datatype_get_info(count, datatype, dt_contig, data_sz, dt_ptr, dt_true_lb); + + if (likely(!syncflag && dt_contig && (data_sz <= MPIDI_Global.max_buffered_send))) + if (noreq) + mpi_errno = MPIDI_OFI_send_lightweight((char *) buf + dt_true_lb, data_sz, + rank, tag, comm, context_offset); + else + mpi_errno = MPIDI_OFI_send_lightweight_request((char *) buf + dt_true_lb, data_sz, + rank, tag, comm, context_offset, + request); + else + mpi_errno = MPIDI_OFI_send_normal(buf, count, datatype, rank, tag, comm, + context_offset, request, dt_contig, + data_sz, dt_ptr, dt_true_lb, syncflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_NM_SEND); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_persistent_send +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__ALWAYS_INLINE__ int MPIDI_OFI_persistent_send(MPIDI_OFI_SENDPARAMS) +{ + MPIR_Request *sreq; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_NM_PSEND); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_NM_PSEND); + + MPIDI_OFI_REQUEST_CREATE(sreq, MPIR_REQUEST_KIND__PREQUEST_SEND); + *request = sreq; + + MPIR_Comm_add_ref(comm); + sreq->comm = comm; + MPIDI_OFI_REQUEST(sreq, util.persist.buf) = (void *) buf; + MPIDI_OFI_REQUEST(sreq, util.persist.count) = count; + MPIDI_OFI_REQUEST(sreq, datatype) = datatype; + MPIDI_OFI_REQUEST(sreq, util.persist.rank) = rank; + MPIDI_OFI_REQUEST(sreq, util.persist.tag) = tag; + MPIDI_OFI_REQUEST(sreq, util_comm) = comm; + MPIDI_OFI_REQUEST(sreq, util_id) = comm->context_id + context_offset; + sreq->u.persist.real_request = NULL; + MPIDI_CH4U_request_complete(sreq); + + if (HANDLE_GET_KIND(datatype) != HANDLE_KIND_BUILTIN) { + MPIR_Datatype *dt_ptr; + MPID_Datatype_get_ptr(datatype, dt_ptr); + MPID_Datatype_add_ref(dt_ptr); + } + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_NM_PSEND); + return MPI_SUCCESS; +} + +#define STARTALL_CASE(CASELABEL,FUNC,CONTEXTID) \ + case CASELABEL: \ + { \ + rc = FUNC(MPIDI_OFI_REQUEST(preq,util.persist.buf), \ + MPIDI_OFI_REQUEST(preq,util.persist.count), \ + MPIDI_OFI_REQUEST(preq,datatype), \ + MPIDI_OFI_REQUEST(preq,util.persist.rank), \ + MPIDI_OFI_REQUEST(preq,util.persist.tag), \ + preq->comm, \ + MPIDI_OFI_REQUEST(preq,util_id) - \ + CONTEXTID, \ + &preq->u.persist.real_request); \ + break; \ + } + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_send +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__ALWAYS_INLINE__ int MPIDI_NM_send(MPIDI_OFI_SENDPARAMS) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_SEND); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_SEND); + mpi_errno = MPIDI_OFI_send(MPIDI_OFI_SENDARGS, 1, 0ULL); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_SEND); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_rsend +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__ALWAYS_INLINE__ int MPIDI_NM_rsend(MPIDI_OFI_SENDPARAMS) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_RSEND); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_RSEND); + mpi_errno = MPIDI_OFI_send(MPIDI_OFI_SENDARGS, 1, 0ULL); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_RSEND); + return mpi_errno; +} + + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_irsend +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__ALWAYS_INLINE__ int MPIDI_NM_irsend(MPIDI_OFI_SENDPARAMS) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_IRSEND); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_IRSEND); + mpi_errno = MPIDI_OFI_send(MPIDI_OFI_SENDARGS, 0, 0ULL); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_IRSEND); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ssend +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__ALWAYS_INLINE__ int MPIDI_NM_ssend(MPIDI_OFI_SENDPARAMS) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_SSEND); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_SSEND); + mpi_errno = MPIDI_OFI_send(MPIDI_OFI_SENDARGS, 0, MPIDI_OFI_SYNC_SEND); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_SSEND); + return mpi_errno; +} + + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_isend +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__ALWAYS_INLINE__ int MPIDI_NM_isend(MPIDI_OFI_SENDPARAMS) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_ISEND); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_ISEND); + mpi_errno = MPIDI_OFI_send(MPIDI_OFI_SENDARGS, 0, 0ULL); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_ISEND); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_issend +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__ALWAYS_INLINE__ int MPIDI_NM_issend(MPIDI_OFI_SENDPARAMS) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_ISSEND); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_ISSEND); + mpi_errno = MPIDI_OFI_send(MPIDI_OFI_SENDARGS, 0, MPIDI_OFI_SYNC_SEND); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_ISSEND); + return mpi_errno; +} + + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_startall +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__ALWAYS_INLINE__ int MPIDI_NM_startall(int count, MPIR_Request * requests[]) +{ + int rc = MPI_SUCCESS, i; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_STARTALL); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_STARTALL); + + for (i = 0; i < count; i++) { + MPIR_Request *const preq = requests[i]; + + switch (MPIDI_OFI_REQUEST(preq, util.persist.type)) { +#ifdef MPIDI_BUILD_CH4_SHM + STARTALL_CASE(MPIDI_PTYPE_RECV, MPIDI_NM_irecv, preq->comm->recvcontext_id); +#else + STARTALL_CASE(MPIDI_PTYPE_RECV, MPIDI_Irecv, preq->comm->recvcontext_id); +#endif + +#ifdef MPIDI_BUILD_CH4_SHM + STARTALL_CASE(MPIDI_PTYPE_SEND, MPIDI_NM_isend, preq->comm->context_id); +#else + STARTALL_CASE(MPIDI_PTYPE_SEND, MPIDI_Isend, preq->comm->context_id); +#endif + STARTALL_CASE(MPIDI_PTYPE_SSEND, MPIDI_Issend, preq->comm->context_id); + + case MPIDI_PTYPE_BSEND:{ + MPI_Request sreq_handle; + rc = MPIR_Ibsend_impl(MPIDI_OFI_REQUEST(preq, util.persist.buf), + MPIDI_OFI_REQUEST(preq, util.persist.count), + MPIDI_OFI_REQUEST(preq, datatype), + MPIDI_OFI_REQUEST(preq, util.persist.rank), + MPIDI_OFI_REQUEST(preq, util.persist.tag), + preq->comm, &sreq_handle); + + if (rc == MPI_SUCCESS) + MPIR_Request_get_ptr(sreq_handle, preq->u.persist.real_request); + + break; + } + + default: + rc = MPIR_Err_create_code(MPI_SUCCESS, MPIR_ERR_FATAL, __FUNCTION__, + __LINE__, MPI_ERR_INTERN, "**ch3|badreqtype", + "**ch3|badreqtype %d", MPIDI_OFI_REQUEST(preq, + util.persist.type)); + } + + if (rc == MPI_SUCCESS) { + preq->status.MPI_ERROR = MPI_SUCCESS; + + if (MPIDI_OFI_REQUEST(preq, util.persist.type) == MPIDI_PTYPE_BSEND) { + preq->cc_ptr = &preq->cc; + MPIR_cc_set(&preq->cc, 0); + } + else + preq->cc_ptr = &preq->u.persist.real_request->cc; + } + else { + preq->u.persist.real_request = NULL; + preq->status.MPI_ERROR = rc; + preq->cc_ptr = &preq->cc; + MPIR_cc_set(&preq->cc, 0); + } + } + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_STARTALL); + return rc; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_send_init +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__ALWAYS_INLINE__ int MPIDI_NM_send_init(MPIDI_OFI_SENDPARAMS) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_SEND_INIT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_SEND_INIT); + mpi_errno = MPIDI_OFI_persistent_send(MPIDI_OFI_SENDARGS); + MPIDI_OFI_REQUEST((*request), util.persist.type) = MPIDI_PTYPE_SEND; + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_SEND_INIT); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ssend_init +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__ALWAYS_INLINE__ int MPIDI_NM_ssend_init(MPIDI_OFI_SENDPARAMS) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_SSEND_INIT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_SSEND_INIT); + mpi_errno = MPIDI_OFI_persistent_send(MPIDI_OFI_SENDARGS); + MPIDI_OFI_REQUEST((*request), util.persist.type) = MPIDI_PTYPE_SSEND; + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_SSEND_INIT); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_bsend_init +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__ALWAYS_INLINE__ int MPIDI_NM_bsend_init(MPIDI_OFI_SENDPARAMS) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_BSEND_INIT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_BSEND_INIT); + mpi_errno = MPIDI_OFI_persistent_send(MPIDI_OFI_SENDARGS); + MPIDI_OFI_REQUEST((*request), util.persist.type) = MPIDI_PTYPE_BSEND; + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_BSEND_INIT); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_rsend_init +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__ALWAYS_INLINE__ int MPIDI_NM_rsend_init(MPIDI_OFI_SENDPARAMS) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_RSEND_INIT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_RSEND_INIT); + mpi_errno = MPIDI_OFI_persistent_send(MPIDI_OFI_SENDARGS); + MPIDI_OFI_REQUEST((*request), util.persist.type) = MPIDI_PTYPE_SEND; + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_RSEND_INIT); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_cancel_send +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__ALWAYS_INLINE__ int MPIDI_NM_cancel_send(MPIR_Request * sreq) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_CANCEL_SEND); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_CANCEL_SEND); + /* Sends cannot be cancelled */ + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_CANCEL_SEND); + return mpi_errno; +} + +#endif /* NETMOD_OFI_SEND_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/ofi/ofi_spawn.h b/src/mpid/ch4/netmod/ofi/ofi_spawn.h new file mode 100644 index 0000000..9b109d8 --- /dev/null +++ b/src/mpid/ch4/netmod/ofi/ofi_spawn.h @@ -0,0 +1,561 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_OFI_DYNPROC_H_INCLUDED +#define NETMOD_OFI_DYNPROC_H_INCLUDED + +#include "ofi_impl.h" + +#define MPIDI_OFI_PORT_NAME_TAG_KEY "tag" +#define MPIDI_OFI_CONNENTRY_TAG_KEY "connentry" + +// FIXME: +#ifdef MPIDI_OFI_CONFIG_USE_AV_TABLE +#define MPIDI_OFI_TABLE_INDEX_INCR() +#else +#define MPIDI_OFI_TABLE_INDEX_INCR() +#endif + +static inline void MPIDI_OFI_free_port_name_tag(int tag) +{ + int index, rem_tag; + + index = tag / (sizeof(int) * 8); + rem_tag = tag - (index * sizeof(int) * 8); + + MPIDI_Global.port_name_tag_mask[index] &= ~(1 << ((8 * sizeof(int)) - 1 - rem_tag)); +} + + +static inline int MPIDI_OFI_get_port_name_tag(int *port_name_tag) +{ + unsigned i, j; + int mpi_errno = MPI_SUCCESS; + + for (i = 0; i < MPIR_MAX_CONTEXT_MASK; i++) + if (MPIDI_Global.port_name_tag_mask[i] != ~0) + break; + + if (i < MPIR_MAX_CONTEXT_MASK) + for (j = 0; j < (8 * sizeof(int)); j++) { + if ((MPIDI_Global.port_name_tag_mask[i] | (1 << ((8 * sizeof(int)) - j - 1))) != + MPIDI_Global.port_name_tag_mask[i]) { + MPIDI_Global.port_name_tag_mask[i] |= (1 << ((8 * sizeof(int)) - j - 1)); + *port_name_tag = ((i * 8 * sizeof(int)) + j); + goto fn_exit; + } + } + else + goto fn_fail; + + fn_exit: + return mpi_errno; + + fn_fail: + *port_name_tag = -1; + mpi_errno = MPI_ERR_OTHER; + goto fn_exit; +} + +static inline int MPIDI_OFI_get_tag_from_port(const char *port_name, int *port_name_tag) +{ + int mpi_errno = MPI_SUCCESS; + int str_errno = MPL_STR_SUCCESS; + + if (strlen(port_name) == 0) + goto fn_exit; + + str_errno = MPL_str_get_int_arg(port_name, MPIDI_OFI_PORT_NAME_TAG_KEY, port_name_tag); + MPIR_ERR_CHKANDJUMP(str_errno, mpi_errno, MPI_ERR_OTHER, "**argstr_no_port_name_tag"); + fn_exit: + return mpi_errno; + fn_fail: + goto fn_exit; +} + + +static inline int MPIDI_OFI_get_conn_name_from_port(const char *port_name, char *connname) +{ + int mpi_errno = MPI_SUCCESS; + int maxlen = MPIDI_KVSAPPSTRLEN; + MPL_str_get_binary_arg(port_name, + MPIDI_OFI_CONNENTRY_TAG_KEY, + connname, MPIDI_Global.addrnamelen, &maxlen); + return mpi_errno; +} + +static inline int MPIDI_OFI_dynproc_create_intercomm(const char *port_name, + char *addr_table, + MPID_Node_id_t * node_table, + int entries, + MPIR_Comm * comm_ptr, + MPIR_Comm ** newcomm, + int is_low_group, char *api) +{ + int i, context_id_offset, mpi_errno = MPI_SUCCESS; + MPIR_Comm *tmp_comm_ptr = NULL; + + MPIDI_OFI_MPI_CALL_POP(MPIDI_OFI_get_tag_from_port(port_name, &context_id_offset)); + MPIDI_OFI_MPI_CALL_POP(MPIR_Comm_create(&tmp_comm_ptr)); + + tmp_comm_ptr->context_id = MPIR_CONTEXT_SET_FIELD(DYNAMIC_PROC, context_id_offset, 1); + tmp_comm_ptr->recvcontext_id = tmp_comm_ptr->context_id; + tmp_comm_ptr->remote_size = entries; + tmp_comm_ptr->local_size = comm_ptr->local_size; + tmp_comm_ptr->rank = comm_ptr->rank; + tmp_comm_ptr->comm_kind = MPIR_COMM_KIND__INTERCOMM; + tmp_comm_ptr->local_comm = comm_ptr; + tmp_comm_ptr->is_low_group = is_low_group; + MPIDII_COMM(tmp_comm_ptr, local_map).mode = MPIDII_COMM(comm_ptr, map).mode; + MPIDII_COMM(tmp_comm_ptr, local_map).size = MPIDII_COMM(comm_ptr, map).size; + MPIDII_COMM(tmp_comm_ptr, local_map).avtid = MPIDII_COMM(comm_ptr, map).avtid; + switch (MPIDII_COMM(comm_ptr, map).mode) { + case MPIDII_RANK_MAP_DIRECT: + case MPIDII_RANK_MAP_DIRECT_INTRA: + break; + case MPIDII_RANK_MAP_OFFSET: + case MPIDII_RANK_MAP_OFFSET_INTRA: + MPIDII_COMM(tmp_comm_ptr, local_map).reg.offset = MPIDII_COMM(comm_ptr, map).reg.offset; + break; + case MPIDII_RANK_MAP_STRIDE: + case MPIDII_RANK_MAP_STRIDE_INTRA: + case MPIDII_RANK_MAP_STRIDE_BLOCK: + case MPIDII_RANK_MAP_STRIDE_BLOCK_INTRA: + MPIDII_COMM(tmp_comm_ptr, local_map).reg.stride.stride = + MPIDII_COMM(comm_ptr, map).reg.stride.stride; + MPIDII_COMM(tmp_comm_ptr, local_map).reg.stride.blocksize = + MPIDII_COMM(comm_ptr, map).reg.stride.blocksize; + MPIDII_COMM(tmp_comm_ptr, local_map).reg.stride.offset = + MPIDII_COMM(comm_ptr, map).reg.stride.offset; + break; + case MPIDII_RANK_MAP_LUT: + case MPIDII_RANK_MAP_LUT_INTRA: + MPIDII_COMM(tmp_comm_ptr, local_map).irreg.lut.t = MPIDII_COMM(comm_ptr, map).irreg.lut.t; + MPIDII_COMM(tmp_comm_ptr, local_map).irreg.lut.lpid = + MPIDII_COMM(comm_ptr, map).irreg.lut.lpid; + MPIR_Object_add_ref(MPIDII_COMM(comm_ptr, map).irreg.lut.t); + break; + case MPIDII_RANK_MAP_MLUT: + MPIDII_COMM(tmp_comm_ptr, local_map).irreg.mlut.t = MPIDII_COMM(comm_ptr, map).irreg.mlut.t; + MPIDII_COMM(tmp_comm_ptr, local_map).irreg.mlut.gpid = + MPIDII_COMM(comm_ptr, map).irreg.mlut.gpid; + MPIR_Object_add_ref(MPIDII_COMM(comm_ptr, map).irreg.mlut.t); + break; + case MPIDII_RANK_MAP_NONE: + MPIR_Assert(0); + break; + } + + int avtid; + avtid = 0; + MPIDIU_new_avt(entries, &avtid); + +#ifdef MPIDI_OFI_CONFIG_USE_AV_TABLE + MPIDI_OFI_CALL(fi_av_insert(MPIDI_Global.av, addr_table, entries, NULL, 0ULL, NULL), avmap); +#else + fi_addr_t *mapped_table; + mapped_table = (fi_addr_t *) MPL_malloc(entries * sizeof(fi_addr_t)); + + MPIDI_OFI_CALL(fi_av_insert(MPIDI_Global.av, addr_table, entries, + mapped_table, 0ULL, NULL), avmap); + for (i = 0; i < entries; i++) { + MPIDI_OFI_AV(&MPIDIU_get_av(avtid, i)).dest = mapped_table[i]; + } + MPL_free(mapped_table); +#endif + + MPIDIU_update_node_map(avtid, entries, node_table); + + /* set mapping for remote group */ + MPIDII_COMM(tmp_comm_ptr, map).mode = MPIDII_RANK_MAP_DIRECT; + MPIDII_COMM(tmp_comm_ptr, map).size = entries; + MPIDII_COMM(tmp_comm_ptr, map).avtid = avtid; + + MPIR_Comm_commit(tmp_comm_ptr); + + MPIDI_OFI_MPI_CALL_POP(MPIR_Comm_dup_impl(tmp_comm_ptr, newcomm)); + + tmp_comm_ptr->local_comm = NULL; /* avoid freeing local comm with comm_release */ + MPIR_Comm_release(tmp_comm_ptr); + + MPL_free(addr_table); + MPL_free(node_table); + + fn_exit: + return mpi_errno; + fn_fail: + goto fn_exit; +} + +static inline int MPIDI_OFI_dynproc_bcast(int root, + MPIR_Comm * comm_ptr, + int *out_root, + ssize_t * out_table_size, + char **out_addr_table, MPID_Node_id_t ** out_node_table) +{ + int entries, mpi_errno = MPI_SUCCESS; + MPIR_Errflag_t errflag = MPIR_ERR_NONE; + + MPIDI_OFI_MPI_CALL_POP(MPIR_Bcast_intra(out_root, 1, MPI_INT, root, comm_ptr, &errflag)); + MPIDI_OFI_MPI_CALL_POP(MPIR_Bcast_intra(out_table_size, 1, MPI_LONG_LONG_INT, + root, comm_ptr, &errflag)); + + if (*out_addr_table == NULL) + *out_addr_table = (char *) MPL_malloc(*out_table_size); + + MPIDI_OFI_MPI_CALL_POP(MPIR_Bcast_intra(*out_addr_table, *out_table_size, + MPI_CHAR, root, comm_ptr, &errflag)); + + entries = *out_table_size / MPIDI_Global.addrnamelen; + + if (*out_node_table == NULL) + *out_node_table = (MPID_Node_id_t *) MPL_malloc(MPIDI_Global.addrnamelen * entries); + + MPIDI_OFI_MPI_CALL_POP(MPIR_Bcast_intra(*out_node_table, entries * sizeof(MPID_Node_id_t), + MPI_CHAR, root, comm_ptr, &errflag)); + + fn_exit: + return mpi_errno; + fn_fail: + goto fn_exit; +} + + +static inline int MPIDI_OFI_dynproc_exchange_map(int root, + int phase, + int port_id, + fi_addr_t * conn, + char *conname, + MPIR_Comm * comm_ptr, + ssize_t * out_table_size, + int *out_root, + char **out_addr_table, + MPID_Node_id_t ** out_node_table) +{ + int i, mpi_errno = MPI_SUCCESS; + + MPIDI_OFI_dynamic_process_request_t req[2]; + uint64_t match_bits = 0; + uint64_t mask_bits = 0; + struct fi_msg_tagged msg; + req[0].done = MPIDI_OFI_PEEK_START; + req[0].event_id = MPIDI_OFI_EVENT_ACCEPT_PROBE; + req[1].done = MPIDI_OFI_PEEK_START; + req[1].event_id = MPIDI_OFI_EVENT_ACCEPT_PROBE; + match_bits = MPIDI_OFI_init_recvtag(&mask_bits, port_id, + MPI_ANY_SOURCE, MPI_ANY_TAG, MPIDI_OFI_ENABLE_DATA); + match_bits |= MPIDI_OFI_DYNPROC_SEND; + + if (phase == 0) { + /* Receive the addresses */ + /* We don't know the size, so probe for table size */ + /* Receive phase updates the connection */ + /* With the probed address */ + msg.msg_iov = NULL; + msg.desc = NULL; + msg.iov_count = 0; + msg.addr = FI_ADDR_UNSPEC; + msg.tag = match_bits; + msg.ignore = mask_bits; + msg.context = (void *) &req[0].context; + msg.data = 0; + + while (req[0].done != MPIDI_OFI_PEEK_FOUND) { + req[0].done = MPIDI_OFI_PEEK_START; + MPIDI_OFI_CALL(fi_trecvmsg + (MPIDI_OFI_EP_RX_TAG(0), &msg, + FI_PEEK | FI_COMPLETION | MPIDI_OFI_ENABLE_DATA), trecv); + MPIDI_OFI_PROGRESS_WHILE(req[0].done == MPIDI_OFI_PEEK_START); + } + + *out_table_size = req[0].msglen; + *out_root = req[0].tag; + *out_addr_table = (char *) MPL_malloc(*out_table_size); + + int entries = req[0].msglen / MPIDI_Global.addrnamelen; + *out_node_table = (MPID_Node_id_t *) MPL_malloc(entries * sizeof(MPID_Node_id_t)); + + req[0].done = 0; + req[0].event_id = MPIDI_OFI_EVENT_DYNPROC_DONE; + req[1].done = 0; + req[1].event_id = MPIDI_OFI_EVENT_DYNPROC_DONE; + + MPIDI_OFI_CALL_RETRY(fi_trecv(MPIDI_OFI_EP_RX_TAG(0), + *out_addr_table, + *out_table_size, + NULL, + FI_ADDR_UNSPEC, + match_bits, + mask_bits, &req[0].context), trecv, MPIDI_OFI_CALL_LOCK); + MPIDI_OFI_CALL_RETRY(fi_trecv(MPIDI_OFI_EP_RX_TAG(0), + *out_node_table, + entries * sizeof(MPID_Node_id_t), + NULL, + FI_ADDR_UNSPEC, + match_bits, + mask_bits, &req[1].context), trecv, MPIDI_OFI_CALL_LOCK); + + MPIDI_OFI_PROGRESS_WHILE(!req[0].done || !req[1].done); + memcpy(conname, *out_addr_table + req[0].source * MPIDI_Global.addrnamelen, + MPIDI_Global.addrnamelen); + } + + if (phase == 1) { + /* Send our table to the child */ + /* Send phase maps the entry */ + char *my_addr_table; + int tag = root; + int tblsz = MPIDI_Global.addrnamelen * comm_ptr->local_size; + my_addr_table = (char *) MPL_malloc(tblsz); + + MPID_Node_id_t *my_node_table; + MPID_Node_id_t nodetblsz = sizeof(*my_node_table) * comm_ptr->local_size; + my_node_table = (MPID_Node_id_t *) MPL_malloc(nodetblsz); + + match_bits = MPIDI_OFI_init_sendtag(port_id, + comm_ptr->rank, + tag, MPIDI_OFI_DYNPROC_SEND, MPIDI_OFI_ENABLE_DATA); + + for (i = 0; i < comm_ptr->local_size; i++) { + size_t sz = MPIDI_Global.addrnamelen; + MPIDI_OFI_CALL(fi_av_lookup(MPIDI_Global.av, + MPIDI_OFI_COMM_TO_PHYS(comm_ptr, i), + my_addr_table + i * MPIDI_Global.addrnamelen, + &sz), avlookup); + MPIR_Assert(sz == MPIDI_Global.addrnamelen); + } + + for (i = 0; i < comm_ptr->local_size; i++) + MPIDI_CH4U_get_node_id(comm_ptr, i, &my_node_table[i]); + + /* fi_av_map here is not quite right for some providers */ + /* we need to get this connection from the sockname */ + req[0].done = 0; + req[0].event_id = MPIDI_OFI_EVENT_DYNPROC_DONE; + req[1].done = 0; + req[1].event_id = MPIDI_OFI_EVENT_DYNPROC_DONE; + mpi_errno = MPIDI_OFI_send_handler(MPIDI_OFI_EP_TX_TAG(0), + my_addr_table, + tblsz, + NULL, + comm_ptr->rank, + *conn, + match_bits, + (void *) &req[0].context, + MPIDI_OFI_DO_SEND, MPIDI_OFI_ENABLE_DATA, + MPIDI_OFI_CALL_LOCK); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + MPIDI_OFI_send_handler(MPIDI_OFI_EP_TX_TAG(0), + my_node_table, + nodetblsz, + NULL, + comm_ptr->rank, + *conn, + match_bits, + (void *) &req[1].context, + MPIDI_OFI_DO_SEND, MPIDI_OFI_ENABLE_DATA, MPIDI_OFI_CALL_LOCK); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + MPIDI_OFI_PROGRESS_WHILE(!req[0].done || !req[1].done); + + MPL_free(my_addr_table); + MPL_free(my_node_table); + } + + fn_exit: + return mpi_errno; + fn_fail: + goto fn_exit; +} + + + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_comm_connect +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_comm_connect(const char *port_name, + MPIR_Info * info, + int root, MPIR_Comm * comm_ptr, MPIR_Comm ** newcomm) +{ + int entries, mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_COMM_CONNECT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_COMM_CONNECT); + + char *parent_addr_table = NULL; + MPID_Node_id_t *parent_node_table = NULL; + ssize_t parent_table_sz = -1LL; + int parent_root = -1; + int rank = comm_ptr->rank; + int port_id; + + MPID_THREAD_CS_ENTER(POBJ, MPIDI_OFI_THREAD_SPAWN_MUTEX); + MPIDI_OFI_MPI_CALL_POP(MPIDI_OFI_get_tag_from_port(port_name, &port_id)); + + if (rank == root) { + fi_addr_t conn; + char conname[FI_NAME_MAX]; + MPIDI_OFI_MPI_CALL_POP(MPIDI_OFI_get_conn_name_from_port(port_name, conname)); + MPIDI_OFI_CALL(fi_av_insert(MPIDI_Global.av, conname, 1, &conn, 0ULL, NULL), avmap); + MPIDI_OFI_MPI_CALL_POP(MPIDI_OFI_dynproc_exchange_map + (root, 1, port_id, &conn, conname, comm_ptr, &parent_table_sz, + &parent_root, &parent_addr_table, &parent_node_table)); + MPIDI_OFI_MPI_CALL_POP(MPIDI_OFI_dynproc_exchange_map + (root, 0, port_id, &conn, conname, comm_ptr, &parent_table_sz, + &parent_root, &parent_addr_table, &parent_node_table)); + MPIDI_OFI_CALL(fi_av_remove(MPIDI_Global.av, &conn, 1, 0ULL), avmap); + } + + /* Map the new address table */ + MPIDI_OFI_MPI_CALL_POP(MPIDI_OFI_dynproc_bcast(root, comm_ptr, &parent_root, + &parent_table_sz, + &parent_addr_table, &parent_node_table)); + + /* Now Create the New Intercomm */ + entries = parent_table_sz / MPIDI_Global.addrnamelen; + MPIDI_OFI_MPI_CALL_POP(MPIDI_OFI_dynproc_create_intercomm(port_name, + parent_addr_table, + parent_node_table, + entries, + comm_ptr, + newcomm, 0, (char *) "Connect")); + fn_exit: + MPID_THREAD_CS_EXIT(POBJ, MPIDI_OFI_THREAD_SPAWN_MUTEX); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_COMM_CONNECT); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_comm_disconnect +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_comm_disconnect(MPIR_Comm * comm_ptr) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Errflag_t errflag = MPIR_ERR_NONE; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_COMM_DISCONNECT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_COMM_DISCONNECT); + + MPIDI_OFI_MPI_CALL_POP(MPIR_Barrier_impl(comm_ptr, &errflag)); + MPIDI_OFI_MPI_CALL_POP(MPIR_Comm_free_impl(comm_ptr)); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_COMM_DISCONNECT); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_comm_open_port +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_open_port(MPIR_Info * info_ptr, char *port_name) +{ + int mpi_errno = MPI_SUCCESS; + int str_errno = MPL_STR_SUCCESS; + int port_name_tag = 0; + int len = MPI_MAX_PORT_NAME; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_COMM_OPEN_PORT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_COMM_OPEN_PORT); + + MPIDI_OFI_MPI_CALL_POP(MPIDI_OFI_get_port_name_tag(&port_name_tag)); + MPIDI_OFI_STR_CALL(MPL_str_add_int_arg(&port_name, &len, MPIDI_OFI_PORT_NAME_TAG_KEY, + port_name_tag), port_str); + MPIDI_OFI_STR_CALL(MPL_str_add_binary_arg(&port_name, &len, MPIDI_OFI_CONNENTRY_TAG_KEY, + MPIDI_Global.addrname, + MPIDI_Global.addrnamelen), port_str); + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_COMM_OPEN_PORT); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_comm_close_port +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_close_port(const char *port_name) +{ + int mpi_errno = MPI_SUCCESS; + int port_name_tag; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_COMM_CLOSE_PORT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_COMM_CLOSE_PORT); + + mpi_errno = MPIDI_OFI_get_tag_from_port(port_name, &port_name_tag); + MPIDI_OFI_free_port_name_tag(port_name_tag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_COMM_CLOSE_PORT); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_comm_close_port +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_comm_accept(const char *port_name, + MPIR_Info * info, + int root, MPIR_Comm * comm_ptr, MPIR_Comm ** newcomm) +{ + int entries, mpi_errno = MPI_SUCCESS; + char *child_addr_table = NULL; + MPID_Node_id_t *child_node_table = NULL; + ssize_t child_table_sz = -1LL; + int child_root = -1; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_COMM_CLOSE_PORT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_COMM_CLOSE_PORT); + + MPID_THREAD_CS_ENTER(POBJ, MPIDI_OFI_THREAD_SPAWN_MUTEX); + int rank = comm_ptr->rank; + + if (rank == root) { + fi_addr_t conn; + char conname[FI_NAME_MAX]; + int port_id; + MPIDI_OFI_MPI_CALL_POP(MPIDI_OFI_get_tag_from_port(port_name, &port_id)); + MPIDI_OFI_MPI_CALL_POP(MPIDI_OFI_dynproc_exchange_map + (root, 0, port_id, &conn, conname, comm_ptr, &child_table_sz, + &child_root, &child_addr_table, &child_node_table)); + MPIDI_OFI_CALL(fi_av_insert(MPIDI_Global.av, conname, 1, &conn, 0ULL, NULL), avmap); + MPIDI_OFI_MPI_CALL_POP(MPIDI_OFI_dynproc_exchange_map + (root, 1, port_id, &conn, conname, comm_ptr, &child_table_sz, + &child_root, &child_addr_table, &child_node_table)); + MPIDI_OFI_CALL(fi_av_remove(MPIDI_Global.av, &conn, 1, 0ULL), avmap); + } + + /* Map the new address table */ + MPIDI_OFI_MPI_CALL_POP(MPIDI_OFI_dynproc_bcast(root, comm_ptr, &child_root, + &child_table_sz, + &child_addr_table, &child_node_table)); + /* Now Create the New Intercomm */ + entries = child_table_sz / MPIDI_Global.addrnamelen; + MPIDI_OFI_MPI_CALL_POP(MPIDI_OFI_dynproc_create_intercomm(port_name, + child_addr_table, + child_node_table, + entries, + comm_ptr, + newcomm, 1, (char *) "Accept")); + fn_exit: + MPID_THREAD_CS_EXIT(POBJ, MPIDI_OFI_THREAD_SPAWN_MUTEX); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_COMM_CLOSE_PORT); + return mpi_errno; + + fn_fail: + goto fn_exit; +} + +#endif /* NETMOD_OFI_WIN_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/ofi/ofi_types.h b/src/mpid/ch4/netmod/ofi/ofi_types.h new file mode 100644 index 0000000..a501d7e --- /dev/null +++ b/src/mpid/ch4/netmod/ofi/ofi_types.h @@ -0,0 +1,545 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_OFI_TYPES_H_INCLUDED +#define NETMOD_OFI_TYPES_H_INCLUDED + +#include <netdb.h> +#include <stddef.h> +#include <inttypes.h> +#ifdef HAVE_STDINT_H +#include <stdint.h> +#endif +#include "ofi_pre.h" +#include "ch4_types.h" +#include "mpidch4r.h" +#include "fi_list.h" + +#define __SHORT_FILE__ \ + (strrchr(__FILE__,'/') \ + ? strrchr(__FILE__,'/')+1 \ + : __FILE__ \ +) +#define MPIDI_OFI_MAP_NOT_FOUND ((void*)(-1UL)) +#define MPIDI_OFI_MAJOR_VERSION 1 +#define MPIDI_OFI_MINOR_VERSION 0 +#define MPIDI_OFI_DEFAULT_SHORT_SEND_SIZE (16 * 1024) +#define MPIDI_OFI_NUM_AM_BUFFERS (8) +#define MPIDI_OFI_AM_BUFF_SZ (1 * 1024 * 1024) +#define MPIDI_OFI_CACHELINE_SIZE (64) +#define MPIDI_OFI_IOV_MAX (32) +#define MPIDI_OFI_BUF_POOL_SIZE (1024) +#define MPIDI_OFI_BUF_POOL_NUM (1024) +#define MPIDI_OFI_NUM_CQ_BUFFERED (1024) +#define MPIDI_OFI_MAX_AM_HANDLERS_TOTAL (24) +#define MPIDI_OFI_INTERNAL_HANDLER_CONTROL (MPIDI_OFI_MAX_AM_HANDLERS_TOTAL-1) +#define MPIDI_OFI_INTERNAL_HANDLER_NEXT (MPIDI_OFI_MAX_AM_HANDLERS_TOTAL-2) +#define MPIDI_OFI_MAX_AM_HANDLERS (MPIDI_OFI_INTERNAL_HANDLER_NEXT-1) + +#ifdef USE_OFI_TAGGED +#define MPIDI_OFI_ENABLE_TAGGED 1 +#define MPIDI_OFI_ENABLE_AM 1 +#define MPIDI_OFI_ENABLE_RMA 1 +#else +#define MPIDI_OFI_ENABLE_TAGGED 0 +#define MPIDI_OFI_ENABLE_AM 1 +#define MPIDI_OFI_ENABLE_RMA 1 +#endif + +#ifdef MPIDI_OFI_CONFIG_USE_SCALABLE_ENDPOINTS +#define MPIDI_OFI_ENABLE_SCALABLE_ENDPOINTS 1 +#else +#define MPIDI_OFI_ENABLE_SCALABLE_ENDPOINTS 0 +#endif + +#ifdef MPIDI_OFI_CONFIG_USE_AV_TABLE +#define MPIDI_OFI_ENABLE_AV_TABLE 1 +#else +#define MPIDI_OFI_ENABLE_AV_TABLE 0 +#endif + +#ifdef USE_OFI_IMMEDIATE_DATA +#define MPIDI_OFI_ENABLE_DATA FI_REMOTE_CQ_DATA +#else +#define MPIDI_OFI_ENABLE_DATA 0 +#endif + +#ifdef USE_OFI_STX_RMA +#define MPIDI_OFI_ENABLE_STX_RMA 1 +#else +#define MPIDI_OFI_ENABLE_STX_RMA 0 +#endif + +#ifdef USE_OFI_MR_SCALABLE +#define MPIDI_OFI_ENABLE_MR_SCALABLE 1 +#else +#define MPIDI_OFI_ENABLE_MR_SCALABLE 0 +#endif + +#ifdef USE_OFI_IMMEDIATE_DATA +/* match/ignore bit manipulation + * + * 0123 4567 01234567 01234567 01234567 01234567 01234567 01234567 01234567 + * | | | + * ^ | Unused | context id | message tag + * | | | | + * +---- protocol + */ +#define MPIDI_OFI_PROTOCOL_MASK (0x9000000000000000ULL) +#define MPIDI_OFI_CONTEXT_MASK (0x00007FFF80000000ULL) +#define MPIDI_OFI_SOURCE_MASK (0x0000000000000000ULL) +#define MPIDI_OFI_TAG_MASK (0x000000007FFFFFFFULL) +#define MPIDI_OFI_SYNC_SEND (0x1000000000000000ULL) +#define MPIDI_OFI_SYNC_SEND_ACK (0x2000000000000000ULL) +#define MPIDI_OFI_DYNPROC_SEND (0x4000000000000000ULL) +#define MPIDI_OFI_TAG_SHIFT (31) +#define MPIDI_OFI_SOURCE_SHIFT (0) +#else +/* match/ignore bit manipulation + * + * 0123 4567 01234567 0123 4567 01234567 0123 4567 01234567 01234567 01234567 + * | | | + * ^ | context id | source | message tag + * | | | | + * +---- protocol + */ +#define MPIDI_OFI_PROTOCOL_MASK (0x9000000000000000ULL) +#define MPIDI_OFI_CONTEXT_MASK (0x0FFFF00000000000ULL) +#define MPIDI_OFI_SOURCE_MASK (0x00000FFFF0000000ULL) +#define MPIDI_OFI_TAG_MASK (0x000000000FFFFFFFULL) +#define MPIDI_OFI_SYNC_SEND (0x1000000000000000ULL) +#define MPIDI_OFI_SYNC_SEND_ACK (0x2000000000000000ULL) +#define MPIDI_OFI_DYNPROC_SEND (0x4000000000000000ULL) +#define MPIDI_OFI_TAG_SHIFT (28) +#define MPIDI_OFI_SOURCE_SHIFT (16) +#endif + +/* RMA Key Space division + * | | | | + * ... Context ID | Huge RMA | Window Instance | + * | | | | + */ +/* 64-bit key space */ +/* 2M window instances per comm */ +/* 2M outstanding huge RMAS per comm */ +/* 4M communicators */ +#define MPIDI_OFI_MAX_WINDOWS_BITS_64 (21) +#define MPIDI_OFI_MAX_HUGE_RMA_BITS_64 (21) +#define MPIDI_OFI_MAX_HUGE_RMAS_64 (1<<(MPIDI_OFI_MAX_HUGE_RMA_BITS_64)) +#define MPIDI_OFI_MAX_WINDOWS_64 (1<<(MPIDI_OFI_MAX_WINDOWS_BITS_64)) +#define MPIDI_OFI_HUGE_RMA_SHIFT_64 (MPIDI_OFI_MAX_WINDOWS_BITS_64) +#define MPIDI_OFI_CONTEXT_SHIFT_64 (MPIDI_OFI_MAX_WINDOWS_BITS_64+MPIDI_OFI_MAX_HUGE_RMA_BITS_64) + +/* 32-bit key space */ +/* 4096 window instances per comm */ +/* 256 outstanding huge RMAS per comm */ +/* 4096 communicators */ +#define MPIDI_OFI_MAX_WINDOWS_BITS_32 (12) +#define MPIDI_OFI_MAX_HUGE_RMA_BITS_32 (8) +#define MPIDI_OFI_MAX_HUGE_RMAS_32 (1<<(MPIDI_OFI_MAX_HUGE_RMA_BITS_32)) +#define MPIDI_OFI_MAX_WINDOWS_32 (1<<(MPIDI_OFI_MAX_WINDOWS_BITS_32)) +#define MPIDI_OFI_HUGE_RMA_SHIFT_32 (MPIDI_OFI_MAX_WINDOWS_BITS_32) +#define MPIDI_OFI_CONTEXT_SHIFT_32 (MPIDI_OFI_MAX_WINDOWS_BITS_32+MPIDI_OFI_MAX_HUGE_RMA_BITS_32) + +/* 16-bit key space */ +/* 64 window instances per comm */ +/* 16 outstanding huge RMAS per comm */ +/* 64 communicators */ +#define MPIDI_OFI_MAX_WINDOWS_BITS_16 (6) +#define MPIDI_OFI_MAX_HUGE_RMA_BITS_16 (4) +#define MPIDI_OFI_MAX_HUGE_RMAS_16 (1<<(MPIDI_OFI_MAX_HUGE_RMA_BITS_16)) +#define MPIDI_OFI_MAX_WINDOWS_16 (1<<(MPIDI_OFI_MAX_WINDOWS_BITS_16)) +#define MPIDI_OFI_HUGE_RMA_SHIFT_16 (MPIDI_OFI_MAX_WINDOWS_BITS_16) +#define MPIDI_OFI_CONTEXT_SHIFT_16 (MPIDI_OFI_MAX_WINDOWS_BITS_16+MPIDI_OFI_MAX_HUGE_RMA_BITS_16) + +#ifdef HAVE_FORTRAN_BINDING +#ifdef MPICH_DEFINE_2COMPLEX +#define MPIDI_OFI_DT_SIZES 62 +#else +#define MPIDI_OFI_DT_SIZES 60 +#endif +#else +#define MPIDI_OFI_DT_SIZES 40 +#endif +#define MPIDI_OFI_OP_SIZES 15 + +#define MPIDI_OFI_API_TAG 0 +#define MPIDI_OFI_API_RMA 1 +#define MPIDI_OFI_API_MSG 2 +#define MPIDI_OFI_API_CTR 3 + +#define MPIDI_OFI_THREAD_UTIL_MUTEX MPIDI_Global.mutexes[0].m +#define MPIDI_OFI_THREAD_PROGRESS_MUTEX MPIDI_Global.mutexes[1].m +#define MPIDI_OFI_THREAD_FI_MUTEX MPIDI_Global.mutexes[2].m +#define MPIDI_OFI_THREAD_SPAWN_MUTEX MPIDI_Global.mutexes[3].m + +/* Field accessor macros */ +#define MPIDI_OFI_GPID(gpid) ((gpid)->dev.netmod.ofi) +#define MPIDI_OFI_OBJECT_HEADER_SIZE offsetof(MPIDI_OFI_offset_checker_t, pad) +#define MPIDI_OFI_AMREQUEST(req,field) ((req)->dev.ch4.ch4u.netmod_am.ofi.field) +#define MPIDI_OFI_AMREQUEST_HDR(req,field) ((req)->dev.ch4.ch4u.netmod_am.ofi.req_hdr->field) +#define MPIDI_OFI_AMREQUEST_HDR_PTR(req) ((req)->dev.ch4.ch4u.netmod_am.ofi.req_hdr) +#define MPIDI_OFI_REQUEST(req,field) ((req)->dev.ch4.netmod.ofi.field) +#define MPIDI_OFI_AV(av) ((av)->netmod.ofi) + +#define MPIDI_OFI_DATATYPE(dt) ((dt)->dev.netmod.ofi) +#define MPIDI_OFI_COMM(comm) ((comm)->dev.ch4.netmod.ofi) + +#ifdef MPIDI_OFI_CONFIG_USE_SCALABLE_ENDPOINTS +#define MPIDI_OFI_COMM_TO_EP(comm,rank) MPIDI_OFI_AV(MPIDIU_comm_rank_to_av(comm, rank)).ep_idx +#define MPIDI_OFI_EP_TX_TAG(x) MPIDI_Global.ctx[x].tx_tag +#define MPIDI_OFI_EP_TX_RMA(x) MPIDI_Global.ctx[x].tx_rma +#define MPIDI_OFI_EP_TX_MSG(x) MPIDI_Global.ctx[x].tx_msg +#define MPIDI_OFI_EP_TX_CTR(x) MPIDI_Global.ctx[x].tx_ctr +#define MPIDI_OFI_EP_RX_TAG(x) MPIDI_Global.ctx[x].rx_tag +#define MPIDI_OFI_EP_RX_RMA(x) MPIDI_Global.ctx[x].rx_rma +#define MPIDI_OFI_EP_RX_MSG(x) MPIDI_Global.ctx[x].rx_msg +#define MPIDI_OFI_EP_RX_CTR(x) MPIDI_Global.ctx[x].rx_ctr +#else +#define MPIDI_OFI_COMM_TO_EP(comm,rank) 0 +#define MPIDI_OFI_EP_TX_TAG(x) MPIDI_Global.ep +#define MPIDI_OFI_EP_TX_RMA(x) MPIDI_Global.ep +#define MPIDI_OFI_EP_TX_MSG(x) MPIDI_Global.ep +#define MPIDI_OFI_EP_TX_CTR(x) MPIDI_Global.ep +#define MPIDI_OFI_EP_RX_TAG(x) MPIDI_Global.ep +#define MPIDI_OFI_EP_RX_RMA(x) MPIDI_Global.ep +#define MPIDI_OFI_EP_RX_MSG(x) MPIDI_Global.ep +#define MPIDI_OFI_EP_RX_CTR(x) MPIDI_Global.ep +#endif + +#define MPIDI_OFI_DO_SEND 0 +#define MPIDI_OFI_DO_INJECT 1 +#define MPIDI_OFI_NUM_CQ_ENTRIES 8 + +/* Typedefs */ +enum { + MPIDI_OFI_CTRL_ASSERT, /**< Lock acknowledge */ + MPIDI_OFI_CTRL_LOCKACK, /**< Lock acknowledge */ + MPIDI_OFI_CTRL_LOCKALLACK,/**< Lock all acknowledge */ + MPIDI_OFI_CTRL_LOCKREQ, /**< Lock window */ + MPIDI_OFI_CTRL_LOCKALLREQ,/**< Lock all window */ + MPIDI_OFI_CTRL_UNLOCK, /**< Unlock window */ + MPIDI_OFI_CTRL_UNLOCKACK, /**< Unlock window */ + MPIDI_OFI_CTRL_UNLOCKALL, /**< Unlock window */ + MPIDI_OFI_CTRL_UNLOCKALLACK, + /**< Unlock window */ + MPIDI_OFI_CTRL_COMPLETE, /**< End a START epoch */ + MPIDI_OFI_CTRL_POST, /**< Begin POST epoch */ + MPIDI_OFI_CTRL_HUGE, /**< Huge message */ + MPIDI_OFI_CTRL_HUGEACK, /**< Huge message ack */ + MPIDI_OFI_CTRL_HUGE_CLEANUP + /**< Huge message cleanup */ +}; + +enum { + MPIDI_OFI_EVENT_ABORT, + MPIDI_OFI_EVENT_SEND, + MPIDI_OFI_EVENT_RECV, + MPIDI_OFI_EVENT_RMA_DONE, + MPIDI_OFI_EVENT_AM_SEND, + MPIDI_OFI_EVENT_AM_RECV, + MPIDI_OFI_EVENT_AM_READ, + MPIDI_OFI_EVENT_AM_MULTI, + MPIDI_OFI_EVENT_PEEK, + MPIDI_OFI_EVENT_RECV_HUGE, + MPIDI_OFI_EVENT_SEND_HUGE, + MPIDI_OFI_EVENT_SSEND_ACK, + MPIDI_OFI_EVENT_GET_HUGE, + MPIDI_OFI_EVENT_CHUNK_DONE, + MPIDI_OFI_EVENT_INJECT_EMU, + MPIDI_OFI_EVENT_DYNPROC_DONE, + MPIDI_OFI_EVENT_ACCEPT_PROBE +}; + +enum { + MPIDI_OFI_REQUEST_LOCK, + MPIDI_OFI_REQUEST_LOCKALL +}; + +enum { + MPIDI_OFI_PEEK_START, + MPIDI_OFI_PEEK_NOT_FOUND, + MPIDI_OFI_PEEK_FOUND +}; + +typedef struct { + char pad[MPIDI_REQUEST_HDR_SIZE]; + struct fi_context context; /* fixed field, do not move */ + int event_id; /* fixed field, do not move */ + int index; +} MPIDI_OFI_am_repost_request_t; + +typedef struct { + char pad[MPIDI_REQUEST_HDR_SIZE]; + struct fi_context context; /* fixed field, do not move */ + int event_id; /* fixed field, do not move */ + MPIR_Request *signal_req; +} MPIDI_OFI_ssendack_request_t; + +typedef struct { + char pad[MPIDI_REQUEST_HDR_SIZE]; + struct fi_context context; /* fixed field, do not move */ + int event_id; /* fixed field, do not move */ + int done; + uint32_t tag; + uint32_t source; + uint64_t msglen; +} MPIDI_OFI_dynamic_process_request_t; + +typedef struct { + uint8_t op; + uint8_t dt; + unsigned atomic_valid:2; + unsigned fetch_atomic_valid:2; + unsigned compare_atomic_valid:2; + unsigned dtsize:10; + uint64_t max_atomic_count; + uint64_t max_compare_atomic_count; + uint64_t max_fetch_atomic_count; +} MPIDI_OFI_atomic_valid_t; + +typedef struct { + struct fid_ep *tx_tag; + struct fid_ep *rx_tag; + + struct fid_ep *tx_rma; + struct fid_ep *rx_rma; + + struct fid_ep *tx_msg; + struct fid_ep *rx_msg; + + struct fid_ep *tx_ctr; + struct fid_ep *rx_ctr; + + int ctx_offset; +} MPIDI_OFI_context_t; + +typedef union { + MPID_Thread_mutex_t m; + char cacheline[MPIDI_OFI_CACHELINE_SIZE]; +} MPIDI_OFI_cacheline_mutex_t __attribute__ ((aligned(MPIDI_OFI_CACHELINE_SIZE))); + +typedef struct { + struct fi_cq_tagged_entry cq_entry; + fi_addr_t source; + struct slist_entry entry; +} MPIDI_OFI_cq_list_t; + +typedef struct { + struct fi_cq_tagged_entry cq_entry; +} MPIDI_OFI_cq_buff_entry_t; + +/* Global state data */ +#define MPIDI_KVSAPPSTRLEN 1024 +typedef struct { + /* OFI objects */ + int avtid; + struct fi_info *prov_use; + struct fid_domain *domain; + struct fid_fabric *fabric; + struct fid_av *av; + struct fid_ep *ep; + struct fid_cq *p2p_cq; + struct fid_cntr *rma_cmpl_cntr; + struct fid_stx *stx_ctx; /* shared TX context for RMA */ + + /* Queryable limits */ + uint64_t max_buffered_send; + uint64_t max_buffered_write; + uint64_t max_send; + uint64_t max_write; + uint64_t max_short_send; + uint64_t max_mr_key_size; + int max_windows_bits; + int max_huge_rma_bits; + int max_huge_rmas; + int huge_rma_shift; + int context_shift; + size_t iov_limit; + size_t rma_iov_limit; + + /* Mutexex and endpoints */ + MPIDI_OFI_cacheline_mutex_t mutexes[4]; + MPIDI_OFI_context_t ctx[MPIDI_OFI_MAX_ENDPOINTS]; + + /* Window/RMA Globals */ + void *win_map; + uint64_t rma_issued_cntr; + MPIDI_OFI_atomic_valid_t win_op_table[MPIDI_OFI_DT_SIZES][MPIDI_OFI_OP_SIZES]; + + /* Active Message Globals */ + struct iovec am_iov[MPIDI_OFI_NUM_AM_BUFFERS]; + struct fi_msg am_msg[MPIDI_OFI_NUM_AM_BUFFERS]; + void *am_bufs[MPIDI_OFI_NUM_AM_BUFFERS]; + MPIDI_OFI_am_repost_request_t am_reqs[MPIDI_OFI_NUM_AM_BUFFERS]; + MPIDI_NM_am_target_handler_fn am_handlers[MPIDI_OFI_MAX_AM_HANDLERS_TOTAL]; + MPIDI_NM_am_origin_handler_fn am_send_cmpl_handlers[MPIDI_OFI_MAX_AM_HANDLERS_TOTAL]; + MPIU_buf_pool_t *am_buf_pool; + OPA_int_t am_inflight_inject_emus; + OPA_int_t am_inflight_rma_send_mrs; + + /* Completion queue buffering */ + MPIDI_OFI_cq_buff_entry_t cq_buffered[MPIDI_OFI_NUM_CQ_BUFFERED]; + struct slist cq_buff_list; + int cq_buff_head; + int cq_buff_tail; + + /* Process management and PMI globals */ + int pname_set; + int pname_len; + int jobid; + char addrname[FI_NAME_MAX]; + size_t addrnamelen; + char kvsname[MPIDI_KVSAPPSTRLEN]; + char pname[MPI_MAX_PROCESSOR_NAME]; + int port_name_tag_mask[MPIR_MAX_CONTEXT_MASK]; +} MPIDI_OFI_global_t; + +typedef struct { + uint32_t index; +} MPIDI_OFI_datatype_t; +/* These control structures have to be the same size */ +typedef struct { + int16_t type; + int16_t lock_type; + int origin_rank; + uint64_t win_id; + int dummy[8]; +} MPIDI_OFI_win_control_t; + +typedef struct { + int16_t type; + int16_t seqno; + int origin_rank; + MPIR_Request *ackreq; + uintptr_t send_buf; + size_t msgsize; + int comm_id; + int endpoint_id; + uint64_t rma_key; +} MPIDI_OFI_send_control_t; + +typedef struct { + MPIR_OBJECT_HEADER; + void *pad; +} MPIDI_OFI_offset_checker_t; + +typedef struct { + uintptr_t target_base_addr; + uintptr_t origin_base_addr; + uintptr_t result_base_addr; + size_t target_count; + size_t origin_count; + size_t result_count; + struct iovec *target_iov; + struct iovec *origin_iov; + struct iovec *result_iov; + size_t target_idx; + uintptr_t target_addr; + uintptr_t target_size; + size_t origin_idx; + uintptr_t origin_addr; + uintptr_t origin_size; + size_t result_idx; + uintptr_t result_addr; + uintptr_t result_size; + size_t buf_limit; + size_t buf_limit_left; +} MPIDI_OFI_iovec_state_t; + +typedef struct { + MPIR_Datatype *pointer; + MPI_Datatype type; + int count; + int contig; + MPI_Aint true_lb; + size_t size; + int num_contig; + DLOOP_VECTOR *map; + DLOOP_VECTOR __map; +} MPIDI_OFI_win_datatype_t; + +typedef struct { + char pad[MPIDI_REQUEST_HDR_SIZE]; + struct fi_context context; /* fixed field, do not move */ + int event_id; /* fixed field, do not move */ + struct MPIDI_Iovec_array *next; + union { + struct { + struct iovec *originv; + struct fi_rma_iov *targetv; + } put_get; + struct { + struct fi_ioc *originv; + struct fi_rma_ioc *targetv; + struct fi_ioc *resultv; + struct fi_ioc *comparev; + } cas; + struct { + struct fi_ioc *originv; + struct fi_rma_ioc *targetv; + } accumulate; + struct { + struct fi_ioc *originv; + struct fi_rma_ioc *targetv; + struct fi_ioc *resultv; + } get_accumulate; + } iov; + char iov_store[0]; /* Flexible array, do not move */ +} MPIDI_OFI_iovec_array_t; + +typedef struct { + MPIDI_OFI_iovec_state_t iovs; + MPIDI_OFI_win_datatype_t origin_dt; + MPIDI_OFI_win_datatype_t target_dt; + MPIDI_OFI_win_datatype_t result_dt; + MPIDI_OFI_iovec_array_t buf; /* Do not move me, flexible array */ +} MPIDI_OFI_win_noncontig_t; + +typedef struct MPIDI_OFI_win_request { + MPIR_OBJECT_HEADER; + char pad[MPIDI_REQUEST_HDR_SIZE - MPIDI_OFI_OBJECT_HEADER_SIZE]; + struct fi_context context; /* fixed field, do not move */ + int event_id; /* fixed field, do not move */ + struct MPIDI_OFI_win_request *next; + int target_rank; + MPIDI_OFI_win_noncontig_t *noncontig; +} MPIDI_OFI_win_request_t; + +typedef struct { + char pad[MPIDI_REQUEST_HDR_SIZE]; + struct fi_context context; /* fixed field, do not move */ + int event_id; /* fixed field, do not move */ + MPIR_Request *parent; /* Parent request */ +} MPIDI_OFI_chunk_request; + +typedef struct { + char pad[MPIDI_REQUEST_HDR_SIZE]; + struct fi_context context; /* fixed field, do not move */ + int event_id; /* fixed field, do not move */ + int (*done_fn) (struct fi_cq_tagged_entry * wc, MPIR_Request * req); + MPIDI_OFI_send_control_t remote_info; + size_t cur_offset; + MPIR_Comm *comm_ptr; + MPIR_Request *localreq; + struct fi_cq_tagged_entry wc; +} MPIDI_OFI_huge_recv_t; + +typedef struct MPIDI_OFI_huge_counter_t { + uint16_t counter; + uint16_t outstanding; + struct fid_mr *mr; +} MPIDI_OFI_huge_counter_t; + +/* Externs */ +extern MPIDI_OFI_global_t MPIDI_Global; +extern int MPIR_Datatype_init_names(void); + +#endif /* NETMOD_OFI_IMPL_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/ofi/ofi_unimpl.h b/src/mpid/ch4/netmod/ofi/ofi_unimpl.h new file mode 100644 index 0000000..db8e66b --- /dev/null +++ b/src/mpid/ch4/netmod/ofi/ofi_unimpl.h @@ -0,0 +1,19 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ + +#ifdef __cplusplus +extern "C" { +#endif + + +#ifdef __cplusplus +} +#endif diff --git a/src/mpid/ch4/netmod/ofi/ofi_win.h b/src/mpid/ch4/netmod/ofi/ofi_win.h new file mode 100644 index 0000000..2975845 --- /dev/null +++ b/src/mpid/ch4/netmod/ofi/ofi_win.h @@ -0,0 +1,1254 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_OFI_WIN_H_INCLUDED +#define NETMOD_OFI_WIN_H_INCLUDED + +#include "ofi_impl.h" +#include <opa_primitives.h> + + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_win_allgather +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_OFI_win_allgather(MPIR_Win * win, void *base, int disp_unit) +{ + int i, same_disp, mpi_errno = MPI_SUCCESS; + uint32_t first; + MPIR_Errflag_t errflag = MPIR_ERR_NONE; + MPIR_Comm *comm_ptr = win->comm_ptr; + int raw_prefix, idx, bitpos; + unsigned gen_id; + MPIDI_OFI_win_targetinfo_t *winfo; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_OFI_WIN_ALLGATHER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_OFI_WIN_ALLGATHER); + + /* Calculate a canonical context id */ + raw_prefix = MPIR_CONTEXT_READ_FIELD(PREFIX, comm_ptr->context_id); + idx = raw_prefix / MPIR_CONTEXT_INT_BITS; + bitpos = raw_prefix % MPIR_CONTEXT_INT_BITS; + gen_id = (idx * MPIR_CONTEXT_INT_BITS) + (31 - bitpos); + + int total_bits_avail = MPIDI_Global.max_mr_key_size * 8; + uint64_t window_instance = (uint64_t) (MPIDI_OFI_WIN(win).win_id) >> 32; + int bits_for_instance_id = MPIDI_Global.max_windows_bits; + int bits_for_context_id; + uint64_t max_contexts_allowed; + uint64_t max_instances_allowed; + + bits_for_context_id = total_bits_avail - + MPIDI_Global.max_windows_bits - MPIDI_Global.max_huge_rma_bits; + max_contexts_allowed = 1 << (bits_for_context_id); + max_instances_allowed = 1 << (bits_for_instance_id); + MPIR_ERR_CHKANDSTMT(gen_id >= max_contexts_allowed, mpi_errno, MPI_ERR_OTHER, + goto fn_fail, "**ofid_mr_reg"); + MPIR_ERR_CHKANDSTMT(window_instance >= max_instances_allowed, mpi_errno, MPI_ERR_OTHER, + goto fn_fail, "**ofid_mr_reg"); + + if (MPIDI_OFI_ENABLE_MR_SCALABLE) { + /* Context id in lower bits, instance in upper bits */ + MPIDI_OFI_WIN(win).mr_key = (gen_id << MPIDI_Global.context_shift) | window_instance; + } + else { + MPIDI_OFI_WIN(win).mr_key = 0; + } + + MPIDI_OFI_CALL(fi_mr_reg(MPIDI_Global.domain, /* In: Domain Object */ + base, /* In: Lower memory address */ + win->size, /* In: Length */ + FI_REMOTE_READ | FI_REMOTE_WRITE, /* In: Expose MR for read */ + 0ULL, /* In: offset(not used) */ + MPIDI_OFI_WIN(win).mr_key, /* In: requested key */ + 0ULL, /* In: flags */ + &MPIDI_OFI_WIN(win).mr, /* Out: memregion object */ + NULL), mr_reg); /* In: context */ + + MPIDI_OFI_WIN(win).winfo = MPL_malloc(sizeof(*winfo) * comm_ptr->local_size); + + winfo = MPIDI_OFI_WIN(win).winfo; + winfo[comm_ptr->rank].disp_unit = disp_unit; + +#ifndef USE_OFI_MR_SCALABLE + /* MR_BASIC */ + MPIDI_OFI_WIN(win).mr_key = fi_mr_key(MPIDI_OFI_WIN(win).mr); + winfo[comm_ptr->rank].mr_key = MPIDI_OFI_WIN(win).mr_key; + winfo[comm_ptr->rank].base = (uintptr_t) base; +#endif + + mpi_errno = MPIR_Allgather_impl(MPI_IN_PLACE, 0, + MPI_DATATYPE_NULL, + winfo, sizeof(*winfo), MPI_BYTE, comm_ptr, &errflag); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + if (MPIDI_OFI_ENABLE_MR_SCALABLE) { + first = winfo[0].disp_unit; + same_disp = 1; + for (i = 1; i < comm_ptr->local_size; i++) { + if (winfo[i].disp_unit != first) { + same_disp = 0; + break; + } + } + if (same_disp) { + MPL_free(MPIDI_OFI_WIN(win).winfo); + MPIDI_OFI_WIN(win).winfo = NULL; + } + } + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_OFI_PROGRESS_WIN_ALLGATHER); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_win_init_generic +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_OFI_win_init_generic(MPI_Aint length, + int disp_unit, + MPIR_Win ** win_ptr, + MPIR_Info * info, + MPIR_Comm * comm_ptr, + int create_flavor, int model, int do_stx_rma) +{ + int mpi_errno = MPI_SUCCESS; + uint64_t window_instance; + MPIR_Win *win; + struct fi_info *finfo; + struct fi_cntr_attr cntr_attr; + + CH4_COMPILE_TIME_ASSERT(sizeof(MPIDI_Devwin_t) >= sizeof(MPIDI_OFI_win_t)); + CH4_COMPILE_TIME_ASSERT(sizeof(MPIDI_Devdt_t) >= sizeof(MPIDI_OFI_datatype_t)); + + /* Note: MPIDI_CH4U_win_init will interpret the info object */ + mpi_errno = MPIDI_CH4R_win_init(length, disp_unit, &win, info, comm_ptr, create_flavor, model); + MPIR_ERR_CHKANDSTMT(mpi_errno != MPI_SUCCESS, + mpi_errno, MPI_ERR_NO_MEM, goto fn_fail, "**nomem"); + *win_ptr = win; + + memset(&MPIDI_OFI_WIN(win), 0, sizeof(MPIDI_OFI_win_t)); + + /* context id lower bits, window instance upper bits */ + window_instance = + MPIDI_OFI_index_allocator_alloc(MPIDI_OFI_COMM(win->comm_ptr).win_id_allocator); + MPIDI_OFI_WIN(win).win_id = ((uint64_t) comm_ptr->context_id) | (window_instance << 32); + MPIDI_OFI_map_set(MPIDI_Global.win_map, MPIDI_OFI_WIN(win).win_id, win); + + if (do_stx_rma && MPIDI_Global.stx_ctx != NULL) { + /* Activate per-window EP/counter */ + int ret; + + finfo = fi_dupinfo(MPIDI_Global.prov_use); + MPIR_Assert(finfo); + finfo->ep_attr->tx_ctx_cnt = FI_SHARED_CONTEXT; /* Request a shared context */ + MPIDI_OFI_CALL_RETURN(fi_endpoint(MPIDI_Global.domain, + finfo, &MPIDI_OFI_WIN(win).ep, NULL), ret); + fi_freeinfo(finfo); + if (ret < 0) { + MPL_DBG_MSG(MPIDI_CH4_DBG_GENERAL, VERBOSE, + "Failed to create per-window EP (with completion), " + "falling back to global EP/counter scheme"); + goto fallback_global; + } + + memset(&cntr_attr, 0, sizeof(cntr_attr)); + cntr_attr.events = FI_CNTR_EVENTS_COMP; + MPIDI_OFI_CALL(fi_cntr_open(MPIDI_Global.domain, /* In: Domain Object */ + &cntr_attr, /* In: Configuration object */ + &MPIDI_OFI_WIN(win).cmpl_cntr, /* Out: Counter Object */ + NULL), openct); /* Context: counter events */ + MPIDI_OFI_WIN(win).issued_cntr = &MPIDI_OFI_WIN(win).issued_cntr_v; + + MPIDI_OFI_CALL(fi_ep_bind(MPIDI_OFI_WIN(win).ep, &MPIDI_Global.stx_ctx->fid, 0), bind); + MPIDI_OFI_CALL(fi_ep_bind(MPIDI_OFI_WIN(win).ep, + &MPIDI_Global.p2p_cq->fid, FI_TRANSMIT | FI_SELECTIVE_COMPLETION), + bind); + MPIDI_OFI_CALL(fi_ep_bind + (MPIDI_OFI_WIN(win).ep, &MPIDI_OFI_WIN(win).cmpl_cntr->fid, + FI_READ | FI_WRITE), bind); + MPIDI_OFI_CALL(fi_ep_bind(MPIDI_OFI_WIN(win).ep, &MPIDI_Global.av->fid, 0), bind); + + MPIDI_OFI_CALL_RETURN(fi_ep_alias(MPIDI_OFI_WIN(win).ep, &MPIDI_OFI_WIN(win).ep_nocmpl, + FI_TRANSMIT), ret); + if (ret < 0) { + MPL_DBG_MSG(MPIDI_CH4_DBG_GENERAL, VERBOSE, + "Failed to create an EP alias, " + "falling back to global EP/counter scheme"); + MPIDI_OFI_CALL(fi_close(&MPIDI_OFI_WIN(win).ep->fid), epclose); + MPIDI_OFI_CALL(fi_close(&MPIDI_OFI_WIN(win).cmpl_cntr->fid), epclose); + goto fallback_global; + } + + MPIDI_OFI_CALL(fi_enable(MPIDI_OFI_WIN(win).ep), ep_enable); + MPIDI_OFI_CALL(fi_enable(MPIDI_OFI_WIN(win).ep_nocmpl), ep_enable); + } + else { + fallback_global: + /* Fallback for the traditional global EP/counter model */ + MPIDI_OFI_WIN(win).ep = MPIDI_OFI_EP_TX_RMA(0); + MPIDI_OFI_WIN(win).ep_nocmpl = MPIDI_OFI_EP_TX_CTR(0); + MPIDI_OFI_WIN(win).cmpl_cntr = MPIDI_Global.rma_cmpl_cntr; + MPIDI_OFI_WIN(win).issued_cntr = &MPIDI_Global.rma_issued_cntr; + } + + fn_exit: + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_win_init +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_OFI_win_init(MPI_Aint length, + int disp_unit, + MPIR_Win ** win_ptr, + MPIR_Info * info, + MPIR_Comm * comm_ptr, int create_flavor, int model) +{ + int mpi_errno; + int use_stx_rma = MPIDI_OFI_ENABLE_SCALABLE_ENDPOINTS ? 0 : MPIDI_OFI_ENABLE_STX_RMA; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_OFI_WIN_INIT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_OFI_WIN_INIT); + + mpi_errno = MPIDI_OFI_win_init_generic(length, + disp_unit, + win_ptr, + info, comm_ptr, create_flavor, model, use_stx_rma); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_OFI_PROGRESS_WIN_INIT); + + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Win_progress_fence +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_Win_progress_fence(MPIR_Win * win) +{ + int mpi_errno = MPI_SUCCESS; + int itercount = 0; + int ret; + uint64_t tcount, donecount; + MPIDI_OFI_win_request_t *r; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_OFI_PROGRESS_WIN_COUNTER_FENCE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_OFI_PROGRESS_WIN_COUNTER_FENCE); + + MPID_THREAD_CS_ENTER(POBJ, MPIDI_OFI_THREAD_FI_MUTEX); + tcount = *MPIDI_OFI_WIN(win).issued_cntr; + donecount = fi_cntr_read(MPIDI_OFI_WIN(win).cmpl_cntr); + + MPIR_Assert(donecount <= tcount); + + while (tcount > donecount) { + MPIR_Assert(donecount <= tcount); + MPID_THREAD_CS_EXIT(POBJ, MPIDI_OFI_THREAD_FI_MUTEX); + MPIDI_OFI_PROGRESS(); + MPID_THREAD_CS_ENTER(POBJ, MPIDI_OFI_THREAD_FI_MUTEX); + donecount = fi_cntr_read(MPIDI_OFI_WIN(win).cmpl_cntr); + itercount++; + + if (itercount == 1000) { + ret = fi_cntr_wait(MPIDI_OFI_WIN(win).cmpl_cntr, tcount, 0); + MPIDI_OFI_ERR(ret < 0 && ret != -FI_ETIMEDOUT, + mpi_errno, + MPI_ERR_RMA_RANGE, + "**ofid_cntr_wait", + "**ofid_cntr_wait %s %d %s %s", + __SHORT_FILE__, __LINE__, FCNAME, fi_strerror(-ret)); + itercount = 0; + } + } + + while (OPA_load_int(&MPIDI_CH4U_WIN(win, outstanding_ops)) != 0) + MPIDI_OFI_PROGRESS(); + + r = MPIDI_OFI_WIN(win).syncQ; + + while (r) { + MPIDI_OFI_win_request_t *next = r->next; + MPIDI_OFI_rma_done_event(NULL, (MPIR_Request *) r); + r = next; + } + + MPIDI_OFI_WIN(win).syncQ = NULL; + fn_exit: + MPID_THREAD_CS_EXIT(POBJ, MPIDI_OFI_THREAD_FI_MUTEX); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_OFI_PROGRESS_WIN_COUNTER_FENCE); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_win_set_info +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_win_set_info(MPIR_Win * win, MPIR_Info * info) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_WIN_SET_INFO); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_WIN_SET_INFO); + + mpi_errno = MPIDI_CH4R_win_set_info(win, info); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_WIN_SET_INFO); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_win_start +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_win_start(MPIR_Group * group, int assert, MPIR_Win * win) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_WIN_START); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_WIN_START); + + MPIDI_CH4U_EPOCH_CHECK_TYPE(win, mpi_errno, goto fn_fail); + + MPIR_Group_add_ref(group); + + MPIDI_OFI_PROGRESS_WHILE(group->size != (int) MPIDI_CH4U_WIN(win, sync).pw.count); + + MPIDI_CH4U_WIN(win, sync).pw.count = 0; + + MPIR_ERR_CHKANDJUMP((MPIDI_CH4U_WIN(win, sync).sc.group != NULL), + mpi_errno, MPI_ERR_GROUP, "**group"); + MPIDI_CH4U_WIN(win, sync).sc.group = group; + MPIDI_CH4U_WIN(win, sync).origin_epoch_type = MPIDI_CH4U_EPOTYPE_START; + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_WIN_START); + return mpi_errno; + fn_fail: + goto fn_exit; +} + + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_win_complete +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_win_complete(MPIR_Win * win) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_WIN_COMPLETE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_WIN_COMPLETE); + + MPIDI_CH4U_EPOCH_START_CHECK2(win, mpi_errno, goto fn_fail); + + MPIDI_OFI_MPI_CALL_POP(MPIDI_Win_progress_fence(win)); + + MPIR_Group *group; + group = MPIDI_CH4U_WIN(win, sync).sc.group; + MPIR_Assert(group != NULL); + MPIDI_OFI_win_control_t msg; + msg.type = MPIDI_OFI_CTRL_COMPLETE; + + int index, peer; + + for (index = 0; index < group->size; ++index) { + peer = group->lrank_to_lpid[index].lpid; + mpi_errno = MPIDI_OFI_do_control_win(&msg, peer, win, 0, 1); + + if (mpi_errno != MPI_SUCCESS) + MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, goto fn_fail, "**rmasync"); + } + + MPIDI_CH4U_EPOCH_TARGET_EVENT(win); + + MPIR_Group_release(MPIDI_CH4U_WIN(win, sync).sc.group); + MPIDI_CH4U_WIN(win, sync).sc.group = NULL; + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_WIN_COMPLETE); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_win_post +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_win_post(MPIR_Group * group, int assert, MPIR_Win * win) +{ + int peer, index, mpi_errno = MPI_SUCCESS; + MPIDI_OFI_win_control_t msg; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_WIN_POST); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_WIN_POST); + + MPIDI_CH4U_EPOCH_POST_CHECK(win, mpi_errno, goto fn_fail); + + MPIR_Group_add_ref(group); + MPIR_ERR_CHKANDJUMP((MPIDI_CH4U_WIN(win, sync).pw.group != NULL), + mpi_errno, MPI_ERR_GROUP, "**group"); + + MPIDI_CH4U_WIN(win, sync).pw.group = group; + MPIR_Assert(group != NULL); + + msg.type = MPIDI_OFI_CTRL_POST; + + for (index = 0; index < group->size; ++index) { + peer = group->lrank_to_lpid[index].lpid; + mpi_errno = MPIDI_OFI_do_control_win(&msg, peer, win, 0, 1); + + if (mpi_errno != MPI_SUCCESS) + MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, goto fn_fail, "**rmasync"); + } + + MPIDI_CH4U_WIN(win, sync).target_epoch_type = MPIDI_CH4U_EPOTYPE_POST; + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_WIN_POST); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_win_wait +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_win_wait(MPIR_Win * win) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_WIN_WAIT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_WIN_WAIT); + + MPIDI_CH4U_EPOCH_TARGET_CHECK(win, MPIDI_CH4U_EPOTYPE_POST, mpi_errno, return mpi_errno); + + MPIR_Group *group; + group = MPIDI_CH4U_WIN(win, sync).pw.group; + + MPIDI_OFI_PROGRESS_WHILE(group->size != (int) MPIDI_CH4U_WIN(win, sync).sc.count); + + MPIDI_CH4U_WIN(win, sync).sc.count = 0; + MPIDI_CH4U_WIN(win, sync).pw.group = NULL; + + MPIR_Group_release(group); + + MPIDI_CH4U_EPOCH_ORIGIN_EVENT(win); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_WIN_WAIT); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_win_test +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_win_test(MPIR_Win * win, int *flag) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_WIN_TEST); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_WIN_TEST); + + MPIDI_CH4U_EPOCH_TARGET_CHECK(win, MPIDI_CH4U_EPOTYPE_POST, mpi_errno, return mpi_errno); + + MPIR_Group *group; + group = MPIDI_CH4U_WIN(win, sync).pw.group; + + if (group->size == (int) MPIDI_CH4U_WIN(win, sync).sc.count) { + MPIDI_CH4U_WIN(win, sync).sc.count = 0; + MPIDI_CH4U_WIN(win, sync).pw.group = NULL; + *flag = 1; + MPIR_Group_release(group); + MPIDI_CH4U_EPOCH_ORIGIN_EVENT(win); + } + else { + MPIDI_OFI_PROGRESS(); + *flag = 0; + } + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_WIN_TEST); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_win_lock +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_win_lock(int lock_type, int rank, int assert, MPIR_Win * win) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_WIN_LOCK); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_WIN_LOCK); + + MPIDI_CH4U_win_sync_lock *slock = &MPIDI_CH4U_WIN(win, sync).lock; + MPIDI_CH4U_EPOCH_CHECK_TYPE(win, mpi_errno, goto fn_fail); + + if (rank == MPI_PROC_NULL) + goto fn_exit0; + + MPIDI_OFI_win_control_t msg; + + msg.type = MPIDI_OFI_CTRL_LOCKREQ; + msg.lock_type = lock_type; + + mpi_errno = MPIDI_OFI_do_control_win(&msg, rank, win, 1, 1); + + if (mpi_errno != MPI_SUCCESS) + MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, goto fn_fail, "**rmasync"); + + MPIDI_OFI_PROGRESS_WHILE(!slock->remote.locked); + + fn_exit0: + MPIDI_CH4U_WIN(win, sync).origin_epoch_type = MPIDI_CH4U_EPOTYPE_LOCK; + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_WIN_LOCK); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_win_unlock +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_win_unlock(int rank, MPIR_Win * win) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_WIN_UNLOCK); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_WIN_UNLOCK); + + MPIDI_CH4U_EPOCH_ORIGIN_CHECK(win, MPIDI_CH4U_EPOTYPE_LOCK, mpi_errno, return mpi_errno); + + if (rank == MPI_PROC_NULL) + goto fn_exit0; + + MPIDI_OFI_MPI_CALL_POP(MPIDI_Win_progress_fence(win)); + + MPIDI_OFI_win_control_t msg; + msg.type = MPIDI_OFI_CTRL_UNLOCK; + mpi_errno = MPIDI_OFI_do_control_win(&msg, rank, win, 1, 1); + + if (mpi_errno != MPI_SUCCESS) + MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, goto fn_fail, "**rmasync"); + + MPIDI_OFI_PROGRESS_WHILE(MPIDI_CH4U_WIN(win, sync).lock.remote.locked); + fn_exit0: + MPIDI_CH4U_WIN(win, sync).origin_epoch_type = MPIDI_CH4U_EPOTYPE_NONE; + MPIDI_CH4U_WIN(win, sync).target_epoch_type = MPIDI_CH4U_EPOTYPE_NONE; + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_WIN_UNLOCK); + return mpi_errno; + fn_fail: + goto fn_exit; + +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_win_get_info +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_win_get_info(MPIR_Win * win, MPIR_Info ** info_p_p) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_WIN_GET_INFO); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_WIN_GET_INFO); + + mpi_errno = MPIDI_CH4R_win_get_info(win, info_p_p); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_WIN_GET_INFO); + return mpi_errno; + fn_fail: + goto fn_exit; +} + + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_win_free +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_win_free(MPIR_Win ** win_ptr) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Errflag_t errflag = MPIR_ERR_NONE; + MPIR_Win *win = *win_ptr; + uint32_t window_instance; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_WIN_FREE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_WIN_FREE); + + MPIDI_CH4U_EPOCH_FREE_CHECK(win, mpi_errno, return mpi_errno); + + mpi_errno = MPIR_Barrier_impl(win->comm_ptr, &errflag); + + if (mpi_errno != MPI_SUCCESS) + goto fn_fail; + + window_instance = (uint32_t) (MPIDI_OFI_WIN(win).win_id >> 32); + + MPIDI_OFI_index_allocator_free(MPIDI_OFI_COMM(win->comm_ptr).win_id_allocator, window_instance); + MPIDI_OFI_map_erase(MPIDI_Global.win_map, MPIDI_OFI_WIN(win).win_id); + if (MPIDI_OFI_WIN(win).ep_nocmpl != MPIDI_OFI_EP_TX_CTR(0)) + MPIDI_OFI_CALL(fi_close(&MPIDI_OFI_WIN(win).ep_nocmpl->fid), epclose); + if (MPIDI_OFI_WIN(win).ep != MPIDI_OFI_EP_TX_RMA(0)) + MPIDI_OFI_CALL(fi_close(&MPIDI_OFI_WIN(win).ep->fid), epclose); + if (MPIDI_OFI_WIN(win).cmpl_cntr != MPIDI_Global.rma_cmpl_cntr) + MPIDI_OFI_CALL(fi_close(&MPIDI_OFI_WIN(win).cmpl_cntr->fid), cqclose); + MPIDI_OFI_CALL(fi_close(&MPIDI_OFI_WIN(win).mr->fid), mr_unreg); + if (MPIDI_OFI_WIN(win).winfo) { + MPL_free(MPIDI_OFI_WIN(win).winfo); + MPIDI_OFI_WIN(win).winfo = NULL; + } + + MPIDI_CH4R_win_finalize(win_ptr); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_WIN_FREE); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_win_fence +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_win_fence(int massert, MPIR_Win * win) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Errflag_t errflag = MPIR_ERR_NONE; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_WIN_FENCE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_WIN_FENCE); + + MPIDI_CH4U_EPOCH_FENCE_CHECK(win, mpi_errno, goto fn_fail); + + MPIDI_OFI_MPI_CALL_POP(MPIDI_Win_progress_fence(win)); + + MPIDI_CH4U_EPOCH_FENCE_EVENT(win, massert); + + /* + * We always make a barrier even if MPI_MODE_NOPRECEDE is specified. + * This is necessary because we no longer defer executions of RMA ops + * until synchronization calls as CH3 did. Otherwise, the code like + * this won't work correctly (cf. f77/rma/wingetf) + * + * Rank 0 Rank 1 + * ---- ---- + * Store to local mem in window + * MPI_Win_fence(MODE_NOPRECEDE) MPI_Win_fence(MODE_NOPRECEDE) + * MPI_Get(from rank 1) + */ + mpi_errno = MPIR_Barrier_impl(win->comm_ptr, &errflag); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_WIN_FENCE); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_win_create +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_win_create(void *base, + MPI_Aint length, + int disp_unit, + MPIR_Info * info, MPIR_Comm * comm_ptr, MPIR_Win ** win_ptr) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Errflag_t errflag = MPIR_ERR_NONE; + MPIR_Win *win; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_WIN_CREATE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_WIN_CREATE); + + mpi_errno = MPIDI_OFI_win_init(length, + disp_unit, + win_ptr, info, comm_ptr, MPI_WIN_FLAVOR_CREATE, MPI_WIN_UNIFIED); + + if (mpi_errno != MPI_SUCCESS) + goto fn_fail; + + win = *win_ptr; + win->base = base; + + mpi_errno = MPIDI_OFI_win_allgather(win, base, disp_unit); + + if (mpi_errno != MPI_SUCCESS) + goto fn_fail; + + mpi_errno = MPIR_Barrier_impl(comm_ptr, &errflag); + + if (mpi_errno != MPI_SUCCESS) + goto fn_fail; + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_WIN_CREATE); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_win_attach +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_win_attach(MPIR_Win * win, void *base, MPI_Aint size) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_WIN_ATTACH); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_WIN_ATTACH); + + MPIR_ERR_CHKANDSTMT((win->create_flavor != MPI_WIN_FLAVOR_DYNAMIC), mpi_errno, + MPI_ERR_RMA_FLAVOR, goto fn_fail, "**rmaflavor"); + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_WIN_ATTACH); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_win_allocate_shared +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_win_allocate_shared(MPI_Aint size, + int disp_unit, + MPIR_Info * info_ptr, + MPIR_Comm * comm_ptr, + void **base_ptr, MPIR_Win ** win_ptr) +{ + int i = 0, fd = -1, rc, first = 0, mpi_errno = MPI_SUCCESS; + MPIR_Errflag_t errflag = MPIR_ERR_NONE; + void *baseP = NULL; + MPIR_Win *win = NULL; + ssize_t total_size = 0LL; + MPI_Aint size_out = 0; + char shm_key[64]; + void *map_ptr; + MPIDI_CH4U_win_shared_info_t *shared_table = NULL; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_WIN_ALLOCATE_SHARED); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_WIN_ALLOCATE_SHARED); + + mpi_errno = MPIDI_OFI_win_init(size, disp_unit, win_ptr, info_ptr, comm_ptr, + MPI_WIN_FLAVOR_SHARED, MPI_WIN_UNIFIED); + if (mpi_errno != MPI_SUCCESS) + MPIR_ERR_POP(mpi_errno); + win = *win_ptr; + MPIDI_CH4U_WIN(win, shared_table) = + (MPIDI_CH4U_win_shared_info_t *) MPL_malloc(sizeof(MPIDI_CH4U_win_shared_info_t) * + comm_ptr->local_size); + shared_table = MPIDI_CH4U_WIN(win, shared_table); + + shared_table[comm_ptr->rank].size = size; + shared_table[comm_ptr->rank].disp_unit = disp_unit; + + mpi_errno = MPIR_Allgather_impl(MPI_IN_PLACE, + 0, + MPI_DATATYPE_NULL, + shared_table, + sizeof(MPIDI_CH4U_win_shared_info_t), + MPI_BYTE, comm_ptr, &errflag); + + if (mpi_errno != MPI_SUCCESS) + goto fn_fail; + + /* No allreduce here because this is a shared memory domain + * and should be a relatively small number of processes + * and a non performance sensitive API. + */ + for (i = 0; i < comm_ptr->local_size; i++) + total_size += shared_table[i].size; + + if (total_size == 0) + goto fn_zero; + + sprintf(shm_key, "/mpi-%X-%" PRIx64, MPIDI_Global.jobid, MPIDI_OFI_WIN(win).win_id); + + rc = shm_open(shm_key, O_CREAT | O_EXCL | O_RDWR, 0600); + first = (rc != -1); + + if (!first) { + rc = shm_open(shm_key, O_RDWR, 0); + + if (rc == -1) { + shm_unlink(shm_key); + MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_NO_MEM, goto fn_fail, "**nomem"); + } + } + + /* Make the addresses symmetric by using MAP_FIXED */ + size_t page_sz, mapsize; + + mapsize = MPIDI_CH4R_get_mapsize(total_size, &page_sz); + fd = rc; + rc = ftruncate(fd, mapsize); + + if (rc == -1) { + close(fd); + + if (first) + shm_unlink(shm_key); + + MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_NO_MEM, goto fn_fail, "**nomem"); + } + + if (comm_ptr->rank == 0) { + map_ptr = MPIDI_CH4R_generate_random_addr(mapsize); + map_ptr = mmap(map_ptr, mapsize, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_FIXED, fd, 0); + + if (map_ptr == NULL || map_ptr == MAP_FAILED) { + close(fd); + + if (first) + shm_unlink(shm_key); + + MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_NO_MEM, goto fn_fail, "**nomem"); + } + + mpi_errno = MPIR_Bcast_impl(&map_ptr, 1, MPI_UNSIGNED_LONG, 0, comm_ptr, &errflag); + + if (mpi_errno != MPI_SUCCESS) + goto fn_fail; + + MPIDI_CH4U_WIN(win, mmap_addr) = map_ptr; + MPIDI_CH4U_WIN(win, mmap_sz) = mapsize; + } + else { + mpi_errno = MPIR_Bcast_impl(&map_ptr, 1, MPI_UNSIGNED_LONG, 0, comm_ptr, &errflag); + + if (mpi_errno != MPI_SUCCESS) + goto fn_fail; + + rc = MPIDI_CH4R_check_maprange_ok(map_ptr, mapsize); + /* If we hit this assert, we need to iterate + * trying more addresses + */ + MPIR_Assert(rc == 1); + map_ptr = mmap(map_ptr, mapsize, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_FIXED, fd, 0); + MPIDI_CH4U_WIN(win, mmap_addr) = map_ptr; + MPIDI_CH4U_WIN(win, mmap_sz) = mapsize; + + if (map_ptr == NULL || map_ptr == MAP_FAILED) { + close(fd); + + if (first) + shm_unlink(shm_key); + + MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_NO_MEM, goto fn_fail, "**nomem"); + } + } + + /* Scan for my offset into the buffer */ + /* Could use exscan if this is expensive at scale */ + for (i = 0; i < comm_ptr->rank; i++) + size_out += shared_table[i].size; + + fn_zero: + + baseP = (size == 0) ? NULL : (void *) ((char *) map_ptr + size_out); + win->base = baseP; + win->size = size; + mpi_errno = MPIDI_OFI_win_allgather(win, baseP, disp_unit); + + if (mpi_errno != MPI_SUCCESS) + return mpi_errno; + + *(void **) base_ptr = (void *) win->base; + mpi_errno = MPIR_Barrier_impl(comm_ptr, &errflag); + + close(fd); + + if (first) + shm_unlink(shm_key); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_WIN_ALLOCATE_SHARED); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_win_detach +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_win_detach(MPIR_Win * win, const void *base) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_WIN_DETACH); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_WIN_DETACH); + MPIR_ERR_CHKANDSTMT((win->create_flavor != MPI_WIN_FLAVOR_DYNAMIC), mpi_errno, + MPI_ERR_RMA_FLAVOR, goto fn_fail, "**rmaflavor"); + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_WIN_DETACH); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_win_shared_query +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_win_shared_query(MPIR_Win * win, + int rank, + MPI_Aint * size, int *disp_unit, void *baseptr) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_WIN_SHARED_QUERY); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_WIN_SHARED_QUERY); + int offset = rank; + int i; + uintptr_t base = (uintptr_t) MPIDI_CH4U_WIN(win, mmap_addr); + + MPIDI_CH4U_win_shared_info_t *shared_table = MPIDI_CH4U_WIN(win, shared_table); + + if (rank < 0) + offset = 0; + + *size = shared_table[offset].size; + *disp_unit = shared_table[offset].disp_unit; + if (*size > 0) { + for (i = 0; i < offset; i++) + base += shared_table[i].size; + *(void **) baseptr = (void *) base; + } + else + *(void **) baseptr = NULL; + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_WIN_SHARED_QUERY); + return mpi_errno; +} + + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_win_allocate +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_win_allocate(MPI_Aint size, + int disp_unit, + MPIR_Info * info, + MPIR_Comm * comm, void *baseptr, MPIR_Win ** win_ptr) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Errflag_t errflag = MPIR_ERR_NONE; + void *baseP; + MPIR_Win *win; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_WIN_ALLOCATE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_WIN_ALLOCATE); + + mpi_errno = MPIDI_OFI_win_init(size, disp_unit, win_ptr, info, comm, + MPI_WIN_FLAVOR_ALLOCATE, MPI_WIN_UNIFIED); + + if (mpi_errno != MPI_SUCCESS) + goto fn_fail; + + mpi_errno = MPIDI_CH4R_get_symmetric_heap(size, comm, &baseP, *win_ptr); + + if (mpi_errno != MPI_SUCCESS) + goto fn_fail; + + win = *win_ptr; + win->base = baseP; + mpi_errno = MPIDI_OFI_win_allgather(win, baseP, disp_unit); + + if (mpi_errno != MPI_SUCCESS) + goto fn_fail; + + *(void **) baseptr = (void *) win->base; + mpi_errno = MPIR_Barrier_impl(comm, &errflag); + + if (mpi_errno != MPI_SUCCESS) + goto fn_fail; + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_WIN_ALLOCATE); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_win_flush +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_win_flush(int rank, MPIR_Win * win) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_WIN_FLUSH); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_WIN_FLUSH); + + MPIDI_CH4U_EPOCH_LOCK_CHECK(win, mpi_errno, goto fn_fail); + + MPIDI_OFI_MPI_CALL_POP(MPIDI_Win_progress_fence(win)); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_WIN_FLUSH); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_win_flush_local_all +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_win_flush_local_all(MPIR_Win * win) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_WIN_FLUSH_LOCAL_ALL); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_WIN_FLUSH_LOCAL_ALL); + + MPIDI_CH4U_EPOCH_LOCK_CHECK(win, mpi_errno, goto fn_fail); + MPIDI_OFI_MPI_CALL_POP(MPIDI_Win_progress_fence(win)); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_WIN_FLUSH_LOCAL_ALL); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_win_unlock_all +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_win_unlock_all(MPIR_Win * win) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_WIN_UNLOCK_ALL); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_WIN_UNLOCK_ALL); + int i; + MPIDI_CH4U_win_lock_info *lockQ; + + MPIDI_CH4U_EPOCH_ORIGIN_CHECK(win, MPIDI_CH4U_EPOTYPE_LOCK_ALL, mpi_errno, return mpi_errno); + + MPIDI_OFI_MPI_CALL_POP(MPIDI_Win_progress_fence(win)); + + MPIR_Assert(MPIDI_CH4U_WIN(win, lockQ) != NULL); + + lockQ = (MPIDI_CH4U_win_lock_info *) MPIDI_CH4U_WIN(win, lockQ); + + for (i = 0; i < win->comm_ptr->local_size; i++) { + MPIDI_OFI_win_control_t msg; + lockQ[i].done = 0; + lockQ[i].peer = i; + lockQ[i].win = win; + msg.type = MPIDI_OFI_CTRL_UNLOCKALL; + mpi_errno = MPIDI_OFI_do_control_win(&msg, lockQ[i].peer, win, 1, 1); + + if (mpi_errno != MPI_SUCCESS) + MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, goto fn_fail, "**rmasync"); + + if (MPIDI_CH4U_WIN(win, sync).lock.remote.allLocked == 1) + lockQ[i].done = 1; + } + + MPIDI_OFI_PROGRESS_WHILE(MPIDI_CH4U_WIN(win, sync).lock.remote.allLocked); + + MPIDI_CH4U_WIN(win, sync).origin_epoch_type = MPIDI_CH4U_EPOTYPE_NONE; + MPIDI_CH4U_WIN(win, sync).target_epoch_type = MPIDI_CH4U_EPOTYPE_NONE; + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_WIN_UNLOCK_ALL); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_win_create_dynamic +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_win_create_dynamic(MPIR_Info * info, + MPIR_Comm * comm, MPIR_Win ** win_ptr) +{ + int mpi_errno = MPI_SUCCESS; + int rc = MPI_SUCCESS; + MPIR_Errflag_t errflag = MPIR_ERR_NONE; + MPIR_Win *win; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_WIN_CREATE_DYNAMIC); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_WIN_CREATE_DYNAMIC); + + rc = MPIDI_OFI_win_init((uintptr_t) UINTPTR_MAX - (uintptr_t) MPI_BOTTOM, + 1, win_ptr, info, comm, MPI_WIN_FLAVOR_DYNAMIC, MPI_WIN_UNIFIED); + + if (rc != MPI_SUCCESS) + goto fn_fail; + + win = *win_ptr; + win->base = MPI_BOTTOM; + + rc = MPIDI_OFI_win_allgather(win, win->base, 1); + + if (rc != MPI_SUCCESS) + goto fn_fail; + + mpi_errno = MPIR_Barrier_impl(comm, &errflag); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_WIN_CREATE_DYNAMIC); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_win_flush_local +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_win_flush_local(int rank, MPIR_Win * win) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_WIN_FLUSH_LOCAL); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_WIN_FLUSH_LOCAL); + + MPIDI_CH4U_EPOCH_LOCK_CHECK(win, mpi_errno, goto fn_fail); + + MPIDI_OFI_MPI_CALL_POP(MPIDI_Win_progress_fence(win)); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_WIN_FLUSH_LOCAL); + return mpi_errno; + fn_fail: + goto fn_exit; +} + + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_win_sync +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_win_sync(MPIR_Win * win) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_WIN_SYNC); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_WIN_SYNC); + + MPIDI_CH4U_EPOCH_LOCK_CHECK(win, mpi_errno, goto fn_fail); + + OPA_read_write_barrier(); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_WIN_SYNC); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_win_flush_all +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_win_flush_all(MPIR_Win * win) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_WIN_FLUSH_ALL); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_WIN_FLUSH_ALL); + + MPIDI_CH4U_EPOCH_LOCK_CHECK(win, mpi_errno, goto fn_fail); + MPIDI_OFI_MPI_CALL_POP(MPIDI_Win_progress_fence(win)); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_WIN_FLUSH_ALL); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_win_lock_all +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_win_lock_all(int assert, MPIR_Win * win) +{ + int mpi_errno = MPI_SUCCESS; + MPIDI_CH4U_win_lock_info *lockQ; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_WIN_LOCK_ALL); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_WIN_LOCK_ALL); + + MPIDI_CH4U_EPOCH_CHECK_TYPE(win, mpi_errno, goto fn_fail); + + int size; + size = win->comm_ptr->local_size; + + if (!MPIDI_CH4U_WIN(win, lockQ)) { + MPIDI_CH4U_WIN(win, lockQ) = + (MPIDI_CH4U_win_lock_info *) MPL_calloc(size, sizeof(MPIDI_CH4U_win_lock_info)); + MPIR_Assert(MPIDI_CH4U_WIN(win, lockQ) != NULL); + } + + lockQ = MPIDI_CH4U_WIN(win, lockQ); + int i; + + for (i = 0; i < size; i++) { + MPIDI_OFI_win_control_t msg; + + lockQ[i].done = 0; + lockQ[i].peer = i; + lockQ[i].win = win; + lockQ[i].lock_type = MPI_LOCK_SHARED; + + msg.type = MPIDI_OFI_CTRL_LOCKALLREQ; + msg.lock_type = MPI_LOCK_SHARED; + mpi_errno = MPIDI_OFI_do_control_win(&msg, lockQ[i].peer, lockQ[i].win, 1, 1); + + if (mpi_errno != MPI_SUCCESS) + MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, goto fn_fail, "**rmasync"); + + if (MPIDI_CH4U_WIN(win, sync).lock.remote.allLocked == 1) + lockQ[i].done = 1; + } + + MPIDI_OFI_PROGRESS_WHILE(size != (int) MPIDI_CH4U_WIN(win, sync).lock.remote.allLocked); + + MPIDI_CH4U_WIN(win, sync).origin_epoch_type = MPIDI_CH4U_EPOTYPE_LOCK_ALL; + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_WIN_LOCK_ALL); + return mpi_errno; + fn_fail: + goto fn_exit; +} + + +#endif /* NETMOD_OFI_WIN_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/ofi/subconfigure.m4 b/src/mpid/ch4/netmod/ofi/subconfigure.m4 new file mode 100644 index 0000000..35e4a03 --- /dev/null +++ b/src/mpid/ch4/netmod/ofi/subconfigure.m4 @@ -0,0 +1,153 @@ +[#] start of __file__ +dnl MPICH_SUBCFG_AFTER=src/mpid/ch4 + +AC_DEFUN([PAC_SUBCFG_PREREQ_]PAC_SUBCFG_AUTO_SUFFIX,[ + AM_COND_IF([BUILD_CH4],[ + for net in $ch4_netmods ; do + AS_CASE([$net],[ofi],[build_ch4_netmod_ofi=yes]) + if test $net = "ofi" ; then + AC_DEFINE(HAVE_CH4_NETMOD_OFI,1,[OFI netmod is built]) + AC_DEFINE(MPIDI_BUILD_CH4_LOCALITY_INFO, 1, [CH4 should build locality info]) + fi + done + + AC_ARG_WITH(ch4-netmod-ofi-args, + [ --with-ch4-netmod-ofi-args=arg1:arg2:arg3 + CH4 OFI netmod arguments: + scalable-endpoints - Use OFI scalable endpoint mode + av-table - Use OFI AV table (logical addressing mode). Default is av-map mode + mr-basic - USE OFI MR_BASIC mode. Default is MR_SCALABLE mode. + direct-provider - USE OFI FI_DIRECT to compile in a single OFI direct provider + no-tagged - Do not use OFI fi_tagged interfaces. + no-data - Disable immediate data field + no-stx-rma - Disable per-window EP & counter for RMA + ], + [ofi_netmod_args=$withval], + [ofi_netmod_args=]) + +dnl Parse the device arguments + SAVE_IFS=$IFS + IFS=':' + args_array=$ofi_netmod_args + do_scalable_endpoints=false + do_direct_provider=false + do_av_table=false + do_tagged=true + do_data=true + do_stx_rma=true + do_mr_scalable=true + echo "Parsing Arguments for OFI Netmod" + for arg in $args_array; do + case ${arg} in + scalable-endpoints) + do_scalable_endpoints=true + echo " ---> CH4::OFI Provider : $arg" + ;; + av_table) + do_av_table=true + echo " ---> CH4::OFI Provider AV table : $arg" + ;; + direct-provider) + do_direct_provider=true + echo " ---> CH4::OFI Direct OFI Provider requested : $arg" + ;; + no-tagged) + do_tagged=false + echo " ---> CH4::OFI Disable fi_tagged interfaces : $arg" + ;; + no-data) + do_data=false + echo " ---> CH4::OFI Disable immediate data field : $arg" + ;; + no-stx-rma) + do_stx_rma=false + echo " ---> CH4::OFI Disable per-window EP & counter for RMA : $arg" + ;; + mr-basic) + do_mr_scalable=false + echo " ---> CH4::OFI Switching to MR_BASIC mode : $arg" + ;; + esac + done + IFS=$SAVE_IFS + + if [test "$do_scalable_endpoints" = "true"]; then + AC_MSG_NOTICE([Enabling OFI netmod scalable endpoints]) + PAC_APPEND_FLAG([-DMPIDI_OFI_CONFIG_USE_SCALABLE_ENDPOINTS], [CPPFLAGS]) + fi + + if [test "$do_av_table" = "true"]; then + AC_MSG_NOTICE([Enabling OFI netmod AV table]) + PAC_APPEND_FLAG([-DMPIDI_OFI_CONFIG_USE_AV_TABLE], [CPPFLAGS]) + fi + + if [test "$do_direct_provider" = "true"]; then + AC_MSG_NOTICE([Enabling OFI netmod direct provider]) + PAC_APPEND_FLAG([-DFABRIC_DIRECT],[CPPFLAGS]) + fi + + if [test "$do_tagged" = "true"]; then + AC_DEFINE([USE_OFI_TAGGED], [1], [Define to use fi_tagged interfaces]) + AC_MSG_NOTICE([Enabling fi_tagged interfaces]) + fi + + if [test "$do_data" = "true"]; then + AC_DEFINE([USE_OFI_IMMEDIATE_DATA], [1], [Define to use immediate data field]) + AC_MSG_NOTICE([Enabling immediate data field]) + fi + + if [test "$do_stx_rma" = "true"]; then + AC_DEFINE([USE_OFI_STX_RMA], [1], [Define to use per-window EP & counter]) + AC_MSG_NOTICE([Enabling per-window EP & counter]) + fi + + if [test "$do_mr_scalable" = "true"]; then + AC_DEFINE([USE_OFI_MR_SCALABLE], [1], [Define to use MR_SCALABLE]) + AC_MSG_NOTICE([Enabling MR_SCALABLE]) + fi +]) + AM_CONDITIONAL([BUILD_CH4_NETMOD_OFI],[test "X$build_ch4_netmod_ofi" = "Xyes"]) +])dnl + +AC_DEFUN([PAC_SUBCFG_BODY_]PAC_SUBCFG_AUTO_SUFFIX,[ +AM_COND_IF([BUILD_CH4_NETMOD_OFI],[ + AC_MSG_NOTICE([RUNNING CONFIGURE FOR ch4:ofi]) + + ofisrcdir="" + AC_SUBST([ofisrcdir]) + ofilib="" + AC_SUBST([ofilib]) + + PAC_SET_HEADER_LIB_PATH(libfabric) + PAC_PUSH_FLAG(LIBS) + PAC_CHECK_HEADER_LIB([rdma/fabric.h], [fabric], [fi_getinfo], [have_libfabric=yes], [have_libfabric=no]) + PAC_POP_FLAG(LIBS) + if test "${have_libfabric}" = "yes" ; then + AC_MSG_NOTICE([CH4 OFI Netmod: Using an external libfabric]) + PAC_APPEND_FLAG([-lfabric],[WRAPPER_LIBS]) + elif test ! -z "${with_libfabric}" ; then + AC_MSG_ERROR([Provided libfabric installation (--with-libfabric=${with_libfabric}) could not be configured.]) + else + # fallback to embedded libfabric + AC_MSG_NOTICE([CH4 OFI Netmod: Using an embedded libfabric]) + PAC_CONFIG_SUBDIR_ARGS([src/mpid/ch4/netmod/ofi/libfabric],[],[],[AC_MSG_ERROR(libfabric configure failed)]) + PAC_APPEND_FLAG([-I${master_top_builddir}/src/mpid/ch4/netmod/ofi/libfabric/include], [CPPFLAGS]) + PAC_APPEND_FLAG([-I${use_top_srcdir}/src/mpid/ch4/netmod/ofi/libfabric/include], [CPPFLAGS]) + + ofisrcdir="${master_top_builddir}/src/mpid/ch4/netmod/ofi/libfabric" + ofilib="src/mpid/ch4/netmod/ofi/libfabric/src/libfabric.la" + fi + + case $host_os in + darwin* ) + PAC_APPEND_FLAG([-lstdc++ -ldl],[WRAPPER_LIBS]) + ;; + *) + PAC_APPEND_FLAG([-lstdc++ -ldl -lrt],[WRAPPER_LIBS]) + ;; + esac + +])dnl end AM_COND_IF(BUILD_CH4_NETMOD_OFI,...) +])dnl end _BODY + +[#] end of __file__ diff --git a/src/mpid/ch4/netmod/ofi/util.c b/src/mpid/ch4/netmod/ofi/util.c new file mode 100644 index 0000000..6c30026 --- /dev/null +++ b/src/mpid/ch4/netmod/ofi/util.c @@ -0,0 +1,822 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ + +#include <mpidimpl.h> +#include "ofi_impl.h" +#include "ofi_events.h" + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_handle_cq_error_util +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +int MPIDI_OFI_handle_cq_error_util(ssize_t ret) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_OFI_HANDLE_CQ_ERROR); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_OFI_HANDLE_CQ_ERROR); + + mpi_errno = MPIDI_OFI_handle_cq_error(ret); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_HANDLE_CQ_ERROR); + return mpi_errno; +} + +int MPIDI_OFI_progress_test_no_inline() +{ + return MPIDI_Progress_test(); +} + +typedef struct { + uint64_t key; + void *value; + MPL_UT_hash_handle hh; /* makes this structure hashable */ +} MPIDI_OFI_map_entry_t; + +typedef struct MPIDI_OFI_map_t { + MPIDI_OFI_map_entry_t *head; + +} MPIDI_OFI_map_t; + +void MPIDI_OFI_map_create(void **out_map) +{ + MPIDI_OFI_map_t *map; + map = MPL_malloc(sizeof(MPIDI_OFI_map_t)); + MPIR_Assert(map != NULL); + map->head = NULL; + *out_map = map; +} + +void MPIDI_OFI_map_destroy(void *in_map) +{ + MPID_THREAD_CS_ENTER(POBJ, MPIDI_OFI_THREAD_UTIL_MUTEX); + MPIDI_OFI_map_t *map = in_map; + MPL_HASH_CLEAR(hh, map->head); + MPL_free(map); + MPID_THREAD_CS_EXIT(POBJ, MPIDI_OFI_THREAD_UTIL_MUTEX); +} + +void MPIDI_OFI_map_set(void *in_map, uint64_t id, void *val) +{ + MPIDI_OFI_map_t *map; + MPIDI_OFI_map_entry_t *map_entry; + MPID_THREAD_CS_ENTER(POBJ, MPIDI_OFI_THREAD_UTIL_MUTEX); + map = (MPIDI_OFI_map_t *) in_map; + map_entry = MPL_malloc(sizeof(MPIDI_OFI_map_entry_t)); + MPIR_Assert(map_entry != NULL); + map_entry->key = id; + map_entry->value = val; + MPL_HASH_ADD(hh, map->head, key, sizeof(uint64_t), map_entry); + MPID_THREAD_CS_EXIT(POBJ, MPIDI_OFI_THREAD_UTIL_MUTEX); +} + +void MPIDI_OFI_map_erase(void *in_map, uint64_t id) +{ + MPIDI_OFI_map_t *map; + MPIDI_OFI_map_entry_t *map_entry; + MPID_THREAD_CS_ENTER(POBJ, MPIDI_OFI_THREAD_UTIL_MUTEX); + map = (MPIDI_OFI_map_t *) in_map; + MPL_HASH_FIND(hh, map->head, &id, sizeof(uint64_t), map_entry); + MPIR_Assert(map_entry != NULL); + MPL_HASH_DELETE(hh, map->head, map_entry); + MPL_free(map_entry); + MPID_THREAD_CS_EXIT(POBJ, MPIDI_OFI_THREAD_UTIL_MUTEX); +} + +void *MPIDI_OFI_map_lookup(void *in_map, uint64_t id) +{ + void *rc; + MPIDI_OFI_map_t *map; + MPIDI_OFI_map_entry_t *map_entry; + + MPID_THREAD_CS_ENTER(POBJ, MPIDI_OFI_THREAD_UTIL_MUTEX); + map = (MPIDI_OFI_map_t *) in_map; + MPL_HASH_FIND(hh, map->head, &id, sizeof(uint64_t), map_entry); + if (map_entry == NULL) + rc = MPIDI_OFI_MAP_NOT_FOUND; + else + rc = map_entry->value; + MPID_THREAD_CS_EXIT(POBJ, MPIDI_OFI_THREAD_UTIL_MUTEX); + return rc; +} + +typedef struct MPIDI_OFI_index_allocator_t { + int chunk_size; + int num_ints; + int start; + int last_free_index; + uint64_t *bitmask; +} MPIDI_OFI_index_allocator_t; + +void MPIDI_OFI_index_allocator_create(void **indexmap, int start) +{ + MPIDI_OFI_index_allocator_t *allocator; + MPID_THREAD_CS_ENTER(POBJ, MPIDI_OFI_THREAD_UTIL_MUTEX); + allocator = MPL_malloc(sizeof(MPIDI_OFI_index_allocator_t)); + allocator->chunk_size = 128; + allocator->num_ints = allocator->chunk_size; + allocator->start = start; + allocator->last_free_index = 0; + allocator->bitmask = MPL_malloc(sizeof(uint64_t) * allocator->num_ints); + memset(allocator->bitmask, 0xFF, sizeof(uint64_t) * allocator->num_ints); + assert(allocator != NULL); + *indexmap = allocator; + MPID_THREAD_CS_EXIT(POBJ, MPIDI_OFI_THREAD_UTIL_MUTEX); +} + +#define MPIDI_OFI_INDEX_CALC(val,nval,shift,mask) \ + if ((val & mask) == 0) { \ + val >>= shift##ULL; \ + nval += shift; \ + } +int MPIDI_OFI_index_allocator_alloc(void *indexmap) +{ + int i; + MPIDI_OFI_index_allocator_t *allocator = indexmap; + MPID_THREAD_CS_ENTER(POBJ, MPIDI_OFI_THREAD_UTIL_MUTEX); + for (i = allocator->last_free_index; i < allocator->num_ints; i++) { + if (allocator->bitmask[i]) { + register uint64_t val, nval; + val = allocator->bitmask[i]; + nval = 2; + MPIDI_OFI_INDEX_CALC(val, nval, 32, 0xFFFFFFFFULL); + MPIDI_OFI_INDEX_CALC(val, nval, 16, 0xFFFFULL); + MPIDI_OFI_INDEX_CALC(val, nval, 8, 0xFFULL); + MPIDI_OFI_INDEX_CALC(val, nval, 4, 0xFULL); + MPIDI_OFI_INDEX_CALC(val, nval, 2, 0x3ULL); + nval -= val & 0x1ULL; + allocator->bitmask[i] &= ~(0x1ULL << (nval - 1)); + allocator->last_free_index = i; + MPID_THREAD_CS_EXIT(POBJ, MPIDI_OFI_THREAD_UTIL_MUTEX); + return i * sizeof(uint64_t) * 8 + (nval - 1) + allocator->start; + } + if (i == allocator->num_ints - 1) { + allocator->num_ints += allocator->chunk_size; + allocator->bitmask = MPL_realloc(allocator->bitmask, + sizeof(uint64_t) * allocator->num_ints); + memset(&allocator->bitmask[i + 1], 0xFF, sizeof(uint64_t) * allocator->chunk_size); + } + } + MPID_THREAD_CS_EXIT(POBJ, MPIDI_OFI_THREAD_UTIL_MUTEX); + return -1; +} + +void MPIDI_OFI_index_allocator_free(void *indexmap, int index) +{ + int int_index, bitpos, numbits; + MPIDI_OFI_index_allocator_t *allocator = indexmap; + MPID_THREAD_CS_ENTER(POBJ, MPIDI_OFI_THREAD_UTIL_MUTEX); + numbits = sizeof(uint64_t) * 8; + int_index = (index + 1 - allocator->start) / numbits; + bitpos = (index - allocator->start) % numbits; + + allocator->last_free_index = MPL_MIN(int_index, allocator->last_free_index); + allocator->bitmask[int_index] |= (0x1ULL << bitpos); + MPID_THREAD_CS_EXIT(POBJ, MPIDI_OFI_THREAD_UTIL_MUTEX); +} + +void MPIDI_OFI_index_allocator_destroy(void *indexmap) +{ + MPIDI_OFI_index_allocator_t *allocator; + MPID_THREAD_CS_ENTER(POBJ, MPIDI_OFI_THREAD_UTIL_MUTEX); + allocator = (MPIDI_OFI_index_allocator_t *) indexmap; + MPL_free(allocator->bitmask); + MPL_free(allocator); + MPID_THREAD_CS_EXIT(POBJ, MPIDI_OFI_THREAD_UTIL_MUTEX); +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_OFI_win_lock_advance +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_OFI_win_lock_advance(MPIR_Win * win) +{ + int mpi_errno = MPI_SUCCESS; + struct MPIDI_CH4U_win_sync_lock *slock = &MPIDI_CH4U_WIN(win, sync).lock; + struct MPIDI_CH4U_win_queue *q = &slock->local.requested; + + if ((q->head != NULL) && + ((slock->local.count == 0) || + ((slock->local.type == MPI_LOCK_SHARED) && (q->head->type == MPI_LOCK_SHARED) +) +) +) { + struct MPIDI_CH4U_win_lock *lock = q->head; + q->head = lock->next; + + if (q->head == NULL) + q->tail = NULL; + + ++slock->local.count; + slock->local.type = lock->type; + + if (lock->mtype == MPIDI_OFI_REQUEST_LOCK) { + MPIDI_OFI_win_control_t info; + info.type = MPIDI_OFI_CTRL_LOCKACK; + mpi_errno = MPIDI_OFI_do_control_win(&info, lock->rank, win, 1, 0); + + if (mpi_errno != MPI_SUCCESS) + MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, goto fn_fail, "**rmasync"); + + } + else if (lock->mtype == MPIDI_OFI_REQUEST_LOCKALL) { + MPIDI_OFI_win_control_t info; + info.type = MPIDI_OFI_CTRL_LOCKALLACK; + mpi_errno = MPIDI_OFI_do_control_win(&info, lock->rank, win, 1, 0); + } + else + MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, goto fn_fail, "**rmasync"); + + MPL_free(lock); + mpi_errno = MPIDI_OFI_win_lock_advance(win); + + if (mpi_errno != MPI_SUCCESS) + MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, goto fn_fail, "**rmasync"); + } + + fn_exit: + return mpi_errno; + fn_fail: + goto fn_exit; +} + +static inline int MPIDI_OFI_win_lock_request_proc(const MPIDI_OFI_win_control_t * info, + MPIR_Win * win, unsigned peer) +{ + int mpi_errno; + struct MPIDI_CH4U_win_lock *lock = + (struct MPIDI_CH4U_win_lock *) MPL_calloc(1, sizeof(struct MPIDI_CH4U_win_lock)); + + if (info->type == MPIDI_OFI_CTRL_LOCKREQ) + lock->mtype = MPIDI_OFI_REQUEST_LOCK; + else if (info->type == MPIDI_OFI_CTRL_LOCKALLREQ) + lock->mtype = MPIDI_OFI_REQUEST_LOCKALL; + + lock->rank = info->origin_rank; + lock->type = info->lock_type; + struct MPIDI_CH4U_win_queue *q = &MPIDI_CH4U_WIN(win, sync).lock.local.requested; + MPIR_Assert((q->head != NULL) ^ (q->tail == NULL)); + + if (q->tail == NULL) + q->head = lock; + else + q->tail->next = lock; + + q->tail = lock; + + mpi_errno = MPIDI_OFI_win_lock_advance(win); + if (mpi_errno != MPI_SUCCESS) + MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, goto fn_fail, "**rmasync"); + + fn_exit: + return mpi_errno; + fn_fail: + goto fn_exit; +} + +static inline void MPIDI_OFI_win_lock_ack_proc(const MPIDI_OFI_win_control_t * info, + MPIR_Win * win, unsigned peer) +{ + if (info->type == MPIDI_OFI_CTRL_LOCKACK) + MPIDI_CH4U_WIN(win, sync).lock.remote.locked = 1; + else if (info->type == MPIDI_OFI_CTRL_LOCKALLACK) + MPIDI_CH4U_WIN(win, sync).lock.remote.allLocked += 1; +} + + +static inline int MPIDI_OFI_win_unlock_proc(const MPIDI_OFI_win_control_t * info, + MPIR_Win * win, unsigned peer) +{ + int mpi_errno; + --MPIDI_CH4U_WIN(win, sync).lock.local.count; + MPIR_Assert((int) MPIDI_CH4U_WIN(win, sync).lock.local.count >= 0); + mpi_errno = MPIDI_OFI_win_lock_advance(win); + if (mpi_errno != MPI_SUCCESS) + MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, goto fn_fail, "**rmasync"); + + MPIDI_OFI_win_control_t new_info; + new_info.type = MPIDI_OFI_CTRL_UNLOCKACK; + mpi_errno = MPIDI_OFI_do_control_win(&new_info, peer, win, 1, 0); + if (mpi_errno != MPI_SUCCESS) + MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, goto fn_fail, "**rmasync"); + fn_exit: + return mpi_errno; + fn_fail: + goto fn_exit; +} + +static inline void MPIDI_OFI_win_complete_proc(const MPIDI_OFI_win_control_t * info, + MPIR_Win * win, unsigned peer) +{ + ++MPIDI_CH4U_WIN(win, sync).sc.count; +} + +static inline void MPIDI_OFI_win_post_proc(const MPIDI_OFI_win_control_t * info, + MPIR_Win * win, unsigned peer) +{ + ++MPIDI_CH4U_WIN(win, sync).pw.count; +} + + +static inline void MPIDI_OFI_win_unlock_done_proc(const MPIDI_OFI_win_control_t * info, + MPIR_Win * win, unsigned peer) +{ + if (MPIDI_CH4U_WIN(win, sync).origin_epoch_type == MPIDI_CH4U_EPOTYPE_LOCK) + MPIDI_CH4U_WIN(win, sync).lock.remote.locked = 0; + else if (MPIDI_CH4U_WIN(win, sync).origin_epoch_type == MPIDI_CH4U_EPOTYPE_LOCK_ALL) { + MPIR_Assert((int) MPIDI_CH4U_WIN(win, sync).lock.remote.allLocked > 0); + MPIDI_CH4U_WIN(win, sync).lock.remote.allLocked -= 1; + } + else + MPIR_Assert(0); + +} + +static inline void MPIDI_OFI_get_huge_cleanup(MPIDI_OFI_send_control_t * info) +{ + MPIDI_OFI_huge_recv_t *recv; + MPIR_Comm *comm_ptr; + uint64_t mapid; + /* Look up the communicator */ + mapid = ((uint64_t) info->endpoint_id << 32) | info->comm_id; + comm_ptr = MPIDI_CH4U_context_id_to_comm(mapid); + /* Look up the per destination receive queue object */ + recv = + (MPIDI_OFI_huge_recv_t *) MPIDI_OFI_map_lookup(MPIDI_OFI_COMM(comm_ptr).huge_recv_counters, + info->origin_rank); + MPIDI_OFI_map_erase(MPIDI_OFI_COMM(comm_ptr).huge_recv_counters, info->origin_rank); + MPL_free(recv); +} + +static inline void MPIDI_OFI_get_huge(MPIDI_OFI_send_control_t * info) +{ + MPIDI_OFI_huge_recv_t *recv; + MPIR_Comm *comm_ptr; + /* Look up the communicator */ + comm_ptr = MPIDI_CH4U_context_id_to_comm(info->comm_id); + /* Look up the per destination receive queue object */ + recv = + (MPIDI_OFI_huge_recv_t *) MPIDI_OFI_map_lookup(MPIDI_OFI_COMM(comm_ptr).huge_recv_counters, + info->origin_rank); + if (recv == MPIDI_OFI_MAP_NOT_FOUND) { + recv = (MPIDI_OFI_huge_recv_t *) MPL_calloc(sizeof(*recv), 1); + MPIDI_OFI_map_set(MPIDI_OFI_COMM(comm_ptr).huge_recv_counters, info->origin_rank, recv); + } + recv->event_id = MPIDI_OFI_EVENT_GET_HUGE; + recv->cur_offset = MPIDI_Global.max_send; + recv->remote_info = *info; + recv->comm_ptr = comm_ptr; + MPIDI_OFI_get_huge_event(NULL, (MPIR_Request *) recv); +} + +int MPIDI_OFI_control_handler(void *am_hdr, + void **data, + size_t * data_sz, + int *is_contig, + MPIDI_NM_am_completion_handler_fn * cmpl_handler_fn, + MPIR_Request ** req) +{ + int senderrank; + int mpi_errno = MPI_SUCCESS; + void *buf = am_hdr; + MPIDI_OFI_win_control_t *control = (MPIDI_OFI_win_control_t *) buf; + *req = NULL; + *cmpl_handler_fn = NULL; + + switch (control->type) { + case MPIDI_OFI_CTRL_HUGEACK:{ + MPIDI_OFI_send_control_t *ctrlsend = (MPIDI_OFI_send_control_t *) buf; + MPIDI_OFI_dispatch_function(NULL, ctrlsend->ackreq, 0); + goto fn_exit; + } + break; + + case MPIDI_OFI_CTRL_HUGE:{ + MPIDI_OFI_send_control_t *ctrlsend = (MPIDI_OFI_send_control_t *) buf; + MPIDI_OFI_get_huge(ctrlsend); + goto fn_exit; + } + break; + + case MPIDI_OFI_CTRL_HUGE_CLEANUP:{ + MPIDI_OFI_send_control_t *ctrlsend = (MPIDI_OFI_send_control_t *) buf; + MPIDI_OFI_get_huge_cleanup(ctrlsend); + goto fn_exit; + } + break; + } + + MPIR_Win *win; + senderrank = control->origin_rank; + win = (MPIR_Win *) MPIDI_OFI_map_lookup(MPIDI_Global.win_map, control->win_id); + MPIR_Assert(win != MPIDI_OFI_MAP_NOT_FOUND); + + switch (control->type) { + case MPIDI_OFI_CTRL_LOCKREQ: + case MPIDI_OFI_CTRL_LOCKALLREQ: + mpi_errno = MPIDI_OFI_win_lock_request_proc(control, win, senderrank); + break; + + case MPIDI_OFI_CTRL_LOCKACK: + case MPIDI_OFI_CTRL_LOCKALLACK: + MPIDI_OFI_win_lock_ack_proc(control, win, senderrank); + break; + + case MPIDI_OFI_CTRL_UNLOCK: + case MPIDI_OFI_CTRL_UNLOCKALL: + mpi_errno = MPIDI_OFI_win_unlock_proc(control, win, senderrank); + break; + + case MPIDI_OFI_CTRL_UNLOCKACK: + case MPIDI_OFI_CTRL_UNLOCKALLACK: + MPIDI_OFI_win_unlock_done_proc(control, win, senderrank); + break; + + case MPIDI_OFI_CTRL_COMPLETE: + MPIDI_OFI_win_complete_proc(control, win, senderrank); + break; + + case MPIDI_OFI_CTRL_POST: + MPIDI_OFI_win_post_proc(control, win, senderrank); + break; + + default: + fprintf(stderr, "Bad control type: 0x%08x %d\n", control->type, control->type); + MPIR_Assert(0); + } + + fn_exit: + return mpi_errno; +} + + +/* MPI Datatype Processing for RMA */ +#define isS_INT(x) ((x)==MPI_INTEGER || \ + (x) == MPI_INT32_T || (x) == MPI_INTEGER4 || \ + (x) == MPI_INT) +#define isUS_INT(x) ((x) == MPI_UINT32_T || (x) == MPI_UNSIGNED) +#define isS_SHORT(x) ((x) == MPI_SHORT || (x) == MPI_INT16_T || \ + (x) == MPI_INTEGER2) +#define isUS_SHORT(x) ((x) == MPI_UNSIGNED_SHORT || (x) == MPI_UINT16_T) +#define isS_CHAR(x) ((x) == MPI_SIGNED_CHAR || (x) == MPI_INT8_T || \ + (x) == MPI_INTEGER1 || (x) == MPI_CHAR) +#define isUS_CHAR(x) ((x) == MPI_BYTE || \ + (x) == MPI_UNSIGNED_CHAR || (x) == MPI_UINT8_T) +#define isS_LONG(x) ((x) == MPI_LONG || (x) == MPI_AINT) +#define isUS_LONG(x) ((x) == MPI_UNSIGNED_LONG) +#define isS_LONG_LONG(x) ((x) == MPI_INT64_T || (x) == MPI_OFFSET || \ + (x) == MPI_INTEGER8 || (x) == MPI_LONG_LONG || \ + (x) == MPI_LONG_LONG_INT || (x) == MPI_COUNT) +#define isUS_LONG_LONG(x) ((x) == MPI_UINT64_T || (x) == MPI_UNSIGNED_LONG_LONG) +#define isFLOAT(x) ((x) == MPI_FLOAT || (x) == MPI_REAL) +#define isDOUBLE(x) ((x) == MPI_DOUBLE || (x) == MPI_DOUBLE_PRECISION) +#define isLONG_DOUBLE(x) ((x) == MPI_LONG_DOUBLE) +#define isLOC_TYPE(x) ((x) == MPI_2REAL || (x) == MPI_2DOUBLE_PRECISION || \ + (x) == MPI_2INTEGER || (x) == MPI_FLOAT_INT || \ + (x) == MPI_DOUBLE_INT || (x) == MPI_LONG_INT || \ + (x) == MPI_2INT || (x) == MPI_SHORT_INT || \ + (x) == MPI_LONG_DOUBLE_INT) +#define isBOOL(x) ((x) == MPI_C_BOOL) +#define isLOGICAL(x) ((x) == MPI_LOGICAL) +#define isSINGLE_COMPLEX(x) ((x) == MPI_COMPLEX || (x) == MPI_C_FLOAT_COMPLEX) +#define isDOUBLE_COMPLEX(x) ((x) == MPI_DOUBLE_COMPLEX || (x) == MPI_COMPLEX8 || \ + (x) == MPI_C_DOUBLE_COMPLEX) + + +#undef FUNCNAME +#define FUNCNAME mpi_to_ofi +#undef FCNAME +#define FCNAME DECL_FUNC(mpi_to_ofi) +static inline int mpi_to_ofi(MPI_Datatype dt, enum fi_datatype *fi_dt, MPI_Op op, enum fi_op *fi_op) +{ + *fi_dt = FI_DATATYPE_LAST; + *fi_op = FI_ATOMIC_OP_LAST; + + if (isS_INT(dt)) + *fi_dt = FI_INT32; + else if (isUS_INT(dt)) + *fi_dt = FI_UINT32; + else if (isFLOAT(dt)) + *fi_dt = FI_FLOAT; + else if (isDOUBLE(dt)) + *fi_dt = FI_DOUBLE; + else if (isLONG_DOUBLE(dt)) + *fi_dt = FI_LONG_DOUBLE; + else if (isS_CHAR(dt)) + *fi_dt = FI_INT8; + else if (isUS_CHAR(dt)) + *fi_dt = FI_UINT8; + else if (isS_SHORT(dt)) + *fi_dt = FI_INT16; + else if (isUS_SHORT(dt)) + *fi_dt = FI_UINT16; + else if (isS_LONG(dt)) + *fi_dt = FI_INT64; + else if (isUS_LONG(dt)) + *fi_dt = FI_UINT64; + else if (isS_LONG_LONG(dt)) + *fi_dt = FI_INT64; + else if (isUS_LONG_LONG(dt)) + *fi_dt = FI_UINT64; + else if (isSINGLE_COMPLEX(dt)) + *fi_dt = FI_FLOAT_COMPLEX; + else if (isDOUBLE_COMPLEX(dt)) + *fi_dt = FI_DOUBLE_COMPLEX; + else if (isLOC_TYPE(dt)) + *fi_dt = FI_DATATYPE_LAST; + else if (isLOGICAL(dt)) + *fi_dt = FI_UINT32; + else if (isBOOL(dt)) + *fi_dt = FI_UINT8; + + if (*fi_dt == FI_DATATYPE_LAST) + goto fn_fail; + + *fi_op = FI_ATOMIC_OP_LAST; + + switch (op) { + case MPI_SUM: + *fi_op = FI_SUM; + goto fn_exit; + break; + + case MPI_PROD: + *fi_op = FI_PROD; + goto fn_exit; + break; + + case MPI_MAX: + *fi_op = FI_MAX; + goto fn_exit; + break; + + case MPI_MIN: + *fi_op = FI_MIN; + goto fn_exit; + break; + + case MPI_BAND: + *fi_op = FI_BAND; + goto fn_exit; + break; + + case MPI_BOR: + *fi_op = FI_BOR; + goto fn_exit; + break; + + case MPI_BXOR: + *fi_op = FI_BXOR; + goto fn_exit; + break; + + case MPI_LAND: + if (isLONG_DOUBLE(dt)) + goto fn_fail; + + *fi_op = FI_LAND; + goto fn_exit; + break; + + case MPI_LOR: + if (isLONG_DOUBLE(dt)) + goto fn_fail; + + *fi_op = FI_LOR; + goto fn_exit; + break; + + case MPI_LXOR: + if (isLONG_DOUBLE(dt)) + goto fn_fail; + + *fi_op = FI_LXOR; + goto fn_exit; + break; + + case MPI_REPLACE:{ + *fi_op = FI_ATOMIC_WRITE; + goto fn_exit; + break; + } + + case MPI_NO_OP:{ + *fi_op = FI_ATOMIC_READ; + goto fn_exit; + break; + } + + case MPI_OP_NULL:{ + *fi_op = FI_CSWAP; + goto fn_exit; + break; + } + + default: + goto fn_fail; + break; + } + + fn_exit: + return MPI_SUCCESS; + fn_fail: + return -1; +} + +static MPI_Datatype mpi_dtypes[] = { + MPI_CHAR, MPI_UNSIGNED_CHAR, MPI_SIGNED_CHAR, MPI_BYTE, + MPI_WCHAR, MPI_SHORT, MPI_UNSIGNED_SHORT, MPI_INT, + MPI_UNSIGNED, MPI_LONG, MPI_UNSIGNED_LONG, MPI_FLOAT, + MPI_DOUBLE, MPI_LONG_DOUBLE, MPI_LONG_LONG, MPI_UNSIGNED_LONG_LONG, + MPI_PACKED, MPI_LB, MPI_UB, MPI_2INT, + + MPI_INT8_T, MPI_INT16_T, MPI_INT32_T, + MPI_INT64_T, MPI_UINT8_T, MPI_UINT16_T, + MPI_UINT32_T, MPI_UINT64_T, MPI_C_BOOL, + MPI_C_FLOAT_COMPLEX, MPI_C_DOUBLE_COMPLEX, MPI_C_LONG_DOUBLE_COMPLEX, + /* address/offset/count types */ + MPI_AINT, MPI_OFFSET, MPI_COUNT, + /* Fortran types */ +#ifdef HAVE_FORTRAN_BINDING + MPI_COMPLEX, MPI_DOUBLE_COMPLEX, MPI_LOGICAL, MPI_REAL, + MPI_DOUBLE_PRECISION, MPI_INTEGER, MPI_2INTEGER, + +#ifdef MPICH_DEFINE_2COMPLEX + MPI_2COMPLEX, MPI_2DOUBLE_COMPLEX, +#endif + MPI_2REAL, MPI_2DOUBLE_PRECISION, MPI_CHARACTER, + MPI_REAL4, MPI_REAL8, MPI_REAL16, MPI_COMPLEX8, MPI_COMPLEX16, + MPI_COMPLEX32, MPI_INTEGER1, MPI_INTEGER2, MPI_INTEGER4, MPI_INTEGER8, + MPI_INTEGER16, +#endif + MPI_FLOAT_INT, MPI_DOUBLE_INT, + MPI_LONG_INT, MPI_SHORT_INT, + MPI_LONG_DOUBLE_INT, + (MPI_Datatype) - 1, +}; + +static MPI_Op mpi_ops[] = { + MPI_MAX, MPI_MIN, MPI_SUM, MPI_PROD, + MPI_LAND, MPI_BAND, MPI_LOR, MPI_BOR, + MPI_LXOR, MPI_BXOR, MPI_MINLOC, MPI_MAXLOC, + MPI_REPLACE, MPI_NO_OP, MPI_OP_NULL, +}; + +#define _TBL MPIDI_Global.win_op_table[i][j] +#define CHECK_ATOMIC(fcn,field1,field2) \ + atomic_count = 0; \ + ret = fcn(MPIDI_OFI_EP_TX_RMA(0), \ + fi_dt, \ + fi_op, \ + &atomic_count); \ + if (ret == 0 && atomic_count != 0) \ + { \ + _TBL.field1 = 1; \ + _TBL.field2 = atomic_count; \ + } + +static inline void create_dt_map() +{ + int i, j; + size_t dtsize[FI_DATATYPE_LAST]; + dtsize[FI_INT8] = sizeof(int8_t); + dtsize[FI_UINT8] = sizeof(uint8_t); + dtsize[FI_INT16] = sizeof(int16_t); + dtsize[FI_UINT16] = sizeof(uint16_t); + dtsize[FI_INT32] = sizeof(int32_t); + dtsize[FI_UINT32] = sizeof(uint32_t); + dtsize[FI_INT64] = sizeof(int64_t); + dtsize[FI_UINT64] = sizeof(uint64_t); + dtsize[FI_FLOAT] = sizeof(float); + dtsize[FI_DOUBLE] = sizeof(double); + dtsize[FI_FLOAT_COMPLEX] = sizeof(float complex); + dtsize[FI_DOUBLE_COMPLEX] = sizeof(double complex); + dtsize[FI_LONG_DOUBLE] = sizeof(long double); + dtsize[FI_LONG_DOUBLE_COMPLEX] = sizeof(long double complex); + + for (i = 0; i < MPIDI_OFI_DT_SIZES; i++) + for (j = 0; j < MPIDI_OFI_OP_SIZES; j++) { + enum fi_datatype fi_dt = (enum fi_datatype) -1; + enum fi_op fi_op = (enum fi_op) -1; + mpi_to_ofi(mpi_dtypes[i], &fi_dt, mpi_ops[j], &fi_op); + MPIR_Assert(fi_dt != (enum fi_datatype) -1); + MPIR_Assert(fi_op != (enum fi_op) -1); + _TBL.dt = fi_dt; + _TBL.op = fi_op; + _TBL.atomic_valid = 0; + _TBL.max_atomic_count = 0; + _TBL.max_fetch_atomic_count = 0; + _TBL.max_compare_atomic_count = 0; + ssize_t ret; + size_t atomic_count; + + if (fi_dt != FI_DATATYPE_LAST && fi_op != FI_ATOMIC_OP_LAST) { + CHECK_ATOMIC(fi_atomicvalid, atomic_valid, max_atomic_count); + CHECK_ATOMIC(fi_fetch_atomicvalid, fetch_atomic_valid, max_fetch_atomic_count); + CHECK_ATOMIC(fi_compare_atomicvalid, compare_atomic_valid, + max_compare_atomic_count); + _TBL.dtsize = dtsize[fi_dt]; + } + } +} + +static inline void add_index(MPI_Datatype datatype, int *index) +{ + MPIR_Datatype *dt_ptr; + MPID_Datatype_get_ptr(datatype, dt_ptr); + MPIDI_OFI_DATATYPE(dt_ptr).index = *index; + (*index)++; +} + +void MPIDI_OFI_index_datatypes() +{ + int index = 0; + + add_index(MPI_CHAR, &index); + add_index(MPI_UNSIGNED_CHAR, &index); + add_index(MPI_SIGNED_CHAR, &index); + add_index(MPI_BYTE, &index); + add_index(MPI_WCHAR, &index); + add_index(MPI_SHORT, &index); + add_index(MPI_UNSIGNED_SHORT, &index); + add_index(MPI_INT, &index); + add_index(MPI_UNSIGNED, &index); + add_index(MPI_LONG, &index); + add_index(MPI_UNSIGNED_LONG, &index); /* 10 */ + + add_index(MPI_FLOAT, &index); + add_index(MPI_DOUBLE, &index); + add_index(MPI_LONG_DOUBLE, &index); + add_index(MPI_LONG_LONG, &index); + add_index(MPI_UNSIGNED_LONG_LONG, &index); + add_index(MPI_PACKED, &index); + add_index(MPI_LB, &index); + add_index(MPI_UB, &index); + add_index(MPI_2INT, &index); + + /* C99 types */ + add_index(MPI_INT8_T, &index); /* 20 */ + add_index(MPI_INT16_T, &index); + add_index(MPI_INT32_T, &index); + add_index(MPI_INT64_T, &index); + add_index(MPI_UINT8_T, &index); + add_index(MPI_UINT16_T, &index); + add_index(MPI_UINT32_T, &index); + add_index(MPI_UINT64_T, &index); + add_index(MPI_C_BOOL, &index); + add_index(MPI_C_FLOAT_COMPLEX, &index); + add_index(MPI_C_DOUBLE_COMPLEX, &index); /* 30 */ + add_index(MPI_C_LONG_DOUBLE_COMPLEX, &index); + + /* address/offset/count types */ + add_index(MPI_AINT, &index); + add_index(MPI_OFFSET, &index); + add_index(MPI_COUNT, &index); + + /* Fortran types */ +#ifdef HAVE_FORTRAN_BINDING + add_index(MPI_COMPLEX, &index); + add_index(MPI_DOUBLE_COMPLEX, &index); + add_index(MPI_LOGICAL, &index); + add_index(MPI_REAL, &index); + add_index(MPI_DOUBLE_PRECISION, &index); + add_index(MPI_INTEGER, &index); /* 40 */ + add_index(MPI_2INTEGER, &index); +#ifdef MPICH_DEFINE_2COMPLEX + add_index(MPI_2COMPLEX, &index); + add_index(MPI_2DOUBLE_COMPLEX, &index); +#endif + add_index(MPI_2REAL, &index); + add_index(MPI_2DOUBLE_PRECISION, &index); + add_index(MPI_CHARACTER, &index); + add_index(MPI_REAL4, &index); + add_index(MPI_REAL8, &index); + add_index(MPI_REAL16, &index); + add_index(MPI_COMPLEX8, &index); /* 50 */ + add_index(MPI_COMPLEX16, &index); + add_index(MPI_COMPLEX32, &index); + add_index(MPI_INTEGER1, &index); + add_index(MPI_INTEGER2, &index); + add_index(MPI_INTEGER4, &index); + add_index(MPI_INTEGER8, &index); + + if (MPI_INTEGER16 == MPI_DATATYPE_NULL) + index++; + else + add_index(MPI_INTEGER16, &index); + +#endif + add_index(MPI_FLOAT_INT, &index); + add_index(MPI_DOUBLE_INT, &index); + add_index(MPI_LONG_INT, &index); + add_index(MPI_SHORT_INT, &index); /* 60 */ + add_index(MPI_LONG_DOUBLE_INT, &index); + create_dt_map(); +} diff --git a/src/mpid/ch4/netmod/portals4/Makefile.mk b/src/mpid/ch4/netmod/portals4/Makefile.mk new file mode 100644 index 0000000..af89854 --- /dev/null +++ b/src/mpid/ch4/netmod/portals4/Makefile.mk @@ -0,0 +1,7 @@ +if BUILD_CH4_NETMOD_PORTALS4 + +mpi_core_sources += src/mpid/ch4/netmod/portals4/globals.c \ + src/mpid/ch4/netmod/portals4/func_table.c +errnames_txt_files += src/mpid/ch4/netmod/portals4/errnames.txt + +endif diff --git a/src/mpid/ch4/netmod/portals4/errnames.txt b/src/mpid/ch4/netmod/portals4/errnames.txt new file mode 100644 index 0000000..ff024e3 --- /dev/null +++ b/src/mpid/ch4/netmod/portals4/errnames.txt @@ -0,0 +1,10 @@ +**PtlInit:PtlInit failed +**PtlInit %s %d %s %s:PtlInit failed (%s %d %s %s) +**PtlNIInit:PtlNIInit failed +**PtlNIInit %s %d %s %s:PtlNIInit failed (%s %d %s %s) +**PtlEQAlloc:PtlEQAlloc failed +**PtlEQAlloc %s %d %s %s:PtlEQAlloc failed (%s %d %s %s) +**PtlPTAlloc:PtlPTAlloc failed +**PtlPTAlloc %s %d %s %s:PtlPTAlloc failed (%s %d %s %s) +**PtlMDBind:PtlMDBind failed +**PtlMDBind %s %d %s %s:PtlMDBind failed (%s %d %s %s) diff --git a/src/mpid/ch4/netmod/portals4/func_table.c b/src/mpid/ch4/netmod/portals4/func_table.c new file mode 100644 index 0000000..1c123eb --- /dev/null +++ b/src/mpid/ch4/netmod/portals4/func_table.c @@ -0,0 +1,157 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2012 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_DIRECT +#define NETMOD_DISABLE_INLINES +#include "mpidimpl.h" +#include "netmod_direct.h" + +MPIDI_NM_funcs_t MPIDI_NM_portals4_funcs = { + MPIDI_NM_init, + MPIDI_NM_finalize, + MPIDI_NM_progress, + MPIDI_NM_comm_connect, + MPIDI_NM_comm_disconnect, + MPIDI_NM_open_port, + MPIDI_NM_close_port, + MPIDI_NM_comm_accept, + MPIDI_NM_comm_get_lpid, + MPIDI_NM_gpid_get, + MPIDI_NM_get_node_id, + MPIDI_NM_get_max_node_id, + MPIDI_NM_getallincomm, + MPIDI_NM_gpid_tolpidarray, + MPIDI_NM_create_intercomm_from_lpids, + MPIDI_NM_comm_create, + MPIDI_NM_comm_destroy, + MPIDI_NM_am_request_init, + MPIDI_NM_am_request_finalize, + MPIDI_NM_reg_hdr_handler, + MPIDI_NM_send_am_hdr, + MPIDI_NM_inject_am_hdr, + MPIDI_NM_send_am, + MPIDI_NM_send_amv, + MPIDI_NM_send_amv_hdr, + MPIDI_NM_send_am_hdr_reply, + MPIDI_NM_inject_am_hdr_reply, + MPIDI_NM_send_am_reply, + MPIDI_NM_send_amv_reply, + MPIDI_NM_am_hdr_max_sz, + MPIDI_NM_am_inject_max_sz, + MPIDI_NM_am_recv, +}; + +MPIDI_NM_native_funcs_t MPIDI_NM_native_portals4_funcs = { + MPIDI_NM_send, + MPIDI_NM_ssend, + MPIDI_NM_startall, + MPIDI_NM_send_init, + MPIDI_NM_ssend_init, + MPIDI_NM_rsend_init, + MPIDI_NM_bsend_init, + MPIDI_NM_isend, + MPIDI_NM_issend, + MPIDI_NM_cancel_send, + MPIDI_NM_recv_init, + MPIDI_NM_recv, + MPIDI_NM_irecv, + MPIDI_NM_imrecv, + MPIDI_NM_cancel_recv, + MPIDI_NM_alloc_mem, + MPIDI_NM_free_mem, + MPIDI_NM_improbe, + MPIDI_NM_iprobe, + MPIDI_NM_win_set_info, + MPIDI_NM_win_shared_query, + MPIDI_NM_put, + MPIDI_NM_win_start, + MPIDI_NM_win_complete, + MPIDI_NM_win_post, + MPIDI_NM_win_wait, + MPIDI_NM_win_test, + MPIDI_NM_win_lock, + MPIDI_NM_win_unlock, + MPIDI_NM_win_get_info, + MPIDI_NM_get, + MPIDI_NM_win_free, + MPIDI_NM_win_fence, + MPIDI_NM_win_create, + MPIDI_NM_accumulate, + MPIDI_NM_win_attach, + MPIDI_NM_win_allocate_shared, + MPIDI_NM_rput, + MPIDI_NM_win_flush_local, + MPIDI_NM_win_detach, + MPIDI_NM_compare_and_swap, + MPIDI_NM_raccumulate, + MPIDI_NM_rget_accumulate, + MPIDI_NM_fetch_and_op, + MPIDI_NM_win_allocate, + MPIDI_NM_win_flush, + MPIDI_NM_win_flush_local_all, + MPIDI_NM_win_unlock_all, + MPIDI_NM_win_create_dynamic, + MPIDI_NM_rget, + MPIDI_NM_win_sync, + MPIDI_NM_win_flush_all, + MPIDI_NM_get_accumulate, + MPIDI_NM_win_lock_all, + MPIDI_NM_rank_is_local, + MPIDI_NM_barrier, + MPIDI_NM_bcast, + MPIDI_NM_allreduce, + MPIDI_NM_allgather, + MPIDI_NM_allgatherv, + MPIDI_NM_scatter, + MPIDI_NM_scatterv, + MPIDI_NM_gather, + MPIDI_NM_gatherv, + MPIDI_NM_alltoall, + MPIDI_NM_alltoallv, + MPIDI_NM_alltoallw, + MPIDI_NM_reduce, + MPIDI_NM_reduce_scatter, + MPIDI_NM_reduce_scatter_block, + MPIDI_NM_scan, + MPIDI_NM_exscan, + MPIDI_NM_neighbor_allgather, + MPIDI_NM_neighbor_allgatherv, + MPIDI_NM_neighbor_alltoall, + MPIDI_NM_neighbor_alltoallv, + MPIDI_NM_neighbor_alltoallw, + MPIDI_NM_ineighbor_allgather, + MPIDI_NM_ineighbor_allgatherv, + MPIDI_NM_ineighbor_alltoall, + MPIDI_NM_ineighbor_alltoallv, + MPIDI_NM_ineighbor_alltoallw, + MPIDI_NM_ibarrier, + MPIDI_NM_ibcast, + MPIDI_NM_iallgather, + MPIDI_NM_iallgatherv, + MPIDI_NM_iallreduce, + MPIDI_NM_ialltoall, + MPIDI_NM_ialltoallv, + MPIDI_NM_ialltoallw, + MPIDI_NM_iexscan, + MPIDI_NM_igather, + MPIDI_NM_igatherv, + MPIDI_NM_ireduce_scatter_block, + MPIDI_NM_ireduce_scatter, + MPIDI_NM_ireduce, + MPIDI_NM_iscan, + MPIDI_NM_iscatter, + MPIDI_NM_iscatterv, + MPIDI_NM_datatype_commit, + MPIDI_NM_datatype_dup, + MPIDI_NM_datatype_destroy, + MPIDI_NM_op_commit, + MPIDI_NM_op_destroy +}; +#endif diff --git a/src/mpid/ch4/netmod/portals4/globals.c b/src/mpid/ch4/netmod/portals4/globals.c new file mode 100644 index 0000000..0275cdd --- /dev/null +++ b/src/mpid/ch4/netmod/portals4/globals.c @@ -0,0 +1,14 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2016 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2012 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#include "ptl_impl.h" +#include "ptl_types.h" + +MPIDI_PTL_global_t MPIDI_PTL_global = { 0 }; diff --git a/src/mpid/ch4/netmod/portals4/netmod_direct.h b/src/mpid/ch4/netmod/portals4/netmod_direct.h new file mode 100644 index 0000000..edae2b3 --- /dev/null +++ b/src/mpid/ch4/netmod/portals4/netmod_direct.h @@ -0,0 +1,30 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2012 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_DIRECT_H_INCLUDED +#define NETMOD_DIRECT_H_INCLUDED +#include "ptl_init.h" +#include "ptl_types.h" +#include "ptl_probe.h" +#include "ptl_progress.h" +#include "ptl_recv.h" +#include "ptl_request.h" +#include "ptl_send.h" +#include "ptl_win.h" +#include "ptl_rma.h" +#include "ptl_am.h" +#include "ptl_spawn.h" +#include "ptl_comm.h" +#include "ptl_unimpl.h" +#include "ptl_proc.h" +#include "ptl_coll.h" +#include "ptl_datatype.h" +#include "ptl_op.h" +#endif /* NETMOD_DIRECT_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/portals4/portals4_pre.h b/src/mpid/ch4/netmod/portals4/portals4_pre.h new file mode 100644 index 0000000..4649cc5 --- /dev/null +++ b/src/mpid/ch4/netmod/portals4/portals4_pre.h @@ -0,0 +1,51 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2012 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ + +#ifndef NETMOD_PORTALS4_PRE_H_INCLUDED +#define NETMOD_PORTALS4_PRE_H_INCLUDED + +#include "portals4.h" + +typedef struct { + int handler_id; + char *pack_buffer; + ptl_handle_md_t md; +} MPIDI_PORTALS4_am_request_t; + +typedef struct { + int dummy; +} MPIDI_PORTALS4_request_t; + +typedef struct { + int dummy; +} MPIDI_PORTALS4_comm_t; + +typedef struct { + int dummy; +} MPIDI_PORTALS4_dt_t; + +typedef struct { + int dummy; +} MPIDI_PORTALS4_win_t; + +typedef struct { + int dummy; +} MPIDI_PORTALS4_gpid_t; + +typedef struct { + int dummy; +} MPIDI_PORTALS4_addr_t; + +typedef struct { + int dummy; +} MPIDI_PORTALS4_op_t; + +#endif /* NETMOD_PORTALS4_PRE_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/portals4/ptl_am.h b/src/mpid/ch4/netmod/portals4/ptl_am.h new file mode 100644 index 0000000..6b12443 --- /dev/null +++ b/src/mpid/ch4/netmod/portals4/ptl_am.h @@ -0,0 +1,370 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2012 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_PTL_AM_H_INCLUDED +#define NETMOD_PTL_AM_H_INCLUDED + +#include "ptl_impl.h" + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_reg_hdr_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_reg_hdr_handler(int handler_id, + MPIDI_NM_am_origin_handler_fn origin_handler_fn, + MPIDI_NM_am_target_handler_fn target_handler_fn) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_REG_HDR_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_REG_HDR_HANDLER); + + MPIDI_PTL_global.am_handlers[handler_id] = target_handler_fn; + MPIDI_PTL_global.send_cmpl_handlers[handler_id] = origin_handler_fn; + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_REG_HDR_HANDLER); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +static inline int MPIDI_NM_send_am_hdr(int rank, + MPIR_Comm * comm, + int handler_id, + const void *am_hdr, + size_t am_hdr_sz, MPIR_Request * sreq, void *netmod_context) +{ + int mpi_errno = MPI_SUCCESS, ret, c; + ptl_hdr_data_t ptl_hdr; + ptl_match_bits_t match_bits; + char *send_buf = NULL; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_SEND_AM); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_SEND_AM); + + ptl_hdr = MPIDI_PTL_init_am_hdr(handler_id, 0); + match_bits = MPIDI_PTL_init_tag(comm->context_id, MPIDI_PTL_AM_TAG); + sreq->dev.ch4.ch4u.netmod_am.portals4.handler_id = handler_id; + + MPIR_cc_incr(sreq->cc_ptr, &c); + + send_buf = MPL_malloc(am_hdr_sz); + MPIR_Memcpy(send_buf, am_hdr, am_hdr_sz); + sreq->dev.ch4.ch4u.netmod_am.portals4.pack_buffer = send_buf; + + ret = PtlPut(MPIDI_PTL_global.md, (ptl_size_t) send_buf, am_hdr_sz, + PTL_ACK_REQ, MPIDI_PTL_global.addr_table[rank].process, + MPIDI_PTL_global.addr_table[rank].pt, match_bits, 0, sreq, ptl_hdr); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_SEND_AM); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +static inline int MPIDI_NM_send_am(int rank, + MPIR_Comm * comm, + int handler_id, + const void *am_hdr, + size_t am_hdr_sz, + const void *data, + MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq, void *netmod_context) +{ + int mpi_errno = MPI_SUCCESS, ret, c; + size_t data_sz; + MPI_Aint dt_true_lb, last; + MPIR_Datatype *dt_ptr; + int dt_contig; + ptl_hdr_data_t ptl_hdr; + ptl_match_bits_t match_bits; + char *send_buf = NULL; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_SEND_AM); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_SEND_AM); + + MPIDI_Datatype_get_info(count, datatype, dt_contig, data_sz, dt_ptr, dt_true_lb); + match_bits = MPIDI_PTL_init_tag(comm->context_id, MPIDI_PTL_AM_TAG); + ptl_hdr = MPIDI_PTL_init_am_hdr(handler_id, data_sz); + sreq->dev.ch4.ch4u.netmod_am.portals4.handler_id = handler_id; + + MPIR_cc_incr(sreq->cc_ptr, &c); + + if (dt_contig) { + /* create a two element iovec and send */ + ptl_md_t md; + ptl_iovec_t iovec[2]; + + send_buf = MPL_malloc(am_hdr_sz); + MPIR_Memcpy(send_buf, am_hdr, am_hdr_sz); + sreq->dev.ch4.ch4u.netmod_am.portals4.pack_buffer = send_buf; + + iovec[0].iov_base = send_buf; + iovec[0].iov_len = am_hdr_sz; + iovec[1].iov_base = (char *) data + dt_true_lb; + iovec[1].iov_len = data_sz; + md.start = iovec; + md.length = 2; + md.options = PTL_IOVEC; + md.eq_handle = MPIDI_PTL_global.eqs[0]; + md.ct_handle = PTL_CT_NONE; + + ret = PtlMDBind(MPIDI_PTL_global.ni, &md, &sreq->dev.ch4.ch4u.netmod_am.portals4.md); + ret = PtlPut(sreq->dev.ch4.ch4u.netmod_am.portals4.md, 0, am_hdr_sz + data_sz, + PTL_ACK_REQ, MPIDI_PTL_global.addr_table[rank].process, + MPIDI_PTL_global.addr_table[rank].pt, match_bits, 0, sreq, ptl_hdr); + } + else { + /* copy everything into pack_buffer */ + MPID_Segment *segment; + MPI_Aint last; + + send_buf = MPL_malloc(am_hdr_sz + data_sz); + MPIR_Memcpy(send_buf, am_hdr, am_hdr_sz); + segment = MPID_Segment_alloc(); + MPID_Segment_init(data, count, datatype, segment, 0); + last = data_sz; + MPID_Segment_pack(segment, 0, &last, send_buf + am_hdr_sz); + MPIR_Assert(last == data_sz); + MPID_Segment_free(segment); + sreq->dev.ch4.ch4u.netmod_am.portals4.pack_buffer = send_buf; + + ret = PtlPut(MPIDI_PTL_global.md, (ptl_size_t) send_buf, am_hdr_sz + data_sz, + PTL_ACK_REQ, MPIDI_PTL_global.addr_table[rank].process, + MPIDI_PTL_global.addr_table[rank].pt, match_bits, 0, sreq, ptl_hdr); + } + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_SEND_AM); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +static inline int MPIDI_NM_send_amv(int rank, + MPIR_Comm * comm, + int handler_id, + struct iovec *am_hdr, + size_t iov_len, + const void *data, + MPI_Count count, + MPI_Datatype datatype, + MPIR_Request * sreq, void *netmod_context) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_send_amv_hdr(int rank, + MPIR_Comm * comm, + int handler_id, + struct iovec *am_hdr, + size_t iov_len, MPIR_Request * sreq, void *netmod_context) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_send_am_hdr_reply(MPIR_Context_id_t context_id, + int src_rank, + int handler_id, + const void *am_hdr, + size_t am_hdr_sz, MPIR_Request * sreq) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_send_am_reply(MPIR_Context_id_t context_id, + int src_rank, + int handler_id, + const void *am_hdr, + size_t am_hdr_sz, + const void *data, + MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq) +{ + int mpi_errno = MPI_SUCCESS, ret, c; + size_t data_sz; + MPI_Aint dt_true_lb, last; + MPIR_Datatype *dt_ptr; + int dt_contig; + ptl_hdr_data_t ptl_hdr; + ptl_match_bits_t match_bits; + MPIR_Comm *use_comm; + char *send_buf = NULL; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_SEND_AM); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_SEND_AM); + + use_comm = MPIDI_CH4U_context_id_to_comm(context_id); + + MPIDI_Datatype_get_info(count, datatype, dt_contig, data_sz, dt_ptr, dt_true_lb); + match_bits = MPIDI_PTL_init_tag(use_comm->context_id, MPIDI_PTL_AM_TAG); + ptl_hdr = MPIDI_PTL_init_am_hdr(handler_id, data_sz); + sreq->dev.ch4.ch4u.netmod_am.portals4.handler_id = handler_id; + + MPIR_cc_incr(sreq->cc_ptr, &c); + + if (dt_contig) { + /* create a two element iovec and send */ + ptl_md_t md; + ptl_iovec_t iovec[2]; + + send_buf = MPL_malloc(am_hdr_sz); + MPIR_Memcpy(send_buf, am_hdr, am_hdr_sz); + sreq->dev.ch4.ch4u.netmod_am.portals4.pack_buffer = send_buf; + + iovec[0].iov_base = send_buf; + iovec[0].iov_len = am_hdr_sz; + iovec[1].iov_base = (char *) data + dt_true_lb; + iovec[1].iov_len = data_sz; + md.start = iovec; + md.length = 2; + md.options = PTL_IOVEC; + md.eq_handle = MPIDI_PTL_global.eqs[0]; + md.ct_handle = PTL_CT_NONE; + + ret = PtlMDBind(MPIDI_PTL_global.ni, &md, &sreq->dev.ch4.ch4u.netmod_am.portals4.md); + ret = PtlPut(sreq->dev.ch4.ch4u.netmod_am.portals4.md, 0, am_hdr_sz + data_sz, + PTL_ACK_REQ, MPIDI_PTL_global.addr_table[src_rank].process, + MPIDI_PTL_global.addr_table[src_rank].pt, match_bits, 0, sreq, ptl_hdr); + } + else { + /* copy everything into pack_buffer */ + MPID_Segment *segment; + MPI_Aint last; + + send_buf = MPL_malloc(am_hdr_sz + data_sz); + MPIR_Memcpy(send_buf, am_hdr, am_hdr_sz); + segment = MPID_Segment_alloc(); + MPID_Segment_init(data, count, datatype, segment, 0); + last = data_sz; + MPID_Segment_pack(segment, 0, &last, send_buf + am_hdr_sz); + MPIR_Assert(last == data_sz); + MPID_Segment_free(segment); + sreq->dev.ch4.ch4u.netmod_am.portals4.pack_buffer = send_buf; + + ret = PtlPut(MPIDI_PTL_global.md, (ptl_size_t) send_buf, am_hdr_sz + data_sz, + PTL_ACK_REQ, MPIDI_PTL_global.addr_table[src_rank].process, + MPIDI_PTL_global.addr_table[src_rank].pt, match_bits, 0, sreq, ptl_hdr); + } + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_SEND_AM); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +static inline int MPIDI_NM_send_amv_reply(MPIR_Context_id_t context_id, + int src_rank, + int handler_id, + struct iovec *am_hdr, + size_t iov_len, + const void *data, + MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline size_t MPIDI_NM_am_hdr_max_sz(void) +{ + MPIR_Assert(0); + return 0; +} + +static inline int MPIDI_NM_inject_am_hdr(int rank, + MPIR_Comm * comm, + int handler_id, + const void *am_hdr, size_t am_hdr_sz, void *netmod_context) +{ + int mpi_errno = MPI_SUCCESS, ret, c; + ptl_hdr_data_t ptl_hdr; + ptl_match_bits_t match_bits; + char *send_buf = NULL; + MPIR_Request *inject_req; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_SEND_AM); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_SEND_AM); + + ptl_hdr = MPIDI_PTL_init_am_hdr(handler_id, 0); + match_bits = MPIDI_PTL_init_tag(comm->context_id, MPIDI_PTL_AM_TAG); + + /* create an internal request for the inject */ + inject_req = MPIR_Request_create(MPIR_REQUEST_KIND__UNDEFINED); + MPIDI_NM_am_request_init(inject_req); + send_buf = MPL_malloc(am_hdr_sz); + MPIR_Memcpy(send_buf, am_hdr, am_hdr_sz); + inject_req->dev.ch4.ch4u.netmod_am.portals4.pack_buffer = send_buf; + + ret = PtlPut(MPIDI_PTL_global.md, (ptl_size_t) send_buf, am_hdr_sz, + PTL_ACK_REQ, MPIDI_PTL_global.addr_table[rank].process, + MPIDI_PTL_global.addr_table[rank].pt, match_bits, 0, inject_req, ptl_hdr); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_SEND_AM); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +static inline int MPIDI_NM_inject_am_hdr_reply(MPIR_Context_id_t context_id, + int src_rank, + int handler_id, const void *am_hdr, size_t am_hdr_sz) +{ + int mpi_errno = MPI_SUCCESS, ret, c; + ptl_hdr_data_t ptl_hdr; + ptl_match_bits_t match_bits; + MPIR_Comm *use_comm; + char *send_buf = NULL; + MPIR_Request *inject_req; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_SEND_AM); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_SEND_AM); + + use_comm = MPIDI_CH4U_context_id_to_comm(context_id); + + ptl_hdr = MPIDI_PTL_init_am_hdr(handler_id, 0); + match_bits = MPIDI_PTL_init_tag(use_comm->context_id, MPIDI_PTL_AM_TAG); + + /* create an internal request for the inject */ + inject_req = MPIR_Request_create(MPIR_REQUEST_KIND__UNDEFINED); + MPIDI_NM_am_request_init(inject_req); + send_buf = MPL_malloc(am_hdr_sz); + MPIR_Memcpy(send_buf, am_hdr, am_hdr_sz); + inject_req->dev.ch4.ch4u.netmod_am.portals4.pack_buffer = send_buf; + + ret = PtlPut(MPIDI_PTL_global.md, (ptl_size_t) send_buf, am_hdr_sz, + PTL_ACK_REQ, MPIDI_PTL_global.addr_table[src_rank].process, + MPIDI_PTL_global.addr_table[src_rank].pt, match_bits, 0, inject_req, ptl_hdr); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_SEND_AM); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +static inline size_t MPIDI_NM_am_inject_max_sz(void) +{ + MPIR_Assert(0); + return 0; +} + +static inline int MPIDI_NM_am_recv(MPIR_Request * req) +{ + MPIR_Assert(0); + return 0; +} + +#endif /* NETMOD_PTL_AM_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/portals4/ptl_coll.h b/src/mpid/ch4/netmod/portals4/ptl_coll.h new file mode 100644 index 0000000..5d7073f --- /dev/null +++ b/src/mpid/ch4/netmod/portals4/ptl_coll.h @@ -0,0 +1,871 @@ + +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2012 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_PTL_COLL_H_INCLUDED +#define NETMOD_PTL_COLL_H_INCLUDED + +#include "ptl_impl.h" + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_barrier +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_barrier(MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_BARRIER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_BARRIER); + + mpi_errno = MPIR_Barrier(comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_BARRIER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_bcast +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_bcast(void *buffer, int count, MPI_Datatype datatype, + int root, MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_BCAST); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_BCAST); + + mpi_errno = MPIR_Bcast(buffer, count, datatype, root, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_BCAST); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_allreduce +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_allreduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_ALLREDUCE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_ALLREDUCE); + + mpi_errno = MPIR_Allreduce(sendbuf, recvbuf, count, datatype, op, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_ALLREDUCE); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_allgather +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_allgather(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_ALLGATHER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_ALLGATHER); + + mpi_errno = MPIR_Allgather(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, + comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_ALLGATHER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_allgatherv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_allgatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, const int *displs, + MPI_Datatype recvtype, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_ALLGATHERV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_ALLGATHERV); + + mpi_errno = MPIR_Allgatherv(sendbuf, sendcount, sendtype, + recvbuf, recvcounts, displs, recvtype, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_ALLGATHERV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_gather +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_gather(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + int root, MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_GATHER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_GATHER); + + mpi_errno = MPIR_Gather(sendbuf, sendcount, sendtype, recvbuf, recvcount, + recvtype, root, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_GATHER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_gatherv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_gatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, const int *displs, + MPI_Datatype recvtype, int root, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_GATHERV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_GATHERV); + + mpi_errno = MPIR_Gatherv(sendbuf, sendcount, sendtype, + recvbuf, recvcounts, displs, recvtype, root, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_GATHERV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_scatter +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_scatter(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + int root, MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_SCATTER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_SCATTER); + + mpi_errno = MPIR_Scatter(sendbuf, sendcount, sendtype, + recvbuf, recvcount, recvtype, root, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_SCATTER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_scatterv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_scatterv(const void *sendbuf, const int *sendcounts, + const int *displs, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + int root, MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_SCATTERV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_SCATTERV); + + mpi_errno = MPIR_Scatterv(sendbuf, sendcounts, displs, + sendtype, recvbuf, recvcount, recvtype, root, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_SCATTERV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_alltoall +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_alltoall(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_ALLTOALL); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_ALLTOALL); + + mpi_errno = MPIR_Alltoall(sendbuf, sendcount, sendtype, recvbuf, recvcount, + recvtype, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_ALLTOALL); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_alltoallv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_alltoallv(const void *sendbuf, const int *sendcounts, + const int *sdispls, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, + const int *rdispls, MPI_Datatype recvtype, + MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_ALLTOALLV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_ALLTOALLV); + + mpi_errno = MPIR_Alltoallv(sendbuf, sendcounts, sdispls, + sendtype, recvbuf, recvcounts, rdispls, recvtype, comm_ptr, errflag); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_ALLTOALLV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_alltoallw +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_alltoallw(const void *sendbuf, const int sendcounts[], + const int sdispls[], const MPI_Datatype sendtypes[], + void *recvbuf, const int recvcounts[], + const int rdispls[], const MPI_Datatype recvtypes[], + MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_ALLTOALLW); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_ALLTOALLW); + + mpi_errno = MPIR_Alltoallw(sendbuf, sendcounts, sdispls, + sendtypes, recvbuf, recvcounts, + rdispls, recvtypes, comm_ptr, errflag); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_ALLTOALLW); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_reduce +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_reduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, int root, + MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_REDUCE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_REDUCE); + + mpi_errno = MPIR_Reduce(sendbuf, recvbuf, count, datatype, op, root, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_REDUCE); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_reduce_scatter +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_reduce_scatter(const void *sendbuf, void *recvbuf, + const int recvcounts[], MPI_Datatype datatype, + MPI_Op op, MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_REDUCE_SCATTER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_REDUCE_SCATTER); + + mpi_errno = MPIR_Reduce_scatter(sendbuf, recvbuf, recvcounts, datatype, op, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_REDUCE_SCATTER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_reduce_scatter_block +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_reduce_scatter_block(const void *sendbuf, void *recvbuf, + int recvcount, MPI_Datatype datatype, + MPI_Op op, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_REDUCE_SCATTER_BLOCK); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_REDUCE_SCATTER_BLOCK); + + mpi_errno = MPIR_Reduce_scatter_block(sendbuf, recvbuf, recvcount, + datatype, op, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_REDUCE_SCATTER_BLOCK); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_scan +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_scan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_SCAN); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_SCAN); + + mpi_errno = MPIR_Scan(sendbuf, recvbuf, count, datatype, op, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_SCAN); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_exscan +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_exscan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_EXSCAN); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_EXSCAN); + + mpi_errno = MPIR_Exscan(sendbuf, recvbuf, count, datatype, op, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_EXSCAN); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_neighbor_allgather +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_neighbor_allgather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, int recvcount, + MPI_Datatype recvtype, MPIR_Comm * comm_ptr) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_NEIGHBOR_ALLGATHER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_NEIGHBOR_ALLGATHER); + + mpi_errno = + MPIR_Neighbor_allgather_impl(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, + comm_ptr); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_NEIGHBOR_ALLGATHER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_neighbor_allgatherv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_neighbor_allgatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int recvcounts[], const int displs[], + MPI_Datatype recvtype, MPIR_Comm * comm_ptr) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_NEIGHBOR_ALLGATHERV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_NEIGHBOR_ALLGATHERV); + + mpi_errno = MPIR_Neighbor_allgatherv_impl(sendbuf, sendcount, sendtype, + recvbuf, recvcounts, displs, recvtype, comm_ptr); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_NEIGHBOR_ALLGATHERV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_neighbor_alltoall +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_neighbor_alltoall(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, int recvcount, + MPI_Datatype recvtype, MPIR_Comm * comm_ptr) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_NEIGHBOR_ALLTOALL); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_NEIGHBOR_ALLTOALL); + + mpi_errno = MPIR_Neighbor_alltoall_impl(sendbuf, sendcount, sendtype, + recvbuf, recvcount, recvtype, comm_ptr); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_NEIGHBOR_ALLTOALL); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_neighbor_alltoallv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_neighbor_alltoallv(const void *sendbuf, const int sendcounts[], + const int sdispls[], MPI_Datatype sendtype, + void *recvbuf, const int recvcounts[], + const int rdispls[], MPI_Datatype recvtype, + MPIR_Comm * comm_ptr) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_NEIGHBOR_ALLTOALLV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_NEIGHBOR_ALLTOALLV); + + mpi_errno = MPIR_Neighbor_alltoallv_impl(sendbuf, sendcounts, sdispls, sendtype, + recvbuf, recvcounts, rdispls, recvtype, comm_ptr); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_NEIGHBOR_ALLTOALLV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_neighbor_alltoallw +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_neighbor_alltoallw(const void *sendbuf, const int sendcounts[], + const MPI_Aint sdispls[], + const MPI_Datatype sendtypes[], void *recvbuf, + const int recvcounts[], const MPI_Aint rdispls[], + const MPI_Datatype recvtypes[], MPIR_Comm * comm_ptr) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_NEIGHBOR_ALLTOALLW); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_NEIGHBOR_ALLTOALLW); + + mpi_errno = MPIR_Neighbor_alltoallw_impl(sendbuf, sendcounts, sdispls, sendtypes, + recvbuf, recvcounts, rdispls, recvtypes, comm_ptr); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_NEIGHBOR_ALLTOALLW); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ineighbor_allgather +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ineighbor_allgather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, int recvcount, + MPI_Datatype recvtype, MPIR_Comm * comm_ptr, + MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_INEIGHBOR_ALLGATHER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_INEIGHBOR_ALLGATHER); + + mpi_errno = MPIR_Ineighbor_allgather_impl(sendbuf, sendcount, sendtype, + recvbuf, recvcount, recvtype, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_INEIGHBOR_ALLGATHER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ineighbor_allgatherv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ineighbor_allgatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int recvcounts[], const int displs[], + MPI_Datatype recvtype, MPIR_Comm * comm_ptr, + MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_INEIGHBOR_ALLGATHERV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_INEIGHBOR_ALLGATHERV); + + mpi_errno = MPIR_Ineighbor_allgatherv_impl(sendbuf, sendcount, sendtype, + recvbuf, recvcounts, displs, recvtype, + comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_INEIGHBOR_ALLGATHERV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ineighbor_alltoall +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ineighbor_alltoall(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_INEIGHBOR_ALLTOALL); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_INEIGHBOR_ALLTOALL); + + mpi_errno = MPIR_Ineighbor_alltoall_impl(sendbuf, sendcount, sendtype, + recvbuf, recvcount, recvtype, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_INEIGHBOR_ALLTOALL); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ineighbor_alltoallv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ineighbor_alltoallv(const void *sendbuf, const int sendcounts[], + const int sdispls[], MPI_Datatype sendtype, + void *recvbuf, const int recvcounts[], + const int rdispls[], MPI_Datatype recvtype, + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_INEIGHBOR_ALLTOALLV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_INEIGHBOR_ALLTOALLV); + + mpi_errno = MPIR_Ineighbor_alltoallv_impl(sendbuf, sendcounts, sdispls, sendtype, + recvbuf, recvcounts, rdispls, recvtype, + comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_INEIGHBOR_ALLTOALLV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ineighbor_alltoallw +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ineighbor_alltoallw(const void *sendbuf, const int sendcounts[], + const MPI_Aint sdispls[], + const MPI_Datatype sendtypes[], void *recvbuf, + const int recvcounts[], const MPI_Aint rdispls[], + const MPI_Datatype recvtypes[], MPIR_Comm * comm_ptr, + MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_INEIGHBOR_ALLTOALLW); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_INEIGHBOR_ALLTOALLW); + + mpi_errno = MPIR_Ineighbor_alltoallw_impl(sendbuf, sendcounts, sdispls, sendtypes, + recvbuf, recvcounts, rdispls, recvtypes, + comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_INEIGHBOR_ALLTOALLW); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ibarrier +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ibarrier(MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IBARRIER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IBARRIER); + + mpi_errno = MPIR_Ibarrier_impl(comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IBARRIER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ibcast +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ibcast(void *buffer, int count, MPI_Datatype datatype, + int root, MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IBCAST); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IBCAST); + + mpi_errno = MPIR_Ibcast_impl(buffer, count, datatype, root, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IBCAST); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_iallgather +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_iallgather(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IALLGATHER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IALLGATHER); + + mpi_errno = MPIR_Iallgather_impl(sendbuf, sendcount, sendtype, recvbuf, + recvcount, recvtype, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IALLGATHER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_iallreduce +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_iallreduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IREDUCE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IREDUCE); + + mpi_errno = MPIR_Iallreduce_impl(sendbuf, recvbuf, count, datatype, op, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IREDUCE); + return mpi_errno; +} + + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_iallgatherv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_iallgatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, const int *displs, + MPI_Datatype recvtype, MPIR_Comm * comm_ptr, + MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IALLGATHERV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IALLGATHERV); + + mpi_errno = MPIR_Iallgatherv_impl(sendbuf, sendcount, sendtype, + recvbuf, recvcounts, displs, recvtype, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IALLGATHERV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ialltoall +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ialltoall(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IALLTOALL); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IALLTOALL); + + mpi_errno = MPIR_Ialltoall_impl(sendbuf, sendcount, sendtype, recvbuf, + recvcount, recvtype, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IALLTOALL); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ialltoallv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ialltoallv(const void *sendbuf, const int *sendcounts, + const int *sdispls, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, + const int *rdispls, MPI_Datatype recvtype, + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IALLTOALLV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IALLTOALLV); + + mpi_errno = MPIR_Ialltoallv_impl(sendbuf, sendcounts, sdispls, + sendtype, recvbuf, recvcounts, + rdispls, recvtype, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IALLTOALLV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ialltoallw +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ialltoallw(const void *sendbuf, const int *sendcounts, + const int *sdispls, const MPI_Datatype sendtypes[], + void *recvbuf, const int *recvcounts, + const int *rdispls, const MPI_Datatype recvtypes[], + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IALLTOALLW); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IALLTOALLW); + + mpi_errno = MPIR_Ialltoallw_impl(sendbuf, sendcounts, sdispls, + sendtypes, recvbuf, recvcounts, + rdispls, recvtypes, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IALLTOALLW); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_iexscan +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_iexscan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm_ptr, + MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IEXSCAN); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IEXSCAN); + + mpi_errno = MPIR_Iexscan_impl(sendbuf, recvbuf, count, datatype, op, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IEXSCAN); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_igather +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_igather(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + int root, MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IGATHER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IGATHER); + + mpi_errno = MPIR_Igather_impl(sendbuf, sendcount, sendtype, recvbuf, + recvcount, recvtype, root, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IGATHER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_igatherv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_igatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, const int *displs, + MPI_Datatype recvtype, int root, MPIR_Comm * comm_ptr, + MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IGATHERV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IGATHERV); + + mpi_errno = MPIR_Igatherv_impl(sendbuf, sendcount, sendtype, + recvbuf, recvcounts, displs, recvtype, root, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IGATHERV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ireduce_scatter_block +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ireduce_scatter_block(const void *sendbuf, void *recvbuf, + int recvcount, MPI_Datatype datatype, + MPI_Op op, MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IREDUCE_SCATTER_BLOCK); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IREDUCE_SCATTER_BLOCK); + + mpi_errno = MPIR_Ireduce_scatter_block_impl(sendbuf, recvbuf, recvcount, + datatype, op, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IREDUCE_SCATTER_BLOCK); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ireduce_scatter +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ireduce_scatter(const void *sendbuf, void *recvbuf, + const int recvcounts[], MPI_Datatype datatype, + MPI_Op op, MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IREDUCE_SCATTER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IREDUCE_SCATTER); + + mpi_errno = MPIR_Ireduce_scatter_impl(sendbuf, recvbuf, recvcounts, datatype, op, + comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IREDUCE_SCATTER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ireduce +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ireduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, int root, + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IREDUCE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IREDUCE); + + mpi_errno = MPIR_Ireduce_impl(sendbuf, recvbuf, count, datatype, op, root, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IREDUCE); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_iscan +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_iscan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm_ptr, + MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_ISCAN); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_ISCAN); + + mpi_errno = MPIR_Iscan_impl(sendbuf, recvbuf, count, datatype, op, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_ISCAN); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_iscatter +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_iscatter(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + int root, MPIR_Comm * comm, MPI_Request * request) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_ISCATTER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_ISCATTER); + + mpi_errno = MPIR_Iscatter_impl(sendbuf, sendcount, sendtype, recvbuf, + recvcount, recvtype, root, comm, request); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_ISCATTER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_iscatterv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_iscatterv(const void *sendbuf, const int *sendcounts, + const int *displs, MPI_Datatype sendtype, + void *recvbuf, int recvcount, + MPI_Datatype recvtype, int root, + MPIR_Comm * comm, MPI_Request * request) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_ISCATTERV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_ISCATTERV); + + mpi_errno = MPIR_Iscatterv_impl(sendbuf, sendcounts, displs, sendtype, + recvbuf, recvcount, recvtype, root, comm, request); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_ISCATTERV); + return mpi_errno; +} + +#endif /* NETMOD_PTL_COLL_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/portals4/ptl_comm.h b/src/mpid/ch4/netmod/portals4/ptl_comm.h new file mode 100644 index 0000000..651ab1f --- /dev/null +++ b/src/mpid/ch4/netmod/portals4/ptl_comm.h @@ -0,0 +1,39 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2012 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_PTL_COMM_H_INCLUDED +#define NETMOD_PTL_COMM_H_INCLUDED + +#include "ptl_impl.h" + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_comm_create +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_comm_create(MPIR_Comm * comm) +{ + int mpi_errno = MPI_SUCCESS; + mpi_errno = MPIDI_CH4U_init_comm(comm); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_comm_destroy +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_comm_destroy(MPIR_Comm * comm) +{ + int mpi_errno = MPI_SUCCESS; + mpi_errno = MPIDI_CH4U_destroy_comm(comm); + return mpi_errno; +} + + +#endif /* NETMOD_PTL_COMM_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/portals4/ptl_datatype.h b/src/mpid/ch4/netmod/portals4/ptl_datatype.h new file mode 100644 index 0000000..3e529d6 --- /dev/null +++ b/src/mpid/ch4/netmod/portals4/ptl_datatype.h @@ -0,0 +1,28 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2016 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + */ +#ifndef NETMOD_PTL_DATATYPE_H_INCLUDED +#define NETMOD_PTL_DATATYPE_H_INCLUDED + +#include "ptl_impl.h" + +static inline void MPIDI_NM_datatype_destroy(MPIR_Datatype * datatype_p) +{ + return; +} + +static inline void MPIDI_NM_datatype_commit(MPIR_Datatype * datatype_p) +{ + return; +} + +static inline void MPIDI_NM_datatype_dup(MPIR_Datatype * old_datatype_p, + MPIR_Datatype * new_datatype_p) +{ + return; +} + +#endif /* NETMOD_PTL_DATATYPE_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/portals4/ptl_impl.h b/src/mpid/ch4/netmod/portals4/ptl_impl.h new file mode 100644 index 0000000..ddcf713 --- /dev/null +++ b/src/mpid/ch4/netmod/portals4/ptl_impl.h @@ -0,0 +1,58 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2012 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_PTL_IMPL_H_INCLUDED +#define NETMOD_PTL_IMPL_H_INCLUDED + +#include <mpidimpl.h> +#include "portals4.h" + +#define MPIDI_PTL_CASE_STR(x) case x: return #x + +static inline const char *MPIDI_PTL_strerror(int ret) +{ + switch (ret) { + MPIDI_PTL_CASE_STR(PTL_OK); + MPIDI_PTL_CASE_STR(PTL_ARG_INVALID); + MPIDI_PTL_CASE_STR(PTL_CT_NONE_REACHED); + MPIDI_PTL_CASE_STR(PTL_EQ_DROPPED); + MPIDI_PTL_CASE_STR(PTL_EQ_EMPTY); + MPIDI_PTL_CASE_STR(PTL_FAIL); + MPIDI_PTL_CASE_STR(PTL_IN_USE); + MPIDI_PTL_CASE_STR(PTL_INTERRUPTED); + MPIDI_PTL_CASE_STR(PTL_IGNORED); + MPIDI_PTL_CASE_STR(PTL_LIST_TOO_LONG); + MPIDI_PTL_CASE_STR(PTL_NO_INIT); + MPIDI_PTL_CASE_STR(PTL_NO_SPACE); + MPIDI_PTL_CASE_STR(PTL_PID_IN_USE); + MPIDI_PTL_CASE_STR(PTL_PT_FULL); + MPIDI_PTL_CASE_STR(PTL_PT_EQ_NEEDED); + MPIDI_PTL_CASE_STR(PTL_PT_IN_USE); + default: + return "UNKNOWN"; + } +} + +#define MPIDI_PTL_ERR MPIR_ERR_CHKANDJUMP4 + +#define MPIDI_PTL_CHK_STATUS(STATUS,STR) \ + do { \ + MPIDI_PTL_ERR(STATUS!=PTL_OK, \ + mpi_errno, \ + MPI_ERR_OTHER, \ + "**"#STR, \ + "**"#STR" %s %d %s %s", \ + __FILE__, \ + __LINE__, \ + FCNAME, \ + MPIDI_PTL_strerror(STATUS)); \ + } while (0) + +#endif /* NETMOD_PTL_IMPL_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/portals4/ptl_init.h b/src/mpid/ch4/netmod/portals4/ptl_init.h new file mode 100644 index 0000000..da36d4e --- /dev/null +++ b/src/mpid/ch4/netmod/portals4/ptl_init.h @@ -0,0 +1,271 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2012 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_PTL_INIT_H_INCLUDED +#define NETMOD_PTL_INIT_H_INCLUDED + +#include "mpidch4r.h" +#include "ptl_types.h" +#include "ptl_impl.h" +#include "portals4.h" + +static inline int MPIDI_PTL_append_overflow(size_t i) +{ + ptl_me_t me; + ptl_process_t id_any; + + id_any.phys.pid = PTL_PID_ANY; + id_any.phys.nid = PTL_NID_ANY; + + me.start = MPIDI_PTL_global.overflow_bufs[i]; + me.length = MPIDI_PTL_OVERFLOW_BUFFER_SZ; + me.ct_handle = PTL_CT_NONE; + me.uid = PTL_UID_ANY; + me.options = (PTL_ME_OP_PUT | PTL_ME_MANAGE_LOCAL | PTL_ME_NO_TRUNCATE | PTL_ME_MAY_ALIGN | + PTL_ME_IS_ACCESSIBLE | PTL_ME_EVENT_LINK_DISABLE); + me.match_id = id_any; + me.match_bits = 0; + me.ignore_bits = ~((ptl_match_bits_t) 0); + me.min_free = MPIDI_PTL_MAX_AM_EAGER_SZ; + + return PtlMEAppend(MPIDI_PTL_global.ni, MPIDI_PTL_global.pt, &me, PTL_OVERFLOW_LIST, (void *) i, + &MPIDI_PTL_global.overflow_me_handles[i]); +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_init +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_init(int rank, + int size, + int appnum, + int *tag_ub, + MPIR_Comm * comm_world, + MPIR_Comm * comm_self, + int spawned, int num_contexts, void **netmod_contexts) +{ + int mpi_errno = MPI_SUCCESS; + int ret; + ptl_md_t md; + ptl_ni_limits_t desired; + ptl_process_t my_ptl_id; + int key_max_sz; + int val_max_sz, val_sz_left; + int name_max_sz; + int len, i; + char *keyS, *valS, *buscard; + + /* Make sure our IOV is the same as portals4's IOV */ + MPIR_Assert(sizeof(ptl_iovec_t) == sizeof(MPL_IOV)); + MPIR_Assert(((void *) &(((ptl_iovec_t *) 0)->iov_base)) == + ((void *) &(((MPL_IOV *) 0)->MPL_IOV_BUF))); + MPIR_Assert(((void *) &(((ptl_iovec_t *) 0)->iov_len)) == + ((void *) &(((MPL_IOV *) 0)->MPL_IOV_LEN))); + MPIR_Assert(sizeof(((ptl_iovec_t *) 0)->iov_len) == sizeof(((MPL_IOV *) 0)->MPL_IOV_LEN)); + + /* init portals */ + ret = PtlInit(); + MPIDI_PTL_CHK_STATUS(ret, PtlInit); + + /* /\* do an interface pre-init to get the default limits struct *\/ */ + /* ret = PtlNIInit(PTL_IFACE_DEFAULT, PTL_NI_MATCHING | PTL_NI_PHYSICAL, */ + /* PTL_PID_ANY, NULL, &desired, &MPIDI_PTL_global.ni_handle); */ + /* MPIDI_PTL_CHK_STATUS(ret, PtlNIInit); */ + + /* /\* finalize the interface so we can re-init with our desired maximums *\/ */ + /* ret = PtlNIFini(MPIDI_PTL_global.ni_handle); */ + /* MPIDI_PTL_CHK_STATUS(ret, PtlNIFini); */ + + /* /\* set higher limits if they are determined to be too low *\/ */ + /* if (desired.max_unexpected_headers < UNEXPECTED_HDR_COUNT && getenv("PTL_LIM_MAX_UNEXPECTED_HEADERS") == NULL) */ + /* desired.max_unexpected_headers = UNEXPECTED_HDR_COUNT; */ + /* if (desired.max_list_size < LIST_SIZE && getenv("PTL_LIM_MAX_LIST_SIZE") == NULL) */ + /* desired.max_list_size = LIST_SIZE; */ + /* if (desired.max_entries < ENTRY_COUNT && getenv("PTL_LIM_MAX_ENTRIES") == NULL) */ + /* desired.max_entries = ENTRY_COUNT; */ + + /* do the real init */ + ret = PtlNIInit(PTL_IFACE_DEFAULT, PTL_NI_MATCHING | PTL_NI_PHYSICAL, + PTL_PID_ANY, NULL, &MPIDI_PTL_global.ni_limits, &MPIDI_PTL_global.ni); + MPIDI_PTL_CHK_STATUS(ret, PtlNIInit); + + /* allocate EQs: 0 is origin, 1 is target */ + ret = PtlEQAlloc(MPIDI_PTL_global.ni, MPIDI_PTL_EVENT_COUNT, &MPIDI_PTL_global.eqs[0]); + MPIDI_PTL_CHK_STATUS(ret, PtlEQAlloc); + ret = PtlEQAlloc(MPIDI_PTL_global.ni, MPIDI_PTL_EVENT_COUNT, &MPIDI_PTL_global.eqs[1]); + MPIDI_PTL_CHK_STATUS(ret, PtlEQAlloc); + + /* allocate portal */ + ret = + PtlPTAlloc(MPIDI_PTL_global.ni, + PTL_PT_ONLY_USE_ONCE | PTL_PT_ONLY_TRUNCATE | PTL_PT_FLOWCTRL, + MPIDI_PTL_global.eqs[1], PTL_PT_ANY, &MPIDI_PTL_global.pt); + MPIDI_PTL_CHK_STATUS(ret, PtlPTAlloc); + + /* create an MD that covers all of memory */ + md.start = NULL; + md.length = PTL_SIZE_MAX; + md.options = 0x0; + md.eq_handle = MPIDI_PTL_global.eqs[0]; + md.ct_handle = PTL_CT_NONE; + ret = PtlMDBind(MPIDI_PTL_global.ni, &md, &MPIDI_PTL_global.md); + MPIDI_PTL_CHK_STATUS(ret, PtlMDBind); + + /* create business card */ + ret = PMI_KVS_Get_key_length_max(&key_max_sz); + ret = PMI_KVS_Get_value_length_max(&val_max_sz); + ret = PMI_KVS_Get_name_length_max(&name_max_sz); + MPIDI_PTL_global.kvsname = MPL_malloc(name_max_sz); + ret = PMI_KVS_Get_my_name(MPIDI_PTL_global.kvsname, name_max_sz); + + keyS = MPL_malloc(key_max_sz); + valS = MPL_malloc(val_max_sz); + buscard = valS; + val_sz_left = val_max_sz; + + ret = PtlGetId(MPIDI_PTL_global.ni, &my_ptl_id); + ret = + MPL_str_add_binary_arg(&buscard, &val_sz_left, "NID", (char *) &my_ptl_id.phys.nid, + sizeof(my_ptl_id.phys.nid)); + ret = + MPL_str_add_binary_arg(&buscard, &val_sz_left, "PID", (char *) &my_ptl_id.phys.pid, + sizeof(my_ptl_id.phys.pid)); + ret = + MPL_str_add_binary_arg(&buscard, &val_sz_left, "PTI", (char *) &MPIDI_PTL_global.pt, + sizeof(MPIDI_PTL_global.pt)); + + sprintf(keyS, "PTL-%d", rank); + buscard = valS; + ret = PMI_KVS_Put(MPIDI_PTL_global.kvsname, keyS, buscard); + ret = PMI_KVS_Commit(MPIDI_PTL_global.kvsname); + ret = PMI_Barrier(); + + /* get and store business cards in address table */ + MPIDI_PTL_global.addr_table = MPL_malloc(size * sizeof(MPIDI_PTL_addr_t)); + for (i = 0; i < size; i++) { + sprintf(keyS, "PTL-%d", i); + ret = PMI_KVS_Get(MPIDI_PTL_global.kvsname, keyS, valS, val_max_sz); + MPL_str_get_binary_arg(valS, "NID", + (char *) &MPIDI_PTL_global.addr_table[i].process.phys.nid, + sizeof(MPIDI_PTL_global.addr_table[i].process.phys.nid), &len); + MPL_str_get_binary_arg(valS, "PID", + (char *) &MPIDI_PTL_global.addr_table[i].process.phys.pid, + sizeof(MPIDI_PTL_global.addr_table[i].process.phys.pid), &len); + MPL_str_get_binary_arg(valS, "PTI", (char *) &MPIDI_PTL_global.addr_table[i].pt, + sizeof(MPIDI_PTL_global.addr_table[i].pt), &len); + } + + /* Setup CH4R Active Messages */ + MPIDI_CH4U_init(comm_world, comm_self, num_contexts, netmod_contexts); + for (i = 0; i < MPIDI_PTL_NUM_OVERFLOW_BUFFERS; i++) { + MPIDI_PTL_global.overflow_bufs[i] = MPL_malloc(MPIDI_PTL_OVERFLOW_BUFFER_SZ); + MPIDI_PTL_append_overflow(i); + } + + MPIDI_PTL_global.node_map = MPL_malloc(size * sizeof(*MPIDI_PTL_global.node_map)); + mpi_errno = + MPIDI_CH4U_build_nodemap(rank, comm_world, size, MPIDI_PTL_global.node_map, + &MPIDI_PTL_global.max_node_id); + + fn_exit: + MPL_free(keyS); + MPL_free(valS); + + return mpi_errno; + fn_fail: + goto fn_exit; +} + +static inline int MPIDI_NM_finalize(void) +{ + int mpi_errno = MPI_SUCCESS; + int ret, i; + + MPIR_Comm_release(MPIR_Process.comm_world); + MPIR_Comm_release(MPIR_Process.comm_self); + + MPIDI_CH4U_finalize(); + + for (i = 0; i < MPIDI_PTL_NUM_OVERFLOW_BUFFERS; i++) { + ret = PtlMEUnlink(MPIDI_PTL_global.overflow_me_handles[i]); + MPL_free(MPIDI_PTL_global.overflow_bufs[i]); + } + ret = PtlMDRelease(MPIDI_PTL_global.md); + ret = PtlPTFree(MPIDI_PTL_global.ni, MPIDI_PTL_global.pt); + ret = PtlEQFree(MPIDI_PTL_global.eqs[1]); + ret = PtlEQFree(MPIDI_PTL_global.eqs[0]); + ret = PtlNIFini(MPIDI_PTL_global.ni); + PtlFini(); + + MPL_free(MPIDI_PTL_global.node_map); + MPL_free(MPIDI_PTL_global.addr_table); + MPL_free(MPIDI_PTL_global.kvsname); + + return mpi_errno; +} + + +static inline int MPIDI_NM_comm_get_lpid(MPIR_Comm * comm_ptr, + int idx, int *lpid_ptr, MPL_bool is_remote) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_gpid_get(MPIR_Comm * comm_ptr, int rank, MPIR_Gpid * gpid) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_get_node_id(MPIR_Comm * comm, int rank, MPID_Node_id_t * id_p) +{ + *id_p = MPIDI_PTL_global.node_map[rank]; + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_get_max_node_id(MPIR_Comm * comm, MPID_Node_id_t * max_id_p) +{ + *max_id_p = MPIDI_PTL_global.max_node_id; + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_getallincomm(MPIR_Comm * comm_ptr, + int local_size, MPIR_Gpid local_gpids[], int *singleAVT) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_gpid_tolpidarray(int size, MPIR_Gpid gpid[], int lpid[]) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_create_intercomm_from_lpids(MPIR_Comm * newcomm_ptr, + int size, const int lpids[]) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_free_mem(void *ptr) +{ + return MPIDI_CH4U_free_mem(ptr); +} + +static inline void *MPIDI_NM_alloc_mem(size_t size, MPIR_Info * info_ptr) +{ + return MPIDI_CH4U_alloc_mem(size, info_ptr); +} + + +#endif /* NETMOD_PTL_INIT_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/portals4/ptl_op.h b/src/mpid/ch4/netmod/portals4/ptl_op.h new file mode 100644 index 0000000..a202468 --- /dev/null +++ b/src/mpid/ch4/netmod/portals4/ptl_op.h @@ -0,0 +1,24 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2016 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + */ +#ifndef NETMOD_PTL_OP_H_INCLUDED +#define NETMOD_PTL_OP_H_INCLUDED + +#include "ptl_impl.h" + +static inline void MPIDI_NM_op_destroy(MPIR_Op * op_p) +{ + MPIR_Assert(0); + return; +} + +static inline void MPIDI_NM_op_commit(MPIR_Op * op_p) +{ + MPIR_Assert(0); + return; +} + +#endif /* NETMOD_PTL_OP_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/portals4/ptl_probe.h b/src/mpid/ch4/netmod/portals4/ptl_probe.h new file mode 100644 index 0000000..df4eae2 --- /dev/null +++ b/src/mpid/ch4/netmod/portals4/ptl_probe.h @@ -0,0 +1,39 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2012 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_PTL_PROBE_H_INCLUDED +#define NETMOD_PTL_PROBE_H_INCLUDED + +#include "ptl_impl.h" + +static inline int MPIDI_NM_probe(int source, + int tag, MPIR_Comm * comm, int context_offset, MPI_Status * status) +{ + return MPIDI_CH4U_probe(source, tag, comm, context_offset, status); +} + +static inline int MPIDI_NM_improbe(int source, + int tag, + MPIR_Comm * comm, + int context_offset, + int *flag, MPIR_Request ** message, MPI_Status * status) +{ + return MPIDI_CH4U_improbe(source, tag, comm, context_offset, flag, message, status); +} + +static inline int MPIDI_NM_iprobe(int source, + int tag, + MPIR_Comm * comm, + int context_offset, int *flag, MPI_Status * status) +{ + return MPIDI_CH4U_iprobe(source, tag, comm, context_offset, flag, status); +} + +#endif /* NETMOD_PTL_PROBE_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/portals4/ptl_proc.h b/src/mpid/ch4/netmod/portals4/ptl_proc.h new file mode 100644 index 0000000..9c59030 --- /dev/null +++ b/src/mpid/ch4/netmod/portals4/ptl_proc.h @@ -0,0 +1,29 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2012 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_PTL_PROC_H_INCLUDED +#define NETMOD_PTL_PROC_H_INCLUDED + +#include "ptl_impl.h" + +static inline int MPIDI_NM_rank_is_local(int rank, MPIR_Comm * comm) +{ + int ret; + MPIR_FUNC_VERBOSE_STATE_DECL(MPIDI_NETMOD_RANK_IS_LOCAL); + MPIR_FUNC_VERBOSE_ENTER(MPIDI_NETMOD_RANK_IS_LOCAL); + + MPIR_Assert(0); + ret = 0; + + MPIR_FUNC_VERBOSE_EXIT(MPIDI_NETMOD_RANK_IS_LOCAL); + return ret; +} + +#endif /* NETMOD_PTL_PROC_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/portals4/ptl_progress.h b/src/mpid/ch4/netmod/portals4/ptl_progress.h new file mode 100644 index 0000000..50163cb --- /dev/null +++ b/src/mpid/ch4/netmod/portals4/ptl_progress.h @@ -0,0 +1,185 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2012 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_PTL_PROGRESS_H_INCLUDED +#define NETMOD_PTL_PROGRESS_H_INCLUDED + +#include "ptl_impl.h" + +static inline int MPIDI_PTL_am_handler(ptl_event_t * e) +{ + int mpi_errno; + MPIR_Request *rreq = NULL; + void *p_data; + void *in_data; + size_t data_sz, in_data_sz; + MPIDI_NM_am_completion_handler_fn cmpl_handler_fn = NULL; + struct iovec *iov; + int i, is_contig, iov_len; + size_t done, curr_len, rem; + + in_data_sz = data_sz = (e->hdr_data & MPIDI_PTL_MSG_SZ_MASK); + in_data = p_data = (e->start + (e->mlength - data_sz)); + int handler_id = e->hdr_data >> 56; + + MPIDI_PTL_global.am_handlers[handler_id] (e->start, + &p_data, &data_sz, + &is_contig, &cmpl_handler_fn, &rreq); + + if (!rreq) + goto fn_exit; + + if ((!p_data || !data_sz) && cmpl_handler_fn) { + cmpl_handler_fn(rreq); + goto fn_exit; + } + + if (is_contig) { + if (in_data_sz > data_sz) { + rreq->status.MPI_ERROR = MPI_ERR_TRUNCATE; + } + else { + rreq->status.MPI_ERROR = MPI_SUCCESS; + } + + data_sz = MPL_MIN(data_sz, in_data_sz); + MPIR_Memcpy(p_data, in_data, data_sz); + MPIR_STATUS_SET_COUNT(rreq->status, data_sz); + } + else { + done = 0; + rem = in_data_sz; + iov = (struct iovec *) p_data; + iov_len = data_sz; + + for (i = 0; i < iov_len && rem > 0; i++) { + curr_len = MPL_MIN(rem, iov[i].iov_len); + MPIR_Memcpy(iov[i].iov_base, (char *) in_data + done, curr_len); + rem -= curr_len; + done += curr_len; + } + + if (rem) { + rreq->status.MPI_ERROR = MPI_ERR_TRUNCATE; + } + else { + rreq->status.MPI_ERROR = MPI_SUCCESS; + } + + MPIR_STATUS_SET_COUNT(rreq->status, done); + } + + if (cmpl_handler_fn) { + cmpl_handler_fn(rreq); + } + + fn_exit: + return mpi_errno; +} + +static inline int MPIDI_NM_progress(void *netmod_context, int blocking) +{ + ptl_event_t e; + unsigned int which; + + while (PtlEQPoll(MPIDI_PTL_global.eqs, 2, 0, &e, &which) != PTL_EQ_EMPTY) { + switch (e.type) { + case PTL_EVENT_PUT: + MPIR_Assert(e.ptl_list == PTL_OVERFLOW_LIST); + MPIDI_PTL_am_handler(&e); + break; + case PTL_EVENT_ACK: + { + int count; + MPIR_Request *sreq = (MPIR_Request *) e.user_ptr; + int handler_id = sreq->dev.ch4.ch4u.netmod_am.portals4.handler_id; + + MPIR_cc_decr(sreq->cc_ptr, &count); + MPIR_Assert(count >= 0); + + if (count == 0) { + MPIDI_CH4U_request_release(sreq); + break; + } + MPIDI_PTL_global.send_cmpl_handlers[handler_id] (sreq); + } + break; + case PTL_EVENT_AUTO_UNLINK: + MPIDI_PTL_global.overflow_me_handles[(size_t) e.user_ptr] = PTL_INVALID_HANDLE; + break; + case PTL_EVENT_AUTO_FREE: + MPIDI_PTL_append_overflow((size_t) e.user_ptr); + break; + case PTL_EVENT_SEND: + break; + default: + printf("ABORT: event = %d\n", e.type); + abort(); + } + } + + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_progress_test(void) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_progress_poke(void) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline void MPIDI_NM_progress_start(MPID_Progress_state * state) +{ + MPIR_Assert(0); + return; +} + +static inline void MPIDI_NM_progress_end(MPID_Progress_state * state) +{ + MPIR_Assert(0); + return; +} + +static inline int MPIDI_NM_progress_wait(MPID_Progress_state * state) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_progress_register(int (*progress_fn) (int *), int *id) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_progress_deregister(int id) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_progress_activate(int id) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_progress_deactivate(int id) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +#endif /* NETMOD_PTL_PROGRESS_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/portals4/ptl_recv.h b/src/mpid/ch4/netmod/portals4/ptl_recv.h new file mode 100644 index 0000000..a239a3a --- /dev/null +++ b/src/mpid/ch4/netmod/portals4/ptl_recv.h @@ -0,0 +1,63 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2012 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_PTL_RECV_H_INCLUDED +#define NETMOD_PTL_RECV_H_INCLUDED + +#include "ptl_impl.h" + +static inline int MPIDI_NM_recv(void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, + int context_offset, MPI_Status * status, MPIR_Request ** request) +{ + return MPIDI_CH4U_recv(buf, count, datatype, rank, tag, comm, context_offset, status, request); +} + +static inline int MPIDI_NM_recv_init(void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_imrecv(void *buf, + int count, + MPI_Datatype datatype, + MPIR_Request * message, MPIR_Request ** rreqp) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_irecv(void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + return MPIDI_CH4U_irecv(buf, count, datatype, rank, tag, comm, context_offset, request); +} + +static inline int MPIDI_NM_cancel_recv(MPIR_Request * rreq) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +#endif /* NETMOD_PTL_RECV_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/portals4/ptl_request.h b/src/mpid/ch4/netmod/portals4/ptl_request.h new file mode 100644 index 0000000..0e4a796 --- /dev/null +++ b/src/mpid/ch4/netmod/portals4/ptl_request.h @@ -0,0 +1,32 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2012 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_PTL_REQUEST_H_INCLUDED +#define NETMOD_PTL_REQUEST_H_INCLUDED + +#include "ptl_impl.h" + +static inline void MPIDI_NM_am_request_init(MPIR_Request * req) +{ + req->dev.ch4.ch4u.netmod_am.portals4.pack_buffer = NULL; + req->dev.ch4.ch4u.netmod_am.portals4.md = PTL_INVALID_HANDLE; +} + +static inline void MPIDI_NM_am_request_finalize(MPIR_Request * req) +{ + if ((req)->dev.ch4.ch4u.netmod_am.portals4.pack_buffer) { + MPL_free((req)->dev.ch4.ch4u.netmod_am.portals4.pack_buffer); + } + if ((req)->dev.ch4.ch4u.netmod_am.portals4.md != PTL_INVALID_HANDLE) { + PtlMDRelease((req)->dev.ch4.ch4u.netmod_am.portals4.md); + } +} + +#endif /* NETMOD_PTL_REQUEST_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/portals4/ptl_rma.h b/src/mpid/ch4/netmod/portals4/ptl_rma.h new file mode 100644 index 0000000..8be0450 --- /dev/null +++ b/src/mpid/ch4/netmod/portals4/ptl_rma.h @@ -0,0 +1,148 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2012 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_PTL_RMA_H_INCLUDED +#define NETMOD_PTL_RMA_H_INCLUDED + +#include "ptl_impl.h" + +static inline int MPIDI_NM_put(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, MPI_Datatype target_datatype, MPIR_Win * win) +{ + return MPIDI_CH4U_put(origin_addr, origin_count, origin_datatype, + target_rank, target_disp, target_count, target_datatype, win); +} + +static inline int MPIDI_NM_get(void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, MPI_Datatype target_datatype, MPIR_Win * win) +{ + return MPIDI_CH4U_get(origin_addr, origin_count, origin_datatype, + target_rank, target_disp, target_count, target_datatype, win); +} + +static inline int MPIDI_NM_rput(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPIR_Win * win, MPIR_Request ** request) +{ + return MPIDI_CH4U_rput(origin_addr, origin_count, origin_datatype, + target_rank, target_disp, target_count, target_datatype, win, request); +} + + +static inline int MPIDI_NM_compare_and_swap(const void *origin_addr, + const void *compare_addr, + void *result_addr, + MPI_Datatype datatype, + int target_rank, MPI_Aint target_disp, MPIR_Win * win) +{ + return MPIDI_CH4U_compare_and_swap(origin_addr, compare_addr, result_addr, + datatype, target_rank, target_disp, win); +} + +static inline int MPIDI_NM_raccumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPI_Op op, MPIR_Win * win, MPIR_Request ** request) +{ + return MPIDI_CH4U_raccumulate(origin_addr, origin_count, origin_datatype, + target_rank, target_disp, target_count, + target_datatype, op, win, request); +} + +static inline int MPIDI_NM_rget_accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + void *result_addr, + int result_count, + MPI_Datatype result_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPI_Op op, MPIR_Win * win, MPIR_Request ** request) +{ + return MPIDI_CH4U_rget_accumulate(origin_addr, origin_count, origin_datatype, + result_addr, result_count, result_datatype, + target_rank, target_disp, target_count, + target_datatype, op, win, request); +} + +static inline int MPIDI_NM_fetch_and_op(const void *origin_addr, + void *result_addr, + MPI_Datatype datatype, + int target_rank, + MPI_Aint target_disp, MPI_Op op, MPIR_Win * win) +{ + return MPIDI_CH4U_fetch_and_op(origin_addr, result_addr, datatype, + target_rank, target_disp, op, win); +} + + +static inline int MPIDI_NM_rget(void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPIR_Win * win, MPIR_Request ** request) +{ + return MPIDI_CH4U_rget(origin_addr, origin_count, origin_datatype, + target_rank, target_disp, target_count, target_datatype, win, request); +} + + +static inline int MPIDI_NM_get_accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + void *result_addr, + int result_count, + MPI_Datatype result_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, MPI_Op op, MPIR_Win * win) +{ + return MPIDI_CH4U_get_accumulate(origin_addr, origin_count, origin_datatype, + result_addr, result_count, result_datatype, + target_rank, target_disp, target_count, + target_datatype, op, win); +} + +static inline int MPIDI_NM_accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, MPI_Op op, MPIR_Win * win) +{ + return MPIDI_CH4U_accumulate(origin_addr, origin_count, origin_datatype, + target_rank, target_disp, target_count, target_datatype, op, win); +} + +#endif /* NETMOD_PTL_RMA_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/portals4/ptl_send.h b/src/mpid/ch4/netmod/portals4/ptl_send.h new file mode 100644 index 0000000..d910222 --- /dev/null +++ b/src/mpid/ch4/netmod/portals4/ptl_send.h @@ -0,0 +1,128 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2012 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_PTL_SEND_H_INCLUDED +#define NETMOD_PTL_SEND_H_INCLUDED + +#include "ptl_impl.h" + +static inline int MPIDI_NM_send(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + return MPIDI_CH4U_send(buf, count, datatype, rank, tag, comm, context_offset, request); +} + +static inline int MPIDI_NM_rsend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + return MPIDI_CH4U_rsend(buf, count, datatype, rank, tag, comm, context_offset, request); +} + + + +static inline int MPIDI_NM_irsend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + return MPIDI_CH4U_irsend(buf, count, datatype, rank, tag, comm, context_offset, request); +} + +static inline int MPIDI_NM_ssend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + return MPIDI_CH4U_ssend(buf, count, datatype, rank, tag, comm, context_offset, request); +} + +static inline int MPIDI_NM_startall(int count, MPIR_Request * requests[]) +{ + return MPIDI_CH4U_startall(count, requests); +} + +static inline int MPIDI_NM_send_init(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + return MPIDI_CH4U_send_init(buf, count, datatype, rank, tag, comm, context_offset, request); +} + +static inline int MPIDI_NM_ssend_init(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + return MPIDI_CH4U_ssend_init(buf, count, datatype, rank, tag, comm, context_offset, request); +} + +static inline int MPIDI_NM_bsend_init(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + return MPIDI_CH4U_bsend_init(buf, count, datatype, rank, tag, comm, context_offset, request); +} + +static inline int MPIDI_NM_rsend_init(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + return MPIDI_CH4U_rsend_init(buf, count, datatype, rank, tag, comm, context_offset, request); +} + +static inline int MPIDI_NM_isend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + return MPIDI_CH4U_isend(buf, count, datatype, rank, tag, comm, context_offset, request); +} + +static inline int MPIDI_NM_issend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + return MPIDI_CH4U_issend(buf, count, datatype, rank, tag, comm, context_offset, request); +} + +static inline int MPIDI_NM_cancel_send(MPIR_Request * sreq) +{ + return MPIDI_CH4U_cancel_send(sreq); +} + +#endif /* NETMOD_PTL_SEND_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/portals4/ptl_spawn.h b/src/mpid/ch4/netmod/portals4/ptl_spawn.h new file mode 100644 index 0000000..b0ef76a --- /dev/null +++ b/src/mpid/ch4/netmod/portals4/ptl_spawn.h @@ -0,0 +1,50 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2012 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_PTL_SPAWN_H_INCLUDED +#define NETMOD_PTL_SPAWN_H_INCLUDED + +#include "ptl_impl.h" + +static inline int MPIDI_NM_comm_connect(const char *port_name, + MPIR_Info * info, + int root, MPIR_Comm * comm, MPIR_Comm ** newcomm_ptr) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_comm_disconnect(MPIR_Comm * comm_ptr) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_open_port(MPIR_Info * info_ptr, char *port_name) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_close_port(const char *port_name) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_comm_accept(const char *port_name, + MPIR_Info * info, + int root, MPIR_Comm * comm, MPIR_Comm ** newcomm_ptr) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +#endif /* NETMOD_PTL_SPAWN_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/portals4/ptl_types.h b/src/mpid/ch4/netmod/portals4/ptl_types.h new file mode 100644 index 0000000..be00d9a --- /dev/null +++ b/src/mpid/ch4/netmod/portals4/ptl_types.h @@ -0,0 +1,76 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * * (C) 2016 by Argonne National Laboratory. + * * See COPYRIGHT in top-level directory. + * * + * */ + +#ifndef NETMOD_PTL_TYPES_H_INCLUDED +#define NETMOD_PTL_TYPES_H_INCLUDED + +#include "mpidimpl.h" +#include "portals4.h" + +/* Portals 4 Limits */ +#define MPIDI_PTL_EVENT_COUNT (1024*64) +#define MPIDI_PTL_UNEXPECTED_HDR_COUNT (1024*64) +#define MPIDI_PTL_LIST_SIZE (1024*64) + +/* Active Message Stuff */ +#define MPIDI_PTL_NUM_OVERFLOW_BUFFERS (8) +#define MPIDI_PTL_OVERFLOW_BUFFER_SZ (1024*1024) +#define MPIDI_PTL_MAX_AM_EAGER_SZ (64*1024) +#define MPIDI_PTL_AM_TAG (1 << 28) +#define MPIDI_PTL_MAX_AM_HANDLERS (64) + +typedef struct { + ptl_process_t process; + ptl_pt_index_t pt; +} MPIDI_PTL_addr_t; + +typedef struct { + MPIDI_PTL_addr_t *addr_table; + MPID_Node_id_t *node_map; + MPID_Node_id_t max_node_id; + char *kvsname; + char pname[MPI_MAX_PROCESSOR_NAME]; + void *overflow_bufs[MPIDI_PTL_NUM_OVERFLOW_BUFFERS]; + ptl_handle_me_t overflow_me_handles[MPIDI_PTL_NUM_OVERFLOW_BUFFERS]; + MPIDI_NM_am_target_handler_fn am_handlers[MPIDI_PTL_MAX_AM_HANDLERS]; + MPIDI_NM_am_origin_handler_fn send_cmpl_handlers[MPIDI_PTL_MAX_AM_HANDLERS]; + ptl_handle_ni_t ni; + ptl_ni_limits_t ni_limits; + ptl_handle_eq_t eqs[2]; + ptl_pt_index_t pt; + ptl_handle_md_t md; +} MPIDI_PTL_global_t; + +extern MPIDI_PTL_global_t MPIDI_PTL_global; + +#define MPIDI_PTL_CONTEXT_ID_BITS 32 +#define MPIDI_PTL_TAG_BITS 32 + +#define MPIDI_PTL_TAG_MASK (0x00000000FFFFFFFFULL) +#define MPIDI_PTL_CTX_MASK (0xFFFFFFFF00000000ULL) +#define MPIDI_PTL_TAG_SHIFT (MPIDI_PTL_TAG_BITS) + +static inline ptl_match_bits_t MPIDI_PTL_init_tag(MPIR_Context_id_t contextid, int tag) +{ + ptl_match_bits_t match_bits = 0; + match_bits = contextid; + match_bits <<= MPIDI_PTL_TAG_SHIFT; + match_bits |= (MPIDI_PTL_TAG_MASK & tag); + return match_bits; +} + +#define MPIDI_PTL_MSG_SZ_MASK (0x00FFFFFFFFFFFFFFULL) + +static inline ptl_hdr_data_t MPIDI_PTL_init_am_hdr(int handler_id, size_t msg_sz) +{ + ptl_hdr_data_t hdr = 0; + hdr = (ptl_hdr_data_t) handler_id << 56; + hdr |= (MPIDI_PTL_MSG_SZ_MASK & msg_sz); + return hdr; +} + +#endif /* NETMOD_PTL_TYPES_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/portals4/ptl_unimpl.h b/src/mpid/ch4/netmod/portals4/ptl_unimpl.h new file mode 100644 index 0000000..2b0cd20 --- /dev/null +++ b/src/mpid/ch4/netmod/portals4/ptl_unimpl.h @@ -0,0 +1,19 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2012 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ + +#ifdef __cplusplus +extern "C" { +#endif + + +#ifdef __cplusplus +} +#endif diff --git a/src/mpid/ch4/netmod/portals4/ptl_win.h b/src/mpid/ch4/netmod/portals4/ptl_win.h new file mode 100644 index 0000000..c325315 --- /dev/null +++ b/src/mpid/ch4/netmod/portals4/ptl_win.h @@ -0,0 +1,160 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2012 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_PTL_WIN_H_INCLUDED +#define NETMOD_PTL_WIN_H_INCLUDED + +#include "ptl_impl.h" + +static inline int MPIDI_NM_win_set_info(MPIR_Win * win, MPIR_Info * info) +{ + return MPIDI_CH4R_win_set_info(win, info); +} + + +static inline int MPIDI_NM_win_start(MPIR_Group * group, int assert, MPIR_Win * win) +{ + return MPIDI_CH4R_win_start(group, assert, win); +} + + +static inline int MPIDI_NM_win_complete(MPIR_Win * win) +{ + return MPIDI_CH4R_win_complete(win); +} + +static inline int MPIDI_NM_win_post(MPIR_Group * group, int assert, MPIR_Win * win) +{ + return MPIDI_CH4R_win_post(group, assert, win); +} + + +static inline int MPIDI_NM_win_wait(MPIR_Win * win) +{ + return MPIDI_CH4R_win_wait(win); +} + + +static inline int MPIDI_NM_win_test(MPIR_Win * win, int *flag) +{ + return MPIDI_CH4R_win_test(win, flag); +} + +static inline int MPIDI_NM_win_lock(int lock_type, int rank, int assert, MPIR_Win * win) +{ + return MPIDI_CH4R_win_lock(lock_type, rank, assert, win); +} + + +static inline int MPIDI_NM_win_unlock(int rank, MPIR_Win * win) +{ + return MPIDI_CH4R_win_unlock(rank, win); +} + +static inline int MPIDI_NM_win_get_info(MPIR_Win * win, MPIR_Info ** info_p_p) +{ + return MPIDI_CH4R_win_get_info(win, info_p_p); +} + + +static inline int MPIDI_NM_win_free(MPIR_Win ** win_ptr) +{ + return MPIDI_CH4R_win_free(win_ptr); +} + +static inline int MPIDI_NM_win_fence(int assert, MPIR_Win * win) +{ + return MPIDI_CH4R_win_fence(assert, win); +} + +static inline int MPIDI_NM_win_create(void *base, + MPI_Aint length, + int disp_unit, + MPIR_Info * info, MPIR_Comm * comm_ptr, MPIR_Win ** win_ptr) +{ + return MPIDI_CH4R_win_create(base, length, disp_unit, info, comm_ptr, win_ptr); +} + +static inline int MPIDI_NM_win_attach(MPIR_Win * win, void *base, MPI_Aint size) +{ + return MPIDI_CH4R_win_attach(win, base, size); +} + +static inline int MPIDI_NM_win_allocate_shared(MPI_Aint size, + int disp_unit, + MPIR_Info * info_ptr, + MPIR_Comm * comm_ptr, + void **base_ptr, MPIR_Win ** win_ptr) +{ + return MPIDI_CH4R_win_allocate_shared(size, disp_unit, info_ptr, comm_ptr, base_ptr, win_ptr); +} + +static inline int MPIDI_NM_win_detach(MPIR_Win * win, const void *base) +{ + return MPIDI_CH4R_win_detach(win, base); +} + +static inline int MPIDI_NM_win_shared_query(MPIR_Win * win, + int rank, + MPI_Aint * size, int *disp_unit, void *baseptr) +{ + return MPIDI_CH4R_win_shared_query(win, rank, size, disp_unit, baseptr); +} + +static inline int MPIDI_NM_win_allocate(MPI_Aint size, + int disp_unit, + MPIR_Info * info, + MPIR_Comm * comm, void *baseptr, MPIR_Win ** win) +{ + return MPIDI_CH4R_win_allocate(size, disp_unit, info, comm, baseptr, win); +} + +static inline int MPIDI_NM_win_flush(int rank, MPIR_Win * win) +{ + return MPIDI_CH4R_win_flush(rank, win); +} + +static inline int MPIDI_NM_win_flush_local_all(MPIR_Win * win) +{ + return MPIDI_CH4R_win_flush_local_all(win); +} + +static inline int MPIDI_NM_win_unlock_all(MPIR_Win * win) +{ + return MPIDI_CH4R_win_unlock_all(win); +} + +static inline int MPIDI_NM_win_create_dynamic(MPIR_Info * info, MPIR_Comm * comm, MPIR_Win ** win) +{ + return MPIDI_CH4R_win_create_dynamic(info, comm, win); +} + +static inline int MPIDI_NM_win_flush_local(int rank, MPIR_Win * win) +{ + return MPIDI_CH4R_win_flush_local(rank, win); +} + +static inline int MPIDI_NM_win_sync(MPIR_Win * win) +{ + return MPIDI_CH4R_win_sync(win); +} + +static inline int MPIDI_NM_win_flush_all(MPIR_Win * win) +{ + return MPIDI_CH4R_win_flush_all(win); +} + +static inline int MPIDI_NM_win_lock_all(int assert, MPIR_Win * win) +{ + return MPIDI_CH4R_win_lock_all(assert, win); +} + + +#endif /* NETMOD_PTL_WIN_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/portals4/subconfigure.m4 b/src/mpid/ch4/netmod/portals4/subconfigure.m4 new file mode 100644 index 0000000..21a60b1 --- /dev/null +++ b/src/mpid/ch4/netmod/portals4/subconfigure.m4 @@ -0,0 +1,33 @@ +[#] start of __file__ +dnl MPICH_SUBCFG_AFTER=src/mpid/ch4 + +AC_DEFUN([PAC_SUBCFG_PREREQ_]PAC_SUBCFG_AUTO_SUFFIX,[ + AM_COND_IF([BUILD_CH4],[ + for net in $ch4_netmods ; do + AS_CASE([$net],[portals4],[build_ch4_netmod_portals4=yes]) + if test $net = "portals4" ; then + AC_DEFINE(HAVE_CH4_NETMOD_PORTALS4,1,[Portals4 netmod is built]) + # if test "$build_ch4_locality_info" != "yes" ; then + # AC_DEFINE(MPIDI_BUILD_CH4_LOCALITY_INFO, 1, [CH4 should build locality info]) + # build_ch4_locality_info="yes" + # fi + fi + done + ]) + AM_CONDITIONAL([BUILD_CH4_NETMOD_PORTALS4],[test "X$build_ch4_netmod_portals4" = "Xyes"]) +])dnl + +AC_DEFUN([PAC_SUBCFG_BODY_]PAC_SUBCFG_AUTO_SUFFIX,[ +AM_COND_IF([BUILD_CH4_NETMOD_PORTALS4],[ + AC_MSG_NOTICE([RUNNING CONFIGURE FOR ch4:portals4]) + + PAC_SET_HEADER_LIB_PATH(portals4) + PAC_PUSH_FLAG(LIBS) + PAC_CHECK_HEADER_LIB_FATAL(portals4, portals4.h, portals, PtlInit) + PAC_APPEND_FLAG([-lportals],[WRAPPER_LIBS]) + PAC_POP_FLAG(LIBS) + +])dnl end AM_COND_IF(BUILD_CH4_NETMOD_PORTALS4,...) +])dnl end _BODY + +[#] end of __file__ diff --git a/src/mpid/ch4/netmod/stubnm/Makefile.mk b/src/mpid/ch4/netmod/stubnm/Makefile.mk new file mode 100644 index 0000000..244ba01 --- /dev/null +++ b/src/mpid/ch4/netmod/stubnm/Makefile.mk @@ -0,0 +1,6 @@ +if BUILD_CH4_NETMOD_STUBNM + +mpi_core_sources += src/mpid/ch4/netmod/stubnm/globals.c +# errnames_txt_files += src/mpid/ch4/netmod/stub/errnames.txt + +endif diff --git a/src/mpid/ch4/netmod/stubnm/globals.c b/src/mpid/ch4/netmod/stubnm/globals.c new file mode 100644 index 0000000..b830d4b --- /dev/null +++ b/src/mpid/ch4/netmod/stubnm/globals.c @@ -0,0 +1,157 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ + +#ifndef NETMOD_DIRECT +#define NETMOD_DISABLE_INLINES +#include <mpidimpl.h> +#include "netmod_direct.h" +MPIDI_NM_funcs_t MPIDI_NM_stubnm_funcs = { + MPIDI_NM_init, + MPIDI_NM_finalize, + MPIDI_NM_progress, + MPIDI_NM_comm_connect, + MPIDI_NM_comm_disconnect, + MPIDI_NM_open_port, + MPIDI_NM_close_port, + MPIDI_NM_comm_accept, + MPIDI_NM_comm_get_lpid, + MPIDI_NM_gpid_get, + MPIDI_NM_get_node_id, + MPIDI_NM_get_max_node_id, + MPIDI_NM_getallincomm, + MPIDI_NM_gpid_tolpidarray, + MPIDI_NM_create_intercomm_from_lpids, + MPIDI_NM_comm_create, + MPIDI_NM_comm_destroy, + MPIDI_NM_am_request_init, + MPIDI_NM_am_request_finalize, + MPIDI_NM_reg_hdr_handler, + MPIDI_NM_send_am_hdr, + MPIDI_NM_inject_am_hdr, + MPIDI_NM_send_am, + MPIDI_NM_send_amv, + MPIDI_NM_send_amv_hdr, + MPIDI_NM_send_am_hdr_reply, + MPIDI_NM_inject_am_hdr_reply, + MPIDI_NM_send_am_reply, + MPIDI_NM_send_amv_reply, + MPIDI_NM_am_hdr_max_sz, + MPIDI_NM_am_inject_max_sz, + MPIDI_NM_am_recv, +}; + +MPIDI_NM_native_funcs_t MPIDI_NM_native_stubnm_funcs = { + MPIDI_NM_send, + MPIDI_NM_ssend, + MPIDI_NM_startall, + MPIDI_NM_send_init, + MPIDI_NM_ssend_init, + MPIDI_NM_rsend_init, + MPIDI_NM_bsend_init, + MPIDI_NM_isend, + MPIDI_NM_issend, + MPIDI_NM_cancel_send, + MPIDI_NM_recv_init, + MPIDI_NM_recv, + MPIDI_NM_irecv, + MPIDI_NM_imrecv, + MPIDI_NM_cancel_recv, + MPIDI_NM_alloc_mem, + MPIDI_NM_free_mem, + MPIDI_NM_improbe, + MPIDI_NM_iprobe, + MPIDI_NM_win_set_info, + MPIDI_NM_win_shared_query, + MPIDI_NM_put, + MPIDI_NM_win_start, + MPIDI_NM_win_complete, + MPIDI_NM_win_post, + MPIDI_NM_win_wait, + MPIDI_NM_win_test, + MPIDI_NM_win_lock, + MPIDI_NM_win_unlock, + MPIDI_NM_win_get_info, + MPIDI_NM_get, + MPIDI_NM_win_free, + MPIDI_NM_win_fence, + MPIDI_NM_win_create, + MPIDI_NM_accumulate, + MPIDI_NM_win_attach, + MPIDI_NM_win_allocate_shared, + MPIDI_NM_rput, + MPIDI_NM_win_flush_local, + MPIDI_NM_win_detach, + MPIDI_NM_compare_and_swap, + MPIDI_NM_raccumulate, + MPIDI_NM_rget_accumulate, + MPIDI_NM_fetch_and_op, + MPIDI_NM_win_allocate, + MPIDI_NM_win_flush, + MPIDI_NM_win_flush_local_all, + MPIDI_NM_win_unlock_all, + MPIDI_NM_win_create_dynamic, + MPIDI_NM_rget, + MPIDI_NM_win_sync, + MPIDI_NM_win_flush_all, + MPIDI_NM_get_accumulate, + MPIDI_NM_win_lock_all, + MPIDI_NM_rank_is_local, + MPIDI_NM_barrier, + MPIDI_NM_bcast, + MPIDI_NM_allreduce, + MPIDI_NM_allgather, + MPIDI_NM_allgatherv, + MPIDI_NM_scatter, + MPIDI_NM_scatterv, + MPIDI_NM_gather, + MPIDI_NM_gatherv, + MPIDI_NM_alltoall, + MPIDI_NM_alltoallv, + MPIDI_NM_alltoallw, + MPIDI_NM_reduce, + MPIDI_NM_reduce_scatter, + MPIDI_NM_reduce_scatter_block, + MPIDI_NM_scan, + MPIDI_NM_exscan, + MPIDI_NM_neighbor_allgather, + MPIDI_NM_neighbor_allgatherv, + MPIDI_NM_neighbor_alltoall, + MPIDI_NM_neighbor_alltoallv, + MPIDI_NM_neighbor_alltoallw, + MPIDI_NM_ineighbor_allgather, + MPIDI_NM_ineighbor_allgatherv, + MPIDI_NM_ineighbor_alltoall, + MPIDI_NM_ineighbor_alltoallv, + MPIDI_NM_ineighbor_alltoallw, + MPIDI_NM_ibarrier, + MPIDI_NM_ibcast, + MPIDI_NM_iallgather, + MPIDI_NM_iallgatherv, + MPIDI_NM_iallreduce, + MPIDI_NM_ialltoall, + MPIDI_NM_ialltoallv, + MPIDI_NM_ialltoallw, + MPIDI_NM_iexscan, + MPIDI_NM_igather, + MPIDI_NM_igatherv, + MPIDI_NM_ireduce_scatter_block, + MPIDI_NM_ireduce_scatter, + MPIDI_NM_ireduce, + MPIDI_NM_iscan, + MPIDI_NM_iscatter, + MPIDI_NM_iscatterv, + MPIDI_NM_datatype_commit, + MPIDI_NM_datatype_dup, + MPIDI_NM_datatype_destroy, + MPIDI_NM_op_commit, + MPIDI_NM_op_destroy, +}; +#endif diff --git a/src/mpid/ch4/netmod/stubnm/netmod_direct.h b/src/mpid/ch4/netmod/stubnm/netmod_direct.h new file mode 100644 index 0000000..f2873ad --- /dev/null +++ b/src/mpid/ch4/netmod/stubnm/netmod_direct.h @@ -0,0 +1,29 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_DIRECT_H_INCLUDED +#define NETMOD_DIRECT_H_INCLUDED +#include "stubnm_init.h" +#include "stubnm_probe.h" +#include "stubnm_progress.h" +#include "stubnm_recv.h" +#include "stubnm_request.h" +#include "stubnm_send.h" +#include "stubnm_win.h" +#include "stubnm_rma.h" +#include "stubnm_am.h" +#include "stubnm_spawn.h" +#include "stubnm_comm.h" +#include "stubnm_unimpl.h" +#include "stubnm_proc.h" +#include "stubnm_coll.h" +#include "stubnm_datatype.h" +#include "stubnm_op.h" +#endif /* NETMOD_DIRECT_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/stubnm/stubnm_am.h b/src/mpid/ch4/netmod/stubnm/stubnm_am.h new file mode 100644 index 0000000..2d7d9f2 --- /dev/null +++ b/src/mpid/ch4/netmod/stubnm/stubnm_am.h @@ -0,0 +1,138 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_STUBNM_AM_H_INCLUDED +#define NETMOD_STUBNM_AM_H_INCLUDED + +#include "stubnm_impl.h" + +static inline int MPIDI_NM_reg_hdr_handler(int handler_id, + MPIDI_NM_am_origin_handler_fn origin_handler_fn, + MPIDI_NM_am_target_handler_fn target_handler_fn) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_send_am_hdr(int rank, + MPIR_Comm * comm, + int handler_id, + const void *am_hdr, + size_t am_hdr_sz, MPIR_Request * sreq, void *netmod_context) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_send_am(int rank, + MPIR_Comm * comm, + int handler_id, + const void *am_hdr, + size_t am_hdr_sz, + const void *data, + MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq, void *netmod_context) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_send_amv(int rank, + MPIR_Comm * comm, + int handler_id, + struct iovec *am_hdr, + size_t iov_len, + const void *data, + MPI_Count count, + MPI_Datatype datatype, + MPIR_Request * sreq, void *netmod_context) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_send_amv_hdr(int rank, + MPIR_Comm * comm, + int handler_id, + struct iovec *am_hdr, + size_t iov_len, MPIR_Request * sreq, void *netmod_context) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_send_am_hdr_reply(MPIR_Context_id_t context_id, int src_rank, + int handler_id, + const void *am_hdr, + size_t am_hdr_sz, MPIR_Request * sreq) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_send_am_reply(MPIR_Context_id_t context_id, int src_rank, + int handler_id, + const void *am_hdr, + size_t am_hdr_sz, + const void *data, + MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_send_amv_reply(MPIR_Context_id_t context_id, int src_rank, + int handler_id, + struct iovec *am_hdr, + size_t iov_len, + const void *data, + MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline size_t MPIDI_NM_am_hdr_max_sz(void) +{ + MPIR_Assert(0); + return 0; +} + +static inline int MPIDI_NM_inject_am_hdr(int rank, + MPIR_Comm * comm, + int handler_id, + const void *am_hdr, size_t am_hdr_sz, void *netmod_context) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_inject_am_hdr_reply(MPIR_Context_id_t context_id, int src_rank, + int handler_id, const void *am_hdr, size_t am_hdr_sz) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline size_t MPIDI_NM_am_inject_max_sz(void) +{ + MPIR_Assert(0); + return 0; +} + +static inline int MPIDI_NM_am_recv(MPIR_Request * req) +{ + MPIR_Assert(0); + return 0; +} + +#endif /* NETMOD_STUBNM_AM_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/stubnm/stubnm_coll.h b/src/mpid/ch4/netmod/stubnm/stubnm_coll.h new file mode 100644 index 0000000..cf76ab6 --- /dev/null +++ b/src/mpid/ch4/netmod/stubnm/stubnm_coll.h @@ -0,0 +1,871 @@ + +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_STUBNM_COLL_H_INCLUDED +#define NETMOD_STUBNM_COLL_H_INCLUDED + +#include "stubnm_impl.h" + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_barrier +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_barrier(MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_BARRIER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_BARRIER); + + mpi_errno = MPIR_Barrier(comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_BARRIER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_bcast +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_bcast(void *buffer, int count, MPI_Datatype datatype, + int root, MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_BCAST); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_BCAST); + + mpi_errno = MPIR_Bcast(buffer, count, datatype, root, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_BCAST); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_allreduce +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_allreduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_ALLREDUCE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_ALLREDUCE); + + mpi_errno = MPIR_Allreduce(sendbuf, recvbuf, count, datatype, op, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_ALLREDUCE); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_allgather +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_allgather(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_ALLGATHER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_ALLGATHER); + + mpi_errno = MPIR_Allgather(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, + comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_ALLGATHER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_allgatherv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_allgatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, const int *displs, + MPI_Datatype recvtype, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_ALLGATHERV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_ALLGATHERV); + + mpi_errno = MPIR_Allgatherv(sendbuf, sendcount, sendtype, + recvbuf, recvcounts, displs, recvtype, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_ALLGATHERV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_gather +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_gather(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + int root, MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_GATHER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_GATHER); + + mpi_errno = MPIR_Gather(sendbuf, sendcount, sendtype, recvbuf, recvcount, + recvtype, root, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_GATHER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_gatherv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_gatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, const int *displs, + MPI_Datatype recvtype, int root, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_GATHERV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_GATHERV); + + mpi_errno = MPIR_Gatherv(sendbuf, sendcount, sendtype, + recvbuf, recvcounts, displs, recvtype, root, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_GATHERV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_scatter +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_scatter(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + int root, MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_SCATTER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_SCATTER); + + mpi_errno = MPIR_Scatter(sendbuf, sendcount, sendtype, + recvbuf, recvcount, recvtype, root, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_SCATTER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_scatterv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_scatterv(const void *sendbuf, const int *sendcounts, + const int *displs, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + int root, MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_SCATTERV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_SCATTERV); + + mpi_errno = MPIR_Scatterv(sendbuf, sendcounts, displs, + sendtype, recvbuf, recvcount, recvtype, root, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_SCATTERV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_alltoall +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_alltoall(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_ALLTOALL); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_ALLTOALL); + + mpi_errno = MPIR_Alltoall(sendbuf, sendcount, sendtype, recvbuf, recvcount, + recvtype, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_ALLTOALL); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_alltoallv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_alltoallv(const void *sendbuf, const int *sendcounts, + const int *sdispls, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, + const int *rdispls, MPI_Datatype recvtype, + MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_ALLTOALLV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_ALLTOALLV); + + mpi_errno = MPIR_Alltoallv(sendbuf, sendcounts, sdispls, + sendtype, recvbuf, recvcounts, rdispls, recvtype, comm_ptr, errflag); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_ALLTOALLV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_alltoallw +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_alltoallw(const void *sendbuf, const int sendcounts[], + const int sdispls[], const MPI_Datatype sendtypes[], + void *recvbuf, const int recvcounts[], + const int rdispls[], const MPI_Datatype recvtypes[], + MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_ALLTOALLW); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_ALLTOALLW); + + mpi_errno = MPIR_Alltoallw(sendbuf, sendcounts, sdispls, + sendtypes, recvbuf, recvcounts, + rdispls, recvtypes, comm_ptr, errflag); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_ALLTOALLW); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_reduce +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_reduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, int root, + MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_REDUCE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_REDUCE); + + mpi_errno = MPIR_Reduce(sendbuf, recvbuf, count, datatype, op, root, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_REDUCE); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_reduce_scatter +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_reduce_scatter(const void *sendbuf, void *recvbuf, + const int recvcounts[], MPI_Datatype datatype, + MPI_Op op, MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_REDUCE_SCATTER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_REDUCE_SCATTER); + + mpi_errno = MPIR_Reduce_scatter(sendbuf, recvbuf, recvcounts, datatype, op, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_REDUCE_SCATTER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_reduce_scatter_block +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_reduce_scatter_block(const void *sendbuf, void *recvbuf, + int recvcount, MPI_Datatype datatype, + MPI_Op op, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_REDUCE_SCATTER_BLOCK); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_REDUCE_SCATTER_BLOCK); + + mpi_errno = MPIR_Reduce_scatter_block(sendbuf, recvbuf, recvcount, + datatype, op, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_REDUCE_SCATTER_BLOCK); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_scan +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_scan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_SCAN); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_SCAN); + + mpi_errno = MPIR_Scan(sendbuf, recvbuf, count, datatype, op, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_SCAN); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_exscan +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_exscan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_EXSCAN); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_EXSCAN); + + mpi_errno = MPIR_Exscan(sendbuf, recvbuf, count, datatype, op, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_EXSCAN); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_neighbor_allgather +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_neighbor_allgather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, int recvcount, + MPI_Datatype recvtype, MPIR_Comm * comm_ptr) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_NEIGHBOR_ALLGATHER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_NEIGHBOR_ALLGATHER); + + mpi_errno = + MPIR_Neighbor_allgather_impl(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, + comm_ptr); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_NEIGHBOR_ALLGATHER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_neighbor_allgatherv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_neighbor_allgatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int recvcounts[], const int displs[], + MPI_Datatype recvtype, MPIR_Comm * comm_ptr) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_NEIGHBOR_ALLGATHERV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_NEIGHBOR_ALLGATHERV); + + mpi_errno = MPIR_Neighbor_allgatherv_impl(sendbuf, sendcount, sendtype, + recvbuf, recvcounts, displs, recvtype, comm_ptr); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_NEIGHBOR_ALLGATHERV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_neighbor_alltoall +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_neighbor_alltoall(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, int recvcount, + MPI_Datatype recvtype, MPIR_Comm * comm_ptr) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_NEIGHBOR_ALLTOALL); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_NEIGHBOR_ALLTOALL); + + mpi_errno = MPIR_Neighbor_alltoall_impl(sendbuf, sendcount, sendtype, + recvbuf, recvcount, recvtype, comm_ptr); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_NEIGHBOR_ALLTOALL); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_neighbor_alltoallv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_neighbor_alltoallv(const void *sendbuf, const int sendcounts[], + const int sdispls[], MPI_Datatype sendtype, + void *recvbuf, const int recvcounts[], + const int rdispls[], MPI_Datatype recvtype, + MPIR_Comm * comm_ptr) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_NEIGHBOR_ALLTOALLV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_NEIGHBOR_ALLTOALLV); + + mpi_errno = MPIR_Neighbor_alltoallv_impl(sendbuf, sendcounts, sdispls, sendtype, + recvbuf, recvcounts, rdispls, recvtype, comm_ptr); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_NEIGHBOR_ALLTOALLV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_neighbor_alltoallw +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_neighbor_alltoallw(const void *sendbuf, const int sendcounts[], + const MPI_Aint sdispls[], + const MPI_Datatype sendtypes[], void *recvbuf, + const int recvcounts[], const MPI_Aint rdispls[], + const MPI_Datatype recvtypes[], MPIR_Comm * comm_ptr) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_NEIGHBOR_ALLTOALLW); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_NEIGHBOR_ALLTOALLW); + + mpi_errno = MPIR_Neighbor_alltoallw_impl(sendbuf, sendcounts, sdispls, sendtypes, + recvbuf, recvcounts, rdispls, recvtypes, comm_ptr); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_NEIGHBOR_ALLTOALLW); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ineighbor_allgather +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ineighbor_allgather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, int recvcount, + MPI_Datatype recvtype, MPIR_Comm * comm_ptr, + MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_INEIGHBOR_ALLGATHER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_INEIGHBOR_ALLGATHER); + + mpi_errno = MPIR_Ineighbor_allgather_impl(sendbuf, sendcount, sendtype, + recvbuf, recvcount, recvtype, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_INEIGHBOR_ALLGATHER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ineighbor_allgatherv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ineighbor_allgatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int recvcounts[], const int displs[], + MPI_Datatype recvtype, MPIR_Comm * comm_ptr, + MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_INEIGHBOR_ALLGATHERV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_INEIGHBOR_ALLGATHERV); + + mpi_errno = MPIR_Ineighbor_allgatherv_impl(sendbuf, sendcount, sendtype, + recvbuf, recvcounts, displs, recvtype, + comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_INEIGHBOR_ALLGATHERV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ineighbor_alltoall +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ineighbor_alltoall(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_INEIGHBOR_ALLTOALL); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_INEIGHBOR_ALLTOALL); + + mpi_errno = MPIR_Ineighbor_alltoall_impl(sendbuf, sendcount, sendtype, + recvbuf, recvcount, recvtype, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_INEIGHBOR_ALLTOALL); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ineighbor_alltoallv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ineighbor_alltoallv(const void *sendbuf, const int sendcounts[], + const int sdispls[], MPI_Datatype sendtype, + void *recvbuf, const int recvcounts[], + const int rdispls[], MPI_Datatype recvtype, + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_INEIGHBOR_ALLTOALLV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_INEIGHBOR_ALLTOALLV); + + mpi_errno = MPIR_Ineighbor_alltoallv_impl(sendbuf, sendcounts, sdispls, sendtype, + recvbuf, recvcounts, rdispls, recvtype, + comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_INEIGHBOR_ALLTOALLV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ineighbor_alltoallw +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ineighbor_alltoallw(const void *sendbuf, const int sendcounts[], + const MPI_Aint sdispls[], + const MPI_Datatype sendtypes[], void *recvbuf, + const int recvcounts[], const MPI_Aint rdispls[], + const MPI_Datatype recvtypes[], MPIR_Comm * comm_ptr, + MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_INEIGHBOR_ALLTOALLW); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_INEIGHBOR_ALLTOALLW); + + mpi_errno = MPIR_Ineighbor_alltoallw_impl(sendbuf, sendcounts, sdispls, sendtypes, + recvbuf, recvcounts, rdispls, recvtypes, + comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_INEIGHBOR_ALLTOALLW); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ibarrier +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ibarrier(MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IBARRIER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IBARRIER); + + mpi_errno = MPIR_Ibarrier_impl(comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IBARRIER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ibcast +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ibcast(void *buffer, int count, MPI_Datatype datatype, + int root, MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IBCAST); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IBCAST); + + mpi_errno = MPIR_Ibcast_impl(buffer, count, datatype, root, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IBCAST); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_iallgather +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_iallgather(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IALLGATHER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IALLGATHER); + + mpi_errno = MPIR_Iallgather_impl(sendbuf, sendcount, sendtype, recvbuf, + recvcount, recvtype, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IALLGATHER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_iallreduce +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_iallreduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IREDUCE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IREDUCE); + + mpi_errno = MPIR_Iallreduce_impl(sendbuf, recvbuf, count, datatype, op, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IREDUCE); + return mpi_errno; +} + + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_iallgatherv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_iallgatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, const int *displs, + MPI_Datatype recvtype, MPIR_Comm * comm_ptr, + MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IALLGATHERV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IALLGATHERV); + + mpi_errno = MPIR_Iallgatherv_impl(sendbuf, sendcount, sendtype, + recvbuf, recvcounts, displs, recvtype, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IALLGATHERV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ialltoall +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ialltoall(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IALLTOALL); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IALLTOALL); + + mpi_errno = MPIR_Ialltoall_impl(sendbuf, sendcount, sendtype, recvbuf, + recvcount, recvtype, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IALLTOALL); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ialltoallv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ialltoallv(const void *sendbuf, const int *sendcounts, + const int *sdispls, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, + const int *rdispls, MPI_Datatype recvtype, + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IALLTOALLV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IALLTOALLV); + + mpi_errno = MPIR_Ialltoallv_impl(sendbuf, sendcounts, sdispls, + sendtype, recvbuf, recvcounts, + rdispls, recvtype, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IALLTOALLV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ialltoallw +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ialltoallw(const void *sendbuf, const int *sendcounts, + const int *sdispls, const MPI_Datatype sendtypes[], + void *recvbuf, const int *recvcounts, + const int *rdispls, const MPI_Datatype recvtypes[], + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IALLTOALLW); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IALLTOALLW); + + mpi_errno = MPIR_Ialltoallw_impl(sendbuf, sendcounts, sdispls, + sendtypes, recvbuf, recvcounts, + rdispls, recvtypes, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IALLTOALLW); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_iexscan +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_iexscan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm_ptr, + MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IEXSCAN); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IEXSCAN); + + mpi_errno = MPIR_Iexscan_impl(sendbuf, recvbuf, count, datatype, op, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IEXSCAN); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_igather +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_igather(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + int root, MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IGATHER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IGATHER); + + mpi_errno = MPIR_Igather_impl(sendbuf, sendcount, sendtype, recvbuf, + recvcount, recvtype, root, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IGATHER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_igatherv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_igatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, const int *displs, + MPI_Datatype recvtype, int root, MPIR_Comm * comm_ptr, + MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IGATHERV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IGATHERV); + + mpi_errno = MPIR_Igatherv_impl(sendbuf, sendcount, sendtype, + recvbuf, recvcounts, displs, recvtype, root, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IGATHERV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ireduce_scatter_block +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ireduce_scatter_block(const void *sendbuf, void *recvbuf, + int recvcount, MPI_Datatype datatype, + MPI_Op op, MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IREDUCE_SCATTER_BLOCK); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IREDUCE_SCATTER_BLOCK); + + mpi_errno = MPIR_Ireduce_scatter_block_impl(sendbuf, recvbuf, recvcount, + datatype, op, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IREDUCE_SCATTER_BLOCK); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ireduce_scatter +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ireduce_scatter(const void *sendbuf, void *recvbuf, + const int recvcounts[], MPI_Datatype datatype, + MPI_Op op, MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IREDUCE_SCATTER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IREDUCE_SCATTER); + + mpi_errno = MPIR_Ireduce_scatter_impl(sendbuf, recvbuf, recvcounts, datatype, op, + comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IREDUCE_SCATTER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ireduce +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ireduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, int root, + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IREDUCE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IREDUCE); + + mpi_errno = MPIR_Ireduce_impl(sendbuf, recvbuf, count, datatype, op, root, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IREDUCE); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_iscan +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_iscan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm_ptr, + MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_ISCAN); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_ISCAN); + + mpi_errno = MPIR_Iscan_impl(sendbuf, recvbuf, count, datatype, op, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_ISCAN); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_iscatter +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_iscatter(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + int root, MPIR_Comm * comm, MPI_Request * request) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_ISCATTER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_ISCATTER); + + mpi_errno = MPIR_Iscatter_impl(sendbuf, sendcount, sendtype, recvbuf, + recvcount, recvtype, root, comm, request); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_ISCATTER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_iscatterv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_iscatterv(const void *sendbuf, const int *sendcounts, + const int *displs, MPI_Datatype sendtype, + void *recvbuf, int recvcount, + MPI_Datatype recvtype, int root, + MPIR_Comm * comm, MPI_Request * request) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_ISCATTERV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_ISCATTERV); + + mpi_errno = MPIR_Iscatterv_impl(sendbuf, sendcounts, displs, sendtype, + recvbuf, recvcount, recvtype, root, comm, request); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_ISCATTERV); + return mpi_errno; +} + +#endif diff --git a/src/mpid/ch4/netmod/stubnm/stubnm_comm.h b/src/mpid/ch4/netmod/stubnm/stubnm_comm.h new file mode 100644 index 0000000..3188e94 --- /dev/null +++ b/src/mpid/ch4/netmod/stubnm/stubnm_comm.h @@ -0,0 +1,39 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_STUBNM_COMM_H_INCLUDED +#define NETMOD_STUBNM_COMM_H_INCLUDED + +#include "stubnm_impl.h" + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_comm_create +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_comm_create(MPIR_Comm * comm) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Assert(0); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_comm_destroy +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_comm_destroy(MPIR_Comm * comm) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Assert(0); + return mpi_errno; +} + + +#endif /* NETMOD_STUBNM_COMM_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/stubnm/stubnm_datatype.h b/src/mpid/ch4/netmod/stubnm/stubnm_datatype.h new file mode 100644 index 0000000..877009b --- /dev/null +++ b/src/mpid/ch4/netmod/stubnm/stubnm_datatype.h @@ -0,0 +1,35 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2012 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_STUBNM_DATATYPE_H_INCLUDED +#define NETMOD_STUBNM_DATATYPE_H_INCLUDED + +#include "stubnm_impl.h" + +static inline void MPIDI_NM_datatype_destroy(MPIR_Datatype * datatype_p) +{ + MPIR_Assert(0); + return; +} + +static inline void MPIDI_NM_datatype_commit(MPIR_Datatype * datatype_p) +{ + MPIR_Assert(0); + return; +} + +static inline void MPIDI_NM_datatype_dup(MPIR_Datatype * old_datatype_p, + MPIR_Datatype * new_datatype_p) +{ + MPIR_Assert(0); + return; +} + +#endif diff --git a/src/mpid/ch4/netmod/stubnm/stubnm_impl.h b/src/mpid/ch4/netmod/stubnm/stubnm_impl.h new file mode 100644 index 0000000..586c6d5 --- /dev/null +++ b/src/mpid/ch4/netmod/stubnm/stubnm_impl.h @@ -0,0 +1,17 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_STUBNM_IMPL_H_INCLUDED +#define NETMOD_STUBNM_IMPL_H_INCLUDED + +#include <mpidimpl.h> +#include "mpidch4r.h" + +#endif /* NETMOD_STUBNM_IMPL_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/stubnm/stubnm_init.h b/src/mpid/ch4/netmod/stubnm/stubnm_init.h new file mode 100644 index 0000000..5f458a6 --- /dev/null +++ b/src/mpid/ch4/netmod/stubnm/stubnm_init.h @@ -0,0 +1,95 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_STUBNM_INIT_H_INCLUDED +#define NETMOD_STUBNM_INIT_H_INCLUDED + +#include "stubnm_impl.h" + +static inline int MPIDI_NM_init(int rank, + int size, + int appnum, + int *tag_ub, + MPIR_Comm * comm_world, + MPIR_Comm * comm_self, + int spawned, int num_contexts, void **netmod_contexts) +{ + int mpi_errno = MPI_SUCCESS; + + + MPIR_Assert(0); + return mpi_errno; +} + +static inline int MPIDI_NM_finalize(void) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Assert(0); + return mpi_errno; +} + + +static inline int MPIDI_NM_comm_get_lpid(MPIR_Comm * comm_ptr, + int idx, int *lpid_ptr, MPL_bool is_remote) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_gpid_get(MPIR_Comm * comm_ptr, int rank, MPIR_Gpid * gpid) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_get_node_id(MPIR_Comm * comm, int rank, MPID_Node_id_t * id_p) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_get_max_node_id(MPIR_Comm * comm, MPID_Node_id_t * max_id_p) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_getallincomm(MPIR_Comm * comm_ptr, + int local_size, MPIR_Gpid local_gpids[], int *singleAVT) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_gpid_tolpidarray(int size, MPIR_Gpid gpid[], int lpid[]) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_create_intercomm_from_lpids(MPIR_Comm * newcomm_ptr, + int size, const int lpids[]) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_free_mem(void *ptr) +{ + return MPIDI_CH4U_free_mem(ptr); +} + +static inline void *MPIDI_NM_alloc_mem(size_t size, MPIR_Info * info_ptr) +{ + return MPIDI_CH4U_alloc_mem(size, info_ptr); +} + + +#endif /* NETMOD_STUBNM_INIT_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/stubnm/stubnm_op.h b/src/mpid/ch4/netmod/stubnm/stubnm_op.h new file mode 100644 index 0000000..ae4eade --- /dev/null +++ b/src/mpid/ch4/netmod/stubnm/stubnm_op.h @@ -0,0 +1,29 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2012 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_STUBNM_OP_H_INCLUDED +#define NETMOD_STUBNM_OP_H_INCLUDED + +#include "stubnm_impl.h" + +static inline void MPIDI_NM_op_destroy(MPIR_Op * op_p) +{ + MPIR_Assert(0); + return; +} + +static inline void MPIDI_NM_op_commit(MPIR_Op * op_p) +{ + MPIR_Assert(0); + return; +} + + +#endif diff --git a/src/mpid/ch4/netmod/stubnm/stubnm_pre.h b/src/mpid/ch4/netmod/stubnm/stubnm_pre.h new file mode 100644 index 0000000..ad23b48 --- /dev/null +++ b/src/mpid/ch4/netmod/stubnm/stubnm_pre.h @@ -0,0 +1,47 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ + +#ifndef NETMOD_STUBNM_PRE_H_INCLUDED +#define NETMOD_STUBNM_PRE_H_INCLUDED + +typedef struct { + int dummy; +} MPIDI_STUBNM_am_request_t; + +typedef struct { + int dummy; +} MPIDI_STUBNM_request_t; + +typedef struct { + int dummy; +} MPIDI_STUBNM_comm_t; + +typedef struct { + int dummy; +} MPIDI_STUBNM_dt_t; + +typedef struct { + int dummy; +} MPIDI_STUBNM_op_t; + +typedef struct { + int dummy; +} MPIDI_STUBNM_win_t; + +typedef struct { + int dummy; +} MPIDI_STUBNM_gpid_t; + +typedef struct { + int dummy; +} MPIDI_STUBNM_addr_t; + +#endif /* NETMOD_STUBNM_PRE_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/stubnm/stubnm_probe.h b/src/mpid/ch4/netmod/stubnm/stubnm_probe.h new file mode 100644 index 0000000..6442fb6 --- /dev/null +++ b/src/mpid/ch4/netmod/stubnm/stubnm_probe.h @@ -0,0 +1,39 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_STUBNM_PROBE_H_INCLUDED +#define NETMOD_STUBNM_PROBE_H_INCLUDED + +#include "stubnm_impl.h" + +static inline int MPIDI_NM_probe(int source, + int tag, MPIR_Comm * comm, int context_offset, MPI_Status * status) +{ + return MPIDI_CH4U_probe(source, tag, comm, context_offset, status); +} + +static inline int MPIDI_NM_improbe(int source, + int tag, + MPIR_Comm * comm, + int context_offset, + int *flag, MPIR_Request ** message, MPI_Status * status) +{ + return MPIDI_CH4U_improbe(source, tag, comm, context_offset, flag, message, status); +} + +static inline int MPIDI_NM_iprobe(int source, + int tag, + MPIR_Comm * comm, + int context_offset, int *flag, MPI_Status * status) +{ + return MPIDI_CH4U_iprobe(source, tag, comm, context_offset, flag, status); +} + +#endif /* NETMOD_STUBNM_PROBE_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/stubnm/stubnm_proc.h b/src/mpid/ch4/netmod/stubnm/stubnm_proc.h new file mode 100644 index 0000000..595aa6b --- /dev/null +++ b/src/mpid/ch4/netmod/stubnm/stubnm_proc.h @@ -0,0 +1,28 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_STUBNM_PROC_H_INCLUDED +#define NETMOD_STUBNM_PROC_H_INCLUDED + +#include "stubnm_impl.h" + +static inline int MPIDI_NM_rank_is_local(int rank, MPIR_Comm * comm) +{ + int ret; + MPIR_FUNC_VERBOSE_STATE_DECL(MPIDI_NETMOD_RANK_IS_LOCAL); + MPIR_FUNC_VERBOSE_ENTER(MPIDI_NETMOD_RANK_IS_LOCAL); + + MPIR_Assert(0); + ret = 0; + + MPIR_FUNC_VERBOSE_EXIT(MPIDI_NETMOD_RANK_IS_LOCAL); + return ret; +} +#endif /*NETMOD_STUBNM_PROC_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/stubnm/stubnm_progress.h b/src/mpid/ch4/netmod/stubnm/stubnm_progress.h new file mode 100644 index 0000000..d43cac5 --- /dev/null +++ b/src/mpid/ch4/netmod/stubnm/stubnm_progress.h @@ -0,0 +1,76 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_STUBNM_PROGRESS_H_INCLUDED +#define NETMOD_STUBNM_PROGRESS_H_INCLUDED + +#include "stubnm_impl.h" + +static inline int MPIDI_NM_progress(void *netmod_context, int blocking) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_progress_test(void) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_progress_poke(void) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline void MPIDI_NM_progress_start(MPID_Progress_state * state) +{ + MPIR_Assert(0); + return; +} + +static inline void MPIDI_NM_progress_end(MPID_Progress_state * state) +{ + MPIR_Assert(0); + return; +} + +static inline int MPIDI_NM_progress_wait(MPID_Progress_state * state) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_progress_register(int (*progress_fn) (int *), int *id) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_progress_deregister(int id) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_progress_activate(int id) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_progress_deactivate(int id) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +#endif /* NETMOD_STUBNM_PROGRESS_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/stubnm/stubnm_recv.h b/src/mpid/ch4/netmod/stubnm/stubnm_recv.h new file mode 100644 index 0000000..18724cc --- /dev/null +++ b/src/mpid/ch4/netmod/stubnm/stubnm_recv.h @@ -0,0 +1,67 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_STUBNM_RECV_H_INCLUDED +#define NETMOD_STUBNM_RECV_H_INCLUDED + +#include "stubnm_impl.h" + +static inline int MPIDI_NM_recv(void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, + int context_offset, MPI_Status * status, MPIR_Request ** request) +{ + int err = MPI_SUCCESS; + MPIR_Assert(0); + return err; +} + +static inline int MPIDI_NM_recv_init(void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_imrecv(void *buf, + int count, + MPI_Datatype datatype, + MPIR_Request * message, MPIR_Request ** rreqp) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_irecv(void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + int err = MPI_SUCCESS; + MPIR_Assert(0); + return err; +} + +static inline int MPIDI_NM_cancel_recv(MPIR_Request * rreq) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +#endif /* NETMOD_STUBNM_RECV_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/stubnm/stubnm_request.h b/src/mpid/ch4/netmod/stubnm/stubnm_request.h new file mode 100644 index 0000000..5909e53 --- /dev/null +++ b/src/mpid/ch4/netmod/stubnm/stubnm_request.h @@ -0,0 +1,26 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_STUBNM_REQUEST_H_INCLUDED +#define NETMOD_STUBNM_REQUEST_H_INCLUDED + +#include "stubnm_impl.h" + +static inline void MPIDI_NM_am_request_init(MPIR_Request * req) +{ + MPIR_Assert(0); +} + +static inline void MPIDI_NM_am_request_finalize(MPIR_Request * req) +{ + MPIR_Assert(0); +} + +#endif /* NETMOD_STUBNM_REQUEST_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/stubnm/stubnm_rma.h b/src/mpid/ch4/netmod/stubnm/stubnm_rma.h new file mode 100644 index 0000000..de8172f --- /dev/null +++ b/src/mpid/ch4/netmod/stubnm/stubnm_rma.h @@ -0,0 +1,148 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_STUBNM_RMA_H_INCLUDED +#define NETMOD_STUBNM_RMA_H_INCLUDED + +#include "stubnm_impl.h" + +static inline int MPIDI_NM_put(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, MPI_Datatype target_datatype, MPIR_Win * win) +{ + return MPIDI_CH4U_put(origin_addr, origin_count, origin_datatype, + target_rank, target_disp, target_count, target_datatype, win); +} + +static inline int MPIDI_NM_get(void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, MPI_Datatype target_datatype, MPIR_Win * win) +{ + return MPIDI_CH4U_get(origin_addr, origin_count, origin_datatype, + target_rank, target_disp, target_count, target_datatype, win); +} + +static inline int MPIDI_NM_rput(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPIR_Win * win, MPIR_Request ** request) +{ + return MPIDI_CH4U_rput(origin_addr, origin_count, origin_datatype, + target_rank, target_disp, target_count, target_datatype, win, request); +} + + +static inline int MPIDI_NM_compare_and_swap(const void *origin_addr, + const void *compare_addr, + void *result_addr, + MPI_Datatype datatype, + int target_rank, MPI_Aint target_disp, MPIR_Win * win) +{ + return MPIDI_CH4U_compare_and_swap(origin_addr, compare_addr, result_addr, + datatype, target_rank, target_disp, win); +} + +static inline int MPIDI_NM_raccumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPI_Op op, MPIR_Win * win, MPIR_Request ** request) +{ + return MPIDI_CH4U_raccumulate(origin_addr, origin_count, origin_datatype, + target_rank, target_disp, target_count, + target_datatype, op, win, request); +} + +static inline int MPIDI_NM_rget_accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + void *result_addr, + int result_count, + MPI_Datatype result_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPI_Op op, MPIR_Win * win, MPIR_Request ** request) +{ + return MPIDI_CH4U_rget_accumulate(origin_addr, origin_count, origin_datatype, + result_addr, result_count, result_datatype, + target_rank, target_disp, target_count, + target_datatype, op, win, request); +} + +static inline int MPIDI_NM_fetch_and_op(const void *origin_addr, + void *result_addr, + MPI_Datatype datatype, + int target_rank, + MPI_Aint target_disp, MPI_Op op, MPIR_Win * win) +{ + return MPIDI_CH4U_fetch_and_op(origin_addr, result_addr, datatype, + target_rank, target_disp, op, win); +} + + +static inline int MPIDI_NM_rget(void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPIR_Win * win, MPIR_Request ** request) +{ + return MPIDI_CH4U_rget(origin_addr, origin_count, origin_datatype, + target_rank, target_disp, target_count, target_datatype, win, request); +} + + +static inline int MPIDI_NM_get_accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + void *result_addr, + int result_count, + MPI_Datatype result_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, MPI_Op op, MPIR_Win * win) +{ + return MPIDI_CH4U_get_accumulate(origin_addr, origin_count, origin_datatype, + result_addr, result_count, result_datatype, + target_rank, target_disp, target_count, + target_datatype, op, win); +} + +static inline int MPIDI_NM_accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, MPI_Op op, MPIR_Win * win) +{ + return MPIDI_CH4U_accumulate(origin_addr, origin_count, origin_datatype, + target_rank, target_disp, target_count, target_datatype, op, win); +} + +#endif /* NETMOD_STUBNM_RMA_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/stubnm/stubnm_send.h b/src/mpid/ch4/netmod/stubnm/stubnm_send.h new file mode 100644 index 0000000..c83867d --- /dev/null +++ b/src/mpid/ch4/netmod/stubnm/stubnm_send.h @@ -0,0 +1,128 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_STUBNM_SEND_H_INCLUDED +#define NETMOD_STUBNM_SEND_H_INCLUDED + +#include "stubnm_impl.h" + +static inline int MPIDI_NM_send(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + return MPIDI_CH4U_send(buf, count, datatype, rank, tag, comm, context_offset, request); +} + +static inline int MPIDI_NM_rsend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + return MPIDI_CH4U_rsend(buf, count, datatype, rank, tag, comm, context_offset, request); +} + + + +static inline int MPIDI_NM_irsend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + return MPIDI_CH4U_irsend(buf, count, datatype, rank, tag, comm, context_offset, request); +} + +static inline int MPIDI_NM_ssend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + return MPIDI_CH4U_ssend(buf, count, datatype, rank, tag, comm, context_offset, request); +} + +static inline int MPIDI_NM_startall(int count, MPIR_Request * requests[]) +{ + return MPIDI_CH4U_startall(count, requests); +} + +static inline int MPIDI_NM_send_init(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + return MPIDI_CH4U_send_init(buf, count, datatype, rank, tag, comm, context_offset, request); +} + +static inline int MPIDI_NM_ssend_init(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + return MPIDI_CH4U_ssend_init(buf, count, datatype, rank, tag, comm, context_offset, request); +} + +static inline int MPIDI_NM_bsend_init(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + return MPIDI_CH4U_bsend_init(buf, count, datatype, rank, tag, comm, context_offset, request); +} + +static inline int MPIDI_NM_rsend_init(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + return MPIDI_CH4U_rsend_init(buf, count, datatype, rank, tag, comm, context_offset, request); +} + +static inline int MPIDI_NM_isend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + return MPIDI_CH4U_isend(buf, count, datatype, rank, tag, comm, context_offset, request); +} + +static inline int MPIDI_NM_issend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + return MPIDI_CH4U_issend(buf, count, datatype, rank, tag, comm, context_offset, request); +} + +static inline int MPIDI_NM_cancel_send(MPIR_Request * sreq) +{ + return MPIDI_CH4U_cancel_send(sreq); +} + +#endif /* NETMOD_STUBNM_SEND_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/stubnm/stubnm_spawn.h b/src/mpid/ch4/netmod/stubnm/stubnm_spawn.h new file mode 100644 index 0000000..2f70a59 --- /dev/null +++ b/src/mpid/ch4/netmod/stubnm/stubnm_spawn.h @@ -0,0 +1,50 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_STUBNM_SPAWN_H_INCLUDED +#define NETMOD_STUBNM_SPAWN_H_INCLUDED + +#include "stubnm_impl.h" + +static inline int MPIDI_NM_comm_connect(const char *port_name, + MPIR_Info * info, + int root, MPIR_Comm * comm, MPIR_Comm ** newcomm_ptr) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_comm_disconnect(MPIR_Comm * comm_ptr) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_open_port(MPIR_Info * info_ptr, char *port_name) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_close_port(const char *port_name) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_comm_accept(const char *port_name, + MPIR_Info * info, + int root, MPIR_Comm * comm, MPIR_Comm ** newcomm_ptr) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +#endif /* NETMOD_STUBNM_WIN_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/stubnm/stubnm_unimpl.h b/src/mpid/ch4/netmod/stubnm/stubnm_unimpl.h new file mode 100644 index 0000000..db8e66b --- /dev/null +++ b/src/mpid/ch4/netmod/stubnm/stubnm_unimpl.h @@ -0,0 +1,19 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ + +#ifdef __cplusplus +extern "C" { +#endif + + +#ifdef __cplusplus +} +#endif diff --git a/src/mpid/ch4/netmod/stubnm/stubnm_win.h b/src/mpid/ch4/netmod/stubnm/stubnm_win.h new file mode 100644 index 0000000..5c8a112 --- /dev/null +++ b/src/mpid/ch4/netmod/stubnm/stubnm_win.h @@ -0,0 +1,160 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_STUBNM_WIN_H_INCLUDED +#define NETMOD_STUBNM_WIN_H_INCLUDED + +#include "stubnm_impl.h" + +static inline int MPIDI_NM_win_set_info(MPIR_Win * win, MPIR_Info * info) +{ + return MPIDI_CH4R_win_set_info(win, info); +} + + +static inline int MPIDI_NM_win_start(MPIR_Group * group, int assert, MPIR_Win * win) +{ + return MPIDI_CH4R_win_start(group, assert, win); +} + + +static inline int MPIDI_NM_win_complete(MPIR_Win * win) +{ + return MPIDI_CH4R_win_complete(win); +} + +static inline int MPIDI_NM_win_post(MPIR_Group * group, int assert, MPIR_Win * win) +{ + return MPIDI_CH4R_win_post(group, assert, win); +} + + +static inline int MPIDI_NM_win_wait(MPIR_Win * win) +{ + return MPIDI_CH4R_win_wait(win); +} + + +static inline int MPIDI_NM_win_test(MPIR_Win * win, int *flag) +{ + return MPIDI_CH4R_win_test(win, flag); +} + +static inline int MPIDI_NM_win_lock(int lock_type, int rank, int assert, MPIR_Win * win) +{ + return MPIDI_CH4R_win_lock(lock_type, rank, assert, win); +} + + +static inline int MPIDI_NM_win_unlock(int rank, MPIR_Win * win) +{ + return MPIDI_CH4R_win_unlock(rank, win); +} + +static inline int MPIDI_NM_win_get_info(MPIR_Win * win, MPIR_Info ** info_p_p) +{ + return MPIDI_CH4R_win_get_info(win, info_p_p); +} + + +static inline int MPIDI_NM_win_free(MPIR_Win ** win_ptr) +{ + return MPIDI_CH4R_win_free(win_ptr); +} + +static inline int MPIDI_NM_win_fence(int assert, MPIR_Win * win) +{ + return MPIDI_CH4R_win_fence(assert, win); +} + +static inline int MPIDI_NM_win_create(void *base, + MPI_Aint length, + int disp_unit, + MPIR_Info * info, MPIR_Comm * comm_ptr, MPIR_Win ** win_ptr) +{ + return MPIDI_CH4R_win_create(base, length, disp_unit, info, comm_ptr, win_ptr); +} + +static inline int MPIDI_NM_win_attach(MPIR_Win * win, void *base, MPI_Aint size) +{ + return MPIDI_CH4R_win_attach(win, base, size); +} + +static inline int MPIDI_NM_win_allocate_shared(MPI_Aint size, + int disp_unit, + MPIR_Info * info_ptr, + MPIR_Comm * comm_ptr, + void **base_ptr, MPIR_Win ** win_ptr) +{ + return MPIDI_CH4R_win_allocate_shared(size, disp_unit, info_ptr, comm_ptr, base_ptr, win_ptr); +} + +static inline int MPIDI_NM_win_detach(MPIR_Win * win, const void *base) +{ + return MPIDI_CH4R_win_detach(win, base); +} + +static inline int MPIDI_NM_win_shared_query(MPIR_Win * win, + int rank, + MPI_Aint * size, int *disp_unit, void *baseptr) +{ + return MPIDI_CH4R_win_shared_query(win, rank, size, disp_unit, baseptr); +} + +static inline int MPIDI_NM_win_allocate(MPI_Aint size, + int disp_unit, + MPIR_Info * info, + MPIR_Comm * comm, void *baseptr, MPIR_Win ** win) +{ + return MPIDI_CH4R_win_allocate(size, disp_unit, info, comm, baseptr, win); +} + +static inline int MPIDI_NM_win_flush(int rank, MPIR_Win * win) +{ + return MPIDI_CH4R_win_flush(rank, win); +} + +static inline int MPIDI_NM_win_flush_local_all(MPIR_Win * win) +{ + return MPIDI_CH4R_win_flush_local_all(win); +} + +static inline int MPIDI_NM_win_unlock_all(MPIR_Win * win) +{ + return MPIDI_CH4R_win_unlock_all(win); +} + +static inline int MPIDI_NM_win_create_dynamic(MPIR_Info * info, MPIR_Comm * comm, MPIR_Win ** win) +{ + return MPIDI_CH4R_win_create_dynamic(info, comm, win); +} + +static inline int MPIDI_NM_win_flush_local(int rank, MPIR_Win * win) +{ + return MPIDI_CH4R_win_flush_local(rank, win); +} + +static inline int MPIDI_NM_win_sync(MPIR_Win * win) +{ + return MPIDI_CH4R_win_sync(win); +} + +static inline int MPIDI_NM_win_flush_all(MPIR_Win * win) +{ + return MPIDI_CH4R_win_flush_all(win); +} + +static inline int MPIDI_NM_win_lock_all(int assert, MPIR_Win * win) +{ + return MPIDI_CH4R_win_lock_all(assert, win); +} + + +#endif /* NETMOD_STUBNM_WIN_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/stubnm/subconfigure.m4 b/src/mpid/ch4/netmod/stubnm/subconfigure.m4 new file mode 100644 index 0000000..4dce4f1 --- /dev/null +++ b/src/mpid/ch4/netmod/stubnm/subconfigure.m4 @@ -0,0 +1,21 @@ +[#] start of __file__ +dnl MPICH_SUBCFG_AFTER=src/mpid/ch4 + +AC_DEFUN([PAC_SUBCFG_PREREQ_]PAC_SUBCFG_AUTO_SUFFIX,[ + AM_COND_IF([BUILD_CH4],[ + for net in $ch4_netmods ; do + AS_CASE([$net],[stubnm],[build_ch4_netmod_stubnm=yes]) + done + ]) + AM_CONDITIONAL([BUILD_CH4_NETMOD_STUBNM],[test "X$build_ch4_netmod_stubnm" = "Xyes"]) +])dnl + +AC_DEFUN([PAC_SUBCFG_BODY_]PAC_SUBCFG_AUTO_SUFFIX,[ +AM_COND_IF([BUILD_CH4_NETMOD_STUBNM],[ + AC_MSG_NOTICE([RUNNING CONFIGURE FOR ch4:stubnm]) + AC_DEFINE([ENABLE_COMM_OVERRIDES], 1, [define to add per-vc function pointers to override send and recv functions]) + AC_DEFINE(MPIDI_BUILD_CH4_LOCALITY_INFO, 1, [CH4 should build locality info]) +])dnl end AM_COND_IF(BUILD_CH4_NETMOD_STUBNM,...) +])dnl end _BODY + +[#] end of __file__ diff --git a/src/mpid/ch4/netmod/ucx/Makefile.mk b/src/mpid/ch4/netmod/ucx/Makefile.mk new file mode 100644 index 0000000..ca88e1a --- /dev/null +++ b/src/mpid/ch4/netmod/ucx/Makefile.mk @@ -0,0 +1,15 @@ +## -*- Mode: Makefile; -*- +## vim: set ft=automake : +## +## (C) 2016 by Argonne National Laboratory. +## See COPYRIGHT in top-level directory. +## +if BUILD_CH4_NETMOD_UCX + +noinst_HEADERS += +mpi_core_sources += src/mpid/ch4/netmod/ucx/func_table.c\ + src/mpid/ch4/netmod/ucx/globals.c + +errnames_txt_files += src/mpid/ch4/netmod/ucx/errnames.txt + +endif diff --git a/src/mpid/ch4/netmod/ucx/errnames.txt b/src/mpid/ch4/netmod/ucx/errnames.txt new file mode 100644 index 0000000..e6058f2 --- /dev/null +++ b/src/mpid/ch4/netmod/ucx/errnames.txt @@ -0,0 +1,16 @@ +**ucx_nm_read_config:ucx_read_config faild +**ucx_nm_read_config %s %d %s %s: ucx_read_config faild(%s %d %s %s) +**ucx_nm_init:ucx_init failed +**ucx_nm_init %s %d %s %s: ucx_init failed (%s %d %s %s) +**ucx_nm_worker_create:ucx_worker_create failed +**ucx_nm_worker_create %s %d %s %s: ucx_worker_create failed (%s %d %s %s) +**ucx_nm_ep_create:failed to create ucp_endpoint +**ucx_nm_ep_create %s %d %s %s: failed to create ucp_endpoint (%s %d %s %s) +**ucx_nm_get_worker_address:ucx failed to get worker address +**ucx_nm_get_worker_address %s %d %s %s: ucx failed to get worker address (%s %s %s %s) +**ucx_nm_tag_nb_send:Failed to start tag send in ucx +**ucx_nm_tag_nb_send %s %d %s %s: Failed to start tag send in ucx (%s %d %s %s) +**ucx_nm_tag_nb_recv:Failed to start tag recv in ucx +**ucx_nm_tag_nb_recv %s %d %s %s: Failed to start tag recv in ucx (%s %d %s %s) +**ucx_nm_other:Other error +**ucx_nm_other %s %d %s %s: Other error (%s %d %s %s) diff --git a/src/mpid/ch4/netmod/ucx/func_table.c b/src/mpid/ch4/netmod/ucx/func_table.c new file mode 100644 index 0000000..7e99f6d --- /dev/null +++ b/src/mpid/ch4/netmod/ucx/func_table.c @@ -0,0 +1,155 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2016 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Mellanox Technologies Ltd. + * Copyright (C) Mellanox Technologies Ltd. 2016. ALL RIGHTS RESERVED + */ + +#ifndef NETMOD_DIRECT +#define NETMOD_DISABLE_INLINES +#include <mpidimpl.h> +#include "netmod_direct.h" +MPIDI_NM_funcs_t MPIDI_NM_ucx_funcs = { + MPIDI_NM_init, + MPIDI_NM_finalize, + MPIDI_NM_progress, + MPIDI_NM_comm_connect, + MPIDI_NM_comm_disconnect, + MPIDI_NM_open_port, + MPIDI_NM_close_port, + MPIDI_NM_comm_accept, + MPIDI_NM_comm_get_lpid, + MPIDI_NM_gpid_get, + MPIDI_NM_get_node_id, + MPIDI_NM_get_max_node_id, + MPIDI_NM_getallincomm, + MPIDI_NM_gpid_tolpidarray, + MPIDI_NM_create_intercomm_from_lpids, + MPIDI_NM_comm_create, + MPIDI_NM_comm_destroy, + MPIDI_NM_am_request_init, + MPIDI_NM_am_request_finalize, + MPIDI_NM_reg_hdr_handler, + MPIDI_NM_send_am_hdr, + MPIDI_NM_inject_am_hdr, + MPIDI_NM_send_am, + MPIDI_NM_send_amv, + MPIDI_NM_send_amv_hdr, + MPIDI_NM_send_am_hdr_reply, + MPIDI_NM_inject_am_hdr_reply, + MPIDI_NM_send_am_reply, + MPIDI_NM_send_amv_reply, + MPIDI_NM_am_hdr_max_sz, + MPIDI_NM_am_inject_max_sz, + MPIDI_NM_am_recv +}; + +MPIDI_NM_native_funcs_t MPIDI_NM_native_ucx_funcs = { + MPIDI_NM_send, + MPIDI_NM_ssend, + MPIDI_NM_startall, + MPIDI_NM_send_init, + MPIDI_NM_ssend_init, + MPIDI_NM_rsend_init, + MPIDI_NM_bsend_init, + MPIDI_NM_isend, + MPIDI_NM_issend, + MPIDI_NM_cancel_send, + MPIDI_NM_recv_init, + MPIDI_NM_recv, + MPIDI_NM_irecv, + MPIDI_NM_imrecv, + MPIDI_NM_cancel_recv, + MPIDI_NM_alloc_mem, + MPIDI_NM_free_mem, + MPIDI_NM_improbe, + MPIDI_NM_iprobe, + MPIDI_NM_win_set_info, + MPIDI_NM_win_shared_query, + MPIDI_NM_put, + MPIDI_NM_win_start, + MPIDI_NM_win_complete, + MPIDI_NM_win_post, + MPIDI_NM_win_wait, + MPIDI_NM_win_test, + MPIDI_NM_win_lock, + MPIDI_NM_win_unlock, + MPIDI_NM_win_get_info, + MPIDI_NM_get, + MPIDI_NM_win_free, + MPIDI_NM_win_fence, + MPIDI_NM_win_create, + MPIDI_NM_accumulate, + MPIDI_NM_win_attach, + MPIDI_NM_win_allocate_shared, + MPIDI_NM_rput, + MPIDI_NM_win_flush_local, + MPIDI_NM_win_detach, + MPIDI_NM_compare_and_swap, + MPIDI_NM_raccumulate, + MPIDI_NM_rget_accumulate, + MPIDI_NM_fetch_and_op, + MPIDI_NM_win_allocate, + MPIDI_NM_win_flush, + MPIDI_NM_win_flush_local_all, + MPIDI_NM_win_unlock_all, + MPIDI_NM_win_create_dynamic, + MPIDI_NM_rget, + MPIDI_NM_win_sync, + MPIDI_NM_win_flush_all, + MPIDI_NM_get_accumulate, + MPIDI_NM_win_lock_all, + MPIDI_NM_rank_is_local, + MPIDI_NM_barrier, + MPIDI_NM_bcast, + MPIDI_NM_allreduce, + MPIDI_NM_allgather, + MPIDI_NM_allgatherv, + MPIDI_NM_scatter, + MPIDI_NM_scatterv, + MPIDI_NM_gather, + MPIDI_NM_gatherv, + MPIDI_NM_alltoall, + MPIDI_NM_alltoallv, + MPIDI_NM_alltoallw, + MPIDI_NM_reduce, + MPIDI_NM_reduce_scatter, + MPIDI_NM_reduce_scatter_block, + MPIDI_NM_scan, + MPIDI_NM_exscan, + MPIDI_NM_neighbor_allgather, + MPIDI_NM_neighbor_allgatherv, + MPIDI_NM_neighbor_alltoall, + MPIDI_NM_neighbor_alltoallv, + MPIDI_NM_neighbor_alltoallw, + MPIDI_NM_ineighbor_allgather, + MPIDI_NM_ineighbor_allgatherv, + MPIDI_NM_ineighbor_alltoall, + MPIDI_NM_ineighbor_alltoallv, + MPIDI_NM_ineighbor_alltoallw, + MPIDI_NM_ibarrier, + MPIDI_NM_ibcast, + MPIDI_NM_iallgather, + MPIDI_NM_iallgatherv, + MPIDI_NM_iallreduce, + MPIDI_NM_ialltoall, + MPIDI_NM_ialltoallv, + MPIDI_NM_ialltoallw, + MPIDI_NM_iexscan, + MPIDI_NM_igather, + MPIDI_NM_igatherv, + MPIDI_NM_ireduce_scatter_block, + MPIDI_NM_ireduce_scatter, + MPIDI_NM_ireduce, + MPIDI_NM_iscan, + MPIDI_NM_iscatter, + MPIDI_NM_iscatterv, + MPIDI_NM_datatype_commit, + MPIDI_NM_datatype_dup, + MPIDI_NM_datatype_destroy, + MPIDI_NM_op_commit, + MPIDI_NM_op_destroy +}; +#endif diff --git a/src/mpid/ch4/netmod/ucx/globals.c b/src/mpid/ch4/netmod/ucx/globals.c new file mode 100644 index 0000000..d914681 --- /dev/null +++ b/src/mpid/ch4/netmod/ucx/globals.c @@ -0,0 +1,13 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2016 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Mellanox Technologies Ltd. + * Copyright (C) Mellanox Technologies Ltd. 2016. ALL RIGHTS RESERVED + */ +#include <mpidimpl.h> +#include "ucx_impl.h" +#include "ucx_types.h" + +MPIDI_UCX_global_t MPIDI_UCX_global = { 0 }; diff --git a/src/mpid/ch4/netmod/ucx/netmod_direct.h b/src/mpid/ch4/netmod/ucx/netmod_direct.h new file mode 100644 index 0000000..0fd8bdf --- /dev/null +++ b/src/mpid/ch4/netmod/ucx/netmod_direct.h @@ -0,0 +1,34 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2016 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Mellanox Technologies Ltd. + * Copyright (C) Mellanox Technologies Ltd. 2016. ALL RIGHTS RESERVED + */ +#ifndef NETMOD_DIRECT_H_INCLUDED +#define NETMOD_DIRECT_H_INCLUDED + +#include "ucx_progress.h" +#include "ucx_request.h" +#include "ucx_probe.h" +#include "ucx_init.h" +#ifdef MPICH_UCX_AM_ONLY +#include "ucx_am_send.h" +#include "ucx_am_recv.h" +#include "ucx_am_win.h" +#include "ucx_am_rma.h" +#else +#include "ucx_send.h" +#include "ucx_recv.h" +#include "ucx_win.h" +#include "ucx_rma.h" +#endif +#include "ucx_am.h" +#include "ucx_spawn.h" +#include "ucx_comm.h" +#include "ucx_datatype.h" +#include "ucx_op.h" +#include "ucx_proc.h" +#include "ucx_coll.h" +#endif /* NETMOD_DIRECT_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/ucx/subconfigure.m4 b/src/mpid/ch4/netmod/ucx/subconfigure.m4 new file mode 100644 index 0000000..7dc5a21 --- /dev/null +++ b/src/mpid/ch4/netmod/ucx/subconfigure.m4 @@ -0,0 +1,61 @@ +[#] start of __file__ +dnl MPICH_SUBCFG_AFTER=src/mpid/ch4 + +AC_DEFUN([PAC_SUBCFG_PREREQ_]PAC_SUBCFG_AUTO_SUFFIX,[ + AM_COND_IF([BUILD_CH4],[ + for net in $ch4_netmods ; do + AS_CASE([$net],[ucx],[build_ch4_netmod_ucx=yes]) + if test $net = "ucx" ; then + AC_DEFINE(HAVE_CH4_NETMOD_UCX,1,[UCX netmod is built]) + if test "$build_ch4_locality_info" != "yes" ; then + AC_DEFINE(MPIDI_BUILD_CH4_LOCALITY_INFO, 1, [CH4 should build locality info]) + build_ch4_locality_info="yes" + fi + fi + done + ]) + AM_CONDITIONAL([BUILD_CH4_NETMOD_UCX],[test "X$build_ch4_netmod_ucx" = "Xyes"]) + + AC_ARG_WITH(ch4-netmod-ucx-args, + [ --with-ch4-netmod-ucx-args=arg1:arg2:arg3 + CH4 OFI netmod arguments: + am-only - Do not use UCX tagged or RMA communication. + ], + [ucx_netmod_args=$withval], + [ucx_netmod_args=]) + +dnl Parse the device arguments + SAVE_IFS=$IFS + IFS=':' + args_array=$ucx_netmod_args + do_am_only=false + echo "Parsing Arguments for OFI Netmod" + for arg in $args_array; do + case ${arg} in + am-only) + do_am_only=true + echo " ---> CH4::UCX Disable native tagged and RMA communication : $arg" + esac + done + IFS=$SAVE_IFS + + if [test "$do_am_only" = "true"]; then + AC_MSG_NOTICE([Disabling native UCX tagged and RMA communication]) + PAC_APPEND_FLAG([-DMPICH_UCX_AM_ONLY], [CPPFLAGS]) + fi +])dnl + +AC_DEFUN([PAC_SUBCFG_BODY_]PAC_SUBCFG_AUTO_SUFFIX,[ +AM_COND_IF([BUILD_CH4_NETMOD_UCX],[ + AC_MSG_NOTICE([RUNNING CONFIGURE FOR ch4:ucx]) + + PAC_SET_HEADER_LIB_PATH(ucx) + PAC_PUSH_FLAG(LIBS) + PAC_CHECK_HEADER_LIB_FATAL(ucx, ucp/api/ucp.h, ucp, ucp_config_read) + PAC_POP_FLAG(LIBS) + PAC_APPEND_FLAG([-lucp],[WRAPPER_LIBS]) + +])dnl end AM_COND_IF(BUILD_CH4_NETMOD_OFI,...) +])dnl end _BODY + +[#] end of __file__ diff --git a/src/mpid/ch4/netmod/ucx/ucx_am.h b/src/mpid/ch4/netmod/ucx/ucx_am.h new file mode 100644 index 0000000..992729d --- /dev/null +++ b/src/mpid/ch4/netmod/ucx/ucx_am.h @@ -0,0 +1,615 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2016 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Mellanox Technologies Ltd. + * Copyright (C) Mellanox Technologies Ltd. 2016. ALL RIGHTS RESERVED + */ +#ifndef NETMOD_UCX_AM_H_INCLUDED +#define NETMOD_UCX_AM_H_INCLUDED + +#include "ucx_impl.h" + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_reg_hdr_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_reg_hdr_handler(int handler_id, + MPIDI_NM_am_origin_handler_fn origin_handler_fn, + MPIDI_NM_am_target_handler_fn target_handler_fn) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_REG_HDR_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_REG_HDR_HANDLER); + + MPIDI_UCX_global.am_handlers[handler_id] = target_handler_fn; + MPIDI_UCX_global.send_cmpl_handlers[handler_id] = origin_handler_fn; + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_REG_HDR_HANDLER); + return mpi_errno; + fn_fail: + goto fn_exit; +} + + +static inline void MPIDI_UCX_send_am_callback(void *request, ucs_status_t status) +{ + MPIDI_UCX_ucp_request_t *ucp_request = (MPIDI_UCX_ucp_request_t *) request; + + if (ucp_request->req) { + MPIR_Request *req = ucp_request->req; + int handler_id = req->dev.ch4.ch4u.netmod_am.ucx.handler_id; + + MPL_free(req->dev.ch4.ch4u.netmod_am.ucx.pack_buffer); + req->dev.ch4.ch4u.netmod_am.ucx.pack_buffer = NULL; + MPIDI_UCX_global.send_cmpl_handlers[handler_id] (req); + ucp_request->req = NULL; + } + else { + ucp_request->req = (void *) TRUE; + } + + fn_exit: + return; + fn_fail: + goto fn_exit; +} + +static inline void MPIDI_UCX_inject_am_callback(void *request, ucs_status_t status) +{ + MPIDI_UCX_ucp_request_t *ucp_request = (MPIDI_UCX_ucp_request_t *) request; + + if (ucp_request->req) { + MPL_free(ucp_request->req); + ucp_request->req = NULL; + ucp_request_release(ucp_request); + } + else { + ucp_request->req = (void *) TRUE; + } + + fn_exit: + return; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_send_am_hdr +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_send_am_hdr(int rank, + MPIR_Comm * comm, + int handler_id, + const void *am_hdr, + size_t am_hdr_sz, MPIR_Request * sreq, void *netmod_context) +{ + int mpi_errno = MPI_SUCCESS, c; + MPIDI_UCX_ucp_request_t *ucp_request; + ucp_ep_h ep; + uint64_t ucx_tag; + char *send_buf; + MPIDI_UCX_am_header_t ucx_hdr; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_SEND_AM_HDR); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_SEND_AM_HDR); + + ep = MPIDI_UCX_COMM_TO_EP(comm, rank); + ucx_tag = MPIDI_UCX_init_tag(0, 0, MPIDI_UCX_AM_TAG); + + /* initialize our portion of the hdr */ + ucx_hdr.handler_id = handler_id; + ucx_hdr.data_sz = 0; + + /* just pack and send for now */ + send_buf = MPL_malloc(am_hdr_sz + sizeof(ucx_hdr)); + MPIR_Memcpy(send_buf, &ucx_hdr, sizeof(ucx_hdr)); + MPIR_Memcpy(send_buf + sizeof(ucx_hdr), am_hdr, am_hdr_sz); + + ucp_request = (MPIDI_UCX_ucp_request_t *) ucp_tag_send_nb(ep, send_buf, + am_hdr_sz + sizeof(ucx_hdr), + ucp_dt_make_contig(1), ucx_tag, + &MPIDI_UCX_send_am_callback); + MPIDI_CH4_UCX_REQUEST(ucp_request, tag_send_nb); + + /* send is done. free all resources and complete the request */ + if (ucp_request == NULL) { + MPL_free(send_buf); + MPIDI_UCX_global.send_cmpl_handlers[handler_id] (sreq); + goto fn_exit; + } + + /* request completed between the UCP call and now. free resources + * and complete the send request */ + if (ucp_request->req) { + MPL_free(send_buf); + MPIDI_UCX_global.send_cmpl_handlers[handler_id] (sreq); + ucp_request->req = NULL; + ucp_request_release(ucp_request); + } + else { + /* set the ch4r request inside the UCP request */ + sreq->dev.ch4.ch4u.netmod_am.ucx.pack_buffer = send_buf; + sreq->dev.ch4.ch4u.netmod_am.ucx.handler_id = handler_id; + ucp_request->req = sreq; + ucp_request_release(ucp_request); + } + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_SEND_AM_HDR); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_send_am +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_send_am(int rank, + MPIR_Comm * comm, + int handler_id, + const void *am_hdr, + size_t am_hdr_sz, + const void *data, + MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq, void *netmod_context) +{ + int mpi_errno = MPI_SUCCESS, c; + MPIDI_UCX_ucp_request_t *ucp_request; + ucp_ep_h ep; + uint64_t ucx_tag; + char *send_buf; + size_t data_sz; + MPI_Aint dt_true_lb, last; + MPIR_Datatype *dt_ptr; + int dt_contig; + MPIDI_UCX_am_header_t ucx_hdr; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_SEND_AM); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_SEND_AM); + + MPIDI_Datatype_get_info(count, datatype, dt_contig, data_sz, dt_ptr, dt_true_lb); + if (handler_id == MPIDI_CH4U_SEND && + am_hdr_sz + sizeof(MPIDI_UCX_am_header_t) + data_sz > MPIDI_UCX_MAX_AM_EAGER_SZ) { + MPIDI_CH4U_send_long_req_msg_t lreq_hdr; + + MPIR_Memcpy(&lreq_hdr.hdr, am_hdr, am_hdr_sz); + lreq_hdr.data_sz = data_sz; + lreq_hdr.sreq_ptr = (uint64_t) sreq; + MPIDI_CH4U_REQUEST(sreq, req->lreq).src_buf = data; + MPIDI_CH4U_REQUEST(sreq, req->lreq).count = count; + dtype_add_ref_if_not_builtin(datatype); + MPIDI_CH4U_REQUEST(sreq, req->lreq).datatype = datatype; + MPIDI_CH4U_REQUEST(sreq, req->lreq).msg_tag = lreq_hdr.hdr.msg_tag; + MPIDI_CH4U_REQUEST(sreq, src_rank) = rank; + mpi_errno = MPIDI_NM_inject_am_hdr(rank, comm, MPIDI_CH4U_SEND_LONG_REQ, + &lreq_hdr, sizeof(lreq_hdr), NULL); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + goto fn_exit; + } + + ep = MPIDI_UCX_COMM_TO_EP(comm, rank); + ucx_tag = MPIDI_UCX_init_tag(0, 0, MPIDI_UCX_AM_TAG); + + /* initialize our portion of the hdr */ + ucx_hdr.handler_id = handler_id; + ucx_hdr.data_sz = data_sz; + + if (dt_contig) { + /* just pack and send for now */ + send_buf = MPL_malloc(data_sz + am_hdr_sz + sizeof(ucx_hdr)); + MPIR_Memcpy(send_buf, &ucx_hdr, sizeof(ucx_hdr)); + MPIR_Memcpy(send_buf + sizeof(ucx_hdr), am_hdr, am_hdr_sz); + MPIR_Memcpy(send_buf + am_hdr_sz + sizeof(ucx_hdr), data + dt_true_lb, data_sz); + } + else { + size_t segment_first; + struct MPIDU_Segment *segment_ptr; + segment_ptr = MPIDU_Segment_alloc(); + MPIR_ERR_CHKANDJUMP1(segment_ptr == NULL, mpi_errno, + MPI_ERR_OTHER, "**nomem", "**nomem %s", "Send MPIDU_Segment_alloc"); + MPIDU_Segment_init(data, count, datatype, segment_ptr, 0); + segment_first = 0; + last = data_sz; + send_buf = MPL_malloc(data_sz + am_hdr_sz + sizeof(ucx_hdr)); + + MPIR_Memcpy(send_buf, &ucx_hdr, sizeof(ucx_hdr)); + MPIR_Memcpy(send_buf + sizeof(ucx_hdr), am_hdr, am_hdr_sz); + MPIDU_Segment_pack(segment_ptr, segment_first, &last, + send_buf + am_hdr_sz + sizeof(ucx_hdr)); + MPIDU_Segment_free(segment_ptr); + } + + ucp_request = (MPIDI_UCX_ucp_request_t *) ucp_tag_send_nb(ep, send_buf, + data_sz + am_hdr_sz + sizeof(ucx_hdr), + ucp_dt_make_contig(1), ucx_tag, + &MPIDI_UCX_send_am_callback); + MPIDI_CH4_UCX_REQUEST(ucp_request, tag_send_nb); + /* send is done. free all resources and complete the request */ + if (ucp_request == NULL) { + MPL_free(send_buf); + MPIDI_UCX_global.send_cmpl_handlers[handler_id] (sreq); + goto fn_exit; + } + + /* request completed between the UCP call and now. free resources + * and complete the send request */ + if (ucp_request->req) { + MPL_free(send_buf); + MPIDI_UCX_global.send_cmpl_handlers[handler_id] (sreq); + ucp_request->req = NULL; + ucp_request_release(ucp_request); + } + else { + /* set the ch4r request inside the UCP request */ + sreq->dev.ch4.ch4u.netmod_am.ucx.pack_buffer = send_buf; + sreq->dev.ch4.ch4u.netmod_am.ucx.handler_id = handler_id; + ucp_request->req = sreq; + ucp_request_release(ucp_request); + } + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_SEND_AM); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_send_amv_hdr +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_send_amv_hdr(int rank, + MPIR_Comm * comm, + int handler_id, + struct iovec *am_hdr, + size_t iov_len, MPIR_Request * sreq, void *netmod_context) +{ + int mpi_errno = MPI_SUCCESS; + size_t am_hdr_sz = 0, i; + char *am_hdr_buf; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_UCX_SEND_AMV_HDR); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_UCX_SEND_AMV_HDR); + + for (i = 0; i < iov_len; i++) { + am_hdr_sz += am_hdr[i].iov_len; + } + + am_hdr_buf = (char *) MPL_malloc(am_hdr_sz); + MPIR_Assert(am_hdr_buf); + am_hdr_sz = 0; + + for (i = 0; i < iov_len; i++) { + MPIR_Memcpy(am_hdr_buf + am_hdr_sz, am_hdr[i].iov_base, am_hdr[i].iov_len); + am_hdr_sz += am_hdr[i].iov_len; + } + + mpi_errno = MPIDI_NM_send_am_hdr(rank, comm, handler_id, am_hdr_buf, am_hdr_sz, + sreq, netmod_context); + MPL_free(am_hdr_buf); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_UCX_SEND_AMV_HDR); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_send_amv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_send_amv(int rank, + MPIR_Comm * comm, + int handler_id, + struct iovec *am_hdr, + size_t iov_len, + const void *data, + MPI_Count count, + MPI_Datatype datatype, + MPIR_Request * sreq, void *netmod_context) +{ + int mpi_errno = MPI_SUCCESS; + size_t am_hdr_sz = 0, i; + char *am_hdr_buf; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_UCX_SEND_AMV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_UCX_SEND_AMV); + + for (i = 0; i < iov_len; i++) { + am_hdr_sz += am_hdr[i].iov_len; + } + + am_hdr_buf = (char *) MPL_malloc(am_hdr_sz); + MPIR_Assert(am_hdr_buf); + am_hdr_sz = 0; + + for (i = 0; i < iov_len; i++) { + MPIR_Memcpy(am_hdr_buf + am_hdr_sz, am_hdr[i].iov_base, am_hdr[i].iov_len); + am_hdr_sz += am_hdr[i].iov_len; + } + + mpi_errno = MPIDI_NM_send_am(rank, comm, handler_id, am_hdr_buf, am_hdr_sz, + data, count, datatype, sreq, netmod_context); + + MPL_free(am_hdr_buf); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_UCX_SEND_AMV); + return mpi_errno; +} + + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_send_am_hdr_reply +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_send_am_hdr_reply(MPIR_Context_id_t context_id, + int src_rank, + int handler_id, + const void *am_hdr, + size_t am_hdr_sz, MPIR_Request * sreq) +{ + + return MPIDI_NM_send_am_hdr(src_rank, MPIDI_CH4U_context_id_to_comm(context_id), handler_id, + am_hdr, am_hdr_sz, sreq, NULL); +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_send_am_reply +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_send_am_reply(MPIR_Context_id_t context_id, + int src_rank, + int handler_id, + const void *am_hdr, + size_t am_hdr_sz, + const void *data, MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq) +{ + int mpi_errno = MPI_SUCCESS, c; + MPIDI_UCX_ucp_request_t *ucp_request; + ucp_ep_h ep; + uint64_t ucx_tag; + char *send_buf; + size_t data_sz; + MPI_Aint dt_true_lb, last; + MPIR_Datatype *dt_ptr; + int dt_contig; + MPIDI_UCX_am_header_t ucx_hdr; + MPIR_Comm *use_comm; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_SEND_AM); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_SEND_AM); + + use_comm = MPIDI_CH4U_context_id_to_comm(context_id); + ep = MPIDI_UCX_COMM_TO_EP(use_comm, src_rank); + ucx_tag = MPIDI_UCX_init_tag(0, 0, MPIDI_UCX_AM_TAG); + + MPIDI_Datatype_get_info(count, datatype, dt_contig, data_sz, dt_ptr, dt_true_lb); + + /* initialize our portion of the hdr */ + ucx_hdr.handler_id = handler_id; + ucx_hdr.data_sz = data_sz; + + if (dt_contig) { + /* just pack and send for now */ + send_buf = MPL_malloc(data_sz + am_hdr_sz + sizeof(ucx_hdr)); + MPIR_Memcpy(send_buf, &ucx_hdr, sizeof(ucx_hdr)); + MPIR_Memcpy(send_buf + sizeof(ucx_hdr), am_hdr, am_hdr_sz); + MPIR_Memcpy(send_buf + am_hdr_sz + sizeof(ucx_hdr), data + dt_true_lb, data_sz); + + ucp_request = (MPIDI_UCX_ucp_request_t *) ucp_tag_send_nb(ep, send_buf, + data_sz + am_hdr_sz + + sizeof(ucx_hdr), + ucp_dt_make_contig(1), ucx_tag, + &MPIDI_UCX_send_am_callback); + MPIDI_CH4_UCX_REQUEST(ucp_request, tag_send_nb); + } + + /* send is done. free all resources and complete the request */ + if (ucp_request == NULL) { + MPL_free(send_buf); + MPIDI_UCX_global.send_cmpl_handlers[handler_id] (sreq); + goto fn_exit; + } + + /* request completed between the UCP call and now. free resources + * and complete the send request */ + if (ucp_request->req) { + MPL_free(send_buf); + MPIDI_UCX_global.send_cmpl_handlers[handler_id] (sreq); + ucp_request->req = NULL; + ucp_request_release(ucp_request); + } + else { + /* set the ch4r request inside the UCP request */ + sreq->dev.ch4.ch4u.netmod_am.ucx.pack_buffer = send_buf; + sreq->dev.ch4.ch4u.netmod_am.ucx.handler_id = handler_id; + ucp_request->req = sreq; + ucp_request_release(ucp_request); + + } + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_SEND_AM); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +static inline int MPIDI_NM_send_amv_reply(MPIR_Context_id_t context_id, + int src_rank, + int handler_id, + struct iovec *am_hdr, + size_t iov_len, + const void *data, MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq) +{ + int mpi_errno = MPI_SUCCESS; + size_t am_hdr_sz = 0, i; + char *am_hdr_buf; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_UCX_SEND_AMV_REPLY); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_UCX_SEND_AMV_REPLY); + + for (i = 0; i < iov_len; i++) { + am_hdr_sz += am_hdr[i].iov_len; + } + + am_hdr_buf = (char *) MPL_malloc(am_hdr_sz); + + MPIR_Assert(am_hdr_buf); + am_hdr_sz = 0; + + for (i = 0; i < iov_len; i++) { + MPIR_Memcpy(am_hdr_buf + am_hdr_sz, am_hdr[i].iov_base, am_hdr[i].iov_len); + am_hdr_sz += am_hdr[i].iov_len; + } + + mpi_errno = MPIDI_NM_send_am_reply(context_id, src_rank, handler_id, am_hdr_buf, am_hdr_sz, + data, count, datatype, sreq); + MPL_free(am_hdr_buf); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_UCX_SEND_AMV_REPLY); + return mpi_errno; +} + +static inline size_t MPIDI_NM_am_hdr_max_sz(void) +{ + return (MPIDI_UCX_MAX_AM_EAGER_SZ - sizeof(MPIDI_UCX_am_header_t)); +} + +static inline int MPIDI_NM_inject_am_hdr(int rank, + MPIR_Comm * comm, + int handler_id, + const void *am_hdr, size_t am_hdr_sz, void *netmod_context) +{ + int mpi_errno = MPI_SUCCESS, c; + MPIDI_UCX_ucp_request_t *ucp_request; + ucp_ep_h ep; + uint64_t ucx_tag; + char *send_buf; + MPIDI_UCX_am_header_t ucx_hdr; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_SEND_AM_HDR); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_SEND_AM_HDR); + + ep = MPIDI_UCX_COMM_TO_EP(comm, rank); + ucx_tag = MPIDI_UCX_init_tag(0, 0, MPIDI_UCX_AM_TAG); + + /* initialize our portion of the hdr */ + ucx_hdr.handler_id = handler_id; + ucx_hdr.data_sz = 0; + + /* just pack and send for now */ + send_buf = MPL_malloc(am_hdr_sz + sizeof(ucx_hdr)); + MPIR_Memcpy(send_buf, &ucx_hdr, sizeof(ucx_hdr)); + MPIR_Memcpy(send_buf + sizeof(ucx_hdr), am_hdr, am_hdr_sz); + + ucp_request = (MPIDI_UCX_ucp_request_t *) ucp_tag_send_nb(ep, send_buf, + am_hdr_sz + sizeof(ucx_hdr), + ucp_dt_make_contig(1), ucx_tag, + &MPIDI_UCX_inject_am_callback); + MPIDI_CH4_UCX_REQUEST(ucp_request, tag_send_nb); + + if (ucp_request == NULL) { + /* inject is done */ + MPL_free(send_buf); + } + else if (ucp_request->req) { + MPL_free(send_buf); + ucp_request->req = NULL; + ucp_request_release(ucp_request); + } + else { + ucp_request->req = send_buf; + } + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_SEND_AM_HDR); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +static inline int MPIDI_NM_inject_am_hdr_reply(MPIR_Context_id_t context_id, + int src_rank, + int handler_id, const void *am_hdr, size_t am_hdr_sz) +{ + int mpi_errno = MPI_SUCCESS, c; + MPIDI_UCX_ucp_request_t *ucp_request; + ucp_ep_h ep; + uint64_t ucx_tag; + char *send_buf; + MPIDI_UCX_am_header_t ucx_hdr; + MPIR_Comm *use_comm; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_INJECT_AM_HDR_REPLY); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_INJECT_AM_HDR_REPLY); + + use_comm = MPIDI_CH4U_context_id_to_comm(context_id); + ep = MPIDI_UCX_COMM_TO_EP(use_comm, src_rank); + ucx_tag = MPIDI_UCX_init_tag(0, 0, MPIDI_UCX_AM_TAG); + + /* initialize our portion of the hdr */ + ucx_hdr.handler_id = handler_id; + + /* just pack and send for now */ + send_buf = MPL_malloc(am_hdr_sz + sizeof(ucx_hdr)); + MPIR_Memcpy(send_buf, &ucx_hdr, sizeof(ucx_hdr)); + MPIR_Memcpy(send_buf + sizeof(ucx_hdr), am_hdr, am_hdr_sz); + ucp_request = (MPIDI_UCX_ucp_request_t *) ucp_tag_send_nb(ep, send_buf, + am_hdr_sz + sizeof(ucx_hdr), + ucp_dt_make_contig(1), ucx_tag, + &MPIDI_UCX_inject_am_callback); + MPIDI_CH4_UCX_REQUEST(ucp_request, tag_send_nb); + + if (ucp_request == NULL) { + /* inject is done */ + MPL_free(send_buf); + } + else if (ucp_request->req) { + MPL_free(send_buf); + ucp_request->req = NULL; + ucp_request_release(ucp_request); + } + else { + ucp_request->req = send_buf; + } + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_SEND_AM); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +static inline size_t MPIDI_NM_am_inject_max_sz(void) +{ + return MPIDI_NM_am_hdr_max_sz(); +} + +static inline int MPIDI_NM_am_recv(MPIR_Request * req) +{ + int mpi_errno = MPI_SUCCESS; + MPIDI_CH4U_send_long_ack_msg_t msg; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_UCX_AM_MATCHED); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_UCX_AM_MATCHED); + + msg.sreq_ptr = (MPIDI_CH4U_REQUEST(req, req->rreq.peer_req_ptr)); + msg.rreq_ptr = (uint64_t) req; + MPIR_Assert((void *) msg.sreq_ptr != NULL); + mpi_errno = MPIDI_NM_inject_am_hdr_reply(MPIDI_CH4U_get_context(MPIDI_CH4U_REQUEST(req, tag)), + MPIDI_CH4U_REQUEST(req, src_rank), + MPIDI_CH4U_SEND_LONG_ACK, &msg, sizeof(msg)); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_UCX_AM_MATCHED); + return mpi_errno; + fn_fail: + goto fn_exit; +} + + +#endif /* NETMOD_UCX_AM_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/ucx/ucx_am_recv.h b/src/mpid/ch4/netmod/ucx/ucx_am_recv.h new file mode 100644 index 0000000..2f4f821 --- /dev/null +++ b/src/mpid/ch4/netmod/ucx/ucx_am_recv.h @@ -0,0 +1,60 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_UCX_AM_RECV_H_INCLUDED +#define NETMOD_UCX_AM_RECV_H_INCLUDED + +#include "ucx_impl.h" + +static inline int MPIDI_NM_recv(void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, + int context_offset, MPI_Status * status, MPIR_Request ** request) +{ + return MPIDI_CH4U_recv(buf, count, datatype, rank, tag, comm, context_offset, status, request); +} + +static inline int MPIDI_NM_recv_init(void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + return MPIDI_CH4U_recv_init(buf, count, datatype, rank, tag, comm, context_offset, request); +} + +static inline int MPIDI_NM_imrecv(void *buf, + int count, + MPI_Datatype datatype, + MPIR_Request * message, MPIR_Request ** rreqp) +{ + return MPIDI_CH4U_imrecv(buf, count, datatype, message, rreqp); +} + +static inline int MPIDI_NM_irecv(void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + return MPIDI_CH4U_irecv(buf, count, datatype, rank, tag, comm, context_offset, request); +} + +static inline int MPIDI_NM_cancel_recv(MPIR_Request * rreq) +{ + return MPIDI_CH4U_cancel_recv(rreq); +} + +#endif /* NETMOD_UCX_RECV_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/ucx/ucx_am_rma.h b/src/mpid/ch4/netmod/ucx/ucx_am_rma.h new file mode 100644 index 0000000..7863464 --- /dev/null +++ b/src/mpid/ch4/netmod/ucx/ucx_am_rma.h @@ -0,0 +1,149 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ + +#ifndef NETMOD_UCX_AM_RMA_H_INCLUDED +#define NETMOD_UCX_AM_RMA_H_INCLUDED + +#include "ucx_impl.h" + +static inline int MPIDI_NM_put(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, MPI_Datatype target_datatype, MPIR_Win * win) +{ + return MPIDI_CH4U_put(origin_addr, origin_count, origin_datatype, + target_rank, target_disp, target_count, target_datatype, win); +} + +static inline int MPIDI_NM_get(void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, MPI_Datatype target_datatype, MPIR_Win * win) +{ + return MPIDI_CH4U_get(origin_addr, origin_count, origin_datatype, + target_rank, target_disp, target_count, target_datatype, win); +} + +static inline int MPIDI_NM_rput(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPIR_Win * win, MPIR_Request ** request) +{ + return MPIDI_CH4U_rput(origin_addr, origin_count, origin_datatype, + target_rank, target_disp, target_count, target_datatype, win, request); +} + + +static inline int MPIDI_NM_compare_and_swap(const void *origin_addr, + const void *compare_addr, + void *result_addr, + MPI_Datatype datatype, + int target_rank, MPI_Aint target_disp, MPIR_Win * win) +{ + return MPIDI_CH4U_compare_and_swap(origin_addr, compare_addr, result_addr, + datatype, target_rank, target_disp, win); +} + +static inline int MPIDI_NM_raccumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPI_Op op, MPIR_Win * win, MPIR_Request ** request) +{ + return MPIDI_CH4U_raccumulate(origin_addr, origin_count, origin_datatype, + target_rank, target_disp, target_count, + target_datatype, op, win, request); +} + +static inline int MPIDI_NM_rget_accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + void *result_addr, + int result_count, + MPI_Datatype result_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPI_Op op, MPIR_Win * win, MPIR_Request ** request) +{ + return MPIDI_CH4U_rget_accumulate(origin_addr, origin_count, origin_datatype, + result_addr, result_count, result_datatype, + target_rank, target_disp, target_count, + target_datatype, op, win, request); +} + +static inline int MPIDI_NM_fetch_and_op(const void *origin_addr, + void *result_addr, + MPI_Datatype datatype, + int target_rank, + MPI_Aint target_disp, MPI_Op op, MPIR_Win * win) +{ + return MPIDI_CH4U_fetch_and_op(origin_addr, result_addr, datatype, + target_rank, target_disp, op, win); +} + + +static inline int MPIDI_NM_rget(void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPIR_Win * win, MPIR_Request ** request) +{ + return MPIDI_CH4U_rget(origin_addr, origin_count, origin_datatype, + target_rank, target_disp, target_count, target_datatype, win, request); +} + + +static inline int MPIDI_NM_get_accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + void *result_addr, + int result_count, + MPI_Datatype result_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, MPI_Op op, MPIR_Win * win) +{ + return MPIDI_CH4U_get_accumulate(origin_addr, origin_count, origin_datatype, + result_addr, result_count, result_datatype, + target_rank, target_disp, target_count, + target_datatype, op, win); +} + +static inline int MPIDI_NM_accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, MPI_Op op, MPIR_Win * win) +{ + return MPIDI_CH4U_accumulate(origin_addr, origin_count, origin_datatype, + target_rank, target_disp, target_count, target_datatype, op, win); +} + +#endif /* NETMOD_UCX_AM_RMA_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/ucx/ucx_am_send.h b/src/mpid/ch4/netmod/ucx/ucx_am_send.h new file mode 100644 index 0000000..9fd977b --- /dev/null +++ b/src/mpid/ch4/netmod/ucx/ucx_am_send.h @@ -0,0 +1,128 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_UCX_AM_SEND_H_INCLUDED +#define NETMOD_UCX_AM_SEND_H_INCLUDED + +#include "ucx_impl.h" + +static inline int MPIDI_NM_send(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + return MPIDI_CH4U_send(buf, count, datatype, rank, tag, comm, context_offset, request); +} + +static inline int MPIDI_NM_rsend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + return MPIDI_CH4U_rsend(buf, count, datatype, rank, tag, comm, context_offset, request); +} + + + +static inline int MPIDI_NM_irsend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + return MPIDI_CH4U_irsend(buf, count, datatype, rank, tag, comm, context_offset, request); +} + +static inline int MPIDI_NM_ssend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + return MPIDI_CH4U_ssend(buf, count, datatype, rank, tag, comm, context_offset, request); +} + +static inline int MPIDI_NM_startall(int count, MPIR_Request * requests[]) +{ + return MPIDI_CH4U_startall(count, requests); +} + +static inline int MPIDI_NM_send_init(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + return MPIDI_CH4U_send_init(buf, count, datatype, rank, tag, comm, context_offset, request); +} + +static inline int MPIDI_NM_ssend_init(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + return MPIDI_CH4U_ssend_init(buf, count, datatype, rank, tag, comm, context_offset, request); +} + +static inline int MPIDI_NM_bsend_init(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + return MPIDI_CH4U_bsend_init(buf, count, datatype, rank, tag, comm, context_offset, request); +} + +static inline int MPIDI_NM_rsend_init(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + return MPIDI_CH4U_rsend_init(buf, count, datatype, rank, tag, comm, context_offset, request); +} + +static inline int MPIDI_NM_isend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + return MPIDI_CH4U_isend(buf, count, datatype, rank, tag, comm, context_offset, request); +} + +static inline int MPIDI_NM_issend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + return MPIDI_CH4U_issend(buf, count, datatype, rank, tag, comm, context_offset, request); +} + +static inline int MPIDI_NM_cancel_send(MPIR_Request * sreq) +{ + return MPIDI_CH4U_cancel_send(sreq); +} + +#endif /* NETMOD_UCX_SEND_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/ucx/ucx_am_win.h b/src/mpid/ch4/netmod/ucx/ucx_am_win.h new file mode 100644 index 0000000..2bf5da9 --- /dev/null +++ b/src/mpid/ch4/netmod/ucx/ucx_am_win.h @@ -0,0 +1,160 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef NETMOD_UCX_AM_WIN_H_INCLUDED +#define NETMOD_UCX_AM_WIN_H_INCLUDED + +#include "ucx_impl.h" + +static inline int MPIDI_NM_win_set_info(MPIR_Win * win, MPIR_Info * info) +{ + return MPIDI_CH4R_win_set_info(win, info); +} + + +static inline int MPIDI_NM_win_start(MPIR_Group * group, int assert, MPIR_Win * win) +{ + return MPIDI_CH4R_win_start(group, assert, win); +} + + +static inline int MPIDI_NM_win_complete(MPIR_Win * win) +{ + return MPIDI_CH4R_win_complete(win); +} + +static inline int MPIDI_NM_win_post(MPIR_Group * group, int assert, MPIR_Win * win) +{ + return MPIDI_CH4R_win_post(group, assert, win); +} + + +static inline int MPIDI_NM_win_wait(MPIR_Win * win) +{ + return MPIDI_CH4R_win_wait(win); +} + + +static inline int MPIDI_NM_win_test(MPIR_Win * win, int *flag) +{ + return MPIDI_CH4R_win_test(win, flag); +} + +static inline int MPIDI_NM_win_lock(int lock_type, int rank, int assert, MPIR_Win * win) +{ + return MPIDI_CH4R_win_lock(lock_type, rank, assert, win); +} + + +static inline int MPIDI_NM_win_unlock(int rank, MPIR_Win * win) +{ + return MPIDI_CH4R_win_unlock(rank, win); +} + +static inline int MPIDI_NM_win_get_info(MPIR_Win * win, MPIR_Info ** info_p_p) +{ + return MPIDI_CH4R_win_get_info(win, info_p_p); +} + + +static inline int MPIDI_NM_win_free(MPIR_Win ** win_ptr) +{ + return MPIDI_CH4R_win_free(win_ptr); +} + +static inline int MPIDI_NM_win_fence(int assert, MPIR_Win * win) +{ + return MPIDI_CH4R_win_fence(assert, win); +} + +static inline int MPIDI_NM_win_create(void *base, + MPI_Aint length, + int disp_unit, + MPIR_Info * info, MPIR_Comm * comm_ptr, MPIR_Win ** win_ptr) +{ + return MPIDI_CH4R_win_create(base, length, disp_unit, info, comm_ptr, win_ptr); +} + +static inline int MPIDI_NM_win_attach(MPIR_Win * win, void *base, MPI_Aint size) +{ + return MPIDI_CH4R_win_attach(win, base, size); +} + +static inline int MPIDI_NM_win_allocate_shared(MPI_Aint size, + int disp_unit, + MPIR_Info * info_ptr, + MPIR_Comm * comm_ptr, + void **base_ptr, MPIR_Win ** win_ptr) +{ + return MPIDI_CH4R_win_allocate_shared(size, disp_unit, info_ptr, comm_ptr, base_ptr, win_ptr); +} + +static inline int MPIDI_NM_win_detach(MPIR_Win * win, const void *base) +{ + return MPIDI_CH4R_win_detach(win, base); +} + +static inline int MPIDI_NM_win_shared_query(MPIR_Win * win, + int rank, + MPI_Aint * size, int *disp_unit, void *baseptr) +{ + return MPIDI_CH4R_win_shared_query(win, rank, size, disp_unit, baseptr); +} + +static inline int MPIDI_NM_win_allocate(MPI_Aint size, + int disp_unit, + MPIR_Info * info, + MPIR_Comm * comm, void *baseptr, MPIR_Win ** win) +{ + return MPIDI_CH4R_win_allocate(size, disp_unit, info, comm, baseptr, win); +} + +static inline int MPIDI_NM_win_flush(int rank, MPIR_Win * win) +{ + return MPIDI_CH4R_win_flush(rank, win); +} + +static inline int MPIDI_NM_win_flush_local_all(MPIR_Win * win) +{ + return MPIDI_CH4R_win_flush_local_all(win); +} + +static inline int MPIDI_NM_win_unlock_all(MPIR_Win * win) +{ + return MPIDI_CH4R_win_unlock_all(win); +} + +static inline int MPIDI_NM_win_create_dynamic(MPIR_Info * info, MPIR_Comm * comm, MPIR_Win ** win) +{ + return MPIDI_CH4R_win_create_dynamic(info, comm, win); +} + +static inline int MPIDI_NM_win_flush_local(int rank, MPIR_Win * win) +{ + return MPIDI_CH4R_win_flush_local(rank, win); +} + +static inline int MPIDI_NM_win_sync(MPIR_Win * win) +{ + return MPIDI_CH4R_win_sync(win); +} + +static inline int MPIDI_NM_win_flush_all(MPIR_Win * win) +{ + return MPIDI_CH4R_win_flush_all(win); +} + +static inline int MPIDI_NM_win_lock_all(int assert, MPIR_Win * win) +{ + return MPIDI_CH4R_win_lock_all(assert, win); +} + + +#endif /* NETMOD_UCX_AM_WIN_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/ucx/ucx_coll.h b/src/mpid/ch4/netmod/ucx/ucx_coll.h new file mode 100644 index 0000000..185c5b9 --- /dev/null +++ b/src/mpid/ch4/netmod/ucx/ucx_coll.h @@ -0,0 +1,867 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2016 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Mellanox Technologies Ltd. + * Copyright (C) Mellanox Technologies Ltd. 2016. ALL RIGHTS RESERVED + */ +#ifndef NETMOD_UCX_COLL_H_INCLUDED +#define NETMOD_UCX_COLL_H_INCLUDED + +#include "ucx_impl.h" + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_barrier +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_barrier(MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_BARRIER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_BARRIER); + + mpi_errno = MPIR_Barrier(comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_BARRIER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_bcast +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_bcast(void *buffer, int count, MPI_Datatype datatype, + int root, MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_BCAST); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_BCAST); + + mpi_errno = MPIR_Bcast(buffer, count, datatype, root, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_BCAST); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_allreduce +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_allreduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_ALLREDUCE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_ALLREDUCE); + + mpi_errno = MPIR_Allreduce(sendbuf, recvbuf, count, datatype, op, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_ALLREDUCE); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_allgather +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_allgather(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_ALLGATHER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_ALLGATHER); + + mpi_errno = MPIR_Allgather(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, + comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_ALLGATHER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_allgatherv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_allgatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, const int *displs, + MPI_Datatype recvtype, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_ALLGATHERV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_ALLGATHERV); + + mpi_errno = MPIR_Allgatherv(sendbuf, sendcount, sendtype, + recvbuf, recvcounts, displs, recvtype, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_ALLGATHERV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_gather +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_gather(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + int root, MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_GATHER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_GATHER); + + mpi_errno = MPIR_Gather(sendbuf, sendcount, sendtype, recvbuf, recvcount, + recvtype, root, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_GATHER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_gatherv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_gatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, const int *displs, + MPI_Datatype recvtype, int root, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_GATHERV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_GATHERV); + + mpi_errno = MPIR_Gatherv(sendbuf, sendcount, sendtype, + recvbuf, recvcounts, displs, recvtype, root, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_GATHERV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_scatter +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_scatter(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + int root, MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_SCATTER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_SCATTER); + + mpi_errno = MPIR_Scatter(sendbuf, sendcount, sendtype, + recvbuf, recvcount, recvtype, root, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_SCATTER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_scatterv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_scatterv(const void *sendbuf, const int *sendcounts, + const int *displs, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + int root, MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_SCATTERV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_SCATTERV); + + mpi_errno = MPIR_Scatterv(sendbuf, sendcounts, displs, + sendtype, recvbuf, recvcount, recvtype, root, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_SCATTERV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_alltoall +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_alltoall(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_ALLTOALL); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_ALLTOALL); + + mpi_errno = MPIR_Alltoall(sendbuf, sendcount, sendtype, recvbuf, recvcount, + recvtype, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_ALLTOALL); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_alltoallv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_alltoallv(const void *sendbuf, const int *sendcounts, + const int *sdispls, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, + const int *rdispls, MPI_Datatype recvtype, + MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_ALLTOALLV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_ALLTOALLV); + + mpi_errno = MPIR_Alltoallv(sendbuf, sendcounts, sdispls, + sendtype, recvbuf, recvcounts, rdispls, recvtype, comm_ptr, errflag); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_ALLTOALLV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_alltoallw +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_alltoallw(const void *sendbuf, const int sendcounts[], + const int sdispls[], const MPI_Datatype sendtypes[], + void *recvbuf, const int recvcounts[], + const int rdispls[], const MPI_Datatype recvtypes[], + MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_ALLTOALLW); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_ALLTOALLW); + + mpi_errno = MPIR_Alltoallw(sendbuf, sendcounts, sdispls, + sendtypes, recvbuf, recvcounts, + rdispls, recvtypes, comm_ptr, errflag); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_ALLTOALLW); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_reduce +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_reduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, int root, + MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_REDUCE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_REDUCE); + + mpi_errno = MPIR_Reduce(sendbuf, recvbuf, count, datatype, op, root, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_REDUCE); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_reduce_scatter +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_reduce_scatter(const void *sendbuf, void *recvbuf, + const int recvcounts[], MPI_Datatype datatype, + MPI_Op op, MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_REDUCE_SCATTER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_REDUCE_SCATTER); + + mpi_errno = MPIR_Reduce_scatter(sendbuf, recvbuf, recvcounts, datatype, op, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_REDUCE_SCATTER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_reduce_scatter_block +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_reduce_scatter_block(const void *sendbuf, void *recvbuf, + int recvcount, MPI_Datatype datatype, + MPI_Op op, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_REDUCE_SCATTER_BLOCK); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_REDUCE_SCATTER_BLOCK); + + mpi_errno = MPIR_Reduce_scatter_block(sendbuf, recvbuf, recvcount, + datatype, op, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_REDUCE_SCATTER_BLOCK); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_scan +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_scan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_SCAN); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_SCAN); + + mpi_errno = MPIR_Scan(sendbuf, recvbuf, count, datatype, op, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_SCAN); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_exscan +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_exscan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_EXSCAN); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_EXSCAN); + + mpi_errno = MPIR_Exscan(sendbuf, recvbuf, count, datatype, op, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_EXSCAN); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_neighbor_allgather +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_neighbor_allgather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, int recvcount, + MPI_Datatype recvtype, MPIR_Comm * comm_ptr) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_NEIGHBOR_ALLGATHER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_NEIGHBOR_ALLGATHER); + + mpi_errno = + MPIR_Neighbor_allgather_impl(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, + comm_ptr); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_NEIGHBOR_ALLGATHER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_neighbor_allgatherv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_neighbor_allgatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int recvcounts[], const int displs[], + MPI_Datatype recvtype, MPIR_Comm * comm_ptr) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_NEIGHBOR_ALLGATHERV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_NEIGHBOR_ALLGATHERV); + + mpi_errno = MPIR_Neighbor_allgatherv_impl(sendbuf, sendcount, sendtype, + recvbuf, recvcounts, displs, recvtype, comm_ptr); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_NEIGHBOR_ALLGATHERV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_neighbor_alltoall +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_neighbor_alltoall(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, int recvcount, + MPI_Datatype recvtype, MPIR_Comm * comm_ptr) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_NEIGHBOR_ALLTOALL); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_NEIGHBOR_ALLTOALL); + + mpi_errno = MPIR_Neighbor_alltoall_impl(sendbuf, sendcount, sendtype, + recvbuf, recvcount, recvtype, comm_ptr); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_NEIGHBOR_ALLTOALL); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_neighbor_alltoallv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_neighbor_alltoallv(const void *sendbuf, const int sendcounts[], + const int sdispls[], MPI_Datatype sendtype, + void *recvbuf, const int recvcounts[], + const int rdispls[], MPI_Datatype recvtype, + MPIR_Comm * comm_ptr) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_NEIGHBOR_ALLTOALLV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_NEIGHBOR_ALLTOALLV); + + mpi_errno = MPIR_Neighbor_alltoallv_impl(sendbuf, sendcounts, sdispls, sendtype, + recvbuf, recvcounts, rdispls, recvtype, comm_ptr); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_NEIGHBOR_ALLTOALLV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_neighbor_alltoallw +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_neighbor_alltoallw(const void *sendbuf, const int sendcounts[], + const MPI_Aint sdispls[], + const MPI_Datatype sendtypes[], void *recvbuf, + const int recvcounts[], const MPI_Aint rdispls[], + const MPI_Datatype recvtypes[], MPIR_Comm * comm_ptr) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_NEIGHBOR_ALLTOALLW); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_NEIGHBOR_ALLTOALLW); + + mpi_errno = MPIR_Neighbor_alltoallw_impl(sendbuf, sendcounts, sdispls, sendtypes, + recvbuf, recvcounts, rdispls, recvtypes, comm_ptr); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_NEIGHBOR_ALLTOALLW); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ineighbor_allgather +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ineighbor_allgather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, int recvcount, + MPI_Datatype recvtype, MPIR_Comm * comm_ptr, + MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_INEIGHBOR_ALLGATHER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_INEIGHBOR_ALLGATHER); + + mpi_errno = MPIR_Ineighbor_allgather_impl(sendbuf, sendcount, sendtype, + recvbuf, recvcount, recvtype, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_INEIGHBOR_ALLGATHER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ineighbor_allgatherv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ineighbor_allgatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int recvcounts[], const int displs[], + MPI_Datatype recvtype, MPIR_Comm * comm_ptr, + MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_INEIGHBOR_ALLGATHERV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_INEIGHBOR_ALLGATHERV); + + mpi_errno = MPIR_Ineighbor_allgatherv_impl(sendbuf, sendcount, sendtype, + recvbuf, recvcounts, displs, recvtype, + comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_INEIGHBOR_ALLGATHERV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ineighbor_alltoall +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ineighbor_alltoall(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_INEIGHBOR_ALLTOALL); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_INEIGHBOR_ALLTOALL); + + mpi_errno = MPIR_Ineighbor_alltoall_impl(sendbuf, sendcount, sendtype, + recvbuf, recvcount, recvtype, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_INEIGHBOR_ALLTOALL); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ineighbor_alltoallv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ineighbor_alltoallv(const void *sendbuf, const int sendcounts[], + const int sdispls[], MPI_Datatype sendtype, + void *recvbuf, const int recvcounts[], + const int rdispls[], MPI_Datatype recvtype, + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_INEIGHBOR_ALLTOALLV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_INEIGHBOR_ALLTOALLV); + + mpi_errno = MPIR_Ineighbor_alltoallv_impl(sendbuf, sendcounts, sdispls, sendtype, + recvbuf, recvcounts, rdispls, recvtype, + comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_INEIGHBOR_ALLTOALLV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ineighbor_alltoallw +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ineighbor_alltoallw(const void *sendbuf, const int sendcounts[], + const MPI_Aint sdispls[], + const MPI_Datatype sendtypes[], void *recvbuf, + const int recvcounts[], const MPI_Aint rdispls[], + const MPI_Datatype recvtypes[], MPIR_Comm * comm_ptr, + MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_INEIGHBOR_ALLTOALLW); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_INEIGHBOR_ALLTOALLW); + + mpi_errno = MPIR_Ineighbor_alltoallw_impl(sendbuf, sendcounts, sdispls, sendtypes, + recvbuf, recvcounts, rdispls, recvtypes, + comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_INEIGHBOR_ALLTOALLW); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ibarrier +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ibarrier(MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IBARRIER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IBARRIER); + + mpi_errno = MPIR_Ibarrier_impl(comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IBARRIER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ibcast +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ibcast(void *buffer, int count, MPI_Datatype datatype, + int root, MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IBCAST); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IBCAST); + + mpi_errno = MPIR_Ibcast_impl(buffer, count, datatype, root, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IBCAST); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_iallgather +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_iallgather(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IALLGATHER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IALLGATHER); + + mpi_errno = MPIR_Iallgather_impl(sendbuf, sendcount, sendtype, recvbuf, + recvcount, recvtype, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IALLGATHER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_iallgatherv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_iallgatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, const int *displs, + MPI_Datatype recvtype, MPIR_Comm * comm_ptr, + MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IALLGATHERV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IALLGATHERV); + + mpi_errno = MPIR_Iallgatherv_impl(sendbuf, sendcount, sendtype, + recvbuf, recvcounts, displs, recvtype, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IALLGATHERV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_iallreduce +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_iallreduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm, + MPI_Request * request) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IALLREDUCE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IALLREDUCE); + + mpi_errno = MPIR_Iallreduce_impl(sendbuf, recvbuf, count, datatype, op, comm, request); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IALLREDUCE); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ialltoall +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ialltoall(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IALLTOALL); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IALLTOALL); + + mpi_errno = MPIR_Ialltoall_impl(sendbuf, sendcount, sendtype, recvbuf, + recvcount, recvtype, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IALLTOALL); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ialltoallv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ialltoallv(const void *sendbuf, const int *sendcounts, + const int *sdispls, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, + const int *rdispls, MPI_Datatype recvtype, + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IALLTOALLV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IALLTOALLV); + + mpi_errno = MPIR_Ialltoallv_impl(sendbuf, sendcounts, sdispls, + sendtype, recvbuf, recvcounts, + rdispls, recvtype, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IALLTOALLV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ialltoallw +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ialltoallw(const void *sendbuf, const int *sendcounts, + const int *sdispls, const MPI_Datatype sendtypes[], + void *recvbuf, const int *recvcounts, + const int *rdispls, const MPI_Datatype recvtypes[], + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IALLTOALLW); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IALLTOALLW); + + mpi_errno = MPIR_Ialltoallw_impl(sendbuf, sendcounts, sdispls, + sendtypes, recvbuf, recvcounts, + rdispls, recvtypes, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IALLTOALLW); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_iexscan +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_iexscan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm_ptr, + MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IEXSCAN); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IEXSCAN); + + mpi_errno = MPIR_Iexscan_impl(sendbuf, recvbuf, count, datatype, op, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IEXSCAN); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_igather +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_igather(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + int root, MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IGATHER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IGATHER); + + mpi_errno = MPIR_Igather_impl(sendbuf, sendcount, sendtype, recvbuf, + recvcount, recvtype, root, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IGATHER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_igatherv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_igatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, const int *displs, + MPI_Datatype recvtype, int root, MPIR_Comm * comm_ptr, + MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IGATHERV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IGATHERV); + + mpi_errno = MPIR_Igatherv_impl(sendbuf, sendcount, sendtype, + recvbuf, recvcounts, displs, recvtype, root, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IGATHERV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ireduce_scatter_block +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ireduce_scatter_block(const void *sendbuf, void *recvbuf, + int recvcount, MPI_Datatype datatype, + MPI_Op op, MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IREDUCE_SCATTER_BLOCK); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IREDUCE_SCATTER_BLOCK); + + mpi_errno = MPIR_Ireduce_scatter_block_impl(sendbuf, recvbuf, recvcount, + datatype, op, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IREDUCE_SCATTER_BLOCK); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ireduce_scatter +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ireduce_scatter(const void *sendbuf, void *recvbuf, + const int recvcounts[], MPI_Datatype datatype, + MPI_Op op, MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IREDUCE_SCATTER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IREDUCE_SCATTER); + + mpi_errno = MPIR_Ireduce_scatter_impl(sendbuf, recvbuf, recvcounts, datatype, op, + comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IREDUCE_SCATTER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_ireduce +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ireduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, int root, + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_IREDUCE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_IREDUCE); + + mpi_errno = MPIR_Ireduce_impl(sendbuf, recvbuf, count, datatype, op, root, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_IREDUCE); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_iscan +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_iscan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm_ptr, + MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_ISCAN); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_ISCAN); + + mpi_errno = MPIR_Iscan_impl(sendbuf, recvbuf, count, datatype, op, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_ISCAN); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_iscatter +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_iscatter(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + int root, MPIR_Comm * comm, MPI_Request * request) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_ISCATTER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_ISCATTER); + + mpi_errno = MPIR_Iscatter_impl(sendbuf, sendcount, sendtype, recvbuf, + recvcount, recvtype, root, comm, request); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_ISCATTER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_iscatterv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_iscatterv(const void *sendbuf, const int *sendcounts, + const int *displs, MPI_Datatype sendtype, + void *recvbuf, int recvcount, + MPI_Datatype recvtype, int root, + MPIR_Comm * comm, MPI_Request * request) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NM_ISCATTERV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NM_ISCATTERV); + + mpi_errno = MPIR_Iscatterv_impl(sendbuf, sendcounts, displs, sendtype, + recvbuf, recvcount, recvtype, root, comm, request); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NM_ISCATTERV); + return mpi_errno; +} + +#endif /* NETMOD_UCX_COLL_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/ucx/ucx_comm.h b/src/mpid/ch4/netmod/ucx/ucx_comm.h new file mode 100644 index 0000000..b152c29 --- /dev/null +++ b/src/mpid/ch4/netmod/ucx/ucx_comm.h @@ -0,0 +1,48 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2016 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Mellanox Technologies Ltd. + * Copyright (C) Mellanox Technologies Ltd. 2016. ALL RIGHTS RESERVED + */ +#ifndef NETMOD_UCX_COMM_H_INCLUDED +#define NETMOD_UCX_COMM_H_INCLUDED + +#include "ucx_impl.h" + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_comm_create +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_comm_create(MPIR_Comm * comm) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_UCX_COMM_CREATE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_UCX_COMM_CREATE); + + mpi_errno = MPIDI_CH4U_init_comm(comm); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_UCX_COMM_CREATE); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_comm_destroy +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_comm_destroy(MPIR_Comm * comm) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_UCX_COMM_DESTROY); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_UCX_COMM_DESTROY); + + mpi_errno = MPIDI_CH4U_destroy_comm(comm); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_UCX_COMM_DESTROY); + return mpi_errno; +} + +#endif /* NETMOD_UCX_COMM_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/ucx/ucx_datatype.h b/src/mpid/ch4/netmod/ucx/ucx_datatype.h new file mode 100644 index 0000000..32f1df1 --- /dev/null +++ b/src/mpid/ch4/netmod/ucx/ucx_datatype.h @@ -0,0 +1,150 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2016 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Mellanox Technologies Ltd. + * Copyright (C) Mellanox Technologies Ltd. 2016. ALL RIGHTS RESERVED + */ +#ifndef NETMOD_UCX_DATATYPE_H_INCLUDED +#define NETMOD_UCX_DATATYPE_H_INCLUDED + +#include "ucx_impl.h" +#include "ucx_types.h" +#include <ucp/api/ucp.h> +struct MPIDI_UCX_pack_state { + + MPID_Segment *segment_ptr; + MPI_Aint packsize; +}; + +static inline void *MPIDI_UCX_Start_pack(void *context, const void *buffer, size_t count) +{ + + MPI_Datatype *datatype = (MPI_Datatype *) context; + MPID_Segment *segment_ptr; + struct MPIDI_UCX_pack_state *state; + MPI_Aint packsize; + state = MPL_malloc(sizeof(struct MPIDI_UCX_pack_state)); + segment_ptr = MPID_Segment_alloc(); + MPIR_Pack_size_impl(count, *datatype, &packsize); +/* Todo: Add error handling */ + MPID_Segment_init(buffer, count, *datatype, segment_ptr, 1); + state->packsize = packsize; + state->segment_ptr = segment_ptr; + return (void *) state; +} + +static inline void *MPIDI_UCX_Start_unpack(void *context, void *buffer, size_t count) +{ + + MPI_Datatype *datatype = (MPI_Datatype *) context; + MPID_Segment *segment_ptr; + struct MPIDI_UCX_pack_state *state; + MPI_Aint packsize; + + state = MPL_malloc(sizeof(struct MPIDI_UCX_pack_state)); + MPIR_Pack_size_impl(count, *datatype, &packsize); + + segment_ptr = MPID_Segment_alloc(); + +/* Todo: Add error handling */ + MPID_Segment_init(buffer, count, *datatype, segment_ptr, 1); + state->packsize = packsize; + state->segment_ptr = segment_ptr; + return (void *) state; + +} + +static inline size_t MPIDI_UCX_Packed_size(void *state) +{ + + struct MPIDI_UCX_pack_state *pack_state = (struct MPIDI_UCX_pack_state *) state; + + return (size_t) pack_state->packsize; +} + +static inline size_t MPIDI_UCX_Pack(void *state, size_t offset, void *dest, size_t max_length) +{ + + struct MPIDI_UCX_pack_state *pack_state = (struct MPIDI_UCX_pack_state *) state; + MPI_Aint last = MPL_MIN(pack_state->packsize, offset + max_length); + + MPID_Segment_pack(pack_state->segment_ptr, offset, &last, dest); + + return (size_t) last - offset; +} + +static inline ucs_status_t MPIDI_UCX_Unpack(void *state, size_t offset, const void *src, + size_t count) +{ + + struct MPIDI_UCX_pack_state *pack_state = (struct MPIDI_UCX_pack_state *) state; + size_t last = MPL_MIN(pack_state->packsize, offset + count); + + MPID_Segment_unpack(pack_state->segment_ptr, offset, &last, (void *) src); + + return UCS_OK; +} + +static inline void MPIDI_UCX_Finish_pack(void *state) +{ + + struct MPIDI_UCX_pack_state *pack_state = (struct MPIDI_UCX_pack_state *) state; + MPID_Segment_free(pack_state->segment_ptr); + MPL_free(pack_state); + +} + + +static ucp_generic_dt_ops_t MPIDI_UCX_datatype_ops = { + .start_pack = MPIDI_UCX_Start_pack, + .start_unpack = MPIDI_UCX_Start_unpack, + .packed_size = MPIDI_UCX_Packed_size, + .pack = MPIDI_UCX_Pack, + .unpack = MPIDI_UCX_Unpack, + .finish = MPIDI_UCX_Finish_pack +}; + + +static inline void MPIDI_NM_datatype_destroy(MPIR_Datatype * datatype_p) +{ + + + if (datatype_p->is_committed && (int) datatype_p->dev.netmod.ucx.ucp_datatype >= 0) { + ucp_dt_destroy(datatype_p->dev.netmod.ucx.ucp_datatype); + datatype_p->dev.netmod.ucx.ucp_datatype = -1; + } + + return; +} + +static inline void MPIDI_NM_datatype_commit(MPIR_Datatype * datatype_p) +{ + ucp_datatype_t ucp_datatype; + ucs_status_t status; + size_t size; + int is_contig; + + + datatype_p->dev.netmod.ucx.ucp_datatype = -1; + MPID_Datatype_is_contig(datatype_p->handle, &is_contig); + + if (!is_contig) { + + status = ucp_dt_create_generic(&MPIDI_UCX_datatype_ops, datatype_p, &ucp_datatype); + MPIR_Assertp(status == UCS_OK); + datatype_p->dev.netmod.ucx.ucp_datatype = ucp_datatype; + + } + + return; +} + +static inline void MPIDI_NM_datatype_dup(MPIR_Datatype * old_datatype_p, + MPIR_Datatype * new_datatype_p) +{ + return; +} + +#endif /* NETMOD_UCX_DATATYPE_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/ucx/ucx_impl.h b/src/mpid/ch4/netmod/ucx/ucx_impl.h new file mode 100644 index 0000000..6f7200d --- /dev/null +++ b/src/mpid/ch4/netmod/ucx/ucx_impl.h @@ -0,0 +1,151 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2016 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Mellanox Technologies Ltd. + * Copyright (C) Mellanox Technologies Ltd. 2016. ALL RIGHTS RESERVED + */ +#ifndef NETMOD_UCX_IMPL_H_INCLUDED +#define NETMOD_UCX_IMPL_H_INCLUDED + +#include <mpidimpl.h> +#include "ucx_types.h" +#include "mpidch4r.h" +#include "ch4_impl.h" + +#include <ucs/type/status.h> + +#define MPIDI_UCX_COMM(comm) ((comm)->dev.ch4.netmod.ucx) +#define MPIDI_UCX_REQ(req) ((req)->dev.ch4.netmod.ucx) +#define COMM_TO_INDEX(comm,rank) MPIDIU_comm_rank_to_pid(comm, rank, NULL, NULL) +#define MPIDI_UCX_COMM_TO_EP(comm,rank) \ + MPIDI_UCX_AV(MPIDIU_comm_rank_to_av(comm, rank)).dest + +#define MPIDI_UCX_WIN(win) ((win)->dev.netmod.ucx) +#define MPIDI_UCX_WIN_INFO(win, rank) MPIDI_UCX_WIN(win).info_table[rank] + +static inline uint64_t MPIDI_UCX_init_tag(MPIR_Context_id_t contextid, int source, uint64_t tag) +{ + uint64_t ucp_tag = 0; + ucp_tag = contextid; + ucp_tag = (ucp_tag << MPIDI_UCX_SOURCE_SHIFT); + ucp_tag |= source; + ucp_tag = (ucp_tag << MPIDI_UCX_TAG_SHIFT); + ucp_tag |= (MPIDI_UCX_TAG_MASK & tag); + return ucp_tag; +} + +#ifndef MPIR_TAG_ERROR_BIT +#define MPIR_TAG_ERROR_BIT (1 << 30) +#endif +#ifndef MPIR_TAG_PROC_FAILURE_BIT +#define MPIR_TAG_PROC_FAILURE_BIT (1 << 29) +#endif + +static inline uint64_t MPIDI_UCX_tag_mask(int mpi_tag, int src) +{ + uint64_t tag_mask; + tag_mask = ~(MPIR_TAG_PROC_FAILURE_BIT | MPIR_TAG_ERROR_BIT); + if (mpi_tag == MPI_ANY_TAG) + tag_mask &= ~MPIDI_UCX_TAG_MASK; + + if (src == MPI_ANY_SOURCE) + tag_mask &= ~(MPIDI_UCX_SOURCE_MASK); + + return tag_mask; +} + +static inline uint64_t MPIDI_UCX_recv_tag(int mpi_tag, int src, MPIR_Context_id_t contextid) +{ + uint64_t ucp_tag = contextid; + + ucp_tag = (ucp_tag << MPIDI_UCX_SOURCE_SHIFT); + if (src != MPI_ANY_SOURCE) + ucp_tag |= (src & UCS_MASK(MPIDI_UCX_CONTEXT_RANK_BITS)); + ucp_tag = ucp_tag << MPIDI_UCX_TAG_SHIFT; + if (mpi_tag != MPI_ANY_TAG) + ucp_tag |= (MPIDI_UCX_TAG_MASK & mpi_tag); + return ucp_tag; +} + +static inline int MPIDI_UCX_get_tag(uint64_t match_bits) +{ + return ((int) (match_bits & MPIDI_UCX_TAG_MASK)); +} + +static inline int MPIDI_UCX_get_source(uint64_t match_bits) +{ + return ((int) ((match_bits & MPIDI_UCX_SOURCE_MASK) >> MPIDI_UCX_TAG_SHIFT)); +} + + +#define MPIDI_UCX_ERR MPIR_ERR_CHKANDJUMP4 + +#define MPIDI_UCX_CHK_STATUS(STATUS,STR) \ + do { \ + MPIDI_UCX_ERR((STATUS!=UCS_OK && STATUS!=UCS_INPROGRESS),\ + mpi_errno, \ + MPI_ERR_OTHER, \ + "**ch4_ucx_nm_"#STR, \ + "**ch4_ucx_nm_"#STR" %s %d %s %s", \ + __SHORT_FILE__, \ + __LINE__, \ + FCNAME, \ + ucs_status_string(STATUS)); \ + } while (0) + + + +#define MPIDI_UCX_PMI_ERROR(_errno,STR) \ + do \ + { \ + MPIDI_UCX_ERR(_errno!=PMI_SUCCESS, \ + mpi_errno, \ + MPI_ERR_OTHER, \ + "**ch4_ucx_nm_pmi"#STR, \ + "**ch4_ucx_nm_mpi"#STR" %s %d %s %s", \ + __SHORT_FILE__, \ + __LINE__, \ + FCNAME, \ + #STR); \ + } while (0) + +#define MPIDI_CH4_UCX_MPI_ERROR(_errno) \ + do \ + { \ + if (unlikely(_errno!=MPI_SUCCESS)) MPIR_ERR_POP(mpi_errno); \ + } while (0) + +#define MPIDI_CH4_UCX_STR_ERRCHK(_errno,STR) \ + do \ + { \ + MPIDI_UCX_ERR(_errno!=MPL_STR_SUCCESS, \ + mpi_errno, \ + MPI_ERR_OTHER, \ + "**ch4_ucx_nm_"#STR, \ + "**ch4_ucx_nm_"#STR" %s %d %s %s", \ + __SHORT_FILE__, \ + __LINE__, \ + FCNAME, \ + #STR); \ + } while (0) + + + +#define MPIDI_CH4_UCX_REQUEST(_req, STR) \ + do { \ + MPIDI_UCX_ERR(UCS_PTR_IS_ERR(_req), \ + mpi_errno, \ + MPI_ERR_OTHER, \ + "**ch4_ucx_nm_"#STR, \ + "**ch4_ucx_nm_"#STR" %s %d %s %s", \ + __SHORT_FILE__, \ + __LINE__, \ + FCNAME, \ + ucs_status_string(UCS_PTR_STATUS(_req))); \ + } while (0) + +extern int MPIR_Datatype_init_names(void); + +#endif /* NETMOD_UCX_IMPL_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/ucx/ucx_init.h b/src/mpid/ch4/netmod/ucx/ucx_init.h new file mode 100644 index 0000000..06b9e7e --- /dev/null +++ b/src/mpid/ch4/netmod/ucx/ucx_init.h @@ -0,0 +1,330 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2016 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Mellanox Technologies Ltd. + * Copyright (C) Mellanox Technologies Ltd. 2016. ALL RIGHTS RESERVED + */ +#ifndef NETMOD_UCX_INIT_H_INCLUDED +#define NETMOD_UCX_INIT_H_INCLUDED + +#include "ucx_impl.h" +#include "mpir_cvars.h" +#include "ucx_types.h" +#include "pmi.h" +#include <ucp/api/ucp.h> +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_init +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_init(int rank, + int size, + int appnum, + int *tag_ub, + MPIR_Comm * comm_world, + MPIR_Comm * comm_self, + int spawned, int num_contexts, void **netmod_contexts) +{ + int mpi_errno = MPI_SUCCESS, thr_err, pmi_errno; + int str_errno = MPL_STR_SUCCESS; + ucp_config_t *config; + ucs_status_t ucx_status; + uint64_t features = 0; + int status; + char valS[MPIDI_UCX_KVSAPPSTRLEN], *val; + char keyS[MPIDI_UCX_KVSAPPSTRLEN]; + char remote_addr[MPIDI_UCX_KVSAPPSTRLEN]; + size_t maxlen = MPIDI_UCX_KVSAPPSTRLEN; + // char *table = NULL; + int i; + ucp_params_t ucp_params; + int avtid = 0, max_n_avts; + + size_t address_length = 0; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_INIT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_INIT); + + ucx_status = ucp_config_read(NULL, NULL, &config); + MPIDI_UCX_CHK_STATUS(ucx_status, read_config); + + /* For now use only the tag feature */ + features = UCP_FEATURE_TAG | UCP_FEATURE_RMA; + ucp_params.features = features; + ucp_params.request_size = sizeof(MPIDI_UCX_ucp_request_t); + ucp_params.request_init = MPIDI_UCX_Request_init_callback; + ucp_params.request_cleanup = NULL; + ucx_status = ucp_init(&ucp_params, config, &MPIDI_UCX_global.context); + MPIDI_UCX_CHK_STATUS(ucx_status, init); + ucp_config_release(config); + + ucx_status = ucp_worker_create(MPIDI_UCX_global.context, UCS_THREAD_MODE_SERIALIZED, + &MPIDI_UCX_global.worker); + MPIDI_UCX_CHK_STATUS(ucx_status, worker_create); + ucx_status = + ucp_worker_get_address(MPIDI_UCX_global.worker, &MPIDI_UCX_global.if_address, + &MPIDI_UCX_global.addrname_len); + MPIDI_UCX_CHK_STATUS(ucx_status, get_worker_address); + + + val = valS; + str_errno = + MPL_str_add_binary_arg(&val, (int *) &maxlen, "UCX", (char *) MPIDI_UCX_global.if_address, + (int) MPIDI_UCX_global.addrname_len); + MPIDI_UCX_global.max_addr_len = MPIDI_UCX_global.addrname_len; + /* MPIDI_CH4_UCX_STR_ERRCHK(str_errno, buscard_len); */ + pmi_errno = PMI_KVS_Get_my_name(MPIDI_UCX_global.kvsname, MPIDI_UCX_KVSAPPSTRLEN); + + val = valS; + sprintf(keyS, "UCX-%d", rank); + pmi_errno = PMI_KVS_Put(MPIDI_UCX_global.kvsname, keyS, val); + MPIDI_UCX_PMI_ERROR(pmi_errno, pmi_put_name); + pmi_errno = PMI_KVS_Commit(MPIDI_UCX_global.kvsname); + MPIDI_UCX_PMI_ERROR(pmi_errno, pmi_commit); + pmi_errno = PMI_Barrier(); + MPIDI_UCX_PMI_ERROR(pmi_errno, pmi_barrier); + + ///table = MPL_malloc(size * MPIDI_UCX_NAME_LEN); + MPIDI_UCX_global.pmi_addr_table = NULL; +// memset(table,0x0, MPIDI_UCX_NAME_LEN*size); + + maxlen = MPIDI_UCX_KVSAPPSTRLEN; + + for (i = 0; i < size; i++) { + sprintf(keyS, "UCX-%d", i); + pmi_errno = PMI_KVS_Get(MPIDI_UCX_global.kvsname, keyS, valS, MPIDI_UCX_KVSAPPSTRLEN); + MPIDI_UCX_PMI_ERROR(pmi_errno, pmi_commit); + str_errno = MPL_str_get_binary_arg(valS, "UCX", remote_addr, + (int) MPIDI_UCX_KVSAPPSTRLEN, (int *) &maxlen); + if (maxlen > MPIDI_UCX_global.max_addr_len) + MPIDI_UCX_global.max_addr_len = maxlen; + /* MPIDI_UCX_STR_ERRCHK(str_errno, buscard_len); */ + ucx_status = ucp_ep_create(MPIDI_UCX_global.worker, + (ucp_address_t *) remote_addr, + &MPIDI_UCX_AV(&MPIDIU_get_av(0, i)).dest); + + MPIDI_UCX_CHK_STATUS(ucx_status, ep_create); + memset(remote_addr, 0x0, maxlen); + } + + MPIDI_CH4U_init(comm_world, comm_self, num_contexts, netmod_contexts); + + mpi_errno = MPIR_Datatype_init_names(); + MPIDI_CH4_UCX_MPI_ERROR(mpi_errno); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_EXIT); + return mpi_errno; + fn_fail: + if (MPIDI_UCX_global.worker != NULL) + ucp_worker_destroy(MPIDI_UCX_global.worker); + + if (MPIDI_UCX_global.context != NULL) + ucp_cleanup(MPIDI_UCX_global.context); + + goto fn_exit; + +} + +static inline int MPIDI_NM_finalize(void) +{ + int mpi_errno = MPI_SUCCESS, thr_err, pmi_errno; + int i, j, max_n_avts; + MPIR_Errflag_t errflag; + MPIR_Comm *comm; + max_n_avts = MPIDIU_get_max_n_avts(); + + for (i = 0; i < max_n_avts; i++) { + for (j = 0; j < MPIDIU_get_av_table(i)->size; j++) + ucp_ep_destroy(MPIDI_UCX_AV(&MPIDIU_get_av(i, j)).dest); + } + pmi_errno = PMI_Barrier(); + MPIDI_UCX_PMI_ERROR(pmi_errno, pmi_barrier); + + + if (MPIDI_UCX_global.worker != NULL) + ucp_worker_destroy(MPIDI_UCX_global.worker); + + if (MPIDI_UCX_global.context != NULL) + ucp_cleanup(MPIDI_UCX_global.context); + + comm = MPIR_Process.comm_world; + MPIR_Comm_release_always(comm); + + comm = MPIR_Process.comm_self; + MPIR_Comm_release_always(comm); + if (MPIDI_UCX_global.pmi_addr_table) + MPL_free(MPIDI_UCX_global.pmi_addr_table); + + MPIDI_CH4U_finalize(); + PMI_Finalize(); + + fn_exit: + return mpi_errno; + fn_fail: + goto fn_exit; + +} + +static inline int MPIDI_NM_comm_get_lpid(MPIR_Comm * comm_ptr, + int idx, int *lpid_ptr, MPL_bool is_remote) +{ + int avtid = 0, lpid = 0; + if (comm_ptr->comm_kind == MPIR_COMM_KIND__INTRACOMM) { + MPIDIU_comm_rank_to_pid(comm_ptr, idx, &lpid, &avtid); + } + else if (is_remote) { + MPIDIU_comm_rank_to_pid(comm_ptr, idx, &lpid, &avtid); + } + else { + MPIDIU_comm_rank_to_pid_local(comm_ptr, idx, &lpid, &avtid); + } + + *lpid_ptr = MPIDIU_LPID_CREATE(avtid, lpid); + return MPI_SUCCESS; + +} + +static inline int allocate_address_table() +{ + + char keyS[MPIDI_UCX_KVSAPPSTRLEN]; + char valS[MPIDI_UCX_KVSAPPSTRLEN]; + int len = MPIDI_UCX_global.max_addr_len; + int i; + int size, maxlen = 1; + size = MPIR_Process.comm_world->local_size; + MPIDI_UCX_global.pmi_addr_table = MPL_malloc(size * len); + memset(MPIDI_UCX_global.pmi_addr_table, 0x0, len * size); + + + for (i = 0; i < size; i++) { + sprintf(keyS, "UCX-%d", i); + PMI_KVS_Get(MPIDI_UCX_global.kvsname, keyS, valS, MPIDI_UCX_KVSAPPSTRLEN); + // MPIDI_UCX_PMI_ERROR(pmi_errno, pmi_commit); + MPL_str_get_binary_arg(valS, "UCX", &MPIDI_UCX_global.pmi_addr_table[len * i], + (int) len, (int *) &maxlen); + } + + +} + +static inline int MPIDI_NM_gpid_get(MPIR_Comm * comm_ptr, int rank, MPIR_Gpid * gpid) +{ + int mpi_errno = MPI_SUCCESS; + int avtid = 0, lpid = 0; + + + int len = MPIDI_UCX_global.max_addr_len; + + MPIDIU_comm_rank_to_pid(comm_ptr, rank, &lpid, &avtid); + MPIR_Assert(rank < comm_ptr->local_size); + + if (MPIDI_UCX_global.pmi_addr_table == NULL) { + allocate_address_table(); + } + memset(MPIDI_UCX_GPID(gpid).addr, 0, len); + memcpy(MPIDI_UCX_GPID(gpid).addr, &MPIDI_UCX_global.pmi_addr_table[lpid * len], len); + MPIR_Assert(len <= sizeof(MPIDI_UCX_GPID(gpid).addr)); + fn_exit: + return mpi_errno; + fn_fail: + goto fn_exit; +} + +static inline int MPIDI_NM_get_node_id(MPIR_Comm * comm, int rank, MPID_Node_id_t * id_p) +{ + MPIDI_CH4U_get_node_id(comm, rank, id_p); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_get_max_node_id(MPIR_Comm * comm, MPID_Node_id_t * max_id_p) +{ + MPIDI_CH4U_get_max_node_id(comm, max_id_p); + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_getallincomm(MPIR_Comm * comm_ptr, + int local_size, MPIR_Gpid local_gpids[], int *singleAVT) +{ + int i; + + for (i = 0; i < comm_ptr->local_size; i++) + MPIDI_GPID_Get(comm_ptr, i, &local_gpids[i]); + + *singleAVT = 0; + return 0; +} + +static inline int MPIDI_NM_gpid_tolpidarray(int size, MPIR_Gpid gpid[], int lpid[]) +{ + + int i, mpi_errno = MPI_SUCCESS; + int *new_avt_procs; + int n_new_procs = 0; + size_t sz; + int max_n_avts; + new_avt_procs = (int *) MPL_malloc(size * sizeof(int)); + max_n_avts = MPIDIU_get_max_n_avts(); + if (MPIDI_UCX_global.pmi_addr_table == NULL) { + allocate_address_table(); + } + + for (i = 0; i < size; i++) { + int j, k; + char tbladdr[128]; + int found = 0; + + for (k = 0; k < max_n_avts; k++) { + if (MPIDIU_get_av_table(k) == NULL) { + continue; + } + for (j = 0; j < MPIDIU_get_av_table(k)->size; j++) { + sz = MPIDI_UCX_global.max_addr_len; // sizeof(MPIDI_UCX_GPID(&gpid[i]).addr); + MPIR_Assert(sz <= sizeof(MPIDI_UCX_GPID(&gpid[i]).addr)); + + if (!memcmp(&MPIDI_UCX_global.pmi_addr_table[j * sz], + MPIDI_UCX_GPID(&gpid[i]).addr, sz)) { + lpid[i] = MPIDIU_LPID_CREATE(k, j); + found = 1; + break; + } + } + } + if (!found) { + new_avt_procs[n_new_procs] = i; + n_new_procs++; + } + } + + /* FIXME: add support for dynamic processes */ + if (n_new_procs > 0) { + mpi_errno = -1; + MPIR_ERR_POP(mpi_errno); + } + + fn_exit: + MPL_free(new_avt_procs); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +static inline int MPIDI_NM_create_intercomm_from_lpids(MPIR_Comm * newcomm_ptr, + int size, const int lpids[]) +{ + return MPI_SUCCESS; +} + +static inline int MPIDI_NM_free_mem(void *ptr) +{ + return MPIDI_CH4U_free_mem(ptr); +} + +static inline void *MPIDI_NM_alloc_mem(size_t size, MPIR_Info * info_ptr) +{ + return MPIDI_CH4U_alloc_mem(size, info_ptr); +} + +#endif /* NETMOD_UCX_INIT_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/ucx/ucx_op.h b/src/mpid/ch4/netmod/ucx/ucx_op.h new file mode 100644 index 0000000..ac10797 --- /dev/null +++ b/src/mpid/ch4/netmod/ucx/ucx_op.h @@ -0,0 +1,24 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2016 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Mellanox Technologies Ltd. + * Copyright (C) Mellanox Technologies Ltd. 2016. ALL RIGHTS RESERVED + */ +#ifndef NETMOD_UCX_OP_H_INCLUDED +#define NETMOD_UCX_OP_H_INCLUDED + +#include "ucx_impl.h" + +static inline void MPIDI_NM_op_destroy(MPIR_Op * op_p) +{ + return; +} + +static inline void MPIDI_NM_op_commit(MPIR_Op * op_p) +{ + return; +} + +#endif /* NETMOD_UCX_OP_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/ucx/ucx_pre.h b/src/mpid/ch4/netmod/ucx/ucx_pre.h new file mode 100644 index 0000000..9779a4d --- /dev/null +++ b/src/mpid/ch4/netmod/ucx/ucx_pre.h @@ -0,0 +1,75 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2016 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Mellanox Technologies Ltd. + * Copyright (C) Mellanox Technologies Ltd. 2016. ALL RIGHTS RESERVED + */ +#ifndef NETMOD_UCX_PRE_H_INCLUDED +#define NETMOD_UCX_PRE_H_INCLUDED + +#include <ucp/api/ucp.h> + +#define HAVE_MPIDI_NM_datatype_commit_hook +#define HAVE_MPIDI_NM_datatype_destroy_hook + +#define MPIDI_UCX_KVSAPPSTRLEN 4096 + +//#define MPIDI_UCX_NAME_LEN (512) +typedef struct { + void *req; +} MPIDI_UCX_ucp_request_t; + +typedef struct { + ucp_datatype_t ucp_datatype; +} MPIDI_UCX_dt_t; + +typedef struct { + union { + ucp_tag_message_h message_handler; + MPIDI_UCX_ucp_request_t *ucp_request; + } a; +} MPIDI_UCX_request_t; + +typedef struct { + int handler_id; + char *pack_buffer; +} MPIDI_UCX_am_request_t; + +typedef struct MPIDI_UCX_am_header_t { + uint64_t handler_id; + uint64_t data_sz; + uint64_t payload[0]; +} MPIDI_UCX_am_header_t; + +typedef struct MPIDI_UCX_win_info { + ucp_rkey_h rkey; + uint64_t addr; + uint32_t disp; +} __attribute__ ((packed)) MPIDI_UCX_win_info_t; + + +typedef struct { + MPIDI_UCX_win_info_t *info_table; + ucp_mem_h mem_h; + int need_local_flush; +} MPIDI_UCX_win_t; + +typedef struct { + char addr[MPIDI_UCX_KVSAPPSTRLEN]; +} MPIDI_UCX_gpid_t; + +typedef struct { + ucp_ep_h dest; +} MPIDI_UCX_addr_t; + +typedef struct { + int dummy; +} MPIDI_UCX_comm_t; + +typedef struct { + int dummy; +} MPIDI_UCX_op_t; + +#endif /* NETMOD_UCX_PRE_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/ucx/ucx_probe.h b/src/mpid/ch4/netmod/ucx/ucx_probe.h new file mode 100644 index 0000000..8806146 --- /dev/null +++ b/src/mpid/ch4/netmod/ucx/ucx_probe.h @@ -0,0 +1,99 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2016 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Mellanox Technologies Ltd. + * Copyright (C) Mellanox Technologies Ltd. 2016. ALL RIGHTS RESERVED + */ +#ifndef NETMOD_UCX_PROBE_H_INCLUDED +#define NETMOD_UCX_PROBE_H_INCLUDED + +#include "ucx_impl.h" +#include "mpidch4.h" + +static inline int ucx_do_iprobe(int source, + int tag, + MPIR_Comm * comm, int context_offset, int *flag, + MPI_Status * status) +{ + int mpi_errno = MPI_SUCCESS; + uint64_t ucp_tag, tag_mask; + MPI_Aint count; + ucp_tag_recv_info_t info; + ucp_tag_message_h message_handler; + tag_mask = MPIDI_UCX_tag_mask(tag, source); + ucp_tag = MPIDI_UCX_recv_tag(tag, source, comm->recvcontext_id + context_offset); + message_handler = ucp_tag_probe_nb(MPIDI_UCX_global.worker, ucp_tag, tag_mask, 0, &info); + if (message_handler == NULL) { + *flag = 0; + goto fn_exit; + } + *flag = true; + if (status == MPI_STATUS_IGNORE) + goto fn_exit; + + status->MPI_ERROR = MPI_SUCCESS; + status->MPI_SOURCE = MPIDI_UCX_get_source(info.sender_tag); + status->MPI_TAG = MPIDI_UCX_get_tag(info.sender_tag); + count = info.length; + MPIR_STATUS_SET_COUNT(*status, count); + fn_exit: + return mpi_errno; + +} + +static inline int MPIDI_NM_improbe(int source, + int tag, + MPIR_Comm * comm, + int context_offset, + int *flag, MPIR_Request ** message, MPI_Status * status) +{ + + + int mpi_errno = MPI_SUCCESS; + uint64_t ucp_tag, tag_mask; + MPI_Aint count; + ucp_tag_recv_info_t info; + ucp_tag_message_h message_handler; + MPIR_Request *req; + + tag_mask = MPIDI_UCX_tag_mask(tag, source); + ucp_tag = MPIDI_UCX_recv_tag(tag, source, comm->recvcontext_id + context_offset); + + message_handler = ucp_tag_probe_nb(MPIDI_UCX_global.worker, ucp_tag, tag_mask, 1, &info); + if (message_handler == NULL) { + *flag = 0; + goto fn_exit; + } + *flag = 1; + req = (MPIR_Request *) MPIR_Request_create(MPIR_REQUEST_KIND__MPROBE); + MPIR_Assert(req); + MPIDI_UCX_REQ(req).a.message_handler = message_handler; + if (status == MPI_STATUS_IGNORE) + goto fn_exit; + + status->MPI_SOURCE = MPIDI_UCX_get_source(info.sender_tag); + status->MPI_TAG = MPIDI_UCX_get_tag(info.sender_tag); + count = info.length; + MPIR_STATUS_SET_COUNT(*status, count); + fn_exit: + *message = req; + return mpi_errno; + fn_fail: + goto fn_exit; +} + + +static inline int MPIDI_NM_iprobe(int source, + int tag, + MPIR_Comm * comm, + int context_offset, int *flag, MPI_Status * status) +{ + int mpi_errno = MPI_SUCCESS; + + mpi_errno = ucx_do_iprobe(source, tag, comm, context_offset, flag, status); + return mpi_errno; +} + +#endif /* NETMOD_UCX_PROBE_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/ucx/ucx_proc.h b/src/mpid/ch4/netmod/ucx/ucx_proc.h new file mode 100644 index 0000000..ca43260 --- /dev/null +++ b/src/mpid/ch4/netmod/ucx/ucx_proc.h @@ -0,0 +1,26 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2016 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Mellanox Technologies Ltd. + * Copyright (C) Mellanox Technologies Ltd. 2016. ALL RIGHTS RESERVED + */ +#ifndef NETMOD_UCX_PROC_H_INCLUDED +#define NETMOD_UCX_PROC_H_INCLUDED + +#include "ucx_impl.h" + +static inline int MPIDI_NM_rank_is_local(int rank, MPIR_Comm * comm) +{ + int ret; + MPIR_FUNC_VERBOSE_STATE_DECL(MPIDI_NETMOD_RANK_IS_LOCAL); + MPIR_FUNC_VERBOSE_ENTER(MPIDI_NETMOD_RANK_IS_LOCAL); + + ret = MPIDI_CH4U_rank_is_local(rank, comm); + + MPIR_FUNC_VERBOSE_EXIT(MPIDI_NETMOD_RANK_IS_LOCAL); + return ret; +} + +#endif /* NETMOD_UCX_PROC_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/ucx/ucx_progress.h b/src/mpid/ch4/netmod/ucx/ucx_progress.h new file mode 100644 index 0000000..c12801d --- /dev/null +++ b/src/mpid/ch4/netmod/ucx/ucx_progress.h @@ -0,0 +1,147 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2016 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Mellanox Technologies Ltd. + * Copyright (C) Mellanox Technologies Ltd. 2016. ALL RIGHTS RESERVED + */ +#ifndef NETMOD_UCX_PROGRESS_H_INCLUDED +#define NETMOD_UCX_PROGRESS_H_INCLUDED + +#include "ucx_impl.h" +//#include "events.h" + +static inline int MPIDI_UCX_am_handler(void *msg, size_t msg_sz) +{ + int mpi_errno; + MPIR_Request *rreq; + void *p_data; + void *in_data; + size_t data_sz, in_data_sz; + MPIDI_NM_am_completion_handler_fn cmpl_handler_fn; + struct iovec *iov; + int i, is_contig, iov_len; + size_t done, curr_len, rem; + MPIDI_UCX_am_header_t *msg_hdr = (MPIDI_UCX_am_header_t *) msg; + + p_data = in_data = (char *) msg_hdr->payload + (msg_sz - msg_hdr->data_sz - sizeof(*msg_hdr)); + in_data_sz = data_sz = msg_hdr->data_sz; + + MPIDI_UCX_global.am_handlers[msg_hdr->handler_id] (msg_hdr->payload, + &p_data, &data_sz, + &is_contig, &cmpl_handler_fn, &rreq); + + if (!rreq) + goto fn_exit; + + if ((!p_data || !data_sz) && cmpl_handler_fn) { + MPIR_STATUS_SET_COUNT(rreq->status, data_sz); + cmpl_handler_fn(rreq); + goto fn_exit; + } + + if (is_contig) { + if (in_data_sz > data_sz) { + rreq->status.MPI_ERROR = MPI_ERR_TRUNCATE; + } + else { + rreq->status.MPI_ERROR = MPI_SUCCESS; + } + + data_sz = MPL_MIN(data_sz, in_data_sz); + MPIR_Memcpy(p_data, in_data, data_sz); + MPIR_STATUS_SET_COUNT(rreq->status, data_sz); + } + else { + done = 0; + rem = in_data_sz; + iov = (struct iovec *) p_data; + iov_len = data_sz; + + for (i = 0; i < iov_len && rem > 0; i++) { + curr_len = MPL_MIN(rem, iov[i].iov_len); + MPIR_Memcpy(iov[i].iov_base, (char *) in_data + done, curr_len); + rem -= curr_len; + done += curr_len; + } + + if (rem) { + rreq->status.MPI_ERROR = MPI_ERR_TRUNCATE; + } + else { + rreq->status.MPI_ERROR = MPI_SUCCESS; + } + + MPIR_STATUS_SET_COUNT(rreq->status, done); + } + + if (cmpl_handler_fn) { + cmpl_handler_fn(rreq); + } + + fn_exit: + return mpi_errno; +} + +static inline void MPIDI_UCX_Handle_am_recv(void *request, ucs_status_t status, + ucp_tag_recv_info_t * info) +{ + int mpi_errno = MPI_SUCCESS; + MPIDI_UCX_ucp_request_t *ucp_request = (MPIDI_UCX_ucp_request_t *) request; + + if (status == UCS_ERR_CANCELED) { + goto fn_exit; + } + fn_exit: + return; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_progress +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_progress(void *netmod_context, int blocking) +{ + int mpi_errno = MPI_SUCCESS; + ucp_tag_recv_info_t info; + MPIDI_UCX_ucp_request_t *ucp_request; + void *am_buf; + ucp_tag_message_h message_handle; + /* check for active messages */ + message_handle = + ucp_tag_probe_nb(MPIDI_UCX_global.worker, MPIDI_UCX_AM_TAG, MPIDI_UCX_AM_TAG, 1, &info); + while (message_handle) { + am_buf = MPL_malloc(info.length); + ucp_request = (MPIDI_UCX_ucp_request_t *) ucp_tag_msg_recv_nb(MPIDI_UCX_global.worker, + am_buf, + info.length, + ucp_dt_make_contig(1), + message_handle, + &MPIDI_UCX_Handle_am_recv); + while (!ucp_request_is_completed(ucp_request)) { + ucp_worker_progress(MPIDI_UCX_global.worker); + } + + ucp_request_release(ucp_request); + MPIDI_UCX_am_handler(am_buf, info.length); + MPL_free(am_buf); + message_handle = + ucp_tag_probe_nb(MPIDI_UCX_global.worker, MPIDI_UCX_AM_TAG, + ~MPIDI_UCX_AM_TAG, 1, &info); + + } + + ucp_worker_progress(MPIDI_UCX_global.worker); + + MPID_THREAD_CS_EXIT(POBJ, MPIDI_THREAD_WORKER_MUTEX); + + fn_exit: + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#endif /* NETMOD_UCX_PROGRESS_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/ucx/ucx_recv.h b/src/mpid/ch4/netmod/ucx/ucx_recv.h new file mode 100644 index 0000000..11eb7cf --- /dev/null +++ b/src/mpid/ch4/netmod/ucx/ucx_recv.h @@ -0,0 +1,244 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2016 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Mellanox Technologies Ltd. + * Copyright (C) Mellanox Technologies Ltd. 2016. ALL RIGHTS RESERVED + */ +#ifndef NETMOD_UCX_RECV_H_INCLUDED +#define NETMOD_UCX_RECV_H_INCLUDED + +#include "ucx_impl.h" + +__ALWAYS_INLINE__ int ucx_irecv_continous(void *buf, + size_t data_sz, + int rank, + int tag, + MPIR_Comm * comm, + int context_offset, MPIR_Request ** request) +{ + + int mpi_errno = MPI_SUCCESS; + uint64_t ucp_tag, tag_mask; + MPIR_Request *req; + MPIDI_UCX_ucp_request_t *ucp_request; +// MPID_THREAD_CS_ENTER(POBJ,MPIDI_THREAD_WORKER_MUTEX); + tag_mask = MPIDI_UCX_tag_mask(tag, rank); + ucp_tag = MPIDI_UCX_recv_tag(tag, rank, comm->recvcontext_id + context_offset); + + ucp_request = (MPIDI_UCX_ucp_request_t *) ucp_tag_recv_nb(MPIDI_UCX_global.worker, + buf, data_sz, ucp_dt_make_contig(1), + ucp_tag, tag_mask, + &MPIDI_UCX_Handle_recv_callback); + + + MPIDI_CH4_UCX_REQUEST(ucp_request, tag_send_nb); + + + if (ucp_request->req == NULL) { + req = MPIR_Request_create(MPIR_REQUEST_KIND__RECV); + MPIR_Request_add_ref(req); + MPIDI_UCX_REQ(req).a.ucp_request = ucp_request; + ucp_request->req = req; + ucp_request_release(ucp_request); + } + else { + req = ucp_request->req; + ucp_request->req = NULL; + MPIDI_UCX_REQ(req).a.ucp_request = NULL; + ucp_request_release(ucp_request); + } + fn_exit: + *request = req; + return mpi_errno; + fn_fail: + goto fn_exit; +} + +__ALWAYS_INLINE__ int ucx_irecv_non_continous(void *buf, + size_t count, + int rank, + int tag, + MPIR_Comm * comm, + int context_offset, MPIR_Request ** request, + MPIR_Datatype * datatype) +{ + + int mpi_errno = MPI_SUCCESS; + uint64_t ucp_tag, tag_mask; + MPIR_Request *req; + MPIDI_UCX_ucp_request_t *ucp_request; +// MPID_THREAD_CS_ENTER(POBJ,MPIDI_THREAD_WORKER_MUTEX); + tag_mask = MPIDI_UCX_tag_mask(tag, rank); + ucp_tag = MPIDI_UCX_recv_tag(tag, rank, comm->recvcontext_id + context_offset); + + ucp_request = (MPIDI_UCX_ucp_request_t *) ucp_tag_recv_nb(MPIDI_UCX_global.worker, + buf, count, + datatype->dev.netmod.ucx.ucp_datatype, + ucp_tag, tag_mask, + &MPIDI_UCX_Handle_recv_callback); + + + MPIDI_CH4_UCX_REQUEST(ucp_request, tag_send_nb); + + + if (ucp_request->req == NULL) { + req = MPIR_Request_create(MPIR_REQUEST_KIND__RECV); + MPIR_Request_add_ref(req); + ucp_request->req = req; + MPIDI_UCX_REQ(req).a.ucp_request = ucp_request; + ucp_request_release(ucp_request); + } + else { + req = ucp_request->req; + + MPIDI_UCX_REQ(req).a.ucp_request = NULL; + ucp_request->req = NULL; + ucp_request_release(ucp_request); + } + (req)->kind = MPIR_REQUEST_KIND__RECV; + fn_exit: + *request = req; + return mpi_errno; + fn_fail: + goto fn_exit; +} + +static inline int do_irecv(void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + int mpi_errno = MPI_SUCCESS; + size_t data_sz; + int dt_contig; + MPIR_Request *req; + MPI_Aint dt_true_lb; + + MPIR_Datatype *dt_ptr; + + MPIDI_Datatype_get_info(count, datatype, dt_contig, data_sz, dt_ptr, dt_true_lb); + if (dt_contig) + mpi_errno = + ucx_irecv_continous(buf + dt_true_lb, data_sz, rank, tag, comm, context_offset, + request); + else + mpi_errno = + ucx_irecv_non_continous(buf, count, rank, tag, comm, context_offset, request, dt_ptr); + fn_exit: + return mpi_errno; + fn_fail: + goto fn_exit; + +} + +__ALWAYS_INLINE__ int MPIDI_NM_recv(void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, + int context_offset, + MPI_Status * status, MPIR_Request ** request) +{ + + return do_irecv(buf, count, datatype, rank, tag, comm, context_offset, request); +} + +__ALWAYS_INLINE__ int MPIDI_NM_recv_init(void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, + int context_offset, MPIR_Request ** request) +{ + return MPIDI_CH4U_recv_init(buf, count, datatype, rank, tag, comm, context_offset, request); +} + +__ALWAYS_INLINE__ int MPIDI_NM_imrecv(void *buf, + int count, + MPI_Datatype datatype, + MPIR_Request * message, MPIR_Request ** rreqp) +{ + ucp_tag_message_h message_handler; + int mpi_errno = MPI_SUCCESS; + size_t data_sz; + int dt_contig; + MPIR_Request *req; + MPI_Aint dt_true_lb; + MPIDI_UCX_ucp_request_t *ucp_request; + + MPIR_Datatype *dt_ptr; + MPIDI_Datatype_get_info(count, datatype, dt_contig, data_sz, dt_ptr, dt_true_lb); + if (message == NULL) { + mpi_errno = MPI_SUCCESS; + MPIDI_CH4U_request_complete(req); + *rreqp = req; + + goto fn_exit; + } + + message_handler = MPIDI_UCX_REQ(message).a.message_handler; + if (dt_contig) + ucp_request = (MPIDI_UCX_ucp_request_t *) ucp_tag_msg_recv_nb(MPIDI_UCX_global.worker, + buf + dt_true_lb, data_sz, + ucp_dt_make_contig(1), + message_handler, + &MPIDI_UCX_Handle_recv_callback); + else + ucp_request = (MPIDI_UCX_ucp_request_t *) ucp_tag_msg_recv_nb(MPIDI_UCX_global.worker, + buf, count, + dt_ptr->dev.netmod.ucx. + ucp_datatype, message_handler, + &MPIDI_UCX_Handle_recv_callback); + + + MPIDI_CH4_UCX_REQUEST(ucp_request, tag_send_nb); + + if (ucp_request->req == NULL) { + req = MPIR_Request_create(MPIR_REQUEST_KIND__RECV); + MPIR_Request_add_ref(req); + ucp_request->req = req; + ucp_request_release(ucp_request); + } + else { + req = ucp_request->req; + ucp_request->req = NULL; + ucp_request_release(ucp_request); + } + + + fn_exit: + *rreqp = req; + return mpi_errno; + fn_fail: + goto fn_exit; +} + +__ALWAYS_INLINE__ int MPIDI_NM_irecv(void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + + + + return do_irecv(buf, count, datatype, rank, tag, comm, context_offset, request); + +} + +static inline int MPIDI_NM_cancel_recv(MPIR_Request * rreq) +{ + + if (MPIDI_UCX_REQ(rreq).a.ucp_request) { + ucp_request_cancel(MPIDI_UCX_global.worker, MPIDI_UCX_REQ(rreq).a.ucp_request); + } + +} + +#endif /* NETMOD_UCX_RECV_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/ucx/ucx_request.h b/src/mpid/ch4/netmod/ucx/ucx_request.h new file mode 100644 index 0000000..35779da --- /dev/null +++ b/src/mpid/ch4/netmod/ucx/ucx_request.h @@ -0,0 +1,116 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2016 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Mellanox Technologies Ltd. + * Copyright (C) Mellanox Technologies Ltd. 2016. ALL RIGHTS RESERVED + */ +#ifndef NETMOD_UCX_REQUEST_H_INCLUDED +#define NETMOD_UCX_REQUEST_H_INCLUDED + +#include "ucx_impl.h" +#include "mpidch4.h" +#include <ucp/api/ucp.h> +#include "mpidch4r.h" + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_request_release +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline void MPIDI_NM_am_request_init(MPIR_Request * req) +{ + req->dev.ch4.ch4u.netmod_am.ucx.pack_buffer = NULL; +} + +static inline void MPIDI_NM_am_request_finalize(MPIR_Request * req) +{ + if ((req)->dev.ch4.ch4u.netmod_am.ucx.pack_buffer) { + MPL_free((req)->dev.ch4.ch4u.netmod_am.ucx.pack_buffer); + } + /* MPIDI_CH4U_request_release(req); */ +} + +static inline void MPIDI_UCX_Request_init_callback(void *request) +{ + + MPIDI_UCX_ucp_request_t *ucp_request = (MPIDI_UCX_ucp_request_t *) request; + ucp_request->req = NULL; + +} + +static inline void MPIDI_UCX_Handle_send_callback(void *request, ucs_status_t status) +{ + int c; + int mpi_errno; + MPIDI_UCX_ucp_request_t *ucp_request = (MPIDI_UCX_ucp_request_t *) request; + MPIR_Request *req = NULL; + if (unlikely(status == UCS_ERR_CANCELED)) { + req = ucp_request->req; + MPIDI_CH4U_request_complete(req); + MPIR_STATUS_SET_CANCEL_BIT(req->status, TRUE); + ucp_request->req = NULL; + goto fn_exit; + } + if (ucp_request->req) { + req = ucp_request->req; + MPIR_cc_decr(req->cc_ptr, &c); + MPIR_Assert(c >= 0); + + if (c == 0) { + MPIR_Request_free(req); + } + ucp_request->req = NULL; + } + else { + req = MPIR_Request_create(MPIR_REQUEST_KIND__SEND); + MPIR_cc_set(&req->cc, 0); + ucp_request->req = req; + } + fn_exit: + return; + fn_fail: + req->status.MPI_ERROR = mpi_errno; +} + +static inline void MPIDI_UCX_Handle_recv_callback(void *request, ucs_status_t status, + ucp_tag_recv_info_t * info) +{ + MPI_Aint count; + int mpi_errno; + MPIDI_UCX_ucp_request_t *ucp_request = (MPIDI_UCX_ucp_request_t *) request; + MPIR_Request *rreq = NULL; + if (unlikely(status == UCS_ERR_CANCELED)) { + rreq = ucp_request->req; + MPIDI_CH4U_request_complete(rreq); + MPIR_STATUS_SET_CANCEL_BIT(rreq->status, TRUE); + ucp_request->req = NULL; + goto fn_exit; + } + if (!ucp_request->req) { + rreq = MPIR_Request_create(MPIR_REQUEST_KIND__RECV); + MPIR_cc_set(&rreq->cc, 0); + rreq->status.MPI_SOURCE = MPIDI_UCX_get_source(info->sender_tag); + rreq->status.MPI_TAG = MPIDI_UCX_get_tag(info->sender_tag); + count = info->length; + MPIR_STATUS_SET_COUNT(rreq->status, count); + ucp_request->req = rreq; + } + else { + rreq = ucp_request->req; + rreq->status.MPI_ERROR = MPI_SUCCESS; + rreq->status.MPI_SOURCE = MPIDI_UCX_get_source(info->sender_tag); + rreq->status.MPI_TAG = MPIDI_UCX_get_tag(info->sender_tag); + count = info->length; + MPIR_STATUS_SET_COUNT(rreq->status, count); + MPIDI_CH4U_request_complete(rreq); + ucp_request->req = NULL; + } + + fn_exit: + return; + fn_fail: + rreq->status.MPI_ERROR = mpi_errno; +} + +#endif /* NETMOD_UCX_REQUEST_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/ucx/ucx_rma.h b/src/mpid/ch4/netmod/ucx/ucx_rma.h new file mode 100644 index 0000000..88aa9af --- /dev/null +++ b/src/mpid/ch4/netmod/ucx/ucx_rma.h @@ -0,0 +1,299 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2016 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Mellanox Technologies Ltd. + * Copyright (C) Mellanox Technologies Ltd. 2016. ALL RIGHTS RESERVED + */ +#ifndef NETMOD_UCX_RMA_H_INCLUDED +#define NETMOD_UCX_RMA_H_INCLUDED + +#include "ucx_impl.h" + +static inline int MPIDI_UCX_contig_put(const void *origin_addr, + size_t size, + int target_rank, + MPI_Aint target_disp, MPI_Aint true_lb, MPIR_Win * win) +{ + + MPIDI_UCX_win_info_t *win_info = &(MPIDI_UCX_WIN_INFO(win, target_rank)); + size_t offset; + uint64_t base; + int mpi_errno = MPI_SUCCESS; + ucs_status_t status; + MPIR_Comm *comm = win->comm_ptr; + ucp_ep_h ep = MPIDI_UCX_COMM_TO_EP(comm, target_rank); + + MPIDI_CH4U_EPOCH_START_CHECK(win, mpi_errno, goto fn_fail); + base = win_info->addr; + offset = target_disp * win_info->disp + true_lb; + + status = ucp_put_nbi(ep, origin_addr, size, base + offset, win_info->rkey); + if (status == UCS_INPROGRESS) + MPIDI_UCX_WIN(win).need_local_flush = 1; + else + MPIDI_UCX_CHK_STATUS(status, ucp_mem_map); + + fn_exit: + return mpi_errno; + fn_fail: + goto fn_exit; + + + +} + +static inline int MPIDI_UCX_contig_get(void *origin_addr, + size_t size, + int target_rank, + MPI_Aint target_disp, MPI_Aint true_lb, MPIR_Win * win) +{ + + MPIDI_UCX_win_info_t *win_info = &(MPIDI_UCX_WIN_INFO(win, target_rank)); + size_t offset; + uint64_t base; + int mpi_errno = MPI_SUCCESS; + ucs_status_t status; + MPIR_Comm *comm = win->comm_ptr; + ucp_ep_h ep = MPIDI_UCX_COMM_TO_EP(comm, target_rank); + + + MPIDI_CH4U_EPOCH_START_CHECK(win, mpi_errno, goto fn_fail); + base = win_info->addr; + offset = target_disp * win_info->disp + true_lb; + + status = ucp_get_nbi(ep, origin_addr, size, base + offset, win_info->rkey); + if (status == UCS_INPROGRESS) + MPIDI_UCX_WIN(win).need_local_flush = 1; + else + MPIDI_UCX_CHK_STATUS(status, ucp_mem_map); + + fn_exit: + return mpi_errno; + fn_fail: + goto fn_exit; + + + +} + +static inline int MPIDI_NM_put(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, MPI_Datatype target_datatype, MPIR_Win * win) +{ + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_UCX_PUT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_UCX_PUT); + int target_contig, origin_contig, mpi_errno = MPI_SUCCESS; + size_t target_bytes, origin_bytes; + MPI_Aint origin_true_lb, target_true_lb; + size_t offset; + if (win->create_flavor == MPI_WIN_FLAVOR_DYNAMIC || win->create_flavor == MPI_WIN_FLAVOR_SHARED) + return MPIDI_CH4U_put(origin_addr, origin_count, origin_datatype, + target_rank, target_disp, target_count, target_datatype, win); + + + + + + MPIDI_Datatype_check_contig_size_lb(target_datatype, target_count, + target_contig, target_bytes, target_true_lb); + MPIDI_Datatype_check_contig_size_lb(origin_datatype, origin_count, + origin_contig, origin_bytes, origin_true_lb); + + MPIR_ERR_CHKANDJUMP((origin_bytes != target_bytes), mpi_errno, MPI_ERR_SIZE, "**rmasize"); + + if (unlikely((origin_bytes == 0) || (target_rank == MPI_PROC_NULL))) + goto fn_exit; + if (!target_contig || !origin_contig || MPIDI_UCX_WIN_INFO(win, target_rank).rkey == NULL) + return MPIDI_CH4U_put(origin_addr, origin_count, origin_datatype, + target_rank, target_disp, target_count, target_datatype, win); + + MPIDI_CH4U_EPOCH_CHECK_SYNC(win, mpi_errno, goto fn_fail); + + if (target_rank == win->comm_ptr->rank) { + offset = win->disp_unit * target_disp; + return MPIR_Localcopy(origin_addr, + origin_count, + origin_datatype, + (char *) win->base + offset, target_count, target_datatype); + } + + + mpi_errno = MPIDI_UCX_contig_put(origin_addr + origin_true_lb, origin_bytes, + target_rank, target_disp, target_true_lb, win); + fn_exit: + return mpi_errno; + fn_fail: + goto fn_exit; + +} + +static inline int MPIDI_NM_get(void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, MPI_Datatype target_datatype, MPIR_Win * win) +{ + + + int origin_contig, target_contig, mpi_errno = MPI_SUCCESS; + size_t origin_bytes, target_bytes; + size_t offset; + + MPI_Aint origin_true_lb, target_true_lb; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_UCX_GET); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_UCX_GET); + + if (win->create_flavor == MPI_WIN_FLAVOR_DYNAMIC || win->create_flavor == MPI_WIN_FLAVOR_SHARED) + return MPIDI_CH4U_get(origin_addr, origin_count, origin_datatype, + target_rank, target_disp, target_count, target_datatype, win); + + MPIDI_Datatype_check_contig_size_lb(target_datatype, target_count, + target_contig, target_bytes, target_true_lb); + MPIDI_Datatype_check_contig_size_lb(origin_datatype, origin_count, + origin_contig, origin_bytes, origin_true_lb); + + if (unlikely((origin_bytes == 0) || (target_rank == MPI_PROC_NULL))) + goto fn_exit; + + + + if (!origin_contig || !target_contig || MPIDI_UCX_WIN_INFO(win, target_rank).rkey == NULL) + return MPIDI_CH4U_get(origin_addr, origin_count, origin_datatype, + target_rank, target_disp, target_count, target_datatype, win); + + MPIDI_CH4U_EPOCH_CHECK_SYNC(win, mpi_errno, goto fn_fail); + + if (target_rank == win->comm_ptr->rank) { + offset = target_disp * win->disp_unit; + return MPIR_Localcopy((char *) win->base + offset, + target_count, + target_datatype, origin_addr, origin_count, origin_datatype); + } + + + return MPIDI_UCX_contig_get(origin_addr + origin_true_lb, origin_bytes, + target_rank, target_disp, target_true_lb, win); + fn_exit: + return mpi_errno; + fn_fail: + goto fn_exit; + +} + +static inline int MPIDI_NM_rput(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPIR_Win * win, MPIR_Request ** request) +{ + return MPIDI_CH4U_rput(origin_addr, origin_count, origin_datatype, + target_rank, target_disp, target_count, target_datatype, win, request); +} + + +static inline int MPIDI_NM_compare_and_swap(const void *origin_addr, + const void *compare_addr, + void *result_addr, + MPI_Datatype datatype, + int target_rank, MPI_Aint target_disp, MPIR_Win * win) +{ + return MPIDI_CH4U_compare_and_swap(origin_addr, compare_addr, result_addr, + datatype, target_rank, target_disp, win); +} + +static inline int MPIDI_NM_raccumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPI_Op op, MPIR_Win * win, MPIR_Request ** request) +{ + return MPIDI_CH4U_raccumulate(origin_addr, origin_count, origin_datatype, + target_rank, target_disp, target_count, + target_datatype, op, win, request); +} + +static inline int MPIDI_NM_rget_accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + void *result_addr, + int result_count, + MPI_Datatype result_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPI_Op op, MPIR_Win * win, MPIR_Request ** request) +{ + return MPIDI_CH4U_rget_accumulate(origin_addr, origin_count, origin_datatype, + result_addr, result_count, result_datatype, + target_rank, target_disp, target_count, + target_datatype, op, win, request); +} + +static inline int MPIDI_NM_fetch_and_op(const void *origin_addr, + void *result_addr, + MPI_Datatype datatype, + int target_rank, + MPI_Aint target_disp, MPI_Op op, MPIR_Win * win) +{ + return MPIDI_CH4U_fetch_and_op(origin_addr, result_addr, datatype, + target_rank, target_disp, op, win); +} + + +static inline int MPIDI_NM_rget(void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPIR_Win * win, MPIR_Request ** request) +{ + return MPIDI_CH4U_rget(origin_addr, origin_count, origin_datatype, + target_rank, target_disp, target_count, target_datatype, win, request); +} + + +static inline int MPIDI_NM_get_accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + void *result_addr, + int result_count, + MPI_Datatype result_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, MPI_Op op, MPIR_Win * win) +{ + return MPIDI_CH4U_get_accumulate(origin_addr, origin_count, origin_datatype, + result_addr, result_count, result_datatype, + target_rank, target_disp, target_count, + target_datatype, op, win); +} + +static inline int MPIDI_NM_accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, MPI_Op op, MPIR_Win * win) +{ + return MPIDI_CH4U_accumulate(origin_addr, origin_count, origin_datatype, + target_rank, target_disp, target_count, target_datatype, op, win); +} + +#endif /* NETMOD_UCX_RMA_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/ucx/ucx_send.h b/src/mpid/ch4/netmod/ucx/ucx_send.h new file mode 100644 index 0000000..6b2f41c --- /dev/null +++ b/src/mpid/ch4/netmod/ucx/ucx_send.h @@ -0,0 +1,488 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2016 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Mellanox Technologies Ltd. + * Copyright (C) Mellanox Technologies Ltd. 2016. ALL RIGHTS RESERVED + */ +#ifndef NETMOD_UCX_SEND_H_INCLUDED +#define NETMOD_UCX_SEND_H_INCLUDED +#include <ucp/api/ucp.h> +#include "ucx_impl.h" +#include "ucx_types.h" + +#undef FUNCNAME +#define FUNCNAME ucx_send_continous +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__ALWAYS_INLINE__ int ucx_send_continous(const void *buf, + size_t data_sz, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request, int have_request) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Request *req; + MPIDI_UCX_ucp_request_t *ucp_request; + ucp_ep_h ep; + uint64_t ucx_tag; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SEND_CONTINOUS); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SEND_CONTINOUS); + + ep = MPIDI_UCX_COMM_TO_EP(comm, rank); + ucx_tag = MPIDI_UCX_init_tag(comm->context_id + context_offset, comm->rank, tag); + + ucp_request = + (MPIDI_UCX_ucp_request_t *) ucp_tag_send_nb(ep, buf, data_sz, ucp_dt_make_contig(1), + ucx_tag, &MPIDI_UCX_Handle_send_callback); + + MPIDI_CH4_UCX_REQUEST(ucp_request, tag_send_nb); + + if (ucp_request == NULL) { + req = MPIR_Request_create(MPIR_REQUEST_KIND__SEND); + MPIR_cc_set(&req->cc, 0); + MPIDI_UCX_REQ(req).a.ucp_request = NULL; + goto fn_exit; + } + + if (ucp_request->req) { + req = ucp_request->req; + ucp_request->req = NULL; + MPIDI_UCX_REQ(req).a.ucp_request = NULL; + ucp_request_release(ucp_request); + } + else { + req = MPIR_Request_create(MPIR_REQUEST_KIND__SEND); + MPIR_Request_add_ref(req); + ucp_request->req = req; + MPIDI_UCX_REQ(req).a.ucp_request = ucp_request; + ucp_request_release(ucp_request); + } + + + fn_exit: + *request = req; + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SEND_CONTINOUS); + return mpi_errno; + fn_fail: + goto fn_exit; + +} + +__ALWAYS_INLINE__ int ucx_sync_send_continous(const void *buf, + size_t data_sz, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request, int have_request) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Request *req; + MPIDI_UCX_ucp_request_t *ucp_request; + ucp_ep_h ep; + uint64_t ucx_tag; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SEND_CONTINOUS); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SEND_CONTINOUS); + + ep = MPIDI_UCX_COMM_TO_EP(comm, rank); + ucx_tag = MPIDI_UCX_init_tag(comm->context_id + context_offset, comm->rank, tag); + + ucp_request = + (MPIDI_UCX_ucp_request_t *) ucp_tag_send_sync_nb(ep, buf, data_sz, ucp_dt_make_contig(1), + ucx_tag, &MPIDI_UCX_Handle_send_callback); + + MPIDI_CH4_UCX_REQUEST(ucp_request, tag_send_nb); + if (ucp_request->req) { + req = ucp_request->req; + ucp_request->req = NULL; + MPIDI_UCX_REQ(req).a.ucp_request = NULL; + ucp_request_release(ucp_request); + } + else { + req = MPIR_Request_create(MPIR_REQUEST_KIND__SEND); + MPIR_Request_add_ref(req); + ucp_request->req = req; + MPIDI_UCX_REQ(req).a.ucp_request = ucp_request; + ucp_request_release(ucp_request); + } + + + fn_exit: + *request = req; + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SEND_CONTINOUS); + return mpi_errno; + fn_fail: + goto fn_exit; + +} + +__ALWAYS_INLINE__ int ucx_sync_send_non_continous(const void *buf, + size_t count, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request, int have_request, + MPIR_Datatype * datatype) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Request *req; + MPIDI_UCX_ucp_request_t *ucp_request; + ucp_ep_h ep; + uint64_t ucx_tag; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SEND_CONTINOUS); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SEND_CONTINOUS); + + ep = MPIDI_UCX_COMM_TO_EP(comm, rank); + ucx_tag = MPIDI_UCX_init_tag(comm->context_id + context_offset, comm->rank, tag); + + ucp_request = + (MPIDI_UCX_ucp_request_t *) ucp_tag_send_sync_nb(ep, buf, count, + datatype->dev.netmod.ucx.ucp_datatype, + ucx_tag, &MPIDI_UCX_Handle_send_callback); + + MPIDI_CH4_UCX_REQUEST(ucp_request, tag_send_nb); + + if (ucp_request->req) { + req = ucp_request->req; + ucp_request->req = NULL; + ucp_request_release(ucp_request); + MPIDI_UCX_REQ(req).a.ucp_request = NULL; + } + else { + req = MPIR_Request_create(MPIR_REQUEST_KIND__SEND); + MPIR_Request_add_ref(req); + ucp_request->req = req; + ucp_request_release(ucp_request); + MPIDI_UCX_REQ(req).a.ucp_request = ucp_request; + } + + + fn_exit: + *request = req; + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SEND_CONTINOUS); + return mpi_errno; + fn_fail: + goto fn_exit; + +} + + + +#undef FUNCNAME +#define FUNCNAME ucx_send_continous +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) + +__ALWAYS_INLINE__ int ucx_send_non_continous(const void *buf, + size_t count, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request, int have_request, + MPIR_Datatype * datatype) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Request *req; + MPIDI_UCX_ucp_request_t *ucp_request; + ucp_ep_h ep; + uint64_t ucx_tag; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SEND_CONTINOUS); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SEND_CONTINOUS); + + ep = MPIDI_UCX_COMM_TO_EP(comm, rank); + ucx_tag = MPIDI_UCX_init_tag(comm->context_id + context_offset, comm->rank, tag); + + ucp_request = + (MPIDI_UCX_ucp_request_t *) ucp_tag_send_nb(ep, buf, count, + datatype->dev.netmod.ucx.ucp_datatype, ucx_tag, + &MPIDI_UCX_Handle_send_callback); + + MPIDI_CH4_UCX_REQUEST(ucp_request, tag_send_nb); + + if (ucp_request == NULL) { + req = MPIR_Request_create(MPIR_REQUEST_KIND__SEND); + MPIDI_UCX_REQ(req).a.ucp_request = NULL; + MPIR_cc_set(&req->cc, 0); + goto fn_exit; + } + + if (ucp_request->req) { + req = ucp_request->req; + ucp_request->req = NULL; + MPIDI_UCX_REQ(req).a.ucp_request = NULL; + ucp_request_release(ucp_request); + } + else { + req = MPIR_Request_create(MPIR_REQUEST_KIND__SEND); + MPIR_Request_add_ref(req); + ucp_request->req = req; + MPIDI_UCX_REQ(req).a.ucp_request = ucp_request; + ucp_request_release(ucp_request); + } + + + fn_exit: + *request = req; + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SEND_CONTINOUS); + return mpi_errno; + fn_fail: + goto fn_exit; + +} + +#undef FUNCNAME +#define FUNCNAME ucx_send +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int ucx_send(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request, + int have_request) +{ + + int dt_contig, mpi_errno; + size_t data_sz; + MPI_Aint dt_true_lb; + MPIR_Datatype *dt_ptr; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SEND); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SEND); + + MPIDI_Datatype_get_info(count, datatype, dt_contig, data_sz, dt_ptr, dt_true_lb); + + if (dt_contig) + mpi_errno = + ucx_send_continous(buf + dt_true_lb, data_sz, rank, tag, comm, context_offset, request, + have_request); + else + mpi_errno = + ucx_send_non_continous(buf, count, rank, tag, comm, context_offset, request, + have_request, dt_ptr); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SEND); + return mpi_errno; + fn_fail: + goto fn_exit; + +} + +static inline int ucx_sync_send(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request, + int have_request) +{ + + int dt_contig, mpi_errno; + size_t data_sz; + MPI_Aint dt_true_lb; + MPIR_Datatype *dt_ptr; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SEND); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SEND); + + MPIDI_Datatype_get_info(count, datatype, dt_contig, data_sz, dt_ptr, dt_true_lb); + + if (dt_contig) + mpi_errno = + ucx_sync_send_continous(buf + dt_true_lb, data_sz, rank, tag, comm, context_offset, + request, have_request); + else + mpi_errno = + ucx_sync_send_non_continous(buf, count, rank, tag, comm, context_offset, request, + have_request, dt_ptr); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SEND); + return mpi_errno; + fn_fail: + goto fn_exit; + +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_netmode_send +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_send(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SEND); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SEND); + mpi_errno = ucx_send(buf, count, datatype, rank, tag, comm, context_offset, request, 0); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SEND); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_netmode_rsend +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_rsend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + + return ucx_send(buf, count, datatype, rank, tag, comm, context_offset, request, 0); +} + + +#undef FUNCNAME +#define FUNCNAME MPIDI_netmode_irsend +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_netmod_irsend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + return ucx_send(buf, count, datatype, rank, tag, comm, context_offset, request, 1); +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_netmode_ssend +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ssend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + return ucx_sync_send(buf, count, datatype, rank, tag, comm, context_offset, request, 0); +} + +static inline int MPIDI_NM_startall(int count, MPIR_Request * requests[]) +{ + return MPIDI_CH4U_startall(count, requests); +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_netmode_send_init +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_send_init(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + return MPIDI_CH4U_send_init(buf, count, datatype, rank, tag, comm, context_offset, request); +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_netmode_ssend_init +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_ssend_init(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + return MPIDI_CH4U_ssend_init(buf, count, datatype, rank, tag, comm, context_offset, request); +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_netmode_bsend_init +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_bsend_init(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + return MPIDI_CH4U_bsend_init(buf, count, datatype, rank, tag, comm, context_offset, request); +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_netmode_rsend_init +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_rsend_init(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + return MPIDI_CH4U_rsend_init(buf, count, datatype, rank, tag, comm, context_offset, request); +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_netmode_isend +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_isend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + + return ucx_send(buf, count, datatype, rank, tag, comm, context_offset, request, 1); + +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_netmode_issend +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_issend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + + return ucx_sync_send(buf, count, datatype, rank, tag, comm, context_offset, request, 1); + +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_netmode_cancel_send +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_cancel_send(MPIR_Request * sreq) +{ + if (MPIDI_UCX_REQ(sreq).a.ucp_request) { + ucp_request_cancel(MPIDI_UCX_global.worker, MPIDI_UCX_REQ(sreq).a.ucp_request); + ucp_request_release(MPIDI_UCX_REQ(sreq).a.ucp_request); + } +} + +#endif /* NETMOD_UCX_SEND_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/ucx/ucx_spawn.h b/src/mpid/ch4/netmod/ucx/ucx_spawn.h new file mode 100644 index 0000000..037b348 --- /dev/null +++ b/src/mpid/ch4/netmod/ucx/ucx_spawn.h @@ -0,0 +1,104 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2016 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Mellanox Technologies Ltd. + * Copyright (C) Mellanox Technologies Ltd. 2016. ALL RIGHTS RESERVED + */ +#ifndef NETMOD_UCX_SPAWN_H_INCLUDED +#define NETMOD_UCX_SPAWN_H_INCLUDED + +#include "ucx_impl.h" + +#undef FUNCNAME +#define FUNCNAME MPIDI_netmod_comm_open_port +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_open_port(MPIR_Info * info_ptr, char *port_name) +{ + int mpi_errno = MPI_SUCCESS; + fn_exit: + return mpi_errno; + fn_fail: + goto fn_exit; + +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_netmod_comm_close_port +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_close_port(const char *port_name) +{ + int mpi_errno = MPI_SUCCESS; + fn_exit: + return mpi_errno; + fn_fail: + goto fn_exit; + +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_comm_connect +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_comm_connect(const char *port_name, + MPIR_Info * info, + int root, MPIR_Comm * comm_ptr, MPIR_Comm ** newcomm) +{ + int mpi_errno = MPI_SUCCESS; + + fn_exit: + return mpi_errno; + fn_fail: + goto fn_exit; + + + +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_NM_comm_disconnect +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_comm_disconnect(MPIR_Comm * comm_ptr) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Errflag_t errflag = MPIR_ERR_NONE; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_COMM_DISCONNECT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_COMM_DISCONNECT); + + + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_COMM_DISCONNECT); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_netmod_comm_close_port +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_NM_comm_accept(const char *port_name, + MPIR_Info * info, + int root, MPIR_Comm * comm_ptr, MPIR_Comm ** newcomm) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_COMM_ACCEPT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_COMM_ACCEPT); + + + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_COMM_ACCEPT); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#endif /* NETMOD_UCX_SPAWN_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/ucx/ucx_types.h b/src/mpid/ch4/netmod/ucx/ucx_types.h new file mode 100644 index 0000000..045f64e --- /dev/null +++ b/src/mpid/ch4/netmod/ucx/ucx_types.h @@ -0,0 +1,66 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2016 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Mellanox Technologies Ltd. + * Copyright (C) Mellanox Technologies Ltd. 2016. ALL RIGHTS RESERVED + */ +#ifndef NETMOD_UCX_TYPES_H_INCLUDED +#define NETMOD_UCX_TYPES_H_INCLUDED +#include <ucp/api/ucp.h> +#include <ucp/api/ucp_def.h> +#include "mpiimpl.h" + +#define __SHORT_FILE__ \ + (strrchr(__FILE__,'/') \ + ? strrchr(__FILE__,'/')+1 \ + : __FILE__ \ +) + +#define UCP_PEER_NAME_MAX HOST_NAME_MAX + +#define MPIDI_MAP_NOT_FOUND ((void*)(-1UL)) + +/* Active Message Stuff */ +#define MPIDI_UCX_NUM_AM_BUFFERS (64) +#define MPIDI_UCX_MAX_AM_EAGER_SZ (16*1024) +#define MPIDI_UCX_AM_TAG (1 << 28) +#define MPIDI_UCX_MAX_AM_HANDLERS (64) + +typedef struct { + int avtid; + ucp_context_h context; + ucp_worker_h worker; + char addrname[UCP_PEER_NAME_MAX]; + char *pmi_addr_table; + size_t addrname_len; + ucp_address_t *if_address; + char kvsname[MPIDI_UCX_KVSAPPSTRLEN]; + char pname[MPI_MAX_PROCESSOR_NAME]; + int max_addr_len; + MPIDI_NM_am_target_handler_fn am_handlers[MPIDI_UCX_MAX_AM_HANDLERS]; + MPIDI_NM_am_origin_handler_fn send_cmpl_handlers[MPIDI_UCX_MAX_AM_HANDLERS]; +} MPIDI_UCX_global_t; + +#define MPIDI_UCX_GPID(gpid) ((gpid)->dev.netmod.ucx) +#define MPIDI_UCX_AV(av) ((av)->netmod.ucx) + +extern MPIDI_UCX_global_t MPIDI_UCX_global; + +/* UCX TAG Layout */ + +/* 01234567 01234567 01234567 01234567 01234567 01234567 01234567 01234567 + * context_id (16) |source rank (16) | Message Tag (32)+ERROR BITS + */ + +#define MPIDI_UCX_CONTEXT_TAG_BITS 16 +#define MPIDI_UCX_CONTEXT_RANK_BITS 16 +#define UCX_TAG_BITS 32 + +#define MPIDI_UCX_TAG_MASK (0x00000000FFFFFFFFULL) +#define MPIDI_UCX_SOURCE_MASK (0x0000FFFF00000000ULL) +#define MPIDI_UCX_TAG_SHIFT (32) +#define MPIDI_UCX_SOURCE_SHIFT (16) + +#endif /* NETMOD_UCX_TYPES_H_INCLUDED */ diff --git a/src/mpid/ch4/netmod/ucx/ucx_win.h b/src/mpid/ch4/netmod/ucx/ucx_win.h new file mode 100644 index 0000000..8ae3fad --- /dev/null +++ b/src/mpid/ch4/netmod/ucx/ucx_win.h @@ -0,0 +1,503 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2016 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Mellanox Technologies Ltd. + * Copyright (C) Mellanox Technologies Ltd. 2016. ALL RIGHTS RESERVED + */ +#ifndef NETMOD_UCX_WIN_H_INCLUDED +#define NETMOD_UCX_WIN_H_INCLUDED + +#include "ucx_impl.h" +struct _UCX_share { + int disp; + MPI_Aint addr; +}; + +char ucx_dummy_buffer[4096]; + +static inline int MPIDI_UCX_Win_allgather(MPIR_Win * win, size_t length, + uint32_t disp_unit, void **base_ptr) +{ + + MPIR_Errflag_t err = MPIR_ERR_NONE; + int mpi_errno = MPI_SUCCESS; + ucs_status_t status; + ucp_mem_h mem_h; + int cntr = 0; + size_t rkey_size; + int *rkey_sizes, *recv_disps, i; + char *rkey_buffer, *rkey_recv_buff = NULL; + struct _UCX_share *share_data; + size_t size; + void *base; + if (length == 0) + size = 1024; + else + size = length; + MPIR_Comm *comm_ptr = win->comm_ptr; + + ucp_context_h ucp_context = MPIDI_UCX_global.context; + + MPIDI_UCX_WIN(win).info_table = MPL_malloc(sizeof(MPIDI_UCX_win_info_t) * comm_ptr->local_size); + if (length == 0) + base = &ucx_dummy_buffer; + else + base = *base_ptr; + + status = ucp_mem_map(MPIDI_UCX_global.context, &base, size, 0, &mem_h); + MPIDI_UCX_CHK_STATUS(status, ucp_mem_map); + if (length > 0) + *base_ptr = base; + + MPIDI_UCX_WIN(win).mem_h = mem_h; + + /* pack the key */ + status = ucp_rkey_pack(ucp_context, mem_h, (void **) &rkey_buffer, &rkey_size); + + MPIDI_UCX_CHK_STATUS(status, ucp_mem_map); + + rkey_sizes = (int *) MPL_malloc(sizeof(int) * comm_ptr->local_size); + rkey_sizes[comm_ptr->rank] = (int) rkey_size; + mpi_errno = MPIR_Allgather_impl(MPI_IN_PLACE, 1, MPI_INT, + rkey_sizes, 1, MPI_INT, comm_ptr, &err); + + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + recv_disps = (int *) MPL_malloc(sizeof(int) * comm_ptr->local_size); + + + for (i = 0; i < comm_ptr->local_size; i++) { + recv_disps[i] = cntr; + cntr += rkey_sizes[i]; + } + + rkey_recv_buff = MPL_malloc(cntr); + + /* allgather */ + mpi_errno = MPIR_Allgatherv_impl(rkey_buffer, rkey_size, MPI_BYTE, + rkey_recv_buff, rkey_sizes, recv_disps, MPI_BYTE, + comm_ptr, &err); + + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + +/* If we use the shared memory support in UCX, we have to distinguish between local + and remote windows (at least now). If win_create is used, the key cannot be unpackt - + then we need our fallback-solution */ + + for (i = 0; i < comm_ptr->local_size; i++) { + status = ucp_ep_rkey_unpack(MPIDI_UCX_COMM_TO_EP(comm_ptr, i), + &rkey_recv_buff[recv_disps[i]], + &(MPIDI_UCX_WIN_INFO(win, i).rkey)); + if (status == UCS_ERR_UNREACHABLE) { + MPIDI_UCX_WIN_INFO(win, i).rkey = NULL; + } + else + MPIDI_UCX_CHK_STATUS(status, ucp_mem_map); + } + share_data = MPL_malloc(comm_ptr->local_size * sizeof(struct _UCX_share)); + + share_data[comm_ptr->rank].disp = disp_unit; + share_data[comm_ptr->rank].addr = (MPI_Aint) base; + + mpi_errno = + MPIR_Allgather(MPI_IN_PLACE, sizeof(struct _UCX_share), MPI_BYTE, share_data, + sizeof(struct _UCX_share), MPI_BYTE, comm_ptr, &err); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + for (i = 0; i < comm_ptr->local_size; i++) { + MPIDI_UCX_WIN_INFO(win, i).disp = share_data[i].disp; + MPIDI_UCX_WIN_INFO(win, i).addr = share_data[i].addr; + } + MPIDI_UCX_WIN(win).need_local_flush = 0; + fn_exit: + /* buffer release */ + if (rkey_buffer) + ucp_rkey_buffer_release(rkey_buffer); + /* free temps */ + MPL_free(share_data); + MPL_free(rkey_sizes); + MPL_free(recv_disps); + MPL_free(rkey_recv_buff); + return mpi_errno; + fn_fail: + goto fn_exit; + +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_UCX_Win_init +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_UCX_Win_init(MPI_Aint length, + int disp_unit, + MPIR_Win ** win_ptr, + MPIR_Info * info, + MPIR_Comm * comm_ptr, int create_flavor, int model) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Win *win; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_UCX_WIN_INIT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_UCX_WIN_INIT); + + mpi_errno = MPIDI_CH4R_win_init(length, disp_unit, &win, info, comm_ptr, create_flavor, model); + MPIR_ERR_CHKANDSTMT(mpi_errno != MPI_SUCCESS, + mpi_errno, MPI_ERR_NO_MEM, goto fn_fail, "**nomem"); + *win_ptr = win; + + memset(&MPIDI_UCX_WIN(win), 0, sizeof(MPIDI_UCX_win_t)); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_UCX_PROGRESS_WIN_INIT); + return mpi_errno; + fn_fail: + goto fn_exit; + +} + +static inline int MPIDI_NM_win_set_info(MPIR_Win * win, MPIR_Info * info) +{ + return MPIDI_CH4R_win_set_info(win, info); +} + + +static inline int MPIDI_NM_win_start(MPIR_Group * group, int assert, MPIR_Win * win) +{ + return MPIDI_CH4R_win_start(group, assert, win); +} + + +static inline int MPIDI_NM_win_complete(MPIR_Win * win) +{ + + ucs_status_t ucp_status; + ucp_status = ucp_worker_flush(MPIDI_UCX_global.worker); + return MPIDI_CH4R_win_complete(win); +} + +static inline int MPIDI_NM_win_post(MPIR_Group * group, int assert, MPIR_Win * win) +{ + + return MPIDI_CH4R_win_post(group, assert, win); +} + + +static inline int MPIDI_NM_win_wait(MPIR_Win * win) +{ + return MPIDI_CH4R_win_wait(win); +} + + +static inline int MPIDI_NM_win_test(MPIR_Win * win, int *flag) +{ + return MPIDI_CH4R_win_test(win, flag); +} + +static inline int MPIDI_NM_win_lock(int lock_type, int rank, int assert, MPIR_Win * win) +{ + return MPIDI_CH4R_win_lock(lock_type, rank, assert, win); +} + + +static inline int MPIDI_NM_win_unlock(int rank, MPIR_Win * win) +{ + + int mpi_errno = MPI_SUCCESS; + ucs_status_t ucp_status; + ucp_ep_h ep = MPIDI_UCX_COMM_TO_EP(win->comm_ptr, rank); + /* make sure all operations are completed */ + ucp_status = ucp_ep_flush(ep); + MPIDI_UCX_CHK_STATUS(ucp_status, ucp_worker_fence); + mpi_errno = MPIDI_CH4R_win_unlock(rank, win); + fn_exit: + return mpi_errno; + fn_fail: + goto fn_exit; +} + +static inline int MPIDI_NM_win_get_info(MPIR_Win * win, MPIR_Info ** info_p_p) +{ + return MPIDI_CH4R_win_get_info(win, info_p_p); +} + + +static inline int MPIDI_NM_win_free(MPIR_Win ** win_ptr) +{ + + int mpi_errno = MPI_SUCCESS; + MPIR_Errflag_t errflag = MPIR_ERR_NONE; + MPIR_Win *win = *win_ptr; + MPIDI_CH4U_EPOCH_FREE_CHECK(win, mpi_errno, return mpi_errno); + mpi_errno = MPIR_Barrier_impl(win->comm_ptr, &errflag); + if (mpi_errno != MPI_SUCCESS) + goto fn_fail; + if (win->create_flavor != MPI_WIN_FLAVOR_SHARED && win->create_flavor != MPI_WIN_FLAVOR_DYNAMIC) { + ucp_mem_unmap(MPIDI_UCX_global.context, MPIDI_UCX_WIN(win).mem_h); + MPL_free(MPIDI_UCX_WIN(win).info_table); + } + if (win->create_flavor == MPI_WIN_FLAVOR_ALLOCATE) + win->base = NULL; + MPIDI_CH4R_win_finalize(win_ptr); + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_WIN_FREE); + return mpi_errno; + fn_fail: + goto fn_exit; + +} + +static inline int MPIDI_NM_win_fence(int assert, MPIR_Win * win) +{ + int mpi_errno; + ucs_status_t ucp_status; + /*keep this for now to fence all none-natice operations */ +/* make sure all local and remote operations are completed */ + ucp_status = ucp_worker_flush(MPIDI_UCX_global.worker); + + + mpi_errno = MPIDI_CH4R_win_fence(assert, win); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + MPIDI_UCX_CHK_STATUS(ucp_status, ucp_worker_fence); + fn_exit: + return mpi_errno; + fn_fail: + goto fn_exit; +} + +static inline int MPIDI_NM_win_create(void *base, + MPI_Aint length, + int disp_unit, + MPIR_Info * info, MPIR_Comm * comm_ptr, MPIR_Win ** win_ptr) +{ + + int mpi_errno = MPI_SUCCESS; + MPIR_Errflag_t errflag = MPIR_ERR_NONE; + MPIR_Win *win; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_UCX_WIN_CREATE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_UCX_WIN_CREATE); + + MPIDI_UCX_Win_init(length, disp_unit, win_ptr, info, + comm_ptr, MPI_WIN_FLAVOR_CREATE, MPI_WIN_UNIFIED); + + win = *win_ptr; + + mpi_errno = MPIDI_UCX_Win_allgather(win, length, disp_unit, &base); + if (mpi_errno != MPI_SUCCESS) + goto fn_fail; + + win->base = base; + + + + mpi_errno = MPIR_Barrier_impl(comm_ptr, &errflag); + + if (mpi_errno != MPI_SUCCESS) + goto fn_fail; + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_UCX_WIN_CREATE); + return mpi_errno; + fn_fail: + goto fn_exit; + + +} + +static inline int MPIDI_NM_win_attach(MPIR_Win * win, void *base, MPI_Aint size) +{ + return MPIDI_CH4R_win_attach(win, base, size); +} + +static inline int MPIDI_NM_win_allocate_shared(MPI_Aint size, + int disp_unit, + MPIR_Info * info_ptr, + MPIR_Comm * comm_ptr, + void **base_ptr, MPIR_Win ** win_ptr) +{ + return MPIDI_CH4R_win_allocate_shared(size, disp_unit, info_ptr, comm_ptr, base_ptr, win_ptr); +} + +static inline int MPIDI_NM_win_detach(MPIR_Win * win, const void *base) +{ + return MPIDI_CH4R_win_detach(win, base); +} + +static inline int MPIDI_NM_win_shared_query(MPIR_Win * win, + int rank, + MPI_Aint * size, int *disp_unit, void *baseptr) +{ + return MPIDI_CH4R_win_shared_query(win, rank, size, disp_unit, baseptr); +} + +static inline int MPIDI_NM_win_allocate(MPI_Aint length, + int disp_unit, + MPIR_Info * info, + MPIR_Comm * comm_ptr, void *baseptr, MPIR_Win ** win_ptr) +{ + + int mpi_errno = MPI_SUCCESS; + MPIR_Errflag_t errflag = MPIR_ERR_NONE; + MPIR_Win *win; + void *base = NULL; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_UCX_WIN_ALLOCATE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_UCX_WIN_WIN_ALLOCATE); + + MPIDI_UCX_Win_init(length, disp_unit, win_ptr, info, + comm_ptr, MPI_WIN_FLAVOR_ALLOCATE, MPI_WIN_UNIFIED); + win = *win_ptr; + mpi_errno = MPIDI_UCX_Win_allgather(win, length, disp_unit, &base); + if (mpi_errno != MPI_SUCCESS) + goto fn_fail; + win->base = base; + + + *(void **) baseptr = (void *) base; + + + mpi_errno = MPIR_Barrier_impl(comm_ptr, &errflag); + + if (mpi_errno != MPI_SUCCESS) + goto fn_fail; + + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_UCX_WIN_ALLOCATE); + return mpi_errno; + fn_fail: + goto fn_exit; + + +} + +static inline int MPIDI_NM_win_flush(int rank, MPIR_Win * win) +{ + + int mpi_errno; + ucs_status_t ucp_status; + + ucp_ep_h ep = MPIDI_UCX_COMM_TO_EP(win->comm_ptr, rank); + + mpi_errno = MPIDI_CH4R_win_flush(rank, win); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); +/* only flush the endpoint */ + ucp_status = ucp_ep_flush(ep); + + MPIDI_UCX_CHK_STATUS(ucp_status, ucp_worker_fence); + + fn_exit: + return mpi_errno; + fn_fail: + goto fn_exit; + +} + +static inline int MPIDI_NM_win_flush_local_all(MPIR_Win * win) +{ + int mpi_errno = MPI_SUCCESS; + ucs_status_t ucp_status; + mpi_errno = MPIDI_CH4R_win_flush_local_all(win); + + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + /* currently, UCP does not support local flush, so we have to call + * a global flush. This is not good for performance - but OK for now */ + if (MPIDI_UCX_WIN(win).need_local_flush == 1) { + ucp_status = ucp_worker_flush(MPIDI_UCX_global.worker); + MPIDI_UCX_CHK_STATUS(ucp_status, ucp_worker_fence); + MPIDI_UCX_WIN(win).need_local_flush = 0; + } + + + fn_exit: + return mpi_errno; + fn_fail: + goto fn_exit; +} + +static inline int MPIDI_NM_win_unlock_all(MPIR_Win * win) +{ + int mpi_errno = MPI_SUCCESS; + ucs_status_t ucp_status; + + /*first we have to make sure that all operations are completed */ + ucp_status = ucp_worker_flush(MPIDI_UCX_global.worker); + MPIDI_UCX_CHK_STATUS(ucp_status, ucp_worker_fence); + mpi_errno = MPIDI_CH4R_win_unlock_all(win); + fn_exit: + return mpi_errno; + fn_fail: + goto fn_exit; +} + +static inline int MPIDI_NM_win_create_dynamic(MPIR_Info * info, MPIR_Comm * comm, MPIR_Win ** win) +{ + return MPIDI_CH4R_win_create_dynamic(info, comm, win); +} + +static inline int MPIDI_NM_win_flush_local(int rank, MPIR_Win * win) +{ + int mpi_errno = MPI_SUCCESS; + ucs_status_t ucp_status; + mpi_errno = MPIDI_CH4R_win_flush_local(rank, win); + + ucp_ep_h ep = MPIDI_UCX_COMM_TO_EP(win->comm_ptr, rank); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + /* currently, UCP does not support local flush, so we have to call + * a global flush. This is not good for performance - but OK for now */ + + if (MPIDI_UCX_WIN(win).need_local_flush == 1) { + ucp_status = ucp_ep_flush(ep); + MPIDI_UCX_CHK_STATUS(ucp_status, ucp_worker_fence); + MPIDI_UCX_WIN(win).need_local_flush = 0; + } + + fn_exit: + return mpi_errno; + fn_fail: + goto fn_exit; + +} + +static inline int MPIDI_NM_win_sync(MPIR_Win * win) +{ + return MPIDI_CH4R_win_sync(win); +} + +static inline int MPIDI_NM_win_flush_all(MPIR_Win * win) +{ + +/*maybe we just flush all eps here? More efficient for smaller communicators...*/ + int mpi_errno = MPI_SUCCESS; + ucs_status_t ucp_status; + mpi_errno = MPIDI_CH4R_win_flush_all(win); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + ucp_status = ucp_worker_flush(MPIDI_UCX_global.worker); + + MPIDI_UCX_CHK_STATUS(ucp_status, ucp_worker_fence); + + fn_exit: + return mpi_errno; + fn_fail: + goto fn_exit; + +} + +static inline int MPIDI_NM_win_lock_all(int assert, MPIR_Win * win) +{ + return MPIDI_CH4R_win_lock_all(assert, win); +} + + +#endif /* NETMOD_UCX_WIN_H_INCLUDED */ diff --git a/src/mpid/ch4/shm/Makefile.mk b/src/mpid/ch4/shm/Makefile.mk new file mode 100644 index 0000000..6bf059b --- /dev/null +++ b/src/mpid/ch4/shm/Makefile.mk @@ -0,0 +1,24 @@ +## -*- Mode: Makefile; -*- +## vim: set ft=automake : +## +## (C) 2016 by Argonne National Laboratory. +## (C) 2014 by Mellanox Technologies, Inc. +## See COPYRIGHT in top-level directory. +## +## Portions of this code were written by Intel Corporation. +## Copyright (C) 2011-2016 Intel Corporation. Intel provides this material +## to Argonne National Laboratory subject to Software Grant and Corporate +## Contributor License Agreement dated February 8, 2012. +## + +if BUILD_CH4_SHM + +AM_CPPFLAGS += -I$(top_srcdir)/src/mpid/ch4/shm/include + +noinst_HEADERS += src/mpid/ch4/shm/include/shm.h +noinst_HEADERS += src/mpid/ch4/shm/include/shm_impl.h + +include $(top_srcdir)/src/mpid/ch4/shm/stubshm/Makefile.mk +include $(top_srcdir)/src/mpid/ch4/shm/posix/Makefile.mk + +endif diff --git a/src/mpid/ch4/shm/include/shm.h b/src/mpid/ch4/shm/include/shm.h new file mode 100644 index 0000000..1fa4138 --- /dev/null +++ b/src/mpid/ch4/shm/include/shm.h @@ -0,0 +1,1192 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +/* ch4 shm functions */ +#ifndef SHM_PROTOTYPES_H_INCLUDED +#define SHM_PROTOTYPES_H_INCLUDED + +#include <mpidimpl.h> + +#define MPIDI_MAX_SHM_STRING_LEN 64 + +typedef int (*MPIDI_SHM_am_completion_handler_fn) (MPIR_Request * req); +typedef int (*MPIDI_SHM_am_origin_handler_fn) (MPIR_Request * req); + +/* Callback function setup by handler register function */ +/* for short cases, output arguments are NULL */ +typedef int (*MPIDI_SHM_am_target_handler_fn) + (void *am_hdr, size_t am_hdr_sz, void **data, /* CH4 manages this buffer - shm only fills with data */ + MPI_Datatype * datatype, MPI_Count * count, int *noncontig, /* if TRUE: data/data_sz are actually iovec/count */ + MPIDI_SHM_am_completion_handler_fn * cmpl_handler_fn, /* completion handler */ + MPIR_Request ** req); /* if allocated, need pointer to completion function */ + +typedef int (*MPIDI_SHM_init_t) (int rank, int size); +typedef int (*MPIDI_SHM_finalize_t) (void); +typedef int (*MPIDI_SHM_progress_t) (int blocking); +typedef int (*MPIDI_SHM_reg_hdr_handler_t) (int handler_id, + MPIDI_SHM_am_origin_handler_fn origin_handler_fn, + MPIDI_SHM_am_target_handler_fn target_handler_fn); +typedef int (*MPIDI_SHM_comm_connect_t) (const char *port_name, MPIR_Info * info, int root, + MPIR_Comm * comm, MPIR_Comm ** newcomm_ptr); +typedef int (*MPIDI_SHM_comm_disconnect_t) (MPIR_Comm * comm_ptr); +typedef int (*MPIDI_SHM_open_port_t) (MPIR_Info * info_ptr, char *port_name); +typedef int (*MPIDI_SHM_close_port_t) (const char *port_name); +typedef int (*MPIDI_SHM_comm_accept_t) (const char *port_name, MPIR_Info * info, int root, + MPIR_Comm * comm, MPIR_Comm ** newcomm_ptr); +typedef int (*MPIDI_SHM_send_am_hdr_t) (int rank, MPIR_Comm * comm, int handler_id, + const void *am_hdr, size_t am_hdr_sz, MPIR_Request * sreq, + void *shm_context); +typedef int (*MPIDI_SHM_inject_am_hdr_t) (int rank, MPIR_Comm * comm, int handler_id, + const void *am_hdr, size_t am_hdr_sz, void *shm_context); +typedef int (*MPIDI_SHM_send_am_t) (int rank, MPIR_Comm * comm, int handler_id, const void *am_hdr, + size_t am_hdr_sz, const void *data, MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq, void *shm_context); +typedef int (*MPIDI_SHM_inject_am_t) (int rank, MPIR_Comm * comm, int handler_id, + const void *am_hdr, size_t am_hdr_sz, const void *data, + MPI_Count count, MPI_Datatype datatype, void *shm_context); +typedef int (*MPIDI_SHM_send_amv_t) (int rank, MPIR_Comm * comm, int handler_id, + struct iovec * am_hdrs, size_t iov_len, const void *data, + MPI_Count count, MPI_Datatype datatype, MPIR_Request * sreq, + void *shm_context); +typedef int (*MPIDI_SHM_inject_amv_t) (int rank, MPIR_Comm * comm, int handler_id, + struct iovec * am_hdrs, size_t iov_len, const void *data, + MPI_Count count, MPI_Datatype datatype, void *shm_context); +typedef int (*MPIDI_SHM_send_am_hdr_reply_t) (MPIR_Context_id_t context_id, int src_rank, + int handler_id, const void *am_hdr, size_t am_hdr_sz, + MPIR_Request * sreq); +typedef int (*MPIDI_SHM_inject_am_hdr_reply_t) (MPIR_Context_id_t context_id, int src_rank, + int handler_id, const void *am_hdr, + size_t am_hdr_sz); +typedef int (*MPIDI_SHM_send_am_reply_t) (MPIR_Context_id_t context_id, int src_rank, + int handler_id, const void *am_hdr, size_t am_hdr_sz, + const void *data, MPI_Count count, MPI_Datatype datatype, + MPIR_Request * sreq); +typedef int (*MPIDI_SHM_inject_am_reply_t) (MPIR_Context_id_t context_id, int src_rank, + int handler_id, const void *am_hdr, size_t am_hdr_sz, + const void *data, MPI_Count count, + MPI_Datatype datatype); +typedef int (*MPIDI_SHM_send_amv_reply_t) (MPIR_Context_id_t context_id, int src_rank, + int handler_id, struct iovec * am_hdr, size_t iov_len, + const void *data, MPI_Count count, MPI_Datatype datatype, + MPIR_Request * sreq); +typedef int (*MPIDI_SHM_inject_amv_reply_t) (MPIR_Context_id_t context_id, int src_rank, + int handler_id, struct iovec * am_hdrs, size_t iov_len, + const void *data, MPI_Count count, + MPI_Datatype datatype); +typedef size_t(*MPIDI_SHM_am_hdr_max_sz_t) (void); +typedef size_t(*MPIDI_SHM_am_inject_max_sz_t) (void); +typedef int (*MPIDI_SHM_am_recv_t) (MPIR_Request * req); +typedef int (*MPIDI_SHM_comm_get_lpid_t) (MPIR_Comm * comm_ptr, int idx, int *lpid_ptr, + MPL_bool is_remote); +typedef int (*MPIDI_SHM_gpid_get_t) (MPIR_Comm * comm_ptr, int rank, MPIR_Gpid * gpid); +typedef int (*MPIDI_SHM_get_node_id_t) (MPIR_Comm * comm, int rank, MPID_Node_id_t * id_p); +typedef int (*MPIDI_SHM_get_max_node_id_t) (MPIR_Comm * comm, MPID_Node_id_t * max_id_p); +typedef int (*MPIDI_SHM_getallincomm_t) (MPIR_Comm * comm_ptr, int local_size, + MPIR_Gpid local_gpid[], int *singleAVT); +typedef int (*MPIDI_SHM_gpid_tolpidarray_t) (int size, MPIR_Gpid gpid[], int lpid[]); +typedef int (*MPIDI_SHM_create_intercomm_from_lpids_t) (MPIR_Comm * newcomm_ptr, int size, + const int lpids[]); +typedef int (*MPIDI_SHM_comm_create_t) (MPIR_Comm * comm); +typedef int (*MPIDI_SHM_comm_destroy_t) (MPIR_Comm * comm); +typedef void (*MPIDI_SHM_am_request_init_t) (MPIR_Request * req); +typedef void (*MPIDI_SHM_am_request_finalize_t) (MPIR_Request * req); +typedef int (*MPIDI_SHM_send_t) (const void *buf, int count, MPI_Datatype datatype, int rank, + int tag, MPIR_Comm * comm, int context_offset, + MPIR_Request ** request); +typedef int (*MPIDI_SHM_ssend_t) (const void *buf, int count, MPI_Datatype datatype, int rank, + int tag, MPIR_Comm * comm, int context_offset, + MPIR_Request ** request); +typedef int (*MPIDI_SHM_startall_t) (int count, MPIR_Request * requests[]); +typedef int (*MPIDI_SHM_send_init_t) (const void *buf, int count, MPI_Datatype datatype, int rank, + int tag, MPIR_Comm * comm, int context_offset, + MPIR_Request ** request); +typedef int (*MPIDI_SHM_ssend_init_t) (const void *buf, int count, MPI_Datatype datatype, int rank, + int tag, MPIR_Comm * comm, int context_offset, + MPIR_Request ** request); +typedef int (*MPIDI_SHM_rsend_init_t) (const void *buf, int count, MPI_Datatype datatype, int rank, + int tag, MPIR_Comm * comm, int context_offset, + MPIR_Request ** request); +typedef int (*MPIDI_SHM_bsend_init_t) (const void *buf, int count, MPI_Datatype datatype, int rank, + int tag, MPIR_Comm * comm, int context_offset, + MPIR_Request ** request); +typedef int (*MPIDI_SHM_isend_t) (const void *buf, int count, MPI_Datatype datatype, int rank, + int tag, MPIR_Comm * comm, int context_offset, + MPIR_Request ** request); +typedef int (*MPIDI_SHM_issend_t) (const void *buf, int count, MPI_Datatype datatype, int rank, + int tag, MPIR_Comm * comm, int context_offset, + MPIR_Request ** request); +typedef int (*MPIDI_SHM_cancel_send_t) (MPIR_Request * sreq); +typedef int (*MPIDI_SHM_recv_init_t) (void *buf, int count, MPI_Datatype datatype, int rank, + int tag, MPIR_Comm * comm, int context_offset, + MPIR_Request ** request); +typedef int (*MPIDI_SHM_recv_t) (void *buf, int count, MPI_Datatype datatype, int rank, int tag, + MPIR_Comm * comm, int context_offset, MPI_Status * status, + MPIR_Request ** request); +typedef int (*MPIDI_SHM_irecv_t) (void *buf, int count, MPI_Datatype datatype, int rank, int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request); +typedef int (*MPIDI_SHM_imrecv_t) (void *buf, int count, MPI_Datatype datatype, + MPIR_Request * message, MPIR_Request ** rreqp); +typedef int (*MPIDI_SHM_cancel_recv_t) (MPIR_Request * rreq); +typedef void *(*MPIDI_SHM_alloc_mem_t) (size_t size, MPIR_Info * info_ptr); +typedef int (*MPIDI_SHM_free_mem_t) (void *ptr); +typedef int (*MPIDI_SHM_improbe_t) (int source, int tag, MPIR_Comm * comm, int context_offset, + int *flag, MPIR_Request ** message, MPI_Status * status); +typedef int (*MPIDI_SHM_iprobe_t) (int source, int tag, MPIR_Comm * comm, int context_offset, + int *flag, MPI_Status * status); +typedef int (*MPIDI_SHM_win_set_info_t) (MPIR_Win * win, MPIR_Info * info); +typedef int (*MPIDI_SHM_win_shared_query_t) (MPIR_Win * win, int rank, MPI_Aint * size, + int *disp_unit, void *baseptr); +typedef int (*MPIDI_SHM_put_t) (const void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, int target_rank, MPI_Aint target_disp, + int target_count, MPI_Datatype target_datatype, MPIR_Win * win); +typedef int (*MPIDI_SHM_win_start_t) (MPIR_Group * group, int assert, MPIR_Win * win); +typedef int (*MPIDI_SHM_win_complete_t) (MPIR_Win * win); +typedef int (*MPIDI_SHM_win_post_t) (MPIR_Group * group, int assert, MPIR_Win * win); +typedef int (*MPIDI_SHM_win_wait_t) (MPIR_Win * win); +typedef int (*MPIDI_SHM_win_test_t) (MPIR_Win * win, int *flag); +typedef int (*MPIDI_SHM_win_lock_t) (int lock_type, int rank, int assert, MPIR_Win * win); +typedef int (*MPIDI_SHM_win_unlock_t) (int rank, MPIR_Win * win); +typedef int (*MPIDI_SHM_win_get_info_t) (MPIR_Win * win, MPIR_Info ** info_p_p); +typedef int (*MPIDI_SHM_get_t) (void *origin_addr, int origin_count, MPI_Datatype origin_datatype, + int target_rank, MPI_Aint target_disp, int target_count, + MPI_Datatype target_datatype, MPIR_Win * win); +typedef int (*MPIDI_SHM_win_free_t) (MPIR_Win ** win_ptr); +typedef int (*MPIDI_SHM_win_fence_t) (int assert, MPIR_Win * win); +typedef int (*MPIDI_SHM_win_create_t) (void *base, MPI_Aint length, int disp_unit, MPIR_Info * info, + MPIR_Comm * comm_ptr, MPIR_Win ** win_ptr); +typedef int (*MPIDI_SHM_accumulate_t) (const void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, int target_rank, + MPI_Aint target_disp, int target_count, + MPI_Datatype target_datatype, MPI_Op op, MPIR_Win * win); +typedef int (*MPIDI_SHM_win_attach_t) (MPIR_Win * win, void *base, MPI_Aint size); +typedef int (*MPIDI_SHM_win_allocate_shared_t) (MPI_Aint size, int disp_unit, MPIR_Info * info_ptr, + MPIR_Comm * comm_ptr, void **base_ptr, + MPIR_Win ** win_ptr); +typedef int (*MPIDI_SHM_rput_t) (const void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, int target_rank, + MPI_Aint target_disp, int target_count, + MPI_Datatype target_datatype, MPIR_Win * win, + MPIR_Request ** request); +typedef int (*MPIDI_SHM_win_flush_local_t) (int rank, MPIR_Win * win); +typedef int (*MPIDI_SHM_win_detach_t) (MPIR_Win * win, const void *base); +typedef int (*MPIDI_SHM_compare_and_swap_t) (const void *origin_addr, const void *compare_addr, + void *result_addr, MPI_Datatype datatype, + int target_rank, MPI_Aint target_disp, MPIR_Win * win); +typedef int (*MPIDI_SHM_raccumulate_t) (const void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, int target_rank, + MPI_Aint target_disp, int target_count, + MPI_Datatype target_datatype, MPI_Op op, MPIR_Win * win, + MPIR_Request ** request); +typedef int (*MPIDI_SHM_rget_accumulate_t) (const void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, void *result_addr, + int result_count, MPI_Datatype result_datatype, + int target_rank, MPI_Aint target_disp, int target_count, + MPI_Datatype target_datatype, MPI_Op op, MPIR_Win * win, + MPIR_Request ** request); +typedef int (*MPIDI_SHM_fetch_and_op_t) (const void *origin_addr, void *result_addr, + MPI_Datatype datatype, int target_rank, + MPI_Aint target_disp, MPI_Op op, MPIR_Win * win); +typedef int (*MPIDI_SHM_win_allocate_t) (MPI_Aint size, int disp_unit, MPIR_Info * info, + MPIR_Comm * comm, void *baseptr, MPIR_Win ** win); +typedef int (*MPIDI_SHM_win_flush_t) (int rank, MPIR_Win * win); +typedef int (*MPIDI_SHM_win_flush_local_all_t) (MPIR_Win * win); +typedef int (*MPIDI_SHM_win_unlock_all_t) (MPIR_Win * win); +typedef int (*MPIDI_SHM_win_create_dynamic_t) (MPIR_Info * info, MPIR_Comm * comm, MPIR_Win ** win); +typedef int (*MPIDI_SHM_rget_t) (void *origin_addr, int origin_count, MPI_Datatype origin_datatype, + int target_rank, MPI_Aint target_disp, int target_count, + MPI_Datatype target_datatype, MPIR_Win * win, + MPIR_Request ** request); +typedef int (*MPIDI_SHM_win_sync_t) (MPIR_Win * win); +typedef int (*MPIDI_SHM_win_flush_all_t) (MPIR_Win * win); +typedef int (*MPIDI_SHM_get_accumulate_t) (const void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, void *result_addr, + int result_count, MPI_Datatype result_datatype, + int target_rank, MPI_Aint target_disp, int target_count, + MPI_Datatype target_datatype, MPI_Op op, MPIR_Win * win); +typedef int (*MPIDI_SHM_win_lock_all_t) (int assert, MPIR_Win * win); +typedef int (*MPIDI_SHM_barrier_t) (MPIR_Comm * comm, MPIR_Errflag_t * errflag); +typedef int (*MPIDI_SHM_bcast_t) (void *buffer, int count, MPI_Datatype datatype, int root, + MPIR_Comm * comm, MPIR_Errflag_t * errflag); +typedef int (*MPIDI_SHM_allreduce_t) (const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm, + MPIR_Errflag_t * errflag); +typedef int (*MPIDI_SHM_allgather_t) (const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, MPIR_Errflag_t * errflag); +typedef int (*MPIDI_SHM_allgatherv_t) (const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, const int *displs, + MPI_Datatype recvtype, MPIR_Comm * comm, + MPIR_Errflag_t * errflag); +typedef int (*MPIDI_SHM_scatter_t) (const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, int root, + MPIR_Comm * comm, MPIR_Errflag_t * errflag); +typedef int (*MPIDI_SHM_scatterv_t) (const void *sendbuf, const int *sendcounts, const int *displs, + MPI_Datatype sendtype, void *recvbuf, int recvcount, + MPI_Datatype recvtype, int root, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag); +typedef int (*MPIDI_SHM_gather_t) (const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, int root, + MPIR_Comm * comm, MPIR_Errflag_t * errflag); +typedef int (*MPIDI_SHM_gatherv_t) (const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, const int *displs, + MPI_Datatype recvtype, int root, MPIR_Comm * comm, + MPIR_Errflag_t * errflag); +typedef int (*MPIDI_SHM_alltoall_t) (const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, MPIR_Errflag_t * errflag); +typedef int (*MPIDI_SHM_alltoallv_t) (const void *sendbuf, const int *sendcounts, + const int *sdispls, MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, const int *rdispls, + MPI_Datatype recvtype, MPIR_Comm * comm, + MPIR_Errflag_t * errflag); +typedef int (*MPIDI_SHM_alltoallw_t) (const void *sendbuf, const int *sendcounts, + const int *sdispls, const MPI_Datatype sendtypes[], + void *recvbuf, const int *recvcounts, const int *rdispls, + const MPI_Datatype recvtypes[], MPIR_Comm * comm, + MPIR_Errflag_t * errflag); +typedef int (*MPIDI_SHM_reduce_t) (const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, int root, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag); +typedef int (*MPIDI_SHM_reduce_scatter_t) (const void *sendbuf, void *recvbuf, + const int *recvcounts, MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag); +typedef int (*MPIDI_SHM_reduce_scatter_block_t) (const void *sendbuf, void *recvbuf, int recvcount, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag); +typedef int (*MPIDI_SHM_scan_t) (const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm, + MPIR_Errflag_t * errflag); +typedef int (*MPIDI_SHM_exscan_t) (const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm, + MPIR_Errflag_t * errflag); +typedef int (*MPIDI_SHM_neighbor_allgather_t) (const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, int recvcount, + MPI_Datatype recvtype, MPIR_Comm * comm, + MPIR_Errflag_t * errflag); +typedef int (*MPIDI_SHM_neighbor_allgatherv_t) (const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, const int *displs, + MPI_Datatype recvtype, MPIR_Comm * comm, + MPIR_Errflag_t * errflag); +typedef int (*MPIDI_SHM_neighbor_alltoallv_t) (const void *sendbuf, const int *sendcounts, + const int *sdispls, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, + const int *rdispls, MPI_Datatype recvtype, + MPIR_Comm * comm, MPIR_Errflag_t * errflag); +typedef int (*MPIDI_SHM_neighbor_alltoallw_t) (const void *sendbuf, const int *sendcounts, + const MPI_Aint * sdispls, + const MPI_Datatype * sendtypes, void *recvbuf, + const int *recvcounts, const MPI_Aint * rdispls, + const MPI_Datatype * recvtypes, MPIR_Comm * comm, + MPIR_Errflag_t * errflag); +typedef int (*MPIDI_SHM_neighbor_alltoall_t) (const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, int recvcount, + MPI_Datatype recvtype, MPIR_Comm * comm, + MPIR_Errflag_t * errflag); +typedef int (*MPIDI_SHM_ineighbor_allgather_t) (const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, int recvcount, + MPI_Datatype recvtype, MPIR_Comm * comm, + MPI_Request * req); +typedef int (*MPIDI_SHM_ineighbor_allgatherv_t) (const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, const int *displs, + MPI_Datatype recvtype, MPIR_Comm * comm, + MPI_Request * req); +typedef int (*MPIDI_SHM_ineighbor_alltoall_t) (const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, int recvcount, + MPI_Datatype recvtype, MPIR_Comm * comm, + MPI_Request * req); +typedef int (*MPIDI_SHM_ineighbor_alltoallv_t) (const void *sendbuf, const int *sendcounts, + const int *sdispls, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, + const int *rdispls, MPI_Datatype recvtype, + MPIR_Comm * comm, MPI_Request * req); +typedef int (*MPIDI_SHM_ineighbor_alltoallw_t) (const void *sendbuf, const int *sendcounts, + const MPI_Aint * sdispls, + const MPI_Datatype * sendtypes, void *recvbuf, + const int *recvcounts, const MPI_Aint * rdispls, + const MPI_Datatype * recvtypes, MPIR_Comm * comm, + MPI_Request * req); +typedef int (*MPIDI_SHM_ibarrier_t) (MPIR_Comm * comm, MPI_Request * req); +typedef int (*MPIDI_SHM_ibcast_t) (void *buffer, int count, MPI_Datatype datatype, int root, + MPIR_Comm * comm, MPI_Request * req); +typedef int (*MPIDI_SHM_iallgather_t) (const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, MPI_Request * req); +typedef int (*MPIDI_SHM_iallgatherv_t) (const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, const int *displs, + MPI_Datatype recvtype, MPIR_Comm * comm, MPI_Request * req); +typedef int (*MPIDI_SHM_iallreduce_t) (const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm, + MPI_Request * req); +typedef int (*MPIDI_SHM_ialltoall_t) (const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, MPI_Request * req); +typedef int (*MPIDI_SHM_ialltoallv_t) (const void *sendbuf, const int *sendcounts, + const int *sdispls, MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, const int *rdispls, + MPI_Datatype recvtype, MPIR_Comm * comm, MPI_Request * req); +typedef int (*MPIDI_SHM_ialltoallw_t) (const void *sendbuf, const int *sendcounts, + const int *sdispls, const MPI_Datatype sendtypes[], + void *recvbuf, const int *recvcounts, const int *rdispls, + const MPI_Datatype recvtypes[], MPIR_Comm * comm, + MPI_Request * req); +typedef int (*MPIDI_SHM_iexscan_t) (const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm, + MPI_Request * req); +typedef int (*MPIDI_SHM_igather_t) (const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, int root, + MPIR_Comm * comm, MPI_Request * req); +typedef int (*MPIDI_SHM_igatherv_t) (const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, const int *displs, + MPI_Datatype recvtype, int root, MPIR_Comm * comm, + MPI_Request * req); +typedef int (*MPIDI_SHM_ireduce_scatter_block_t) (const void *sendbuf, void *recvbuf, int recvcount, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, MPI_Request * req); +typedef int (*MPIDI_SHM_ireduce_scatter_t) (const void *sendbuf, void *recvbuf, + const int *recvcounts, MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, MPI_Request * req); +typedef int (*MPIDI_SHM_ireduce_t) (const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, int root, + MPIR_Comm * comm_ptr, MPI_Request * req); +typedef int (*MPIDI_SHM_iscan_t) (const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm, + MPI_Request * req); +typedef int (*MPIDI_SHM_iscatter_t) (const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, int root, + MPIR_Comm * comm, MPI_Request * req); +typedef int (*MPIDI_SHM_iscatterv_t) (const void *sendbuf, const int *sendcounts, const int *displs, + MPI_Datatype sendtype, void *recvbuf, int recvcount, + MPI_Datatype recvtype, int root, MPIR_Comm * comm_ptr, + MPI_Request * req); + +typedef struct MPIDI_SHM_funcs { + MPIDI_SHM_init_t init; + MPIDI_SHM_finalize_t finalize; + MPIDI_SHM_progress_t progress; + MPIDI_SHM_reg_hdr_handler_t reg_hdr_handler; + MPIDI_SHM_comm_connect_t comm_connect; + MPIDI_SHM_comm_disconnect_t comm_disconnect; + MPIDI_SHM_open_port_t open_port; + MPIDI_SHM_close_port_t close_port; + MPIDI_SHM_comm_accept_t comm_accept; + MPIDI_SHM_send_am_hdr_t send_am_hdr; + MPIDI_SHM_inject_am_hdr_t inject_am_hdr; + MPIDI_SHM_send_am_t send_am; + MPIDI_SHM_inject_am_t inject_am; + MPIDI_SHM_send_amv_t send_amv; + MPIDI_SHM_inject_amv_t inject_amv; + MPIDI_SHM_send_am_hdr_reply_t send_am_hdr_reply; + MPIDI_SHM_inject_am_hdr_reply_t inject_am_hdr_reply; + MPIDI_SHM_send_am_reply_t send_am_reply; + MPIDI_SHM_inject_am_reply_t inject_am_reply; + MPIDI_SHM_send_amv_reply_t send_amv_reply; + MPIDI_SHM_inject_amv_reply_t inject_amv_reply; + MPIDI_SHM_am_hdr_max_sz_t am_hdr_max_sz; + MPIDI_SHM_am_inject_max_sz_t am_inject_max_sz; + MPIDI_SHM_am_recv_t am_recv; + /* Routines that handle addressing */ + MPIDI_SHM_comm_get_lpid_t comm_get_lpid; + MPIDI_SHM_gpid_get_t gpid_get; + MPIDI_SHM_get_node_id_t get_node_id; + MPIDI_SHM_get_max_node_id_t get_max_node_id; + MPIDI_SHM_getallincomm_t getallincomm; + MPIDI_SHM_gpid_tolpidarray_t gpid_tolpidarray; + MPIDI_SHM_create_intercomm_from_lpids_t create_intercomm_from_lpids; + MPIDI_SHM_comm_create_t comm_create; + MPIDI_SHM_comm_destroy_t comm_destroy; + /* Request allocation routines */ + MPIDI_SHM_am_request_init_t am_request_init; + MPIDI_SHM_am_request_finalize_t am_request_finalize; +} MPIDI_SHM_funcs_t; + +typedef struct MPIDI_SHM_native_funcs { + MPIDI_SHM_send_t send; + MPIDI_SHM_ssend_t ssend; + MPIDI_SHM_startall_t startall; + MPIDI_SHM_send_init_t send_init; + MPIDI_SHM_ssend_init_t ssend_init; + MPIDI_SHM_rsend_init_t rsend_init; + MPIDI_SHM_bsend_init_t bsend_init; + MPIDI_SHM_isend_t isend; + MPIDI_SHM_issend_t issend; + MPIDI_SHM_cancel_send_t cancel_send; + MPIDI_SHM_recv_init_t recv_init; + MPIDI_SHM_recv_t recv; + MPIDI_SHM_irecv_t irecv; + MPIDI_SHM_imrecv_t imrecv; + MPIDI_SHM_cancel_recv_t cancel_recv; + MPIDI_SHM_alloc_mem_t alloc_mem; + MPIDI_SHM_free_mem_t free_mem; + MPIDI_SHM_improbe_t improbe; + MPIDI_SHM_iprobe_t iprobe; + MPIDI_SHM_win_set_info_t win_set_info; + MPIDI_SHM_win_shared_query_t win_shared_query; + MPIDI_SHM_put_t put; + MPIDI_SHM_win_start_t win_start; + MPIDI_SHM_win_complete_t win_complete; + MPIDI_SHM_win_post_t win_post; + MPIDI_SHM_win_wait_t win_wait; + MPIDI_SHM_win_test_t win_test; + MPIDI_SHM_win_lock_t win_lock; + MPIDI_SHM_win_unlock_t win_unlock; + MPIDI_SHM_win_get_info_t win_get_info; + MPIDI_SHM_get_t get; + MPIDI_SHM_win_free_t win_free; + MPIDI_SHM_win_fence_t win_fence; + MPIDI_SHM_win_create_t win_create; + MPIDI_SHM_accumulate_t accumulate; + MPIDI_SHM_win_attach_t win_attach; + MPIDI_SHM_win_allocate_shared_t win_allocate_shared; + MPIDI_SHM_rput_t rput; + MPIDI_SHM_win_flush_local_t win_flush_local; + MPIDI_SHM_win_detach_t win_detach; + MPIDI_SHM_compare_and_swap_t compare_and_swap; + MPIDI_SHM_raccumulate_t raccumulate; + MPIDI_SHM_rget_accumulate_t rget_accumulate; + MPIDI_SHM_fetch_and_op_t fetch_and_op; + MPIDI_SHM_win_allocate_t win_allocate; + MPIDI_SHM_win_flush_t win_flush; + MPIDI_SHM_win_flush_local_all_t win_flush_local_all; + MPIDI_SHM_win_unlock_all_t win_unlock_all; + MPIDI_SHM_win_create_dynamic_t win_create_dynamic; + MPIDI_SHM_rget_t rget; + MPIDI_SHM_win_sync_t win_sync; + MPIDI_SHM_win_flush_all_t win_flush_all; + MPIDI_SHM_get_accumulate_t get_accumulate; + MPIDI_SHM_win_lock_all_t win_lock_all; + /* Collectives */ + MPIDI_SHM_barrier_t barrier; + MPIDI_SHM_bcast_t bcast; + MPIDI_SHM_allreduce_t allreduce; + MPIDI_SHM_allgather_t allgather; + MPIDI_SHM_allgatherv_t allgatherv; + MPIDI_SHM_scatter_t scatter; + MPIDI_SHM_scatterv_t scatterv; + MPIDI_SHM_gather_t gather; + MPIDI_SHM_gatherv_t gatherv; + MPIDI_SHM_alltoall_t alltoall; + MPIDI_SHM_alltoallv_t alltoallv; + MPIDI_SHM_alltoallw_t alltoallw; + MPIDI_SHM_reduce_t reduce; + MPIDI_SHM_reduce_scatter_t reduce_scatter; + MPIDI_SHM_reduce_scatter_block_t reduce_scatter_block; + MPIDI_SHM_scan_t scan; + MPIDI_SHM_exscan_t exscan; + MPIDI_SHM_neighbor_allgather_t neighbor_allgather; + MPIDI_SHM_neighbor_allgatherv_t neighbor_allgatherv; + MPIDI_SHM_neighbor_alltoall_t neighbor_alltoall; + MPIDI_SHM_neighbor_alltoallv_t neighbor_alltoallv; + MPIDI_SHM_neighbor_alltoallw_t neighbor_alltoallw; + MPIDI_SHM_ineighbor_allgather_t ineighbor_allgather; + MPIDI_SHM_ineighbor_allgatherv_t ineighbor_allgatherv; + MPIDI_SHM_ineighbor_alltoall_t ineighbor_alltoall; + MPIDI_SHM_ineighbor_alltoallv_t ineighbor_alltoallv; + MPIDI_SHM_ineighbor_alltoallw_t ineighbor_alltoallw; + MPIDI_SHM_ibarrier_t ibarrier; + MPIDI_SHM_ibcast_t ibcast; + MPIDI_SHM_iallgather_t iallgather; + MPIDI_SHM_iallgatherv_t iallgatherv; + MPIDI_SHM_iallreduce_t iallreduce; + MPIDI_SHM_ialltoall_t ialltoall; + MPIDI_SHM_ialltoallv_t ialltoallv; + MPIDI_SHM_ialltoallw_t ialltoallw; + MPIDI_SHM_iexscan_t iexscan; + MPIDI_SHM_igather_t igather; + MPIDI_SHM_igatherv_t igatherv; + MPIDI_SHM_ireduce_scatter_block_t ireduce_scatter_block; + MPIDI_SHM_ireduce_scatter_t ireduce_scatter; + MPIDI_SHM_ireduce_t ireduce; + MPIDI_SHM_iscan_t iscan; + MPIDI_SHM_iscatter_t iscatter; + MPIDI_SHM_iscatterv_t iscatterv; +} MPIDI_SHM_native_funcs_t; + +extern MPIDI_SHM_funcs_t *MPIDI_SHM_funcs[]; +extern MPIDI_SHM_funcs_t *MPIDI_SHM_func; +extern MPIDI_SHM_native_funcs_t *MPIDI_SHM_native_funcs[]; +extern MPIDI_SHM_native_funcs_t *MPIDI_SHM_native_func; +extern int MPIDI_num_shms; +extern char MPIDI_SHM_strings[][MPIDI_MAX_SHM_STRING_LEN]; + +#ifndef MPIDI_SHM_STATIC_INLINE_PREFIX +#define MPIDI_SHM_STATIC_INLINE_PREFIX __attribute__((always_inline)) static inline +#endif + +#ifndef MPIDI_SHM_STATIC_INLINE_SUFFIX +#define MPIDI_SHM_STATIC_INLINE_SUFFIX __attribute__((always_inline)) +#endif + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_init(int rank, + int size) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_finalize(void) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_progress(int blocking) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_reg_hdr_handler(int handler_id, + MPIDI_SHM_am_origin_handler_fn + origin_handler_fn, + MPIDI_SHM_am_target_handler_fn + target_handler_fn) + MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_connect(const char *port_name, MPIR_Info * info, + int root, MPIR_Comm * comm, + MPIR_Comm ** + newcomm_ptr) + MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_disconnect(MPIR_Comm * + comm_ptr) + MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_open_port(MPIR_Info * info_ptr, + char *port_name) + MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_close_port(const char *port_name) + MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_accept(const char *port_name, MPIR_Info * info, + int root, MPIR_Comm * comm, + MPIR_Comm ** + newcomm_ptr) + MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_send_am_hdr(int rank, MPIR_Comm * comm, int handler_id, + const void *am_hdr, size_t am_hdr_sz, + MPIR_Request * sreq, + void *shm_context) + MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_am_hdr(int rank, MPIR_Comm * comm, + int handler_id, const void *am_hdr, + size_t am_hdr_sz, + void *shm_context) + MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_send_am(int rank, MPIR_Comm * comm, int handler_id, + const void *am_hdr, size_t am_hdr_sz, + const void *data, MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq, + void *shm_context) + MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_am(int rank, MPIR_Comm * comm, int handler_id, + const void *am_hdr, size_t am_hdr_sz, + const void *data, MPI_Count count, + MPI_Datatype datatype, + void *shm_context) + MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_send_amv(int rank, MPIR_Comm * comm, int handler_id, + struct iovec *am_hdrs, size_t iov_len, + const void *data, MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq, + void *shm_context) + MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_amv(int rank, MPIR_Comm * comm, int handler_id, + struct iovec *am_hdrs, size_t iov_len, + const void *data, MPI_Count count, + MPI_Datatype datatype, + void *shm_context) + MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_send_am_hdr_reply(MPIR_Context_id_t context_id, + int src_rank, int handler_id, + const void *am_hdr, size_t am_hdr_sz, + MPIR_Request * + sreq) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_am_hdr_reply(MPIR_Context_id_t context_id, + int src_rank, int handler_id, + const void *am_hdr, + size_t am_hdr_sz) + MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_send_am_reply(MPIR_Context_id_t context_id, + int src_rank, int handler_id, + const void *am_hdr, size_t am_hdr_sz, + const void *data, MPI_Count count, + MPI_Datatype datatype, + MPIR_Request * + sreq) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_am_reply(MPIR_Context_id_t context_id, + int src_rank, int handler_id, + const void *am_hdr, size_t am_hdr_sz, + const void *data, MPI_Count count, + MPI_Datatype datatype) + MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_send_amv_reply(MPIR_Context_id_t context_id, + int src_rank, int handler_id, + struct iovec *am_hdr, size_t iov_len, + const void *data, MPI_Count count, + MPI_Datatype datatype, + MPIR_Request * + sreq) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_amv_reply(MPIR_Context_id_t context_id, + int src_rank, int handler_id, + struct iovec *am_hdrs, size_t iov_len, + const void *data, MPI_Count count, + MPI_Datatype datatype) + MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX size_t MPIDI_SHM_am_hdr_max_sz(void) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX size_t MPIDI_SHM_am_inject_max_sz(void) + MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_am_recv(MPIR_Request * + req) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_get_lpid(MPIR_Comm * comm_ptr, int idx, + int *lpid_ptr, + MPL_bool is_remote) + MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_gpid_get(MPIR_Comm * comm_ptr, int rank, + MPIR_Gpid * + gpid) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_get_node_id(MPIR_Comm * comm, int rank, + MPID_Node_id_t * + id_p) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_get_max_node_id(MPIR_Comm * comm, + MPID_Node_id_t * + max_id_p) + MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_getallincomm(MPIR_Comm * comm_ptr, int local_size, + MPIR_Gpid local_gpid[], + int *singleAVT) + MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_gpid_tolpidarray(int size, MPIR_Gpid gpid[], + int lpid[]) + MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_create_intercomm_from_lpids(MPIR_Comm * newcomm_ptr, + int size, + const int lpids[]) + MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_create(MPIR_Comm * + comm) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_destroy(MPIR_Comm * + comm) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX void MPIDI_SHM_am_request_init(MPIR_Request * + req) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX void MPIDI_SHM_am_request_finalize(MPIR_Request * + req) + MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_send(const void *buf, int count, MPI_Datatype datatype, + int rank, int tag, MPIR_Comm * comm, + int context_offset, + MPIR_Request ** + request) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ssend(const void *buf, int count, + MPI_Datatype datatype, int rank, int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** + request) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_startall(int count, + MPIR_Request * + requests[]) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_send_init(const void *buf, int count, + MPI_Datatype datatype, int rank, int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** + request) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ssend_init(const void *buf, int count, + MPI_Datatype datatype, int rank, int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** + request) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_rsend_init(const void *buf, int count, + MPI_Datatype datatype, int rank, int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** + request) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_bsend_init(const void *buf, int count, + MPI_Datatype datatype, int rank, int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** + request) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_isend(const void *buf, int count, + MPI_Datatype datatype, int rank, int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** + request) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_issend(const void *buf, int count, + MPI_Datatype datatype, int rank, int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** + request) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_cancel_send(MPIR_Request * + sreq) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_recv_init(void *buf, int count, MPI_Datatype datatype, + int rank, int tag, MPIR_Comm * comm, + int context_offset, + MPIR_Request ** + request) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_recv(void *buf, int count, MPI_Datatype datatype, + int rank, int tag, MPIR_Comm * comm, + int context_offset, MPI_Status * status, + MPIR_Request ** + request) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_irecv(void *buf, int count, MPI_Datatype datatype, + int rank, int tag, MPIR_Comm * comm, + int context_offset, + MPIR_Request ** + request) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_imrecv(void *buf, int count, MPI_Datatype datatype, + MPIR_Request * message, + MPIR_Request ** + rreqp) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_cancel_recv(MPIR_Request * + rreq) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX void *MPIDI_SHM_alloc_mem(size_t size, + MPIR_Info * + info_ptr) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_free_mem(void *ptr) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_improbe(int source, int tag, MPIR_Comm * comm, + int context_offset, int *flag, + MPIR_Request ** message, + MPI_Status * + status) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_iprobe(int source, int tag, MPIR_Comm * comm, + int context_offset, int *flag, + MPI_Status * + status) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_set_info(MPIR_Win * win, + MPIR_Info * + info) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_shared_query(MPIR_Win * win, int rank, + MPI_Aint * size, int *disp_unit, + void *baseptr) + MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_put(const void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, int target_rank, + MPI_Aint target_disp, int target_count, + MPI_Datatype target_datatype, + MPIR_Win * win) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_start(MPIR_Group * group, int assert, + MPIR_Win * + win) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_complete(MPIR_Win * + win) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_post(MPIR_Group * group, int assert, + MPIR_Win * + win) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_wait(MPIR_Win * + win) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_test(MPIR_Win * win, + int *flag) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_lock(int lock_type, int rank, int assert, + MPIR_Win * + win) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_unlock(int rank, + MPIR_Win * + win) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_get_info(MPIR_Win * win, + MPIR_Info ** + info_p_p) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_get(void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, int target_rank, + MPI_Aint target_disp, int target_count, + MPI_Datatype target_datatype, + MPIR_Win * win) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_free(MPIR_Win ** + win_ptr) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_fence(int assert, + MPIR_Win * + win) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_create(void *base, MPI_Aint length, int disp_unit, + MPIR_Info * info, MPIR_Comm * comm_ptr, + MPIR_Win ** + win_ptr) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_accumulate(const void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, + int target_rank, MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, MPI_Op op, + MPIR_Win * + win) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_attach(MPIR_Win * win, void *base, + MPI_Aint size) + MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_allocate_shared(MPI_Aint size, int disp_unit, + MPIR_Info * info_ptr, + MPIR_Comm * comm_ptr, + void **base_ptr, + MPIR_Win ** + win_ptr) + MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_rput(const void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, int target_rank, + MPI_Aint target_disp, int target_count, + MPI_Datatype target_datatype, MPIR_Win * win, + MPIR_Request ** + request) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_flush_local(int rank, + MPIR_Win * + win) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_detach(MPIR_Win * win, + const void *base) + MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_compare_and_swap(const void *origin_addr, + const void *compare_addr, + void *result_addr, + MPI_Datatype datatype, + int target_rank, MPI_Aint target_disp, + MPIR_Win * + win) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_raccumulate(const void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, + int target_rank, MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, MPI_Op op, + MPIR_Win * win, + MPIR_Request ** + request) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_rget_accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + void *result_addr, int result_count, + MPI_Datatype result_datatype, + int target_rank, MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPI_Op op, MPIR_Win * win, + MPIR_Request ** + request) + MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_fetch_and_op(const void *origin_addr, + void *result_addr, MPI_Datatype datatype, + int target_rank, MPI_Aint target_disp, + MPI_Op op, + MPIR_Win * + win) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_allocate(MPI_Aint size, int disp_unit, + MPIR_Info * info, MPIR_Comm * comm, + void *baseptr, + MPIR_Win ** + win) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_flush(int rank, + MPIR_Win * + win) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_flush_local_all(MPIR_Win * + win) + MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_unlock_all(MPIR_Win * + win) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_create_dynamic(MPIR_Info * info, MPIR_Comm * comm, + MPIR_Win ** + win) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_rget(void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, int target_rank, + MPI_Aint target_disp, int target_count, + MPI_Datatype target_datatype, MPIR_Win * win, + MPIR_Request ** + request) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_sync(MPIR_Win * + win) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_flush_all(MPIR_Win * + win) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_get_accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + void *result_addr, int result_count, + MPI_Datatype result_datatype, + int target_rank, MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, MPI_Op op, + MPIR_Win * + win) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_lock_all(int assert, + MPIR_Win * + win) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_barrier(MPIR_Comm * comm, + MPIR_Errflag_t * + errflag) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_bcast(void *buffer, int count, MPI_Datatype datatype, + int root, MPIR_Comm * comm, + MPIR_Errflag_t * + errflag) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_allreduce(const void *sendbuf, void *recvbuf, + int count, MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, + MPIR_Errflag_t * + errflag) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_allgather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, + MPIR_Errflag_t * + errflag) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_allgatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, const int *displs, + MPI_Datatype recvtype, MPIR_Comm * comm, + MPIR_Errflag_t * + errflag) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_scatter(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, int root, + MPIR_Comm * comm, + MPIR_Errflag_t * + errflag) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_scatterv(const void *sendbuf, const int *sendcounts, + const int *displs, MPI_Datatype sendtype, + void *recvbuf, int recvcount, + MPI_Datatype recvtype, int root, + MPIR_Comm * comm_ptr, + MPIR_Errflag_t * + errflag) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_gather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, int root, + MPIR_Comm * comm, + MPIR_Errflag_t * + errflag) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_gatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, const int *displs, + MPI_Datatype recvtype, int root, + MPIR_Comm * comm, + MPIR_Errflag_t * + errflag) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_alltoall(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, + MPIR_Errflag_t * + errflag) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_alltoallv(const void *sendbuf, const int *sendcounts, + const int *sdispls, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, + const int *rdispls, MPI_Datatype recvtype, + MPIR_Comm * comm, + MPIR_Errflag_t * + errflag) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_alltoallw(const void *sendbuf, const int *sendcounts, + const int *sdispls, + const MPI_Datatype sendtypes[], + void *recvbuf, const int *recvcounts, + const int *rdispls, + const MPI_Datatype recvtypes[], + MPIR_Comm * comm, + MPIR_Errflag_t * + errflag) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_reduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, int root, + MPIR_Comm * comm_ptr, + MPIR_Errflag_t * + errflag) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_reduce_scatter(const void *sendbuf, void *recvbuf, + const int *recvcounts, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm_ptr, + MPIR_Errflag_t * + errflag) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_reduce_scatter_block(const void *sendbuf, + void *recvbuf, int recvcount, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm_ptr, + MPIR_Errflag_t * + errflag) + MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_scan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, + MPIR_Errflag_t * + errflag) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_exscan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, + MPIR_Errflag_t * + errflag) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_neighbor_allgather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, + void *recvbuf, int recvcount, + MPI_Datatype recvtype, + MPIR_Comm * comm, + MPIR_Errflag_t * + errflag) + MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_neighbor_allgatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, + void *recvbuf, + const int *recvcounts, + const int *displs, + MPI_Datatype recvtype, + MPIR_Comm * comm, + MPIR_Errflag_t * + errflag) + MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_neighbor_alltoallv(const void *sendbuf, + const int *sendcounts, + const int *sdispls, + MPI_Datatype sendtype, + void *recvbuf, + const int *recvcounts, + const int *rdispls, + MPI_Datatype recvtype, + MPIR_Comm * comm, + MPIR_Errflag_t * + errflag) + MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_neighbor_alltoallw(const void *sendbuf, + const int *sendcounts, + const MPI_Aint * sdispls, + const MPI_Datatype * sendtypes, + void *recvbuf, + const int *recvcounts, + const MPI_Aint * rdispls, + const MPI_Datatype * recvtypes, + MPIR_Comm * comm, + MPIR_Errflag_t * + errflag) + MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_neighbor_alltoall(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, + MPIR_Errflag_t * + errflag) + MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ineighbor_allgather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, + void *recvbuf, int recvcount, + MPI_Datatype recvtype, + MPIR_Comm * comm, + MPI_Request * + req) + MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ineighbor_allgatherv(const void *sendbuf, + int sendcount, + MPI_Datatype sendtype, + void *recvbuf, + const int *recvcounts, + const int *displs, + MPI_Datatype recvtype, + MPIR_Comm * comm, + MPI_Request * + req) + MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ineighbor_alltoall(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, + void *recvbuf, int recvcount, + MPI_Datatype recvtype, + MPIR_Comm * comm, + MPI_Request * + req) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ineighbor_alltoallv(const void *sendbuf, + const int *sendcounts, + const int *sdispls, + MPI_Datatype sendtype, + void *recvbuf, + const int *recvcounts, + const int *rdispls, + MPI_Datatype recvtype, + MPIR_Comm * comm, + MPI_Request * + req) + MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ineighbor_alltoallw(const void *sendbuf, + const int *sendcounts, + const MPI_Aint * sdispls, + const MPI_Datatype * sendtypes, + void *recvbuf, + const int *recvcounts, + const MPI_Aint * rdispls, + const MPI_Datatype * recvtypes, + MPIR_Comm * comm, + MPI_Request * + req) + MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ibarrier(MPIR_Comm * comm, + MPI_Request * + req) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ibcast(void *buffer, int count, MPI_Datatype datatype, + int root, MPIR_Comm * comm, + MPI_Request * + req) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_iallgather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, + MPI_Request * + req) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_iallgatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, const int *displs, + MPI_Datatype recvtype, MPIR_Comm * comm, + MPI_Request * + req) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_iallreduce(const void *sendbuf, void *recvbuf, + int count, MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, + MPI_Request * + req) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ialltoall(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, + MPI_Request * + req) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ialltoallv(const void *sendbuf, const int *sendcounts, + const int *sdispls, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, + const int *rdispls, MPI_Datatype recvtype, + MPIR_Comm * comm, + MPI_Request * + req) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ialltoallw(const void *sendbuf, const int *sendcounts, + const int *sdispls, + const MPI_Datatype sendtypes[], + void *recvbuf, const int *recvcounts, + const int *rdispls, + const MPI_Datatype recvtypes[], + MPIR_Comm * comm, + MPI_Request * + req) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_iexscan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, + MPI_Request * + req) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_igather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, int root, + MPIR_Comm * comm, + MPI_Request * + req) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_igatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, const int *displs, + MPI_Datatype recvtype, int root, + MPIR_Comm * comm, + MPI_Request * + req) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ireduce_scatter_block(const void *sendbuf, + void *recvbuf, int recvcount, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, + MPI_Request * + req) + MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ireduce_scatter(const void *sendbuf, void *recvbuf, + const int *recvcounts, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, + MPI_Request * + req) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ireduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, int root, + MPIR_Comm * comm_ptr, + MPI_Request * + req) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_iscan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, + MPI_Request * + req) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_iscatter(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + int root, MPIR_Comm * comm, + MPI_Request * + req) MPIDI_SHM_STATIC_INLINE_SUFFIX; +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_iscatterv(const void *sendbuf, const int *sendcounts, + const int *displs, MPI_Datatype sendtype, + void *recvbuf, int recvcount, + MPI_Datatype recvtype, int root, + MPIR_Comm * comm_ptr, + MPI_Request * + req) MPIDI_SHM_STATIC_INLINE_SUFFIX; + +#endif /* SHM_PROTOTYPES_H_INCLUDED */ diff --git a/src/mpid/ch4/shm/include/shm_impl.h b/src/mpid/ch4/shm/include/shm_impl.h new file mode 100644 index 0000000..a6d0baa --- /dev/null +++ b/src/mpid/ch4/shm/include/shm_impl.h @@ -0,0 +1,1109 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +/* ch4 shm functions */ +#ifndef SHM_IMPL_PROTOTYPES_H_INCLUDED +#define SHM_IMPL_PROTOTYPES_H_INCLUDED + +#ifndef SHM_DIRECT +#ifndef SHM_DISABLE_INLINES + +#ifndef MPIDI_SHM_STATIC_INLINE_PREFIX +#define MPIDI_SHM_STATIC_INLINE_PREFIX __attribute__((always_inline)) static inline +#endif + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_init(int rank, int size) +{ + return MPIDI_SHM_func->init(rank, size); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_finalize(void) +{ + return MPIDI_SHM_func->finalize(); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_progress(int blocking) +{ + return MPIDI_SHM_func->progress(blocking); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_reg_hdr_handler(int handler_id, + MPIDI_SHM_am_origin_handler_fn + origin_handler_fn, + MPIDI_SHM_am_target_handler_fn + target_handler_fn) +{ + return MPIDI_SHM_func->reg_hdr_handler(handler_id, origin_handler_fn, target_handler_fn); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_connect(const char *port_name, MPIR_Info * info, + int root, MPIR_Comm * comm, + MPIR_Comm ** newcomm_ptr) +{ + return MPIDI_SHM_func->comm_connect(port_name, info, root, comm, newcomm_ptr); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_disconnect(MPIR_Comm * comm_ptr) +{ + return MPIDI_SHM_func->comm_disconnect(comm_ptr); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_open_port(MPIR_Info * info_ptr, char *port_name) +{ + return MPIDI_SHM_func->open_port(info_ptr, port_name); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_close_port(const char *port_name) +{ + return MPIDI_SHM_func->close_port(port_name); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_accept(const char *port_name, MPIR_Info * info, + int root, MPIR_Comm * comm, + MPIR_Comm ** newcomm_ptr) +{ + return MPIDI_SHM_func->comm_accept(port_name, info, root, comm, newcomm_ptr); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_send_am_hdr(int rank, MPIR_Comm * comm, int handler_id, + const void *am_hdr, size_t am_hdr_sz, + MPIR_Request * sreq, void *shm_context) +{ + return MPIDI_SHM_func->send_am_hdr(rank, comm, handler_id, am_hdr, am_hdr_sz, sreq, + shm_context); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_am_hdr(int rank, MPIR_Comm * comm, + int handler_id, const void *am_hdr, + size_t am_hdr_sz, void *shm_context) +{ + return MPIDI_SHM_func->inject_am_hdr(rank, comm, handler_id, am_hdr, am_hdr_sz, shm_context); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_send_am(int rank, MPIR_Comm * comm, int handler_id, + const void *am_hdr, size_t am_hdr_sz, + const void *data, MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq, + void *shm_context) +{ + return MPIDI_SHM_func->send_am(rank, comm, handler_id, am_hdr, am_hdr_sz, data, count, datatype, + sreq, shm_context); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_am(int rank, MPIR_Comm * comm, int handler_id, + const void *am_hdr, size_t am_hdr_sz, + const void *data, MPI_Count count, + MPI_Datatype datatype, void *shm_context) +{ + return MPIDI_SHM_func->inject_am(rank, comm, handler_id, am_hdr, am_hdr_sz, data, count, + datatype, shm_context); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_send_amv(int rank, MPIR_Comm * comm, int handler_id, + struct iovec *am_hdrs, size_t iov_len, + const void *data, MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq, + void *shm_context) +{ + return MPIDI_SHM_func->send_amv(rank, comm, handler_id, am_hdrs, iov_len, data, count, datatype, + sreq, shm_context); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_amv(int rank, MPIR_Comm * comm, int handler_id, + struct iovec *am_hdrs, size_t iov_len, + const void *data, MPI_Count count, + MPI_Datatype datatype, void *shm_context) +{ + return MPIDI_SHM_func->inject_amv(rank, comm, handler_id, am_hdrs, iov_len, data, count, + datatype, shm_context); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_send_am_hdr_reply(MPIR_Context_id_t context_id, + int src_rank, int handler_id, + const void *am_hdr, size_t am_hdr_sz, + MPIR_Request * sreq) +{ + return MPIDI_SHM_func->send_am_hdr_reply(context_id, src_rank, handler_id, am_hdr, am_hdr_sz, + sreq); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_am_hdr_reply(MPIR_Context_id_t context_id, + int src_rank, int handler_id, + const void *am_hdr, + size_t am_hdr_sz) +{ + return MPIDI_SHM_func->inject_am_hdr_reply(context_id, src_rank, handler_id, am_hdr, am_hdr_sz); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_send_am_reply(MPIR_Context_id_t context_id, + int src_rank, int handler_id, + const void *am_hdr, size_t am_hdr_sz, + const void *data, MPI_Count count, + MPI_Datatype datatype, + MPIR_Request * sreq) +{ + return MPIDI_SHM_func->send_am_reply(context_id, src_rank, handler_id, am_hdr, am_hdr_sz, data, + count, datatype, sreq); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_am_reply(MPIR_Context_id_t context_id, + int src_rank, int handler_id, + const void *am_hdr, size_t am_hdr_sz, + const void *data, MPI_Count count, + MPI_Datatype datatype) +{ + return MPIDI_SHM_func->inject_am_reply(context_id, src_rank, handler_id, am_hdr, am_hdr_sz, + data, count, datatype); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_send_amv_reply(MPIR_Context_id_t context_id, + int src_rank, int handler_id, + struct iovec *am_hdr, size_t iov_len, + const void *data, MPI_Count count, + MPI_Datatype datatype, + MPIR_Request * sreq) +{ + return MPIDI_SHM_func->send_amv_reply(context_id, src_rank, handler_id, am_hdr, iov_len, data, + count, datatype, sreq); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_inject_amv_reply(MPIR_Context_id_t context_id, + int src_rank, int handler_id, + struct iovec *am_hdrs, size_t iov_len, + const void *data, MPI_Count count, + MPI_Datatype datatype) +{ + return MPIDI_SHM_func->inject_amv_reply(context_id, src_rank, handler_id, am_hdrs, iov_len, + data, count, datatype); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX size_t MPIDI_SHM_am_hdr_max_sz(void) +{ + return MPIDI_SHM_func->am_hdr_max_sz(); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX size_t MPIDI_SHM_am_inject_max_sz(void) +{ + return MPIDI_SHM_func->am_inject_max_sz(); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_am_recv(MPIR_Request * req) +{ + return MPIDI_SHM_func->am_recv(); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_get_lpid(MPIR_Comm * comm_ptr, int idx, + int *lpid_ptr, MPL_bool is_remote) +{ + return MPIDI_SHM_func->comm_get_lpid(comm_ptr, idx, lpid_ptr, is_remote); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_gpid_get(MPIR_Comm * comm_ptr, int rank, + MPIR_Gpid * gpid) +{ + return MPIDI_SHM_func->gpid_get(comm_ptr, rank, gpid); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_get_node_id(MPIR_Comm * comm, int rank, + MPID_Node_id_t * id_p) +{ + return MPIDI_SHM_func->get_node_id(comm, rank, id_p); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_get_max_node_id(MPIR_Comm * comm, + MPID_Node_id_t * max_id_p) +{ + return MPIDI_SHM_func->get_max_node_id(comm, max_id_p); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_getallincomm(MPIR_Comm * comm_ptr, int local_size, + MPIR_Gpid local_gpid[], int *singleAVT) +{ + return MPIDI_SHM_func->getallincomm(comm_ptr, local_size, local_gpid, singleAVT); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_gpid_tolpidarray(int size, MPIR_Gpid gpid[], + int lpid[]) +{ + return MPIDI_SHM_func->gpid_tolpidarray(size, gpid, lpid); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_create_intercomm_from_lpids(MPIR_Comm * newcomm_ptr, + int size, + const int lpids[]) +{ + return MPIDI_SHM_func->create_intercomm_from_lpids(newcomm_ptr, size, lpids); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_create(MPIR_Comm * comm) +{ + return MPIDI_SHM_func->comm_create(comm); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_comm_destroy(MPIR_Comm * comm) +{ + return MPIDI_SHM_func->comm_destroy(comm); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX void MPIDI_SHM_am_request_init(MPIR_Request * req) +{ + return MPIDI_SHM_func->am_request_init(req); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX void MPIDI_SHM_am_request_finalize(MPIR_Request * req) +{ + return MPIDI_SHM_func->am_request_finalize(req); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_send(const void *buf, int count, MPI_Datatype datatype, + int rank, int tag, MPIR_Comm * comm, + int context_offset, MPIR_Request ** request) +{ + return MPIDI_SHM_native_func->send(buf, count, datatype, rank, tag, comm, context_offset, + request); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ssend(const void *buf, int count, + MPI_Datatype datatype, int rank, int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) +{ + return MPIDI_SHM_native_func->ssend(buf, count, datatype, rank, tag, comm, context_offset, + request); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_startall(int count, MPIR_Request * requests[]) +{ + return MPIDI_SHM_native_func->startall(count, requests); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_send_init(const void *buf, int count, + MPI_Datatype datatype, int rank, int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) +{ + return MPIDI_SHM_native_func->send_init(buf, count, datatype, rank, tag, comm, context_offset, + request); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ssend_init(const void *buf, int count, + MPI_Datatype datatype, int rank, int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) +{ + return MPIDI_SHM_native_func->ssend_init(buf, count, datatype, rank, tag, comm, context_offset, + request); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_rsend_init(const void *buf, int count, + MPI_Datatype datatype, int rank, int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) +{ + return MPIDI_SHM_native_func->rsend_init(buf, count, datatype, rank, tag, comm, context_offset, + request); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_bsend_init(const void *buf, int count, + MPI_Datatype datatype, int rank, int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) +{ + return MPIDI_SHM_native_func->bsend_init(buf, count, datatype, rank, tag, comm, context_offset, + request); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_isend(const void *buf, int count, + MPI_Datatype datatype, int rank, int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) +{ + return MPIDI_SHM_native_func->isend(buf, count, datatype, rank, tag, comm, context_offset, + request); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_issend(const void *buf, int count, + MPI_Datatype datatype, int rank, int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) +{ + return MPIDI_SHM_native_func->issend(buf, count, datatype, rank, tag, comm, context_offset, + request); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_cancel_send(MPIR_Request * sreq) +{ + return MPIDI_SHM_native_func->cancel_send(sreq); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_recv_init(void *buf, int count, MPI_Datatype datatype, + int rank, int tag, MPIR_Comm * comm, + int context_offset, MPIR_Request ** request) +{ + return MPIDI_SHM_native_func->recv_init(buf, count, datatype, rank, tag, comm, context_offset, + request); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_recv(void *buf, int count, MPI_Datatype datatype, + int rank, int tag, MPIR_Comm * comm, + int context_offset, MPI_Status * status, + MPIR_Request ** request) +{ + return MPIDI_SHM_native_func->recv(buf, count, datatype, rank, tag, comm, context_offset, + status, request); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_irecv(void *buf, int count, MPI_Datatype datatype, + int rank, int tag, MPIR_Comm * comm, + int context_offset, MPIR_Request ** request) +{ + return MPIDI_SHM_native_func->irecv(buf, count, datatype, rank, tag, comm, context_offset, + request); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_imrecv(void *buf, int count, MPI_Datatype datatype, + MPIR_Request * message, MPIR_Request ** rreqp) +{ + return MPIDI_SHM_native_func->imrecv(buf, count, datatype, message, rreqp); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_cancel_recv(MPIR_Request * rreq) +{ + return MPIDI_SHM_native_func->cancel_recv(rreq); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX void *MPIDI_SHM_alloc_mem(size_t size, MPIR_Info * info_ptr) +{ + return MPIDI_SHM_native_func->alloc_mem(size, info_ptr); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_free_mem(void *ptr) +{ + return MPIDI_SHM_native_func->free_mem(ptr); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_improbe(int source, int tag, MPIR_Comm * comm, + int context_offset, int *flag, + MPIR_Request ** message, MPI_Status * status) +{ + return MPIDI_SHM_native_func->improbe(source, tag, comm, context_offset, flag, message, status); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_iprobe(int source, int tag, MPIR_Comm * comm, + int context_offset, int *flag, + MPI_Status * status) +{ + return MPIDI_SHM_native_func->iprobe(source, tag, comm, context_offset, flag, status); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_set_info(MPIR_Win * win, MPIR_Info * info) +{ + return MPIDI_SHM_native_func->win_set_info(win, info); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_shared_query(MPIR_Win * win, int rank, + MPI_Aint * size, int *disp_unit, + void *baseptr) +{ + return MPIDI_SHM_native_func->win_shared_query(win, rank, size, disp_unit, baseptr); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_put(const void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, int target_rank, + MPI_Aint target_disp, int target_count, + MPI_Datatype target_datatype, MPIR_Win * win) +{ + return MPIDI_SHM_native_func->put(origin_addr, origin_count, origin_datatype, target_rank, + target_disp, target_count, target_datatype, win); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_start(MPIR_Group * group, int assert, + MPIR_Win * win) +{ + return MPIDI_SHM_native_func->win_start(group, assert, win); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_complete(MPIR_Win * win) +{ + return MPIDI_SHM_native_func->win_complete(win); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_post(MPIR_Group * group, int assert, + MPIR_Win * win) +{ + return MPIDI_SHM_native_func->win_post(group, assert, win); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_wait(MPIR_Win * win) +{ + return MPIDI_SHM_native_func->win_wait(win); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_test(MPIR_Win * win, int *flag) +{ + return MPIDI_SHM_native_func->win_test(win, flag); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_lock(int lock_type, int rank, int assert, + MPIR_Win * win) +{ + return MPIDI_SHM_native_func->win_lock(lock_type, rank, assert, win); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_unlock(int rank, MPIR_Win * win) +{ + return MPIDI_SHM_native_func->win_unlock(rank, win); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_get_info(MPIR_Win * win, MPIR_Info ** info_p_p) +{ + return MPIDI_SHM_native_func->win_get_info(win, info_p_p); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_get(void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, int target_rank, + MPI_Aint target_disp, int target_count, + MPI_Datatype target_datatype, MPIR_Win * win) +{ + return MPIDI_SHM_native_func->get(origin_addr, origin_count, origin_datatype, target_rank, + target_disp, target_count, target_datatype, win); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_free(MPIR_Win ** win_ptr) +{ + return MPIDI_SHM_native_func->win_free(win_ptr); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_fence(int assert, MPIR_Win * win) +{ + return MPIDI_SHM_native_func->win_fence(assert, win); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_create(void *base, MPI_Aint length, int disp_unit, + MPIR_Info * info, MPIR_Comm * comm_ptr, + MPIR_Win ** win_ptr) +{ + return MPIDI_SHM_native_func->win_create(base, length, disp_unit, info, comm_ptr, win_ptr); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_accumulate(const void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, + int target_rank, MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, MPI_Op op, + MPIR_Win * win) +{ + return MPIDI_SHM_native_func->accumulate(origin_addr, origin_count, origin_datatype, + target_rank, target_disp, target_count, + target_datatype, op, win); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_attach(MPIR_Win * win, void *base, MPI_Aint size) +{ + return MPIDI_SHM_native_func->win_attach(win, base, size); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_allocate_shared(MPI_Aint size, int disp_unit, + MPIR_Info * info_ptr, + MPIR_Comm * comm_ptr, + void **base_ptr, + MPIR_Win ** win_ptr) +{ + return MPIDI_SHM_native_func->win_allocate_shared(size, disp_unit, info_ptr, comm_ptr, base_ptr, + win_ptr); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_rput(const void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, int target_rank, + MPI_Aint target_disp, int target_count, + MPI_Datatype target_datatype, MPIR_Win * win, + MPIR_Request ** request) +{ + return MPIDI_SHM_native_func->rput(origin_addr, origin_count, origin_datatype, target_rank, + target_disp, target_count, target_datatype, win, request); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_flush_local(int rank, MPIR_Win * win) +{ + return MPIDI_SHM_native_func->win_flush_local(rank, win); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_detach(MPIR_Win * win, const void *base) +{ + return MPIDI_SHM_native_func->win_detach(win, base); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_compare_and_swap(const void *origin_addr, + const void *compare_addr, + void *result_addr, + MPI_Datatype datatype, + int target_rank, MPI_Aint target_disp, + MPIR_Win * win) +{ + return MPIDI_SHM_native_func->compare_and_swap(origin_addr, compare_addr, result_addr, datatype, + target_rank, target_disp, win); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_raccumulate(const void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, + int target_rank, MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, MPI_Op op, + MPIR_Win * win, MPIR_Request ** request) +{ + return MPIDI_SHM_native_func->raccumulate(origin_addr, origin_count, origin_datatype, + target_rank, target_disp, target_count, + target_datatype, op, win, request); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_rget_accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + void *result_addr, int result_count, + MPI_Datatype result_datatype, + int target_rank, MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPI_Op op, MPIR_Win * win, + MPIR_Request ** request) +{ + return MPIDI_SHM_native_func->rget_accumulate(origin_addr, origin_count, origin_datatype, + result_addr, result_count, result_datatype, + target_rank, target_disp, target_count, + target_datatype, op, win, request); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_fetch_and_op(const void *origin_addr, + void *result_addr, MPI_Datatype datatype, + int target_rank, MPI_Aint target_disp, + MPI_Op op, MPIR_Win * win) +{ + return MPIDI_SHM_native_func->fetch_and_op(origin_addr, result_addr, datatype, target_rank, + target_disp, op, win); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_allocate(MPI_Aint size, int disp_unit, + MPIR_Info * info, MPIR_Comm * comm, + void *baseptr, MPIR_Win ** win) +{ + return MPIDI_SHM_native_func->win_allocate(size, disp_unit, info, comm, baseptr, win); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_flush(int rank, MPIR_Win * win) +{ + return MPIDI_SHM_native_func->win_flush(rank, win); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_flush_local_all(MPIR_Win * win) +{ + return MPIDI_SHM_native_func->win_flush_local_all(win); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_unlock_all(MPIR_Win * win) +{ + return MPIDI_SHM_native_func->win_unlock_all(win); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_create_dynamic(MPIR_Info * info, MPIR_Comm * comm, + MPIR_Win ** win) +{ + return MPIDI_SHM_native_func->win_create_dynamic(info, comm, win); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_rget(void *origin_addr, int origin_count, + MPI_Datatype origin_datatype, int target_rank, + MPI_Aint target_disp, int target_count, + MPI_Datatype target_datatype, MPIR_Win * win, + MPIR_Request ** request) +{ + return MPIDI_SHM_native_func->rget(origin_addr, origin_count, origin_datatype, target_rank, + target_disp, target_count, target_datatype, win, request); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_sync(MPIR_Win * win) +{ + return MPIDI_SHM_native_func->win_sync(win); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_flush_all(MPIR_Win * win) +{ + return MPIDI_SHM_native_func->win_flush_all(win); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_get_accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + void *result_addr, int result_count, + MPI_Datatype result_datatype, + int target_rank, MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, MPI_Op op, + MPIR_Win * win) +{ + return MPIDI_SHM_native_func->get_accumulate(origin_addr, origin_count, origin_datatype, + result_addr, result_count, result_datatype, + target_rank, target_disp, target_count, + target_datatype, op, win); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_win_lock_all(int assert, MPIR_Win * win) +{ + return MPIDI_SHM_native_func->win_lock_all(assert, win); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_barrier(MPIR_Comm * comm, MPIR_Errflag_t * errflag) +{ + return MPIDI_SHM_native_func->barrier(comm, errflag); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_bcast(void *buffer, int count, MPI_Datatype datatype, + int root, MPIR_Comm * comm, + MPIR_Errflag_t * errflag) +{ + return MPIDI_SHM_native_func->bcast(buffer, count, datatype, root, comm, errflag); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_allreduce(const void *sendbuf, void *recvbuf, + int count, MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, MPIR_Errflag_t * errflag) +{ + return MPIDI_SHM_native_func->allreduce(sendbuf, recvbuf, count, datatype, op, comm, errflag); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_allgather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, MPIR_Errflag_t * errflag) +{ + return MPIDI_SHM_native_func->allgather(sendbuf, sendcount, sendtype, recvbuf, recvcount, + recvtype, comm, errflag); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_allgatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, const int *displs, + MPI_Datatype recvtype, MPIR_Comm * comm, + MPIR_Errflag_t * errflag) +{ + return MPIDI_SHM_native_func->allgatherv(sendbuf, sendcount, sendtype, recvbuf, recvcounts, + displs, recvtype, comm, errflag); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_scatter(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, int root, + MPIR_Comm * comm, MPIR_Errflag_t * errflag) +{ + return MPIDI_SHM_native_func->scatter(sendbuf, sendcount, sendtype, recvbuf, recvcount, + recvtype, root, comm, errflag); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_scatterv(const void *sendbuf, const int *sendcounts, + const int *displs, MPI_Datatype sendtype, + void *recvbuf, int recvcount, + MPI_Datatype recvtype, int root, + MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) +{ + return MPIDI_SHM_native_func->scatterv(sendbuf, sendcounts, displs, sendtype, recvbuf, + recvcount, recvtype, root, comm_ptr, errflag); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_gather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, int root, + MPIR_Comm * comm, MPIR_Errflag_t * errflag) +{ + return MPIDI_SHM_native_func->gather(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, + root, comm, errflag); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_gatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, const int *displs, + MPI_Datatype recvtype, int root, + MPIR_Comm * comm, MPIR_Errflag_t * errflag) +{ + return MPIDI_SHM_native_func->gatherv(sendbuf, sendcount, sendtype, recvbuf, recvcounts, displs, + recvtype, root, comm, errflag); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_alltoall(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, MPIR_Errflag_t * errflag) +{ + return MPIDI_SHM_native_func->alltoall(sendbuf, sendcount, sendtype, recvbuf, recvcount, + recvtype, comm, errflag); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_alltoallv(const void *sendbuf, const int *sendcounts, + const int *sdispls, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, + const int *rdispls, MPI_Datatype recvtype, + MPIR_Comm * comm, MPIR_Errflag_t * errflag) +{ + return MPIDI_SHM_native_func->alltoallv(sendbuf, sendcounts, sdispls, sendtype, recvbuf, + recvcounts, rdispls, recvtype, comm, errflag); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_alltoallw(const void *sendbuf, const int *sendcounts, + const int *sdispls, + const MPI_Datatype sendtypes[], + void *recvbuf, const int *recvcounts, + const int *rdispls, + const MPI_Datatype recvtypes[], + MPIR_Comm * comm, MPIR_Errflag_t * errflag) +{ + return MPIDI_SHM_native_func->alltoallw(sendbuf, sendcounts, sdispls, sendtypes, recvbuf, + recvcounts, rdispls, recvtypes, comm, errflag); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_reduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, int root, + MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + return MPIDI_SHM_native_func->reduce(sendbuf, recvbuf, count, datatype, op, root, comm_ptr, + errflag); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_reduce_scatter(const void *sendbuf, void *recvbuf, + const int *recvcounts, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) +{ + return MPIDI_SHM_native_func->reduce_scatter(sendbuf, recvbuf, recvcounts, datatype, op, + comm_ptr, errflag); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_reduce_scatter_block(const void *sendbuf, + void *recvbuf, int recvcount, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) +{ + return MPIDI_SHM_native_func->reduce_scatter_block(sendbuf, recvbuf, recvcount, datatype, op, + comm_ptr, errflag); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_scan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, MPIR_Errflag_t * errflag) +{ + return MPIDI_SHM_native_func->scan(sendbuf, recvbuf, count, datatype, op, comm, errflag); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_exscan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, MPIR_Errflag_t * errflag) +{ + return MPIDI_SHM_native_func->exscan(sendbuf, recvbuf, count, datatype, op, comm, errflag); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_neighbor_allgather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, + void *recvbuf, int recvcount, + MPI_Datatype recvtype, + MPIR_Comm * comm, + MPIR_Errflag_t * errflag) +{ + return MPIDI_SHM_native_func->neighbor_allgather(sendbuf, sendcount, sendtype, recvbuf, + recvcount, recvtype, comm, errflag); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_neighbor_allgatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, + void *recvbuf, + const int *recvcounts, + const int *displs, + MPI_Datatype recvtype, + MPIR_Comm * comm, + MPIR_Errflag_t * errflag) +{ + return MPIDI_SHM_native_func->neighbor_allgatherv(sendbuf, sendcount, sendtype, recvbuf, + recvcounts, displs, recvtype, comm, errflag); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_neighbor_alltoallv(const void *sendbuf, + const int *sendcounts, + const int *sdispls, + MPI_Datatype sendtype, + void *recvbuf, + const int *recvcounts, + const int *rdispls, + MPI_Datatype recvtype, + MPIR_Comm * comm, + MPIR_Errflag_t * errflag) +{ + return MPIDI_SHM_native_func->neighbor_alltoallv(sendbuf, sendcounts, sdispls, sendtype, + recvbuf, recvcounts, rdispls, recvtype, comm, + errflag); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_neighbor_alltoallw(const void *sendbuf, + const int *sendcounts, + const MPI_Aint * sdispls, + const MPI_Datatype * sendtypes, + void *recvbuf, + const int *recvcounts, + const MPI_Aint * rdispls, + const MPI_Datatype * recvtypes, + MPIR_Comm * comm, + MPIR_Errflag_t * errflag) +{ + return MPIDI_SHM_native_func->neighbor_alltoallw(sendbuf, sendcounts, sdispls, sendtypes, + recvbuf, recvcounts, rdispls, recvtypes, comm, + errflag); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_neighbor_alltoall(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, + MPIR_Errflag_t * errflag) +{ + return MPIDI_SHM_native_func->neighbor_alltoall(sendbuf, sendcount, sendtype, recvbuf, + recvcount, recvtype, comm, errflag); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ineighbor_allgather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, + void *recvbuf, int recvcount, + MPI_Datatype recvtype, + MPIR_Comm * comm, + MPI_Request * req) +{ + return MPIDI_SHM_native_func->ineighbor_allgather(sendbuf, sendcount, sendtype, recvbuf, + recvcount, recvtype, comm, req); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ineighbor_allgatherv(const void *sendbuf, + int sendcount, + MPI_Datatype sendtype, + void *recvbuf, + const int *recvcounts, + const int *displs, + MPI_Datatype recvtype, + MPIR_Comm * comm, + MPI_Request * req) +{ + return MPIDI_SHM_native_func->ineighbor_allgatherv(sendbuf, sendcount, sendtype, recvbuf, + recvcounts, displs, recvtype, comm, req); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ineighbor_alltoall(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, + void *recvbuf, int recvcount, + MPI_Datatype recvtype, + MPIR_Comm * comm, MPI_Request * req) +{ + return MPIDI_SHM_native_func->ineighbor_alltoall(sendbuf, sendcount, sendtype, recvbuf, + recvcount, recvtype, comm, req); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ineighbor_alltoallv(const void *sendbuf, + const int *sendcounts, + const int *sdispls, + MPI_Datatype sendtype, + void *recvbuf, + const int *recvcounts, + const int *rdispls, + MPI_Datatype recvtype, + MPIR_Comm * comm, + MPI_Request * req) +{ + return MPIDI_SHM_native_func->ineighbor_alltoallv(sendbuf, sendcounts, sdispls, sendtype, + recvbuf, recvcounts, rdispls, recvtype, comm, + req); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ineighbor_alltoallw(const void *sendbuf, + const int *sendcounts, + const MPI_Aint * sdispls, + const MPI_Datatype * sendtypes, + void *recvbuf, + const int *recvcounts, + const MPI_Aint * rdispls, + const MPI_Datatype * recvtypes, + MPIR_Comm * comm, + MPI_Request * req) +{ + return MPIDI_SHM_native_func->ineighbor_alltoallw(sendbuf, sendcounts, sdispls, sendtypes, + recvbuf, recvcounts, rdispls, recvtypes, comm, + req); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ibarrier(MPIR_Comm * comm, MPI_Request * req) +{ + return MPIDI_SHM_native_func->ibarrier(comm, req); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ibcast(void *buffer, int count, MPI_Datatype datatype, + int root, MPIR_Comm * comm, MPI_Request * req) +{ + return MPIDI_SHM_native_func->ibcast(buffer, count, datatype, root, comm, req); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_iallgather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, MPI_Request * req) +{ + return MPIDI_SHM_native_func->iallgather(sendbuf, sendcount, sendtype, recvbuf, recvcount, + recvtype, comm, req); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_iallgatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, const int *displs, + MPI_Datatype recvtype, MPIR_Comm * comm, + MPI_Request * req) +{ + return MPIDI_SHM_native_func->iallgatherv(sendbuf, sendcount, sendtype, recvbuf, recvcounts, + displs, recvtype, comm, req); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_iallreduce(const void *sendbuf, void *recvbuf, + int count, MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, MPI_Request * req) +{ + return MPIDI_SHM_native_func->iallreduce(sendbuf, recvbuf, count, datatype, op, comm, req); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ialltoall(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, MPI_Request * req) +{ + return MPIDI_SHM_native_func->ialltoall(sendbuf, sendcount, sendtype, recvbuf, recvcount, + recvtype, comm, req); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ialltoallv(const void *sendbuf, const int *sendcounts, + const int *sdispls, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, + const int *rdispls, MPI_Datatype recvtype, + MPIR_Comm * comm, MPI_Request * req) +{ + return MPIDI_SHM_native_func->ialltoallv(sendbuf, sendcounts, sdispls, sendtype, recvbuf, + recvcounts, rdispls, recvtype, comm, req); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ialltoallw(const void *sendbuf, const int *sendcounts, + const int *sdispls, + const MPI_Datatype sendtypes[], + void *recvbuf, const int *recvcounts, + const int *rdispls, + const MPI_Datatype recvtypes[], + MPIR_Comm * comm, MPI_Request * req) +{ + return MPIDI_SHM_native_func->ialltoallw(sendbuf, sendcounts, sdispls, sendtypes, recvbuf, + recvcounts, rdispls, recvtypes, comm, req); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_iexscan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, MPI_Request * req) +{ + return MPIDI_SHM_native_func->iexscan(sendbuf, recvbuf, count, datatype, op, comm, req); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_igather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, int root, + MPIR_Comm * comm, MPI_Request * req) +{ + return MPIDI_SHM_native_func->igather(sendbuf, sendcount, sendtype, recvbuf, recvcount, + recvtype, root, comm, req); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_igatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, const int *displs, + MPI_Datatype recvtype, int root, + MPIR_Comm * comm, MPI_Request * req) +{ + return MPIDI_SHM_native_func->igatherv(sendbuf, sendcount, sendtype, recvbuf, recvcounts, + displs, recvtype, root, comm, req); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ireduce_scatter_block(const void *sendbuf, + void *recvbuf, int recvcount, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, + MPI_Request * req) +{ + return MPIDI_SHM_native_func->ireduce_scatter_block(sendbuf, recvbuf, recvcount, datatype, op, + comm, req); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ireduce_scatter(const void *sendbuf, void *recvbuf, + const int *recvcounts, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, MPI_Request * req) +{ + return MPIDI_SHM_native_func->ireduce_scatter(sendbuf, recvbuf, recvcounts, datatype, op, comm, + req); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_ireduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, int root, + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + return MPIDI_SHM_native_func->ireduce(sendbuf, recvbuf, count, datatype, op, root, comm_ptr, + req); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_iscan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm, MPI_Request * req) +{ + return MPIDI_SHM_native_func->iscan(sendbuf, recvbuf, count, datatype, op, comm, req); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_iscatter(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + int root, MPIR_Comm * comm, MPI_Request * req) +{ + return MPIDI_SHM_native_func->iscatter(sendbuf, sendcount, sendtype, recvbuf, recvcount, + recvtype, root, comm, req); +}; + +MPIDI_SHM_STATIC_INLINE_PREFIX int MPIDI_SHM_iscatterv(const void *sendbuf, const int *sendcounts, + const int *displs, MPI_Datatype sendtype, + void *recvbuf, int recvcount, + MPI_Datatype recvtype, int root, + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + return MPIDI_SHM_native_func->iscatterv(sendbuf, sendcounts, displs, sendtype, recvbuf, + recvcount, recvtype, root, comm_ptr, req); +}; + +#endif /* SHM_DISABLE_INLINES */ + +#else + +#define __shm_direct_stubshm__ 0 +#define __shm_direct_posix__ 1 + +#if SHM_DIRECT==__shm_direct_stubshm__ +#include "../stubshm/shm_direct.h" +#elif SHM_DIRECT==__shm_direct_posix__ +#include "../posix/shm_direct.h" +#else +#error "No direct shm included" +#endif + + +#endif /* SHM_DIRECT */ + +#endif diff --git a/src/mpid/ch4/shm/posix/Makefile.mk b/src/mpid/ch4/shm/posix/Makefile.mk new file mode 100644 index 0000000..2a0fcb3 --- /dev/null +++ b/src/mpid/ch4/shm/posix/Makefile.mk @@ -0,0 +1,38 @@ +## -*- Mode: Makefile; -*- +## vim: set ft=automake : +## +## (C) 2016 by Argonne National Laboratory. +## See COPYRIGHT in top-level directory. +## +## Portions of this code were written by Intel Corporation. +## Copyright (C) 2011-2016 Intel Corporation. Intel provides this material +## to Argonne National Laboratory subject to Software Grant and Corporate +## Contributor License Agreement dated February 8, 2012. +## + +if BUILD_SHM_POSIX + +noinst_HEADERS += src/mpid/ch4/shm/posix/posix_am.h \ + src/mpid/ch4/shm/posix/posix_coll.h \ + src/mpid/ch4/shm/posix/posix_datatypes.h \ + src/mpid/ch4/shm/posix/shm_direct.h \ + src/mpid/ch4/shm/posix/posix_init.h \ + src/mpid/ch4/shm/posix/posix_progress.h \ + src/mpid/ch4/shm/posix/posix_recv.h \ + src/mpid/ch4/shm/posix/posix_rma.h \ + src/mpid/ch4/shm/posix/posix_spawn.h \ + src/mpid/ch4/shm/posix/posix_win.h \ + src/mpid/ch4/shm/posix/posix_comm.h \ + src/mpid/ch4/shm/posix/posix_defs.h \ + src/mpid/ch4/shm/posix/posix_impl.h \ + src/mpid/ch4/shm/posix/posix_probe.h \ + src/mpid/ch4/shm/posix/posix_queue.h \ + src/mpid/ch4/shm/posix/posix_request.h \ + src/mpid/ch4/shm/posix/posix_send.h \ + src/mpid/ch4/shm/posix/posix_unimpl.h + +mpi_core_sources += src/mpid/ch4/shm/posix/globals.c \ + src/mpid/ch4/shm/posix/func_table.c \ + src/mpid/ch4/shm/posix/barrier.c + +endif diff --git a/src/mpid/ch4/shm/posix/barrier.c b/src/mpid/ch4/shm/posix/barrier.c new file mode 100644 index 0000000..151a50b --- /dev/null +++ b/src/mpid/ch4/shm/posix/barrier.c @@ -0,0 +1,42 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2016 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ + +#include <mpidimpl.h> +#include "posix_impl.h" + +/* ------------------------------------------------------- */ +/* from mpid/ch3/channels/nemesis/src/ch3i_comm.c */ +/* ------------------------------------------------------- */ + +#undef FUNCNAME +#define FUNCNAME MPIDI_POSIX_barrier_vars_init +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +int MPIDI_POSIX_barrier_vars_init(MPIDI_POSIX_barrier_vars_t * barrier_region) +{ + int mpi_errno = MPI_SUCCESS; + int i; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDI_POSIX_BARRIER_VARS_INIT); + + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDI_POSIX_BARRIER_VARS_INIT); + + if (MPIDI_POSIX_mem_region.local_rank == 0) + for (i = 0; i < MPIDI_POSIX_NUM_BARRIER_VARS; ++i) { + OPA_store_int(&barrier_region[i].context_id, -1); + OPA_store_int(&barrier_region[i].usage_cnt, 0); + OPA_store_int(&barrier_region[i].cnt, 0); + OPA_store_int(&barrier_region[i].sig0, 0); + OPA_store_int(&barrier_region[i].sig, 0); + } + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDI_POSIX_BARRIER_VARS_INIT); + return mpi_errno; +} diff --git a/src/mpid/ch4/shm/posix/func_table.c b/src/mpid/ch4/shm/posix/func_table.c new file mode 100644 index 0000000..9b8a049 --- /dev/null +++ b/src/mpid/ch4/shm/posix/func_table.c @@ -0,0 +1,152 @@ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ + +#ifndef SHM_DIRECT +#define SHM_DISABLE_INLINES +#include <mpidimpl.h> +#include "shm_direct.h" +MPIDI_SHM_funcs_t MPIDI_SHM_posix_funcs = { + MPIDI_SHM_init, + MPIDI_SHM_finalize, + MPIDI_SHM_progress, + MPIDI_SHM_reg_hdr_handler, + MPIDI_SHM_comm_connect, + MPIDI_SHM_comm_disconnect, + MPIDI_SHM_open_port, + MPIDI_SHM_close_port, + MPIDI_SHM_comm_accept, + MPIDI_SHM_send_am_hdr, + MPIDI_SHM_inject_am_hdr, + MPIDI_SHM_send_am, + MPIDI_SHM_inject_am, + MPIDI_SHM_send_amv, + MPIDI_SHM_inject_amv, + MPIDI_SHM_send_am_hdr_reply, + MPIDI_SHM_inject_am_hdr_reply, + MPIDI_SHM_send_am_reply, + MPIDI_SHM_inject_am_reply, + MPIDI_SHM_send_amv_reply, + MPIDI_SHM_inject_amv_reply, + MPIDI_SHM_am_hdr_max_sz, + MPIDI_SHM_am_inject_max_sz, + MPIDI_SHM_am_recv, + MPIDI_SHM_comm_get_lpid, + MPIDI_SHM_gpid_get, + MPIDI_SHM_get_node_id, + MPIDI_SHM_get_max_node_id, + MPIDI_SHM_getallincomm, + MPIDI_SHM_gpid_tolpidarray, + MPIDI_SHM_create_intercomm_from_lpids, + MPIDI_SHM_comm_create, + MPIDI_SHM_comm_destroy, + MPIDI_SHM_am_request_init, +}; + +MPIDI_SHM_native_funcs_t MPIDI_SHM_native_posix_funcs = { + MPIDI_SHM_send, + MPIDI_SHM_ssend, + MPIDI_SHM_startall, + MPIDI_SHM_send_init, + MPIDI_SHM_ssend_init, + MPIDI_SHM_rsend_init, + MPIDI_SHM_bsend_init, + MPIDI_SHM_isend, + MPIDI_SHM_issend, + MPIDI_SHM_cancel_send, + MPIDI_SHM_recv_init, + MPIDI_SHM_recv, + MPIDI_SHM_irecv, + MPIDI_SHM_imrecv, + MPIDI_SHM_cancel_recv, + MPIDI_SHM_alloc_mem, + MPIDI_SHM_free_mem, + MPIDI_SHM_improbe, + MPIDI_SHM_iprobe, + MPIDI_SHM_win_set_info, + MPIDI_SHM_win_shared_query, + MPIDI_SHM_put, + MPIDI_SHM_win_start, + MPIDI_SHM_win_complete, + MPIDI_SHM_win_post, + MPIDI_SHM_win_wait, + MPIDI_SHM_win_test, + MPIDI_SHM_win_lock, + MPIDI_SHM_win_unlock, + MPIDI_SHM_win_get_info, + MPIDI_SHM_get, + MPIDI_SHM_win_free, + MPIDI_SHM_win_fence, + MPIDI_SHM_win_create, + MPIDI_SHM_accumulate, + MPIDI_SHM_win_attach, + MPIDI_SHM_win_allocate_shared, + MPIDI_SHM_rput, + MPIDI_SHM_win_flush_local, + MPIDI_SHM_win_detach, + MPIDI_SHM_compare_and_swap, + MPIDI_SHM_raccumulate, + MPIDI_SHM_rget_accumulate, + MPIDI_SHM_fetch_and_op, + MPIDI_SHM_win_allocate, + MPIDI_SHM_win_flush, + MPIDI_SHM_win_flush_local_all, + MPIDI_SHM_win_unlock_all, + MPIDI_SHM_win_create_dynamic, + MPIDI_SHM_rget, + MPIDI_SHM_win_sync, + MPIDI_SHM_win_flush_all, + MPIDI_SHM_get_accumulate, + MPIDI_SHM_win_lock_all, + MPIDI_SHM_barrier, + MPIDI_SHM_bcast, + MPIDI_SHM_allreduce, + MPIDI_SHM_allgather, + MPIDI_SHM_allgatherv, + MPIDI_SHM_scatter, + MPIDI_SHM_scatterv, + MPIDI_SHM_gather, + MPIDI_SHM_gatherv, + MPIDI_SHM_alltoall, + MPIDI_SHM_alltoallv, + MPIDI_SHM_alltoallw, + MPIDI_SHM_reduce, + MPIDI_SHM_reduce_scatter, + MPIDI_SHM_reduce_scatter_block, + MPIDI_SHM_scan, + MPIDI_SHM_exscan, + MPIDI_SHM_neighbor_allgather, + MPIDI_SHM_neighbor_allgatherv, + MPIDI_SHM_neighbor_alltoall, + MPIDI_SHM_neighbor_alltoallv, + MPIDI_SHM_neighbor_alltoallw, + MPIDI_SHM_ineighbor_allgather, + MPIDI_SHM_ineighbor_allgatherv, + MPIDI_SHM_ineighbor_alltoall, + MPIDI_SHM_ineighbor_alltoallv, + MPIDI_SHM_ineighbor_alltoallw, + MPIDI_SHM_ibarrier, + MPIDI_SHM_ibcast, + MPIDI_SHM_iallgather, + MPIDI_SHM_iallgatherv, + MPIDI_SHM_iallreduce, + MPIDI_SHM_ialltoall, + MPIDI_SHM_ialltoallv, + MPIDI_SHM_ialltoallw, + MPIDI_SHM_iexscan, + MPIDI_SHM_igather, + MPIDI_SHM_igatherv, + MPIDI_SHM_ireduce_scatter_block, + MPIDI_SHM_ireduce_scatter, + MPIDI_SHM_ireduce, + MPIDI_SHM_iscan, + MPIDI_SHM_iscatter, + MPIDI_SHM_iscatterv, +}; +#endif diff --git a/src/mpid/ch4/shm/posix/globals.c b/src/mpid/ch4/shm/posix/globals.c new file mode 100644 index 0000000..84f538a --- /dev/null +++ b/src/mpid/ch4/shm/posix/globals.c @@ -0,0 +1,22 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ + +#include <mpidimpl.h> +#include "posix_impl.h" + +MPIDI_POSIX_request_queue_t MPIDI_POSIX_sendq = { NULL, NULL }; +MPIDI_POSIX_request_queue_t MPIDI_POSIX_recvq_posted = { NULL, NULL }; +MPIDI_POSIX_request_queue_t MPIDI_POSIX_recvq_unexpected = { NULL, NULL }; +MPIDI_POSIX_mem_region_t MPIDI_POSIX_mem_region = { {0} +}; + +char *MPIDI_POSIX_asym_base_addr = 0; +MPID_Thread_mutex_t MPID_shm_mutex; diff --git a/src/mpid/ch4/shm/posix/posix_am.h b/src/mpid/ch4/shm/posix/posix_am.h new file mode 100644 index 0000000..06b2351 --- /dev/null +++ b/src/mpid/ch4/shm/posix/posix_am.h @@ -0,0 +1,173 @@ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef SHM_POSIX_AM_H_INCLUDED +#define SHM_POSIX_AM_H_INCLUDED + +#include "posix_impl.h" + +static inline int MPIDI_SHM_reg_hdr_handler(int handler_id, + MPIDI_SHM_am_origin_handler_fn origin_handler_fn, + MPIDI_SHM_am_target_handler_fn target_handler_fn) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_send_am_hdr(int rank, + MPIR_Comm * comm, + int handler_id, + const void *am_hdr, + size_t am_hdr_sz, MPIR_Request * sreq, void *shm_context) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_send_am(int rank, + MPIR_Comm * comm, + int handler_id, + const void *am_hdr, + size_t am_hdr_sz, + const void *data, + MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq, void *shm_context) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_send_amv(int rank, + MPIR_Comm * comm, + int handler_id, + struct iovec *am_hdr, + size_t iov_len, + const void *data, + MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq, void *shm_context) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_send_am_hdr_reply(MPIR_Context_id_t context_id, int src_rank, + int handler_id, + const void *am_hdr, + size_t am_hdr_sz, MPIR_Request * sreq) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_send_am_reply(MPIR_Context_id_t context_id, int src_rank, + int handler_id, + const void *am_hdr, + size_t am_hdr_sz, + const void *data, + MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_send_amv_reply(MPIR_Context_id_t context_id, int src_rank, + int handler_id, + struct iovec *am_hdr, + size_t iov_len, + const void *data, + MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline size_t MPIDI_SHM_am_hdr_max_sz(void) +{ + MPIR_Assert(0); + return 0; +} + +static inline int MPIDI_SHM_inject_am_hdr(int rank, + MPIR_Comm * comm, + int handler_id, + const void *am_hdr, size_t am_hdr_sz, void *shm_context) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_inject_am(int rank, + MPIR_Comm * comm, + int handler_id, + const void *am_hdr, + size_t am_hdr_sz, + const void *data, + MPI_Count count, MPI_Datatype datatype, void *shm_context) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_inject_amv(int rank, + MPIR_Comm * comm, + int handler_id, + struct iovec *am_hdr, + size_t iov_len, + const void *data, + MPI_Count count, MPI_Datatype datatype, void *shm_context) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_inject_am_hdr_reply(MPIR_Context_id_t context_id, int src_rank, + int handler_id, + const void *am_hdr, size_t am_hdr_sz) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_inject_am_reply(MPIR_Context_id_t context_id, int src_rank, + int handler_id, + const void *am_hdr, + size_t am_hdr_sz, + const void *data, + MPI_Count count, MPI_Datatype datatype) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_inject_amv_reply(MPIR_Context_id_t context_id, int src_rank, + int handler_id, + struct iovec *am_hdr, + size_t iov_len, + const void *data, + MPI_Count count, MPI_Datatype datatype) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline size_t MPIDI_SHM_am_inject_max_sz(void) +{ + MPIR_Assert(0); + return 0; +} + +static inline int MPIDI_SHM_am_recv(MPIR_Request * req) +{ + MPIR_Assert(0); + return 0; +} + +#endif /* SHM_POSIX_AM_H_INCLUDED */ diff --git a/src/mpid/ch4/shm/posix/posix_coll.h b/src/mpid/ch4/shm/posix/posix_coll.h new file mode 100644 index 0000000..52e3228 --- /dev/null +++ b/src/mpid/ch4/shm/posix/posix_coll.h @@ -0,0 +1,876 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef SHM_POSIX_COLL_H_INCLUDED +#define SHM_POSIX_COLL_H_INCLUDED + +#include "posix_impl.h" +#include "ch4_impl.h" + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_barrier +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_barrier(MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_BARRIER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_BARRIER); + + mpi_errno = MPIR_Barrier(comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_BARRIER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_bcast +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_bcast(void *buffer, int count, MPI_Datatype datatype, + int root, MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_BCAST); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_BCAST); + + mpi_errno = MPIR_Bcast(buffer, count, datatype, root, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_BCAST); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_allreduce +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_allreduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_ALLREDUCE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_ALLREDUCE); + + mpi_errno = MPIR_Allreduce(sendbuf, recvbuf, count, datatype, op, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_ALLREDUCE); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_allgather +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_allgather(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_ALLGATHER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_ALLGATHER); + + mpi_errno = MPIR_Allgather(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, + comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_ALLGATHER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_allgatherv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_allgatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, const int *displs, + MPI_Datatype recvtype, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_ALLGATHERV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_ALLGATHERV); + + mpi_errno = MPIR_Allgatherv(sendbuf, sendcount, sendtype, + recvbuf, recvcounts, displs, recvtype, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_ALLGATHERV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_gather +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_gather(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + int root, MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_GATHER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_GATHER); + + mpi_errno = MPIR_Gather(sendbuf, sendcount, sendtype, recvbuf, recvcount, + recvtype, root, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_GATHER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_gatherv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_gatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, const int *displs, + MPI_Datatype recvtype, int root, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_GATHERV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_GATHERV); + + mpi_errno = MPIR_Gatherv(sendbuf, sendcount, sendtype, + recvbuf, recvcounts, displs, recvtype, root, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_GATHERV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_scatter +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_scatter(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + int root, MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_SCATTER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_SCATTER); + + mpi_errno = MPIR_Scatter(sendbuf, sendcount, sendtype, + recvbuf, recvcount, recvtype, root, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_SCATTER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_scatterv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_scatterv(const void *sendbuf, const int *sendcounts, + const int *displs, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + int root, MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_SCATTERV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_SCATTERV); + + mpi_errno = MPIR_Scatterv(sendbuf, sendcounts, displs, + sendtype, recvbuf, recvcount, recvtype, root, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_SCATTERV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_alltoall +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_alltoall(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_ALLTOALL); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_ALLTOALL); + + mpi_errno = MPIR_Alltoall(sendbuf, sendcount, sendtype, recvbuf, recvcount, + recvtype, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_ALLTOALL); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_alltoallv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_alltoallv(const void *sendbuf, const int *sendcounts, + const int *sdispls, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, + const int *rdispls, MPI_Datatype recvtype, + MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_ALLTOALLV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_ALLTOALLV); + + mpi_errno = MPIR_Alltoallv(sendbuf, sendcounts, sdispls, + sendtype, recvbuf, recvcounts, rdispls, recvtype, comm_ptr, errflag); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_ALLTOALLV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_alltoallw +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_alltoallw(const void *sendbuf, const int sendcounts[], + const int sdispls[], const MPI_Datatype sendtypes[], + void *recvbuf, const int recvcounts[], + const int rdispls[], const MPI_Datatype recvtypes[], + MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_ALLTOALLW); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_ALLTOALLW); + + mpi_errno = MPIR_Alltoallw(sendbuf, sendcounts, sdispls, + sendtypes, recvbuf, recvcounts, + rdispls, recvtypes, comm_ptr, errflag); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_ALLTOALLW); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_reduce +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_reduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, int root, + MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_REDUCE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_REDUCE); + + mpi_errno = MPIR_Reduce(sendbuf, recvbuf, count, datatype, op, root, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_REDUCE); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_reduce_scatter +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_reduce_scatter(const void *sendbuf, void *recvbuf, + const int recvcounts[], MPI_Datatype datatype, + MPI_Op op, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_REDUCE_SCATTER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_REDUCE_SCATTER); + + mpi_errno = MPIR_Reduce_scatter(sendbuf, recvbuf, recvcounts, datatype, op, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_REDUCE_SCATTER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_reduce_scatter_block +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_reduce_scatter_block(const void *sendbuf, void *recvbuf, + int recvcount, MPI_Datatype datatype, + MPI_Op op, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_REDUCE_SCATTER_BLOCK); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_REDUCE_SCATTER_BLOCK); + + mpi_errno = MPIR_Reduce_scatter_block(sendbuf, recvbuf, recvcount, + datatype, op, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_REDUCE_SCATTER_BLOCK); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_scan +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_scan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_SCAN); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_SCAN); + + mpi_errno = MPIR_Scan(sendbuf, recvbuf, count, datatype, op, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_SCAN); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_exscan +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_exscan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_EXSCAN); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_EXSCAN); + + mpi_errno = MPIR_Exscan(sendbuf, recvbuf, count, datatype, op, comm_ptr, errflag); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_EXSCAN); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_neighbor_allgather +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_neighbor_allgather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, int recvcount, + MPI_Datatype recvtype, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_NEIGHBOR_ALLGATHER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_NEIGHBOR_ALLGATHER); + + mpi_errno = + MPIR_Neighbor_allgather_impl(sendbuf, sendcount, sendtype, recvbuf, recvcount, recvtype, + comm_ptr); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_NEIGHBOR_ALLGATHER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_neighbor_allgatherv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_neighbor_allgatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int recvcounts[], const int displs[], + MPI_Datatype recvtype, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_NEIGHBOR_ALLGATHERV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_NEIGHBOR_ALLGATHERV); + + mpi_errno = MPIR_Neighbor_allgatherv_impl(sendbuf, sendcount, sendtype, + recvbuf, recvcounts, displs, recvtype, comm_ptr); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_NEIGHBOR_ALLGATHERV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_neighbor_alltoall +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_neighbor_alltoall(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, int recvcount, + MPI_Datatype recvtype, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_NEIGHBOR_ALLTOALL); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_NEIGHBOR_ALLTOALL); + + mpi_errno = MPIR_Neighbor_alltoall_impl(sendbuf, sendcount, sendtype, + recvbuf, recvcount, recvtype, comm_ptr); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_NEIGHBOR_ALLTOALL); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_neighbor_alltoallv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_neighbor_alltoallv(const void *sendbuf, const int sendcounts[], + const int sdispls[], MPI_Datatype sendtype, + void *recvbuf, const int recvcounts[], + const int rdispls[], MPI_Datatype recvtype, + MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_NEIGHBOR_ALLTOALLV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_NEIGHBOR_ALLTOALLV); + + mpi_errno = MPIR_Neighbor_alltoallv_impl(sendbuf, sendcounts, sdispls, sendtype, + recvbuf, recvcounts, rdispls, recvtype, comm_ptr); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_NEIGHBOR_ALLTOALLV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_neighbor_alltoallw +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_neighbor_alltoallw(const void *sendbuf, const int sendcounts[], + const MPI_Aint sdispls[], + const MPI_Datatype sendtypes[], void *recvbuf, + const int recvcounts[], const MPI_Aint rdispls[], + const MPI_Datatype recvtypes[], MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_NEIGHBOR_ALLTOALLW); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_NEIGHBOR_ALLTOALLW); + + mpi_errno = MPIR_Neighbor_alltoallw_impl(sendbuf, sendcounts, sdispls, sendtypes, + recvbuf, recvcounts, rdispls, recvtypes, comm_ptr); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_NEIGHBOR_ALLTOALLW); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_ineighbor_allgather +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_ineighbor_allgather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, int recvcount, + MPI_Datatype recvtype, MPIR_Comm * comm_ptr, + MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_INEIGHBOR_ALLGATHER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_INEIGHBOR_ALLGATHER); + + mpi_errno = MPIR_Ineighbor_allgather_impl(sendbuf, sendcount, sendtype, + recvbuf, recvcount, recvtype, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_INEIGHBOR_ALLGATHER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_ineighbor_allgatherv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_ineighbor_allgatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int recvcounts[], const int displs[], + MPI_Datatype recvtype, MPIR_Comm * comm_ptr, + MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_INEIGHBOR_ALLGATHERV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_INEIGHBOR_ALLGATHERV); + + mpi_errno = MPIR_Ineighbor_allgatherv_impl(sendbuf, sendcount, sendtype, + recvbuf, recvcounts, displs, recvtype, + comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_INEIGHBOR_ALLGATHERV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_ineighbor_alltoall +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_ineighbor_alltoall(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_INEIGHBOR_ALLTOALL); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_INEIGHBOR_ALLTOALL); + + mpi_errno = MPIR_Ineighbor_alltoall_impl(sendbuf, sendcount, sendtype, + recvbuf, recvcount, recvtype, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_INEIGHBOR_ALLTOALL); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_ineighbor_alltoallv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_ineighbor_alltoallv(const void *sendbuf, const int sendcounts[], + const int sdispls[], MPI_Datatype sendtype, + void *recvbuf, const int recvcounts[], + const int rdispls[], MPI_Datatype recvtype, + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_INEIGHBOR_ALLTOALLV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_INEIGHBOR_ALLTOALLV); + + mpi_errno = MPIR_Ineighbor_alltoallv_impl(sendbuf, sendcounts, sdispls, sendtype, + recvbuf, recvcounts, rdispls, recvtype, + comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_INEIGHBOR_ALLTOALLV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_ineighbor_alltoallw +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_ineighbor_alltoallw(const void *sendbuf, const int sendcounts[], + const MPI_Aint sdispls[], + const MPI_Datatype sendtypes[], void *recvbuf, + const int recvcounts[], const MPI_Aint rdispls[], + const MPI_Datatype recvtypes[], + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_INEIGHBOR_ALLTOALLW); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_INEIGHBOR_ALLTOALLW); + + mpi_errno = MPIR_Ineighbor_alltoallw_impl(sendbuf, sendcounts, sdispls, sendtypes, + recvbuf, recvcounts, rdispls, recvtypes, + comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_INEIGHBOR_ALLTOALLW); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_ibarrier +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_ibarrier(MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_IBARRIER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_IBARRIER); + + mpi_errno = MPIR_Ibarrier_impl(comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_IBARRIER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_ibcast +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_ibcast(void *buffer, int count, MPI_Datatype datatype, + int root, MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_IBCAST); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_IBCAST); + + mpi_errno = MPIR_Ibcast_impl(buffer, count, datatype, root, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_IBCAST); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_iallgather +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_iallgather(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_IALLGATHER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_IALLGATHER); + + mpi_errno = MPIR_Iallgather_impl(sendbuf, sendcount, sendtype, recvbuf, + recvcount, recvtype, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_IALLGATHER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_iallgatherv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_iallgatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, const int *displs, + MPI_Datatype recvtype, MPIR_Comm * comm_ptr, + MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_IALLGATHERV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_IALLGATHERV); + + mpi_errno = MPIR_Iallgatherv_impl(sendbuf, sendcount, sendtype, + recvbuf, recvcounts, displs, recvtype, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_IALLGATHERV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_ialltoall +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_ialltoall(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_IALLTOALL); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_IALLTOALL); + + mpi_errno = MPIR_Ialltoall_impl(sendbuf, sendcount, sendtype, recvbuf, + recvcount, recvtype, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_IALLTOALL); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_ialltoallv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_ialltoallv(const void *sendbuf, const int *sendcounts, + const int *sdispls, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, + const int *rdispls, MPI_Datatype recvtype, + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_IALLTOALLV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_IALLTOALLV); + + mpi_errno = MPIR_Ialltoallv_impl(sendbuf, sendcounts, sdispls, + sendtype, recvbuf, recvcounts, + rdispls, recvtype, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_IALLTOALLV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_ialltoallw +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_ialltoallw(const void *sendbuf, const int *sendcounts, + const int *sdispls, const MPI_Datatype sendtypes[], + void *recvbuf, const int *recvcounts, + const int *rdispls, const MPI_Datatype recvtypes[], + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_IALLTOALLW); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_IALLTOALLW); + + mpi_errno = MPIR_Ialltoallw_impl(sendbuf, sendcounts, sdispls, + sendtypes, recvbuf, recvcounts, + rdispls, recvtypes, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_IALLTOALLW); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_iexscan +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_iexscan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm_ptr, + MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_IEXSCAN); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_IEXSCAN); + + mpi_errno = MPIR_Iexscan_impl(sendbuf, recvbuf, count, datatype, op, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_IEXSCAN); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_igather +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_igather(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + int root, MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_IGATHER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_IGATHER); + + mpi_errno = MPIR_Igather_impl(sendbuf, sendcount, sendtype, recvbuf, + recvcount, recvtype, root, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_IGATHER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_igatherv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_igatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, const int *displs, + MPI_Datatype recvtype, int root, MPIR_Comm * comm_ptr, + MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_IGATHERV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_IGATHERV); + + mpi_errno = MPIR_Igatherv_impl(sendbuf, sendcount, sendtype, + recvbuf, recvcounts, displs, recvtype, root, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_IGATHERV); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_ireduce_scatter_block +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_ireduce_scatter_block(const void *sendbuf, void *recvbuf, + int recvcount, MPI_Datatype datatype, + MPI_Op op, MPIR_Comm * comm_ptr, + MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_IREDUCE_SCATTER_BLOCK); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_IREDUCE_SCATTER_BLOCK); + + mpi_errno = MPIR_Ireduce_scatter_block_impl(sendbuf, recvbuf, recvcount, + datatype, op, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_IREDUCE_SCATTER_BLOCK); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_ireduce_scatter +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_ireduce_scatter(const void *sendbuf, void *recvbuf, + const int recvcounts[], MPI_Datatype datatype, + MPI_Op op, MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_IREDUCE_SCATTER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_IREDUCE_SCATTER); + + mpi_errno = MPIR_Ireduce_scatter_impl(sendbuf, recvbuf, recvcounts, datatype, op, + comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_IREDUCE_SCATTER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_ireduce +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_ireduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, int root, + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_IREDUCE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_IREDUCE); + + mpi_errno = MPIR_Ireduce_impl(sendbuf, recvbuf, count, datatype, op, root, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_IREDUCE); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_iallreduce +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_iallreduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_IALLREDUCE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_IALLREDUCE); + + mpi_errno = MPIR_Iallreduce_impl(sendbuf, recvbuf, count, datatype, op, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_IALLREDUCE); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_iscan +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_iscan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm_ptr, + MPI_Request * req) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_ISCAN); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_ISCAN); + + mpi_errno = MPIR_Iscan_impl(sendbuf, recvbuf, count, datatype, op, comm_ptr, req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_ISCAN); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_iscatter +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_iscatter(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + int root, MPIR_Comm * comm, MPI_Request * request) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_ISCATTER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_ISCATTER); + + mpi_errno = MPIR_Iscatter_impl(sendbuf, sendcount, sendtype, recvbuf, + recvcount, recvtype, root, comm, request); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_ISCATTER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_iscatterv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_iscatterv(const void *sendbuf, const int *sendcounts, + const int *displs, MPI_Datatype sendtype, + void *recvbuf, int recvcount, + MPI_Datatype recvtype, int root, + MPIR_Comm * comm, MPI_Request * request) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_ISCATTERV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_ISCATTERV); + + mpi_errno = MPIR_Iscatterv_impl(sendbuf, sendcounts, displs, sendtype, + recvbuf, recvcount, recvtype, root, comm, request); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_ISCATTERV); + return mpi_errno; +} + +#endif diff --git a/src/mpid/ch4/shm/posix/posix_comm.h b/src/mpid/ch4/shm/posix/posix_comm.h new file mode 100644 index 0000000..336cd74 --- /dev/null +++ b/src/mpid/ch4/shm/posix/posix_comm.h @@ -0,0 +1,45 @@ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef SHM_POSIX_COMM_H_INCLUDED +#define SHM_POSIX_COMM_H_INCLUDED + +#include "posix_impl.h" +#include "mpl_utlist.h" + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_comm_create +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_comm_create(MPIR_Comm * comm) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_COMM_CREATE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_COMM_CREATE); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_COMM_CREATE); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_comm_destroy +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_comm_destroy(MPIR_Comm * comm) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_COMM_DESTROY); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_COMM_DESTROY); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_COMM_DESTROY); + return mpi_errno; +} + + +#endif /* SHM_POSIX_COMM_H_INCLUDED */ diff --git a/src/mpid/ch4/shm/posix/posix_datatypes.h b/src/mpid/ch4/shm/posix/posix_datatypes.h new file mode 100644 index 0000000..e99e853 --- /dev/null +++ b/src/mpid/ch4/shm/posix/posix_datatypes.h @@ -0,0 +1,185 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2016 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef SHM_DATATYPES_H +#define SHM_DATATYPES_H + +/* ************************************************************************** */ +/* from mpid/ch3/channels/nemesis/include/mpid_nem_datatypes.h */ +/* ************************************************************************** */ + +#define MPIDI_POSIX_OFFSETOF(struc, field) ((int)(&((struc *)0)->field)) +#define MPIDI_POSIX_CACHE_LINE_LEN (64) +#define MPIDI_POSIX_NUM_CELLS (64) +#define MPIDI_POSIX_CELL_LEN (64*1024) + +#if (SIZEOF_OPA_PTR_T > 8) +#if (SIZEOF_OPA_PTR_T > 16) +#error unexpected size for OPA_ptr_t +#endif +#define MPIDI_POSIX_CELL_HEAD_LEN 16 /* We use this to keep elements 64-bit aligned */ +#else /* (SIZEOF_OPA_PTR_T <= 8) */ +#define MPIDI_POSIX_CELL_HEAD_LEN 8 /* We use this to keep elements 64-bit aligned */ +#endif + +/* cell header with matching info now is cache line */ +#define MPIDI_POSIX_CELL_PAYLOAD_LEN (MPIDI_POSIX_CELL_LEN - MPIDI_POSIX_CACHE_LINE_LEN + MPIDI_POSIX_MPICH_HEAD_LEN) + +#define MPIDI_POSIX_CALC_CELL_LEN(cellp) (MPIDI_POSIX_CELL_HEAD_LEN + MPIDI_POSIX_MPICH_HEAD_LEN + MPIDI_POSIX_CELL_DLEN (cell)) + +#define MPIDI_POSIX_ALIGNED(addr, bytes) ((((unsigned long)addr) & (((unsigned long)bytes)-1)) == 0) + +#define MPIDI_POSIX_PKT_UNKNOWN 0 +#define MPIDI_POSIX_PKT_MPICH 1 +#define MPIDI_POSIX_PKT_MPICH_HEAD 2 + +#define MPIDI_POSIX_FBOX_SOURCE(cell) (MPIDI_POSIX_mem_region.local_procs[(cell)->pkt.mpich.source]) +#define MPIDI_POSIX_CELL_SOURCE(cell) ((cell)->pkt.mpich.source) +#define MPIDI_POSIX_CELL_DEST(cell) ((cell)->pkt.mpich.dest) +#define MPIDI_POSIX_CELL_DLEN(cell) ((cell)->pkt.mpich.datalen) +#define MPIDI_POSIX_CELL_SEQN(cell) ((cell)->pkt.mpich.seqno) + +#define MPIDI_POSIX_MPICH_HEAD_LEN sizeof(MPIDI_POSIX_pkt_header_t) +#define MPIDI_POSIX_DATA_LEN (MPIDI_POSIX_CELL_PAYLOAD_LEN - MPIDI_POSIX_MPICH_HEAD_LEN) + +#define MPIDI_POSIX_PKT_HEADER_FIELDS \ + int source; \ + int dest; \ + uintptr_t datalen; \ + unsigned short seqno; \ + unsigned short type; /* currently used only with checkpointing */ + +typedef struct MPIDI_POSIX_pkt_header { + MPIDI_POSIX_PKT_HEADER_FIELDS; +} MPIDI_POSIX_pkt_header_t; + +typedef struct MPIDI_POSIX_pkt_mpich { + MPIDI_POSIX_PKT_HEADER_FIELDS; + union { + char payload[MPIDI_POSIX_DATA_LEN]; + double dummy; /* align paylod to double */ + } p; +} MPIDI_POSIX_pkt_mpich_t; + +typedef union { + MPIDI_POSIX_pkt_header_t header; + MPIDI_POSIX_pkt_mpich_t mpich; +} MPIDI_POSIX_pkt_t; + +/* Nemesis cells which are to be used in shared memory need to use + * "relative pointers" because the absolute pointers to a cell from + * different processes may be different. Relative pointers are + * offsets from the beginning of the mmapped region where they live. + * We use different types for relative and absolute pointers to help + * catch errors. Use MPIDI_POSIX_REL_TO_ABS and MPIDI_POSIX_ABS_TO_REL to + * convert between relative and absolute pointers. */ + +/* This should always be exactly the size of a pointer */ +typedef struct MPIDI_POSIX_cell_rel_ptr { + OPA_ptr_t p; +} MPIDI_POSIX_cell_rel_ptr_t; + +/* MPIDI_POSIX_cell and MPIDI_POSIX_abs_cell must be kept in sync so that we + * can cast between them. MPIDI_POSIX_abs_cell should only be used when + * a cell is enqueued on a queue local to a single process (e.g., a + * queue in a network module) where relative pointers are not + * needed. */ + +typedef struct MPIDI_POSIX_cell { + MPIDI_POSIX_cell_rel_ptr_t next; +#if (MPIDI_POSIX_CELL_HEAD_LEN > SIZEOF_OPA_PTR_T) + char padding[MPIDI_POSIX_CELL_HEAD_LEN - sizeof(MPIDI_POSIX_cell_rel_ptr_t)]; +#endif + int my_rank; + int rank; + int tag; + int context_id; + MPIR_Request *pending; +#if MPIDI_POSIX_CACHE_LINE_LEN != 0 + char padding[MPIDI_POSIX_CACHE_LINE_LEN - MPIDI_POSIX_CELL_HEAD_LEN - MPIDI_POSIX_MPICH_HEAD_LEN - 4 * sizeof(int) - sizeof(MPIR_Request *)]; /* should be 64-16-16-16-8 = 8 */ +#endif + volatile MPIDI_POSIX_pkt_t pkt; +} MPIDI_POSIX_cell_t; +typedef MPIDI_POSIX_cell_t *MPIDI_POSIX_cell_ptr_t; + +typedef struct MPIDI_POSIX_abs_cell { + struct MPIDI_POSIX_abs_cell *next; +#if (MPIDI_POSIX_CELL_HEAD_LEN > SIZEOF_VOID_P) + char padding[MPIDI_POSIX_CELL_HEAD_LEN - sizeof(struct MPIDI_POSIX_abs_cell *)]; +#endif + volatile MPIDI_POSIX_pkt_t pkt; +} MPIDI_POSIX_abs_cell_t; +typedef MPIDI_POSIX_abs_cell_t *MPIDI_POSIX_abs_cell_ptr_t; + +#define MPIDI_POSIX_CELL_TO_PACKET(cellp) (&(cellp)->pkt) +#define MPIDI_POSIX_PACKET_TO_CELL(packetp) \ + ((MPIDI_POSIX_cell_ptr_t) ((char*)(packetp) - (char *)MPIDI_POSIX_CELL_TO_PACKET((MPIDI_POSIX_cell_ptr_t)0))) +#define MPIDI_POSIX_MIN_PACKET_LEN (sizeof (MPIDI_POSIX_pkt_header_t)) +#define MPIDI_POSIX_MAX_PACKET_LEN (sizeof (MPIDI_POSIX_pkt_t)) +#define MPIDI_POSIX_PACKET_LEN(pkt) ((pkt)->mpich.datalen + MPIDI_POSIX_MPICH_HEAD_LEN) + +#define MPIDI_POSIX_OPT_LOAD 16 +#define MPIDI_POSIX_OPT_SIZE ((sizeof(MPIDI_CH3_Pkt_t)) + (MPIDI_POSIX_OPT_LOAD)) +#define MPIDI_POSIX_OPT_HEAD_LEN ((MPIDI_POSIX_MPICH_HEAD_LEN) + (MPIDI_POSIX_OPT_SIZE)) + +#define MPIDI_POSIX_PACKET_OPT_LEN(pkt) \ + (((pkt)->mpich.datalen < MPIDI_POSIX_OPT_SIZE) ? (MPIDI_POSIX_OPT_HEAD_LEN) : (MPIDI_POSIX_PACKET_LEN(pkt))) + +#define MPIDI_POSIX_PACKET_PAYLOAD(pkt) ((pkt)->mpich.payload) + +typedef struct MPIDI_POSIX_queue { + MPIDI_POSIX_cell_rel_ptr_t head; + MPIDI_POSIX_cell_rel_ptr_t tail; +#if (MPIDI_POSIX_CACHE_LINE_LEN > (2 * SIZEOF_OPA_PTR_T)) + char padding1[MPIDI_POSIX_CACHE_LINE_LEN - 2 * sizeof(MPIDI_POSIX_cell_rel_ptr_t)]; +#endif + MPIDI_POSIX_cell_rel_ptr_t my_head; +#if (MPIDI_POSIX_CACHE_LINE_LEN > SIZEOF_OPA_PTR_T) + char padding2[MPIDI_POSIX_CACHE_LINE_LEN - sizeof(MPIDI_POSIX_cell_rel_ptr_t)]; +#endif +#if !defined(MPIDI_POSIX_USE_LOCK_FREE_QUEUES) + /* see FIXME in mpid_nem_queue.h */ +#define MPIDI_POSIX_queue_mutex_t MPID_Thread_mutex_t + MPIDI_POSIX_queue_mutex_t lock; + char padding3[MPIDI_POSIX_CACHE_LINE_LEN - sizeof(MPID_Thread_mutex_t)]; +#endif +} +MPIDI_POSIX_queue_t, *MPIDI_POSIX_queue_ptr_t; + +/* Fast Boxes*/ +typedef union { + OPA_int_t value; +#if MPIDI_POSIX_CACHE_LINE_LEN != 0 + char padding[MPIDI_POSIX_CACHE_LINE_LEN]; +#endif +} MPIDI_POSIX_opt_volint_t; + +typedef struct MPIDI_POSIX_fbox_common { + MPIDI_POSIX_opt_volint_t flag; +} MPIDI_POSIX_fbox_common_t, *MPIDI_POSIX_fbox_common_ptr_t; + +typedef struct MPIDI_POSIX_fbox_mpich { + MPIDI_POSIX_opt_volint_t flag; + MPIDI_POSIX_cell_t cell; +} MPIDI_POSIX_fbox_mpich_t; + +#define MPIDI_POSIX_FBOX_DATALEN MPIDI_POSIX_DATA_LEN + +typedef union { + MPIDI_POSIX_fbox_common_t common; + MPIDI_POSIX_fbox_mpich_t mpich; +} MPIDI_POSIX_fastbox_t; + +typedef struct MPIDI_POSIX_fbox_arrays { + MPIDI_POSIX_fastbox_t **in; + MPIDI_POSIX_fastbox_t **out; +} MPIDI_POSIX_fbox_arrays_t; + +#endif /* ifndef SHM_DATATYPES_H */ diff --git a/src/mpid/ch4/shm/posix/posix_defs.h b/src/mpid/ch4/shm/posix/posix_defs.h new file mode 100644 index 0000000..459666d --- /dev/null +++ b/src/mpid/ch4/shm/posix/posix_defs.h @@ -0,0 +1,115 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2016 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef SHM_DEFS_H +#define SHM_DEFS_H + +/* ************************************************************************** */ +/* from mpid/ch3/channels/nemesis/include/mpid_nem_defs.h */ +/* ************************************************************************** */ + +#include "mpidu_shm.h" +#define MPIDI_POSIX_MAX_FNAME_LEN 256 + +/* FIXME: This definition should be gotten from mpidi_ch3_impl.h */ +#ifndef MPIDI_POSIX_MAX_HOSTNAME_LEN +#define MPIDI_POSIX_MAX_HOSTNAME_LEN 256 +#endif /* MPIDI_POSIX_MAX_HOSTNAME_LEN */ + +extern char MPIDI_POSIX_hostname[MPIDI_POSIX_MAX_HOSTNAME_LEN]; + +#define MPIDI_POSIX_RET_OK 1 +#define MPIDI_POSIX_RET_NG -1 +#define MPIDI_POSIX_KEY 632236 +#define MPIDI_POSIX_ANY_SOURCE -1 +#define MPIDI_POSIX_IN 1 +#define MPIDI_POSIX_OUT 0 + +#define MPIDI_POSIX_POLL_IN 0 +#define MPIDI_POSIX_POLL_OUT 1 + +#define MPIDI_POSIX_ASYMM_NULL_VAL 64 +typedef MPI_Aint MPIDI_POSIX_addr_t; +extern char *MPIDI_POSIX_asym_base_addr; + +#define MPIDI_POSIX_REL_NULL (0x0) +#define MPIDI_POSIX_IS_REL_NULL(rel_ptr) (OPA_load_ptr(&(rel_ptr).p) == MPIDI_POSIX_REL_NULL) +#define MPIDI_POSIX_SET_REL_NULL(rel_ptr) (OPA_store_ptr(&((rel_ptr).p), MPIDI_POSIX_REL_NULL)) +#define MPIDI_POSIX_REL_ARE_EQUAL(rel_ptr1, rel_ptr2) \ + (OPA_load_ptr(&(rel_ptr1).p) == OPA_load_ptr(&(rel_ptr2).p)) + +#ifndef MPIDI_POSIX_SYMMETRIC_QUEUES + +static inline MPIDI_POSIX_cell_ptr_t MPIDI_POSIX_REL_TO_ABS(MPIDI_POSIX_cell_rel_ptr_t r) +{ + return (MPIDI_POSIX_cell_ptr_t) ((char *) OPA_load_ptr(&r.p) + + (MPIDI_POSIX_addr_t) MPIDI_POSIX_asym_base_addr); +} + +static inline MPIDI_POSIX_cell_rel_ptr_t MPIDI_POSIX_ABS_TO_REL(MPIDI_POSIX_cell_ptr_t a) +{ + MPIDI_POSIX_cell_rel_ptr_t ret; + OPA_store_ptr(&ret.p, (char *) a - (MPIDI_POSIX_addr_t) MPIDI_POSIX_asym_base_addr); + return ret; +} + +#else /*MPIDI_POSIX_SYMMETRIC_QUEUES */ +#define MPIDI_POSIX_REL_TO_ABS(ptr) (ptr) +#define MPIDI_POSIX_ABS_TO_REL(ptr) (ptr) +#endif /*MPIDI_POSIX_SYMMETRIC_QUEUES */ + +/* NOTE: MPIDI_POSIX_IS_LOCAL should only be used when the process is known to be + in your comm_world (such as at init time). This will generally not work for + dynamic processes. Check vc_ch->is_local instead. If that is true, then + it's safe to use MPIDI_POSIX_LOCAL_RANK. */ +#define MPIDI_POSIX_NON_LOCAL -1 +#define MPIDI_POSIX_IS_LOCAL(grank) (MPIDI_POSIX_mem_region.local_ranks[grank] != MPIDI_POSIX_NON_LOCAL) +#define MPIDI_POSIX_LOCAL_RANK(grank) (MPIDI_POSIX_mem_region.local_ranks[grank]) +#define MPIDI_POSIX_NUM_BARRIER_VARS 16 +#define MPIDI_POSIX_SHM_MUTEX MPID_shm_mutex +typedef struct MPIDI_POSIX_barrier_vars { + OPA_int_t context_id; + OPA_int_t usage_cnt; + OPA_int_t cnt; +#if MPIDI_POSIX_CACHE_LINE_LEN != SIZEOF_INT + char padding0[MPIDI_POSIX_CACHE_LINE_LEN - sizeof(int)]; +#endif + OPA_int_t sig0; + OPA_int_t sig; + char padding1[MPIDI_POSIX_CACHE_LINE_LEN - 2 * sizeof(int)]; +} MPIDI_POSIX_barrier_vars_t; + +typedef struct MPIDI_POSIX_mem_region { + MPIDU_shm_seg_t memory; + MPIDU_shm_seg_info_t *seg; + int num_seg; + int map_lock; + int num_local; + int num_procs; + int *local_procs; /* local_procs[lrank] gives the global rank of proc with local rank lrank */ + int local_rank; + int *local_ranks; /* local_ranks[grank] gives the local rank of proc with global rank grank or MPIDI_POSIX_NON_LOCAL */ + int ext_procs; /* Number of non-local processes */ + int *ext_ranks; /* Ranks of non-local processes */ + MPIDI_POSIX_fbox_arrays_t mailboxes; + MPIDI_POSIX_cell_ptr_t Elements; + MPIDI_POSIX_queue_ptr_t *FreeQ; + MPIDI_POSIX_queue_ptr_t *RecvQ; + MPIDU_shm_barrier_t *barrier; + MPIDI_POSIX_queue_ptr_t my_freeQ; + MPIDI_POSIX_queue_ptr_t my_recvQ; + MPIDI_POSIX_barrier_vars_t *barrier_vars; + int rank; + struct MPIDI_POSIX_mem_region *next; +} MPIDI_POSIX_mem_region_t, *MPIDI_POSIX_mem_region_ptr_t; +extern MPIDI_POSIX_mem_region_t MPIDI_POSIX_mem_region; +extern MPID_Thread_mutex_t MPID_shm_mutex; + +#endif /* ifndef SHM_DEFS_H */ diff --git a/src/mpid/ch4/shm/posix/posix_impl.h b/src/mpid/ch4/shm/posix/posix_impl.h new file mode 100644 index 0000000..97f3e68 --- /dev/null +++ b/src/mpid/ch4/shm/posix/posix_impl.h @@ -0,0 +1,174 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef SHM_POSIX_IMPL_H_INCLUDED +#define SHM_POSIX_IMPL_H_INCLUDED + +#include <mpidimpl.h> +#include "mpidch4r.h" + +#include "pmi.h" + +#include "mpidu_shm.h" + +/* ---------------------------------------------------- */ +/* temp headers */ +/* ---------------------------------------------------- */ +#include "posix_datatypes.h" /* MPID_nem datatypes like cell, fastbox defined here */ +#include "posix_defs.h" /* MPID_nem objects like shared memory region defined here */ +#include "posix_queue.h" /* MPIDI_POSIX_queue functions defined here */ + +/* ---------------------------------------------------- */ +/* constants */ +/* ---------------------------------------------------- */ +#define MPIDI_POSIX_EAGER_THRESHOLD MPIDI_POSIX_DATA_LEN +#define MPIDI_POSIX_TYPESTANDARD 0 +#define MPIDI_POSIX_TYPEEAGER 1 +#define MPIDI_POSIX_TYPELMT 2 +#define MPIDI_POSIX_TYPESYNC 3 +#define MPIDI_POSIX_TYPEBUFFERED 4 +#define MPIDI_POSIX_TYPEREADY 5 +#define MPIDI_POSIX_TYPEACK 6 +#define MPIDI_POSIX_REQUEST(req) (&(req)->dev.ch4.shm.posix) + +/* ---------------------------------------------------- */ +/* shm specific object data */ +/* ---------------------------------------------------- */ +/* VCR Table Data */ +typedef struct { + unsigned int avt_rank; +} MPIDI_POSIX_vcr_t; + +struct MPIDI_POSIX_vcrt_t { + MPIR_OBJECT_HEADER; + unsigned size; /**< Number of entries in the table */ + MPIDI_POSIX_vcr_t vcr_table[0]; /**< Array of virtual connection references */ +}; +/* ---------------------------------------------------- */ +/* general send/recv queue types, macros and objects */ +/* ---------------------------------------------------- */ +typedef struct { + MPIR_Request *head; + MPIR_Request *tail; +} MPIDI_POSIX_request_queue_t; + +#define MPIDI_POSIX_REQUEST_COMPLETE(req_) \ +{ \ + int incomplete__; \ + MPIR_cc_decr((req_)->cc_ptr, &incomplete__); \ + dtype_release_if_not_builtin(MPIDI_POSIX_REQUEST(req_)->datatype); \ + if (!incomplete__) \ + MPIDI_CH4U_request_release(req_); \ +} + +#define MPIDI_POSIX_REQUEST_ENQUEUE(req,queue) \ +{ \ + if ((queue).tail != NULL) \ + MPIDI_POSIX_REQUEST((queue).tail)->next = req; \ + else \ + (queue).head = req; \ + (queue).tail = req; \ +} + +#define MPIDI_POSIX_REQUEST_DEQUEUE(req_p,prev_req,queue) \ +{ \ + MPIR_Request *next = MPIDI_POSIX_REQUEST(*(req_p))->next; \ + if ((queue).head == *(req_p)) \ + (queue).head = next; \ + else \ + MPIDI_POSIX_REQUEST(prev_req)->next = next; \ + if ((queue).tail == *(req_p)) \ + (queue).tail = prev_req; \ + MPIDI_POSIX_REQUEST(*(req_p))->next = NULL; \ +} + +#define MPIDI_POSIX_REQUEST_DEQUEUE_AND_SET_ERROR(req_p,prev_req,queue,err) \ +{ \ + MPIR_Request *next = MPIDI_POSIX_REQUEST(*(req_p))->next; \ + if ((queue).head == *(req_p)) \ + (queue).head = next; \ + else \ + MPIDI_POSIX_REQUEST(prev_req)->next = next; \ + if ((queue).tail == *(req_p)) \ + (queue).tail = prev_req; \ + (*(req_p))->status.MPI_ERROR = err; \ + MPIDI_POSIX_REQUEST_COMPLETE(*(req_p)); \ + *(req_p) = next; \ +} + +#define MPIDI_POSIX_REQUEST_CREATE_SREQ(sreq_) \ +{ \ + (sreq_) = MPIR_Request_create(MPIR_REQUEST_KIND__SEND); \ + MPIR_Request_add_ref((sreq_)); \ + (sreq_)->u.persist.real_request = NULL; \ +} + +#define MPIDI_POSIX_REQUEST_CREATE_RREQ(rreq_) \ +{ \ + (rreq_) = MPIR_Request_create(MPIR_REQUEST_KIND__RECV); \ + MPIR_Request_add_ref((rreq_)); \ + (rreq_)->u.persist.real_request = NULL; \ +} + +/* ---------------------------------------------------- */ +/* matching macros */ +/* ---------------------------------------------------- */ +#define MPIDI_POSIX_ENVELOPE_SET(ptr_,rank_,tag_,context_id_) \ +{ \ + (ptr_)->rank = rank_; \ + (ptr_)->tag = tag_; \ + (ptr_)->context_id = context_id_; \ +} + +#define MPIDI_POSIX_ENVELOPE_GET(ptr_,rank_,tag_,context_id_) \ +{ \ + rank_ = (ptr_)->rank; \ + tag_ = (ptr_)->tag; \ + context_id_ = (ptr_)->context_id; \ +} + +#define MPIDI_POSIX_ENVELOPE_MATCH(ptr_,rank_,tag_,context_id_) \ + (((ptr_)->rank == (rank_) || (rank_) == MPI_ANY_SOURCE) && \ + ((ptr_)->tag == (tag_) || (tag_) == MPI_ANY_TAG) && \ + (ptr_)->context_id == (context_id_)) + +/* + * Helper routines and macros for request completion + */ +#define DECL_FUNC(FUNCNAME) MPL_QUOTE(FUNCNAME) + +#undef FUNCNAME +#define FUNCNAME nothing +#define BEGIN_FUNC(FUNCNAME) \ + MPIR_FUNC_VERBOSE_STATE_DECL(FUNCNAME); \ + MPIR_FUNC_VERBOSE_ENTER(FUNCNAME); +#define END_FUNC(FUNCNAME) \ + MPIR_FUNC_VERBOSE_EXIT(FUNCNAME); +#define END_FUNC_RC(FUNCNAME) \ + fn_exit: \ + MPIR_FUNC_VERBOSE_EXIT(FUNCNAME); \ + return mpi_errno; \ +fn_fail: \ + goto fn_exit; + +#define __SHORT_FILE__ \ + (strrchr(__FILE__,'/') \ + ? strrchr(__FILE__,'/')+1 \ + : __FILE__ \ +) + +int MPIDI_POSIX_barrier_vars_init(MPIDI_POSIX_barrier_vars_t * barrier_region); +extern MPIDI_POSIX_request_queue_t MPIDI_POSIX_sendq; +extern MPIDI_POSIX_request_queue_t MPIDI_POSIX_recvq_unexpected; +extern MPIDI_POSIX_request_queue_t MPIDI_POSIX_recvq_posted; + + + +#endif /* SHM_POSIX_IMPL_H_INCLUDED */ diff --git a/src/mpid/ch4/shm/posix/posix_init.h b/src/mpid/ch4/shm/posix/posix_init.h new file mode 100644 index 0000000..30723a6 --- /dev/null +++ b/src/mpid/ch4/shm/posix/posix_init.h @@ -0,0 +1,324 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef SHM_POSIX_INIT_H_INCLUDED +#define SHM_POSIX_INIT_H_INCLUDED + +#include "posix_impl.h" +#include "ch4_types.h" +#include "mpidu_shm.h" + +/* ------------------------------------------------------- */ +/* from mpid/ch3/channels/nemesis/src/mpid_nem_init.c */ +/* ------------------------------------------------------- */ +extern MPIDI_POSIX_mem_region_t MPIDI_POSIX_mem_region; +extern char *MPIDI_POSIX_asym_base_addr; + +#undef FCNAME +#define FCNAME DECL_FUNC(MPIDI_SHM_init) +static inline int MPIDI_SHM_init(int rank, int size) +{ + int mpi_errno = MPI_SUCCESS; + int num_local = 0; + int local_rank = -1; + int *local_procs = NULL; + int *local_ranks = NULL; + int i; + int grank; + MPIDI_POSIX_fastbox_t *fastboxes_p = NULL; + MPIDI_POSIX_cell_t(*cells_p)[MPIDI_POSIX_NUM_CELLS]; + MPIDI_POSIX_queue_t *recv_queues_p = NULL; + MPIDI_POSIX_queue_t *free_queues_p = NULL; + MPIR_CHKPMEM_DECL(9); + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDI_SHM_INIT); + + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDI_SHM_INIT); + + MPIDI_POSIX_mem_region.num_seg = 1; + MPIR_CHKPMEM_MALLOC(MPIDI_POSIX_mem_region.seg, MPIDU_shm_seg_info_ptr_t, + MPIDI_POSIX_mem_region.num_seg * sizeof(MPIDU_shm_seg_info_t), mpi_errno, + "mem_region segments"); + MPIR_CHKPMEM_MALLOC(local_procs, int *, size * sizeof(int), mpi_errno, + "local process index array"); + MPIR_CHKPMEM_MALLOC(local_ranks, int *, size * sizeof(int), mpi_errno, + "mem_region local ranks"); + + for (i = 0; i < size; i++) { + if (MPIDI_CH4_rank_is_local(i, MPIR_Process.comm_world)) { + if (i == rank) { + local_rank = num_local; + } + + local_procs[num_local] = i; + local_ranks[i] = num_local; + num_local++; + } + } + + MPIDI_POSIX_mem_region.rank = rank; + MPIDI_POSIX_mem_region.num_local = num_local; + MPIDI_POSIX_mem_region.num_procs = size; + MPIDI_POSIX_mem_region.local_procs = local_procs; + MPIDI_POSIX_mem_region.local_ranks = local_ranks; + MPIDI_POSIX_mem_region.local_rank = local_rank; + MPIDI_POSIX_mem_region.next = NULL; + + /* Request fastboxes region */ + mpi_errno = + MPIDU_shm_seg_alloc(MAX + ((num_local * ((num_local - 1) * sizeof(MPIDI_POSIX_fastbox_t))), + MPIDI_POSIX_ASYMM_NULL_VAL), (void **) &fastboxes_p); + + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + /* Request data cells region */ + mpi_errno = + MPIDU_shm_seg_alloc(num_local * MPIDI_POSIX_NUM_CELLS * sizeof(MPIDI_POSIX_cell_t), + (void **) &cells_p); + + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + /* Request free q region */ + mpi_errno = + MPIDU_shm_seg_alloc(num_local * sizeof(MPIDI_POSIX_queue_t), (void **) &free_queues_p); + + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + /* Request recv q region */ + mpi_errno = + MPIDU_shm_seg_alloc(num_local * sizeof(MPIDI_POSIX_queue_t), (void **) &recv_queues_p); + + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + /* Request shared collectives barrier vars region */ + mpi_errno = + MPIDU_shm_seg_alloc(MPIDI_POSIX_NUM_BARRIER_VARS * sizeof(MPIDI_POSIX_barrier_vars_t), + (void **) &MPIDI_POSIX_mem_region.barrier_vars); + + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + /* Actually allocate the segment and assign regions to the pointers */ + mpi_errno = + MPIDU_shm_seg_commit(&MPIDI_POSIX_mem_region.memory, &MPIDI_POSIX_mem_region.barrier, + num_local, local_rank, MPIDI_POSIX_mem_region.local_procs[0], + MPIDI_POSIX_mem_region.rank); + + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + /* post check_alloc steps */ + if (MPIDI_POSIX_mem_region.memory.symmetrical == 1) { + MPIDI_POSIX_asym_base_addr = NULL; + } + else { + MPIDI_POSIX_asym_base_addr = MPIDI_POSIX_mem_region.memory.base_addr; +#ifdef MPIDI_POSIX_SYMMETRIC_QUEUES + MPIR_ERR_INTERNALANDJUMP(mpi_errno, "queues are not symmetrically allocated as expected"); +#endif + } + + /* init shared collectives barrier region */ + mpi_errno = MPIDI_POSIX_barrier_vars_init(MPIDI_POSIX_mem_region.barrier_vars); + + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + /* local procs barrier */ + mpi_errno = MPIDU_shm_barrier(MPIDI_POSIX_mem_region.barrier, num_local); + + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + /* find our cell region */ + MPIDI_POSIX_mem_region.Elements = cells_p[local_rank]; + + /* Tables of pointers to shared memory Qs */ + MPIR_CHKPMEM_MALLOC(MPIDI_POSIX_mem_region.FreeQ, MPIDI_POSIX_queue_ptr_t *, + size * sizeof(MPIDI_POSIX_queue_ptr_t), mpi_errno, "FreeQ"); + MPIR_CHKPMEM_MALLOC(MPIDI_POSIX_mem_region.RecvQ, MPIDI_POSIX_queue_ptr_t *, + size * sizeof(MPIDI_POSIX_queue_ptr_t), mpi_errno, "RecvQ"); + + /* Init table entry for our Qs */ + MPIDI_POSIX_mem_region.FreeQ[rank] = &free_queues_p[local_rank]; + MPIDI_POSIX_mem_region.RecvQ[rank] = &recv_queues_p[local_rank]; + + /* Init our queues */ + MPIDI_POSIX_queue_init(MPIDI_POSIX_mem_region.RecvQ[rank]); + MPIDI_POSIX_queue_init(MPIDI_POSIX_mem_region.FreeQ[rank]); + + /* Init and enqueue our free cells */ + for (i = 0; i < MPIDI_POSIX_NUM_CELLS; ++i) { + MPIDI_POSIX_cell_init(&(MPIDI_POSIX_mem_region.Elements[i]), rank); + MPIDI_POSIX_queue_enqueue(MPIDI_POSIX_mem_region.FreeQ[rank], + &(MPIDI_POSIX_mem_region.Elements[i])); + } + + /* set route for local procs through shmem */ + for (i = 0; i < num_local; i++) { + grank = local_procs[i]; + MPIDI_POSIX_mem_region.FreeQ[grank] = &free_queues_p[i]; + MPIDI_POSIX_mem_region.RecvQ[grank] = &recv_queues_p[i]; + + MPIR_Assert(MPIDI_POSIX_ALIGNED + (MPIDI_POSIX_mem_region.FreeQ[grank], MPIDI_POSIX_CACHE_LINE_LEN)); + MPIR_Assert(MPIDI_POSIX_ALIGNED + (MPIDI_POSIX_mem_region.RecvQ[grank], MPIDI_POSIX_CACHE_LINE_LEN)); + } + + /* make pointers to our queues global so we don't have to dereference the array */ + MPIDI_POSIX_mem_region.my_freeQ = MPIDI_POSIX_mem_region.FreeQ[rank]; + MPIDI_POSIX_mem_region.my_recvQ = MPIDI_POSIX_mem_region.RecvQ[rank]; + + /* local barrier */ + mpi_errno = MPIDU_shm_barrier(MPIDI_POSIX_mem_region.barrier, num_local); + + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + /* Allocate table of pointers to fastboxes */ + MPIR_CHKPMEM_MALLOC(MPIDI_POSIX_mem_region.mailboxes.in, MPIDI_POSIX_fastbox_t **, + num_local * sizeof(MPIDI_POSIX_fastbox_t *), mpi_errno, "fastboxes"); + MPIR_CHKPMEM_MALLOC(MPIDI_POSIX_mem_region.mailboxes.out, MPIDI_POSIX_fastbox_t **, + num_local * sizeof(MPIDI_POSIX_fastbox_t *), mpi_errno, "fastboxes"); + + MPIR_Assert(num_local > 0); + +#define MPIDI_POSIX_MAILBOX_INDEX(sender, receiver) (((sender) > (receiver)) ? ((num_local-1) * (sender) + (receiver)) : \ + (((sender) < (receiver)) ? ((num_local-1) * (sender) + ((receiver)-1)) : 0)) + + /* fill in tables */ + for (i = 0; i < num_local; ++i) { + if (i == local_rank) { + /* No fastboxs to myself */ + MPIDI_POSIX_mem_region.mailboxes.in[i] = NULL; + MPIDI_POSIX_mem_region.mailboxes.out[i] = NULL; + } + else { + MPIDI_POSIX_mem_region.mailboxes.in[i] = + &fastboxes_p[MPIDI_POSIX_MAILBOX_INDEX(i, local_rank)]; + MPIDI_POSIX_mem_region.mailboxes.out[i] = + &fastboxes_p[MPIDI_POSIX_MAILBOX_INDEX(local_rank, i)]; + OPA_store_int(&MPIDI_POSIX_mem_region.mailboxes.in[i]->common.flag.value, 0); + OPA_store_int(&MPIDI_POSIX_mem_region.mailboxes.out[i]->common.flag.value, 0); + } + } + +#undef MPIDI_POSIX_MAILBOX_INDEX + + MPIR_CHKPMEM_COMMIT(); + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDI_SHM_INIT); + return mpi_errno; + fn_fail: + /* --BEGIN ERROR HANDLING-- */ + MPIR_CHKPMEM_REAP(); + goto fn_exit; + /* --END ERROR HANDLING-- */ +} + +#undef FCNAME +#define FCNAME DECL_FUNC(MPIDI_SHM_finalize) +static inline int MPIDI_SHM_finalize(void) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDI_SHM_FINALIZE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDI_SHM_FINALIZE); + + /* local barrier */ + mpi_errno = MPIDU_shm_barrier(MPIDI_POSIX_mem_region.barrier, MPIDI_POSIX_mem_region.num_local); + + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + /* from MPIDI_POSIX_init */ + MPL_free(MPIDI_POSIX_mem_region.FreeQ); + MPL_free(MPIDI_POSIX_mem_region.RecvQ); + MPL_free(MPIDI_POSIX_mem_region.local_ranks); + MPL_free(MPIDI_POSIX_mem_region.seg); + MPL_free(MPIDI_POSIX_mem_region.mailboxes.out); + MPL_free(MPIDI_POSIX_mem_region.mailboxes.in); + MPL_free(MPIDI_POSIX_mem_region.local_procs); + + mpi_errno = + MPIDU_shm_seg_destroy(&MPIDI_POSIX_mem_region.memory, MPIDI_POSIX_mem_region.num_local); + + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDI_SHM_FINALIZE); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +static inline void *MPIDI_SHM_alloc_mem(size_t size, MPIR_Info * info_ptr) +{ + MPIR_Assert(0); + return NULL; +} + +static inline int MPIDI_SHM_free_mem(void *ptr) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_comm_get_lpid(MPIR_Comm * comm_ptr, + int idx, int *lpid_ptr, MPL_bool is_remote) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_gpid_get(MPIR_Comm * comm_ptr, int rank, MPIR_Gpid * gpid) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_get_node_id(MPIR_Comm * comm, int rank, MPID_Node_id_t * id_p) +{ + *id_p = (MPID_Node_id_t) 0; + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_get_max_node_id(MPIR_Comm * comm, MPID_Node_id_t * max_id_p) +{ + *max_id_p = (MPID_Node_id_t) 1; + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_getallincomm(MPIR_Comm * comm_ptr, + int local_size, MPIR_Gpid local_gpids[], int *singleAVT) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_gpid_tolpidarray(int size, MPIR_Gpid gpid[], int lpid[]) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_create_intercomm_from_lpids(MPIR_Comm * newcomm_ptr, + int size, const int lpids[]) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +#endif /* SHM_POSIX_INIT_H_INCLUDED */ diff --git a/src/mpid/ch4/shm/posix/posix_pre.h b/src/mpid/ch4/shm/posix/posix_pre.h new file mode 100644 index 0000000..cd4cec1 --- /dev/null +++ b/src/mpid/ch4/shm/posix/posix_pre.h @@ -0,0 +1,41 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ + +#ifndef SHM_POSIX_PRE_H_INCLUDED +#define SHM_POSIX_PRE_H_INCLUDED + +#include <mpi.h> + +struct MPIR_Request; +struct MPIDU_Segment; + +typedef struct { + struct MPIR_Request *next; + struct MPIR_Request *pending; + int dest; + int rank; + int tag; + int context_id; + char *user_buf; + size_t data_sz; + int type; + int user_count; + MPI_Datatype datatype; + struct MPIDU_Segment *segment_ptr; + size_t segment_first; + size_t segment_size; +} MPIDI_POSIX_request_t; + +typedef struct { + int dummy; +} MPIDI_POSIX_comm_t; + +#endif /* SHM_POSIX_PRE_H_INCLUDED */ diff --git a/src/mpid/ch4/shm/posix/posix_probe.h b/src/mpid/ch4/shm/posix/posix_probe.h new file mode 100644 index 0000000..5fe8e11 --- /dev/null +++ b/src/mpid/ch4/shm/posix/posix_probe.h @@ -0,0 +1,146 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef SHM_POSIX_PROBE_H_INCLUDED +#define SHM_POSIX_PROBE_H_INCLUDED + +#include "posix_impl.h" + + +static inline int MPIDI_SHM_improbe(int source, + int tag, + MPIR_Comm * comm, + int context_offset, + int *flag, MPIR_Request ** message, MPI_Status * status) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Request *req, *matched_req = NULL; + int count = 0; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDI_SHM_IMPROBE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDI_SHM_IMPROBE); + + MPID_THREAD_CS_ENTER(POBJ, MPIDI_POSIX_SHM_MUTEX); + *message = NULL; + + if (unlikely(source == MPI_PROC_NULL)) { + MPIR_Status_set_procnull(status); + *flag = true; + goto fn_exit; + } + + for (req = MPIDI_POSIX_recvq_unexpected.head; req; req = MPIDI_POSIX_REQUEST(req)->next) { + if (MPIDI_POSIX_ENVELOPE_MATCH + (MPIDI_POSIX_REQUEST(req), source, tag, comm->recvcontext_id + context_offset)) { + if (!matched_req) + matched_req = req; + + if (req && MPIDI_POSIX_REQUEST(req)->type == MPIDI_POSIX_TYPEEAGER) { + *message = matched_req; + break; + } + } + } + + if (*message) { + MPIDI_POSIX_request_queue_t mqueue = { NULL, NULL }; + MPIR_Request *prev_req = NULL, *next_req = NULL; + req = MPIDI_POSIX_recvq_unexpected.head; + + while (req) { + next_req = MPIDI_POSIX_REQUEST(req)->next; + + if (MPIDI_POSIX_ENVELOPE_MATCH + (MPIDI_POSIX_REQUEST(req), source, tag, comm->recvcontext_id + context_offset)) { + if (mqueue.head == NULL) + MPIR_Assert(req == matched_req); + + count += MPIR_STATUS_GET_COUNT(req->status); + MPIDI_POSIX_REQUEST_DEQUEUE(&req, prev_req, MPIDI_POSIX_recvq_unexpected); + MPIDI_POSIX_REQUEST_ENQUEUE(req, mqueue); + + if (req && MPIDI_POSIX_REQUEST(req)->type == MPIDI_POSIX_TYPEEAGER) + break; + } + else + prev_req = req; + + req = next_req; + } + + *flag = 1; + matched_req->kind = MPIR_REQUEST_KIND__MPROBE; + matched_req->comm = comm; + MPIR_Comm_add_ref(comm); + status->MPI_TAG = matched_req->status.MPI_TAG; + status->MPI_SOURCE = matched_req->status.MPI_SOURCE; + MPIR_STATUS_SET_COUNT(*status, count); + } + else { + *flag = 0; + MPIDI_Progress_test(); + } + + fn_exit: + MPID_THREAD_CS_EXIT(POBJ, MPIDI_POSIX_SHM_MUTEX); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDI_SHM_IMPROBE); + return mpi_errno; +} + +static inline int MPIDI_SHM_iprobe(int source, + int tag, + MPIR_Comm * comm, + int context_offset, int *flag, MPI_Status * status) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Request *req, *matched_req = NULL; + int count = 0; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDI_SHM_IPROBE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDI_SHM_IPROBE); + MPID_THREAD_CS_ENTER(POBJ, MPIDI_POSIX_SHM_MUTEX); + + if (unlikely(source == MPI_PROC_NULL)) { + MPIR_Status_set_procnull(status); + *flag = true; + goto fn_exit; + } + + for (req = MPIDI_POSIX_recvq_unexpected.head; req; req = MPIDI_POSIX_REQUEST(req)->next) { + if (MPIDI_POSIX_ENVELOPE_MATCH + (MPIDI_POSIX_REQUEST(req), source, tag, comm->recvcontext_id + context_offset)) { + count += MPIR_STATUS_GET_COUNT(req->status); + + if (MPIDI_POSIX_REQUEST(req)->type == MPIDI_POSIX_TYPEEAGER) { + matched_req = req; + break; + } + } + } + + if (matched_req) { + *flag = 1; + status->MPI_TAG = matched_req->status.MPI_TAG; + status->MPI_SOURCE = matched_req->status.MPI_SOURCE; + MPIR_STATUS_SET_COUNT(*status, count); + } + else { + *flag = 0; + MPID_THREAD_CS_EXIT(POBJ, MPIDI_POSIX_SHM_MUTEX); + MPIDI_Progress_test(); + MPID_THREAD_CS_ENTER(POBJ, MPIDI_POSIX_SHM_MUTEX); + } + + fn_exit: + MPID_THREAD_CS_EXIT(POBJ, MPIDI_POSIX_SHM_MUTEX); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDI_SHM_IPROBE); + return mpi_errno; +} + +#endif /* SHM_POSIX_PROBE_H_INCLUDED */ diff --git a/src/mpid/ch4/shm/posix/posix_progress.h b/src/mpid/ch4/shm/posix/posix_progress.h new file mode 100644 index 0000000..4732f1b --- /dev/null +++ b/src/mpid/ch4/shm/posix/posix_progress.h @@ -0,0 +1,454 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef SHM_POSIX_PROGRESS_H_INCLUDED +#define SHM_POSIX_PROGRESS_H_INCLUDED + +#include "posix_impl.h" + +/* ----------------------------------------------------- */ +/* MPIDI_POSIX_progress_recv */ +/* ----------------------------------------------------- */ +#undef FCNAME +#define FCNAME DECL_FUNC(MPIDI_POSIX_progress_recv) +static inline int MPIDI_POSIX_progress_recv(int blocking, int *completion_count) +{ + int mpi_errno = MPI_SUCCESS; + size_t data_sz; + int in_cell = 0; + MPIDI_POSIX_cell_ptr_t cell = NULL; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDI_SHM_DO_PROGRESS_RECV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDI_SHM_DO_PROGRESS_RECV); + /* try to match with unexpected */ + MPIR_Request *sreq = MPIDI_POSIX_recvq_unexpected.head; + MPIR_Request *prev_sreq = NULL; + unexpected_l: + + if (sreq != NULL) { + goto match_l; + } + + /* try to receive from recvq */ + if (MPIDI_POSIX_mem_region.my_recvQ && + !MPIDI_POSIX_queue_empty(MPIDI_POSIX_mem_region.my_recvQ)) { + MPIDI_POSIX_queue_dequeue(MPIDI_POSIX_mem_region.my_recvQ, &cell); + in_cell = 1; + goto match_l; + } + + goto fn_exit; + match_l:{ + /* traverse posted receive queue */ + MPIR_Request *req = MPIDI_POSIX_recvq_posted.head; + MPIR_Request *prev_req = NULL; + int continue_matching = 1; + char *send_buffer = + in_cell ? (char *) cell->pkt.mpich.p.payload : (char *) MPIDI_POSIX_REQUEST(sreq)-> + user_buf; + int type = in_cell ? cell->pkt.mpich.type : MPIDI_POSIX_REQUEST(sreq)->type; + MPIR_Request *pending = in_cell ? cell->pending : MPIDI_POSIX_REQUEST(sreq)->pending; + + if (type == MPIDI_POSIX_TYPEACK) { + /* ACK message doesn't have a matching receive! */ + int c; + MPIR_Assert(in_cell); + MPIR_Assert(pending); + MPIR_cc_decr(pending->cc_ptr, &c); + MPIDI_CH4U_request_release(pending); + goto release_cell_l; + } + + while (req) { + int sender_rank, tag, context_id; + MPI_Count count; + MPIDI_POSIX_ENVELOPE_GET(MPIDI_POSIX_REQUEST(req), sender_rank, tag, context_id); + MPL_DBG_MSG_FMT(MPIR_DBG_HANDLE, TYPICAL, + (MPL_DBG_FDEST, "Posted from grank %d to %d in progress %d,%d,%d\n", + MPIDI_CH4U_rank_to_lpid(sender_rank, req->comm), + MPIDI_POSIX_mem_region.rank, sender_rank, tag, context_id)); + + if ((in_cell && MPIDI_POSIX_ENVELOPE_MATCH(cell, sender_rank, tag, context_id)) || + (sreq && + MPIDI_POSIX_ENVELOPE_MATCH(MPIDI_POSIX_REQUEST(sreq), sender_rank, tag, + context_id))) { + + /* Request matched */ + + continue_matching = 1; + + if (MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(req)) { + MPIDI_CH4R_anysource_matched(MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(req), + MPIDI_CH4R_SHM, &continue_matching); + MPIDI_CH4U_request_release(MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(req)); + + /* Decouple requests */ + MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(req)) + = NULL; + MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(req) = NULL; + + if (continue_matching) + break; + } + + char *recv_buffer = (char *) MPIDI_POSIX_REQUEST(req)->user_buf; + + if (pending) { + /* we must send ACK */ + int srank = in_cell ? cell->rank : sreq->status.MPI_SOURCE; + MPIR_Request *req_ack = NULL; + MPIDI_POSIX_REQUEST_CREATE_SREQ(req_ack); + MPIR_Object_set_ref(req_ack, 1); + req_ack->comm = req->comm; + MPIR_Comm_add_ref(req->comm); + + MPIDI_POSIX_ENVELOPE_SET(MPIDI_POSIX_REQUEST(req_ack), req->comm->rank, tag, + context_id); + MPIDI_POSIX_REQUEST(req_ack)->user_buf = NULL; + MPIDI_POSIX_REQUEST(req_ack)->user_count = 0; + MPIDI_POSIX_REQUEST(req_ack)->datatype = MPI_BYTE; + MPIDI_POSIX_REQUEST(req_ack)->data_sz = 0; + MPIDI_POSIX_REQUEST(req_ack)->type = MPIDI_POSIX_TYPEACK; + MPIDI_POSIX_REQUEST(req_ack)->dest = srank; + MPIDI_POSIX_REQUEST(req_ack)->next = NULL; + MPIDI_POSIX_REQUEST(req_ack)->segment_ptr = NULL; + MPIDI_POSIX_REQUEST(req_ack)->pending = pending; + /* enqueue req_ack */ + MPIDI_POSIX_REQUEST_ENQUEUE(req_ack, MPIDI_POSIX_sendq); + } + + if (type == MPIDI_POSIX_TYPEEAGER) + /* eager message */ + data_sz = + in_cell ? cell->pkt.mpich.datalen : MPIDI_POSIX_REQUEST(sreq)->data_sz; + else if (type == MPIDI_POSIX_TYPELMT) + data_sz = MPIDI_POSIX_EAGER_THRESHOLD; + else { + data_sz = 0; /* unused warning */ + MPIR_Assert(0); + } + /* check for user buffer overflow */ + size_t user_data_sz = MPIDI_POSIX_REQUEST(req)->data_sz; + if (user_data_sz < data_sz) { + req->status.MPI_ERROR = MPI_ERR_TRUNCATE; + data_sz = user_data_sz; + } + + /* copy to user buffer */ + if (MPIDI_POSIX_REQUEST(req)->segment_ptr) { + /* non-contig */ + size_t last = MPIDI_POSIX_REQUEST(req)->segment_first + data_sz; + MPID_Segment_unpack(MPIDI_POSIX_REQUEST(req)->segment_ptr, + MPIDI_POSIX_REQUEST(req)->segment_first, + (MPI_Aint *) & last, send_buffer); + if (last != MPIDI_POSIX_REQUEST(req)->segment_first + data_sz) + req->status.MPI_ERROR = MPI_ERR_TYPE; + if (type == MPIDI_POSIX_TYPEEAGER) + MPID_Segment_free(MPIDI_POSIX_REQUEST(req)->segment_ptr); + else + MPIDI_POSIX_REQUEST(req)->segment_first = last; + } + else + /* contig */ + if (send_buffer) + MPIR_Memcpy(recv_buffer, (void *) send_buffer, data_sz); + MPIDI_POSIX_REQUEST(req)->data_sz -= data_sz; + MPIDI_POSIX_REQUEST(req)->user_buf += data_sz; + + /* set status and dequeue receive request if done */ + count = MPIR_STATUS_GET_COUNT(req->status) + (MPI_Count) data_sz; + MPIR_STATUS_SET_COUNT(req->status, count); + if (type == MPIDI_POSIX_TYPEEAGER) { + if (in_cell) { + req->status.MPI_SOURCE = cell->rank; + req->status.MPI_TAG = cell->tag; + } + else { + req->status.MPI_SOURCE = sreq->status.MPI_SOURCE; + req->status.MPI_TAG = sreq->status.MPI_TAG; + } + MPIDI_POSIX_REQUEST_DEQUEUE_AND_SET_ERROR(&req, prev_req, + MPIDI_POSIX_recvq_posted, + req->status.MPI_ERROR); + } + + goto release_cell_l; + } /* if matched */ + + prev_req = req; + req = MPIDI_POSIX_REQUEST(req)->next; + } + + /* unexpected message, no posted matching req */ + if (in_cell) { + /* free the cell, move to unexpected queue */ + MPIR_Request *rreq; + MPIDI_POSIX_REQUEST_CREATE_RREQ(rreq); + MPIR_Object_set_ref(rreq, 1); + /* set status */ + rreq->status.MPI_SOURCE = cell->rank; + rreq->status.MPI_TAG = cell->tag; + MPIR_STATUS_SET_COUNT(rreq->status, cell->pkt.mpich.datalen); + MPIDI_POSIX_ENVELOPE_SET(MPIDI_POSIX_REQUEST(rreq), cell->rank, cell->tag, + cell->context_id); + data_sz = cell->pkt.mpich.datalen; + MPIDI_POSIX_REQUEST(rreq)->data_sz = data_sz; + MPIDI_POSIX_REQUEST(rreq)->type = cell->pkt.mpich.type; + + if (data_sz > 0) { + MPIDI_POSIX_REQUEST(rreq)->user_buf = (char *) MPL_malloc(data_sz); + MPIR_Memcpy(MPIDI_POSIX_REQUEST(rreq)->user_buf, (void *) cell->pkt.mpich.p.payload, + data_sz); + } + else { + MPIDI_POSIX_REQUEST(rreq)->user_buf = NULL; + } + + MPIDI_POSIX_REQUEST(rreq)->datatype = MPI_BYTE; + MPIDI_POSIX_REQUEST(rreq)->next = NULL; + MPIDI_POSIX_REQUEST(rreq)->pending = cell->pending; + /* enqueue rreq */ + MPIDI_POSIX_REQUEST_ENQUEUE(rreq, MPIDI_POSIX_recvq_unexpected); + MPL_DBG_MSG_FMT(MPIR_DBG_HANDLE, TYPICAL, + (MPL_DBG_FDEST, "Unexpected from grank %d to %d in progress %d,%d,%d\n", + cell->my_rank, MPIDI_POSIX_mem_region.rank, + cell->rank, cell->tag, cell->context_id)); + } + else { + /* examine another message in unexpected queue */ + prev_sreq = sreq; + sreq = MPIDI_POSIX_REQUEST(sreq)->next; + goto unexpected_l; + } + } + release_cell_l: + + if (in_cell) { + /* release cell */ + MPL_DBG_MSG_FMT(MPIR_DBG_HANDLE, TYPICAL, + (MPL_DBG_FDEST, "Received from grank %d to %d in progress %d,%d,%d\n", + cell->my_rank, MPIDI_POSIX_mem_region.rank, cell->rank, cell->tag, + cell->context_id)); + cell->pending = NULL; + { + MPIDI_POSIX_queue_enqueue(MPIDI_POSIX_mem_region.FreeQ[cell->my_rank], cell); + } + } + else { + /* destroy unexpected req */ + MPIDI_POSIX_REQUEST(sreq)->pending = NULL; + MPL_free(MPIDI_POSIX_REQUEST(sreq)->user_buf); + MPIDI_POSIX_REQUEST_DEQUEUE_AND_SET_ERROR(&sreq, prev_sreq, MPIDI_POSIX_recvq_unexpected, + mpi_errno); + } + + (*completion_count)++; + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDI_SHM_DO_PROGRESS_RECV); + return mpi_errno; +} + +/* ----------------------------------------------------- */ +/* MPIDI_POSIX_progress_send */ +/* ----------------------------------------------------- */ +#undef FCNAME +#define FCNAME DECL_FUNC(MPIDI_POSIX_progress_send) +static inline int MPIDI_POSIX_progress_send(int blocking, int *completion_count) +{ + int mpi_errno = MPI_SUCCESS; + int dest; + MPIDI_POSIX_cell_ptr_t cell = NULL; + MPIR_Request *sreq = MPIDI_POSIX_sendq.head; + MPIR_Request *prev_sreq = NULL; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDI_SHM_DO_PROGRESS_SEND); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDI_SHM_DO_PROGRESS_SEND); + + if (sreq == NULL) + goto fn_exit; + + /* try to send via freeq */ + if (!MPIDI_POSIX_queue_empty(MPIDI_POSIX_mem_region.my_freeQ)) { + MPIDI_POSIX_queue_dequeue(MPIDI_POSIX_mem_region.my_freeQ, &cell); + MPIDI_POSIX_ENVELOPE_GET(MPIDI_POSIX_REQUEST(sreq), cell->rank, cell->tag, + cell->context_id); + dest = MPIDI_POSIX_REQUEST(sreq)->dest; + char *recv_buffer = (char *) cell->pkt.mpich.p.payload; + size_t data_sz = MPIDI_POSIX_REQUEST(sreq)->data_sz; + /* + * TODO: make request field dest_lpid (or even recvQ[dest_lpid]) instead of dest - no need to do rank_to_lpid each time + */ + int grank = MPIDI_CH4U_rank_to_lpid(dest, sreq->comm); + cell->pending = NULL; + + if (MPIDI_POSIX_REQUEST(sreq)->type == MPIDI_POSIX_TYPESYNC) { + /* increase req cc in order to release req only after ACK, do it once per SYNC request */ + /* non-NULL pending req signal receiver about sending ACK back */ + /* the pending req should be sent back for sender to decrease cc, for it is dequeued already */ + int c; + cell->pending = sreq; + MPIR_cc_incr(sreq->cc_ptr, &c); + MPIDI_POSIX_REQUEST(sreq)->type = MPIDI_POSIX_TYPESTANDARD; + } + + if (data_sz <= MPIDI_POSIX_EAGER_THRESHOLD) { + cell->pkt.mpich.datalen = data_sz; + + if (MPIDI_POSIX_REQUEST(sreq)->type == MPIDI_POSIX_TYPEACK) { + cell->pkt.mpich.type = MPIDI_POSIX_TYPEACK; + cell->pending = MPIDI_POSIX_REQUEST(sreq)->pending; + } + else { + /* eager message */ + if (MPIDI_POSIX_REQUEST(sreq)->segment_ptr) { + /* non-contig */ + MPID_Segment_pack(MPIDI_POSIX_REQUEST(sreq)->segment_ptr, + MPIDI_POSIX_REQUEST(sreq)->segment_first, + (MPI_Aint *) & MPIDI_POSIX_REQUEST(sreq)->segment_size, + recv_buffer); + MPID_Segment_free(MPIDI_POSIX_REQUEST(sreq)->segment_ptr); + } + else { + /* contig */ + MPIR_Memcpy((void *) recv_buffer, MPIDI_POSIX_REQUEST(sreq)->user_buf, data_sz); + } + + cell->pkt.mpich.type = MPIDI_POSIX_TYPEEAGER; + /* set status */ + /* + * TODO: incorrect count for LMT - set to a last chunk of data + * is send status required? + */ + sreq->status.MPI_SOURCE = cell->rank; + sreq->status.MPI_TAG = cell->tag; + MPIR_STATUS_SET_COUNT(sreq->status, data_sz); + } + + /* dequeue sreq */ + MPIDI_POSIX_REQUEST_DEQUEUE_AND_SET_ERROR(&sreq, prev_sreq, MPIDI_POSIX_sendq, + mpi_errno); + } + else { + /* long message */ + if (MPIDI_POSIX_REQUEST(sreq)->segment_ptr) { + /* non-contig */ + size_t last = + MPIDI_POSIX_REQUEST(sreq)->segment_first + MPIDI_POSIX_EAGER_THRESHOLD; + MPID_Segment_pack(MPIDI_POSIX_REQUEST(sreq)->segment_ptr, + MPIDI_POSIX_REQUEST(sreq)->segment_first, (MPI_Aint *) & last, + recv_buffer); + MPIDI_POSIX_REQUEST(sreq)->segment_first = last; + } + else { + /* contig */ + MPIR_Memcpy((void *) recv_buffer, MPIDI_POSIX_REQUEST(sreq)->user_buf, + MPIDI_POSIX_EAGER_THRESHOLD); + MPIDI_POSIX_REQUEST(sreq)->user_buf += MPIDI_POSIX_EAGER_THRESHOLD; + } + + cell->pkt.mpich.datalen = MPIDI_POSIX_EAGER_THRESHOLD; + MPIDI_POSIX_REQUEST(sreq)->data_sz -= MPIDI_POSIX_EAGER_THRESHOLD; + cell->pkt.mpich.type = MPIDI_POSIX_TYPELMT; + } + + MPL_DBG_MSG_FMT(MPIR_DBG_HANDLE, TYPICAL, + (MPL_DBG_FDEST, "Sent to grank %d from %d in progress %d,%d,%d\n", grank, + cell->my_rank, cell->rank, cell->tag, cell->context_id)); + MPIDI_POSIX_queue_enqueue(MPIDI_POSIX_mem_region.RecvQ[grank], cell); + (*completion_count)++; + } + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDI_SHM_DO_PROGRESS_SEND); + return mpi_errno; +} + +#undef FCNAME +#define FCNAME DECL_FUNC(MPIDI_SHM_progress) +static inline int MPIDI_SHM_progress(int blocking) +{ + int complete = 0; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDI_SHM_PROGRESS); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDI_SHM_PROGRESS); + + do { + /* Receieve progress */ + MPID_THREAD_CS_ENTER(POBJ, MPIDI_POSIX_SHM_MUTEX); + MPIDI_POSIX_progress_recv(blocking, &complete); + MPID_THREAD_CS_EXIT(POBJ, MPIDI_POSIX_SHM_MUTEX); + /* Send progress */ + MPID_THREAD_CS_ENTER(POBJ, MPIDI_POSIX_SHM_MUTEX); + MPIDI_POSIX_progress_send(blocking, &complete); + MPID_THREAD_CS_EXIT(POBJ, MPIDI_POSIX_SHM_MUTEX); + + MPID_THREAD_CS_EXIT(GLOBAL, MPIR_THREAD_GLOBAL_ALLFUNC_MUTEX); + MPID_THREAD_CS_ENTER(GLOBAL, MPIR_THREAD_GLOBAL_ALLFUNC_MUTEX); + + if (complete > 0) + break; + } while (blocking); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDI_SHM_PROGRESS); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_progress_test(void) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_progress_poke(void) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline void MPIDI_SHM_progress_start(MPID_Progress_state * state) +{ + MPIR_Assert(0); + return; +} + +static inline void MPIDI_SHM_progress_end(MPID_Progress_state * state) +{ + MPIR_Assert(0); + return; +} + +static inline int MPIDI_SHM_progress_wait(MPID_Progress_state * state) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_progress_register(int (*progress_fn) (int *)) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_progress_deregister(int id) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_progress_activate(int id) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_progress_deactivate(int id) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +#endif /* SHM_POSIX_PROGRESS_H_INCLUDED */ diff --git a/src/mpid/ch4/shm/posix/posix_queue.h b/src/mpid/ch4/shm/posix/posix_queue.h new file mode 100644 index 0000000..be68f48 --- /dev/null +++ b/src/mpid/ch4/shm/posix/posix_queue.h @@ -0,0 +1,343 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2016 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef SHM_QUEUE_H +#define SHM_QUEUE_H + +/* ------------------------------------------------------- */ +/* from mpid/ch3/channels/nemesis/include/mpid_nem_debug.h */ +/* ------------------------------------------------------- */ + +/*#define MPIDI_POSIX_YIELD_IN_SKIP*/ +#ifdef MPIDI_POSIX_YIELD_IN_SKIP +#define MPIDI_POSIX_SKIP MPL_sched_yield() +#warning "MPIDI_POSIX_SKIP is yield" +#else /* MPIDI_POSIX_YIELD_IN_SKIP */ +#define MPIDI_POSIX_SKIP do {} while (0) +/*#warning "MPIDI_POSIX_SKIP is do ...while" */ +#endif /* MPIDI_POSIX_YIELD_IN_SKIP */ + +/* ------------------------------------------------------- */ +/* from mpid/ch3/channels/nemesis/include/mpid_nem_queue.h */ +/* ------------------------------------------------------- */ + +/* Assertion macros for nemesis queues. We don't use the normal + * assertion macros because we don't usually want to assert several + * times per queue operation. These assertions serve more as structured + * comments that can easily transformed into being real assertions */ +#define MPIDI_POSIX_Q_assert(a_) \ + do {/*nothing*/} while (0) + +#undef FUNCNAME +#define FUNCNAME MPIDI_POSIX_cell_init +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline void MPIDI_POSIX_cell_init(MPIDI_POSIX_cell_ptr_t cell, int rank) +{ + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDI_POSIX_CELL_INIT); + + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDI_POSIX_CELL_INIT); + + MPIDI_POSIX_SET_REL_NULL(cell->next); + memset((void *) &cell->pkt, 0, sizeof(MPIDI_POSIX_pkt_header_t)); + cell->my_rank = rank; + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDI_POSIX_CELL_INIT); +} + +#if defined(MPIDI_POSIX_USE_LOCK_FREE_QUEUES) + +#undef FUNCNAME +#define FUNCNAME MPIDI_POSIX_queue_init +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline void MPIDI_POSIX_queue_init(MPIDI_POSIX_queue_ptr_t qhead) +{ + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDI_POSIX_QUEUE_INIT); + + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDI_POSIX_QUEUE_INIT); + + MPIDI_POSIX_SET_REL_NULL(qhead->head); + MPIDI_POSIX_SET_REL_NULL(qhead->my_head); + MPIDI_POSIX_SET_REL_NULL(qhead->tail); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDI_POSIX_QUEUE_INIT); +} + +#define MPIDI_POSIX_USE_SHADOW_HEAD + +static inline MPIDI_POSIX_cell_rel_ptr_t MPIDI_POSIX_SWAP_REL(MPIDI_POSIX_cell_rel_ptr_t * ptr, + MPIDI_POSIX_cell_rel_ptr_t val) +{ + MPIDI_POSIX_cell_rel_ptr_t ret; + OPA_store_ptr(&ret.p, OPA_swap_ptr(&(ptr->p), OPA_load_ptr(&val.p))); + return ret; +} + +/* do a compare-and-swap with MPIDI_POSIX_RELNULL */ +static inline MPIDI_POSIX_cell_rel_ptr_t MPIDI_POSIX_CAS_REL_NULL(MPIDI_POSIX_cell_rel_ptr_t * ptr, + MPIDI_POSIX_cell_rel_ptr_t oldv) +{ + MPIDI_POSIX_cell_rel_ptr_t ret; + OPA_store_ptr(&ret.p, OPA_cas_ptr(&(ptr->p), OPA_load_ptr(&oldv.p), MPIDI_POSIX_REL_NULL)); + return ret; +} + +static inline void MPIDI_POSIX_queue_enqueue(MPIDI_POSIX_queue_ptr_t qhead, + MPIDI_POSIX_cell_ptr_t element) +{ + MPIDI_POSIX_cell_rel_ptr_t r_prev; + MPIDI_POSIX_cell_rel_ptr_t r_element = MPIDI_POSIX_ABS_TO_REL(element); + + /* the _dequeue can break if this does not hold */ + MPIDI_POSIX_Q_assert(MPIDI_POSIX_IS_REL_NULL(element->next)); + + /* Orders payload and e->next=NULL w.r.t. the SWAP, updating head, and + * updating prev->next. We assert e->next==NULL above, but it may have been + * done by us in the preceding _dequeue operation. + * + * The SWAP itself does not need to be ordered w.r.t. the payload because + * the consumer does not directly inspect the tail. But the subsequent + * update to the head or e->next field does need to be ordered w.r.t. the + * payload or the consumer may read incorrect data. */ + OPA_write_barrier(); + + /* enqueue at tail */ + r_prev = MPIDI_POSIX_SWAP_REL(&(qhead->tail), r_element); + + if (MPIDI_POSIX_IS_REL_NULL(r_prev)) { + /* queue was empty, element is the new head too */ + + /* no write barrier needed, we believe atomic SWAP with a control + * dependence (if) will enforce ordering between the SWAP and the head + * assignment */ + qhead->head = r_element; + } + else { + /* queue was not empty, swing old tail's next field to point to + * our element */ + MPIDI_POSIX_Q_assert(MPIDI_POSIX_IS_REL_NULL(MPIDI_POSIX_REL_TO_ABS(r_prev)->next)); + + /* no write barrier needed, we believe atomic SWAP with a control + * dependence (if/else) will enforce ordering between the SWAP and the + * prev->next assignment */ + MPIDI_POSIX_REL_TO_ABS(r_prev)->next = r_element; + } +} + +/* This operation is only safe because this is a single-dequeuer queue impl. + Assumes that MPIDI_POSIX_queue_empty was called immediately prior to fix up any + shadow head issues. */ +static inline MPIDI_POSIX_cell_ptr_t MPIDI_POSIX_queue_head(MPIDI_POSIX_queue_ptr_t qhead) +{ + MPIDI_POSIX_Q_assert(MPIDI_POSIX_IS_REL_NULL(qhead->head)); + return MPIDI_POSIX_REL_TO_ABS(qhead->my_head); +} + +static inline int MPIDI_POSIX_queue_empty(MPIDI_POSIX_queue_ptr_t qhead) +{ + /* outside of this routine my_head and head should never both + * contain a non-null value */ + MPIDI_POSIX_Q_assert(MPIDI_POSIX_IS_REL_NULL(qhead->my_head) || + MPIDI_POSIX_IS_REL_NULL(qhead->head)); + + if (MPIDI_POSIX_IS_REL_NULL(qhead->my_head)) { + /* the order of comparison between my_head and head does not + * matter, no read barrier needed here */ + if (MPIDI_POSIX_IS_REL_NULL(qhead->head)) { + /* both null, nothing in queue */ + return 1; + } + else { + /* shadow head null and head has value, move the value to + * our private shadow head and zero the real head */ + qhead->my_head = qhead->head; + /* no barrier needed, my_head is entirely private to consumer */ + MPIDI_POSIX_SET_REL_NULL(qhead->head); + } + } + + /* the following assertions are present at the beginning of _dequeue: + * MPIDI_POSIX_Q_assert(!MPIDI_POSIX_IS_REL_NULL(qhead->my_head)); + * MPIDI_POSIX_Q_assert(MPIDI_POSIX_IS_REL_NULL(qhead->head)); + */ + return 0; +} + + +/* Gets the head */ +static inline void MPIDI_POSIX_queue_dequeue(MPIDI_POSIX_queue_ptr_t qhead, + MPIDI_POSIX_cell_ptr_t * e) +{ + MPIDI_POSIX_cell_ptr_t _e; + MPIDI_POSIX_cell_rel_ptr_t _r_e; + + /* _empty always called first, moving head-->my_head */ + MPIDI_POSIX_Q_assert(!MPIDI_POSIX_IS_REL_NULL(qhead->my_head)); + MPIDI_POSIX_Q_assert(MPIDI_POSIX_IS_REL_NULL(qhead->head)); + + _r_e = qhead->my_head; + _e = MPIDI_POSIX_REL_TO_ABS(_r_e); + + /* no barrier needed, my_head is private to consumer, plus + * head/my_head and _e->next are ordered by a data dependency */ + if (!MPIDI_POSIX_IS_REL_NULL(_e->next)) { + qhead->my_head = _e->next; + } + else { + /* we've reached the end (tail) of the queue */ + MPIDI_POSIX_cell_rel_ptr_t old_tail; + + MPIDI_POSIX_SET_REL_NULL(qhead->my_head); + /* no barrier needed, the caller doesn't need any ordering w.r.t. + * my_head or the tail */ + old_tail = MPIDI_POSIX_CAS_REL_NULL(&(qhead->tail), _r_e); + + if (!MPIDI_POSIX_REL_ARE_EQUAL(old_tail, _r_e)) { + /* FIXME is a barrier needed here because of the control-only dependency? */ + while (MPIDI_POSIX_IS_REL_NULL(_e->next)) { + MPIDI_POSIX_SKIP; + } + + /* no read barrier needed between loads from the same location */ + qhead->my_head = _e->next; + } + } + + MPIDI_POSIX_SET_REL_NULL(_e->next); + + /* Conservative read barrier here to ensure loads from head are ordered + * w.r.t. payload reads by the caller. The McKenney "whymb" document's + * Figure 11 indicates that we don't need a barrier, but we are currently + * unconvinced of this. Further work, ideally using more formal methods, + * should justify removing this. (note that this barrier won't cost us + * anything on many platforms, esp. x86) */ + OPA_read_barrier(); + + *e = _e; +} + +#else /* !defined(MPIDI_POSIX_USE_LOCK_FREE_QUEUES) */ + +/* FIXME We shouldn't really be using the MPID_Thread_mutex_* code but the + * MPIDU_Process_locks code is a total mess right now. In the long term we need + to resolve this, but in the short run it should be safe on most (all?) + platforms to use these instead. Usually they will both boil down to a + pthread_mutex_t and and associated functions. */ +#define MPIDI_POSIX_queue_mutex_create MPID_Thread_mutex_create +#define MPIDI_POSIX_queue_mutex_lock MPID_Thread_mutex_lock +#define MPIDI_POSIX_queue_mutex_unlock MPID_Thread_mutex_unlock + +/* must be called by exactly one process per queue */ +#undef FUNCNAME +#define FUNCNAME MPIDI_POSIX_queue_init +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline void MPIDI_POSIX_queue_init(MPIDI_POSIX_queue_ptr_t qhead) +{ + int err = 0; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDI_POSIX_QUEUE_INIT); + + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDI_POSIX_QUEUE_INIT); + + MPIDI_POSIX_SET_REL_NULL(qhead->head); + MPIDI_POSIX_SET_REL_NULL(qhead->my_head); + MPIDI_POSIX_SET_REL_NULL(qhead->tail); + MPIDI_POSIX_queue_mutex_create(&qhead->lock, &err); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDI_POSIX_QUEUE_INIT); +} + +static inline void MPIDI_POSIX_queue_enqueue(MPIDI_POSIX_queue_ptr_t qhead, + MPIDI_POSIX_cell_ptr_t element) +{ + int err = 0; + MPIDI_POSIX_cell_rel_ptr_t r_prev; + MPIDI_POSIX_cell_rel_ptr_t r_element = MPIDI_POSIX_ABS_TO_REL(element); + + MPIDI_POSIX_queue_mutex_lock(&qhead->lock, &err); + + r_prev = qhead->tail; + qhead->tail = r_element; + + if (MPIDI_POSIX_IS_REL_NULL(r_prev)) { + qhead->head = r_element; + } + else { + MPIDI_POSIX_REL_TO_ABS(r_prev)->next = r_element; + } + + MPIDI_POSIX_queue_mutex_unlock(&qhead->lock, &err); +} + +/* This operation is only safe because this is a single-dequeuer queue impl. */ +static inline MPIDI_POSIX_cell_ptr_t MPIDI_POSIX_queue_head(MPIDI_POSIX_queue_ptr_t qhead) +{ + return MPIDI_POSIX_REL_TO_ABS(qhead->my_head); +} + +/* Assumption: regular loads & stores are atomic. This may not be univerally + true, but it's not uncommon. We often need to use these "lock-ful" queues on + platforms where atomics are not yet implemented, so we can't rely on the + atomics to provide atomic load/store operations for us. */ +static inline int MPIDI_POSIX_queue_empty(MPIDI_POSIX_queue_ptr_t qhead) +{ + if (MPIDI_POSIX_IS_REL_NULL(qhead->my_head)) { + if (MPIDI_POSIX_IS_REL_NULL(qhead->head)) { + return 1; + } + else { + qhead->my_head = qhead->head; + MPIDI_POSIX_SET_REL_NULL(qhead->head); /* reset it for next time */ + } + } + + return 0; +} + +static inline void MPIDI_POSIX_queue_dequeue(MPIDI_POSIX_queue_ptr_t qhead, + MPIDI_POSIX_cell_ptr_t * e) +{ + int err = 0; + MPIDI_POSIX_cell_ptr_t _e; + MPIDI_POSIX_cell_rel_ptr_t _r_e; + + _r_e = qhead->my_head; + _e = MPIDI_POSIX_REL_TO_ABS(_r_e); + + + if (MPIDI_POSIX_IS_REL_NULL(_e->next)) { + /* a REL_NULL _e->next or writing qhead->tail both require locking */ + MPIDI_POSIX_queue_mutex_lock(&qhead->lock, &err); + qhead->my_head = _e->next; + + /* We have to check _e->next again because it may have changed between + * the time we checked it without the lock and the time that we acquired + * the lock. */ + if (MPIDI_POSIX_IS_REL_NULL(_e->next)) { + MPIDI_POSIX_SET_REL_NULL(qhead->tail); + } + + MPIDI_POSIX_queue_mutex_unlock(&qhead->lock, &err); + } + else { /* !MPIDI_POSIX_IS_REL_NULL(_e->next) */ + /* We don't need to lock because a non-null _e->next can't be changed by + * anyone but us (the dequeuer) and we don't need to modify qhead->tail + * because we aren't removing the last element from the queue. */ + qhead->my_head = _e->next; + } + + MPIDI_POSIX_SET_REL_NULL(_e->next); + *e = _e; +} + +#endif /* !defined(MPIDI_POSIX_USE_LOCK_FREE_QUEUES) */ + +#endif /* ifndef SHM_QUEUE_H */ diff --git a/src/mpid/ch4/shm/posix/posix_recv.h b/src/mpid/ch4/shm/posix/posix_recv.h new file mode 100644 index 0000000..d51e8e4 --- /dev/null +++ b/src/mpid/ch4/shm/posix/posix_recv.h @@ -0,0 +1,355 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef SHM_POSIX_RECV_H_INCLUDED +#define SHM_POSIX_RECV_H_INCLUDED + +#include "posix_impl.h" +#include "ch4_impl.h" + +/* ---------------------------------------------------- */ +/* general queues */ +/* ---------------------------------------------------- */ +extern MPIDI_POSIX_request_queue_t MPIDI_POSIX_recvq_posted; +extern MPIDI_POSIX_request_queue_t MPIDI_POSIX_recvq_unexpected; + +/* ---------------------------------------------------- */ +/* MPIDI_POSIX_do_irecv */ +/* ---------------------------------------------------- */ +#undef FCNAME +#define FCNAME DECL_FUNC(MPIDI_POSIX_do_irecv) +static inline int MPIDI_POSIX_do_irecv(void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request) +{ + int mpi_errno = MPI_SUCCESS, dt_contig; + size_t data_sz; + MPI_Aint dt_true_lb; + MPIR_Datatype *dt_ptr; + MPIR_Request *rreq = NULL; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_DO_IRECV); + + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_DO_IRECV); + + MPIDI_POSIX_REQUEST_CREATE_RREQ(rreq); + + if (unlikely(rank == MPI_PROC_NULL)) { + MPIR_Status_set_procnull(&(rreq->status)); + *request = rreq; + goto fn_exit; + } + + MPIDI_Datatype_get_info(count, datatype, dt_contig, data_sz, dt_ptr, dt_true_lb); + MPIDI_POSIX_ENVELOPE_SET(MPIDI_POSIX_REQUEST(rreq), rank, tag, + comm->context_id + context_offset); + rreq->comm = comm; + MPIR_Comm_add_ref(comm); + MPIDI_POSIX_REQUEST(rreq)->user_buf = (char *) buf + dt_true_lb; + MPIDI_POSIX_REQUEST(rreq)->user_count = count; + MPIDI_POSIX_REQUEST(rreq)->datatype = datatype; + MPIDI_POSIX_REQUEST(rreq)->data_sz = data_sz; + MPIDI_POSIX_REQUEST(rreq)->next = NULL; + MPIDI_POSIX_REQUEST(rreq)->segment_ptr = NULL; + MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(rreq) = NULL; + MPIR_STATUS_SET_COUNT(rreq->status, 0); + + if (!dt_contig) { + MPIDI_POSIX_REQUEST(rreq)->segment_ptr = MPID_Segment_alloc(); + MPIR_ERR_CHKANDJUMP1((MPIDI_POSIX_REQUEST(rreq)->segment_ptr == NULL), mpi_errno, + MPI_ERR_OTHER, "**nomem", "**nomem %s", "MPID_Segment_alloc"); + MPID_Segment_init((char *) buf, MPIDI_POSIX_REQUEST(rreq)->user_count, + MPIDI_POSIX_REQUEST(rreq)->datatype, + MPIDI_POSIX_REQUEST(rreq)->segment_ptr, 0); + MPIDI_POSIX_REQUEST(rreq)->segment_first = 0; + MPIDI_POSIX_REQUEST(rreq)->segment_size = data_sz; + } + + dtype_add_ref_if_not_builtin(datatype); + /* enqueue rreq */ + MPIDI_POSIX_REQUEST_ENQUEUE(rreq, MPIDI_POSIX_recvq_posted); + MPL_DBG_MSG_FMT(MPIR_DBG_HANDLE, TYPICAL, + (MPL_DBG_FDEST, + "Enqueued from grank %d to %d (comm_kind %d) in recv %d,%d,%d\n", + MPIDI_CH4U_rank_to_lpid(rank, comm), MPIDI_POSIX_mem_region.rank, + comm->comm_kind, rank, tag, comm->context_id + context_offset)); + *request = rreq; + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_DO_IRECV); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FCNAME +#define FCNAME DECL_FUNC(MPIDI_SHM_recv) +static inline int MPIDI_SHM_recv(void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, + int context_offset, MPI_Status * status, MPIR_Request ** request) +{ + int mpi_errno = MPI_SUCCESS, dt_contig __attribute__ ((__unused__)); + size_t data_sz __attribute__ ((__unused__)); + MPIR_FUNC_VERBOSE_STATE_DECL(MPIDI_SHM_RECV); + + MPIR_FUNC_VERBOSE_ENTER(MPIDI_SHM_RECV); + + /* create a request */ + MPID_THREAD_CS_ENTER(POBJ, MPIDI_POSIX_SHM_MUTEX); + mpi_errno = + MPIDI_POSIX_do_irecv(buf, count, datatype, rank, tag, comm, context_offset, request); + MPID_THREAD_CS_EXIT(POBJ, MPIDI_POSIX_SHM_MUTEX); + MPIR_FUNC_VERBOSE_EXIT(MPIDI_SHM_RECV); + return mpi_errno; +} + +#undef FCNAME +#define FCNAME DECL_FUNC(MPIDI_SHM_recv) +static inline int MPIDI_SHM_recv_init(void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Request *rreq = NULL; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDI_SHM_RECV_INIT); + + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDI_SHM_RECV_INIT); + + MPIDI_POSIX_REQUEST_CREATE_RREQ(rreq); + MPIR_Object_set_ref(rreq, 1); + MPIR_cc_set(&rreq->cc, 0); + rreq->kind = MPIR_REQUEST_KIND__PREQUEST_RECV; + rreq->comm = comm; + MPIR_Comm_add_ref(comm); + MPIDI_POSIX_ENVELOPE_SET(MPIDI_POSIX_REQUEST(rreq), rank, tag, + comm->context_id + context_offset); + MPIDI_POSIX_REQUEST(rreq)->user_buf = (char *) buf; + MPIDI_POSIX_REQUEST(rreq)->user_count = count; + MPIDI_POSIX_REQUEST(rreq)->datatype = datatype; + *request = rreq; + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDI_SHM_RECV_INIT); + return mpi_errno; +} + + +static inline int MPIDI_SHM_imrecv(void *buf, + int count, + MPI_Datatype datatype, + MPIR_Request * message, MPIR_Request ** rreqp) +{ + int mpi_errno = MPI_SUCCESS; + int dt_contig; + size_t data_sz; + MPI_Aint dt_true_lb; + MPIR_Datatype *dt_ptr; + MPIR_Request *rreq = NULL, *sreq = NULL; + int rank, tag, context_id; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPIDI_SHM_IMRECV); + MPIR_FUNC_VERBOSE_ENTER(MPIDI_SHM_IMRECV); + MPID_THREAD_CS_ENTER(POBJ, MPIDI_POSIX_SHM_MUTEX); + + if (message == NULL) { + MPIDI_Request_create_null_rreq(rreq, mpi_errno, goto fn_fail); + *rreqp = rreq; + goto fn_exit; + } + + MPIR_Assert(message != NULL); + + MPIDI_Datatype_get_info(count, datatype, dt_contig, data_sz, dt_ptr, dt_true_lb); + MPIDI_POSIX_REQUEST_CREATE_RREQ(rreq); + MPIR_Object_set_ref(rreq, 1); + MPIR_cc_set(&rreq->cc, 0); + MPIDI_POSIX_ENVELOPE_GET(MPIDI_POSIX_REQUEST(message), rank, tag, context_id); + MPIDI_POSIX_ENVELOPE_SET(MPIDI_POSIX_REQUEST(rreq), rank, tag, context_id); + rreq->comm = message->comm; + MPIR_Comm_add_ref(message->comm); + MPIDI_POSIX_REQUEST(rreq)->user_buf = (char *) buf + dt_true_lb; + MPIDI_POSIX_REQUEST(rreq)->user_count = count; + MPIDI_POSIX_REQUEST(rreq)->datatype = datatype; + MPIDI_POSIX_REQUEST(rreq)->next = NULL; + MPIDI_POSIX_REQUEST(rreq)->segment_ptr = NULL; + MPIR_STATUS_SET_COUNT(rreq->status, 0); + + if (!dt_contig) { + MPIDI_POSIX_REQUEST(rreq)->segment_ptr = MPID_Segment_alloc(); + MPIR_ERR_CHKANDJUMP1((MPIDI_POSIX_REQUEST(rreq)->segment_ptr == NULL), mpi_errno, + MPI_ERR_OTHER, "**nomem", "**nomem %s", "MPID_Segment_alloc"); + MPID_Segment_init((char *) buf, MPIDI_POSIX_REQUEST(rreq)->user_count, + MPIDI_POSIX_REQUEST(rreq)->datatype, + MPIDI_POSIX_REQUEST(rreq)->segment_ptr, 0); + MPIDI_POSIX_REQUEST(rreq)->segment_first = 0; + MPIDI_POSIX_REQUEST(rreq)->segment_size = data_sz; + } + + if (MPIDI_POSIX_REQUEST(message)->pending) { + /* Sync send - we must send ACK */ + int srank = message->status.MPI_SOURCE; + MPIR_Request *req_ack = NULL; + MPIDI_POSIX_REQUEST_CREATE_SREQ(req_ack); + MPIR_Object_set_ref(req_ack, 1); + req_ack->comm = message->comm; + MPIR_Comm_add_ref(message->comm); + MPIDI_POSIX_ENVELOPE_SET(MPIDI_POSIX_REQUEST(req_ack), message->comm->rank, tag, + context_id); + MPIDI_POSIX_REQUEST(req_ack)->user_buf = NULL; + MPIDI_POSIX_REQUEST(req_ack)->user_count = 0; + MPIDI_POSIX_REQUEST(req_ack)->datatype = MPI_BYTE; + MPIDI_POSIX_REQUEST(req_ack)->data_sz = 0; + MPIDI_POSIX_REQUEST(req_ack)->type = MPIDI_POSIX_TYPEACK; + MPIDI_POSIX_REQUEST(req_ack)->dest = srank; + MPIDI_POSIX_REQUEST(req_ack)->next = NULL; + MPIDI_POSIX_REQUEST(req_ack)->segment_ptr = NULL; + MPIDI_POSIX_REQUEST(req_ack)->pending = MPIDI_POSIX_REQUEST(message)->pending; + /* enqueue req_ack */ + MPIDI_POSIX_REQUEST_ENQUEUE(req_ack, MPIDI_POSIX_sendq); + } + + for (sreq = message; sreq;) { + MPIR_Request *next_req = NULL; + char *send_buffer = MPIDI_POSIX_REQUEST(sreq)->user_buf; + char *recv_buffer = (char *) MPIDI_POSIX_REQUEST(rreq)->user_buf; + + if (MPIDI_POSIX_REQUEST(sreq)->type == MPIDI_POSIX_TYPEEAGER) { + /* eager message */ + data_sz = MPIDI_POSIX_REQUEST(sreq)->data_sz; + + if (MPIDI_POSIX_REQUEST(rreq)->segment_ptr) { + /* non-contig */ + size_t last = MPIDI_POSIX_REQUEST(rreq)->segment_first + data_sz; + MPID_Segment_unpack(MPIDI_POSIX_REQUEST(rreq)->segment_ptr, + MPIDI_POSIX_REQUEST(rreq)->segment_first, (MPI_Aint *) & last, + send_buffer); + MPID_Segment_free(MPIDI_POSIX_REQUEST(rreq)->segment_ptr); + } + else + /* contig */ + if (send_buffer) + MPIR_Memcpy(recv_buffer, (void *) send_buffer, data_sz); + + /* set status */ + rreq->status.MPI_SOURCE = sreq->status.MPI_SOURCE; + rreq->status.MPI_TAG = sreq->status.MPI_TAG; + count = MPIR_STATUS_GET_COUNT(rreq->status) + (MPI_Count) data_sz; + MPIR_STATUS_SET_COUNT(rreq->status, count); + } + else if (MPIDI_POSIX_REQUEST(sreq)->type == MPIDI_POSIX_TYPELMT) { + /* long message */ + if (MPIDI_POSIX_REQUEST(rreq)->segment_ptr) { + /* non-contig */ + size_t last = + MPIDI_POSIX_REQUEST(rreq)->segment_first + MPIDI_POSIX_EAGER_THRESHOLD; + MPID_Segment_unpack(MPIDI_POSIX_REQUEST(rreq)->segment_ptr, + MPIDI_POSIX_REQUEST(rreq)->segment_first, (MPI_Aint *) & last, + send_buffer); + MPIDI_POSIX_REQUEST(rreq)->segment_first = last; + } + else + /* contig */ + if (send_buffer) + MPIR_Memcpy(recv_buffer, (void *) send_buffer, MPIDI_POSIX_EAGER_THRESHOLD); + + MPIDI_POSIX_REQUEST(rreq)->data_sz -= MPIDI_POSIX_EAGER_THRESHOLD; + MPIDI_POSIX_REQUEST(rreq)->user_buf += MPIDI_POSIX_EAGER_THRESHOLD; + count = MPIR_STATUS_GET_COUNT(rreq->status) + (MPI_Count) MPIDI_POSIX_EAGER_THRESHOLD; + MPIR_STATUS_SET_COUNT(rreq->status, count); + } + + /* destroy unexpected req */ + MPIDI_POSIX_REQUEST(sreq)->pending = NULL; + MPL_free(MPIDI_POSIX_REQUEST(sreq)->user_buf); + next_req = MPIDI_POSIX_REQUEST(sreq)->next; + MPIDI_POSIX_REQUEST_COMPLETE(sreq); + sreq = next_req; + } + + *rreqp = rreq; + + fn_exit: + MPID_THREAD_CS_EXIT(POBJ, MPIDI_POSIX_SHM_MUTEX); + MPIR_FUNC_VERBOSE_EXIT(MPIDI_SHM_IMRECV); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FCNAME +#define FCNAME DECL_FUNC(MPIDI_SHM_irecv) +static inline int MPIDI_SHM_irecv(void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPIDI_SHM_IRECV); + + MPIR_FUNC_VERBOSE_ENTER(MPIDI_SHM_IRECV); + MPID_THREAD_CS_ENTER(POBJ, MPIDI_POSIX_SHM_MUTEX); + mpi_errno = + MPIDI_POSIX_do_irecv(buf, count, datatype, rank, tag, comm, context_offset, request); + MPID_THREAD_CS_EXIT(POBJ, MPIDI_POSIX_SHM_MUTEX); + MPIR_FUNC_VERBOSE_EXIT(MPIDI_SHM_IRECV); + return mpi_errno; +} + +#undef FCNAME +#define FCNAME DECL_FUNC(MPIDI_SHM_cancel_recv) +static inline int MPIDI_SHM_cancel_recv(MPIR_Request * rreq) +{ + MPID_THREAD_CS_ENTER(POBJ, MPIDI_POSIX_SHM_MUTEX); + MPIR_Request *req = MPIDI_POSIX_recvq_posted.head; + MPIR_Request *prev_req = NULL; + + while (req) { + + if (req == rreq) { + /* Remove request from shm posted receive queue */ + + if (prev_req) { + MPIDI_POSIX_REQUEST(prev_req)->next = MPIDI_POSIX_REQUEST(req)->next; + } + else { + MPIDI_POSIX_recvq_posted.head = MPIDI_POSIX_REQUEST(req)->next; + } + + if (req == MPIDI_POSIX_recvq_posted.tail) { + MPIDI_POSIX_recvq_posted.tail = prev_req; + } + + MPIR_STATUS_SET_CANCEL_BIT(req->status, TRUE); + MPIR_STATUS_SET_COUNT(req->status, 0); + MPIDI_POSIX_REQUEST_COMPLETE(req); + + break; + } + + prev_req = req; + req = MPIDI_POSIX_REQUEST(req)->next; + } + + MPID_THREAD_CS_EXIT(POBJ, MPIDI_POSIX_SHM_MUTEX); + return MPI_SUCCESS; +} + +#endif /* SHM_POSIX_RECV_H_INCLUDED */ diff --git a/src/mpid/ch4/shm/posix/posix_request.h b/src/mpid/ch4/shm/posix/posix_request.h new file mode 100644 index 0000000..abf78e6 --- /dev/null +++ b/src/mpid/ch4/shm/posix/posix_request.h @@ -0,0 +1,26 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef SHM_POSIX_REQUEST_H_INCLUDED +#define SHM_POSIX_REQUEST_H_INCLUDED + +#include "posix_impl.h" + +static inline void MPIDI_SHM_am_request_init(MPIR_Request * req) +{ + MPIR_Assert(0); +} + +static inline void MPIDI_SHM_am_request_finalize(MPIR_Request * req) +{ + MPIR_Assert(0); +} + +#endif /* SHM_POSIX_REQUEST_H_INCLUDED */ diff --git a/src/mpid/ch4/shm/posix/posix_rma.h b/src/mpid/ch4/shm/posix/posix_rma.h new file mode 100644 index 0000000..8ea78bf --- /dev/null +++ b/src/mpid/ch4/shm/posix/posix_rma.h @@ -0,0 +1,143 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef SHM_POSIX_RMA_H_INCLUDED +#define SHM_POSIX_RMA_H_INCLUDED + +#include "posix_impl.h" + +static inline int MPIDI_SHM_put(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, MPI_Datatype target_datatype, MPIR_Win * win) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_get(void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, MPI_Datatype target_datatype, MPIR_Win * win) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_rput(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPIR_Win * win, MPIR_Request ** request) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + + +static inline int MPIDI_SHM_compare_and_swap(const void *origin_addr, + const void *compare_addr, + void *result_addr, + MPI_Datatype datatype, + int target_rank, MPI_Aint target_disp, MPIR_Win * win) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_raccumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPI_Op op, MPIR_Win * win, MPIR_Request ** request) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_rget_accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + void *result_addr, + int result_count, + MPI_Datatype result_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPI_Op op, MPIR_Win * win, MPIR_Request ** request) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_fetch_and_op(const void *origin_addr, + void *result_addr, + MPI_Datatype datatype, + int target_rank, + MPI_Aint target_disp, MPI_Op op, MPIR_Win * win) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + + +static inline int MPIDI_SHM_rget(void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPIR_Win * win, MPIR_Request ** request) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + + +static inline int MPIDI_SHM_get_accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + void *result_addr, + int result_count, + MPI_Datatype result_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, MPI_Op op, MPIR_Win * win) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, MPI_Op op, MPIR_Win * win) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +#endif /* SHM_POSIX_RMA_H_INCLUDED */ diff --git a/src/mpid/ch4/shm/posix/posix_send.h b/src/mpid/ch4/shm/posix/posix_send.h new file mode 100644 index 0000000..afde708 --- /dev/null +++ b/src/mpid/ch4/shm/posix/posix_send.h @@ -0,0 +1,462 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef SHM_POSIX_SEND_H_INCLUDED +#define SHM_POSIX_SEND_H_INCLUDED + +#include "posix_impl.h" +#include "ch4_impl.h" +#include <../mpi/pt2pt/bsendutil.h> +/* ---------------------------------------------------- */ +/* from mpid/ch3/channels/nemesis/include/mpid_nem_impl.h */ +/* ---------------------------------------------------- */ +/* assumes value!=0 means the fbox is full. Contains acquire barrier to + * ensure that later operations that are dependent on this check don't + * escape earlier than this check. */ +#define MPIDI_POSIX_fbox_is_full(pbox_) (OPA_load_acquire_int(&(pbox_)->flag.value)) + +/* ---------------------------------------------------- */ +/* MPIDI_POSIX_do_isend */ +/* ---------------------------------------------------- */ +#undef FCNAME +#define FCNAME DECL_FUNC(MPIDI_POSIX_do_isend) +static inline int MPIDI_POSIX_do_isend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, + MPIR_Request ** request, int type) +{ + int dt_contig, mpi_errno = MPI_SUCCESS; + MPI_Aint dt_true_lb; + MPIR_Request *sreq = NULL; + size_t data_sz; + MPIR_Datatype *dt_ptr; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_SHM_DO_ISEND); + + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_SHM_DO_ISEND); + + MPIDI_Datatype_get_info(count, datatype, dt_contig, data_sz, dt_ptr, dt_true_lb); + MPIDI_POSIX_REQUEST_CREATE_SREQ(sreq); + sreq->comm = comm; + MPIR_Comm_add_ref(comm); + MPIDI_POSIX_ENVELOPE_SET(MPIDI_POSIX_REQUEST(sreq), comm->rank, tag, + comm->context_id + context_offset); + MPIDI_POSIX_REQUEST(sreq)->user_buf = (char *) buf + dt_true_lb; + MPIDI_POSIX_REQUEST(sreq)->user_count = count; + MPIDI_POSIX_REQUEST(sreq)->datatype = datatype; + MPIDI_POSIX_REQUEST(sreq)->data_sz = data_sz; + MPIDI_POSIX_REQUEST(sreq)->type = type; + MPIDI_POSIX_REQUEST(sreq)->dest = rank; + MPIDI_POSIX_REQUEST(sreq)->next = NULL; + MPIDI_POSIX_REQUEST(sreq)->pending = NULL; + MPIDI_POSIX_REQUEST(sreq)->segment_ptr = NULL; + + if (!dt_contig) { + MPIDI_POSIX_REQUEST(sreq)->segment_ptr = MPID_Segment_alloc(); + MPIR_ERR_CHKANDJUMP1((MPIDI_POSIX_REQUEST(sreq)->segment_ptr == NULL), mpi_errno, + MPI_ERR_OTHER, "**nomem", "**nomem %s", "MPID_Segment_alloc"); + MPID_Segment_init((char *) buf, MPIDI_POSIX_REQUEST(sreq)->user_count, + MPIDI_POSIX_REQUEST(sreq)->datatype, + MPIDI_POSIX_REQUEST(sreq)->segment_ptr, 0); + MPIDI_POSIX_REQUEST(sreq)->segment_first = 0; + MPIDI_POSIX_REQUEST(sreq)->segment_size = data_sz; + } + + dtype_add_ref_if_not_builtin(datatype); + /* enqueue sreq */ + MPIDI_POSIX_REQUEST_ENQUEUE(sreq, MPIDI_POSIX_sendq); + *request = sreq; + MPL_DBG_MSG_FMT(MPIR_DBG_HANDLE, TYPICAL, + (MPL_DBG_FDEST, + "Enqueued to grank %d from %d (comm_kind %d) in recv %d,%d,%d\n", + MPIDI_CH4U_rank_to_lpid(rank, comm), MPIDI_POSIX_mem_region.rank, + comm->comm_kind, comm->rank, tag, comm->context_id + context_offset)); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_SHM_DO_ISEND); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FCNAME +#define FCNAME DECL_FUNC(MPIDI_SHM_SEND) +static inline int MPIDI_SHM_send(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + int dt_contig __attribute__ ((__unused__)), mpi_errno = MPI_SUCCESS; + MPI_Aint dt_true_lb; + size_t data_sz; + MPIR_Datatype *dt_ptr; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDI_SHM_SEND); + + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDI_SHM_SEND); + MPID_THREAD_CS_ENTER(POBJ, MPIDI_POSIX_SHM_MUTEX); + MPIDI_Datatype_get_info(count, datatype, dt_contig, data_sz, dt_ptr, dt_true_lb); + + /* try to send immediately, contig, short message */ + if (dt_contig && data_sz <= MPIDI_POSIX_EAGER_THRESHOLD) { + /* eager message */ + int grank = MPIDI_CH4U_rank_to_lpid(rank, comm); + + /* Try freeQ */ + if (!MPIDI_POSIX_queue_empty(MPIDI_POSIX_mem_region.my_freeQ)) { + MPIDI_POSIX_cell_ptr_t cell; + MPIDI_POSIX_queue_dequeue(MPIDI_POSIX_mem_region.my_freeQ, &cell); + MPIDI_POSIX_ENVELOPE_SET(cell, comm->rank, tag, comm->context_id + context_offset); + cell->pkt.mpich.datalen = data_sz; + cell->pkt.mpich.type = MPIDI_POSIX_TYPEEAGER; + MPIR_Memcpy((void *) cell->pkt.mpich.p.payload, (char *) buf + dt_true_lb, data_sz); + cell->pending = NULL; + MPIDI_POSIX_queue_enqueue(MPIDI_POSIX_mem_region.RecvQ[grank], cell); + *request = NULL; + MPL_DBG_MSG_FMT(MPIR_DBG_HANDLE, TYPICAL, + (MPL_DBG_FDEST, "Sent to grank %d from %d in send %d,%d,%d\n", grank, + cell->my_rank, cell->rank, cell->tag, cell->context_id)); + goto fn_exit; + } + } + + /* Long message or */ + /* Failed to send immediately - create and return request */ + mpi_errno = + MPIDI_POSIX_do_isend(buf, count, datatype, rank, tag, comm, context_offset, request, + MPIDI_POSIX_TYPESTANDARD); + + fn_exit: + MPID_THREAD_CS_EXIT(POBJ, MPIDI_POSIX_SHM_MUTEX); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDI_SHM_SEND); + return mpi_errno; +} + + + + +static inline int MPIDI_SHM_irsend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDI_SHM_ISEND); + + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDI_SHM_ISEND); + MPID_THREAD_CS_ENTER(POBJ, MPIDI_POSIX_SHM_MUTEX); + mpi_errno = + MPIDI_POSIX_do_isend(buf, count, datatype, rank, tag, comm, context_offset, request, + MPIDI_POSIX_TYPEREADY); + MPID_THREAD_CS_EXIT(POBJ, MPIDI_POSIX_SHM_MUTEX); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDI_SHM_ISEND); + return mpi_errno; +} + +#undef FCNAME +#define FCNAME DECL_FUNC(MPIDI_SHM_SSEND) +static inline int MPIDI_SHM_ssend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDI_SHM_SSEND); + + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDI_SHM_SSEND); + MPID_THREAD_CS_ENTER(POBJ, MPIDI_POSIX_SHM_MUTEX); + mpi_errno = + MPIDI_POSIX_do_isend(buf, count, datatype, rank, tag, comm, context_offset, request, + MPIDI_POSIX_TYPESYNC); + MPID_THREAD_CS_EXIT(POBJ, MPIDI_POSIX_SHM_MUTEX); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDI_SHM_SSEND); + return mpi_errno; +} + +#undef FCNAME +#define FCNAME DECL_FUNC(MPIDI_SHM_STARTALL) +static inline int MPIDI_SHM_startall(int count, MPIR_Request * requests[]) +{ + int i, mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDI_SHM_STARTALL); + + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDI_SHM_STARTALL); + MPID_THREAD_CS_ENTER(POBJ, MPIDI_POSIX_SHM_MUTEX); + + for (i = 0; i < count; i++) { + MPIR_Request *preq = requests[i]; + + if (preq->kind == MPIR_REQUEST_KIND__PREQUEST_SEND) { + if (MPIDI_POSIX_REQUEST(preq)->type != MPIDI_POSIX_TYPEBUFFERED) { + mpi_errno = + MPIDI_POSIX_do_isend(MPIDI_POSIX_REQUEST(preq)->user_buf, + MPIDI_POSIX_REQUEST(preq)->user_count, + MPIDI_POSIX_REQUEST(preq)->datatype, + MPIDI_POSIX_REQUEST(preq)->dest, + MPIDI_POSIX_REQUEST(preq)->tag, preq->comm, + MPIDI_POSIX_REQUEST(preq)->context_id - + preq->comm->context_id, &preq->u.persist.real_request, + MPIDI_POSIX_REQUEST(preq)->type); + } + else { + MPI_Request sreq_handle; + mpi_errno = + MPIR_Ibsend_impl(MPIDI_POSIX_REQUEST(preq)->user_buf, + MPIDI_POSIX_REQUEST(preq)->user_count, + MPIDI_POSIX_REQUEST(preq)->datatype, + MPIDI_POSIX_REQUEST(preq)->dest, + MPIDI_POSIX_REQUEST(preq)->tag, preq->comm, &sreq_handle); + + if (mpi_errno == MPI_SUCCESS) + MPIR_Request_get_ptr(sreq_handle, preq->u.persist.real_request); + } + } + else if (preq->kind == MPIR_REQUEST_KIND__PREQUEST_RECV) { + mpi_errno = + MPIDI_POSIX_do_irecv(MPIDI_POSIX_REQUEST(preq)->user_buf, + MPIDI_POSIX_REQUEST(preq)->user_count, + MPIDI_POSIX_REQUEST(preq)->datatype, + MPIDI_POSIX_REQUEST(preq)->rank, + MPIDI_POSIX_REQUEST(preq)->tag, preq->comm, + MPIDI_POSIX_REQUEST(preq)->context_id - preq->comm->context_id, + &preq->u.persist.real_request); + } + else { + MPIR_Assert(0); + } + + if (mpi_errno == MPI_SUCCESS) { + preq->status.MPI_ERROR = MPI_SUCCESS; + + if (MPIDI_POSIX_REQUEST(preq)->type == MPIDI_POSIX_TYPEBUFFERED) { + preq->cc_ptr = &preq->cc; + MPIR_cc_set(&preq->cc, 0); + } + else + preq->cc_ptr = &preq->u.persist.real_request->cc; + } + else { + preq->u.persist.real_request = NULL; + preq->status.MPI_ERROR = mpi_errno; + preq->cc_ptr = &preq->cc; + MPIR_cc_set(&preq->cc, 0); + } + } + + MPID_THREAD_CS_EXIT(POBJ, MPIDI_POSIX_SHM_MUTEX); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDI_SHM_STARTALL); + return mpi_errno; +} + +#undef FCNAME +#define FCNAME DECL_FUNC(MPIDI_SHM_SEND_INIT) +static inline int MPIDI_SHM_send_init(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Request *sreq = NULL; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDI_SHM_SEND_INIT); + + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDI_SHM_SEND_INIT); + MPIDI_POSIX_REQUEST_CREATE_SREQ(sreq); + MPIR_Object_set_ref(sreq, 1); + MPIR_cc_set(&(sreq)->cc, 0); + sreq->kind = MPIR_REQUEST_KIND__PREQUEST_SEND; + sreq->comm = comm; + MPIR_Comm_add_ref(comm); + MPIDI_POSIX_ENVELOPE_SET(MPIDI_POSIX_REQUEST(sreq), comm->rank, tag, + comm->context_id + context_offset); + MPIDI_POSIX_REQUEST(sreq)->user_buf = (char *) buf; + MPIDI_POSIX_REQUEST(sreq)->user_count = count; + MPIDI_POSIX_REQUEST(sreq)->dest = rank; + MPIDI_POSIX_REQUEST(sreq)->datatype = datatype; + MPIDI_POSIX_REQUEST(sreq)->type = MPIDI_POSIX_TYPESTANDARD; + *request = sreq; + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDI_SHM_SEND_INIT); + return mpi_errno; +} + +#undef FCNAME +#define FCNAME DECL_FUNC(MPIDI_SHM_SSEND_INIT) +static inline int MPIDI_SHM_ssend_init(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, + int context_offset, MPIR_Request ** request) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Request *sreq = NULL; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDI_SHM_SEND_INIT); + + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDI_SHM_SEND_INIT); + MPIDI_POSIX_REQUEST_CREATE_SREQ(sreq); + MPIR_Object_set_ref(sreq, 1); + sreq->kind = MPIR_REQUEST_KIND__PREQUEST_SEND; + sreq->comm = comm; + MPIR_Comm_add_ref(comm); + MPIDI_POSIX_ENVELOPE_SET(MPIDI_POSIX_REQUEST(sreq), comm->rank, tag, + comm->context_id + context_offset); + MPIDI_POSIX_REQUEST(sreq)->user_buf = (char *) buf; + MPIDI_POSIX_REQUEST(sreq)->user_count = count; + MPIDI_POSIX_REQUEST(sreq)->dest = rank; + MPIDI_POSIX_REQUEST(sreq)->datatype = datatype; + MPIDI_POSIX_REQUEST(sreq)->type = MPIDI_POSIX_TYPESYNC; + *request = sreq; + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDI_SHM_SEND_INIT); + return mpi_errno; +} + +static inline int MPIDI_SHM_bsend_init(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, + int context_offset, MPIR_Request ** request) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Request *sreq = NULL; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDI_SHM_SEND_INIT); + + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDI_SHM_SEND_INIT); + MPIDI_POSIX_REQUEST_CREATE_SREQ(sreq); + MPIR_Object_set_ref(sreq, 1); + sreq->kind = MPIR_REQUEST_KIND__PREQUEST_SEND; + sreq->comm = comm; + MPIR_Comm_add_ref(comm); + MPIDI_POSIX_ENVELOPE_SET(MPIDI_POSIX_REQUEST(sreq), comm->rank, tag, + comm->context_id + context_offset); + MPIDI_POSIX_REQUEST(sreq)->user_buf = (char *) buf; + MPIDI_POSIX_REQUEST(sreq)->user_count = count; + MPIDI_POSIX_REQUEST(sreq)->dest = rank; + MPIDI_POSIX_REQUEST(sreq)->datatype = datatype; + MPIDI_POSIX_REQUEST(sreq)->type = MPIDI_POSIX_TYPEBUFFERED; + *request = sreq; + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDI_SHM_SEND_INIT); + return mpi_errno; +} + +static inline int MPIDI_SHM_rsend_init(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, + int context_offset, MPIR_Request ** request) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Request *sreq = NULL; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDI_SHM_RSEND_INIT); + + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDI_SHM_RSEND_INIT); + MPIDI_POSIX_REQUEST_CREATE_SREQ(sreq); + MPIR_Object_set_ref(sreq, 1); + MPIR_cc_set(&(sreq)->cc, 0); + sreq->kind = MPIR_REQUEST_KIND__PREQUEST_SEND; + sreq->comm = comm; + MPIR_Comm_add_ref(comm); + MPIDI_POSIX_ENVELOPE_SET(MPIDI_POSIX_REQUEST(sreq), comm->rank, tag, + comm->context_id + context_offset); + MPIDI_POSIX_REQUEST(sreq)->user_buf = (char *) buf; + MPIDI_POSIX_REQUEST(sreq)->user_count = count; + MPIDI_POSIX_REQUEST(sreq)->dest = rank; + MPIDI_POSIX_REQUEST(sreq)->datatype = datatype; + MPIDI_POSIX_REQUEST(sreq)->type = MPIDI_POSIX_TYPEREADY; + *request = sreq; + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDI_SHM_RSEND_INIT); + return mpi_errno; +} + +#undef FCNAME +#define FCNAME DECL_FUNC(MPIDI_SHM_isend) +static inline int MPIDI_SHM_isend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDI_SHM_ISEND); + + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDI_SHM_ISEND); + MPID_THREAD_CS_ENTER(POBJ, MPIDI_POSIX_SHM_MUTEX); + mpi_errno = + MPIDI_POSIX_do_isend(buf, count, datatype, rank, tag, comm, context_offset, request, + MPIDI_POSIX_TYPESTANDARD); + MPID_THREAD_CS_EXIT(POBJ, MPIDI_POSIX_SHM_MUTEX); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDI_SHM_ISEND); + return mpi_errno; +} + +static inline int MPIDI_SHM_issend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDI_SHM_ISSEND); + + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDI_SHM_ISSEND); + MPID_THREAD_CS_ENTER(POBJ, MPIDI_POSIX_SHM_MUTEX); + mpi_errno = + MPIDI_POSIX_do_isend(buf, count, datatype, rank, tag, comm, context_offset, request, + MPIDI_POSIX_TYPESYNC); + MPID_THREAD_CS_EXIT(POBJ, MPIDI_POSIX_SHM_MUTEX); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDI_SHM_ISSEND); + return mpi_errno; +} + +static inline int MPIDI_SHM_cancel_send(MPIR_Request * sreq) +{ + MPID_THREAD_CS_ENTER(POBJ, MPIDI_POSIX_SHM_MUTEX); + MPIR_Request *req = MPIDI_POSIX_sendq.head; + MPIR_Request *prev_req = NULL; + int mpi_errno = MPI_SUCCESS; + + while (req) { + + if (req == sreq) { + MPIR_STATUS_SET_CANCEL_BIT(sreq->status, TRUE); + MPIR_STATUS_SET_COUNT(sreq->status, 0); + MPIDI_POSIX_REQUEST_COMPLETE(sreq); + MPIDI_POSIX_REQUEST_DEQUEUE_AND_SET_ERROR(&sreq, prev_req, MPIDI_POSIX_sendq, + mpi_errno); + break; + } + + prev_req = req; + req = MPIDI_POSIX_REQUEST(req)->next; + } + + MPID_THREAD_CS_EXIT(POBJ, MPIDI_POSIX_SHM_MUTEX); + return mpi_errno; +} + +#endif /* SHM_POSIX_SEND_H_INCLUDED */ diff --git a/src/mpid/ch4/shm/posix/posix_spawn.h b/src/mpid/ch4/shm/posix/posix_spawn.h new file mode 100644 index 0000000..2a2e001 --- /dev/null +++ b/src/mpid/ch4/shm/posix/posix_spawn.h @@ -0,0 +1,50 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef SHM_POSIX_DYNPROC_H_INCLUDED +#define SHM_POSIX_DYNPROC_H_INCLUDED + +#include "posix_impl.h" + +static inline int MPIDI_SHM_comm_connect(const char *port_name, + MPIR_Info * info, + int root, MPIR_Comm * comm, MPIR_Comm ** newcomm_ptr) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_comm_disconnect(MPIR_Comm * comm_ptr) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_open_port(MPIR_Info * info_ptr, char *port_name) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_close_port(const char *port_name) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_comm_accept(const char *port_name, + MPIR_Info * info, + int root, MPIR_Comm * comm, MPIR_Comm ** newcomm_ptr) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +#endif /* SHM_POSIX_WIN_H_INCLUDED */ diff --git a/src/mpid/ch4/shm/posix/posix_unimpl.h b/src/mpid/ch4/shm/posix/posix_unimpl.h new file mode 100644 index 0000000..db8e66b --- /dev/null +++ b/src/mpid/ch4/shm/posix/posix_unimpl.h @@ -0,0 +1,19 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ + +#ifdef __cplusplus +extern "C" { +#endif + + +#ifdef __cplusplus +} +#endif diff --git a/src/mpid/ch4/shm/posix/posix_win.h b/src/mpid/ch4/shm/posix/posix_win.h new file mode 100644 index 0000000..aeac908 --- /dev/null +++ b/src/mpid/ch4/shm/posix/posix_win.h @@ -0,0 +1,185 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef SHM_POSIX_WIN_H_INCLUDED +#define SHM_POSIX_WIN_H_INCLUDED + +#include "posix_impl.h" + +static inline int MPIDI_SHM_win_set_info(MPIR_Win * win, MPIR_Info * info) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + + +static inline int MPIDI_SHM_win_start(MPIR_Group * group, int assert, MPIR_Win * win) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + + +static inline int MPIDI_SHM_win_complete(MPIR_Win * win) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_win_post(MPIR_Group * group, int assert, MPIR_Win * win) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + + +static inline int MPIDI_SHM_win_wait(MPIR_Win * win) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + + +static inline int MPIDI_SHM_win_test(MPIR_Win * win, int *flag) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_win_lock(int lock_type, int rank, int assert, MPIR_Win * win) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + + +static inline int MPIDI_SHM_win_unlock(int rank, MPIR_Win * win) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_win_get_info(MPIR_Win * win, MPIR_Info ** info_p_p) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + + +static inline int MPIDI_SHM_win_free(MPIR_Win ** win_ptr) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_win_fence(int assert, MPIR_Win * win) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_win_create(void *base, + MPI_Aint length, + int disp_unit, + MPIR_Info * info, MPIR_Comm * comm_ptr, MPIR_Win ** win_ptr) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_win_attach(MPIR_Win * win, void *base, MPI_Aint size) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_win_allocate_shared(MPI_Aint size, + int disp_unit, + MPIR_Info * info_ptr, + MPIR_Comm * comm_ptr, + void **base_ptr, MPIR_Win ** win_ptr) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_win_detach(MPIR_Win * win, const void *base) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_win_shared_query(MPIR_Win * win, + int rank, + MPI_Aint * size, int *disp_unit, void *baseptr) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_win_allocate(MPI_Aint size, + int disp_unit, + MPIR_Info * info, + MPIR_Comm * comm, void *baseptr, MPIR_Win ** win) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_win_flush(int rank, MPIR_Win * win) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_win_flush_local_all(MPIR_Win * win) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_win_unlock_all(MPIR_Win * win) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_win_create_dynamic(MPIR_Info * info, MPIR_Comm * comm, MPIR_Win ** win) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_win_flush_local(int rank, MPIR_Win * win) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_win_sync(MPIR_Win * win) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_win_flush_all(MPIR_Win * win) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_win_lock_all(int assert, MPIR_Win * win) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + + +#endif /* SHM_POSIX_WIN_H_INCLUDED */ diff --git a/src/mpid/ch4/shm/posix/shm_direct.h b/src/mpid/ch4/shm/posix/shm_direct.h new file mode 100644 index 0000000..9173da2 --- /dev/null +++ b/src/mpid/ch4/shm/posix/shm_direct.h @@ -0,0 +1,27 @@ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef SHM_DIRECT_H_INCLUDED +#define SHM_DIRECT_H_INCLUDED + +#include "posix_init.h" +#include "posix_probe.h" +#include "posix_progress.h" +#include "posix_recv.h" +#include "posix_request.h" +#include "posix_send.h" +#include "posix_win.h" +#include "posix_rma.h" +#include "posix_am.h" +#include "posix_spawn.h" +#include "posix_comm.h" +#include "posix_coll.h" +#include "posix_unimpl.h" + +#endif /* SHM_DIRECT_H_INCLUDED */ diff --git a/src/mpid/ch4/shm/posix/subconfigure.m4 b/src/mpid/ch4/shm/posix/subconfigure.m4 new file mode 100644 index 0000000..0a32273 --- /dev/null +++ b/src/mpid/ch4/shm/posix/subconfigure.m4 @@ -0,0 +1,53 @@ +[#] start of __file__ +dnl MPICH_SUBCFG_BEFORE=src/mpid/common/shm +dnl MPICH_SUBCFG_AFTER=src/mpid/ch4 + +AC_DEFUN([PAC_SUBCFG_PREREQ_]PAC_SUBCFG_AUTO_SUFFIX,[ + AM_COND_IF([BUILD_CH4],[ + for shm in $ch4_shm ; do + AS_CASE([$shm],[posix],[build_ch4_shm_posix=yes]) + done + + AC_ARG_WITH(ch4-shmmod-posix-args, + [ --with-ch4-shmmod-posix-args=arg1:arg2:arg3 + CH4 POSIX shmmod arguments: + disable-lock-free-queues - Disable atomics and lock-free queues + ], + [posix_shmmod_args=$withval], + [posix_shmmod_args=]) + +dnl Parse the shmmod arguments + SAVE_IFS=$IFS + IFS=':' + args_array=$posix_shmmod_args + do_disable_lock_free_queues=false + echo "Parsing Arguments for POSIX shmmod" + for arg in $args_array; do + case ${arg} in + disable-lock-free-queues) + do_disable_lock_free_queues=true + echo " ---> CH4::SHM::POSIX : $arg" + ;; + esac + done + IFS=$SAVE_IFS + + if [test "$do_disable_lock_free_queues" = "true"]; then + AC_MSG_NOTICE([Disabling POSIX shared memory lock free queues]) + else + AC_MSG_NOTICE([Enabling POSIX shared memory lock free queues]) + PAC_APPEND_FLAG([-DMPIDI_POSIX_USE_LOCK_FREE_QUEUES],[CPPFLAGS]) + fi + # the POSIX shmmod depends on the common shm code + build_mpid_common_shm=yes + ]) + AM_CONDITIONAL([BUILD_SHM_POSIX],[test "X$build_ch4_shm_posix" = "Xyes"]) +])dnl + +AC_DEFUN([PAC_SUBCFG_BODY_]PAC_SUBCFG_AUTO_SUFFIX,[ +AM_COND_IF([BUILD_SHM_POSIX],[ + AC_MSG_NOTICE([RUNNING CONFIGURE FOR ch4:shm:posix]) +])dnl end AM_COND_IF(BUILD_SHM_POSIX,...) +])dnl end _BODY + +[#] end of __file__ diff --git a/src/mpid/ch4/shm/stubshm/Makefile.mk b/src/mpid/ch4/shm/stubshm/Makefile.mk new file mode 100644 index 0000000..b1dcb01 --- /dev/null +++ b/src/mpid/ch4/shm/stubshm/Makefile.mk @@ -0,0 +1,7 @@ +if BUILD_SHM_STUBSHM + +mpi_core_sources += src/mpid/ch4/shm/stubshm/globals.c \ + src/mpid/ch4/shm/stubshm/func_table.c +# errnames_txt_files += src/mpid/ch4/shm/stub/errnames.txt + +endif diff --git a/src/mpid/ch4/shm/stubshm/func_table.c b/src/mpid/ch4/shm/stubshm/func_table.c new file mode 100644 index 0000000..1244e17 --- /dev/null +++ b/src/mpid/ch4/shm/stubshm/func_table.c @@ -0,0 +1,152 @@ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ + +#ifndef SHM_DIRECT +#define SHM_DISABLE_INLINES +#include <mpidimpl.h> +#include "shm_direct.h" +MPIDI_SHM_funcs_t MPIDI_SHM_stubshm_funcs = { + MPIDI_SHM_init, + MPIDI_SHM_finalize, + MPIDI_SHM_progress, + MPIDI_SHM_reg_hdr_handler, + MPIDI_SHM_comm_connect, + MPIDI_SHM_comm_disconnect, + MPIDI_SHM_open_port, + MPIDI_SHM_close_port, + MPIDI_SHM_comm_accept, + MPIDI_SHM_send_am_hdr, + MPIDI_SHM_inject_am_hdr, + MPIDI_SHM_send_am, + MPIDI_SHM_inject_am, + MPIDI_SHM_send_amv, + MPIDI_SHM_inject_amv, + MPIDI_SHM_send_am_hdr_reply, + MPIDI_SHM_inject_am_hdr_reply, + MPIDI_SHM_send_am_reply, + MPIDI_SHM_inject_am_reply, + MPIDI_SHM_send_amv_reply, + MPIDI_SHM_inject_amv_reply, + MPIDI_SHM_am_hdr_max_sz, + MPIDI_SHM_am_inject_max_sz, + MPIDI_SHM_am_recv, + MPIDI_SHM_comm_get_lpid, + MPIDI_SHM_gpid_get, + MPIDI_SHM_get_node_id, + MPIDI_SHM_get_max_node_id, + MPIDI_SHM_getallincomm, + MPIDI_SHM_gpid_tolpidarray, + MPIDI_SHM_create_intercomm_from_lpids, + MPIDI_SHM_comm_create, + MPIDI_SHM_comm_destroy, + MPIDI_SHM_am_request_init, +}; + +MPIDI_SHM_native_funcs_t MPIDI_SHM_native_stubshm_funcs = { + MPIDI_SHM_send, + MPIDI_SHM_ssend, + MPIDI_SHM_startall, + MPIDI_SHM_send_init, + MPIDI_SHM_ssend_init, + MPIDI_SHM_rsend_init, + MPIDI_SHM_bsend_init, + MPIDI_SHM_isend, + MPIDI_SHM_issend, + MPIDI_SHM_cancel_send, + MPIDI_SHM_recv_init, + MPIDI_SHM_recv, + MPIDI_SHM_irecv, + MPIDI_SHM_imrecv, + MPIDI_SHM_cancel_recv, + MPIDI_SHM_alloc_mem, + MPIDI_SHM_free_mem, + MPIDI_SHM_improbe, + MPIDI_SHM_iprobe, + MPIDI_SHM_win_set_info, + MPIDI_SHM_win_shared_query, + MPIDI_SHM_put, + MPIDI_SHM_win_start, + MPIDI_SHM_win_complete, + MPIDI_SHM_win_post, + MPIDI_SHM_win_wait, + MPIDI_SHM_win_test, + MPIDI_SHM_win_lock, + MPIDI_SHM_win_unlock, + MPIDI_SHM_win_get_info, + MPIDI_SHM_get, + MPIDI_SHM_win_free, + MPIDI_SHM_win_fence, + MPIDI_SHM_win_create, + MPIDI_SHM_accumulate, + MPIDI_SHM_win_attach, + MPIDI_SHM_win_allocate_shared, + MPIDI_SHM_rput, + MPIDI_SHM_win_flush_local, + MPIDI_SHM_win_detach, + MPIDI_SHM_compare_and_swap, + MPIDI_SHM_raccumulate, + MPIDI_SHM_rget_accumulate, + MPIDI_SHM_fetch_and_op, + MPIDI_SHM_win_allocate, + MPIDI_SHM_win_flush, + MPIDI_SHM_win_flush_local_all, + MPIDI_SHM_win_unlock_all, + MPIDI_SHM_win_create_dynamic, + MPIDI_SHM_rget, + MPIDI_SHM_win_sync, + MPIDI_SHM_win_flush_all, + MPIDI_SHM_get_accumulate, + MPIDI_SHM_win_lock_all, + MPIDI_SHM_barrier, + MPIDI_SHM_bcast, + MPIDI_SHM_allreduce, + MPIDI_SHM_allgather, + MPIDI_SHM_allgatherv, + MPIDI_SHM_scatter, + MPIDI_SHM_scatterv, + MPIDI_SHM_gather, + MPIDI_SHM_gatherv, + MPIDI_SHM_alltoall, + MPIDI_SHM_alltoallv, + MPIDI_SHM_alltoallw, + MPIDI_SHM_reduce, + MPIDI_SHM_reduce_scatter, + MPIDI_SHM_reduce_scatter_block, + MPIDI_SHM_scan, + MPIDI_SHM_exscan, + MPIDI_SHM_neighbor_allgather, + MPIDI_SHM_neighbor_allgatherv, + MPIDI_SHM_neighbor_alltoall, + MPIDI_SHM_neighbor_alltoallv, + MPIDI_SHM_neighbor_alltoallw, + MPIDI_SHM_ineighbor_allgather, + MPIDI_SHM_ineighbor_allgatherv, + MPIDI_SHM_ineighbor_alltoall, + MPIDI_SHM_ineighbor_alltoallv, + MPIDI_SHM_ineighbor_alltoallw, + MPIDI_SHM_ibarrier, + MPIDI_SHM_ibcast, + MPIDI_SHM_iallgather, + MPIDI_SHM_iallgatherv, + MPIDI_SHM_iallreduce, + MPIDI_SHM_ialltoall, + MPIDI_SHM_ialltoallv, + MPIDI_SHM_ialltoallw, + MPIDI_SHM_iexscan, + MPIDI_SHM_igather, + MPIDI_SHM_igatherv, + MPIDI_SHM_ireduce_scatter_block, + MPIDI_SHM_ireduce_scatter, + MPIDI_SHM_ireduce, + MPIDI_SHM_iscan, + MPIDI_SHM_iscatter, + MPIDI_SHM_iscatterv, +}; +#endif diff --git a/src/mpid/ch4/shm/stubshm/globals.c b/src/mpid/ch4/shm/stubshm/globals.c new file mode 100644 index 0000000..74ad1a3 --- /dev/null +++ b/src/mpid/ch4/shm/stubshm/globals.c @@ -0,0 +1,13 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ + +#include <mpidimpl.h> +#include "stubshm_impl.h" diff --git a/src/mpid/ch4/shm/stubshm/shm_direct.h b/src/mpid/ch4/shm/stubshm/shm_direct.h new file mode 100644 index 0000000..69fd12b --- /dev/null +++ b/src/mpid/ch4/shm/stubshm/shm_direct.h @@ -0,0 +1,27 @@ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef SHM_DIRECT_H_INCLUDED +#define SHM_DIRECT_H_INCLUDED + +#include "stubshm_init.h" +#include "stubshm_probe.h" +#include "stubshm_progress.h" +#include "stubshm_recv.h" +#include "stubshm_request.h" +#include "stubshm_send.h" +#include "stubshm_win.h" +#include "stubshm_rma.h" +#include "stubshm_am.h" +#include "stubshm_spawn.h" +#include "stubshm_comm.h" +#include "stubshm_coll.h" +#include "stubshm_unimpl.h" + +#endif /* SHM_DIRECT_H_INCLUDED */ diff --git a/src/mpid/ch4/shm/stubshm/stubshm_am.h b/src/mpid/ch4/shm/stubshm/stubshm_am.h new file mode 100644 index 0000000..1ff7dab --- /dev/null +++ b/src/mpid/ch4/shm/stubshm/stubshm_am.h @@ -0,0 +1,173 @@ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef SHM_STUBSHM_AM_H_INCLUDED +#define SHM_STUBSHM_AM_H_INCLUDED + +#include "stubshm_impl.h" + +static inline int MPIDI_SHM_reg_hdr_handler(int handler_id, + MPIDI_SHM_am_origin_handler_fn origin_handler_fn, + MPIDI_SHM_am_target_handler_fn target_handler_fn) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_send_am_hdr(int rank, + MPIR_Comm * comm, + int handler_id, + const void *am_hdr, + size_t am_hdr_sz, MPIR_Request * sreq, void *shm_context) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_send_am(int rank, + MPIR_Comm * comm, + int handler_id, + const void *am_hdr, + size_t am_hdr_sz, + const void *data, + MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq, void *shm_context) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_send_amv(int rank, + MPIR_Comm * comm, + int handler_id, + struct iovec *am_hdr, + size_t iov_len, + const void *data, + MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq, void *shm_context) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_send_am_hdr_reply(MPIR_Context_id_t context_id, int src_rank, + int handler_id, + const void *am_hdr, + size_t am_hdr_sz, MPIR_Request * sreq) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_send_am_reply(MPIR_Context_id_t context_id, int src_rank, + int handler_id, + const void *am_hdr, + size_t am_hdr_sz, + const void *data, + MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_send_amv_reply(MPIR_Context_id_t context_id, int src_rank, + int handler_id, + struct iovec *am_hdr, + size_t iov_len, + const void *data, + MPI_Count count, + MPI_Datatype datatype, MPIR_Request * sreq) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline size_t MPIDI_SHM_am_hdr_max_sz(void) +{ + MPIR_Assert(0); + return 0; +} + +static inline int MPIDI_SHM_inject_am_hdr(int rank, + MPIR_Comm * comm, + int handler_id, + const void *am_hdr, size_t am_hdr_sz, void *shm_context) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_inject_am(int rank, + MPIR_Comm * comm, + int handler_id, + const void *am_hdr, + size_t am_hdr_sz, + const void *data, + MPI_Count count, MPI_Datatype datatype, void *shm_context) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_inject_amv(int rank, + MPIR_Comm * comm, + int handler_id, + struct iovec *am_hdr, + size_t iov_len, + const void *data, + MPI_Count count, MPI_Datatype datatype, void *shm_context) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_inject_am_hdr_reply(MPIR_Context_id_t context_id, int src_rank, + int handler_id, + const void *am_hdr, size_t am_hdr_sz) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_inject_am_reply(MPIR_Context_id_t context_id, int src_rank, + int handler_id, + const void *am_hdr, + size_t am_hdr_sz, + const void *data, + MPI_Count count, MPI_Datatype datatype) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_inject_amv_reply(MPIR_Context_id_t context_id, int src_rank, + int handler_id, + struct iovec *am_hdr, + size_t iov_len, + const void *data, + MPI_Count count, MPI_Datatype datatype) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline size_t MPIDI_SHM_am_inject_max_sz(void) +{ + MPIR_Assert(0); + return 0; +} + +static inline int MPIDI_SHM_am_recv(MPIR_Request * req) +{ + MPIR_Assert(0); + return 0; +} + +#endif /* SHM_STUBSHM_AM_H_INCLUDED */ diff --git a/src/mpid/ch4/shm/stubshm/stubshm_coll.h b/src/mpid/ch4/shm/stubshm/stubshm_coll.h new file mode 100644 index 0000000..73ef4e8 --- /dev/null +++ b/src/mpid/ch4/shm/stubshm/stubshm_coll.h @@ -0,0 +1,695 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef SHM_STUBSHM_COLL_H_INCLUDED +#define SHM_STUBSHM_COLL_H_INCLUDED + +#include "stubshm_impl.h" +#include "ch4_impl.h" + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_barrier +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_barrier(MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_Assert(0); + + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_bcast +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_bcast(void *buffer, int count, MPI_Datatype datatype, + int root, MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_Assert(0); + + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_allreduce +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_allreduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_Assert(0); + + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_allgather +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_allgather(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_Assert(0); + + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_allgatherv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_allgatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, const int *displs, + MPI_Datatype recvtype, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_Assert(0); + + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_gather +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_gather(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + int root, MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_Assert(0); + + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_gatherv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_gatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, const int *displs, + MPI_Datatype recvtype, int root, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_Assert(0); + + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_scatter +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_scatter(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + int root, MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_Assert(0); + + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_scatterv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_scatterv(const void *sendbuf, const int *sendcounts, + const int *displs, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + int root, MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_Assert(0); + + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_alltoall +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_alltoall(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_Assert(0); + + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_alltoallv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_alltoallv(const void *sendbuf, const int *sendcounts, + const int *sdispls, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, + const int *rdispls, MPI_Datatype recvtype, + MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_Assert(0); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_alltoallw +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_alltoallw(const void *sendbuf, const int sendcounts[], + const int sdispls[], const MPI_Datatype sendtypes[], + void *recvbuf, const int recvcounts[], + const int rdispls[], const MPI_Datatype recvtypes[], + MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Assert(0); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_reduce +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_reduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, int root, + MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_Assert(0); + + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_reduce_scatter +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_reduce_scatter(const void *sendbuf, void *recvbuf, + const int recvcounts[], MPI_Datatype datatype, + MPI_Op op, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Assert(0); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_reduce_scatter_block +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_reduce_scatter_block(const void *sendbuf, void *recvbuf, + int recvcount, MPI_Datatype datatype, + MPI_Op op, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Assert(0); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_scan +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_scan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_Assert(0); + + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_exscan +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_exscan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_Assert(0); + + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_neighbor_allgather +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_neighbor_allgather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, int recvcount, + MPI_Datatype recvtype, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_Assert(0); + + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_neighbor_allgatherv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_neighbor_allgatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int recvcounts[], const int displs[], + MPI_Datatype recvtype, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_Assert(0); + + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_neighbor_alltoall +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_neighbor_alltoall(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, int recvcount, + MPI_Datatype recvtype, MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_Assert(0); + + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_neighbor_alltoallv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_neighbor_alltoallv(const void *sendbuf, const int sendcounts[], + const int sdispls[], MPI_Datatype sendtype, + void *recvbuf, const int recvcounts[], + const int rdispls[], MPI_Datatype recvtype, + MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_Assert(0); + + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_neighbor_alltoallw +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_neighbor_alltoallw(const void *sendbuf, const int sendcounts[], + const MPI_Aint sdispls[], + const MPI_Datatype sendtypes[], void *recvbuf, + const int recvcounts[], const MPI_Aint rdispls[], + const MPI_Datatype recvtypes[], MPIR_Comm * comm_ptr, + MPIR_Errflag_t * errflag) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_Assert(0); + + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_ineighbor_allgather +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_ineighbor_allgather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, int recvcount, + MPI_Datatype recvtype, MPIR_Comm * comm_ptr, + MPI_Request * req) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_Assert(0); + + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_ineighbor_allgatherv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_ineighbor_allgatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int recvcounts[], const int displs[], + MPI_Datatype recvtype, MPIR_Comm * comm_ptr, + MPI_Request * req) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_Assert(0); + + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_ineighbor_alltoall +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_ineighbor_alltoall(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_Assert(0); + + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_ineighbor_alltoallv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_ineighbor_alltoallv(const void *sendbuf, const int sendcounts[], + const int sdispls[], MPI_Datatype sendtype, + void *recvbuf, const int recvcounts[], + const int rdispls[], MPI_Datatype recvtype, + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_Assert(0); + + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_ineighbor_alltoallw +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_ineighbor_alltoallw(const void *sendbuf, const int sendcounts[], + const MPI_Aint sdispls[], + const MPI_Datatype sendtypes[], void *recvbuf, + const int recvcounts[], const MPI_Aint rdispls[], + const MPI_Datatype recvtypes[], + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_Assert(0); + + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_ibarrier +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_ibarrier(MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_Assert(0); + + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_ibcast +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_ibcast(void *buffer, int count, MPI_Datatype datatype, + int root, MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_Assert(0); + + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_iallgather +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_iallgather(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_Assert(0); + + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_iallgatherv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_iallgatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, const int *displs, + MPI_Datatype recvtype, MPIR_Comm * comm_ptr, + MPI_Request * req) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_Assert(0); + + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_ialltoall +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_ialltoall(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_Assert(0); + + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_ialltoallv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_ialltoallv(const void *sendbuf, const int *sendcounts, + const int *sdispls, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, + const int *rdispls, MPI_Datatype recvtype, + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_Assert(0); + + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_ialltoallw +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_ialltoallw(const void *sendbuf, const int *sendcounts, + const int *sdispls, const MPI_Datatype sendtypes[], + void *recvbuf, const int *recvcounts, + const int *rdispls, const MPI_Datatype recvtypes[], + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Assert(0); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_iexscan +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_iexscan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm_ptr, + MPI_Request * req) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_Assert(0); + + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_igather +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_igather(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + int root, MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_Assert(0); + + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_igatherv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_igatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, const int *displs, + MPI_Datatype recvtype, int root, MPIR_Comm * comm_ptr, + MPI_Request * req) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_Assert(0); + + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_ireduce_scatter_block +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_ireduce_scatter_block(const void *sendbuf, void *recvbuf, + int recvcount, MPI_Datatype datatype, + MPI_Op op, MPIR_Comm * comm_ptr, + MPI_Request * req) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_Assert(0); + + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_ireduce_scatter +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_ireduce_scatter(const void *sendbuf, void *recvbuf, + const int recvcounts[], MPI_Datatype datatype, + MPI_Op op, MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_Assert(0); + + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_ireduce +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_ireduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, int root, + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_Assert(0); + + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_iallreduce +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_iallreduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm_ptr, MPI_Request * req) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_Assert(0); + + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_iscan +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_iscan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm_ptr, + MPI_Request * req) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_Assert(0); + + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_iscatter +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_iscatter(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + int root, MPIR_Comm * comm, MPI_Request * request) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Assert(0); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_SHM_iscatterv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_SHM_iscatterv(const void *sendbuf, const int *sendcounts, + const int *displs, MPI_Datatype sendtype, + void *recvbuf, int recvcount, + MPI_Datatype recvtype, int root, + MPIR_Comm * comm, MPI_Request * request) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Assert(0); + return mpi_errno; +} + +#endif /* SHM_STUBSHM_COLL_H_INCLUDED */ diff --git a/src/mpid/ch4/shm/stubshm/stubshm_comm.h b/src/mpid/ch4/shm/stubshm/stubshm_comm.h new file mode 100644 index 0000000..db0b3a6 --- /dev/null +++ b/src/mpid/ch4/shm/stubshm/stubshm_comm.h @@ -0,0 +1,29 @@ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef SHM_STUBSHM_COMM_H_INCLUDED +#define SHM_STUBSHM_COMM_H_INCLUDED + +#include "stubshm_impl.h" +static inline int MPIDI_SHM_comm_create(MPIR_Comm * comm) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Assert(0); + return mpi_errno; +} + +static inline int MPIDI_SHM_comm_destroy(MPIR_Comm * comm) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Assert(0); + return mpi_errno; +} + + +#endif /* SHM_STUBSHM_COMM_H_INCLUDED */ diff --git a/src/mpid/ch4/shm/stubshm/stubshm_impl.h b/src/mpid/ch4/shm/stubshm/stubshm_impl.h new file mode 100644 index 0000000..a0b274c --- /dev/null +++ b/src/mpid/ch4/shm/stubshm/stubshm_impl.h @@ -0,0 +1,14 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef SHM_STUBSHM_IMPL_H_INCLUDED +#define SHM_STUBSHM_IMPL_H_INCLUDED + +#endif /* SHM_STUBSHM_IMPL_H_INCLUDED */ diff --git a/src/mpid/ch4/shm/stubshm/stubshm_init.h b/src/mpid/ch4/shm/stubshm/stubshm_init.h new file mode 100644 index 0000000..cfe95fe --- /dev/null +++ b/src/mpid/ch4/shm/stubshm/stubshm_init.h @@ -0,0 +1,85 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef SHM_STUBSHM_INIT_H_INCLUDED +#define SHM_STUBSHM_INIT_H_INCLUDED + +#include "stubshm_impl.h" + +static inline int MPIDI_SHM_init(int rank, int size) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_finalize(void) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline void *MPIDI_SHM_alloc_mem(size_t size, MPIR_Info * info_ptr) +{ + MPIR_Assert(0); + return NULL; +} + +static inline int MPIDI_SHM_free_mem(void *ptr) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_comm_get_lpid(MPIR_Comm * comm_ptr, + int idx, int *lpid_ptr, MPL_bool is_remote) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_gpid_get(MPIR_Comm * comm_ptr, int rank, MPIR_Gpid * gpid) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_get_node_id(MPIR_Comm * comm, int rank, MPID_Node_id_t * id_p) +{ + *id_p = (MPID_Node_id_t) 0; + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_get_max_node_id(MPIR_Comm * comm, MPID_Node_id_t * max_id_p) +{ + *max_id_p = (MPID_Node_id_t) 1; + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_getallincomm(MPIR_Comm * comm_ptr, + int local_size, MPIR_Gpid local_gpids[], int *singleAVT) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_gpid_tolpidarray(int size, MPIR_Gpid gpid[], int lpid[]) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_create_intercomm_from_lpids(MPIR_Comm * newcomm_ptr, + int size, const int lpids[]) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +#endif /* SHM_STUBSHM_INIT_H_INCLUDED */ diff --git a/src/mpid/ch4/shm/stubshm/stubshm_pre.h b/src/mpid/ch4/shm/stubshm/stubshm_pre.h new file mode 100644 index 0000000..fc0f625 --- /dev/null +++ b/src/mpid/ch4/shm/stubshm/stubshm_pre.h @@ -0,0 +1,27 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ + +#ifndef SHM_STUBSHM_PRE_H_INCLUDED +#define SHM_STUBSHM_PRE_H_INCLUDED + +typedef struct { + int dummy; +} MPIDI_STUBSHM_am_request_t; + +typedef struct { + int dummy; +} MPIDI_STUBSHM_request_t; + +typedef struct MPIDI_STUBSHM_comm_t { + int dummy; +} MPIDI_STUBSHM_comm_t; + +#endif /* SHM_STUBSHM_PRE_H_INCLUDED */ diff --git a/src/mpid/ch4/shm/stubshm/stubshm_probe.h b/src/mpid/ch4/shm/stubshm/stubshm_probe.h new file mode 100644 index 0000000..0ba72f6 --- /dev/null +++ b/src/mpid/ch4/shm/stubshm/stubshm_probe.h @@ -0,0 +1,36 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef SHM_STUBSHM_PROBE_H_INCLUDED +#define SHM_STUBSHM_PROBE_H_INCLUDED + +#include "stubshm_impl.h" + + +static inline int MPIDI_SHM_improbe(int source, + int tag, + MPIR_Comm * comm, + int context_offset, + int *flag, MPIR_Request ** message, MPI_Status * status) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_iprobe(int source, + int tag, + MPIR_Comm * comm, + int context_offset, int *flag, MPI_Status * status) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +#endif /* SHM_STUBSHM_PROBE_H_INCLUDED */ diff --git a/src/mpid/ch4/shm/stubshm/stubshm_proc.h b/src/mpid/ch4/shm/stubshm/stubshm_proc.h new file mode 100644 index 0000000..984002e --- /dev/null +++ b/src/mpid/ch4/shm/stubshm/stubshm_proc.h @@ -0,0 +1,21 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef SHM_STUBSHM_PROC_H_INCLUDED +#define SHM_STUBSHM_PROC_H_INCLUDED + +#include "stubshm_impl.h" + +static inline int MPIDI_SHM_rank_is_local(int rank, MPIR_Comm * comm) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} +#endif /* SHM_STUBSHM_PROC_H_INCLUDED */ diff --git a/src/mpid/ch4/shm/stubshm/stubshm_progress.h b/src/mpid/ch4/shm/stubshm/stubshm_progress.h new file mode 100644 index 0000000..5c39ce7 --- /dev/null +++ b/src/mpid/ch4/shm/stubshm/stubshm_progress.h @@ -0,0 +1,88 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef SHM_STUBSHM_PROGRESS_H_INCLUDED +#define SHM_STUBSHM_PROGRESS_H_INCLUDED + +#include "stubshm_impl.h" + +static inline int MPIDI_SHM_do_progress_recv(int blocking, int *completion_count) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_do_progress_send(int blocking, int *completion_count) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_progress(int blocking) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_progress_test(void) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_progress_poke(void) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline void MPIDI_SHM_progress_start(MPID_Progress_state * state) +{ + MPIR_Assert(0); + return; +} + +static inline void MPIDI_SHM_progress_end(MPID_Progress_state * state) +{ + MPIR_Assert(0); + return; +} + +static inline int MPIDI_SHM_progress_wait(MPID_Progress_state * state) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_progress_register(int (*progress_fn) (int *)) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_progress_deregister(int id) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_progress_activate(int id) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_progress_deactivate(int id) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +#endif /* SHM_STUBSHM_PROGRESS_H_INCLUDED */ diff --git a/src/mpid/ch4/shm/stubshm/stubshm_recv.h b/src/mpid/ch4/shm/stubshm/stubshm_recv.h new file mode 100644 index 0000000..63eaf7a --- /dev/null +++ b/src/mpid/ch4/shm/stubshm/stubshm_recv.h @@ -0,0 +1,70 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef SHM_STUBSHM_RECV_H_INCLUDED +#define SHM_STUBSHM_RECV_H_INCLUDED + +#include "stubshm_impl.h" + +#undef FCNAME +#define FCNAME DECL_FUNC(MPIDI_SHM_recv) +static inline int MPIDI_SHM_recv(void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, + int context_offset, MPI_Status * status, MPIR_Request ** request) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_recv_init(void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + + +static inline int MPIDI_SHM_imrecv(void *buf, + int count, + MPI_Datatype datatype, + MPIR_Request * message, MPIR_Request ** rreqp) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +#undef FCNAME +#define FCNAME DECL_FUNC(MPIDI_SHM_irecv) +static inline int MPIDI_SHM_irecv(void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_cancel_recv(MPIR_Request * rreq) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +#endif /* SHM_STUBSHM_RECV_H_INCLUDED */ diff --git a/src/mpid/ch4/shm/stubshm/stubshm_request.h b/src/mpid/ch4/shm/stubshm/stubshm_request.h new file mode 100644 index 0000000..edbb381 --- /dev/null +++ b/src/mpid/ch4/shm/stubshm/stubshm_request.h @@ -0,0 +1,26 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef SHM_STUBSHM_REQUEST_H_INCLUDED +#define SHM_STUBSHM_REQUEST_H_INCLUDED + +#include "stubshm_impl.h" + +static inline void MPIDI_SHM_am_request_init(MPIR_Request * req) +{ + MPIR_Assert(0); +} + +static inline void MPIDI_SHM_am_request_finalize(MPIR_Request * req) +{ + MPIR_Assert(0); +} + +#endif /* SHM_STUBSHM_REQUEST_H_INCLUDED */ diff --git a/src/mpid/ch4/shm/stubshm/stubshm_rma.h b/src/mpid/ch4/shm/stubshm/stubshm_rma.h new file mode 100644 index 0000000..34256d1 --- /dev/null +++ b/src/mpid/ch4/shm/stubshm/stubshm_rma.h @@ -0,0 +1,143 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef SHM_STUBSHM_RMA_H_INCLUDED +#define SHM_STUBSHM_RMA_H_INCLUDED + +#include "stubshm_impl.h" + +static inline int MPIDI_SHM_put(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, MPI_Datatype target_datatype, MPIR_Win * win) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_get(void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, MPI_Datatype target_datatype, MPIR_Win * win) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_rput(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPIR_Win * win, MPIR_Request ** request) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + + +static inline int MPIDI_SHM_compare_and_swap(const void *origin_addr, + const void *compare_addr, + void *result_addr, + MPI_Datatype datatype, + int target_rank, MPI_Aint target_disp, MPIR_Win * win) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_raccumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPI_Op op, MPIR_Win * win, MPIR_Request ** request) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_rget_accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + void *result_addr, + int result_count, + MPI_Datatype result_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPI_Op op, MPIR_Win * win, MPIR_Request ** request) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_fetch_and_op(const void *origin_addr, + void *result_addr, + MPI_Datatype datatype, + int target_rank, + MPI_Aint target_disp, MPI_Op op, MPIR_Win * win) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + + +static inline int MPIDI_SHM_rget(void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPIR_Win * win, MPIR_Request ** request) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + + +static inline int MPIDI_SHM_get_accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + void *result_addr, + int result_count, + MPI_Datatype result_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, MPI_Op op, MPIR_Win * win) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, MPI_Op op, MPIR_Win * win) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +#endif /* SHM_STUBSHM_RMA_H_INCLUDED */ diff --git a/src/mpid/ch4/shm/stubshm/stubshm_send.h b/src/mpid/ch4/shm/stubshm/stubshm_send.h new file mode 100644 index 0000000..446959f --- /dev/null +++ b/src/mpid/ch4/shm/stubshm/stubshm_send.h @@ -0,0 +1,140 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef SHM_STUBSHM_SEND_H_INCLUDED +#define SHM_STUBSHM_SEND_H_INCLUDED + +#include "stubshm_impl.h" + +static inline int MPIDI_SHM_send(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + int err = MPI_SUCCESS; + MPIR_Assert(0); + return err; +} + + + + +static inline int MPIDI_SHM_irsend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_ssend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + int err = MPI_SUCCESS; + MPIR_Assert(0); + + return err; +} + +static inline int MPIDI_SHM_startall(int count, MPIR_Request * requests[]) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_send_init(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_ssend_init(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, + int context_offset, MPIR_Request ** request) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_bsend_init(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, + int context_offset, MPIR_Request ** request) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_rsend_init(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, + int context_offset, MPIR_Request ** request) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +#undef FCNAME +#define FCNAME DECL_FUNC(MPIDI_SHM_isend) +static inline int MPIDI_SHM_isend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + int err = MPI_SUCCESS; + MPIR_Assert(0); + return err; +} + +static inline int MPIDI_SHM_issend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + int err = MPI_SUCCESS; + MPIR_Assert(0); + return err; +} + +static inline int MPIDI_SHM_cancel_send(MPIR_Request * sreq) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +#endif /* SHM_STUBSHM_SEND_H_INCLUDED */ diff --git a/src/mpid/ch4/shm/stubshm/stubshm_spawn.h b/src/mpid/ch4/shm/stubshm/stubshm_spawn.h new file mode 100644 index 0000000..cedb3f4 --- /dev/null +++ b/src/mpid/ch4/shm/stubshm/stubshm_spawn.h @@ -0,0 +1,50 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef SHM_STUBSHM_SPAWN_H_INCLUDED +#define SHM_STUBSHM_SPAWN_H_INCLUDED + +#include "stubshm_impl.h" + +static inline int MPIDI_SHM_comm_connect(const char *port_name, + MPIR_Info * info, + int root, MPIR_Comm * comm, MPIR_Comm ** newcomm_ptr) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_comm_disconnect(MPIR_Comm * comm_ptr) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_open_port(MPIR_Info * info_ptr, char *port_name) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_close_port(const char *port_name) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_comm_accept(const char *port_name, + MPIR_Info * info, + int root, MPIR_Comm * comm, MPIR_Comm ** newcomm_ptr) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +#endif /* SHM_STUBSHM_SPAWN_H_INCLUDED */ diff --git a/src/mpid/ch4/shm/stubshm/stubshm_unimpl.h b/src/mpid/ch4/shm/stubshm/stubshm_unimpl.h new file mode 100644 index 0000000..db8e66b --- /dev/null +++ b/src/mpid/ch4/shm/stubshm/stubshm_unimpl.h @@ -0,0 +1,19 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ + +#ifdef __cplusplus +extern "C" { +#endif + + +#ifdef __cplusplus +} +#endif diff --git a/src/mpid/ch4/shm/stubshm/stubshm_win.h b/src/mpid/ch4/shm/stubshm/stubshm_win.h new file mode 100644 index 0000000..24a9b1b --- /dev/null +++ b/src/mpid/ch4/shm/stubshm/stubshm_win.h @@ -0,0 +1,185 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef SHM_STUBSHM_WIN_H_INCLUDED +#define SHM_STUBSHM_WIN_H_INCLUDED + +#include "stubshm_impl.h" + +static inline int MPIDI_SHM_win_set_info(MPIR_Win * win, MPIR_Info * info) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + + +static inline int MPIDI_SHM_win_start(MPIR_Group * group, int assert, MPIR_Win * win) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + + +static inline int MPIDI_SHM_win_complete(MPIR_Win * win) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_win_post(MPIR_Group * group, int assert, MPIR_Win * win) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + + +static inline int MPIDI_SHM_win_wait(MPIR_Win * win) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + + +static inline int MPIDI_SHM_win_test(MPIR_Win * win, int *flag) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_win_lock(int lock_type, int rank, int assert, MPIR_Win * win) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + + +static inline int MPIDI_SHM_win_unlock(int rank, MPIR_Win * win) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_win_get_info(MPIR_Win * win, MPIR_Info ** info_p_p) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + + +static inline int MPIDI_SHM_win_free(MPIR_Win ** win_ptr) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_win_fence(int assert, MPIR_Win * win) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_win_create(void *base, + MPI_Aint length, + int disp_unit, + MPIR_Info * info, MPIR_Comm * comm_ptr, MPIR_Win ** win_ptr) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_win_attach(MPIR_Win * win, void *base, MPI_Aint size) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_win_allocate_shared(MPI_Aint size, + int disp_unit, + MPIR_Info * info_ptr, + MPIR_Comm * comm_ptr, + void **base_ptr, MPIR_Win ** win_ptr) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_win_detach(MPIR_Win * win, const void *base) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_win_shared_query(MPIR_Win * win, + int rank, + MPI_Aint * size, int *disp_unit, void *baseptr) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_win_allocate(MPI_Aint size, + int disp_unit, + MPIR_Info * info, + MPIR_Comm * comm, void *baseptr, MPIR_Win ** win) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_win_flush(int rank, MPIR_Win * win) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_win_flush_local_all(MPIR_Win * win) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_win_unlock_all(MPIR_Win * win) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_win_create_dynamic(MPIR_Info * info, MPIR_Comm * comm, MPIR_Win ** win) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_win_flush_local(int rank, MPIR_Win * win) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_win_sync(MPIR_Win * win) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_win_flush_all(MPIR_Win * win) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +static inline int MPIDI_SHM_win_lock_all(int assert, MPIR_Win * win) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + + +#endif /* SHM_STUBSHM_WIN_H_INCLUDED */ diff --git a/src/mpid/ch4/shm/stubshm/subconfigure.m4 b/src/mpid/ch4/shm/stubshm/subconfigure.m4 new file mode 100644 index 0000000..9e6d42a --- /dev/null +++ b/src/mpid/ch4/shm/stubshm/subconfigure.m4 @@ -0,0 +1,19 @@ +[#] start of __file__ +dnl MPICH_SUBCFG_AFTER=src/mpid/ch4 + +AC_DEFUN([PAC_SUBCFG_PREREQ_]PAC_SUBCFG_AUTO_SUFFIX,[ + AM_COND_IF([BUILD_CH4],[ + for shm in $ch4_shm ; do + AS_CASE([$shm],[stubshm],[build_ch4_shm_stubshm=yes]) + done + ]) + AM_CONDITIONAL([BUILD_SHM_STUBSHM],[test "X$build_ch4_shm_stubshm" = "Xyes"]) +])dnl + +AC_DEFUN([PAC_SUBCFG_BODY_]PAC_SUBCFG_AUTO_SUFFIX,[ +AM_COND_IF([BUILD_SHM_STUBSHM],[ + AC_MSG_NOTICE([RUNNING CONFIGURE FOR ch4:shm:stubshm]) +])dnl end AM_COND_IF(BUILD_SHM_STUBSHM,...) +])dnl end _BODY + +[#] end of __file__ diff --git a/src/mpid/ch4/src/Makefile.mk b/src/mpid/ch4/src/Makefile.mk new file mode 100644 index 0000000..db04a3b --- /dev/null +++ b/src/mpid/ch4/src/Makefile.mk @@ -0,0 +1,44 @@ +## -*- Mode: Makefile; -*- +## vim: set ft=automake : +## +## (C) 2016 by Argonne National Laboratory. +## See COPYRIGHT in top-level directory. +## +## Portions of this code were written by Intel Corporation. +## Copyright (C) 2011-2016 Intel Corporation. Intel provides this material +## to Argonne National Laboratory subject to Software Grant and Corporate +## Contributor License Agreement dated February 8, 2012. +## + +AM_CPPFLAGS += -I$(top_srcdir)/src/mpid/ch4/src + +noinst_HEADERS += src/mpid/ch4/src/ch4_comm.h \ + src/mpid/ch4/src/ch4_init.h \ + src/mpid/ch4/src/ch4_progress.h \ + src/mpid/ch4/src/ch4_request.h \ + src/mpid/ch4/src/ch4_send.h \ + src/mpid/ch4/src/ch4_types.h \ + src/mpid/ch4/src/ch4_impl.h \ + src/mpid/ch4/src/ch4_probe.h \ + src/mpid/ch4/src/ch4_proc.h \ + src/mpid/ch4/src/ch4_recv.h \ + src/mpid/ch4/src/ch4_rma.h \ + src/mpid/ch4/src/ch4_spawn.h \ + src/mpid/ch4/src/ch4_win.h \ + src/mpid/ch4/src/ch4r_probe.h \ + src/mpid/ch4/src/ch4r_recv.h \ + src/mpid/ch4/src/ch4r_rma.h \ + src/mpid/ch4/src/ch4r_win.h \ + src/mpid/ch4/src/ch4r_init.h \ + src/mpid/ch4/src/ch4r_proc.h \ + src/mpid/ch4/src/ch4i_comm.h \ + src/mpid/ch4/src/ch4r_recvq.h \ + src/mpid/ch4/src/ch4i_util.h \ + src/mpid/ch4/src/ch4r_symheap.h \ + src/mpid/ch4/src/ch4r_send.h \ + src/mpid/ch4/src/ch4r_buf.h \ + src/mpid/ch4/src/ch4r_request.h + +mpi_core_sources += src/mpid/ch4/src/ch4_globals.c \ + src/mpid/ch4/src/mpid_ch4_net_array.c \ + src/mpid/ch4/src/mpid_ch4_shm_array.c diff --git a/src/mpid/ch4/src/ch4_coll.h b/src/mpid/ch4/src/ch4_coll.h new file mode 100644 index 0000000..03dae8f --- /dev/null +++ b/src/mpid/ch4/src/ch4_coll.h @@ -0,0 +1,378 @@ +/* -*- Mode: C ; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef CH4_COLL_H_INCLUDED +#define CH4_COLL_H_INCLUDED + +#include "ch4_impl.h" +#include "ch4r_proc.h" + +__CH4_INLINE__ int MPIDI_Barrier(MPIR_Comm * comm, MPIR_Errflag_t * errflag) +{ + return MPIDI_NM_barrier(comm, errflag); +} + +__CH4_INLINE__ int MPIDI_Bcast(void *buffer, int count, MPI_Datatype datatype, + int root, MPIR_Comm * comm, MPIR_Errflag_t * errflag) +{ + return MPIDI_NM_bcast(buffer, count, datatype, root, comm, errflag); +} + +__CH4_INLINE__ int MPIDI_Allreduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm, + MPIR_Errflag_t * errflag) +{ + return MPIDI_NM_allreduce(sendbuf, recvbuf, count, datatype, op, comm, errflag); +} + +__CH4_INLINE__ int MPIDI_Allgather(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, MPIR_Errflag_t * errflag) +{ + return MPIDI_NM_allgather(sendbuf, sendcount, sendtype, recvbuf, + recvcount, recvtype, comm, errflag); +} + +__CH4_INLINE__ int MPIDI_Allgatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, const int *displs, + MPI_Datatype recvtype, MPIR_Comm * comm, + MPIR_Errflag_t * errflag) +{ + return MPIDI_NM_allgatherv(sendbuf, sendcount, sendtype, recvbuf, + recvcounts, displs, recvtype, comm, errflag); +} + +__CH4_INLINE__ int MPIDI_Scatter(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + int root, MPIR_Comm * comm, MPIR_Errflag_t * errflag) +{ + return MPIDI_NM_scatter(sendbuf, sendcount, sendtype, recvbuf, + recvcount, recvtype, root, comm, errflag); +} + +__CH4_INLINE__ int MPIDI_Scatterv(const void *sendbuf, const int *sendcounts, + const int *displs, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + int root, MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + return MPIDI_NM_scatterv(sendbuf, sendcounts, displs, sendtype, + recvbuf, recvcount, recvtype, root, comm_ptr, errflag); +} + +__CH4_INLINE__ int MPIDI_Gather(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + int root, MPIR_Comm * comm, MPIR_Errflag_t * errflag) +{ + return MPIDI_NM_gather(sendbuf, sendcount, sendtype, recvbuf, + recvcount, recvtype, root, comm, errflag); +} + +__CH4_INLINE__ int MPIDI_Gatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, + const int *displs, MPI_Datatype recvtype, + int root, MPIR_Comm * comm, MPIR_Errflag_t * errflag) +{ + return MPIDI_NM_gatherv(sendbuf, sendcount, sendtype, recvbuf, + recvcounts, displs, recvtype, root, comm, errflag); +} + +__CH4_INLINE__ int MPIDI_Alltoall(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, MPIR_Errflag_t * errflag) +{ + return MPIDI_NM_alltoall(sendbuf, sendcount, sendtype, recvbuf, + recvcount, recvtype, comm, errflag); +} + +__CH4_INLINE__ int MPIDI_Alltoallv(const void *sendbuf, const int *sendcounts, + const int *sdispls, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, + const int *rdispls, MPI_Datatype recvtype, + MPIR_Comm * comm, MPIR_Errflag_t * errflag) +{ + return MPIDI_NM_alltoallv(sendbuf, sendcounts, sdispls, sendtype, + recvbuf, recvcounts, rdispls, recvtype, comm, errflag); +} + +__CH4_INLINE__ int MPIDI_Alltoallw(const void *sendbuf, const int sendcounts[], + const int sdispls[], const MPI_Datatype sendtypes[], + void *recvbuf, const int recvcounts[], + const int rdispls[], const MPI_Datatype recvtypes[], + MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + return MPIDI_NM_alltoallw(sendbuf, sendcounts, sdispls, sendtypes, + recvbuf, recvcounts, rdispls, recvtypes, comm_ptr, errflag); +} + +__CH4_INLINE__ int MPIDI_Reduce(const void *sendbuf, void *recvbuf, + int count, MPI_Datatype datatype, MPI_Op op, + int root, MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + return MPIDI_NM_reduce(sendbuf, recvbuf, count, datatype, op, root, comm_ptr, errflag); +} + +__CH4_INLINE__ int MPIDI_Reduce_scatter(const void *sendbuf, void *recvbuf, + const int recvcounts[], MPI_Datatype datatype, + MPI_Op op, MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + return MPIDI_NM_reduce_scatter(sendbuf, recvbuf, recvcounts, datatype, op, comm_ptr, errflag); +} + +__CH4_INLINE__ int MPIDI_Reduce_scatter_block(const void *sendbuf, void *recvbuf, int recvcount, + MPI_Datatype datatype, MPI_Op op, + MPIR_Comm * comm_ptr, MPIR_Errflag_t * errflag) +{ + return MPIDI_NM_reduce_scatter_block(sendbuf, recvbuf, recvcount, + datatype, op, comm_ptr, errflag); +} + +__CH4_INLINE__ int MPIDI_Scan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm, + MPIR_Errflag_t * errflag) +{ + return MPIDI_NM_scan(sendbuf, recvbuf, count, datatype, op, comm, errflag); +} + +__CH4_INLINE__ int MPIDI_Exscan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm, + MPIR_Errflag_t * errflag) +{ + return MPIDI_NM_exscan(sendbuf, recvbuf, count, datatype, op, comm, errflag); +} + +__CH4_INLINE__ int MPIDI_Neighbor_allgather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, int recvcount, + MPI_Datatype recvtype, MPIR_Comm * comm) +{ + return MPIDI_NM_neighbor_allgather(sendbuf, sendcount, sendtype, + recvbuf, recvcount, recvtype, comm); +} + +__CH4_INLINE__ int MPIDI_Neighbor_allgatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, const int *displs, + MPI_Datatype recvtype, MPIR_Comm * comm) +{ + return MPIDI_NM_neighbor_allgatherv(sendbuf, sendcount, sendtype, + recvbuf, recvcounts, displs, recvtype, comm); +} + +__CH4_INLINE__ int MPIDI_Neighbor_alltoallv(const void *sendbuf, const int *sendcounts, + const int *sdispls, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, + const int *rdispls, MPI_Datatype recvtype, + MPIR_Comm * comm) +{ + return MPIDI_NM_neighbor_alltoallv(sendbuf, sendcounts, sdispls, + sendtype, recvbuf, recvcounts, rdispls, recvtype, comm); +} + +__CH4_INLINE__ int MPIDI_Neighbor_alltoallw(const void *sendbuf, const int *sendcounts, + const MPI_Aint * sdispls, + const MPI_Datatype * sendtypes, void *recvbuf, + const int *recvcounts, const MPI_Aint * rdispls, + const MPI_Datatype * recvtypes, MPIR_Comm * comm) +{ + return MPIDI_NM_neighbor_alltoallw(sendbuf, sendcounts, sdispls, + sendtypes, recvbuf, recvcounts, rdispls, recvtypes, comm); +} + +__CH4_INLINE__ int MPIDI_Neighbor_alltoall(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, int recvcount, + MPI_Datatype recvtype, MPIR_Comm * comm) +{ + return MPIDI_NM_neighbor_alltoall(sendbuf, sendcount, sendtype, + recvbuf, recvcount, recvtype, comm); +} + +__CH4_INLINE__ int MPIDI_Ineighbor_allgather(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, MPI_Request * req) +{ + return MPIDI_NM_ineighbor_allgather(sendbuf, sendcount, sendtype, + recvbuf, recvcount, recvtype, comm, req); +} + +__CH4_INLINE__ int MPIDI_Ineighbor_allgatherv(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + const int *recvcounts, const int *displs, + MPI_Datatype recvtype, MPIR_Comm * comm, + MPI_Request * req) +{ + return MPIDI_NM_ineighbor_allgatherv(sendbuf, sendcount, sendtype, + recvbuf, recvcounts, displs, recvtype, comm, req); +} + +__CH4_INLINE__ int MPIDI_Ineighbor_alltoall(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, + int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, MPI_Request * req) +{ + return MPIDI_NM_ineighbor_alltoall(sendbuf, sendcount, sendtype, + recvbuf, recvcount, recvtype, comm, req); +} + +__CH4_INLINE__ int MPIDI_Ineighbor_alltoallv(const void *sendbuf, const int *sendcounts, + const int *sdispls, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, + const int *rdispls, MPI_Datatype recvtype, + MPIR_Comm * comm, MPI_Request * req) +{ + return MPIDI_NM_ineighbor_alltoallv(sendbuf, sendcounts, sdispls, + sendtype, recvbuf, recvcounts, rdispls, recvtype, comm, + req); +} + +__CH4_INLINE__ int MPIDI_Ineighbor_alltoallw(const void *sendbuf, const int *sendcounts, + const MPI_Aint * sdispls, + const MPI_Datatype * sendtypes, void *recvbuf, + const int *recvcounts, const MPI_Aint * rdispls, + const MPI_Datatype * recvtypes, MPIR_Comm * comm, + MPI_Request * req) +{ + return MPIDI_NM_ineighbor_alltoallw(sendbuf, sendcounts, sdispls, + sendtypes, recvbuf, recvcounts, rdispls, recvtypes, comm, + req); +} + +__CH4_INLINE__ int MPIDI_Ibarrier(MPIR_Comm * comm, MPI_Request * req) +{ + return MPIDI_NM_ibarrier(comm, req); +} + +__CH4_INLINE__ int MPIDI_Ibcast(void *buffer, int count, MPI_Datatype datatype, + int root, MPIR_Comm * comm, MPI_Request * req) +{ + return MPIDI_NM_ibcast(buffer, count, datatype, root, comm, req); +} + +__CH4_INLINE__ int MPIDI_Iallgather(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, MPI_Request * req) +{ + return MPIDI_NM_iallgather(sendbuf, sendcount, sendtype, recvbuf, + recvcount, recvtype, comm, req); +} + +__CH4_INLINE__ int MPIDI_Iallgatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, const int *displs, + MPI_Datatype recvtype, MPIR_Comm * comm, MPI_Request * req) +{ + return MPIDI_NM_iallgatherv(sendbuf, sendcount, sendtype, recvbuf, + recvcounts, displs, recvtype, comm, req); +} + +__CH4_INLINE__ int MPIDI_Iallreduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm, + MPI_Request * req) +{ + return MPIDI_NM_iallreduce(sendbuf, recvbuf, count, datatype, op, comm, req); +} + +__CH4_INLINE__ int MPIDI_Ialltoall(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + MPIR_Comm * comm, MPI_Request * req) +{ + return MPIDI_NM_ialltoall(sendbuf, sendcount, sendtype, recvbuf, + recvcount, recvtype, comm, req); +} + +__CH4_INLINE__ int MPIDI_Ialltoallv(const void *sendbuf, const int *sendcounts, + const int *sdispls, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, + const int *rdispls, MPI_Datatype recvtype, + MPIR_Comm * comm, MPI_Request * req) +{ + return MPIDI_NM_ialltoallv(sendbuf, sendcounts, sdispls, sendtype, + recvbuf, recvcounts, rdispls, recvtype, comm, req); +} + +__CH4_INLINE__ int MPIDI_Ialltoallw(const void *sendbuf, const int *sendcounts, + const int *sdispls, const MPI_Datatype * sendtypes, + void *recvbuf, const int *recvcounts, + const int *rdispls, const MPI_Datatype * recvtypes, + MPIR_Comm * comm, MPI_Request * req) +{ + return MPIDI_NM_ialltoallw(sendbuf, sendcounts, sdispls, sendtypes, + recvbuf, recvcounts, rdispls, recvtypes, comm, req); +} + +__CH4_INLINE__ int MPIDI_Iexscan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm, + MPI_Request * req) +{ + return MPIDI_NM_iexscan(sendbuf, recvbuf, count, datatype, op, comm, req); +} + +__CH4_INLINE__ int MPIDI_Igather(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + int root, MPIR_Comm * comm, MPI_Request * req) +{ + return MPIDI_NM_igather(sendbuf, sendcount, sendtype, recvbuf, + recvcount, recvtype, root, comm, req); +} + +__CH4_INLINE__ int MPIDI_Igatherv(const void *sendbuf, int sendcount, MPI_Datatype sendtype, + void *recvbuf, const int *recvcounts, const int *displs, + MPI_Datatype recvtype, int root, MPIR_Comm * comm, + MPI_Request * req) +{ + return MPIDI_NM_igatherv(sendbuf, sendcount, sendtype, recvbuf, + recvcounts, displs, recvtype, root, comm, req); +} + +__CH4_INLINE__ int MPIDI_Ireduce_scatter_block(const void *sendbuf, void *recvbuf, int recvcount, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm, + MPI_Request * req) +{ + return MPIDI_NM_ireduce_scatter_block(sendbuf, recvbuf, recvcount, datatype, op, comm, req); +} + +__CH4_INLINE__ int MPIDI_Ireduce_scatter(const void *sendbuf, void *recvbuf, + const int *recvcounts, MPI_Datatype datatype, + MPI_Op op, MPIR_Comm * comm, MPI_Request * req) +{ + return MPIDI_NM_ireduce_scatter(sendbuf, recvbuf, recvcounts, datatype, op, comm, req); +} + +__CH4_INLINE__ int MPIDI_Ireduce(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, int root, + MPIR_Comm * comm, MPI_Request * req) +{ + return MPIDI_NM_ireduce(sendbuf, recvbuf, count, datatype, op, root, comm, req); +} + +__CH4_INLINE__ int MPIDI_Iscan(const void *sendbuf, void *recvbuf, int count, + MPI_Datatype datatype, MPI_Op op, MPIR_Comm * comm, + MPI_Request * req) +{ + return MPIDI_NM_iscan(sendbuf, recvbuf, count, datatype, op, comm, req); +} + +__CH4_INLINE__ int MPIDI_Iscatter(const void *sendbuf, int sendcount, + MPI_Datatype sendtype, void *recvbuf, int recvcount, + MPI_Datatype recvtype, int root, MPIR_Comm * comm, + MPI_Request * req) +{ + return MPIDI_NM_iscatter(sendbuf, sendcount, sendtype, recvbuf, + recvcount, recvtype, root, comm, req); +} + +__CH4_INLINE__ int MPIDI_Iscatterv(const void *sendbuf, const int *sendcounts, + const int *displs, MPI_Datatype sendtype, + void *recvbuf, int recvcount, MPI_Datatype recvtype, + int root, MPIR_Comm * comm, MPI_Request * req) +{ + return MPIDI_NM_iscatterv(sendbuf, sendcounts, displs, sendtype, + recvbuf, recvcount, recvtype, root, comm, req); +} + +#endif /* CH4_COLL_H_INCLUDED */ diff --git a/src/mpid/ch4/src/ch4_comm.h b/src/mpid/ch4/src/ch4_comm.h new file mode 100644 index 0000000..695566d --- /dev/null +++ b/src/mpid/ch4/src/ch4_comm.h @@ -0,0 +1,249 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef CH4_COMM_H_INCLUDED +#define CH4_COMM_H_INCLUDED + +#include "ch4_impl.h" +#include "ch4i_comm.h" + +__CH4_INLINE__ int MPIDI_Comm_AS_enabled(MPIR_Comm * comm) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +__CH4_INLINE__ int MPIDI_Comm_reenable_anysource(MPIR_Comm * comm, MPIR_Group ** failed_group_ptr) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +__CH4_INLINE__ int MPIDI_Comm_remote_group_failed(MPIR_Comm * comm, MPIR_Group ** failed_group_ptr) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +__CH4_INLINE__ int MPIDI_Comm_group_failed(MPIR_Comm * comm_ptr, MPIR_Group ** failed_group_ptr) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +__CH4_INLINE__ int MPIDI_Comm_failure_ack(MPIR_Comm * comm_ptr) +{ + MPIR_Assert(0); + return 0; +} + +__CH4_INLINE__ int MPIDI_Comm_failure_get_acked(MPIR_Comm * comm_ptr, + MPIR_Group ** failed_group_ptr) +{ + MPIR_Assert(0); + return 0; +} + +__CH4_INLINE__ int MPIDI_Comm_revoke(MPIR_Comm * comm_ptr, int is_remote) +{ + MPIR_Assert(0); + return 0; +} + +__CH4_INLINE__ int MPIDI_Comm_get_all_failed_procs(MPIR_Comm * comm_ptr, MPIR_Group ** failed_group, + int tag) +{ + MPIR_Assert(0); + return 0; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Comm_split_type +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Comm_split_type(MPIR_Comm * comm_ptr, + int split_type, + int key, MPIR_Info * info_ptr, MPIR_Comm ** newcomm_ptr) +{ + int mpi_errno = MPI_SUCCESS; + int idx; + MPID_Node_id_t node_id; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_COMM_SPLIT_TYPE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_COMM_SPLIT_TYPE); + + if (split_type == MPI_COMM_TYPE_SHARED) { + MPIDI_Comm_get_lpid(comm_ptr, comm_ptr->rank, &idx, FALSE); + MPIDI_Get_node_id(comm_ptr, comm_ptr->rank, &node_id); + mpi_errno = MPIR_Comm_split_impl(comm_ptr, node_id, key, newcomm_ptr); + } + else + mpi_errno = MPIR_Comm_split_impl(comm_ptr, MPI_UNDEFINED, key, newcomm_ptr); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_COMM_SPLIT_TYPE); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Comm_create +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Comm_create(MPIR_Comm * comm) +{ + int mpi_errno; + int i, *uniq_avtids; + int max_n_avts; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_COMM_CREATE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_COMM_CREATE); + mpi_errno = MPIDI_NM_comm_create(comm); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } +#if defined(MPIDI_BUILD_CH4_SHM) + mpi_errno = MPIDI_SHM_comm_create(comm); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } +#endif + + /* comm_world and comm_self are already initialized */ + if (comm != MPIR_Process.comm_world && comm != MPIR_Process.comm_self) { + MPIDII_comm_create_rank_map(comm); + /* add ref to avts */ + switch (MPIDII_COMM(comm, map).mode) { + case MPIDII_RANK_MAP_NONE: + break; + case MPIDII_RANK_MAP_MLUT: + max_n_avts = MPIDIU_get_max_n_avts(); + uniq_avtids = (int *) MPL_malloc(max_n_avts * sizeof(int)); + memset(uniq_avtids, 0, max_n_avts); + for (i = 0; i < MPIDII_COMM(comm, map).size; i++) { + if (uniq_avtids[MPIDII_COMM(comm, map).irreg.mlut.gpid[i].avtid] == 0) { + uniq_avtids[MPIDII_COMM(comm, map).irreg.mlut.gpid[i].avtid] = 1; + MPIDIU_avt_add_ref(MPIDII_COMM(comm, map).irreg.mlut.gpid[i].avtid); + } + } + MPL_free(uniq_avtids); + break; + default: + MPIDIU_avt_add_ref(MPIDII_COMM(comm, map).avtid); + } + + switch (MPIDII_COMM(comm, local_map).mode) { + case MPIDII_RANK_MAP_NONE: + break; + case MPIDII_RANK_MAP_MLUT: + max_n_avts = MPIDIU_get_max_n_avts(); + uniq_avtids = (int *) MPL_malloc(max_n_avts * sizeof(int)); + memset(uniq_avtids, 0, max_n_avts); + for (i = 0; i < MPIDII_COMM(comm, local_map).size; i++) { + if (uniq_avtids[MPIDII_COMM(comm, local_map).irreg.mlut.gpid[i].avtid] == 0) { + uniq_avtids[MPIDII_COMM(comm, local_map).irreg.mlut.gpid[i].avtid] = 1; + MPIDIU_avt_add_ref(MPIDII_COMM(comm, local_map).irreg.mlut.gpid[i].avtid); + } + } + MPL_free(uniq_avtids); + break; + default: + MPIDIU_avt_add_ref(MPIDII_COMM(comm, local_map).avtid); + } + } + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_COMM_CREATE); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Comm_destroy +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Comm_destroy(MPIR_Comm * comm) +{ + int mpi_errno; + int i, *uniq_avtids; + int max_n_avts; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_COMM_DESTROY); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_COMM_DESTROY); + /* release ref to avts */ + switch (MPIDII_COMM(comm, map).mode) { + case MPIDII_RANK_MAP_NONE: + break; + case MPIDII_RANK_MAP_MLUT: + max_n_avts = MPIDIU_get_max_n_avts(); + uniq_avtids = (int *) MPL_malloc(max_n_avts * sizeof(int)); + memset(uniq_avtids, 0, max_n_avts); + for (i = 0; i < MPIDII_COMM(comm, map).size; i++) { + if (uniq_avtids[MPIDII_COMM(comm, map).irreg.mlut.gpid[i].avtid] == 0) { + uniq_avtids[MPIDII_COMM(comm, map).irreg.mlut.gpid[i].avtid] = 1; + MPIDIU_avt_release_ref(MPIDII_COMM(comm, map).irreg.mlut.gpid[i].avtid); + } + } + MPL_free(uniq_avtids); + break; + default: + MPIDIU_avt_release_ref(MPIDII_COMM(comm, map).avtid); + } + + switch (MPIDII_COMM(comm, local_map).mode) { + case MPIDII_RANK_MAP_NONE: + break; + case MPIDII_RANK_MAP_MLUT: + max_n_avts = MPIDIU_get_max_n_avts(); + uniq_avtids = (int *) MPL_malloc(max_n_avts * sizeof(int)); + memset(uniq_avtids, 0, max_n_avts); + for (i = 0; i < MPIDII_COMM(comm, local_map).size; i++) { + if (uniq_avtids[MPIDII_COMM(comm, local_map).irreg.mlut.gpid[i].avtid] == 0) { + uniq_avtids[MPIDII_COMM(comm, local_map).irreg.mlut.gpid[i].avtid] = 1; + MPIDIU_avt_release_ref(MPIDII_COMM(comm, local_map).irreg.mlut.gpid[i].avtid); + } + } + MPL_free(uniq_avtids); + break; + default: + MPIDIU_avt_release_ref(MPIDII_COMM(comm, local_map).avtid); + } + + mpi_errno = MPIDI_NM_comm_destroy(comm); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } +#if defined(MPIDI_BUILD_CH4_SHM) + mpi_errno = MPIDI_SHM_comm_destroy(comm); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } +#endif + + if (MPIDII_COMM(comm, map).mode == MPIDII_RANK_MAP_LUT + || MPIDII_COMM(comm, map).mode == MPIDII_RANK_MAP_LUT_INTRA) { + MPIDIU_release_lut(MPIDII_COMM(comm, map).irreg.lut.t); + } + if (MPIDII_COMM(comm, local_map).mode == MPIDII_RANK_MAP_LUT + || MPIDII_COMM(comm, local_map).mode == MPIDII_RANK_MAP_LUT_INTRA) { + MPIDIU_release_lut(MPIDII_COMM(comm, local_map).irreg.lut.t); + } + if (MPIDII_COMM(comm, map).mode == MPIDII_RANK_MAP_MLUT) { + MPIDIU_release_mlut(MPIDII_COMM(comm, map).irreg.mlut.t); + } + if (MPIDII_COMM(comm, local_map).mode == MPIDII_RANK_MAP_MLUT) { + MPIDIU_release_mlut(MPIDII_COMM(comm, local_map).irreg.mlut.t); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_COMM_DESTROY); + return mpi_errno; + fn_fail: + goto fn_exit; +} + + +#endif /* CH4_COMM_H_INCLUDED */ diff --git a/src/mpid/ch4/src/ch4_globals.c b/src/mpid/ch4/src/ch4_globals.c new file mode 100644 index 0000000..44ccbcd --- /dev/null +++ b/src/mpid/ch4/src/ch4_globals.c @@ -0,0 +1,62 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ + +/* All global ADI data structures need to go in this file */ +/* reference them with externs from other files */ + +#include <mpidimpl.h> +#include "ch4_impl.h" + +MPIDI_CH4_Global_t MPIDI_CH4_Global; +MPIDII_av_table_t **MPIDII_av_table; +MPIDII_av_table_t *MPIDII_av_table0; + +MPIDI_NM_funcs_t *MPIDI_NM_func; +MPIDI_NM_native_funcs_t *MPIDI_NM_native_func; + +#ifdef MPIDI_BUILD_CH4_SHM +MPIDI_SHM_funcs_t *MPIDI_SHM_func; +MPIDI_SHM_native_funcs_t *MPIDI_SHM_native_func; +#endif + +#ifdef MPID_DEVICE_DEFINES_THREAD_CS +pthread_mutex_t MPIDI_Mutex_lock[MPIDI_NUM_LOCKS]; +#endif + +/* The MPID_Abort ADI is strangely defined by the upper layers */ +/* We should fix the upper layer to define MPID_Abort like any */ +/* Other ADI */ +#ifdef MPID_Abort +#define MPID_TMP MPID_Abort +#undef MPID_Abort +int MPID_Abort(MPIR_Comm * comm, int mpi_errno, int exit_code, const char *error_msg) +{ + return MPIDI_Abort(comm, mpi_errno, exit_code, error_msg); +} + +#define MPID_Abort MPID_TMP +#endif + +/* Another weird ADI that doesn't follow convention */ +static void init_comm() __attribute__ ((constructor)); +static void init_comm() +{ + MPIR_Comm_fns = &MPIDI_CH4_Global.MPIR_Comm_fns_store; + MPIR_Comm_fns->split_type = MPIDI_Comm_split_type; +} + +MPL_dbg_class MPIDI_CH4_DBG_GENERAL; + +#if defined(MPL_USE_DBG_LOGGING) +MPL_dbg_class MPIDI_CH4_DBG_GENERAL; +MPL_dbg_class MPIDI_CH4_DBG_MAP; +MPL_dbg_class MPIDI_CH4_DBG_MEMORY; +#endif diff --git a/src/mpid/ch4/src/ch4_impl.h b/src/mpid/ch4/src/ch4_impl.h new file mode 100644 index 0000000..f8a8b75 --- /dev/null +++ b/src/mpid/ch4/src/ch4_impl.h @@ -0,0 +1,553 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef CH4_IMPL_H_INCLUDED +#define CH4_IMPL_H_INCLUDED + +#include "ch4_types.h" +#include <mpidch4.h> + +/* Static inlines */ +static inline int MPIDI_CH4U_get_tag(uint64_t match_bits) +{ + int tag = (match_bits & MPIDI_CH4U_TAG_MASK); + /* Left shift and right shift by MPIDI_CH4U_TAG_SHIFT_UNPACK is to make sure the sign of tag is retained */ + return ((tag << MPIDI_CH4U_TAG_SHIFT_UNPACK) >> MPIDI_CH4U_TAG_SHIFT_UNPACK); +} + +static inline int MPIDI_CH4U_get_context(uint64_t match_bits) +{ + return ((int) ((match_bits & MPIDI_CH4U_CONTEXT_MASK) >> + (MPIDI_CH4U_TAG_SHIFT + MPIDI_CH4U_SOURCE_SHIFT))); +} + +static inline int MPIDI_CH4U_get_context_index(uint64_t context_id) +{ + int raw_prefix, idx, bitpos, gen_id; + raw_prefix = MPIR_CONTEXT_READ_FIELD(PREFIX, context_id); + idx = raw_prefix / MPIR_CONTEXT_INT_BITS; + bitpos = raw_prefix % MPIR_CONTEXT_INT_BITS; + gen_id = (idx * MPIR_CONTEXT_INT_BITS) + (31 - bitpos); + return gen_id; +} + +static inline MPIR_Comm *MPIDI_CH4U_context_id_to_comm(uint64_t context_id) +{ + int comm_idx = MPIDI_CH4U_get_context_index(context_id); + int subcomm_type = MPIR_CONTEXT_READ_FIELD(SUBCOMM, context_id); + int is_localcomm = MPIR_CONTEXT_READ_FIELD(IS_LOCALCOMM, context_id); + MPIR_Assert(subcomm_type <= 3); + MPIR_Assert(is_localcomm <= 2); + return MPIDI_CH4_Global.comm_req_lists[comm_idx].comm[is_localcomm][subcomm_type]; +} + +static inline MPIDI_CH4U_rreq_t **MPIDI_CH4U_context_id_to_uelist(uint64_t context_id) +{ + int comm_idx = MPIDI_CH4U_get_context_index(context_id); + int subcomm_type = MPIR_CONTEXT_READ_FIELD(SUBCOMM, context_id); + int is_localcomm = MPIR_CONTEXT_READ_FIELD(IS_LOCALCOMM, context_id); + MPIR_Assert(subcomm_type <= 3); + MPIR_Assert(is_localcomm <= 2); + return &MPIDI_CH4_Global.comm_req_lists[comm_idx].uelist[is_localcomm][subcomm_type]; +} + +static inline uint64_t MPIDI_CH4U_generate_win_id(MPIR_Comm * comm_ptr) +{ + /* context id lower bits, window instance upper bits */ + return 1 + (((uint64_t) comm_ptr->context_id) | + ((uint64_t) ((MPIDI_CH4U_COMM(comm_ptr, window_instance))++) << 32)); +} + +static inline MPIR_Context_id_t MPIDI_CH4U_win_id_to_context(uint64_t win_id) +{ + /* pick the lower 32-bit to extract context id */ + return (win_id - 1) & 0xffffffff; +} + +static inline MPIR_Context_id_t MPIDI_CH4U_win_to_context(const MPIR_Win * win) +{ + return MPIDI_CH4U_win_id_to_context(MPIDI_CH4U_WIN(win, win_id)); +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_request_release +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ void MPIDI_CH4U_request_release(MPIR_Request * req) +{ + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4R_REQUEST_RELEASE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4R_REQEUST_RELEASE); + + if (req->kind == MPIR_REQUEST_KIND__PREQUEST_RECV && + NULL != MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(req)) { + MPIR_Request_free(MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(req)); + } + MPIR_Request_free(req); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4R_REQUEST_RELEASE); +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_request_complete +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ void MPIDI_CH4U_request_complete(MPIR_Request * req) +{ + int incomplete; + MPIR_cc_decr(req->cc_ptr, &incomplete); + if (!incomplete) + MPIDI_CH4U_request_release(req); +} + +#ifndef dtype_add_ref_if_not_builtin +#define dtype_add_ref_if_not_builtin(datatype_) \ + do { \ + if ((datatype_) != MPI_DATATYPE_NULL && \ + HANDLE_GET_KIND((datatype_)) != HANDLE_KIND_BUILTIN) \ + { \ + MPIR_Datatype *dtp_ = NULL; \ + MPID_Datatype_get_ptr((datatype_), dtp_); \ + MPID_Datatype_add_ref(dtp_); \ + } \ + } while (0) +#endif + +#ifndef dtype_release_if_not_builtin +#define dtype_release_if_not_builtin(datatype_) \ + do { \ + if ((datatype_) != MPI_DATATYPE_NULL && \ + HANDLE_GET_KIND((datatype_)) != HANDLE_KIND_BUILTIN) \ + { \ + MPIR_Datatype *dtp_ = NULL; \ + MPID_Datatype_get_ptr((datatype_), dtp_); \ + MPID_Datatype_release(dtp_); \ + } \ + } while (0) +#endif + +#define MPIDI_Datatype_get_info(_count, _datatype, \ + _dt_contig_out, _data_sz_out, \ + _dt_ptr, _dt_true_lb) \ + do { \ + if (IS_BUILTIN(_datatype)) \ + { \ + (_dt_ptr) = NULL; \ + (_dt_contig_out) = TRUE; \ + (_dt_true_lb) = 0; \ + (_data_sz_out) = (size_t)(_count) * \ + MPID_Datatype_get_basic_size(_datatype); \ + } \ + else \ + { \ + MPID_Datatype_get_ptr((_datatype), (_dt_ptr)); \ + if (_dt_ptr) \ + { \ + (_dt_contig_out) = (_dt_ptr)->is_contig; \ + (_dt_true_lb) = (_dt_ptr)->true_lb; \ + (_data_sz_out) = (size_t)(_count) * \ + (_dt_ptr)->size; \ + } \ + else \ + { \ + (_dt_contig_out) = 1; \ + (_dt_true_lb) = 0; \ + (_data_sz_out) = 0; \ + } \ + } \ + } while (0) + +#define MPIDI_Datatype_get_size_dt_ptr(_count, _datatype, \ + _data_sz_out, _dt_ptr) \ + do { \ + if (IS_BUILTIN(_datatype)) \ + { \ + (_dt_ptr) = NULL; \ + (_data_sz_out) = (size_t)(_count) * \ + MPID_Datatype_get_basic_size(_datatype); \ + } \ + else \ + { \ + MPID_Datatype_get_ptr((_datatype), (_dt_ptr)); \ + (_data_sz_out) = (_dt_ptr) ? (size_t)(_count) * \ + (_dt_ptr)->size : 0; \ + } \ + } while (0) + +#define MPIDI_Datatype_check_contig(_datatype,_dt_contig_out) \ + do { \ + if (IS_BUILTIN(_datatype)) \ + { \ + (_dt_contig_out) = TRUE; \ + } \ + else \ + { \ + MPIR_Datatype *_dt_ptr; \ + MPID_Datatype_get_ptr((_datatype), (_dt_ptr)); \ + (_dt_contig_out) = (_dt_ptr) ? (_dt_ptr)->is_contig : 1; \ + } \ + } while (0) + +#define MPIDI_Datatype_check_contig_size(_datatype,_count, \ + _dt_contig_out, \ + _data_sz_out) \ + do { \ + if (IS_BUILTIN(_datatype)) \ + { \ + (_dt_contig_out) = TRUE; \ + (_data_sz_out) = (size_t)(_count) * \ + MPID_Datatype_get_basic_size(_datatype); \ + } \ + else \ + { \ + MPIR_Datatype *_dt_ptr; \ + MPID_Datatype_get_ptr((_datatype), (_dt_ptr)); \ + if (_dt_ptr) \ + { \ + (_dt_contig_out) = (_dt_ptr)->is_contig; \ + (_data_sz_out) = (size_t)(_count) * \ + (_dt_ptr)->size; \ + } \ + else \ + { \ + (_dt_contig_out) = 1; \ + (_data_sz_out) = 0; \ + } \ + } \ + } while (0) + +#define MPIDI_Datatype_check_size(_datatype,_count,_data_sz_out) \ + do { \ + if (IS_BUILTIN(_datatype)) \ + { \ + (_data_sz_out) = (size_t)(_count) * \ + MPID_Datatype_get_basic_size(_datatype); \ + } \ + else \ + { \ + MPIR_Datatype *_dt_ptr; \ + MPID_Datatype_get_ptr((_datatype), (_dt_ptr)); \ + (_data_sz_out) = (_dt_ptr) ? (size_t)(_count) * \ + (_dt_ptr)->size : 0; \ + } \ + } while (0) + +#define MPIDI_Datatype_check_contig_size_lb(_datatype,_count, \ + _dt_contig_out, \ + _data_sz_out, \ + _dt_true_lb) \ + do { \ + if (IS_BUILTIN(_datatype)) \ + { \ + (_dt_contig_out) = TRUE; \ + (_data_sz_out) = (size_t)(_count) * \ + MPID_Datatype_get_basic_size(_datatype); \ + (_dt_true_lb) = 0; \ + } \ + else \ + { \ + MPIR_Datatype *_dt_ptr; \ + MPID_Datatype_get_ptr((_datatype), (_dt_ptr)); \ + if (_dt_ptr) \ + { \ + (_dt_contig_out) = (_dt_ptr)->is_contig; \ + (_data_sz_out) = (size_t)(_count) * \ + (_dt_ptr)->size; \ + (_dt_true_lb) = (_dt_ptr)->true_lb; \ + } \ + else \ + { \ + (_dt_contig_out) = 1; \ + (_data_sz_out) = 0; \ + (_dt_true_lb) = 0; \ + } \ + } \ + } while (0) + +#define MPIDI_Request_create_null_rreq(rreq_, mpi_errno_, FAIL_) \ + do { \ + (rreq_) = MPIR_Request_create(MPIR_REQUEST_KIND__RECV); \ + if ((rreq_) != NULL) { \ + MPIR_cc_set(&(rreq_)->cc, 0); \ + MPIR_Status_set_procnull(&(rreq_)->status); \ + } \ + else { \ + MPIR_ERR_SETANDJUMP(mpi_errno_,MPI_ERR_OTHER,"**nomemreq"); \ + } \ + } while (0) + +#define IS_BUILTIN(_datatype) \ + (HANDLE_GET_KIND(_datatype) == HANDLE_KIND_BUILTIN) + +#ifndef container_of +#define container_of(ptr, type, field) \ + ((type *) ((char *)ptr - offsetof(type, field))) +#endif + +static inline uint64_t MPIDI_CH4U_init_send_tag(MPIR_Context_id_t contextid, int source, int tag) +{ + uint64_t match_bits; + match_bits = contextid; + match_bits = (match_bits << MPIDI_CH4U_SOURCE_SHIFT); + match_bits |= (source & (MPIDI_CH4U_SOURCE_MASK >> MPIDI_CH4U_TAG_SHIFT)); + match_bits = (match_bits << MPIDI_CH4U_TAG_SHIFT); + match_bits |= (MPIDI_CH4U_TAG_MASK & tag); + return match_bits; +} + +static inline uint64_t MPIDI_CH4U_init_recvtag(uint64_t * mask_bits, + MPIR_Context_id_t contextid, int source, int tag) +{ + uint64_t match_bits = 0; + *mask_bits = MPIDI_CH4U_PROTOCOL_MASK; + match_bits = contextid; + match_bits = (match_bits << MPIDI_CH4U_SOURCE_SHIFT); + + if (MPI_ANY_SOURCE == source) { + match_bits = (match_bits << MPIDI_CH4U_TAG_SHIFT); + *mask_bits |= MPIDI_CH4U_SOURCE_MASK; + } + else { + match_bits |= source; + match_bits = (match_bits << MPIDI_CH4U_TAG_SHIFT); + } + + if (MPI_ANY_TAG == tag) + *mask_bits |= MPIDI_CH4U_TAG_MASK; + else + match_bits |= (MPIDI_CH4U_TAG_MASK & tag); + + return match_bits; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4I_valid_group_rank +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4I_valid_group_rank(MPIR_Comm * comm, int rank, MPIR_Group * grp) +{ + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4I_VALID_GROUP_RANK); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4I_VALID_GROUP_RANK); + + int lpid; + int size = grp->size; + int z; + int ret; + + if (unlikely(rank == MPI_PROC_NULL)) { + /* Treat PROC_NULL as always valid */ + ret = 1; + goto fn_exit; + } + + MPIDI_NM_comm_get_lpid(comm, rank, &lpid, FALSE); + + for (z = 0; z < size && lpid != grp->lrank_to_lpid[z].lpid; ++z) { + } + + ret = (z < size); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4I_VALID_GROUP_RANK); + fn_exit: + return ret; +} + +#define MPIDI_CH4R_PROGRESS() \ + do { \ + mpi_errno = MPIDI_Progress_test(); \ + if (mpi_errno != MPI_SUCCESS) MPIR_ERR_POP(mpi_errno); \ + } while (0) + +#define MPIDI_CH4R_PROGRESS_WHILE(cond) \ + do { \ + while (cond) \ + MPIDI_CH4R_PROGRESS(); \ + } while (0) + +#ifdef HAVE_ERROR_CHECKING +#define MPIDI_CH4U_EPOCH_CHECK_SYNC(win, mpi_errno, stmt) \ + do { \ + MPID_BEGIN_ERROR_CHECKS; \ + if (MPIDI_CH4U_WIN(win, sync).origin_epoch_type == MPIDI_CH4U_WIN(win, sync).target_epoch_type && \ + MPIDI_CH4U_WIN(win, sync).origin_epoch_type == MPIDI_CH4U_EPOTYPE_REFENCE) \ + { \ + MPIDI_CH4U_WIN(win, sync).origin_epoch_type = MPIDI_CH4U_EPOTYPE_FENCE; \ + MPIDI_CH4U_WIN(win, sync).target_epoch_type = MPIDI_CH4U_EPOTYPE_FENCE; \ + } \ + if (MPIDI_CH4U_WIN(win, sync).origin_epoch_type == MPIDI_CH4U_EPOTYPE_NONE || \ + MPIDI_CH4U_WIN(win, sync).origin_epoch_type == MPIDI_CH4U_EPOTYPE_POST) \ + MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, \ + stmt, "**rmasync"); \ + MPID_END_ERROR_CHECKS; \ + } while (0) + +#define MPIDI_CH4U_EPOCH_CHECK_TYPE(win,mpi_errno,stmt) \ + do { \ + MPID_BEGIN_ERROR_CHECKS; \ + if (MPIDI_CH4U_WIN(win, sync).origin_epoch_type != MPIDI_CH4U_EPOTYPE_NONE && \ + MPIDI_CH4U_WIN(win, sync).origin_epoch_type != MPIDI_CH4U_EPOTYPE_REFENCE) \ + MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, \ + stmt, "**rmasync"); \ + MPID_END_ERROR_CHECKS; \ + } while (0) + +#define MPIDI_CH4U_EPOCH_START_CHECK(win,mpi_errno,stmt) \ + do { \ + MPID_BEGIN_ERROR_CHECKS; \ + if (MPIDI_CH4U_WIN(win, sync).origin_epoch_type == MPIDI_CH4U_EPOTYPE_START && \ + !MPIDI_CH4I_valid_group_rank(win->comm_ptr, target_rank, \ + MPIDI_CH4U_WIN(win, sync).sc.group)) \ + MPIR_ERR_SETANDSTMT(mpi_errno, \ + MPI_ERR_RMA_SYNC, \ + stmt, \ + "**rmasync"); \ + MPID_END_ERROR_CHECKS; \ + } while (0) + +#define MPIDI_CH4U_EPOCH_START_CHECK2(win,mpi_errno,stmt) \ + do { \ + MPID_BEGIN_ERROR_CHECKS; \ + if (MPIDI_CH4U_WIN(win, sync).origin_epoch_type != MPIDI_CH4U_EPOTYPE_START) { \ + MPIR_ERR_SETANDSTMT(mpi_errno, \ + MPI_ERR_RMA_SYNC, \ + stmt, \ + "**rmasync"); \ + } \ + MPID_END_ERROR_CHECKS; \ + } while (0) + +#define MPIDI_CH4U_EPOCH_FENCE_CHECK(win,mpi_errno,stmt) \ + do { \ + MPID_BEGIN_ERROR_CHECKS; \ + if (MPIDI_CH4U_WIN(win, sync).origin_epoch_type != MPIDI_CH4U_WIN(win, sync).target_epoch_type) \ + MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, \ + stmt, "**rmasync"); \ + if (!(massert & MPI_MODE_NOPRECEDE) && \ + MPIDI_CH4U_WIN(win, sync).origin_epoch_type != MPIDI_CH4U_EPOTYPE_FENCE && \ + MPIDI_CH4U_WIN(win, sync).origin_epoch_type != MPIDI_CH4U_EPOTYPE_REFENCE && \ + MPIDI_CH4U_WIN(win, sync).origin_epoch_type != MPIDI_CH4U_EPOTYPE_NONE) \ + MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, \ + stmt, "**rmasync"); \ + MPID_END_ERROR_CHECKS; \ + } while (0) + +#define MPIDI_CH4U_EPOCH_POST_CHECK(win,mpi_errno,stmt) \ + do { \ + MPID_BEGIN_ERROR_CHECKS; \ + if (MPIDI_CH4U_WIN(win, sync).target_epoch_type != MPIDI_CH4U_EPOTYPE_NONE && \ + MPIDI_CH4U_WIN(win, sync).target_epoch_type != MPIDI_CH4U_EPOTYPE_REFENCE) \ + MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, \ + stmt, "**rmasync"); \ + MPID_END_ERROR_CHECKS; \ + } while (0) + +#define MPIDI_CH4U_EPOCH_LOCK_CHECK(win,mpi_errno,stmt) \ +do { \ + MPID_BEGIN_ERROR_CHECKS; \ + if ((MPIDI_CH4U_WIN(win, sync).origin_epoch_type != MPIDI_CH4U_EPOTYPE_LOCK) && \ + (MPIDI_CH4U_WIN(win, sync).origin_epoch_type != MPIDI_CH4U_EPOTYPE_LOCK_ALL)) \ + MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, \ + stmt, "**rmasync"); \ + MPID_END_ERROR_CHECKS; \ +} while (0) + +#define MPIDI_CH4U_EPOCH_FREE_CHECK(win,mpi_errno,stmt) \ + do { \ + MPID_BEGIN_ERROR_CHECKS; \ + if (MPIDI_CH4U_WIN(win, sync).origin_epoch_type != MPIDI_CH4U_WIN(win, sync).target_epoch_type || \ + (MPIDI_CH4U_WIN(win, sync).origin_epoch_type != MPIDI_CH4U_EPOTYPE_NONE && \ + MPIDI_CH4U_WIN(win, sync).origin_epoch_type != MPIDI_CH4U_EPOTYPE_REFENCE)) \ + MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, stmt, "**rmasync"); \ + MPID_END_ERROR_CHECKS; \ + } while (0) + +#define MPIDI_CH4U_EPOCH_ORIGIN_CHECK(win, epoch_type, mpi_errno, stmt) \ + do { \ + MPID_BEGIN_ERROR_CHECKS; \ + if (MPIDI_CH4U_WIN(win, sync).origin_epoch_type != epoch_type) \ + MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, \ + stmt, "**rmasync"); \ + MPID_END_ERROR_CHECKS; \ + } while (0) + +#define MPIDI_CH4U_EPOCH_TARGET_CHECK(win, epoch_type, mpi_errno, stmt) \ + do { \ + MPID_BEGIN_ERROR_CHECKS; \ + if (MPIDI_CH4U_WIN(win, sync).target_epoch_type != epoch_type) \ + MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, \ + stmt, "**rmasync"); \ + MPID_END_ERROR_CHECKS; \ + } while (0) + +#else /* HAVE_ERROR_CHECKING */ +#define MPIDI_CH4U_EPOCH_CHECK_SYNC(win, mpi_errno, stmt) if (0) goto fn_fail; +#define MPIDI_CH4U_EPOCH_CHECK_TYPE(win, mpi_errno, stmt) if (0) goto fn_fail; +#define MPIDI_CH4U_EPOCH_START_CHECK(win, mpi_errno, stmt) if (0) goto fn_fail; +#define MPIDI_CH4U_EPOCH_START_CHECK2(win, mpi_errno, stmt) if (0) goto fn_fail; +#define MPIDI_CH4U_EPOCH_FENCE_CHECK(win, mpi_errno, stmt) if (0) goto fn_fail; +#define MPIDI_CH4U_EPOCH_POST_CHECK(win, mpi_errno, stmt) if (0) goto fn_fail; +#define MPIDI_CH4U_EPOCH_LOCK_CHECK(win, mpi_errno, stmt) if (0) goto fn_fail; +#define MPIDI_CH4U_EPOCH_FREE_CHECK(win, mpi_errno, stmt) if (0) goto fn_fail; +#define MPIDI_CH4U_EPOCH_ORIGIN_CHECK(win, epoch_type, mpi_errno, stmt) if (0) goto fn_fail; +#define MPIDI_CH4U_EPOCH_TARGET_CHECK(win, epoch_type, mpi_errno, stmt) if (0) goto fn_fail; +#endif /* HAVE_ERROR_CHECKING */ + +#define MPIDI_CH4U_EPOCH_FENCE_EVENT(win, massert) \ + do { \ + if (massert & MPI_MODE_NOSUCCEED) \ + { \ + MPIDI_CH4U_WIN(win, sync).origin_epoch_type = MPIDI_CH4U_EPOTYPE_NONE; \ + MPIDI_CH4U_WIN(win, sync).target_epoch_type = MPIDI_CH4U_EPOTYPE_NONE; \ + } \ + else \ + { \ + MPIDI_CH4U_WIN(win, sync).origin_epoch_type = MPIDI_CH4U_EPOTYPE_REFENCE; \ + MPIDI_CH4U_WIN(win, sync).target_epoch_type = MPIDI_CH4U_EPOTYPE_REFENCE; \ + } \ + } while (0) + +#define MPIDI_CH4U_EPOCH_TARGET_EVENT(win) \ + do { \ + if (MPIDI_CH4U_WIN(win, sync).target_epoch_type == MPIDI_CH4U_EPOTYPE_REFENCE) \ + MPIDI_CH4U_WIN(win, sync).origin_epoch_type = MPIDI_CH4U_EPOTYPE_REFENCE; \ + else \ + MPIDI_CH4U_WIN(win, sync).origin_epoch_type = MPIDI_CH4U_EPOTYPE_NONE; \ + } while (0) + +#define MPIDI_CH4U_EPOCH_ORIGIN_EVENT(Win) \ + do { \ + if (MPIDI_CH4U_WIN(win, sync).origin_epoch_type == MPIDI_CH4U_EPOTYPE_REFENCE) \ + MPIDI_CH4U_WIN(win, sync).target_epoch_type = MPIDI_CH4U_EPOTYPE_REFENCE; \ + else \ + MPIDI_CH4U_WIN(win, sync).target_epoch_type = MPIDI_CH4U_EPOTYPE_NONE; \ + } while (0) + +/* + Calculate base address of the target window at the origin side + Return zero to let the target side calculate the actual address + (only offset from window base is given to the target in this case) +*/ +static inline uintptr_t MPIDI_CH4I_win_base_at_origin(const MPIR_Win * win, int target_rank) +{ + /* TODO: In future we may want to calculate the full virtual address + * in the target at the origin side. It can be done by looking at + * MPIDI_CH4U_WINFO(win, target_rank)->base_addr */ + return 0; +} + +/* + Calculate base address of the window at the target side + If MPIDI_CH4I_win_base_at_origin calculates the full virtual address + this function must return zero +*/ +static inline uintptr_t MPIDI_CH4I_win_base_at_target(const MPIR_Win * win) +{ + return (uintptr_t) win->base; +} + +#endif /* CH4_IMPL_H_INCLUDED */ diff --git a/src/mpid/ch4/src/ch4_init.h b/src/mpid/ch4/src/ch4_init.h new file mode 100644 index 0000000..c26db6b --- /dev/null +++ b/src/mpid/ch4/src/ch4_init.h @@ -0,0 +1,751 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef CH4_INIT_H_INCLUDED +#define CH4_INIT_H_INCLUDED + +#include "ch4_impl.h" +#include "ch4r_proc.h" +#include "ch4i_comm.h" + +/* +=== BEGIN_MPI_T_CVAR_INFO_BLOCK === + +categories: + - name : CH4 + description : cvars that control behavior of the CH4 device + +cvars: + - name : MPIR_CVAR_CH4_NETMOD + category : CH4 + type : string + default : "" + class : device + verbosity : MPI_T_VERBOSITY_USER_BASIC + scope : MPI_T_SCOPE_ALL_EQ + description : >- + If non-empty, this cvar specifies which network module to use + + - name : MPIR_CVAR_CH4_SHM + category : CH4 + type : string + default : "" + class : device + verbosity : MPI_T_VERBOSITY_USER_BASIC + scope : MPI_T_SCOPE_ALL_EQ + description : >- + If non-empty, this cvar specifies which shm module to use + +=== END_MPI_T_CVAR_INFO_BLOCK === +*/ + +#undef FUNCNAME +#define FUNCNAME MPIDI_choose_netmod +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_choose_netmod(void) +{ + int i, mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_CHOOSE_NETMOD); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_CHOOSE_NETMOD); + + MPIR_Assert(MPIR_CVAR_CH4_NETMOD != NULL); + + if (strcmp(MPIR_CVAR_CH4_NETMOD, "") == 0) { + /* netmod not specified, using the default */ + MPIDI_NM_func = MPIDI_NM_funcs[0]; + MPIDI_NM_native_func = MPIDI_NM_native_funcs[0]; + goto fn_exit; + } + + for (i = 0; i < MPIDI_num_netmods; ++i) { + /* use MPL variant of strncasecmp if we get one */ + if (!strncasecmp(MPIR_CVAR_CH4_NETMOD, MPIDI_NM_strings[i], MPIDI_MAX_NETMOD_STRING_LEN)) { + MPIDI_NM_func = MPIDI_NM_funcs[i]; + MPIDI_NM_native_func = MPIDI_NM_native_funcs[i]; + goto fn_exit; + } + } + + MPIR_ERR_SETANDJUMP1(mpi_errno, MPI_ERR_OTHER, "**invalid_netmod", "**invalid_netmod %s", + MPIR_CVAR_CH4_NETMOD); + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_CHOOSE_NETMOD); + return mpi_errno; + fn_fail: + + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_choose_shm +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_choose_shm(void) +{ + + int mpi_errno = MPI_SUCCESS; +#if defined(MPIDI_BUILD_CH4_SHM) + int i; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_CHOOSE_SHM); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_CHOOSE_SHM); + + + MPIR_Assert(MPIR_CVAR_CH4_SHM != NULL); + + if (strcmp(MPIR_CVAR_CH4_SHM, "") == 0) { + /* shm not specified, using the default */ + MPIDI_SHM_func = MPIDI_SHM_funcs[0]; + MPIDI_SHM_native_func = MPIDI_SHM_native_funcs[0]; + goto fn_exit; + } + + for (i = 0; i < MPIDI_num_shms; ++i) { + /* use MPL variant of strncasecmp if we get one */ + if (!strncasecmp(MPIR_CVAR_CH4_SHM, MPIDI_SHM_strings[i], MPIDI_MAX_SHM_STRING_LEN)) { + MPIDI_SHM_func = MPIDI_SHM_funcs[i]; + MPIDI_SHM_native_func = MPIDI_SHM_native_funcs[i]; + goto fn_exit; + } + } + + MPIR_ERR_SETANDJUMP1(mpi_errno, MPI_ERR_OTHER, "**ch4|invalid_shm", "**ch4|invalid_shm %s", + MPIR_CVAR_CH4_SHM); + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_CHOOSE_SHM); + return mpi_errno; + fn_fail: + goto fn_exit; +#else + return mpi_errno; +#endif +} + + +#if (MPICH_THREAD_GRANULARITY == MPICH_THREAD_GRANULARITY__POBJ) +#define MAX_THREAD_MODE MPI_THREAD_MULTIPLE +#elif (MPICH_THREAD_GRANULARITY == MPICH_THREAD_GRANULARITY__GLOBAL) +#define MAX_THREAD_MODE MPI_THREAD_MULTIPLE +#elif (MPICH_THREAD_GRANULARITY == MPICH_THREAD_GRANULARITY__SINGLE) +#define MAX_THREAD_MODE MPI_THREAD_SERIALIZED +#elif (MPICH_THREAD_GRANULARITY == MPICH_THREAD_GRANULARITY__LOCKFREE) +#define MAX_THREAD_MODE MPI_THREAD_SERIALIZED +#else +#error "Thread Granularity: Invalid" +#endif + +#undef FUNCNAME +#define FUNCNAME MPIDI_Init +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Init(int *argc, + char ***argv, + int requested, int *provided, int *has_args, int *has_env) +{ + int pmi_errno, mpi_errno = MPI_SUCCESS, rank, has_parent, size, appnum, thr_err; + void *netmod_contexts; + int avtid, max_n_avts; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_INIT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_INIT); + +#ifdef MPL_USE_DBG_LOGGING + MPIDI_CH4_DBG_GENERAL = MPL_dbg_class_alloc("CH4", "ch4"); + MPIDI_CH4_DBG_MAP = MPL_dbg_class_alloc("CH4_MAP", "ch4_map"); + MPIDI_CH4_DBG_MEMORY = MPL_dbg_class_alloc("CH4_MEMORY", "ch4_memory"); +#endif + MPIDI_choose_netmod(); + pmi_errno = PMI_Init(&has_parent); + + if (pmi_errno != PMI_SUCCESS) { + MPIR_ERR_SETANDJUMP1(pmi_errno, MPI_ERR_OTHER, "**pmi_init", "**pmi_init %d", pmi_errno); + } + + pmi_errno = PMI_Get_rank(&rank); + + if (pmi_errno != PMI_SUCCESS) { + MPIR_ERR_SETANDJUMP1(pmi_errno, MPI_ERR_OTHER, "**pmi_get_rank", + "**pmi_get_rank %d", pmi_errno); + } + + pmi_errno = PMI_Get_size(&size); + + if (pmi_errno != 0) { + MPIR_ERR_SETANDJUMP1(pmi_errno, MPI_ERR_OTHER, "**pmi_get_size", + "**pmi_get_size %d", pmi_errno); + } + + pmi_errno = PMI_Get_appnum(&appnum); + + if (pmi_errno != PMI_SUCCESS) { + MPIR_ERR_SETANDJUMP1(pmi_errno, MPI_ERR_OTHER, "**pmi_get_appnum", + "**pmi_get_appnum %d", pmi_errno); + } + + MPID_Thread_mutex_create(&MPIDI_CH4I_THREAD_PROGRESS_MUTEX, &thr_err); + MPID_Thread_mutex_create(&MPIDI_CH4I_THREAD_PROGRESS_HOOK_MUTEX, &thr_err); + + /* ---------------------------------- */ + /* Initialize MPI_COMM_SELF */ + /* ---------------------------------- */ + MPIR_Process.comm_self->rank = 0; + MPIR_Process.comm_self->remote_size = 1; + MPIR_Process.comm_self->local_size = 1; + + /* ---------------------------------- */ + /* Initialize MPI_COMM_WORLD */ + /* ---------------------------------- */ + MPIR_Process.comm_world->rank = rank; + MPIR_Process.comm_world->remote_size = size; + MPIR_Process.comm_world->local_size = size; + + mpi_errno = MPIDI_choose_shm(); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POPFATAL(mpi_errno); + } + + MPIDI_CH4_Global.allocated_max_n_avts = 0; + MPIDIU_avt_init(); + MPIDIU_get_next_avtid(&avtid); + MPIR_Assert(avtid == 0); + max_n_avts = MPIDIU_get_max_n_avts(); + + MPIDII_av_table = (MPIDII_av_table_t **) + MPL_malloc(max_n_avts * sizeof(MPIDII_av_table_t *)); + + MPIDII_av_table[0] = (MPIDII_av_table_t *) + MPL_malloc(size * sizeof(MPIDII_av_entry_t) + + sizeof(MPIDII_av_table_t)); + + MPIDII_av_table[0]->size = size; + MPIR_Object_set_ref(MPIDII_av_table[0], 1); + + MPIDIU_alloc_globals_for_avtid(avtid); + + MPIDII_av_table0 = MPIDII_av_table[0]; + + /* initialize rank_map */ + MPIDII_COMM(MPIR_Process.comm_world, map).mode = MPIDII_RANK_MAP_DIRECT_INTRA; + MPIDII_COMM(MPIR_Process.comm_world, map).avtid = 0; + MPIDII_COMM(MPIR_Process.comm_world, map).size = size; + MPIDII_COMM(MPIR_Process.comm_world, local_map).mode = MPIDII_RANK_MAP_NONE; + MPIDIU_avt_add_ref(0); + + MPIDII_COMM(MPIR_Process.comm_self, map).mode = MPIDII_RANK_MAP_OFFSET_INTRA; + MPIDII_COMM(MPIR_Process.comm_self, map).avtid = 0; + MPIDII_COMM(MPIR_Process.comm_self, map).size = 1; + MPIDII_COMM(MPIR_Process.comm_self, map).reg.offset = rank; + MPIDII_COMM(MPIR_Process.comm_self, local_map).mode = MPIDII_RANK_MAP_NONE; + MPIDIU_avt_add_ref(0); + + MPIR_Process.attrs.tag_ub = (1ULL << MPIDI_CH4U_TAG_SHIFT) - 1; + /* discuss */ + + mpi_errno = MPIDI_NM_init(rank, size, appnum, &MPIR_Process.attrs.tag_ub, + MPIR_Process.comm_world, + MPIR_Process.comm_self, has_parent, 1, &netmod_contexts); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POPFATAL(mpi_errno); + } + +#ifdef MPIDI_BUILD_CH4_LOCALITY_INFO + int i; + for (i = 0; i < MPIR_Process.comm_world->local_size; i++) { + MPIDII_av_table0->table[i].is_local = 0; + } + MPIDI_CH4U_build_nodemap(MPIR_Process.comm_world->rank, + MPIR_Process.comm_world, + MPIR_Process.comm_world->local_size, + MPIDI_CH4_Global.node_map[0], &MPIDI_CH4_Global.max_node_id); + + for (i = 0; i < MPIR_Process.comm_world->local_size; i++) { + MPIDII_av_table0->table[i].is_local = + (MPIDI_CH4_Global.node_map[0][i] == + MPIDI_CH4_Global.node_map[0][MPIR_Process.comm_world->rank]) ? 1 : 0; + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_GENERAL, VERBOSE, + (MPL_DBG_FDEST, "WORLD RANK %d %s local", i, + MPIDII_av_table0->table[i].is_local ? "is" : "is not")); + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_GENERAL, VERBOSE, + (MPL_DBG_FDEST, "Node id (i) (me) %d %d", MPIDI_CH4_Global.node_map[0][i], + MPIDI_CH4_Global.node_map[0][MPIR_Process.comm_world->rank])); + } +#endif + +#ifdef MPIDI_BUILD_CH4_SHM + mpi_errno = MPIDI_SHM_init(rank, size); + + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POPFATAL(mpi_errno); + } +#endif + + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + + MPIR_Process.attrs.appnum = appnum; + MPIR_Process.attrs.wtime_is_global = 1; + MPIR_Process.attrs.io = MPI_ANY_SOURCE; + + MPIR_Comm_commit(MPIR_Process.comm_self); + MPIR_Comm_commit(MPIR_Process.comm_world); + + /* -------------------------------- */ + /* Return MPICH Parameters */ + /* -------------------------------- */ + switch (requested) { + case MPI_THREAD_SINGLE: + case MPI_THREAD_SERIALIZED: + case MPI_THREAD_FUNNELED: + *provided = requested; + break; + + case MPI_THREAD_MULTIPLE: + *provided = MAX_THREAD_MODE; + break; + } + + *has_args = TRUE; + *has_env = TRUE; + MPIDI_CH4_Global.is_initialized = 0; + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_INIT); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_InitCompleted +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_InitCompleted(void) +{ + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_INITCOMPLETED); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_INITCOMPLETED); + MPIDI_CH4_Global.is_initialized = 1; + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_INITCOMPLETED); + return MPI_SUCCESS; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Finalize +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Finalize(void) +{ + int mpi_errno, thr_err; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_FINALIZE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_FINALIZE); + + mpi_errno = MPIDI_NM_finalize(); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); +#ifdef MPIDI_BUILD_CH4_SHM + mpi_errno = MPIDI_SHM_finalize(); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); +#endif + + int i; + int max_n_avts; + max_n_avts = MPIDIU_get_max_n_avts(); + for (i = 0; i < max_n_avts; i++) { + if (MPIDII_av_table[i] != NULL) { + MPIDIU_avt_release_ref(i); + } + } + MPL_free(MPIDII_av_table); + MPL_free(MPIDI_CH4_Global.node_map); + + MPIDIU_avt_destroy(); + + MPID_Thread_mutex_destroy(&MPIDI_CH4I_THREAD_PROGRESS_MUTEX, &thr_err); + MPID_Thread_mutex_destroy(&MPIDI_CH4I_THREAD_PROGRESS_HOOK_MUTEX, &thr_err); + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_FINALIZE); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Get_universe_size +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Get_universe_size(int *universe_size) +{ + int mpi_errno = MPI_SUCCESS; + int pmi_errno = PMI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_GET_UNIVERSE_SIZE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_GET_UNIVERSE_SIZE); + + + pmi_errno = PMI_Get_universe_size(universe_size); + + if (pmi_errno != PMI_SUCCESS) + MPIR_ERR_SETANDJUMP1(mpi_errno, MPI_ERR_OTHER, + "**pmi_get_universe_size", "**pmi_get_universe_size %d", pmi_errno); + + if (*universe_size < 0) + *universe_size = MPIR_UNIVERSE_SIZE_NOT_AVAILABLE; + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_GET_UNIVERSE_SIZE); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Get_processor_name +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Get_processor_name(char *name, int namelen, int *resultlen) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_GET_PROCESSOR_NAME); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_GET_PROCESSOR_NAME); + + if (!MPIDI_CH4_Global.pname_set) { +#if defined(HAVE_GETHOSTNAME) + + if (gethostname(MPIDI_CH4_Global.pname, MPI_MAX_PROCESSOR_NAME) == 0) + MPIDI_CH4_Global.pname_len = (int) strlen(MPIDI_CH4_Global.pname); + +#elif defined(HAVE_SYSINFO) + + if (sysinfo(SI_HOSTNAME, MPIDI_CH4_Global.pname, MPI_MAX_PROCESSOR_NAME) == 0) + MPIDI_CH4_Global.pname_len = (int) strlen(MPIDI_CH4_Global.pname); + +#else + MPL_snprintf(MPIDI_CH4_Global.pname, MPI_MAX_PROCESSOR_NAME, "%d", + MPIR_Process.comm_world->rank); + MPIDI_CH4_Global.pname_len = (int) strlen(MPIDI_CH4_Global.pname); +#endif + MPIDI_CH4_Global.pname_set = 1; + } + + MPIR_ERR_CHKANDJUMP(MPIDI_CH4_Global.pname_len <= 0, + mpi_errno, MPI_ERR_OTHER, "**procnamefailed"); + MPL_strncpy(name, MPIDI_CH4_Global.pname, namelen); + + if (resultlen) + *resultlen = MPIDI_CH4_Global.pname_len; + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_GET_PROCESSOR_NAME); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Abort +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Abort(MPIR_Comm * comm, + int mpi_errno, int exit_code, const char *error_msg) +{ + char sys_str[MPI_MAX_ERROR_STRING + 5] = ""; + char comm_str[MPI_MAX_ERROR_STRING] = ""; + char world_str[MPI_MAX_ERROR_STRING] = ""; + char error_str[2 * MPI_MAX_ERROR_STRING + 128]; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_ABORT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_ABORT); + + if (MPIR_Process.comm_world) { + int rank = MPIR_Process.comm_world->rank; + snprintf(world_str, sizeof(world_str), " on node %d", rank); + } + + if (comm) { + int rank = comm->rank; + int context_id = comm->context_id; + snprintf(comm_str, sizeof(comm_str), " (rank %d in comm %d)", rank, context_id); + } + + if (!error_msg) + error_msg = "Internal error"; + + if (mpi_errno != MPI_SUCCESS) { + char msg[MPI_MAX_ERROR_STRING] = ""; + MPIR_Err_get_string(mpi_errno, msg, MPI_MAX_ERROR_STRING, NULL); + snprintf(sys_str, sizeof(msg), " (%s)", msg); + } + MPL_snprintf(error_str, sizeof(error_str), "Abort(%d)%s%s: %s%s\n", + exit_code, world_str, comm_str, error_msg, sys_str); + MPL_error_printf("%s", error_str); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_ABORT); + fflush(stderr); + fflush(stdout); + PMI_Abort(exit_code, error_msg); + return 0; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Alloc_mem +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ void *MPIDI_Alloc_mem(size_t size, MPIR_Info * info_ptr) +{ + void *p; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_ALLOC_MEM); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_ALLOC_MEM); + + p = MPIDI_NM_alloc_mem(size, info_ptr); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_ALLOC_MEM); + return p; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Free_mem +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Free_mem(void *ptr) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_FREE_MEM); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_FREE_MEM); + mpi_errno = MPIDI_NM_free_mem(ptr); + + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_FREE_MEM); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Comm_get_lpid +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Comm_get_lpid(MPIR_Comm * comm_ptr, + int idx, int *lpid_ptr, MPL_bool is_remote) +{ + int mpi_errno = MPI_SUCCESS; + int avtid = 0, lpid = 0; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_COMM_GET_LPID); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_COMM_GET_LPID); + + if (comm_ptr->comm_kind == MPIR_COMM_KIND__INTRACOMM) + MPIDIU_comm_rank_to_pid(comm_ptr, idx, &lpid, &avtid); + else if (is_remote) + MPIDIU_comm_rank_to_pid(comm_ptr, idx, &lpid, &avtid); + else { + MPIDIU_comm_rank_to_pid_local(comm_ptr, idx, &lpid, &avtid); + } + + *lpid_ptr = MPIDIU_LPID_CREATE(avtid, lpid); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_COMM_GET_LPID); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_GPID_Get +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_GPID_Get(MPIR_Comm * comm_ptr, int rank, MPIR_Gpid * gpid) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_GPID_GET); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_GPID_GET); + + mpi_errno = MPIDI_NM_gpid_get(comm_ptr, rank, gpid); + MPIDI_CH4U_get_node_id(comm_ptr, rank, &MPIDII_GPID(gpid).node); + + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_GPID_GET); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Get_node_id +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Get_node_id(MPIR_Comm * comm, int rank, MPID_Node_id_t * id_p) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_GET_NODE_ID); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_GET_NODE_ID); + + MPIDI_CH4U_get_node_id(comm, rank, id_p); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_GET_NODE_ID); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Get_max_node_id +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Get_max_node_id(MPIR_Comm * comm, MPID_Node_id_t * max_id_p) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_GET_MAX_NODE_ID); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_GET_MAX_NODE_ID); + + MPIDI_CH4U_get_max_node_id(comm, max_id_p); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_GET_MAX_NODE_ID); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_GetAllInComm +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_GPID_GetAllInComm(MPIR_Comm * comm_ptr, + int local_size, MPIR_Gpid local_gpids[], int *singleAVT) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_GETALLINCOMM); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_GETALLINCOMM); + + mpi_errno = MPIDI_NM_getallincomm(comm_ptr, local_size, local_gpids, singleAVT); + + if (MPIDII_COMM(comm_ptr, map).mode == MPIDII_RANK_MAP_MLUT) { + *singleAVT = FALSE; + } + else { + *singleAVT = TRUE; + } + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_GETALLINCOMM); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_GPID_ToLpidArray +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_GPID_ToLpidArray(int size, MPIR_Gpid gpid[], int lpid[]) +{ + int mpi_errno = MPI_SUCCESS, i; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_GPID_TOLPIDARRAY); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_GPID_TOLPIDARRAY); + + mpi_errno = MPIDI_NM_gpid_tolpidarray(size, gpid, lpid); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + + /* update node_map */ + for (i = 0; i < size; i++) { + int _avtid = 0, _lpid = 0; + /* if this is a new process, update node_map and locality */ + if (MPIDIU_LPID_IS_NEW_AVT(lpid[i])) { + MPIDIU_LPID_CLEAR_NEW_AVT_MARK(lpid[i]); + _avtid = MPIDIU_LPID_GET_AVTID(lpid[i]); + _lpid = MPIDIU_LPID_GET_LPID(lpid[i]); + MPIDI_CH4_Global.node_map[_avtid][_lpid] = MPIDII_GPID(&gpid[i]).node; + /* new process groups are always assumed to be remote */ +#ifdef MPIDI_BUILD_CH4_LOCALITY_INFO + if (_avtid != 0) { + MPIDII_av_table[_avtid]->table[_lpid].is_local = 0; + } +#endif + } + } + + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_GPID_TOLPIDARRAY); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Create_intercomm_from_lpids +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Create_intercomm_from_lpids(MPIR_Comm * newcomm_ptr, + int size, const int lpids[]) +{ + int mpi_errno = MPI_SUCCESS, i; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_CREATE_INTERCOMM_FROM_LPIDS); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_CREATE_INTERCOMM_FROM_LPIDS); + + MPIDII_rank_map_mlut_t *mlut = NULL; + MPIDII_COMM(newcomm_ptr, map).mode = MPIDII_RANK_MAP_MLUT; + MPIDII_COMM(newcomm_ptr, map).avtid = -1; + mpi_errno = MPIDII_alloc_mlut(&mlut, size); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + MPIDII_COMM(newcomm_ptr, map).size = size; + MPIDII_COMM(newcomm_ptr, map).irreg.mlut.t = mlut; + MPIDII_COMM(newcomm_ptr, map).irreg.mlut.gpid = mlut->gpid; + + for (i = 0; i < size; i++) { + MPIDII_COMM(newcomm_ptr, map).irreg.mlut.gpid[i].avtid = MPIDIU_LPID_GET_AVTID(lpids[i]); + MPIDII_COMM(newcomm_ptr, map).irreg.mlut.gpid[i].lpid = MPIDIU_LPID_GET_LPID(lpids[i]); + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, + (MPL_DBG_FDEST, " remote rank=%d, avtid=%d, lpid=%d", i, + MPIDII_COMM(newcomm_ptr, map).irreg.mlut.gpid[i].avtid, + MPIDII_COMM(newcomm_ptr, map).irreg.mlut.gpid[i].lpid)); + } + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_CREATE_INTERCOMM_FROM_LPIDS); + return mpi_errno; + fn_fail: + goto fn_exit; +} + + +#undef FUNCNAME +#define FUNCNAME MPIDI_Aint_add +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ MPI_Aint MPIDI_Aint_add(MPI_Aint base, MPI_Aint disp) +{ + MPI_Aint result; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_AINT_ADD); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_AINT_ADD); + result = MPIR_VOID_PTR_CAST_TO_MPI_AINT((char *) MPIR_AINT_CAST_TO_VOID_PTR(base) + disp); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_AINT_ADD); + return result; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Aint_diff +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ MPI_Aint MPIDI_Aint_diff(MPI_Aint addr1, MPI_Aint addr2) +{ + MPI_Aint result; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_AINT_DIFF); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_AINT_DIFF); + + result = MPIR_PTR_DISP_CAST_TO_MPI_AINT((char *) MPIR_AINT_CAST_TO_VOID_PTR(addr1) + - (char *) MPIR_AINT_CAST_TO_VOID_PTR(addr2)); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_AINT_DIFF); + return result; +} + +#endif /* CH4_INIT_H_INCLUDED */ diff --git a/src/mpid/ch4/src/ch4_probe.h b/src/mpid/ch4/src/ch4_probe.h new file mode 100644 index 0000000..89b7bc9 --- /dev/null +++ b/src/mpid/ch4/src/ch4_probe.h @@ -0,0 +1,220 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef CH4_PROBE_H_INCLUDED +#define CH4_PROBE_H_INCLUDED + +#include "ch4_impl.h" + +#undef FUNCNAME +#define FUNCNAME MPIDI_Probe +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Probe(int source, + int tag, MPIR_Comm * comm, int context_offset, MPI_Status * status) +{ + int mpi_errno, flag = 0; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_PROBE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_PROBE); + + if (unlikely(source == MPI_PROC_NULL)) { + MPIR_Status_set_procnull(status); + mpi_errno = MPI_SUCCESS; + goto fn_exit; + } + + while (!flag) { +#ifndef MPIDI_CH4_EXCLUSIVE_SHM + mpi_errno = MPIDI_NM_iprobe(source, tag, comm, context_offset, &flag, status); +#else + if (unlikely(source == MPI_ANY_SOURCE)) { + mpi_errno = MPIDI_SHM_iprobe(source, tag, comm, context_offset, &flag, status); + if (!flag) + mpi_errno = MPIDI_NM_iprobe(source, tag, comm, context_offset, &flag, status); + } + else if (MPIDI_CH4_rank_is_local(source, comm)) + mpi_errno = MPIDI_SHM_iprobe(source, tag, comm, context_offset, &flag, status); + else + mpi_errno = MPIDI_NM_iprobe(source, tag, comm, context_offset, &flag, status); +#endif + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } +#ifndef MPIDI_CH4_EXCLUSIVE_SHM + MPIDI_NM_progress(MPIDI_CH4_Global.netmod_context[0], 0); +#else + if (unlikely(source == MPI_ANY_SOURCE)) { + MPIDI_SHM_progress(0); + MPIDI_NM_progress(MPIDI_CH4_Global.netmod_context[0], 0); + } + if (MPIDI_CH4_rank_is_local(source, comm)) + MPIDI_SHM_progress(0); + else + MPIDI_NM_progress(MPIDI_CH4_Global.netmod_context[0], 0); +#endif + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_PROBE); + return mpi_errno; + fn_fail: + goto fn_exit; +} + + +#undef FUNCNAME +#define FUNCNAME MPIDI_Mprobe +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Mprobe(int source, + int tag, + MPIR_Comm * comm, + int context_offset, MPIR_Request ** message, MPI_Status * status) +{ + int mpi_errno = MPI_SUCCESS, flag = 0; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_MPROBE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_MPROBE); + + if (source == MPI_PROC_NULL) { + MPIR_Status_set_procnull(status); + *message = NULL; /* should be interpreted as MPI_MESSAGE_NO_PROC */ + mpi_errno = MPI_SUCCESS; + goto fn_exit; + } + + while (!flag) { +#ifndef MPIDI_CH4_EXCLUSIVE_SHM + mpi_errno = MPIDI_NM_improbe(source, tag, comm, context_offset, &flag, message, status); +#else + if (unlikely(source == MPI_ANY_SOURCE)) { + mpi_errno = + MPIDI_SHM_improbe(source, tag, comm, context_offset, &flag, message, status); + if (!flag) + mpi_errno = + MPIDI_NM_improbe(source, tag, comm, context_offset, &flag, message, status); + } + else if (MPIDI_CH4_rank_is_local(source, comm)) + mpi_errno = + MPIDI_SHM_improbe(source, tag, comm, context_offset, &flag, message, status); + else + mpi_errno = MPIDI_NM_improbe(source, tag, comm, context_offset, &flag, message, status); +#endif + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } +#ifndef MPIDI_CH4_EXCLUSIVE_SHM + MPIDI_NM_progress(MPIDI_CH4_Global.netmod_context[0], 0); +#else + if (unlikely(source == MPI_ANY_SOURCE)) { + MPIDI_SHM_progress(0); + MPIDI_NM_progress(MPIDI_CH4_Global.netmod_context[0], 0); + } + else if (MPIDI_CH4_rank_is_local(source, comm)) + MPIDI_SHM_progress(0); + else + MPIDI_NM_progress(MPIDI_CH4_Global.netmod_context[0], 0); +#endif + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_MPROBE); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Improbe +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Improbe(int source, + int tag, + MPIR_Comm * comm, + int context_offset, + int *flag, MPIR_Request ** message, MPI_Status * status) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_IMPROBE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_IMPROBE); + + if (source == MPI_PROC_NULL) { + MPIR_Status_set_procnull(status); + *flag = 1; + *message = NULL; /* should be interpreted as MPI_MESSAGE_NO_PROC */ + mpi_errno = MPI_SUCCESS; + goto fn_exit; + } + +#ifndef MPIDI_CH4_EXCLUSIVE_SHM + mpi_errno = MPIDI_NM_improbe(source, tag, comm, context_offset, flag, message, status); +#else + if (unlikely(source == MPI_ANY_SOURCE)) { + mpi_errno = MPIDI_SHM_improbe(source, tag, comm, context_offset, flag, message, status); + if (!*flag) + mpi_errno = MPIDI_NM_improbe(source, tag, comm, context_offset, flag, message, status); + } + else if (MPIDI_CH4_rank_is_local(source, comm)) + mpi_errno = MPIDI_SHM_improbe(source, tag, comm, context_offset, flag, message, status); + else + mpi_errno = MPIDI_NM_improbe(source, tag, comm, context_offset, flag, message, status); +#endif + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_IMPROBE); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Iprobe +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Iprobe(int source, + int tag, + MPIR_Comm * comm, + int context_offset, int *flag, MPI_Status * status) +{ + + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_IPROBE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_IPROBE); + + if (unlikely(source == MPI_PROC_NULL)) { + MPIR_Status_set_procnull(status); + *flag = 1; + mpi_errno = MPI_SUCCESS; + goto fn_exit; + } + +#ifndef MPIDI_CH4_EXCLUSIVE_SHM + mpi_errno = MPIDI_NM_iprobe(source, tag, comm, context_offset, flag, status); +#else + if (unlikely(source == MPI_ANY_SOURCE)) { + mpi_errno = MPIDI_SHM_iprobe(source, tag, comm, context_offset, flag, status); + if (!*flag) + mpi_errno = MPIDI_NM_iprobe(source, tag, comm, context_offset, flag, status); + } + else if (MPIDI_CH4_rank_is_local(source, comm)) + mpi_errno = MPIDI_SHM_iprobe(source, tag, comm, context_offset, flag, status); + else + mpi_errno = MPIDI_NM_iprobe(source, tag, comm, context_offset, flag, status); +#endif + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_IPROBE); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#endif /* CH4_PROBE_H_INCLUDED */ diff --git a/src/mpid/ch4/src/ch4_proc.h b/src/mpid/ch4/src/ch4_proc.h new file mode 100644 index 0000000..4aefcf0 --- /dev/null +++ b/src/mpid/ch4/src/ch4_proc.h @@ -0,0 +1,37 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef CH4_PROC_H_INCLUDED +#define CH4_PROC_H_INCLUDED + +#include "ch4_impl.h" + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4_rank_is_local +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_CH4_rank_is_local(int rank, MPIR_Comm * comm) +{ + int ret; + MPIR_FUNC_VERBOSE_STATE_DECL(MPIDI_STATE_CH4_RANK_IS_LOCAL); + MPIR_FUNC_VERBOSE_ENTER(MPIDI_STATE_CH4_RANK_IS_LOCAL); + +#ifndef MPIDI_CH4_EXCLUSIVE_SHM + /* Ask the netmod for locality information. If it decided not to build it, + * it will call back up to the CH4U function to get the infomration. */ + ret = MPIDI_NM_rank_is_local(rank, comm); +#else + ret = MPIDI_CH4U_rank_is_local(rank, comm); +#endif + + MPIR_FUNC_VERBOSE_EXIT(MPIDI_STATE_CH4_RANK_IS_LOCAL); + return ret; +} +#endif /* CH4_PROC_H_INCLUDED */ diff --git a/src/mpid/ch4/src/ch4_progress.h b/src/mpid/ch4/src/ch4_progress.h new file mode 100644 index 0000000..37bffef --- /dev/null +++ b/src/mpid/ch4/src/ch4_progress.h @@ -0,0 +1,185 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef CH4_PROGRESS_H_INCLUDED +#define CH4_PROGRESS_H_INCLUDED + +#include "ch4_impl.h" + +#undef FUNCNAME +#define FUNCNAME MPIDI_Progress_test +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Progress_test(void) +{ + int mpi_errno, made_progress, i; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_PROGRESS_TEST); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_PROGRESS_TEST); + + + if (OPA_load_int(&MPIDI_CH4_Global.active_progress_hooks)) { + MPID_THREAD_CS_ENTER(POBJ, MPIDI_CH4I_THREAD_PROGRESS_MUTEX); + for (i = 0; i < MAX_PROGRESS_HOOKS; i++) { + if (MPIDI_CH4_Global.progress_hooks[i].active == TRUE) { + MPIR_Assert(MPIDI_CH4_Global.progress_hooks[i].func_ptr != NULL); + mpi_errno = MPIDI_CH4_Global.progress_hooks[i].func_ptr(&made_progress); + if (mpi_errno) { + MPID_THREAD_CS_EXIT(POBJ, MPIDI_CH4I_THREAD_PROGRESS_MUTEX); + MPIR_ERR_POP(mpi_errno); + } + } + } + MPID_THREAD_CS_EXIT(POBJ, MPIDI_CH4I_THREAD_PROGRESS_MUTEX); + } + /* todo: progress unexp_list */ + mpi_errno = MPIDI_NM_progress(MPIDI_CH4_Global.netmod_context[0], 0); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } +#ifdef MPIDI_CH4_EXCLUSIVE_SHM + mpi_errno = MPIDI_SHM_progress(0); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } +#endif + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_PROGRESS_TEST); + return mpi_errno; + fn_fail: + goto fn_exit;; +} + +__CH4_INLINE__ int MPIDI_Progress_poke(void) +{ + return MPIDI_Progress_test(); +} + +__CH4_INLINE__ void MPIDI_Progress_start(MPID_Progress_state * state) +{ + return; +} + +__CH4_INLINE__ void MPIDI_Progress_end(MPID_Progress_state * state) +{ + return; +} + +__CH4_INLINE__ int MPIDI_Progress_wait(MPID_Progress_state * state) +{ + return MPIDI_Progress_test(); +} + + +#undef FUNCNAME +#define FUNCNAME MPIDI_Progress_register +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Progress_register(int (*progress_fn) (int *), int *id) +{ + int mpi_errno = MPI_SUCCESS; + int i; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDI_PROGRESS_REGISTER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDI_PROGRESS_REGISTER); + + MPID_THREAD_CS_ENTER(POBJ, MPIDI_CH4I_THREAD_PROGRESS_HOOK_MUTEX); + for (i = 0; i < MAX_PROGRESS_HOOKS; i++) { + if (MPIDI_CH4_Global.progress_hooks[i].func_ptr == NULL) { + MPIDI_CH4_Global.progress_hooks[i].func_ptr = progress_fn; + MPIDI_CH4_Global.progress_hooks[i].active = FALSE; + break; + } + } + + if (i >= MAX_PROGRESS_HOOKS) + goto fn_fail; + + OPA_incr_int(&MPIDI_CH4_Global.active_progress_hooks); + + (*id) = i; + + fn_exit: + MPID_THREAD_CS_EXIT(POBJ, MPIDI_CH4I_THREAD_PROGRESS_HOOK_MUTEX); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDI_PROGRESS_REGISTER); + return mpi_errno; + fn_fail: + mpi_errno = MPIR_Err_create_code(MPI_SUCCESS, MPIR_ERR_RECOVERABLE, + "MPIDI_Progress_register", __LINE__, + MPI_ERR_INTERN, "**progresshookstoomany", 0); + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Progress_deregister +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Progress_deregister(int id) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDI_PROGRESS_DEREGISTER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDI_PROGRESS_DEREGISTER); + + MPID_THREAD_CS_ENTER(POBJ, MPIDI_CH4I_THREAD_PROGRESS_HOOK_MUTEX); + MPIR_Assert(id >= 0); + MPIR_Assert(id < MAX_PROGRESS_HOOKS); + MPIR_Assert(MPIDI_CH4_Global.progress_hooks[id].func_ptr != NULL); + MPIDI_CH4_Global.progress_hooks[id].func_ptr = NULL; + MPIDI_CH4_Global.progress_hooks[id].active = FALSE; + + OPA_decr_int(&MPIDI_CH4_Global.active_progress_hooks); + MPID_THREAD_CS_EXIT(POBJ, MPIDI_CH4I_THREAD_PROGRESS_HOOK_MUTEX); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDI_PROGRESS_DEREGISTER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Progress_activate +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Progress_activate(int id) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDI_PROGRESS_ACTIVATE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDI_PROGRESS_ACTIVATE); + + MPID_THREAD_CS_ENTER(POBJ, MPIDI_CH4I_THREAD_PROGRESS_HOOK_MUTEX); + MPIR_Assert(id >= 0); + MPIR_Assert(id < MAX_PROGRESS_HOOKS); + MPIR_Assert(MPIDI_CH4_Global.progress_hooks[id].active == FALSE); + MPIR_Assert(MPIDI_CH4_Global.progress_hooks[id].func_ptr != NULL); + MPIDI_CH4_Global.progress_hooks[id].active = TRUE; + + MPID_THREAD_CS_EXIT(POBJ, MPIDI_CH4I_THREAD_PROGRESS_HOOK_MUTEX); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDI_PROGRESS_ACTIVATE); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Progress_deactivate +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Progress_deactivate(int id) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDI_PROGRESS_DEACTIVATE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDI_PROGRESS_DEACTIVATE); + + MPID_THREAD_CS_ENTER(POBJ, MPIDI_CH4I_THREAD_PROGRESS_HOOK_MUTEX); + MPIR_Assert(id >= 0); + MPIR_Assert(id < MAX_PROGRESS_HOOKS); + MPIR_Assert(MPIDI_CH4_Global.progress_hooks[id].active == TRUE); + MPIR_Assert(MPIDI_CH4_Global.progress_hooks[id].func_ptr != NULL); + MPIDI_CH4_Global.progress_hooks[id].active = FALSE; + + MPID_THREAD_CS_EXIT(POBJ, MPIDI_CH4I_THREAD_PROGRESS_HOOK_MUTEX); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDI_PROGRESS_DEACTIVATE); + return mpi_errno; +} + +#endif /* CH4_PROGRESS_H_INCLUDED */ diff --git a/src/mpid/ch4/src/ch4_recv.h b/src/mpid/ch4/src/ch4_recv.h new file mode 100644 index 0000000..71af9db --- /dev/null +++ b/src/mpid/ch4/src/ch4_recv.h @@ -0,0 +1,411 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef CH4_RECV_H_INCLUDED +#define CH4_RECV_H_INCLUDED + +#include "ch4_impl.h" + +#undef FUNCNAME +#define FUNCNAME MPIDI_Recv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Recv(void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, + int context_offset, MPI_Status * status, MPIR_Request ** request) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_RECV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_RECV); + + if (unlikely(rank == MPI_PROC_NULL)) { + MPIR_Request *rreq = MPIR_Request_create(MPIR_REQUEST_KIND__RECV); + *request = rreq; + MPIR_Request_add_ref(rreq); + rreq->status.MPI_SOURCE = rank; + rreq->status.MPI_TAG = tag; + MPIDI_CH4U_request_complete(rreq); + mpi_errno = MPI_SUCCESS; + goto fn_exit; + } + +#ifndef MPIDI_CH4_EXCLUSIVE_SHM + mpi_errno = + MPIDI_NM_recv(buf, count, datatype, rank, tag, comm, context_offset, status, request); +#else + if (unlikely(rank == MPI_ANY_SOURCE)) { + mpi_errno = MPIDI_SHM_irecv(buf, count, datatype, rank, tag, comm, context_offset, request); + + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + + mpi_errno = MPIDI_NM_irecv(buf, count, datatype, rank, tag, comm, context_offset, + &(MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(*request))); + + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + /* cancel the shm request if netmod/am handles the request from unexpected queue. */ + else if (*request) { + if (MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(*request)->status.MPI_SOURCE != MPI_UNDEFINED) { + mpi_errno = MPIDI_SHM_cancel_recv(*request); + if (MPIR_STATUS_GET_CANCEL_BIT((*request)->status)) { + (*request)->status = MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(*request)->status; + } + goto fn_exit; + } + MPIDI_CH4I_REQUEST(*request, is_local) = 1; + MPIDI_CH4I_REQUEST(MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(*request), is_local) = 0; + } + + MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(*request)) = + *request; + } + else { + int r; + if ((r = MPIDI_CH4_rank_is_local(rank, comm))) + mpi_errno = + MPIDI_SHM_recv(buf, count, datatype, rank, tag, comm, context_offset, status, + request); + else + mpi_errno = + MPIDI_NM_recv(buf, count, datatype, rank, tag, comm, context_offset, status, + request); + if (mpi_errno == MPI_SUCCESS && *request) { + MPIDI_CH4I_REQUEST(*request, is_local) = r; + MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(*request) = NULL; + } + } +#endif + + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_RECV); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Recv_init +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Recv_init(void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_RECV_INIT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_RECV_INIT); +#ifndef MPIDI_CH4_EXCLUSIVE_SHM + mpi_errno = MPIDI_NM_recv_init(buf, count, datatype, rank, tag, comm, context_offset, request); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_RECV_INIT); + return mpi_errno; +#else + if (unlikely(rank == MPI_ANY_SOURCE)) { + mpi_errno = + MPIDI_SHM_recv_init(buf, count, datatype, rank, tag, comm, context_offset, request); + + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + + mpi_errno = MPIDI_NM_recv_init(buf, count, datatype, rank, tag, comm, context_offset, + &(MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(*request))); + + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + + MPIDI_CH4I_REQUEST(*request, is_local) = 1; + MPIDI_CH4I_REQUEST(MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(*request), is_local) = 0; + + MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(*request)) = + *request; + } + else { + int r; + if ((r = MPIDI_CH4_rank_is_local(rank, comm))) + mpi_errno = MPIDI_SHM_recv_init(buf, count, datatype, rank, tag, + comm, context_offset, request); + else + mpi_errno = MPIDI_NM_recv_init(buf, count, datatype, rank, tag, + comm, context_offset, request); + if (mpi_errno == MPI_SUCCESS) { + MPIDI_CH4I_REQUEST(*request, is_local) = r; + MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(*request) = NULL; + } + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_RECV_INIT); + return mpi_errno; + fn_fail: + goto fn_exit; +#endif + +} + + +#undef FUNCNAME +#define FUNCNAME MPIDI_Mrecv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Mrecv(void *buf, + int count, + MPI_Datatype datatype, MPIR_Request * message, MPI_Status * status) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_MRECV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_MRECV); + + MPI_Request req_handle; + int active_flag; + MPIR_Request *rreq = NULL; + + if (message == NULL) { + /* treat as though MPI_MESSAGE_NO_PROC was passed */ + MPIR_Status_set_procnull(status); + mpi_errno = MPI_SUCCESS; + goto fn_exit; + } + + if (unlikely(message->status.MPI_SOURCE == MPI_PROC_NULL)) { + MPIR_Request *rreq = message; + rreq->status.MPI_SOURCE = message->status.MPI_SOURCE; + rreq->status.MPI_TAG = message->status.MPI_TAG; + MPIDI_CH4U_request_complete(rreq); + mpi_errno = MPI_SUCCESS; + goto fn_exit; + } + +#ifndef MPIDI_CH4_EXCLUSIVE_SHM + mpi_errno = MPIDI_NM_imrecv(buf, count, datatype, message, &rreq); +#else + + if (unlikely(message->status.MPI_SOURCE == MPI_ANY_SOURCE)) { + mpi_errno = MPIDI_SHM_imrecv(buf, count, datatype, message, &rreq); + + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + + mpi_errno = + MPIDI_NM_imrecv(buf, count, datatype, message, + &(MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(rreq))); + + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + + MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(rreq)) = rreq; + MPIDI_CH4I_REQUEST(rreq, is_local) = 1; + MPIDI_CH4I_REQUEST(MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(rreq), is_local) = 0; + } + else { + int local = MPIDI_CH4_rank_is_local(message->status.MPI_SOURCE, message->comm); + if (local) + mpi_errno = MPIDI_SHM_imrecv(buf, count, datatype, message, &rreq); + else + mpi_errno = MPIDI_NM_imrecv(buf, count, datatype, message, &rreq); + + if (mpi_errno == MPI_SUCCESS) { + MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(rreq) = NULL; + MPIDI_CH4I_REQUEST(rreq, is_local) = local; + } + } +#endif + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + while (!MPIR_Request_is_complete(rreq)) { + MPIDI_NM_progress(MPIDI_CH4_Global.netmod_context[0], 0); +#ifdef MPIDI_CH4_EXCLUSIVE_SHM + MPIDI_SHM_progress(0); +#endif + } + + /* This should probably be moved to MPICH (above device) level */ + /* Someone neglected to put the blocking at the MPICH level */ + MPIR_Request_extract_status(rreq, status); + MPIR_Request_complete(&req_handle, rreq, status, &active_flag); + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_MRECV); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Imrecv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Imrecv(void *buf, + int count, + MPI_Datatype datatype, + MPIR_Request * message, MPIR_Request ** rreqp) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_IMRECV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_IMRECV); + + if (message == NULL) { + MPIR_Request *rreq; + MPIDI_Request_create_null_rreq(rreq, mpi_errno, goto fn_fail); + *rreqp = rreq; + goto fn_exit; + } + + if (unlikely(message->status.MPI_SOURCE == MPI_PROC_NULL)) { + MPIR_Request *rreq = message; + rreq->status.MPI_SOURCE = message->status.MPI_SOURCE; + rreq->status.MPI_TAG = message->status.MPI_TAG; + MPIDI_CH4U_request_complete(rreq); + *rreqp = rreq; + mpi_errno = MPI_SUCCESS; + goto fn_exit; + } + +#ifndef MPIDI_CH4_EXCLUSIVE_SHM + mpi_errno = MPIDI_NM_imrecv(buf, count, datatype, message, rreqp); +#else + { + int local = MPIDI_CH4_rank_is_local(message->status.MPI_SOURCE, message->comm); + if (local) + mpi_errno = MPIDI_SHM_imrecv(buf, count, datatype, message, rreqp); + else + mpi_errno = MPIDI_NM_imrecv(buf, count, datatype, message, rreqp); + if (mpi_errno == MPI_SUCCESS) { + MPIDI_CH4I_REQUEST(*rreqp, is_local) = local; + MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(*rreqp) = NULL; + } + } +#endif + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_IMRECV); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Irecv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Irecv(void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_IRECV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_IRECV); + + if (unlikely(rank == MPI_PROC_NULL)) { + MPIR_Request *rreq = MPIR_Request_create(MPIR_REQUEST_KIND__RECV); + *request = rreq; + MPIR_Request_add_ref(rreq); + rreq->status.MPI_SOURCE = rank; + rreq->status.MPI_TAG = tag; + MPIDI_CH4U_request_complete(rreq); + mpi_errno = MPI_SUCCESS; + goto fn_exit; + } + +#ifndef MPIDI_CH4_EXCLUSIVE_SHM + mpi_errno = MPIDI_NM_irecv(buf, count, datatype, rank, tag, comm, context_offset, request); +#else + if (unlikely(rank == MPI_ANY_SOURCE)) { + mpi_errno = MPIDI_SHM_irecv(buf, count, datatype, rank, tag, comm, context_offset, request); + + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + + mpi_errno = MPIDI_NM_irecv(buf, count, datatype, rank, tag, comm, context_offset, + &(MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(*request))); + + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + else if (*request) { + MPIDI_CH4I_REQUEST(*request, is_local) = 1; + MPIDI_CH4I_REQUEST(MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(*request), is_local) = 0; + } + + MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(*request)) = + *request; + } + else { + int r; + if ((r = MPIDI_CH4_rank_is_local(rank, comm))) + mpi_errno = + MPIDI_SHM_irecv(buf, count, datatype, rank, tag, comm, context_offset, request); + else + mpi_errno = + MPIDI_NM_irecv(buf, count, datatype, rank, tag, comm, context_offset, request); + if (mpi_errno == MPI_SUCCESS && *request) { + MPIDI_CH4I_REQUEST(*request, is_local) = r; + MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(*request) = NULL; + } + } +#endif + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_IRECV); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Cancel_Recv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Cancel_recv(MPIR_Request * rreq) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_CANCEL_RECV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_CANCEL_RECV); +#ifndef MPIDI_CH4_EXCLUSIVE_SHM + mpi_errno = MPIDI_NM_cancel_recv(rreq); +#else + if (MPIDI_CH4I_REQUEST(rreq, is_local)) + mpi_errno = MPIDI_SHM_cancel_recv(rreq); + else + mpi_errno = MPIDI_NM_cancel_recv(rreq); +#endif + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_CANCEL_RECV); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#endif /* CH4_RECV_H_INCLUDED */ diff --git a/src/mpid/ch4/src/ch4_request.h b/src/mpid/ch4/src/ch4_request.h new file mode 100644 index 0000000..ac11dac --- /dev/null +++ b/src/mpid/ch4/src/ch4_request.h @@ -0,0 +1,91 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef CH4_REQUEST_H_INCLUDED +#define CH4_REQUEST_H_INCLUDED + +#include "ch4_impl.h" +#include "ch4r_buf.h" + +__CH4_INLINE__ int MPIDI_Request_is_anysource(MPIR_Request * req) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +__CH4_INLINE__ int MPIDI_Request_is_pending_failure(MPIR_Request * req) +{ + MPIR_Assert(0); + return MPI_SUCCESS; +} + +__CH4_INLINE__ void MPIDI_Request_set_completed(MPIR_Request * req) +{ + MPIR_cc_set(&req->cc, 0); + return; +} + +__CH4_INLINE__ void MPIDI_Request_add_ref(MPIR_Request * req) +{ + MPIR_Request_add_ref(req); + return; +} + +__CH4_INLINE__ void MPIDI_Request_release_ref(MPIR_Request * req) +{ + int inuse; + MPIR_Request_release_ref(req, &inuse); + return; +} + +/* These request functions should be called by the MPI layer only + since they only do base initialization of the request object. + A few notes: + + It is each layer's responsibility to initialize a request + properly. + + The CH4I_request functions are even more bare bones. + They create request objects that are not useable by the + lower layers until further initialization takes place. + + CH4R_request_xxx functions can be used to create and destroy + request objects at any CH4 layer, including shmmod and netmod. + These functions create and initialize a base request with + the appropriate "above device" fields initialized, and any + required CH4 layer fields initialized. + + The net/shm mods can upcall to CH4R to create a request, or + they can iniitalize their own requests internally, but note + that all the fields from the upper layers must be initialized + properly. + + Note that the request_release function is used by the MPI + layer to release the ref on a request object. It is important + for the netmods to release any memory pointed to by the request + when the internal completion counters hits zero, NOT when the + ref hits zero or there will be a memory leak. The generic + release function will not release any memory pointed to by + the request because it does not know about the internals of + the ch4r/netmod/shmmod fields of the request. +*/ +#undef FUNCNAME +#define FUNCNAME MPIDI_request_complete +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Request_complete(MPIR_Request * req) +{ + int incomplete; + MPIR_cc_decr(req->cc_ptr, &incomplete); + MPIR_Request_free(req); + return MPI_SUCCESS; +} + +#endif /* CH4_REQUEST_H_INCLUDED */ diff --git a/src/mpid/ch4/src/ch4_rma.h b/src/mpid/ch4/src/ch4_rma.h new file mode 100644 index 0000000..a85cdb8 --- /dev/null +++ b/src/mpid/ch4/src/ch4_rma.h @@ -0,0 +1,297 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef CH4_RMA_H_INCLUDED +#define CH4_RMA_H_INCLUDED + +#include "ch4_impl.h" + +#undef FUNCNAME +#define FUNCNAME MPIDI_Put +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Put(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, MPI_Datatype target_datatype, MPIR_Win * win) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_PUT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_PUT); + mpi_errno = MPIDI_NM_put(origin_addr, origin_count, origin_datatype, + target_rank, target_disp, target_count, target_datatype, win); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_PUT); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Get +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Get(void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, MPI_Datatype target_datatype, MPIR_Win * win) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_GET); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_GET); + mpi_errno = MPIDI_NM_get(origin_addr, origin_count, origin_datatype, + target_rank, target_disp, target_count, target_datatype, win); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_GET); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Accumulate +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, MPI_Op op, MPIR_Win * win) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_ACCUMULATE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_ACCUMULATE); + mpi_errno = MPIDI_NM_accumulate(origin_addr, origin_count, origin_datatype, + target_rank, target_disp, target_count, + target_datatype, op, win); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_ACCUMULATE); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Compare_and_swap +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Compare_and_swap(const void *origin_addr, + const void *compare_addr, + void *result_addr, + MPI_Datatype datatype, + int target_rank, MPI_Aint target_disp, MPIR_Win * win) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_COMPARE_AND_SWAP); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_COMPARE_AND_SWAP); + mpi_errno = MPIDI_NM_compare_and_swap(origin_addr, compare_addr, result_addr, + datatype, target_rank, target_disp, win); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_COMPARE_AND_SWAP); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Raccumulate +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Raccumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPI_Op op, MPIR_Win * win, MPIR_Request ** request) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_RACCUMULATE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_RACCUMULATE); + mpi_errno = MPIDI_NM_raccumulate(origin_addr, origin_count, origin_datatype, + target_rank, target_disp, target_count, + target_datatype, op, win, request); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_RACCUMULATE); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Rget_accumulate +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Rget_accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + void *result_addr, + int result_count, + MPI_Datatype result_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPI_Op op, MPIR_Win * win, MPIR_Request ** request) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_RGET_ACCUMULATE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_RGET_ACCUMULATE); + mpi_errno = MPIDI_NM_rget_accumulate(origin_addr, origin_count, origin_datatype, + result_addr, result_count, result_datatype, + target_rank, target_disp, target_count, + target_datatype, op, win, request); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_RGET_ACCUMULATE); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Fetch_and_op +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Fetch_and_op(const void *origin_addr, + void *result_addr, + MPI_Datatype datatype, + int target_rank, + MPI_Aint target_disp, MPI_Op op, MPIR_Win * win) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_FETCH_AND_OP); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_FETCH_AND_OP); + mpi_errno = MPIDI_NM_fetch_and_op(origin_addr, result_addr, + datatype, target_rank, target_disp, op, win); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_FETCH_AND_OP); + return mpi_errno; + fn_fail: + goto fn_exit; +} + + +#undef FUNCNAME +#define FUNCNAME MPIDI_Rget +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Rget(void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, MPIR_Win * win, MPIR_Request ** request) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_RGET); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_RGET); + mpi_errno = MPIDI_NM_rget(origin_addr, origin_count, origin_datatype, + target_rank, target_disp, target_count, + target_datatype, win, request); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_RGET); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Rput +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Rput(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, MPIR_Win * win, MPIR_Request ** request) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_RPUT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_RPUT); + mpi_errno = MPIDI_NM_rput(origin_addr, origin_count, origin_datatype, + target_rank, target_disp, target_count, + target_datatype, win, request); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_RPUT); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Get_accumulate +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Get_accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + void *result_addr, + int result_count, + MPI_Datatype result_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, MPI_Op op, MPIR_Win * win) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_GET_ACCUMULATE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_GET_ACCUMULATE); + mpi_errno = MPIDI_NM_get_accumulate(origin_addr, origin_count, origin_datatype, + result_addr, result_count, result_datatype, + target_rank, target_disp, target_count, target_datatype, + op, win); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_GET_ACCUMULATE); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#endif /* CH4_RMA_H_INCLUDED */ diff --git a/src/mpid/ch4/src/ch4_send.h b/src/mpid/ch4/src/ch4_send.h new file mode 100644 index 0000000..318e79b --- /dev/null +++ b/src/mpid/ch4/src/ch4_send.h @@ -0,0 +1,515 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef CH4_SEND_H_INCLUDED +#define CH4_SEND_H_INCLUDED + +#include "ch4_impl.h" +#include "ch4r_proc.h" +#undef FUNCNAME +#define FUNCNAME MPIDI_Send +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Send(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_SEND); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_SEND); + + if (unlikely(rank == MPI_PROC_NULL)) { + MPIR_Request *rreq = MPIR_Request_create(MPIR_REQUEST_KIND__SEND); + MPIR_Request_add_ref(rreq); + *request = rreq; + MPIDI_CH4U_request_complete(rreq); + mpi_errno = MPI_SUCCESS; + goto fn_exit; + } + +#ifndef MPIDI_CH4_EXCLUSIVE_SHM + mpi_errno = MPIDI_NM_send(buf, count, datatype, rank, tag, comm, context_offset, request); +#else + int r; + if ((r = MPIDI_CH4_rank_is_local(rank, comm))) + mpi_errno = MPIDI_SHM_send(buf, count, datatype, rank, tag, comm, context_offset, request); + else + mpi_errno = MPIDI_NM_send(buf, count, datatype, rank, tag, comm, context_offset, request); + if (mpi_errno == MPI_SUCCESS && *request) + MPIDI_CH4I_REQUEST(*request, is_local) = r; +#endif + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_SEND); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Isend +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Isend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_ISEND); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_ISEND); + + if (unlikely(rank == MPI_PROC_NULL)) { + MPIR_Request *rreq = MPIR_Request_create(MPIR_REQUEST_KIND__SEND); + MPIR_Request_add_ref(rreq); + *request = rreq; + MPIDI_CH4U_request_complete(rreq); + mpi_errno = MPI_SUCCESS; + goto fn_exit; + } + +#ifndef MPIDI_CH4_EXCLUSIVE_SHM + mpi_errno = MPIDI_NM_isend(buf, count, datatype, rank, tag, comm, context_offset, request); +#else + int r; + if ((r = MPIDI_CH4_rank_is_local(rank, comm))) + mpi_errno = MPIDI_SHM_isend(buf, count, datatype, rank, tag, comm, context_offset, request); + else + mpi_errno = MPIDI_NM_isend(buf, count, datatype, rank, tag, comm, context_offset, request); + if (mpi_errno == MPI_SUCCESS) + MPIDI_CH4I_REQUEST(*request, is_local) = r; +#endif + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_ISEND); + return mpi_errno; + fn_fail: + goto fn_exit; +} + + +#undef FUNCNAME +#define FUNCNAME MPIDI_Rsend +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Rsend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_RSEND); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_RSEND); + + if (unlikely(rank == MPI_PROC_NULL)) { + MPIR_Request *rreq = MPIR_Request_create(MPIR_REQUEST_KIND__SEND); + MPIR_Request_add_ref(rreq); + *request = rreq; + MPIDI_CH4U_request_complete(rreq); + mpi_errno = MPI_SUCCESS; + goto fn_exit; + } + +#ifndef MPIDI_CH4_EXCLUSIVE_SHM + mpi_errno = MPIDI_NM_send(buf, count, datatype, rank, tag, comm, context_offset, request); +#else + int r; + if ((r = MPIDI_CH4_rank_is_local(rank, comm))) + mpi_errno = MPIDI_SHM_send(buf, count, datatype, rank, tag, comm, context_offset, request); + else + mpi_errno = MPIDI_NM_send(buf, count, datatype, rank, tag, comm, context_offset, request); + if (mpi_errno == MPI_SUCCESS && *request) + MPIDI_CH4I_REQUEST(*request, is_local) = r; +#endif + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_RSEND); + return mpi_errno; + fn_fail: + goto fn_exit; +} + + +#undef FUNCNAME +#define FUNCNAME MPIDI_Irsend +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Irsend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_IRSEND); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_IRSEND); + + if (unlikely(rank == MPI_PROC_NULL)) { + MPIR_Request *rreq = MPIR_Request_create(MPIR_REQUEST_KIND__SEND); + MPIR_Request_add_ref(rreq); + *request = rreq; + MPIDI_CH4U_request_complete(rreq); + mpi_errno = MPI_SUCCESS; + goto fn_exit; + } + +#ifndef MPIDI_CH4_EXCLUSIVE_SHM + mpi_errno = MPIDI_NM_isend(buf, count, datatype, rank, tag, comm, context_offset, request); +#else + int r; + if ((r = MPIDI_CH4_rank_is_local(rank, comm))) + mpi_errno = MPIDI_SHM_isend(buf, count, datatype, rank, tag, comm, context_offset, request); + else + mpi_errno = MPIDI_NM_isend(buf, count, datatype, rank, tag, comm, context_offset, request); + if (mpi_errno == MPI_SUCCESS) + MPIDI_CH4I_REQUEST(*request, is_local) = r; +#endif + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_SEND); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Ssend +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Ssend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_SSEND); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_SSEND); + + if (unlikely(rank == MPI_PROC_NULL)) { + MPIR_Request *rreq = MPIR_Request_create(MPIR_REQUEST_KIND__SEND); + MPIR_Request_add_ref(rreq); + *request = rreq; + MPIDI_CH4U_request_complete(rreq); + mpi_errno = MPI_SUCCESS; + goto fn_exit; + } + +#ifndef MPIDI_CH4_EXCLUSIVE_SHM + mpi_errno = MPIDI_NM_ssend(buf, count, datatype, rank, tag, comm, context_offset, request); +#else + int r; + if ((r = MPIDI_CH4_rank_is_local(rank, comm))) + mpi_errno = MPIDI_SHM_ssend(buf, count, datatype, rank, tag, comm, context_offset, request); + else + mpi_errno = MPIDI_NM_ssend(buf, count, datatype, rank, tag, comm, context_offset, request); + if (mpi_errno == MPI_SUCCESS && *request) + MPIDI_CH4I_REQUEST(*request, is_local) = r; +#endif + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_SSEND); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Issend +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Issend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_ISSEND); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_ISSEND); + + if (unlikely(rank == MPI_PROC_NULL)) { + MPIR_Request *rreq = MPIR_Request_create(MPIR_REQUEST_KIND__SEND); + MPIR_Request_add_ref(rreq); + *request = rreq; + MPIDI_CH4U_request_complete(rreq); + mpi_errno = MPI_SUCCESS; + goto fn_exit; + } + +#ifndef MPIDI_CH4_EXCLUSIVE_SHM + mpi_errno = MPIDI_NM_issend(buf, count, datatype, rank, tag, comm, context_offset, request); +#else + int r; + if ((r = MPIDI_CH4_rank_is_local(rank, comm))) + mpi_errno = + MPIDI_SHM_issend(buf, count, datatype, rank, tag, comm, context_offset, request); + else + mpi_errno = MPIDI_NM_issend(buf, count, datatype, rank, tag, comm, context_offset, request); + if (mpi_errno == MPI_SUCCESS) + MPIDI_CH4I_REQUEST(*request, is_local) = r; +#endif + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_ISSEND); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Startall +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Startall(int count, MPIR_Request * requests[]) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_STARTALL); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_STARTALL); +#ifndef MPIDI_CH4_EXCLUSIVE_SHM + mpi_errno = MPIDI_NM_startall(count, requests); +#else + int i; + for (i = 0; i < count; i++) { + /* This is sub-optimal, can we do better? */ + if (MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(requests[i])) { + mpi_errno = MPIDI_SHM_startall(1, &requests[i]); + if (mpi_errno == MPI_SUCCESS) { + mpi_errno = + MPIDI_NM_startall(1, &MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(requests[i])); + MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(requests[i]->u.persist.real_request) = + MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(requests[i])->u.persist.real_request; + MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER + (requests[i])->u.persist.real_request) = + requests[i]->u.persist.real_request; + } + } + else if (MPIDI_CH4I_REQUEST(requests[i], is_local)) + mpi_errno = MPIDI_SHM_startall(1, &requests[i]); + else + mpi_errno = MPIDI_NM_startall(1, &requests[i]); + } +#endif + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_STARTALL); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Send_init +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Send_init(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_SEND_INIT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_SEND_INIT); +#ifndef MPIDI_CH4_EXCLUSIVE_SHM + mpi_errno = MPIDI_NM_send_init(buf, count, datatype, rank, tag, comm, context_offset, request); +#else + int r; + if ((r = MPIDI_CH4_rank_is_local(rank, comm))) + mpi_errno = MPIDI_SHM_send_init(buf, count, datatype, rank, tag, + comm, context_offset, request); + else + mpi_errno = MPIDI_NM_send_init(buf, count, datatype, rank, tag, + comm, context_offset, request); + if (mpi_errno == MPI_SUCCESS) + MPIDI_CH4I_REQUEST(*request, is_local) = r; + MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(*request) = NULL; +#endif + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_SEND_INIT); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Ssend_init +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Ssend_init(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_SSEND_INIT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_SSEND_INIT); +#ifndef MPIDI_CH4_EXCLUSIVE_SHM + mpi_errno = MPIDI_NM_ssend_init(buf, count, datatype, rank, tag, comm, context_offset, request); +#else + int r; + if ((r = MPIDI_CH4_rank_is_local(rank, comm))) + mpi_errno = MPIDI_SHM_ssend_init(buf, count, datatype, rank, tag, + comm, context_offset, request); + else + mpi_errno = MPIDI_NM_ssend_init(buf, count, datatype, rank, tag, + comm, context_offset, request); + if (mpi_errno == MPI_SUCCESS && *request) { + MPIDI_CH4I_REQUEST(*request, is_local) = r; + MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(*request) = NULL; + } +#endif + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_SSEND_INIT); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Bsend_init +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Bsend_init(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_BSEND_INIT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_BSEND_INIT); +#ifndef MPIDI_CH4_EXCLUSIVE_SHM + mpi_errno = MPIDI_NM_bsend_init(buf, count, datatype, rank, tag, comm, context_offset, request); +#else + int r; + if ((r = MPIDI_CH4_rank_is_local(rank, comm))) + mpi_errno = MPIDI_SHM_bsend_init(buf, count, datatype, rank, tag, + comm, context_offset, request); + else + mpi_errno = MPIDI_NM_bsend_init(buf, count, datatype, rank, tag, + comm, context_offset, request); + if (mpi_errno == MPI_SUCCESS && *request) { + MPIDI_CH4I_REQUEST(*request, is_local) = r; + MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(*request) = NULL; + } +#endif + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_BSEND_INIT); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Rsend_init +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Rsend_init(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_RSEND_INIT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_RSEND_INIT); +#ifndef MPIDI_CH4_EXCLUSIVE_SHM + mpi_errno = MPIDI_NM_rsend_init(buf, count, datatype, rank, tag, comm, context_offset, request); +#else + int r; + if ((r = MPIDI_CH4_rank_is_local(rank, comm))) + mpi_errno = MPIDI_SHM_rsend_init(buf, count, datatype, rank, tag, + comm, context_offset, request); + else + mpi_errno = MPIDI_NM_rsend_init(buf, count, datatype, rank, tag, + comm, context_offset, request); + if (mpi_errno == MPI_SUCCESS && *request) { + MPIDI_CH4I_REQUEST(*request, is_local) = r; + MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(*request) = NULL; + } +#endif + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_RSEND_INIT); + return mpi_errno; + fn_fail: + goto fn_exit; + +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Cancel_send +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Cancel_send(MPIR_Request * sreq) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_CANCEL_SEND); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_CANCEL_SEND); +#ifndef MPIDI_CH4_EXCLUSIVE_SHM + mpi_errno = MPIDI_NM_cancel_send(sreq); +#else + if (MPIDI_CH4I_REQUEST(sreq, is_local)) + mpi_errno = MPIDI_SHM_cancel_send(sreq); + else + mpi_errno = MPIDI_NM_cancel_send(sreq); +#endif + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_CANCEL_SEND); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#endif /* CH4_SEND_H_INCLUDED */ diff --git a/src/mpid/ch4/src/ch4_spawn.h b/src/mpid/ch4/src/ch4_spawn.h new file mode 100644 index 0000000..d4968a1 --- /dev/null +++ b/src/mpid/ch4/src/ch4_spawn.h @@ -0,0 +1,319 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef CH4_SPAWN_H_INCLUDED +#define CH4_SPAWN_H_INCLUDED + +#include "ch4_impl.h" + +static inline int MPIDI_mpi_to_pmi_keyvals(MPIR_Info * info_ptr, + PMI_keyval_t ** kv_ptr, int *nkeys_ptr) +{ + char key[MPI_MAX_INFO_KEY]; + PMI_keyval_t *kv = 0; + int i, nkeys = 0, vallen, flag, mpi_errno = MPI_SUCCESS; + + if (!info_ptr || info_ptr->handle == MPI_INFO_NULL) + goto fn_exit; + + MPIR_Info_get_nkeys_impl(info_ptr, &nkeys); + + if (nkeys == 0) + goto fn_exit; + + kv = (PMI_keyval_t *) MPL_malloc(nkeys * sizeof(PMI_keyval_t)); + + for (i = 0; i < nkeys; i++) { + mpi_errno = MPIR_Info_get_nthkey_impl(info_ptr, i, key); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + MPIR_Info_get_valuelen_impl(info_ptr, key, &vallen, &flag); + kv[i].key = (const char *) MPL_strdup(key); + kv[i].val = (char *) MPL_malloc(vallen + 1); + MPIR_Info_get_impl(info_ptr, key, vallen + 1, kv[i].val, &flag); + } + + fn_fail: + fn_exit: + *kv_ptr = kv; + *nkeys_ptr = nkeys; + return mpi_errno; +} + +static inline void MPIDI_free_pmi_keyvals(PMI_keyval_t ** kv, int size, int *counts) +{ + int i, j; + + for (i = 0; i < size; i++) { + for (j = 0; j < counts[i]; j++) { + if (kv[i][j].key != NULL) + MPL_free((char *) kv[i][j].key); + + if (kv[i][j].val != NULL) + MPL_free(kv[i][j].val); + } + + if (kv[i] != NULL) + MPL_free(kv[i]); + } + +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Comm_spawn_multiple +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Comm_spawn_multiple(int count, + char *commands[], + char **argvs[], + const int maxprocs[], + MPIR_Info * info_ptrs[], + int root, + MPIR_Comm * comm_ptr, + MPIR_Comm ** intercomm, int errcodes[]) +{ + char port_name[MPI_MAX_PORT_NAME]; + int *info_keyval_sizes = 0, i, mpi_errno = MPI_SUCCESS; + PMI_keyval_t **info_keyval_vectors = 0, preput_keyval_vector; + int *pmi_errcodes = 0, pmi_errno = 0; + int total_num_processes, should_accept = 1; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_COMM_SPAWN_MULTIPLE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_COMM_SPAWN_MULTIPLE); + + memset(port_name, 0, sizeof(port_name)); + + if (comm_ptr->rank == root) { + total_num_processes = 0; + + for (i = 0; i < count; i++) + total_num_processes += maxprocs[i]; + + pmi_errcodes = (int *) MPL_malloc(sizeof(int) * total_num_processes); + MPIR_ERR_CHKANDJUMP(!pmi_errcodes, mpi_errno, MPI_ERR_OTHER, "**nomem"); + + for (i = 0; i < total_num_processes; i++) + pmi_errcodes[i] = 0; + + mpi_errno = MPIDI_Open_port(NULL, port_name); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + info_keyval_sizes = (int *) MPL_malloc(count * sizeof(int)); + MPIR_ERR_CHKANDJUMP(!info_keyval_sizes, mpi_errno, MPI_ERR_OTHER, "**nomem"); + info_keyval_vectors = (PMI_keyval_t **) MPL_malloc(count * sizeof(PMI_keyval_t *)); + MPIR_ERR_CHKANDJUMP(!info_keyval_vectors, mpi_errno, MPI_ERR_OTHER, "**nomem"); + + if (!info_ptrs) + for (i = 0; i < count; i++) { + info_keyval_vectors[i] = 0; + info_keyval_sizes[i] = 0; + } + else + for (i = 0; i < count; i++) { + mpi_errno = MPIDI_mpi_to_pmi_keyvals(info_ptrs[i], + &info_keyval_vectors[i], + &info_keyval_sizes[i]); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + } + + preput_keyval_vector.key = MPIDI_PARENT_PORT_KVSKEY; + preput_keyval_vector.val = port_name; + pmi_errno = PMI_Spawn_multiple(count, (const char **) + commands, + (const char ** *) argvs, + maxprocs, info_keyval_sizes, (const PMI_keyval_t **) + info_keyval_vectors, 1, &preput_keyval_vector, pmi_errcodes); + + if (pmi_errno != PMI_SUCCESS) + MPIR_ERR_SETANDJUMP1(mpi_errno, MPI_ERR_OTHER, + "**pmi_spawn_multiple", "**pmi_spawn_multiple %d", pmi_errno); + + if (errcodes != MPI_ERRCODES_IGNORE) { + for (i = 0; i < total_num_processes; i++) { + errcodes[i] = pmi_errcodes[0]; + should_accept = should_accept && errcodes[i]; + } + + should_accept = !should_accept; + } + } + + if (errcodes != MPI_ERRCODES_IGNORE) { + MPIR_Errflag_t errflag = MPIR_ERR_NONE; + mpi_errno = MPIR_Bcast_impl(&should_accept, 1, MPI_INT, root, comm_ptr, &errflag); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + mpi_errno = MPIR_Bcast_impl(&pmi_errno, 1, MPI_INT, root, comm_ptr, &errflag); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + mpi_errno = MPIR_Bcast_impl(&total_num_processes, 1, MPI_INT, root, comm_ptr, &errflag); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + mpi_errno = MPIR_Bcast_impl(errcodes, total_num_processes, MPI_INT, + root, comm_ptr, &errflag); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + } + + if (should_accept) { + mpi_errno = MPIDI_Comm_accept(port_name, NULL, root, comm_ptr, intercomm); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + } + else { + if ((pmi_errno == PMI_SUCCESS) && (errcodes[0] != 0)) + MPIR_Comm_create(intercomm); + } + + if (comm_ptr->rank == root) { + mpi_errno = MPIDI_Close_port(port_name); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + } + + fn_exit: + + if (info_keyval_vectors) { + MPIDI_free_pmi_keyvals(info_keyval_vectors, count, info_keyval_sizes); + MPL_free(info_keyval_vectors); + } + + if (info_keyval_sizes) + MPL_free(info_keyval_sizes); + + if (pmi_errcodes) + MPL_free(pmi_errcodes); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_COMM_SPAWN_MULTIPLE); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Comm_connect +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Comm_connect(const char *port_name, + MPIR_Info * info, + int root, MPIR_Comm * comm, MPIR_Comm ** newcomm_ptr) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_COMM_CONNECT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_COMM_CONNECT); + mpi_errno = MPIDI_NM_comm_connect(port_name, info, root, comm, newcomm_ptr); + + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_COMM_CONNECT); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Comm_disconnect +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Comm_disconnect(MPIR_Comm * comm_ptr) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_COMM_DISCONNECT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_COMM_DISCONNECT); + mpi_errno = MPIDI_NM_comm_disconnect(comm_ptr); + + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_COMM_DISCONNECT); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Open_port +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Open_port(MPIR_Info * info_ptr, char *port_name) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_OPEN_PORT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_OPEN_PORT); + mpi_errno = MPIDI_NM_open_port(info_ptr, port_name); + + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_OPEN_PORT); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Close_port +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Close_port(const char *port_name) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_CLOSE_PORT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_CLOSE_PORT); + mpi_errno = MPIDI_NM_close_port(port_name); + + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_CLOSE_PORT); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Comm_accept +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Comm_accept(const char *port_name, + MPIR_Info * info, + int root, MPIR_Comm * comm, MPIR_Comm ** newcomm_ptr) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_COMM_ACCEPT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_COMM_ACCEPT); + mpi_errno = MPIDI_NM_comm_accept(port_name, info, root, comm, newcomm_ptr); + + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_COMM_ACCEPT); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#endif /* CH4_SPAWN_H_INCLUDED */ diff --git a/src/mpid/ch4/src/ch4_types.h b/src/mpid/ch4/src/ch4_types.h new file mode 100644 index 0000000..061122e --- /dev/null +++ b/src/mpid/ch4/src/ch4_types.h @@ -0,0 +1,285 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef CH4_TYPES_H_INCLUDED +#define CH4_TYPES_H_INCLUDED + +#include <mpidimpl.h> +#include <stdio.h> +#include "mpir_cvars.h" +#include "pmi.h" + +/* Macros and inlines */ +/* match/ignore bit manipulation + * + * 0123 4567 01234567 0123 4567 01234567 0123 4567 01234567 01234567 01234567 + * | | | + * ^ | context id | source | message tag + * | | | | + * +---- protocol + */ +#define MPIDI_CH4U_PROTOCOL_MASK (0x9000000000000000ULL) +#define MPIDI_CH4U_CONTEXT_MASK (0x0FFFF00000000000ULL) +#define MPIDI_CH4U_SOURCE_MASK (0x00000FFFF0000000ULL) +#define MPIDI_CH4U_TAG_MASK (0x000000000FFFFFFFULL) +#define MPIDI_CH4U_DYNPROC_SEND (0x4000000000000000ULL) +#define MPIDI_CH4U_TAG_SHIFT (28) +#define MPIDI_CH4U_SOURCE_SHIFT (16) +#define MPIDI_CH4U_SOURCE_SHIFT_UNPACK (sizeof(int)*8 - MPIDI_CH4U_SOURCE_SHIFT) +#define MPIDI_CH4U_TAG_SHIFT_UNPACK (sizeof(int)*8 - MPIDI_CH4U_TAG_SHIFT) + +#define MPIDI_CH4I_MAP_NOT_FOUND ((void*)(-1UL)) + +#define MAX_NETMOD_CONTEXTS 8 +#define MAX_PROGRESS_HOOKS 4 + +#define MPIDI_CH4I_BUF_POOL_NUM (1024) +#define MPIDI_CH4I_BUF_POOL_SZ (256) + +typedef int (*progress_func_ptr_t) (int *made_progress); +typedef struct progress_hook_slot { + progress_func_ptr_t func_ptr; + int active; +} progress_hook_slot_t; + +typedef enum { + MPIDI_CH4U_SEND = 0, /* Eager send */ + + MPIDI_CH4U_SEND_LONG_REQ, /* Rendezvous send RTS (request to send) */ + MPIDI_CH4U_SEND_LONG_ACK, /* Rendezvous send CTS (clear to send) */ + MPIDI_CH4U_SEND_LONG_LMT, /* Rendezvous send LMT */ + + MPIDI_CH4U_SSEND_REQ, + MPIDI_CH4U_SSEND_ACK, + + MPIDI_CH4U_WIN_CTRL, + + MPIDI_CH4U_PUT_REQ, + MPIDI_CH4U_PUT_ACK, + MPIDI_CH4U_PUT_IOV_REQ, + MPIDI_CH4U_PUT_DAT_REQ, + MPIDI_CH4U_PUT_IOV_ACK, + + MPIDI_CH4U_GET_REQ, + MPIDI_CH4U_GET_ACK, + + MPIDI_CH4U_ACC_REQ, + MPIDI_CH4U_ACC_ACK, + MPIDI_CH4U_ACC_IOV_REQ, + MPIDI_CH4U_ACC_DAT_REQ, + MPIDI_CH4U_ACC_IOV_ACK, + MPIDI_CH4U_GET_ACC_ACK, + + MPIDI_CH4U_CSWAP_REQ, + MPIDI_CH4U_CSWAP_ACK, + MPIDI_CH4U_FETCH_OP +} MPIDI_CH4U_TYPE; + +typedef enum { + MPIDI_CH4U_WIN_COMPLETE, + MPIDI_CH4U_WIN_POST, + MPIDI_CH4U_WIN_LOCK, + MPIDI_CH4U_WIN_LOCK_ACK, + MPIDI_CH4U_WIN_UNLOCK, + MPIDI_CH4U_WIN_UNLOCK_ACK, + MPIDI_CH4U_WIN_LOCKALL, + MPIDI_CH4U_WIN_LOCKALL_ACK, + MPIDI_CH4U_WIN_UNLOCKALL, + MPIDI_CH4U_WIN_UNLOCKALL_ACK +} MPIDI_CH4U_WIN_CTRL_MSG_TYPE; + +enum { + MPIDI_CH4U_EPOTYPE_NONE = 0, /**< No epoch in affect */ + MPIDI_CH4U_EPOTYPE_LOCK = 1, /**< MPI_Win_lock access epoch */ + MPIDI_CH4U_EPOTYPE_START = 2, /**< MPI_Win_start access epoch */ + MPIDI_CH4U_EPOTYPE_POST = 3, /**< MPI_Win_post exposure epoch */ + MPIDI_CH4U_EPOTYPE_FENCE = 4, /**< MPI_Win_fence access/exposure epoch */ + MPIDI_CH4U_EPOTYPE_REFENCE = 5, /**< MPI_Win_fence possible access/exposure epoch */ + MPIDI_CH4U_EPOTYPE_LOCK_ALL = 6 /**< MPI_Win_lock_all access epoch */ +}; + +/* Enum for calling types between netmod and shm */ +enum { + MPIDI_CH4R_NETMOD = 0, + MPIDI_CH4R_SHM = 1 +}; + +typedef struct MPIDI_CH4U_hdr_t { + uint64_t msg_tag; + int src_rank; +} MPIDI_CH4U_hdr_t; + +typedef struct MPIDI_CH4U_send_long_req_msg_t { + MPIDI_CH4U_hdr_t hdr; + size_t data_sz; /* Message size in bytes */ + uint64_t sreq_ptr; /* Pointer value of the request object at the sender side */ +} MPIDI_CH4U_send_long_req_msg_t; + +typedef struct MPIDI_CH4U_send_long_ack_msg_t { + uint64_t sreq_ptr; + uint64_t rreq_ptr; +} MPIDI_CH4U_send_long_ack_msg_t; + +typedef struct MPIDI_CH4U_send_long_lmt_msg_t { + uint64_t rreq_ptr; +} MPIDI_CH4U_send_long_lmt_msg_t; + +typedef struct MPIDI_CH4U_ssend_req_msg_t { + MPIDI_CH4U_hdr_t hdr; + uint64_t sreq_ptr; +} MPIDI_CH4U_ssend_req_msg_t; + +typedef struct MPIDI_CH4U_ssend_ack_msg_t { + uint64_t sreq_ptr; +} MPIDI_CH4U_ssend_ack_msg_t; + +typedef struct MPIDI_CH4U_win_cntrl_msg_t { + uint64_t win_id; + uint32_t origin_rank; + int16_t lock_type; + int16_t type; +} MPIDI_CH4U_win_cntrl_msg_t; + +typedef struct MPIDI_CH4U_put_msg_t { + int src_rank; + uint64_t win_id; + uint64_t preq_ptr; + MPI_Aint target_disp; + uint64_t count; + MPI_Datatype datatype; + int n_iov; +} MPIDI_CH4U_put_msg_t; + +typedef struct MPIDI_CH4U_put_iov_ack_msg_t { + int src_rank; + uint64_t target_preq_ptr; + uint64_t origin_preq_ptr; +} MPIDI_CH4U_put_iov_ack_msg_t; +typedef MPIDI_CH4U_put_iov_ack_msg_t MPIDI_CH4U_acc_iov_ack_msg_t; + +typedef struct MPIDI_CH4U_put_dat_msg_t { + uint64_t preq_ptr; +} MPIDI_CH4U_put_dat_msg_t; +typedef MPIDI_CH4U_put_dat_msg_t MPIDI_CH4U_acc_dat_msg_t; + +typedef struct MPIDI_CH4U_put_ack_msg_t { + uint64_t preq_ptr; +} MPIDI_CH4U_put_ack_msg_t; + +typedef struct MPIDI_CH4U_get_req_msg_t { + int src_rank; + uint64_t win_id; + uint64_t greq_ptr; + MPI_Aint target_disp; + uint64_t count; + MPI_Datatype datatype; + int n_iov; +} MPIDI_CH4U_get_req_msg_t; + +typedef struct MPIDI_CH4U_get_ack_msg_t { + uint64_t greq_ptr; +} MPIDI_CH4U_get_ack_msg_t; + +typedef struct MPIDI_CH4U_cswap_req_msg_t { + int src_rank; + uint64_t win_id; + uint64_t req_ptr; + MPI_Aint target_disp; + MPI_Datatype datatype; +} MPIDI_CH4U_cswap_req_msg_t; + +typedef struct MPIDI_CH4U_cswap_ack_msg_t { + uint64_t req_ptr; +} MPIDI_CH4U_cswap_ack_msg_t; + +typedef struct MPIDI_CH4U_acc_req_msg_t { + int src_rank; + uint64_t win_id; + uint64_t req_ptr; + int origin_count; + MPI_Datatype origin_datatype; + int target_count; + MPI_Datatype target_datatype; + MPI_Op op; + int do_get; + MPI_Aint target_disp; + uint64_t result_data_sz; + int n_iov; +} MPIDI_CH4U_acc_req_msg_t; + +typedef struct MPIDI_CH4U_acc_ack_msg_t { + uint64_t req_ptr; +} MPIDI_CH4U_acc_ack_msg_t; + +typedef struct MPIDI_CH4U_comm_req_list_t { + MPIR_Comm *comm[2][4]; + MPIDI_CH4U_rreq_t *uelist[2][4]; +} MPIDI_CH4U_comm_req_list_t; + +typedef struct MPIU_buf_pool_t { + int size; + int num; + void *memory_region; + struct MPIU_buf_pool_t *next; + struct MPIU_buf_t *head; + pthread_mutex_t lock; +} MPIU_buf_pool_t; + +typedef struct MPIU_buf_t { + struct MPIU_buf_t *next; + MPIU_buf_pool_t *pool; + char data[]; +} MPIU_buf_t; + +typedef struct { + int max_n_avts; + int n_avts; + int next_avtid; + int *free_avtid; +} MPIDI_CH4_avt_manager; + +typedef struct MPIDI_CH4_Global_t { + MPIR_Request *request_test; + MPIR_Comm *comm_test; + int pname_set; + int pname_len; + char pname[MPI_MAX_PROCESSOR_NAME]; + int is_initialized; + int allocated_max_n_avts; + MPIDI_CH4_avt_manager avt_mgr; + int is_ch4u_initialized; + MPID_Node_id_t **node_map, max_node_id; + MPIDI_CH4U_comm_req_list_t *comm_req_lists; + OPA_int_t active_progress_hooks; + MPIR_Commops MPIR_Comm_fns_store; + progress_hook_slot_t progress_hooks[MAX_PROGRESS_HOOKS]; + MPID_Thread_mutex_t m[2]; + MPIR_Win *win_hash; + int jobid; +#ifndef MPIDI_CH4U_USE_PER_COMM_QUEUE + MPIDI_CH4U_rreq_t *posted_list; + MPIDI_CH4U_rreq_t *unexp_list; +#endif + MPIDI_CH4U_req_ext_t *cmpl_list; + OPA_int_t exp_seq_no; + OPA_int_t nxt_seq_no; + void *netmod_context[8]; + MPIU_buf_pool_t *buf_pool; +} MPIDI_CH4_Global_t; +extern MPIDI_CH4_Global_t MPIDI_CH4_Global; +#ifdef MPL_USE_DBG_LOGGING +extern MPL_dbg_class MPIDI_CH4_DBG_GENERAL; +extern MPL_dbg_class MPIDI_CH4_DBG_MAP; +extern MPL_dbg_class MPIDI_CH4_DBG_MEMORY; +#endif +#define MPIDI_CH4I_THREAD_PROGRESS_MUTEX MPIDI_CH4_Global.m[0] +#define MPIDI_CH4I_THREAD_PROGRESS_HOOK_MUTEX MPIDI_CH4_Global.m[1] + +#endif /* CH4_TYPES_H_INCLUDED */ diff --git a/src/mpid/ch4/src/ch4_win.h b/src/mpid/ch4/src/ch4_win.h new file mode 100644 index 0000000..222f7fe --- /dev/null +++ b/src/mpid/ch4/src/ch4_win.h @@ -0,0 +1,530 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef CH4_WIN_H_INCLUDED +#define CH4_WIN_H_INCLUDED + +#include "ch4_impl.h" + +#undef FUNCNAME +#define FUNCNAME MPIDI_Win_set_info +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Win_set_info(MPIR_Win * win, MPIR_Info * info) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_WIN_SET_INFO); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_WIN_SET_INFO); + mpi_errno = MPIDI_NM_win_set_info(win, info); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_WIN_SET_INFO); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Win_start +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Win_start(MPIR_Group * group, int assert, MPIR_Win * win) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_WIN_START); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_WIN_START); + mpi_errno = MPIDI_NM_win_start(group, assert, win); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_WIN_START); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Win_complete +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Win_complete(MPIR_Win * win) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_WIN_COMPLETE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_WIN_COMPLETE); + mpi_errno = MPIDI_NM_win_complete(win); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_WIN_COMPLETE); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Win_post +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Win_post(MPIR_Group * group, int assert, MPIR_Win * win) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_WIN_POST); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_WIN_POST); + mpi_errno = MPIDI_NM_win_post(group, assert, win); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_WIN_POST); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Win_wait +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Win_wait(MPIR_Win * win) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_WIN_WAIT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_WIN_WAIT); + mpi_errno = MPIDI_NM_win_wait(win); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_WIN_WAIT); + return mpi_errno; + fn_fail: + goto fn_exit; +} + + +#undef FUNCNAME +#define FUNCNAME MPIDI_Win_test +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Win_test(MPIR_Win * win, int *flag) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_WIN_TEST); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_WIN_TEST); + mpi_errno = MPIDI_NM_win_test(win, flag); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_WIN_TEST); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Win_lock +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Win_lock(int lock_type, int rank, int assert, MPIR_Win * win) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_WIN_LOCK); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_WIN_LOCK); + mpi_errno = MPIDI_NM_win_lock(lock_type, rank, assert, win); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_WIN_LOCK); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Win_unlock +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Win_unlock(int rank, MPIR_Win * win) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_WIN_UNLOCK); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_WIN_UNLOCK); + mpi_errno = MPIDI_NM_win_unlock(rank, win); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_WIN_UNLOCK); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Win_get_info +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Win_get_info(MPIR_Win * win, MPIR_Info ** info_p_p) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_WIN_GET_INFO); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_WIN_GET_INFO); + mpi_errno = MPIDI_NM_win_get_info(win, info_p_p); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_WIN_GET_INFO); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Win_free +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Win_free(MPIR_Win ** win_ptr) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_WIN_FREE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_WIN_FREE); + mpi_errno = MPIDI_NM_win_free(win_ptr); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_WIN_FREE); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Win_fence +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Win_fence(int assert, MPIR_Win * win) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_WIN_FENCE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_WIN_FENCE); + mpi_errno = MPIDI_NM_win_fence(assert, win); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_WIN_FENCE); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Win_create +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Win_create(void *base, + MPI_Aint length, + int disp_unit, + MPIR_Info * info, MPIR_Comm * comm_ptr, MPIR_Win ** win_ptr) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_WIN_CREATE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_WIN_CREATE); + mpi_errno = MPIDI_NM_win_create(base, length, disp_unit, info, comm_ptr, win_ptr); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_WIN_CREATE); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Win_attach +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Win_attach(MPIR_Win * win, void *base, MPI_Aint size) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_WIN_ATTACH); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_WIN_ATTACH); + mpi_errno = MPIDI_NM_win_attach(win, base, size); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_WIN_ATTACH); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Win_allocate_shared +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Win_allocate_shared(MPI_Aint size, + int disp_unit, + MPIR_Info * info_ptr, + MPIR_Comm * comm_ptr, + void **base_ptr, MPIR_Win ** win_ptr) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_WIN_ALLOCATE_SHARED); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_WIN_ALLOCATE_SHARED); + mpi_errno = MPIDI_NM_win_allocate_shared(size, disp_unit, + info_ptr, comm_ptr, base_ptr, win_ptr); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_WIN_ALLOCATE_SHARED); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Win_flush_local +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Win_flush_local(int rank, MPIR_Win * win) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_WIN_FLUSH_LOCAL); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_WIN_FLUSH_LOCAL); + mpi_errno = MPIDI_NM_win_flush_local(rank, win); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_WIN_FLUSH_LOCAL); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Win_detach +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Win_detach(MPIR_Win * win, const void *base) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_WIN_DETACH); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_WIN_DETACH); + mpi_errno = MPIDI_NM_win_detach(win, base); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_WIN_DETACH); + return mpi_errno; + fn_fail: + goto fn_exit; +} + + +#undef FUNCNAME +#define FUNCNAME MPIDI_Win_shared_query +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Win_shared_query(MPIR_Win * win, + int rank, MPI_Aint * size, int *disp_unit, void *baseptr) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_WIN_SHARED_QUERY); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_WIN_SHARED_QUERY); + mpi_errno = MPIDI_NM_win_shared_query(win, rank, size, disp_unit, baseptr); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_WIN_SHARED_QUERY); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Win_allocate +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Win_allocate(MPI_Aint size, + int disp_unit, + MPIR_Info * info, + MPIR_Comm * comm, void *baseptr, MPIR_Win ** win) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_WIN_ALLOCATE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_WIN_ALLOCATE); + mpi_errno = MPIDI_NM_win_allocate(size, disp_unit, info, comm, baseptr, win); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_WIN_ALLOCATE); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Win_flush +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Win_flush(int rank, MPIR_Win * win) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_WIN_FLUSH); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_WIN_FLUSH); + mpi_errno = MPIDI_NM_win_flush(rank, win); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_WIN_FLUSH); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Win_flush_local_all +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Win_flush_local_all(MPIR_Win * win) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_WIN_FLUSH_LOCAL_ALL); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_WIN_FLUSH_LOCAL_ALL); + mpi_errno = MPIDI_NM_win_flush_local_all(win); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_WIN_FLUSH_LOCAL_ALL); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Win_unlock_all +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Win_unlock_all(MPIR_Win * win) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_WIN_UNLOCK_ALL); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_WIN_UNLOCK_ALL); + mpi_errno = MPIDI_NM_win_unlock_all(win); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_WIN_UNLOCK_ALL); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Win_create_dynamic +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Win_create_dynamic(MPIR_Info * info, MPIR_Comm * comm, MPIR_Win ** win) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_WIN_CREATE_DYNAMIC); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_WIN_CREATE_DYNAMIC); + mpi_errno = MPIDI_NM_win_create_dynamic(info, comm, win); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_WIN_CREATE_DYNAMIC); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Win_sync +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Win_sync(MPIR_Win * win) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_WIN_SYNC); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_WIN_SYNC); + mpi_errno = MPIDI_NM_win_sync(win); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_WIN_SYNC); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Win_flush_all +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Win_flush_all(MPIR_Win * win) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_WIN_FLUSH_ALL); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_WIN_FLUSH_ALL); + mpi_errno = MPIDI_NM_win_flush_all(win); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_WIN_FLUSH_ALL); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Win_lock_all +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_Win_lock_all(int assert, MPIR_Win * win) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_WIN_LOCK_ALL); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_WIN_LOCK_ALL); + mpi_errno = MPIDI_NM_win_lock_all(assert, win); + if (mpi_errno != MPI_SUCCESS) { + MPIR_ERR_POP(mpi_errno); + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_WIN_LOCK_ALL); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#endif /* CH4_WIN_H_INCLUDED */ diff --git a/src/mpid/ch4/src/ch4i_comm.h b/src/mpid/ch4/src/ch4i_comm.h new file mode 100644 index 0000000..b34405c --- /dev/null +++ b/src/mpid/ch4/src/ch4i_comm.h @@ -0,0 +1,1161 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2016 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + */ +#ifndef CH4I_COMM_H_INCLUDED +#define CH4I_COMM_H_INCLUDED + +#include "ch4_types.h" +#include "mpl_utlist.h" + +#undef FUNCNAME +#define FUNCNAME MPIDII_alloc_lut +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDII_alloc_lut(MPIDII_rank_map_lut_t ** lut, int size) +{ + int mpi_errno = MPI_SUCCESS; + MPIDII_rank_map_lut_t *new_lut = NULL; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDIU_ALLOC_LUT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDIU_ALLOC_LUT); + + new_lut = (MPIDII_rank_map_lut_t *) MPL_malloc(sizeof(MPIDII_rank_map_lut_t) + + size * sizeof(MPIDII_lpid_t)); + if (new_lut == NULL) { + *lut = NULL; + MPIR_ERR_SETANDJUMP(mpi_errno, MPI_ERR_OTHER, "**nomem"); + } + + MPIR_Object_set_ref(new_lut, 1); + *lut = new_lut; + + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MEMORY, VERBOSE, + (MPL_DBG_FDEST, "alloc lut %p, size %ld, refcount=%d", + new_lut, size * sizeof(MPIDII_lpid_t), MPIR_Object_get_ref(new_lut))); + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDIU_ALLOC_LUT); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDIU_release_lut +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDIU_release_lut(MPIDII_rank_map_lut_t * lut) +{ + int mpi_errno = MPI_SUCCESS; + int count = 0; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDIU_RELEASE_LUT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDIU_RELEASE_LUT); + + MPIR_Object_release_ref(lut, &count); + if (count == 0) { + MPL_free(lut); + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MEMORY, VERBOSE, (MPL_DBG_FDEST, "free lut %p", lut)); + } + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDIU_RELEASE_LUT); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDII_alloc_mlut +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDII_alloc_mlut(MPIDII_rank_map_mlut_t ** mlut, int size) +{ + int mpi_errno = MPI_SUCCESS; + MPIDII_rank_map_mlut_t *new_mlut = NULL; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDIU_ALLOC_MLUT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDIU_ALLOC_MLUT); + + new_mlut = (MPIDII_rank_map_mlut_t *) MPL_malloc(sizeof(MPIDII_rank_map_mlut_t) + + size * sizeof(MPIDII_gpid_t)); + if (new_mlut == NULL) { + *mlut = NULL; + MPIR_ERR_SETANDJUMP(mpi_errno, MPI_ERR_OTHER, "**nomem"); + } + + MPIR_Object_set_ref(new_mlut, 1); + *mlut = new_mlut; + + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MEMORY, VERBOSE, + (MPL_DBG_FDEST, "alloc mlut %p, size %ld, refcount=%d", + new_mlut, size * sizeof(MPIDII_gpid_t), MPIR_Object_get_ref(new_mlut))); + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDIU_ALLOC_MLUT); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDIU_release_mlut +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDIU_release_mlut(MPIDII_rank_map_mlut_t * mlut) +{ + int mpi_errno = MPI_SUCCESS; + int count = 0; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDIU_RELEASE_MLUT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDIU_RELEASE_MLUT); + + MPIR_Object_release_ref(mlut, &count); + if (count == 0) { + MPL_free(mlut); + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MEMORY, VERBOSE, (MPL_DBG_FDEST, "free mlut %p", mlut)); + } + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDIU_RELEASE_MLUT); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDII_map_size +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDII_map_size(MPIR_Comm_map_t map) +{ + if (map.type == MPIR_COMM_MAP_TYPE__IRREGULAR) + return map.src_mapping_size; + else if (map.dir == MPIR_COMM_MAP_DIR__L2L || map.dir == MPIR_COMM_MAP_DIR__L2R) + return map.src_comm->local_size; + else + return map.src_comm->remote_size; +} + +/* + * This enum is used exclusively in this header file + */ +enum MPIDII_src_mapper_models { + MPIDII_SRC_MAPPER_IRREGULAR = 0, + MPIDII_SRC_MAPPER_DIRECT = 1, + MPIDII_SRC_MAPPER_OFFSET = 2, + MPIDII_SRC_MAPPER_STRIDE = 3 +}; + +#undef FUNCNAME +#define FUNCNAME MPIDII_detect_regular_model +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDII_detect_regular_model(int *lpid, int size, + int *offset, int *blocksize, int *stride) +{ + int off = 0, bs = 0, st = 0; + int i; + + off = lpid[0]; + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, (MPL_DBG_FDEST, "\tdetect model: offset %d", off)); + + for (i = 0; i < size; i++) { + if (lpid[i] != i + off) { + break; + } + bs++; + } + + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, + (MPL_DBG_FDEST, "\tdetect model: blocksize %d", bs)); + if (bs == size) { + if (off == 0) { + return MPIDII_SRC_MAPPER_DIRECT; + } + else { + *offset = off; + return MPIDII_SRC_MAPPER_OFFSET; + } + } + + /* blocksize less than total size, try if this is stride */ + st = lpid[bs] - lpid[0]; + if (st < 0 || st <= bs) { + return MPIDII_SRC_MAPPER_IRREGULAR; + } + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, (MPL_DBG_FDEST, "\tdetect model: stride %d", bs)); + for (i = bs; i < size; i++) { + if (lpid[i] != MPIDII_CALC_STRIDE(i, st, bs, off)) { + return MPIDII_SRC_MAPPER_IRREGULAR; + } + } + *offset = off; + *blocksize = bs; + *stride = st; + return MPIDII_SRC_MAPPER_STRIDE; +} + +#undef FUNCNAME +#define FUNCNAME MPIDII_src_comm_to_lut +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDII_src_comm_to_lut(MPIDII_rank_map_t * src, + MPIDII_rank_map_t * dest, + int size, int total_mapper_size, int mapper_offset) +{ + int mpi_errno = MPI_SUCCESS, i; + MPIDII_rank_map_lut_t *lut = NULL; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDII_SRC_COMM_TO_LUT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDII_SRC_COMM_TO_LUT); + + if (!mapper_offset) { + mpi_errno = MPIDII_alloc_lut(&lut, total_mapper_size); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + dest->size = total_mapper_size; + dest->mode = MPIDII_RANK_MAP_LUT; + dest->avtid = src->avtid; + dest->irreg.lut.t = lut; + dest->irreg.lut.lpid = lut->lpid; + } + + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, + (MPL_DBG_FDEST, " source mode %d", (int) src->mode)); + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, (MPL_DBG_FDEST, " size %d", size)); + switch (src->mode) { + case MPIDII_RANK_MAP_DIRECT: + case MPIDII_RANK_MAP_DIRECT_INTRA: + for (i = 0; i < size; i++) { + dest->irreg.lut.lpid[i + mapper_offset] = i; + } + break; + case MPIDII_RANK_MAP_OFFSET: + case MPIDII_RANK_MAP_OFFSET_INTRA: + for (i = 0; i < size; i++) { + dest->irreg.lut.lpid[i + mapper_offset] = i + src->reg.offset; + } + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, + (MPL_DBG_FDEST, " source offset %d", src->reg.offset)); + break; + case MPIDII_RANK_MAP_STRIDE: + case MPIDII_RANK_MAP_STRIDE_INTRA: + for (i = 0; i < size; i++) { + dest->irreg.lut.lpid[i + mapper_offset] = MPIDII_CALC_STRIDE_SIMPLE(i, + src->reg.stride. + stride, + src->reg.stride. + offset); + } + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, + (MPL_DBG_FDEST, " source stride %d blocksize %d offset %d", + src->reg.stride.stride, src->reg.stride.blocksize, + src->reg.stride.offset)); + break; + case MPIDII_RANK_MAP_STRIDE_BLOCK: + case MPIDII_RANK_MAP_STRIDE_BLOCK_INTRA: + for (i = 0; i < size; i++) { + dest->irreg.lut.lpid[i + mapper_offset] = MPIDII_CALC_STRIDE(i, + src->reg.stride.stride, + src->reg.stride.blocksize, + src->reg.stride.offset); + } + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, + (MPL_DBG_FDEST, " source stride %d blocksize %d offset %d", + src->reg.stride.stride, src->reg.stride.blocksize, + src->reg.stride.offset)); + break; + case MPIDII_RANK_MAP_LUT: + case MPIDII_RANK_MAP_LUT_INTRA: + for (i = 0; i < size; i++) { + dest->irreg.lut.lpid[i + mapper_offset] = src->irreg.lut.lpid[i]; + } + break; + case MPIDII_RANK_MAP_MLUT: + case MPIDII_RANK_MAP_NONE: + MPIR_Assert(0); + break; + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDII_SRC_COMM_TO_LUT); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDII_src_comm_to_mlut +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDII_src_comm_to_mlut(MPIDII_rank_map_t * src, + MPIDII_rank_map_t * dest, + int size, int total_mapper_size, int mapper_offset) +{ + int mpi_errno = MPI_SUCCESS, i; + MPIDII_rank_map_mlut_t *mlut = NULL; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDII_SRC_COMM_TO_MLUT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDII_SRC_COMM_TO_MLUT); + + if (!mapper_offset) { + mpi_errno = MPIDII_alloc_mlut(&mlut, total_mapper_size); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + dest->size = total_mapper_size; + dest->mode = MPIDII_RANK_MAP_MLUT; + dest->avtid = -1; + dest->irreg.mlut.t = mlut; + dest->irreg.mlut.gpid = mlut->gpid; + } + + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, + (MPL_DBG_FDEST, " source mode %d", (int) src->mode)); + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, (MPL_DBG_FDEST, " size %d", size)); + switch (src->mode) { + case MPIDII_RANK_MAP_DIRECT: + case MPIDII_RANK_MAP_DIRECT_INTRA: + for (i = 0; i < size; i++) { + dest->irreg.mlut.gpid[i + mapper_offset].lpid = i; + dest->irreg.mlut.gpid[i + mapper_offset].avtid = src->avtid; + } + break; + case MPIDII_RANK_MAP_OFFSET: + case MPIDII_RANK_MAP_OFFSET_INTRA: + for (i = 0; i < size; i++) { + dest->irreg.mlut.gpid[i + mapper_offset].lpid = i + src->reg.offset; + dest->irreg.mlut.gpid[i + mapper_offset].avtid = src->avtid; + } + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, + (MPL_DBG_FDEST, " source offset %d", src->reg.offset)); + break; + case MPIDII_RANK_MAP_STRIDE: + case MPIDII_RANK_MAP_STRIDE_INTRA: + for (i = 0; i < size; i++) { + dest->irreg.mlut.gpid[i + mapper_offset].lpid = MPIDII_CALC_STRIDE_SIMPLE(i, + src->reg. + stride.stride, + src->reg. + stride. + offset); + dest->irreg.mlut.gpid[i + mapper_offset].avtid = src->avtid; + } + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, + (MPL_DBG_FDEST, " source stride %d blocksize %d offset %d", + src->reg.stride.stride, src->reg.stride.blocksize, + src->reg.stride.offset)); + break; + case MPIDII_RANK_MAP_STRIDE_BLOCK: + case MPIDII_RANK_MAP_STRIDE_BLOCK_INTRA: + for (i = 0; i < size; i++) { + dest->irreg.mlut.gpid[i + mapper_offset].lpid = MPIDII_CALC_STRIDE(i, + src->reg.stride. + stride, + src->reg.stride. + blocksize, + src->reg.stride. + offset); + dest->irreg.mlut.gpid[i + mapper_offset].avtid = src->avtid; + } + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, + (MPL_DBG_FDEST, " source stride %d blocksize %d offset %d", + src->reg.stride.stride, src->reg.stride.blocksize, + src->reg.stride.offset)); + break; + case MPIDII_RANK_MAP_LUT: + case MPIDII_RANK_MAP_LUT_INTRA: + for (i = 0; i < size; i++) { + dest->irreg.mlut.gpid[i + mapper_offset].lpid = src->irreg.lut.lpid[i]; + dest->irreg.mlut.gpid[i + mapper_offset].avtid = src->avtid; + } + break; + case MPIDII_RANK_MAP_MLUT: + for (i = 0; i < size; i++) { + dest->irreg.mlut.gpid[i + mapper_offset].lpid = src->irreg.mlut.gpid[i].lpid; + dest->irreg.mlut.gpid[i + mapper_offset].avtid = src->irreg.mlut.gpid[i].avtid; + } + break; + case MPIDII_RANK_MAP_NONE: + MPIR_Assert(0); + break; + } + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDII_SRC_COMM_TO_MLUT); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDII_src_mlut_to_mlut +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDII_src_mlut_to_mlut(MPIDII_rank_map_t * src, + MPIDII_rank_map_t * dest, + MPIR_Comm_map_t * mapper, + int total_mapper_size, int mapper_offset) +{ + int mpi_errno = MPI_SUCCESS, i; + int size = MPIDII_map_size(*mapper); + MPIDII_rank_map_mlut_t *mlut = NULL; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDII_MLUT_TO_MLUT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDII_MLUT_TO_MLUT); + + if (!mapper_offset) { + mpi_errno = MPIDII_alloc_mlut(&mlut, total_mapper_size); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + dest->size = total_mapper_size; + } + + dest->mode = MPIDII_RANK_MAP_MLUT; + dest->mode = src->mode; + dest->irreg.mlut.t = mlut; + dest->irreg.mlut.gpid = mlut->gpid; + for (i = 0; i < size; i++) { + dest->irreg.mlut.gpid[i + mapper_offset].avtid = + src->irreg.mlut.gpid[mapper->src_mapping[i]].avtid; + dest->irreg.mlut.gpid[i + mapper_offset].lpid = + src->irreg.mlut.gpid[mapper->src_mapping[i]].lpid; + } + fn_exit: + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, + (MPL_DBG_FDEST, " src mode %d, dest mode %d", + (int) src->mode, (int) dest->mode)); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDII_MLUT_TO_MLUT); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDII_src_map_to_lut +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDII_src_map_to_lut(MPIDII_rank_map_t * src, + MPIDII_rank_map_t * dest, + MPIR_Comm_map_t * mapper, + int total_mapper_size, int mapper_offset) +{ + int mpi_errno = MPI_SUCCESS, i; + int size = MPIDII_map_size(*mapper); + MPIDII_rank_map_lut_t *lut = NULL; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDII_MAP_TO_LUT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDII_MAP_TO_LUT); + + if (!mapper_offset) { + mpi_errno = MPIDII_alloc_lut(&lut, total_mapper_size); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + dest->size = total_mapper_size; + } + + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, + (MPL_DBG_FDEST, " source mode %d", (int) src->mode)); + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, + (MPL_DBG_FDEST, " size %d, mapper->src_mapping_size %d", + size, mapper->src_mapping_size)); + dest->mode = MPIDII_RANK_MAP_LUT; + dest->avtid = src->avtid; + dest->irreg.lut.t = lut; + dest->irreg.lut.lpid = lut->lpid; + switch (src->mode) { + case MPIDII_RANK_MAP_DIRECT: + case MPIDII_RANK_MAP_DIRECT_INTRA: + for (i = 0; i < size; i++) { + dest->irreg.lut.lpid[i + mapper_offset] = mapper->src_mapping[i]; + } + break; + case MPIDII_RANK_MAP_OFFSET: + case MPIDII_RANK_MAP_OFFSET_INTRA: + for (i = 0; i < size; i++) { + dest->irreg.lut.lpid[i + mapper_offset] = mapper->src_mapping[i] + src->reg.offset; + } + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, + (MPL_DBG_FDEST, " source offset %d", src->reg.offset)); + break; + case MPIDII_RANK_MAP_STRIDE: + case MPIDII_RANK_MAP_STRIDE_INTRA: + for (i = 0; i < size; i++) { + dest->irreg.lut.lpid[i + mapper_offset] = + MPIDII_CALC_STRIDE_SIMPLE(mapper->src_mapping[i], src->reg.stride.stride, + src->reg.stride.offset); + } + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, + (MPL_DBG_FDEST, " source stride %d blocksize %d offset %d", + src->reg.stride.stride, src->reg.stride.blocksize, + src->reg.stride.offset)); + break; + case MPIDII_RANK_MAP_STRIDE_BLOCK: + case MPIDII_RANK_MAP_STRIDE_BLOCK_INTRA: + for (i = 0; i < size; i++) { + dest->irreg.lut.lpid[i + mapper_offset] = MPIDII_CALC_STRIDE(mapper->src_mapping[i], + src->reg.stride.stride, + src->reg.stride.blocksize, + src->reg.stride.offset); + } + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, + (MPL_DBG_FDEST, " source stride %d blocksize %d offset %d", + src->reg.stride.stride, src->reg.stride.blocksize, + src->reg.stride.offset)); + break; + case MPIDII_RANK_MAP_LUT: + case MPIDII_RANK_MAP_LUT_INTRA: + for (i = 0; i < size; i++) { + dest->irreg.lut.lpid[i + mapper_offset] = src->irreg.lut.lpid[mapper->src_mapping[i]]; + } + break; + default: + mpi_errno = 1; + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, + (MPL_DBG_FDEST, " cannot convert mode %d to lut", (int) src->mode)); + goto fn_fail; + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDII_MAP_TO_LUT); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDII_direct_of_src_rmap +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline void MPIDII_direct_of_src_rmap(MPIDII_rank_map_t * src, + MPIDII_rank_map_t * dest, MPIR_Comm_map_t * mapper) +{ + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDII_DIRECT_OF_SRC_RMAP); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDII_DIRECT_OF_SRC_RMAP); + dest->mode = src->mode; + dest->size = MPIDII_map_size(*mapper); + dest->avtid = src->avtid; + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, + (MPL_DBG_FDEST, " source mode %d", (int) src->mode)); + switch (src->mode) { + case MPIDII_RANK_MAP_DIRECT: + case MPIDII_RANK_MAP_DIRECT_INTRA: + break; + case MPIDII_RANK_MAP_OFFSET: + case MPIDII_RANK_MAP_OFFSET_INTRA: + dest->reg.offset = src->reg.offset; + break; + case MPIDII_RANK_MAP_STRIDE: + case MPIDII_RANK_MAP_STRIDE_INTRA: + case MPIDII_RANK_MAP_STRIDE_BLOCK: + case MPIDII_RANK_MAP_STRIDE_BLOCK_INTRA: + dest->reg.stride.stride = src->reg.stride.stride; + dest->reg.stride.blocksize = src->reg.stride.blocksize; + dest->reg.stride.offset = src->reg.stride.offset; + break; + case MPIDII_RANK_MAP_LUT: + case MPIDII_RANK_MAP_LUT_INTRA: + dest->irreg.lut.t = src->irreg.lut.t; + dest->irreg.lut.lpid = src->irreg.lut.lpid; + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, + (MPL_DBG_FDEST, "\tref count %d", MPIR_Object_get_ref(src->irreg.lut.t))); + MPIR_Object_add_ref(src->irreg.lut.t); + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, (MPL_DBG_FDEST, "\tadd ref to src lut")); + break; + case MPIDII_RANK_MAP_MLUT: + dest->irreg.mlut.t = src->irreg.mlut.t; + dest->irreg.mlut.gpid = src->irreg.mlut.gpid; + MPIR_Object_add_ref(src->irreg.mlut.t); + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, (MPL_DBG_FDEST, "\tadd ref to src mlut")); + break; + case MPIDII_RANK_MAP_NONE: + MPIR_Assert(0); + break; + } + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDII_DIRECT_OF_SRC_RMAP); +} + +#undef FUNCNAME +#define FUNCNAME MPIDII_offset_of_src_rmap +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline void MPIDII_offset_of_src_rmap(MPIDII_rank_map_t * src, + MPIDII_rank_map_t * dest, + MPIR_Comm_map_t * mapper, int offset) +{ + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDII_OFFSET_OF_SRC_RMAP); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDII_OFFSET_OF_SRC_RMAP); + dest->avtid = src->avtid; + dest->size = MPIDII_map_size(*mapper); + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, + (MPL_DBG_FDEST, " source mode %d", (int) src->mode)); + switch (src->mode) { + case MPIDII_RANK_MAP_DIRECT_INTRA: + dest->mode = MPIDII_RANK_MAP_OFFSET_INTRA; + dest->reg.offset = offset; + break; + case MPIDII_RANK_MAP_DIRECT: + dest->mode = MPIDII_RANK_MAP_OFFSET; + dest->reg.offset = offset; + break; + case MPIDII_RANK_MAP_OFFSET: + dest->mode = MPIDII_RANK_MAP_OFFSET; + dest->reg.offset = src->reg.offset + offset; + break; + case MPIDII_RANK_MAP_OFFSET_INTRA: + dest->mode = MPIDII_RANK_MAP_OFFSET_INTRA; + dest->reg.offset = src->reg.offset + offset; + break; + case MPIDII_RANK_MAP_STRIDE: + dest->mode = MPIDII_RANK_MAP_STRIDE; + dest->reg.stride.stride = src->reg.stride.stride; + dest->reg.stride.blocksize = src->reg.stride.blocksize; + dest->reg.stride.offset = src->reg.stride.offset + offset * src->reg.stride.stride; + break; + case MPIDII_RANK_MAP_STRIDE_INTRA: + dest->mode = MPIDII_RANK_MAP_STRIDE_INTRA; + dest->reg.stride.stride = src->reg.stride.stride; + dest->reg.stride.blocksize = src->reg.stride.blocksize; + dest->reg.stride.offset = src->reg.stride.offset + offset * src->reg.stride.stride; + break; + case MPIDII_RANK_MAP_STRIDE_BLOCK: + case MPIDII_RANK_MAP_STRIDE_BLOCK_INTRA: + MPIDII_src_map_to_lut(src, dest, mapper, mapper->src_mapping_size, 0); + break; + case MPIDII_RANK_MAP_LUT: + case MPIDII_RANK_MAP_LUT_INTRA: + dest->mode = src->mode; + dest->irreg.lut.t = src->irreg.lut.t; + dest->irreg.lut.lpid = &src->irreg.lut.lpid[offset]; + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, + (MPL_DBG_FDEST, "\tref count %d", MPIR_Object_get_ref(src->irreg.lut.t))); + MPIR_Object_add_ref(src->irreg.lut.t); + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, (MPL_DBG_FDEST, "\tadd ref to src lut")); + break; + case MPIDII_RANK_MAP_MLUT: + dest->mode = src->mode; + dest->irreg.mlut.t = src->irreg.mlut.t; + dest->irreg.mlut.gpid = &src->irreg.mlut.gpid[offset]; + MPIR_Object_add_ref(src->irreg.mlut.t); + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, (MPL_DBG_FDEST, "\tadd ref to src mlut")); + break; + case MPIDII_RANK_MAP_NONE: + MPIR_Assert(0); + break; + } + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDII_OFFSET_OF_SRC_RMAP); +} + +#undef FUNCNAME +#define FUNCNAME MPIDII_stride_of_src_rmap +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline void MPIDII_stride_of_src_rmap(MPIDII_rank_map_t * src, + MPIDII_rank_map_t * dest, + MPIR_Comm_map_t * mapper, + int stride, int blocksize, int offset) +{ + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDII_STRIDE_OF_SRC_RMAP); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDII_STRIDE_OF_SRC_RMAP); + dest->avtid = src->avtid; + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, + (MPL_DBG_FDEST, " source mode %d", (int) src->mode)); + switch (src->mode) { + case MPIDII_RANK_MAP_DIRECT_INTRA: + if (blocksize == 1) { + dest->mode = MPIDII_RANK_MAP_STRIDE_INTRA; + } + else { + dest->mode = MPIDII_RANK_MAP_STRIDE_BLOCK_INTRA; + } + dest->size = MPIDII_map_size(*mapper); + dest->reg.stride.stride = stride; + dest->reg.stride.blocksize = blocksize; + dest->reg.stride.offset = offset; + MPIR_Assert(stride > 0); + MPIR_Assert(blocksize > 0); + break; + case MPIDII_RANK_MAP_DIRECT: + if (blocksize == 1) { + dest->mode = MPIDII_RANK_MAP_STRIDE; + } + else { + dest->mode = MPIDII_RANK_MAP_STRIDE_BLOCK; + } + dest->size = MPIDII_map_size(*mapper); + dest->reg.stride.stride = stride; + dest->reg.stride.blocksize = blocksize; + dest->reg.stride.offset = offset; + MPIR_Assert(stride > 0); + MPIR_Assert(blocksize > 0); + break; + case MPIDII_RANK_MAP_OFFSET: + if (blocksize == 1) { + dest->mode = MPIDII_RANK_MAP_STRIDE; + } + else { + dest->mode = MPIDII_RANK_MAP_STRIDE_BLOCK; + } + dest->size = MPIDII_map_size(*mapper); + dest->reg.stride.stride = stride; + dest->reg.stride.blocksize = blocksize; + dest->reg.stride.offset = offset + src->reg.offset; + break; + case MPIDII_RANK_MAP_OFFSET_INTRA: + if (blocksize == 1) { + dest->mode = MPIDII_RANK_MAP_STRIDE_INTRA; + } + else { + dest->mode = MPIDII_RANK_MAP_STRIDE_BLOCK_INTRA; + } + dest->size = MPIDII_map_size(*mapper); + dest->reg.stride.stride = stride; + dest->reg.stride.blocksize = blocksize; + dest->reg.stride.offset = offset + src->reg.offset; + break; + case MPIDII_RANK_MAP_STRIDE: + if (blocksize == 1) { + dest->mode = MPIDII_RANK_MAP_STRIDE; + dest->reg.stride.stride = src->reg.stride.stride * stride; + dest->reg.stride.blocksize = blocksize; + dest->reg.stride.offset = src->reg.stride.stride * offset + src->reg.stride.offset; + } + else { + MPIDII_src_map_to_lut(src, dest, mapper, mapper->src_mapping_size, 0); + } + break; + case MPIDII_RANK_MAP_STRIDE_INTRA: + if (blocksize == 1) { + dest->mode = MPIDII_RANK_MAP_STRIDE_INTRA; + dest->reg.stride.stride = src->reg.stride.stride * stride; + dest->reg.stride.blocksize = blocksize; + dest->reg.stride.offset = src->reg.stride.stride * offset + src->reg.stride.offset; + } + else { + MPIDII_src_map_to_lut(src, dest, mapper, mapper->src_mapping_size, 0); + } + break; + case MPIDII_RANK_MAP_STRIDE_BLOCK: + case MPIDII_RANK_MAP_STRIDE_BLOCK_INTRA: + MPIDII_src_map_to_lut(src, dest, mapper, mapper->src_mapping_size, 0); + break; + case MPIDII_RANK_MAP_LUT: + case MPIDII_RANK_MAP_LUT_INTRA: + MPIDII_src_map_to_lut(src, dest, mapper, mapper->src_mapping_size, 0); + break; + case MPIDII_RANK_MAP_MLUT: + MPIDII_src_mlut_to_mlut(src, dest, mapper, mapper->src_mapping_size, 0); + break; + case MPIDII_RANK_MAP_NONE: + MPIR_Assert(0); + break; + } + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDII_STRIDE_OF_SRC_RMAP); +} + +#undef FUNCNAME +#define FUNCNAME MPIDII_check_convert_mlut_to_lut +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDII_check_convert_mlut_to_lut(MPIDII_rank_map_t * src) +{ + int mpi_errno = MPI_SUCCESS, i; + int flag = 1; + int avtid; + MPIDII_rank_map_mlut_t *mlut = NULL; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDII_CONVERT_MLUT_TO_LUT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDII_CONVERT_MLUT_TO_LUT); + + if (src->mode != MPIDII_RANK_MAP_MLUT) { + goto fn_exit; + } + + /* check if all mlut item has the same avtid */ + avtid = src->irreg.mlut.gpid[0].avtid; + for (i = 1; i < src->size; i++) { + if (src->irreg.mlut.gpid[i].avtid != avtid) { + flag = 0; + break; + } + } + if (!flag) { /* multiple avtid */ + goto fn_exit; + } + + src->avtid = avtid; + if (avtid == 0) { + src->mode = MPIDII_RANK_MAP_LUT_INTRA; + } + else { + src->mode = MPIDII_RANK_MAP_LUT; + } + mlut = src->irreg.mlut.t; + mpi_errno = MPIDII_alloc_lut(&src->irreg.lut.t, src->size); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + src->irreg.lut.lpid = src->irreg.lut.t->lpid; + for (i = 0; i < src->size; i++) { + src->irreg.lut.lpid[i] = mlut->gpid[i].lpid; + } + MPIDIU_release_mlut(mlut); + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, (MPL_DBG_FDEST, " avtid %d", src->avtid)); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDII_CONVERT_MLUT_TO_LUT); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDII_check_convert_lut_to_regular +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDII_check_convert_lut_to_regular(MPIDII_rank_map_t * src) +{ + int mpi_errno = MPI_SUCCESS; + int mode_detected, offset, blocksize, stride; + MPIDII_rank_map_lut_t *lut = NULL; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDII_CONVERT_LUT_TO_REGULAR); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDII_CONVERT_LUT_TO_REGULAR); + + if (src->mode != MPIDII_RANK_MAP_LUT && src->mode != MPIDII_RANK_MAP_LUT_INTRA) { + goto fn_exit; + } + + lut = src->irreg.lut.t; + mode_detected = MPIDII_detect_regular_model(src->irreg.lut.lpid, src->size, + &offset, &blocksize, &stride); + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, + (MPL_DBG_FDEST, " detected mode: %d", mode_detected)); + + + switch (mode_detected) { + case MPIDII_SRC_MAPPER_DIRECT: + src->mode = MPIDII_RANK_MAP_DIRECT; + if (src->avtid == 0) { + src->mode = MPIDII_RANK_MAP_DIRECT_INTRA; + } + src->irreg.lut.t = NULL; + src->irreg.lut.lpid = NULL; + MPIDIU_release_lut(lut); + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, + (MPL_DBG_FDEST, "\tlut to mode %d", (int) src->mode)); + break; + case MPIDII_SRC_MAPPER_OFFSET: + src->mode = MPIDII_RANK_MAP_OFFSET; + if (src->avtid == 0) { + src->mode = MPIDII_RANK_MAP_OFFSET_INTRA; + } + src->reg.offset = offset; + src->irreg.lut.t = NULL; + src->irreg.lut.lpid = NULL; + MPIDIU_release_lut(lut); + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, + (MPL_DBG_FDEST, " lut to mode %d", (int) src->mode)); + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, + (MPL_DBG_FDEST, "\toffset: %d", src->reg.offset)); + break; + case MPIDII_SRC_MAPPER_STRIDE: + if (blocksize == 1) { + src->mode = MPIDII_RANK_MAP_STRIDE; + if (src->avtid == 0) { + src->mode = MPIDII_RANK_MAP_STRIDE_INTRA; + } + } + else { + src->mode = MPIDII_RANK_MAP_STRIDE_BLOCK; + if (src->avtid == 0) { + src->mode = MPIDII_RANK_MAP_STRIDE_BLOCK_INTRA; + } + } + src->reg.stride.stride = stride; + src->reg.stride.blocksize = blocksize; + src->reg.stride.offset = offset; + src->irreg.lut.t = NULL; + src->irreg.lut.lpid = NULL; + MPIDIU_release_lut(lut); + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, + (MPL_DBG_FDEST, " lut to mode %d", (int) src->mode)); + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, + (MPL_DBG_FDEST, "\toffset: %d", src->reg.stride.offset)); + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, + (MPL_DBG_FDEST, "\toffset: %d", src->reg.stride.blocksize)); + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, + (MPL_DBG_FDEST, "\toffset: %d", src->reg.stride.stride)); + break; + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDII_CONVERT_LUT_TO_REGULAR); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDII_set_map +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDII_set_map(MPIDII_rank_map_t * src_rmap, + MPIDII_rank_map_t * dest_rmap, + MPIR_Comm_map_t * mapper, + int src_comm_size, int total_mapper_size, int mapper_offset) +{ + int mpi_errno = MPI_SUCCESS; + MPIDII_rank_map_mode src_mode; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDII_SET_MAP); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDII_SET_MAP); + + src_mode = src_rmap->mode; + + /* Simplest case: MAP_DUP, exact duplication of src_comm */ + if (mapper->type == MPIR_COMM_MAP_TYPE__DUP && src_comm_size == total_mapper_size) { + MPIDII_direct_of_src_rmap(src_rmap, dest_rmap, mapper); + goto fn_exit; + } + /* single src_comm, newcomm is smaller than src_comm, only one mapper */ + else if (mapper->type == MPIR_COMM_MAP_TYPE__IRREGULAR && + mapper->src_mapping_size == total_mapper_size) { + /* check if new comm has the same mapping as src_comm */ + /* detect src_mapping_offset for direct_to_direct and offset_to_offset */ + int mode_detected, offset, blocksize, stride; + mode_detected = MPIDII_detect_regular_model(mapper->src_mapping, mapper->src_mapping_size, + &offset, &blocksize, &stride); + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, + (MPL_DBG_FDEST, "\tdetected mode: %d", mode_detected)); + + switch (mode_detected) { + case MPIDII_SRC_MAPPER_DIRECT: + MPIDII_direct_of_src_rmap(src_rmap, dest_rmap, mapper); + break; + case MPIDII_SRC_MAPPER_OFFSET: + MPIDII_offset_of_src_rmap(src_rmap, dest_rmap, mapper, offset); + break; + case MPIDII_SRC_MAPPER_STRIDE: + MPIDII_stride_of_src_rmap(src_rmap, dest_rmap, mapper, stride, blocksize, offset); + break; + default: + if (src_rmap->mode == MPIDII_RANK_MAP_MLUT) { + MPIDII_src_mlut_to_mlut(src_rmap, dest_rmap, mapper, total_mapper_size, + mapper_offset); + } + else { /* src_mode != MPIDII_RANK_MAP_MLUT */ + MPIDII_src_map_to_lut(src_rmap, dest_rmap, mapper, mapper->src_mapping_size, + mapper_offset); + } + } + goto fn_exit; + } + + /* more complex case: multiple mappers + * We always alloc lut (or mlut is src_rmap is mlut). We will check if a + * lut mapping can be converted to something simpler after all the mapper + * are processed + */ + + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, (MPL_DBG_FDEST, " multiple mapper")); + if (mapper->type == MPIR_COMM_MAP_TYPE__DUP) { + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, + (MPL_DBG_FDEST, " check map_size %d, src_comm_size %d", + MPIDII_map_size(*mapper), src_comm_size)); + if (src_mode == MPIDII_RANK_MAP_MLUT) { + MPIDII_src_comm_to_mlut(src_rmap, dest_rmap, src_comm_size, + total_mapper_size, mapper_offset); + } + else { /* src_mode != MPIDII_RANK_MAP_MLUT */ + MPIDII_src_comm_to_lut(src_rmap, dest_rmap, src_comm_size, + total_mapper_size, mapper_offset); + } + } + else { /* mapper->type == MPIR_COMM_MAP_TYPE__IRREGULAR */ + if (src_mode == MPIDII_RANK_MAP_MLUT) { + MPIDII_src_mlut_to_mlut(src_rmap, dest_rmap, mapper, total_mapper_size, mapper_offset); + } + else { /* src_mode != MPIDII_RANK_MAP_MLUT */ + MPIDII_src_map_to_lut(src_rmap, dest_rmap, mapper, total_mapper_size, mapper_offset); + } + } + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDII_SET_MAP); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDII_comm_create_rank_map +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDII_comm_create_rank_map(MPIR_Comm * comm) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Comm_map_t *mapper; + MPIR_Comm *src_comm; + int total_mapper_size, mapper_offset; + + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDII_COMM_CREATE_RANK_MAP); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDII_COMM_CREATE_RANK_MAP); + + /* do some sanity checks */ + MPL_LL_FOREACH(comm->mapper_head, mapper) { + if (mapper->src_comm->comm_kind == MPIR_COMM_KIND__INTRACOMM) + MPIR_Assert(mapper->dir == MPIR_COMM_MAP_DIR__L2L || + mapper->dir == MPIR_COMM_MAP_DIR__L2R); + + if (comm->comm_kind == MPIR_COMM_KIND__INTRACOMM) + MPIR_Assert(mapper->dir == MPIR_COMM_MAP_DIR__L2L || + mapper->dir == MPIR_COMM_MAP_DIR__R2L); + } + + /* First, handle all the mappers that contribute to the local part + * of the comm */ + total_mapper_size = 0; + MPL_LL_FOREACH(comm->mapper_head, mapper) { + if (mapper->dir == MPIR_COMM_MAP_DIR__L2R || mapper->dir == MPIR_COMM_MAP_DIR__R2R) + continue; + + total_mapper_size += MPIDII_map_size(*mapper); + } + mapper_offset = 0; + MPL_LL_FOREACH(comm->mapper_head, mapper) { + src_comm = mapper->src_comm; + + if (mapper->dir == MPIR_COMM_MAP_DIR__L2R || mapper->dir == MPIR_COMM_MAP_DIR__R2R) + continue; + + if (mapper->dir == MPIR_COMM_MAP_DIR__L2L) { + if (src_comm->comm_kind == MPIR_COMM_KIND__INTRACOMM && + comm->comm_kind == MPIR_COMM_KIND__INTRACOMM) { + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, + (MPL_DBG_FDEST, + " intra->intra, L2L, size=%d, total_mapper_size=%d, mapper_offset=%d", + src_comm->local_size, total_mapper_size, mapper_offset)); + MPIDII_set_map(&MPIDII_COMM(src_comm, map), &MPIDII_COMM(comm, map), mapper, + src_comm->local_size, total_mapper_size, mapper_offset); + } + else if (src_comm->comm_kind == MPIR_COMM_KIND__INTRACOMM && + comm->comm_kind == MPIR_COMM_KIND__INTERCOMM) { + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, + (MPL_DBG_FDEST, + " intra->inter, L2L, size=%d, total_mapper_size=%d, mapper_offset=%d", + src_comm->local_size, total_mapper_size, mapper_offset)); + MPIDII_set_map(&MPIDII_COMM(src_comm, map), &MPIDII_COMM(comm, local_map), mapper, + src_comm->local_size, total_mapper_size, mapper_offset); + } + else if (src_comm->comm_kind == MPIR_COMM_KIND__INTERCOMM && + comm->comm_kind == MPIR_COMM_KIND__INTRACOMM) { + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, + (MPL_DBG_FDEST, + " inter->intra, L2L, size=%d, total_mapper_size=%d, mapper_offset=%d", + src_comm->local_size, total_mapper_size, mapper_offset)); + MPIDII_set_map(&MPIDII_COMM(src_comm, local_map), &MPIDII_COMM(comm, map), mapper, + src_comm->local_size, total_mapper_size, mapper_offset); + } + else { /* src_comm->comm_kind == MPIR_COMM_KIND__INTERCOMM && comm->comm_kind == MPIR_COMM_KIND__INTERCOMM */ + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, + (MPL_DBG_FDEST, + " inter->inter, L2L, size=%d, total_mapper_size=%d, mapper_offset=%d", + src_comm->local_size, total_mapper_size, mapper_offset)); + MPIDII_set_map(&MPIDII_COMM(src_comm, local_map), &MPIDII_COMM(comm, local_map), + mapper, src_comm->local_size, total_mapper_size, mapper_offset); + } + } + else { /* mapper->dir == MPIR_COMM_MAP_DIR__R2L */ + MPIR_Assert(src_comm->comm_kind == MPIR_COMM_KIND__INTERCOMM); + + if (comm->comm_kind == MPIR_COMM_KIND__INTRACOMM) { + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, + (MPL_DBG_FDEST, + " ->intra, R2L, size=%d, total_mapper_size=%d, mapper_offset=%d", + src_comm->remote_size, total_mapper_size, mapper_offset)); + MPIDII_set_map(&MPIDII_COMM(src_comm, map), &MPIDII_COMM(comm, map), mapper, + src_comm->remote_size, total_mapper_size, mapper_offset); + } + else { + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, + (MPL_DBG_FDEST, + " ->inter, R2L, size=%d, total_mapper_size=%d, mapper_offset=%d", + src_comm->remote_size, total_mapper_size, mapper_offset)); + MPIDII_set_map(&MPIDII_COMM(src_comm, map), &MPIDII_COMM(comm, local_map), mapper, + src_comm->remote_size, total_mapper_size, mapper_offset); + } + } + + mapper_offset += MPIDII_map_size(*mapper); + } + + /* Next, handle all the mappers that contribute to the remote part + * of the comm (only valid for intercomms) + */ + total_mapper_size = 0; + MPL_LL_FOREACH(comm->mapper_head, mapper) { + if (mapper->dir == MPIR_COMM_MAP_DIR__L2L || mapper->dir == MPIR_COMM_MAP_DIR__R2L) + continue; + + total_mapper_size += MPIDII_map_size(*mapper); + } + mapper_offset = 0; + MPL_LL_FOREACH(comm->mapper_head, mapper) { + src_comm = mapper->src_comm; + + if (mapper->dir == MPIR_COMM_MAP_DIR__L2L || mapper->dir == MPIR_COMM_MAP_DIR__R2L) + continue; + + MPIR_Assert(comm->comm_kind == MPIR_COMM_KIND__INTERCOMM); + + if (mapper->dir == MPIR_COMM_MAP_DIR__L2R) { + if (src_comm->comm_kind == MPIR_COMM_KIND__INTRACOMM) { + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, + (MPL_DBG_FDEST, + " intra->, L2R, size=%d, total_mapper_size=%d, mapper_offset=%d", + src_comm->local_size, total_mapper_size, mapper_offset)); + MPIDII_set_map(&MPIDII_COMM(src_comm, map), &MPIDII_COMM(comm, map), mapper, + src_comm->local_size, total_mapper_size, mapper_offset); + } + else { /* src_comm->comm_kind == MPIR_COMM_KIND__INTERCOMM */ + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, + (MPL_DBG_FDEST, + " inter->, L2R, size=%d, total_mapper_size=%d, mapper_offset=%d", + src_comm->local_size, total_mapper_size, mapper_offset)); + MPIDII_set_map(&MPIDII_COMM(src_comm, local_map), &MPIDII_COMM(comm, map), mapper, + src_comm->local_size, total_mapper_size, mapper_offset); + } + } + else { /* mapper->dir == MPIR_COMM_MAP_DIR__R2R */ + MPIR_Assert(src_comm->comm_kind == MPIR_COMM_KIND__INTERCOMM); + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, + (MPL_DBG_FDEST, + " inter->, R2R, size=%d, total_mapper_size=%d, mapper_offset=%d", + src_comm->remote_size, total_mapper_size, mapper_offset)); + MPIDII_set_map(&MPIDII_COMM(src_comm, map), &MPIDII_COMM(comm, map), mapper, + src_comm->remote_size, total_mapper_size, mapper_offset); + } + + mapper_offset += MPIDII_map_size(*mapper); + } + + /* check before finishing + * 1. if mlut can be converted to lut: all avtids are the same + * 2. if lut can be converted to regular modes: direct, offset, and more + */ + MPIDII_check_convert_mlut_to_lut(&MPIDII_COMM(comm, map)); + MPIDII_check_convert_lut_to_regular(&MPIDII_COMM(comm, map)); + if (comm->comm_kind == MPIR_COMM_KIND__INTERCOMM) { + MPIDII_check_convert_mlut_to_lut(&MPIDII_COMM(comm, local_map)); + MPIDII_check_convert_lut_to_regular(&MPIDII_COMM(comm, local_map)); + } + + if (comm->comm_kind == MPIR_COMM_KIND__INTERCOMM) { + /* setup the lut for the local_comm in the intercomm */ + if (comm->local_comm) { + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, + (MPL_DBG_FDEST, "\t create local_comm using src_comm")); + MPIDII_direct_of_src_rmap(&MPIDII_COMM(comm, local_map), + &MPIDII_COMM(comm->local_comm, map), mapper); + + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MEMORY, VERBOSE, + (MPL_DBG_FDEST, "create local_comm using src_comm")); + } + } + + if (comm->comm_kind == MPIR_COMM_KIND__INTRACOMM) { + MPIDII_COMM(comm, local_map).mode = MPIDII_RANK_MAP_NONE; + } + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDII_COMM_CREATE_RANK_MAP); + return mpi_errno; +} + +#endif /* ifndef CH4I_COMM_H_INCLUDED */ diff --git a/src/mpid/ch4/src/ch4i_util.h b/src/mpid/ch4/src/ch4i_util.h new file mode 100644 index 0000000..e6a636d --- /dev/null +++ b/src/mpid/ch4/src/ch4i_util.h @@ -0,0 +1,20 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef CH4I_UTIL_H_INCLUDED +#define CH4I_UTIL_H_INCLUDED + +void MPIDI_CH4I_map_create(void **_map); +void MPIDI_CH4I_map_destroy(void *_map); +void MPIDI_CH4I_map_set(void *_map, uint64_t id, void *val); +void MPIDI_CH4I_map_erase(void *_map, uint64_t id); +void *MPIDI_CH4I_map_lookup(void *_map, uint64_t id); + +#endif /* CH4I_UTIL_H_INCLUDED */ diff --git a/src/mpid/ch4/src/ch4r_buf.h b/src/mpid/ch4/src/ch4r_buf.h new file mode 100644 index 0000000..4b8ea31 --- /dev/null +++ b/src/mpid/ch4/src/ch4r_buf.h @@ -0,0 +1,168 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef CH4R_BUF_H_INCLUDED +#define CH4R_BUF_H_INCLUDED + +#include "ch4_impl.h" +#include "ch4i_util.h" +#include <pthread.h> + +/* + initial prototype of buffer pool. + + TODO: + - align buffer region + - add garbage collection + - use huge pages +*/ + +static inline MPIU_buf_pool_t *create_buf_pool(int num, int size, MPIU_buf_pool_t * parent_pool) +{ + int i; + MPIU_buf_pool_t *buf_pool; + MPIU_buf_t *curr, *next; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_CREATE_BUF_POOL); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_CREATE_BUF_POOL); + + buf_pool = (MPIU_buf_pool_t *) MPL_malloc(sizeof(*buf_pool)); + MPIR_Assert(buf_pool); + pthread_mutex_init(&buf_pool->lock, NULL); + + buf_pool->size = size; + buf_pool->num = num; + buf_pool->next = NULL; + buf_pool->memory_region = MPL_malloc(num * (sizeof(MPIU_buf_t) + size)); + MPIR_Assert(buf_pool->memory_region); + + curr = (MPIU_buf_t *) buf_pool->memory_region; + buf_pool->head = curr; + for (i = 0; i < num - 1; i++) { + next = (MPIU_buf_t *) ((char *) curr + size + sizeof(MPIU_buf_t)); + curr->next = next; + curr->pool = parent_pool ? parent_pool : buf_pool; + curr = curr->next; + } + curr->next = NULL; + curr->pool = parent_pool ? parent_pool : buf_pool; + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_CREATE_BUF_POOL); + return buf_pool; +} + +static inline MPIU_buf_pool_t *MPIDI_CH4U_create_buf_pool(int num, int size) +{ + MPIU_buf_pool_t *buf_pool; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_CREATE_BUF_POOL); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_CREATE_BUF_POOL); + + buf_pool = create_buf_pool(num, size, NULL); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_CREATE_BUF_POOL); + return buf_pool; +} + +static inline void *MPIDI_CH4U_get_head_buf(MPIU_buf_pool_t * pool) +{ + void *buf; + MPIU_buf_t *curr; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_GET_HEAD_BUF); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_GET_HEAD_BUF); + + curr = pool->head; + pool->head = curr->next; + buf = curr->data; + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_GET_HEAD_BUF); + return buf; +} + +static inline void *MPIDI_CH4R_get_buf_safe(MPIU_buf_pool_t * pool) +{ + void *buf; + MPIU_buf_pool_t *curr_pool; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_GET_BUF_SAFE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_GET_BUF_SAFE); + + if (pool->head) { + buf = MPIDI_CH4U_get_head_buf(pool); + goto fn_exit; + } + + curr_pool = pool; + while (curr_pool->next) + curr_pool = curr_pool->next; + + curr_pool->next = create_buf_pool(pool->num, pool->size, pool); + MPIR_Assert(curr_pool->next); + pool->head = curr_pool->next->head; + buf = MPIDI_CH4U_get_head_buf(pool); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_GET_BUF_SAFE); + return buf; +} + + +static inline void *MPIDI_CH4R_get_buf(MPIU_buf_pool_t * pool) +{ + void *buf; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_GET_BUF); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_GET_BUF); + + pthread_mutex_lock(&pool->lock); + buf = MPIDI_CH4R_get_buf_safe(pool); + pthread_mutex_unlock(&pool->lock); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_GET_BUF); + return buf; +} + +static inline void MPIDI_CH4R_release_buf_safe(void *buf) +{ + MPIU_buf_t *curr_buf; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_RELEASE_BUF_SAFE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_RELEASE_BUF_SAFE); + + curr_buf = container_of(buf, MPIU_buf_t, data); + curr_buf->next = curr_buf->pool->head; + curr_buf->pool->head = curr_buf; + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_RELEASE_BUF_SAFE); +} + +static inline void MPIDI_CH4R_release_buf(void *buf) +{ + MPIU_buf_t *curr_buf; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_RELEASE_BUF); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_RELEASE_BUF); + + curr_buf = container_of(buf, MPIU_buf_t, data); + pthread_mutex_lock(&curr_buf->pool->lock); + curr_buf->next = curr_buf->pool->head; + curr_buf->pool->head = curr_buf; + pthread_mutex_unlock(&curr_buf->pool->lock); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_RELEASE_BUF); +} + + +static inline void MPIDI_CH4R_destroy_buf_pool(MPIU_buf_pool_t * pool) +{ + if (pool->next) + MPIDI_CH4R_destroy_buf_pool(pool->next); + + MPL_free(pool->memory_region); + MPL_free(pool); +} + +#endif /* CH4R_BUF_H_INCLUDED */ diff --git a/src/mpid/ch4/src/ch4r_callbacks.h b/src/mpid/ch4/src/ch4r_callbacks.h new file mode 100644 index 0000000..db0ec20 --- /dev/null +++ b/src/mpid/ch4/src/ch4r_callbacks.h @@ -0,0 +1,2836 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef CH4R_CALLBACKS_H_INCLUDED +#define CH4R_CALLBACKS_H_INCLUDED + +#include "ch4r_request.h" + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_progress_cmpl_list +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline void MPIDI_CH4U_progress_cmpl_list(void) +{ + MPIR_Request *req; + MPIDI_CH4U_req_ext_t *curr, *tmp; + MPIDI_NM_am_completion_handler_fn cmpl_handler_fn; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_PROGRESS_CMPL_LIST); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_PROGRESS_CMPL_LIST); + + /* MPIDI_CS_ENTER(); */ + do_check_again: + MPL_DL_FOREACH_SAFE(MPIDI_CH4_Global.cmpl_list, curr, tmp) { + if (curr->seq_no == (uint64_t) OPA_load_int(&MPIDI_CH4_Global.exp_seq_no)) { + MPL_DL_DELETE(MPIDI_CH4_Global.cmpl_list, curr); + req = (MPIR_Request *) curr->request; + cmpl_handler_fn = (MPIDI_NM_am_completion_handler_fn) curr->cmpl_handler_fn; + cmpl_handler_fn(req); + goto do_check_again; + } + } + /* MPIDI_CS_EXIT(); */ + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_PROGRESS_CMPL_LIST); +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_check_cmpl_order +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_check_cmpl_order(MPIR_Request * req, + MPIDI_NM_am_completion_handler_fn cmpl_handler_fn) +{ + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_CHECK_CMPL_ORDER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_CHECK_CMPL_ORDER); + + if (MPIDI_CH4U_REQUEST(req, req->seq_no) == + (uint64_t) OPA_load_int(&MPIDI_CH4_Global.exp_seq_no)) { + OPA_incr_int(&MPIDI_CH4_Global.exp_seq_no); + return 1; + } + + MPIDI_CH4U_REQUEST(req, req->cmpl_handler_fn) = (void *) cmpl_handler_fn; + MPIDI_CH4U_REQUEST(req, req->request) = (uint64_t) req; + /* MPIDI_CS_ENTER(); */ + MPL_DL_APPEND(MPIDI_CH4_Global.cmpl_list, req->dev.ch4.ch4u.req); + /* MPIDI_CS_EXIT(); */ + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_CHECK_CMPL_ORDER); + return 0; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_send_origin_cmpl_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_send_origin_cmpl_handler(MPIR_Request * sreq) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_SEND_TX_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_SEND_TX_HANDLER); + MPIDI_CH4I_am_request_complete(sreq); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_SEND_TX_HANDLER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_send_long_lmt_origin_cmpl_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_send_long_lmt_origin_cmpl_handler(MPIR_Request * sreq) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_SEND_LONG_LMT_TX_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_SEND_LONG_LMT_TX_HANDLER); + dtype_release_if_not_builtin(MPIDI_CH4U_REQUEST(sreq, req->lreq).datatype); + MPIDI_CH4I_am_request_complete(sreq); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_SEND_LONG_LMT_TX_HANDLER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_ssend_ack_origin_cmpl_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_ssend_ack_origin_cmpl_handler(MPIR_Request * req) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_SSEND_ACK_TX_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_SSEND_ACK_TX_HANDLER); + MPIDI_CH4I_am_request_complete(req); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_SSEND_ACK_TX_HANDLER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_put_ack_origin_cmpl_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_put_ack_origin_cmpl_handler(MPIR_Request * req) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_PUT_ACK_TX_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_PUT_ACK_TX_HANDLER); + MPIDI_CH4I_am_request_complete(req); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_PUT_ACK_TX_HANDLER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_get_cmpl_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_get_cmpl_handler(MPIR_Request * req) +{ + int mpi_errno = MPI_SUCCESS, i, c; + size_t data_sz, offset; + MPIDI_CH4U_get_ack_msg_t get_ack; + struct iovec *iov; + char *p_data; + uintptr_t base; + MPIR_Win *win; + MPIR_Context_id_t context_id; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_GET_CMPL_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_GET_CMPL_HANDLER); + + if (!MPIDI_CH4U_check_cmpl_order(req, MPIDI_CH4U_get_cmpl_handler)) + return mpi_errno; + + base = MPIDI_CH4U_REQUEST(req, req->greq.addr); + + MPIR_cc_incr(req->cc_ptr, &c); + get_ack.greq_ptr = MPIDI_CH4U_REQUEST(req, req->greq.greq_ptr); + win = MPIDI_CH4U_REQUEST(req, req->greq.win_ptr); + context_id = MPIDI_CH4U_win_to_context(win); + if (MPIDI_CH4U_REQUEST(req, req->greq.n_iov) == 0) { + mpi_errno = MPIDI_NM_send_am_reply(context_id, + MPIDI_CH4U_REQUEST(req, src_rank), + MPIDI_CH4U_GET_ACK, + &get_ack, sizeof(get_ack), + (void *) MPIDI_CH4U_REQUEST(req, req->greq.addr), + MPIDI_CH4U_REQUEST(req, req->greq.count), + MPIDI_CH4U_REQUEST(req, req->greq.datatype), req); + MPIDI_CH4I_am_request_complete(req); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + goto fn_exit; + } + + iov = (struct iovec *) MPIDI_CH4U_REQUEST(req, req->greq.dt_iov); + + data_sz = 0; + for (i = 0; i < MPIDI_CH4U_REQUEST(req, req->greq.n_iov); i++) { + data_sz += iov[i].iov_len; + } + + p_data = (char *) MPL_malloc(data_sz); + MPIR_Assert(p_data); + + offset = 0; + for (i = 0; i < MPIDI_CH4U_REQUEST(req, req->greq.n_iov); i++) { + /* Adjust a window base address */ + iov[i].iov_base = (char *) iov[i].iov_base + base; + MPIR_Memcpy(p_data + offset, iov[i].iov_base, iov[i].iov_len); + offset += iov[i].iov_len; + } + + MPL_free(MPIDI_CH4U_REQUEST(req, req->greq.dt_iov)); + MPIDI_CH4U_REQUEST(req, req->greq.dt_iov) = (void *) p_data; + + mpi_errno = MPIDI_NM_send_am_reply(context_id, + MPIDI_CH4U_REQUEST(req, src_rank), + MPIDI_CH4U_GET_ACK, + &get_ack, sizeof(get_ack), p_data, data_sz, MPI_BYTE, req); + MPIDI_CH4I_am_request_complete(req); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + MPIDI_CH4U_progress_cmpl_list(); + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_GET_CMPL_HANDLER); + return mpi_errno; + fn_fail: + goto fn_exit; +} + + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_acc_ack_origin_cmpl_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_acc_ack_origin_cmpl_handler(MPIR_Request * req) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_ACC_ACK_TX_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_ACC_ACK_TX_HANDLER); + MPIDI_CH4I_am_request_complete(req); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_ACC_ACK_TX_HANDLER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_get_acc_ack_origin_cmpl_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_get_acc_ack_origin_cmpl_handler(MPIR_Request * req) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Win *win; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_GET_ACC_ACK_TX_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_GET_ACC_ACK_TX_HANDLER); + MPL_free(MPIDI_CH4U_REQUEST(req, req->areq.data)); + + win = MPIDI_CH4U_REQUEST(req, req->areq.win_ptr); + /* MPIDI_CS_ENTER(); */ + OPA_decr_int(&MPIDI_CH4U_WIN(win, outstanding_ops)); + /* MPIDI_CS_EXIT(); */ + + MPIDI_CH4I_am_request_complete(req); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_GET_ACC_ACK_TX_HANDLER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_cswap_ack_origin_cmpl_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_cswap_ack_origin_cmpl_handler(MPIR_Request * req) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Win *win; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_CSWAP_ACK_TX_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_CSWAP_ACK_TX_HANDLER); + + MPL_free(MPIDI_CH4U_REQUEST(req, req->creq.data)); + win = MPIDI_CH4U_REQUEST(req, req->creq.win_ptr); + /* MPIDI_CS_ENTER(); */ + OPA_decr_int(&MPIDI_CH4U_WIN(win, outstanding_ops)); + /* MPIDI_CS_EXIT(); */ + + MPIDI_CH4I_am_request_complete(req); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_CSWAP_ACK_TX_HANDLER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_get_ack_origin_cmpl_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_get_ack_origin_cmpl_handler(MPIR_Request * req) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Win *win; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_GET_ACK_TX_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_GET_ACK_TX_HANDLER); + + if (MPIDI_CH4U_REQUEST(req, req->greq.dt_iov)) { + MPL_free(MPIDI_CH4U_REQUEST(req, req->greq.dt_iov)); + } + + win = MPIDI_CH4U_REQUEST(req, req->greq.win_ptr); + /* MPIDI_CS_ENTER(); */ + OPA_decr_int(&MPIDI_CH4U_WIN(win, outstanding_ops)); + /* MPIDI_CS_EXIT(); */ + + MPIDI_CH4I_am_request_complete(req); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_GET_ACK_TX_HANDLER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_put_origin_cmpl_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_put_origin_cmpl_handler(MPIR_Request * sreq) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_PUT_TX_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_PUT_TX_HANDLER); + MPIDI_CH4I_am_request_complete(sreq); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_PUT_TX_HANDLER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_cswap_origin_cmpl_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_cswap_origin_cmpl_handler(MPIR_Request * sreq) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_CSWAP_TX_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_CSWAP_TX_HANDLER); + MPIDI_CH4I_am_request_complete(sreq); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_CSWAP_TX_HANDLER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_acc_origin_cmpl_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_acc_origin_cmpl_handler(MPIR_Request * sreq) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_ACC_TX_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_ACC_TX_HANDLER); + MPIDI_CH4I_am_request_complete(sreq); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_ACC_TX_HANDLER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_put_data_origin_cmpl_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_put_data_origin_cmpl_handler(MPIR_Request * sreq) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_PUT_DATA_TX_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_PUT_DATA_TX_HANDLER); + MPIDI_CH4I_am_request_complete(sreq); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_PUT_DATA_TX_HANDLER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_acc_data_origin_cmpl_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_acc_data_origin_cmpl_handler(MPIR_Request * sreq) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_ACC_DATA_TX_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_ACC_DATA_TX_HANDLER); + MPIDI_CH4I_am_request_complete(sreq); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_ACC_DATA_TX_HANDLER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_put_iov_origin_cmpl_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_put_iov_origin_cmpl_handler(MPIR_Request * sreq) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_PUT_IOV_TX_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_PUT_IOV_TX_HANDLER); + MPIDI_CH4I_am_request_complete(sreq); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_PUT_IOV_TX_HANDLER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_acc_iov_origin_cmpl_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_acc_iov_origin_cmpl_handler(MPIR_Request * sreq) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_ACC_IOV_TX_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_ACC_IOV_TX_HANDLER); + MPIDI_CH4I_am_request_complete(sreq); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_ACC_IOV_TX_HANDLER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_get_origin_cmpl_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_get_origin_cmpl_handler(MPIR_Request * sreq) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_GET_TX_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_GET_TX_HANDLER); + MPIDI_CH4I_am_request_complete(sreq); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_GET_TX_HANDLER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_reply_ssend +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_reply_ssend(MPIR_Request * rreq) +{ + int mpi_errno = MPI_SUCCESS, c; + MPIDI_CH4U_ssend_ack_msg_t ack_msg; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_REPLY_SSEND); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_REPLY_SSEND); + MPIR_cc_incr(rreq->cc_ptr, &c); + ack_msg.sreq_ptr = MPIDI_CH4U_REQUEST(rreq, req->rreq.peer_req_ptr); + + mpi_errno = MPIDI_NM_send_am_hdr_reply(MPIDI_CH4U_get_context(MPIDI_CH4U_REQUEST(rreq, tag)), + MPIDI_CH4U_REQUEST(rreq, src_rank), + MPIDI_CH4U_SSEND_ACK, &ack_msg, sizeof(ack_msg), rreq); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_REPLY_SSEND); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_ack_put +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_ack_put(MPIR_Request * rreq) +{ + int mpi_errno = MPI_SUCCESS; + MPIDI_CH4U_put_ack_msg_t ack_msg; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_ACK_PUT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_ACK_PUT); + + ack_msg.preq_ptr = MPIDI_CH4U_REQUEST(rreq, req->preq.preq_ptr); + mpi_errno = + MPIDI_NM_inject_am_hdr_reply(MPIDI_CH4U_win_to_context + (MPIDI_CH4U_REQUEST(rreq, req->preq.win_ptr)), + MPIDI_CH4U_REQUEST(rreq, src_rank), MPIDI_CH4U_PUT_ACK, + &ack_msg, sizeof(ack_msg)); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_ACK_PUT); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_ack_cswap +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_ack_cswap(MPIR_Request * rreq) +{ + int mpi_errno = MPI_SUCCESS, c; + MPIDI_CH4U_cswap_ack_msg_t ack_msg; + void *result_addr; + size_t data_sz; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_ACK_CSWAP); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_ACK_CSWAP); + + MPIDI_Datatype_check_size(MPIDI_CH4U_REQUEST(rreq, req->creq.datatype), 1, data_sz); + result_addr = ((char *) MPIDI_CH4U_REQUEST(rreq, req->creq.data)) + data_sz; + + MPIR_cc_incr(rreq->cc_ptr, &c); + ack_msg.req_ptr = MPIDI_CH4U_REQUEST(rreq, req->creq.creq_ptr); + + mpi_errno = + MPIDI_NM_send_am_reply(MPIDI_CH4U_win_to_context + (MPIDI_CH4U_REQUEST(rreq, req->creq.win_ptr)), + MPIDI_CH4U_REQUEST(rreq, src_rank), MPIDI_CH4U_CSWAP_ACK, &ack_msg, + sizeof(ack_msg), result_addr, 1, MPIDI_CH4U_REQUEST(rreq, + req->creq. + datatype), rreq); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_ACK_CSWAP); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_ack_acc +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_ack_acc(MPIR_Request * rreq) +{ + int mpi_errno = MPI_SUCCESS; + MPIDI_CH4U_acc_ack_msg_t ack_msg; + MPIR_Win *win; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_ACK_ACC); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_ACK_ACC); + + ack_msg.req_ptr = MPIDI_CH4U_REQUEST(rreq, req->areq.req_ptr); + mpi_errno = + MPIDI_NM_inject_am_hdr_reply(MPIDI_CH4U_win_to_context + (MPIDI_CH4U_REQUEST(rreq, req->areq.win_ptr)), + MPIDI_CH4U_REQUEST(rreq, src_rank), MPIDI_CH4U_ACC_ACK, + &ack_msg, sizeof(ack_msg)); + + win = MPIDI_CH4U_REQUEST(rreq, req->areq.win_ptr); + /* MPIDI_CS_ENTER(); */ + OPA_decr_int(&MPIDI_CH4U_WIN(win, outstanding_ops)); + /* MPIDI_CS_EXIT(); */ + + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_ACK_ACC); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_ack_get_acc +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_ack_get_acc(MPIR_Request * rreq) +{ + int mpi_errno = MPI_SUCCESS, c; + MPIDI_CH4U_acc_ack_msg_t ack_msg; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_ACK_GET_ACC); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_ACK_GET_ACC); + + MPIR_cc_incr(rreq->cc_ptr, &c); + ack_msg.req_ptr = MPIDI_CH4U_REQUEST(rreq, req->areq.req_ptr); + + mpi_errno = + MPIDI_NM_send_am_reply(MPIDI_CH4U_win_to_context + (MPIDI_CH4U_REQUEST(rreq, req->areq.win_ptr)), + MPIDI_CH4U_REQUEST(rreq, src_rank), MPIDI_CH4U_GET_ACC_ACK, &ack_msg, + sizeof(ack_msg), MPIDI_CH4U_REQUEST(rreq, req->areq.data), + MPIDI_CH4U_REQUEST(rreq, req->areq.data_sz), MPI_BYTE, rreq); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_ACK_GET_ACC); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_unexp_mrecv_cmpl_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_unexp_mrecv_cmpl_handler(MPIR_Request * rreq) +{ + int mpi_errno = MPI_SUCCESS; + uint64_t msg_tag; + size_t message_sz; + MPI_Aint last; + int dt_contig; + MPI_Aint dt_true_lb; + MPIR_Datatype *dt_ptr; + size_t data_sz, dt_sz; + MPID_Segment *segment_ptr; + void *buf; + int count; + MPI_Datatype datatype; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_UNEXP_MRECV_CMPL_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_UNEXP_MRECV_CMPL_HANDLER); + + msg_tag = MPIDI_CH4U_REQUEST(rreq, tag); + rreq->status.MPI_SOURCE = MPIDI_CH4U_REQUEST(rreq, src_rank); + rreq->status.MPI_TAG = MPIDI_CH4U_get_tag(msg_tag); + + buf = MPIDI_CH4U_REQUEST(rreq, req->rreq.mrcv_buffer); + count = MPIDI_CH4U_REQUEST(rreq, req->rreq.mrcv_count); + datatype = MPIDI_CH4U_REQUEST(rreq, req->rreq.mrcv_datatype); + + message_sz = MPIDI_CH4U_REQUEST(rreq, count); + MPID_Datatype_get_size_macro(datatype, dt_sz); + + if (message_sz > count * dt_sz) { + rreq->status.MPI_ERROR = MPI_ERR_TRUNCATE; + } + else { + rreq->status.MPI_ERROR = MPI_SUCCESS; + count = message_sz / dt_sz; + } + + MPIR_STATUS_SET_COUNT(rreq->status, count * dt_sz); + MPIDI_Datatype_get_info(count, datatype, dt_contig, data_sz, dt_ptr, dt_true_lb); + + if (!dt_contig) { + segment_ptr = MPIDU_Segment_alloc(); + MPIR_ERR_CHKANDJUMP1(segment_ptr == NULL, mpi_errno, + MPI_ERR_OTHER, "**nomem", "**nomem %s", "Recv MPIDU_Segment_alloc"); + MPIDU_Segment_init(buf, count, datatype, segment_ptr, 0); + + last = count * dt_sz; + MPIDU_Segment_unpack(segment_ptr, 0, &last, MPIDI_CH4U_REQUEST(rreq, buffer)); + MPIDU_Segment_free(segment_ptr); + if (last != (MPI_Aint) (count * dt_sz)) { + mpi_errno = MPIR_Err_create_code(MPI_SUCCESS, MPIR_ERR_RECOVERABLE, + __FUNCTION__, __LINE__, + MPI_ERR_TYPE, "**dtypemismatch", 0); + rreq->status.MPI_ERROR = mpi_errno; + } + } + else { + MPIR_Memcpy((char *) buf + dt_true_lb, MPIDI_CH4U_REQUEST(rreq, buffer), data_sz); + } + + MPL_free(MPIDI_CH4U_REQUEST(rreq, buffer)); + rreq->kind = MPIR_REQUEST_KIND__RECV; + + if (MPIDI_CH4U_REQUEST(rreq, req->status) & MPIDI_CH4U_REQ_PEER_SSEND) { + mpi_errno = MPIDI_CH4U_reply_ssend(rreq); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + } + MPIDI_CH4I_am_request_complete(rreq); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_UNEXP_MRECV_CMPL_HANDLER); + return mpi_errno; + fn_fail: + goto fn_exit; +} + + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_unexp_cmpl_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_unexp_cmpl_handler(MPIR_Request * rreq) +{ + int mpi_errno = MPI_SUCCESS, c; + MPIR_Comm *root_comm; + MPIR_Request *match_req = NULL; + uint64_t msg_tag; + size_t count; + MPI_Aint last; + int dt_contig; + MPI_Aint dt_true_lb; + MPIR_Datatype *dt_ptr; + size_t dt_sz; + MPID_Segment *segment_ptr; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_UNEXP_CMPL_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_UNEXP_CMPL_HANDLER); + + /* MPIDI_CS_ENTER(); */ + if (MPIDI_CH4U_REQUEST(rreq, req->status) & MPIDI_CH4U_REQ_UNEXP_DQUED) { + if (MPIDI_CH4U_REQUEST(rreq, req->status) & MPIDI_CH4U_REQ_UNEXP_CLAIMED) { + MPIDI_CH4U_unexp_mrecv_cmpl_handler(rreq); + } + /* MPIDI_CS_EXIT(); */ + goto fn_exit; + } + /* MPIDI_CS_EXIT(); */ + + msg_tag = MPIDI_CH4U_REQUEST(rreq, tag); + root_comm = MPIDI_CH4U_context_id_to_comm(MPIDI_CH4U_get_context(msg_tag)); + + if (MPIDI_CH4U_REQUEST(rreq, req->status) & MPIDI_CH4U_REQ_MATCHED) { + match_req = (MPIR_Request *) MPIDI_CH4U_REQUEST(rreq, req->rreq.match_req); + } + else { + /* MPIDI_CS_ENTER(); */ + if (root_comm) + match_req = + MPIDI_CH4U_dequeue_posted(msg_tag, &MPIDI_CH4U_COMM(root_comm, posted_list)); + + if (match_req) { + MPIDI_CH4U_delete_unexp(rreq, &MPIDI_CH4U_COMM(root_comm, unexp_list)); + /* Decrement the counter twice, one for posted_list and the other for unexp_list */ + MPIR_Comm_release(root_comm); + MPIR_Comm_release(root_comm); + } + /* MPIDI_CS_EXIT(); */ + } + + if (!match_req) { + MPIDI_CH4U_REQUEST(rreq, req->status) &= ~MPIDI_CH4U_REQ_BUSY; + goto fn_exit; + } + + match_req->status.MPI_SOURCE = MPIDI_CH4U_REQUEST(rreq, src_rank); + match_req->status.MPI_TAG = MPIDI_CH4U_get_tag(msg_tag); + + MPIDI_Datatype_get_info(MPIDI_CH4U_REQUEST(match_req, count), + MPIDI_CH4U_REQUEST(match_req, datatype), + dt_contig, dt_sz, dt_ptr, dt_true_lb); + MPID_Datatype_get_size_macro(MPIDI_CH4U_REQUEST(match_req, datatype), dt_sz); + + if (MPIDI_CH4U_REQUEST(rreq, count) > dt_sz * MPIDI_CH4U_REQUEST(match_req, count)) { + rreq->status.MPI_ERROR = MPI_ERR_TRUNCATE; + count = MPIDI_CH4U_REQUEST(match_req, count); + } + else { + rreq->status.MPI_ERROR = MPI_SUCCESS; + count = MPIDI_CH4U_REQUEST(rreq, count) / dt_sz; + } + + MPIR_STATUS_SET_COUNT(match_req->status, count * dt_sz); + MPIDI_CH4U_REQUEST(rreq, count) = count; + + if (!dt_contig) { + segment_ptr = MPIDU_Segment_alloc(); + MPIR_ERR_CHKANDJUMP1(segment_ptr == NULL, mpi_errno, + MPI_ERR_OTHER, "**nomem", "**nomem %s", "Recv MPIDU_Segment_alloc"); + MPIDU_Segment_init(MPIDI_CH4U_REQUEST(match_req, buffer), count, + MPIDI_CH4U_REQUEST(match_req, datatype), segment_ptr, 0); + + last = count * dt_sz; + MPIDU_Segment_unpack(segment_ptr, 0, &last, MPIDI_CH4U_REQUEST(rreq, buffer)); + MPIDU_Segment_free(segment_ptr); + if (last != (MPI_Aint) (count * dt_sz)) { + mpi_errno = MPIR_Err_create_code(MPI_SUCCESS, MPIR_ERR_RECOVERABLE, + __FUNCTION__, __LINE__, + MPI_ERR_TYPE, "**dtypemismatch", 0); + match_req->status.MPI_ERROR = mpi_errno; + } + } + else { + MPIR_Memcpy((char *) MPIDI_CH4U_REQUEST(match_req, buffer) + dt_true_lb, + MPIDI_CH4U_REQUEST(rreq, buffer), count * dt_sz); + } + + MPIDI_CH4U_REQUEST(rreq, req->status) &= ~MPIDI_CH4U_REQ_UNEXPECTED; + if (MPIDI_CH4U_REQUEST(rreq, req->status) & MPIDI_CH4U_REQ_PEER_SSEND) { + mpi_errno = MPIDI_CH4U_reply_ssend(rreq); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + } + + dtype_release_if_not_builtin(MPIDI_CH4U_REQUEST(match_req, datatype)); + MPL_free(MPIDI_CH4U_REQUEST(rreq, buffer)); + MPIR_Object_release_ref(rreq, &c); + MPIDI_CH4I_am_request_complete(rreq); + MPIDI_CH4I_am_request_complete(match_req); + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_UNEXP_CMPL_HANDLER); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_recv_cmpl_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_recv_cmpl_handler(MPIR_Request * rreq) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_RECV_CMPL_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_RECV_CMPL_HANDLER); + + if (!MPIDI_CH4U_check_cmpl_order(rreq, MPIDI_CH4U_recv_cmpl_handler)) + return mpi_errno; + + if (MPIDI_CH4U_REQUEST(rreq, req->status) & MPIDI_CH4U_REQ_RCV_NON_CONTIG) { + MPL_free(MPIDI_CH4U_REQUEST(rreq, req->iov)); + } + + if (MPIDI_CH4U_REQUEST(rreq, req->status) & MPIDI_CH4U_REQ_UNEXPECTED) { + mpi_errno = MPIDI_CH4U_unexp_cmpl_handler(rreq); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + goto fn_exit; + } + + rreq->status.MPI_SOURCE = MPIDI_CH4U_REQUEST(rreq, src_rank); + rreq->status.MPI_TAG = MPIDI_CH4U_get_tag(MPIDI_CH4U_REQUEST(rreq, tag)); + + if (MPIDI_CH4U_REQUEST(rreq, req->status) & MPIDI_CH4U_REQ_PEER_SSEND) { + mpi_errno = MPIDI_CH4U_reply_ssend(rreq); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + } + +#ifdef MPIDI_BUILD_CH4_SHM + if (MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(rreq)) { + int continue_matching = 1; + MPIDI_CH4R_anysource_matched(MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(rreq), MPIDI_CH4R_NETMOD, + &continue_matching); + if (unlikely(MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(rreq))) { + MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(rreq)) = NULL; + MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(rreq) = NULL; + } + } +#endif + + dtype_release_if_not_builtin(MPIDI_CH4U_REQUEST(rreq, datatype)); + MPIDI_CH4I_am_request_complete(rreq); + fn_exit: + MPIDI_CH4U_progress_cmpl_list(); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_RECV_CMPL_HANDLER); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_get_acc_ack_cmpl_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_get_acc_ack_cmpl_handler(MPIR_Request * areq) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Win *win; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_GET_ACC_ACK_CMPL_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_GET_ACC_ACK_CMPL_HANDLER); + + if (!MPIDI_CH4U_check_cmpl_order(areq, MPIDI_CH4U_get_acc_ack_cmpl_handler)) + return mpi_errno; + + if (MPIDI_CH4U_REQUEST(areq, req->status) & MPIDI_CH4U_REQ_RCV_NON_CONTIG) { + MPL_free(MPIDI_CH4U_REQUEST(areq, req->iov)); + } + + win = MPIDI_CH4U_REQUEST(areq, req->areq.win_ptr); + /* MPIDI_CS_ENTER(); */ + OPA_decr_int(&MPIDI_CH4U_WIN(win, outstanding_ops)); + /* MPIDI_CS_EXIT(); */ + + dtype_release_if_not_builtin(MPIDI_CH4U_REQUEST(areq, req->areq.result_datatype)); + MPIDI_CH4I_am_request_complete(areq); + + MPIDI_CH4U_progress_cmpl_list(); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_GET_ACC_ACK_CMPL_HANDLER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_put_cmpl_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_put_cmpl_handler(MPIR_Request * rreq) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Win *win; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_PUT_CMPL_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_PUT_CMPL_HANDLER); + + if (!MPIDI_CH4U_check_cmpl_order(rreq, MPIDI_CH4U_put_cmpl_handler)) + return mpi_errno; + + if (MPIDI_CH4U_REQUEST(rreq, req->status) & MPIDI_CH4U_REQ_RCV_NON_CONTIG) { + MPL_free(MPIDI_CH4U_REQUEST(rreq, req->iov)); + } + + if (MPIDI_CH4U_REQUEST(rreq, req->preq.dt_iov)) { + MPL_free(MPIDI_CH4U_REQUEST(rreq, req->preq.dt_iov)); + } + + mpi_errno = MPIDI_CH4U_ack_put(rreq); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + win = (MPIR_Win *) MPIDI_CH4U_REQUEST(rreq, req->preq.win_ptr); + + /* MPIDI_CS_ENTER(); */ + OPA_decr_int(&MPIDI_CH4U_WIN(win, outstanding_ops)); + /* MPIDI_CS_EXIT(); */ + + MPIDI_CH4I_am_request_complete(rreq); + MPIDI_CH4U_progress_cmpl_list(); + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_PUT_CMPL_HANDLER); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_put_iov_cmpl_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_put_iov_cmpl_handler(MPIR_Request * rreq) +{ + int mpi_errno = MPI_SUCCESS; + MPIDI_CH4U_put_iov_ack_msg_t ack_msg; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_PUT_IOV_CMPL_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_PUT_IOV_CMPL_HANDLER); + + ack_msg.src_rank = MPIDI_CH4U_REQUEST(rreq, src_rank); + ack_msg.origin_preq_ptr = (uint64_t) MPIDI_CH4U_REQUEST(rreq, req->preq.preq_ptr); + ack_msg.target_preq_ptr = (uint64_t) rreq; + + mpi_errno = + MPIDI_NM_inject_am_hdr_reply(MPIDI_CH4U_win_to_context + (MPIDI_CH4U_REQUEST(rreq, req->preq.win_ptr)), + MPIDI_CH4U_REQUEST(rreq, src_rank), MPIDI_CH4U_PUT_IOV_ACK, + &ack_msg, sizeof(ack_msg)); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_PUT_IOV_CMPL_HANDLER); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_acc_iov_cmpl_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_acc_iov_cmpl_handler(MPIR_Request * rreq) +{ + int mpi_errno = MPI_SUCCESS; + MPIDI_CH4U_acc_iov_ack_msg_t ack_msg; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_ACC_IOV_CMPL_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_ACC_IOV_CMPL_HANDLER); + + ack_msg.origin_preq_ptr = (uint64_t) MPIDI_CH4U_REQUEST(rreq, req->areq.req_ptr); + ack_msg.target_preq_ptr = (uint64_t) rreq; + + mpi_errno = + MPIDI_NM_inject_am_hdr_reply(MPIDI_CH4U_win_to_context + (MPIDI_CH4U_REQUEST(rreq, req->areq.win_ptr)), + MPIDI_CH4U_REQUEST(rreq, src_rank), MPIDI_CH4U_ACC_IOV_ACK, + &ack_msg, sizeof(ack_msg)); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_ACC_IOV_CMPL_HANDLER); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_cswap_cmpl_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_cswap_cmpl_handler(MPIR_Request * rreq) +{ + int mpi_errno = MPI_SUCCESS; + void *compare_addr; + void *origin_addr; + size_t data_sz; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_CSWAP_CMPL_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_CSWAP_CMPL_HANDLER); + + if (!MPIDI_CH4U_check_cmpl_order(rreq, MPIDI_CH4U_cswap_cmpl_handler)) + return mpi_errno; + + MPIDI_Datatype_check_size(MPIDI_CH4U_REQUEST(rreq, req->creq.datatype), 1, data_sz); + origin_addr = MPIDI_CH4U_REQUEST(rreq, req->creq.data); + compare_addr = ((char *) MPIDI_CH4U_REQUEST(rreq, req->creq.data)) + data_sz; + + /* MPIDI_CS_ENTER(); */ + + if (MPIR_Compare_equal((void *) MPIDI_CH4U_REQUEST(rreq, req->creq.addr), compare_addr, + MPIDI_CH4U_REQUEST(rreq, req->creq.datatype))) { + MPIR_Memcpy(compare_addr, (void *) MPIDI_CH4U_REQUEST(rreq, req->creq.addr), data_sz); + MPIR_Memcpy((void *) MPIDI_CH4U_REQUEST(rreq, req->creq.addr), origin_addr, data_sz); + } + else { + MPIR_Memcpy(compare_addr, (void *) MPIDI_CH4U_REQUEST(rreq, req->creq.addr), data_sz); + } + + /* MPIDI_CS_EXIT(); */ + + mpi_errno = MPIDI_CH4U_ack_cswap(rreq); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + MPIDI_CH4I_am_request_complete(rreq); + MPIDI_CH4U_progress_cmpl_list(); + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_CSWAP_CMPL_HANDLER); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_do_accumulate_op +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_do_accumulate_op(void *source_buf, int source_count, + MPI_Datatype source_dtp, void *target_buf, + int target_count, MPI_Datatype target_dtp, + MPI_Aint stream_offset, MPI_Op acc_op) +{ + int mpi_errno = MPI_SUCCESS; + MPI_User_function *uop = NULL; + MPI_Aint source_dtp_size = 0, source_dtp_extent = 0; + int is_empty_source = FALSE; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_DO_ACCUMULATE_OP); + + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_DO_ACCUMULATE_OP); + + /* first Judge if source buffer is empty */ + if (acc_op == MPI_NO_OP) + is_empty_source = TRUE; + + if (is_empty_source == FALSE) { + MPIR_Assert(MPIR_DATATYPE_IS_PREDEFINED(source_dtp)); + MPID_Datatype_get_size_macro(source_dtp, source_dtp_size); + MPID_Datatype_get_extent_macro(source_dtp, source_dtp_extent); + } + + if (HANDLE_GET_KIND(acc_op) == HANDLE_KIND_BUILTIN) { + /* get the function by indexing into the op table */ + uop = MPIR_OP_HDL_TO_FN(acc_op); + } + else { + /* --BEGIN ERROR HANDLING-- */ + mpi_errno = MPIR_Err_create_code(MPI_SUCCESS, MPIR_ERR_RECOVERABLE, + FCNAME, __LINE__, MPI_ERR_OP, + "**opnotpredefined", "**opnotpredefined %d", acc_op); + return mpi_errno; + /* --END ERROR HANDLING-- */ + } + + + if (is_empty_source == TRUE || MPIR_DATATYPE_IS_PREDEFINED(target_dtp)) { + /* directly apply op if target dtp is predefined dtp OR source buffer is empty */ + MPI_Aint real_stream_offset; + void *curr_target_buf; + + if (is_empty_source == FALSE) { + MPIR_Assert(source_dtp == target_dtp); + real_stream_offset = (stream_offset / source_dtp_size) * source_dtp_extent; + curr_target_buf = (void *) ((char *) target_buf + real_stream_offset); + } + else { + curr_target_buf = target_buf; + } + + (*uop) (source_buf, curr_target_buf, &source_count, &source_dtp); + } + else { + /* derived datatype */ + MPID_Segment *segp; + DLOOP_VECTOR *dloop_vec; + MPI_Aint first, last; + int vec_len, i, count; + MPI_Aint type_extent, type_size; + MPI_Datatype type; + MPIR_Datatype *dtp; + MPI_Aint curr_len; + void *curr_loc; + int accumulated_count; + + segp = MPIDU_Segment_alloc(); + /* --BEGIN ERROR HANDLING-- */ + if (!segp) { + mpi_errno = + MPIR_Err_create_code(MPI_SUCCESS, MPIR_ERR_RECOVERABLE, FCNAME, __LINE__, + MPI_ERR_OTHER, "**nomem", 0); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_DO_ACCUMULATE_OP); + return mpi_errno; + } + /* --END ERROR HANDLING-- */ + MPIDU_Segment_init(NULL, target_count, target_dtp, segp, 0); + first = stream_offset; + last = first + source_count * source_dtp_size; + + MPID_Datatype_get_ptr(target_dtp, dtp); + vec_len = dtp->max_contig_blocks * target_count + 1; + /* +1 needed because Rob says so */ + dloop_vec = (DLOOP_VECTOR *) + MPL_malloc(vec_len * sizeof(DLOOP_VECTOR)); + /* --BEGIN ERROR HANDLING-- */ + if (!dloop_vec) { + mpi_errno = + MPIR_Err_create_code(MPI_SUCCESS, MPIR_ERR_RECOVERABLE, FCNAME, __LINE__, + MPI_ERR_OTHER, "**nomem", 0); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_DO_ACCUMULATE_OP); + return mpi_errno; + } + /* --END ERROR HANDLING-- */ + + MPIDU_Segment_pack_vector(segp, first, &last, dloop_vec, &vec_len); + + type = dtp->basic_type; + MPIR_Assert(type != MPI_DATATYPE_NULL); + + MPIR_Assert(type == source_dtp); + type_size = source_dtp_size; + type_extent = source_dtp_extent; + + i = 0; + curr_loc = dloop_vec[0].DLOOP_VECTOR_BUF; + curr_len = dloop_vec[0].DLOOP_VECTOR_LEN; + accumulated_count = 0; + while (i != vec_len) { + if (curr_len < type_size) { + MPIR_Assert(i != vec_len); + i++; + curr_len += dloop_vec[i].DLOOP_VECTOR_LEN; + continue; + } + + MPIR_Assign_trunc(count, curr_len / type_size, int); + + (*uop) ((char *) source_buf + type_extent * accumulated_count, + (char *) target_buf + MPIR_Ptr_to_aint(curr_loc), &count, &type); + + if (curr_len % type_size == 0) { + i++; + if (i != vec_len) { + curr_loc = dloop_vec[i].DLOOP_VECTOR_BUF; + curr_len = dloop_vec[i].DLOOP_VECTOR_LEN; + } + } + else { + curr_loc = (void *) ((char *) curr_loc + type_extent * count); + curr_len -= type_size * count; + } + + accumulated_count += count; + } + + MPIDU_Segment_free(segp); + MPL_free(dloop_vec); + } + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_DO_ACCUMULATE_OP); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_handle_acc_cmpl +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_handle_acc_cmpl(MPIR_Request * rreq, int do_get) +{ + int mpi_errno = MPI_SUCCESS, i; + MPI_Aint basic_sz, count, offset = 0; + struct iovec *iov; + char *src_ptr, *original = NULL; + size_t data_sz; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_HANDLE_ACC_CMPL); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_HANDLE_ACC_CMPL); + + MPID_Datatype_get_size_macro(MPIDI_CH4U_REQUEST(rreq, req->areq.target_datatype), basic_sz); + data_sz = MPIDI_CH4U_REQUEST(rreq, req->areq.data_sz); + + /* MPIDI_CS_ENTER(); */ + + if (do_get) { + original = (char *) MPL_malloc(data_sz); + MPIR_Assert(original); + } + + if (MPIDI_CH4U_REQUEST(rreq, req->areq.op) == MPI_NO_OP) { + MPIDI_CH4U_REQUEST(rreq, req->areq.origin_count) = + MPIDI_CH4U_REQUEST(rreq, req->areq.target_count); + MPIDI_CH4U_REQUEST(rreq, req->areq.data_sz) = data_sz; + } + + if (MPIDI_CH4U_REQUEST(rreq, req->areq.dt_iov) == NULL) { + + if (original) { + MPIR_Memcpy(original, MPIDI_CH4U_REQUEST(rreq, req->areq.target_addr), + basic_sz * MPIDI_CH4U_REQUEST(rreq, req->areq.target_count)); + } + + mpi_errno = MPIDI_CH4U_do_accumulate_op(MPIDI_CH4U_REQUEST(rreq, req->areq.data), + MPIDI_CH4U_REQUEST(rreq, req->areq.origin_count), + MPIDI_CH4U_REQUEST(rreq, req->areq.origin_datatype), + MPIDI_CH4U_REQUEST(rreq, req->areq.target_addr), + MPIDI_CH4U_REQUEST(rreq, req->areq.target_count), + MPIDI_CH4U_REQUEST(rreq, req->areq.target_datatype), + 0, MPIDI_CH4U_REQUEST(rreq, req->areq.op)); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + } + else { + iov = (struct iovec *) MPIDI_CH4U_REQUEST(rreq, req->areq.dt_iov); + src_ptr = (char *) MPIDI_CH4U_REQUEST(rreq, req->areq.data); + for (i = 0; i < MPIDI_CH4U_REQUEST(rreq, req->areq.n_iov); i++) { + count = iov[i].iov_len / basic_sz; + MPIR_Assert(count > 0); + + if (original) { + MPIR_Memcpy(original + offset, iov[i].iov_base, count * basic_sz); + offset += count * basic_sz; + } + + mpi_errno = MPIDI_CH4U_do_accumulate_op(src_ptr, count, + MPIDI_CH4U_REQUEST(rreq, + req->areq.origin_datatype), + iov[i].iov_base, count, MPIDI_CH4U_REQUEST(rreq, + req-> + areq. + target_datatype), + 0, MPIDI_CH4U_REQUEST(rreq, req->areq.op)); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + src_ptr += count * basic_sz; + } + MPL_free(iov); + } + + /* MPIDI_CS_EXIT(); */ + if (MPIDI_CH4U_REQUEST(rreq, req->areq.data)) + MPL_free(MPIDI_CH4U_REQUEST(rreq, req->areq.data)); + + if (original) { + MPIDI_CH4U_REQUEST(rreq, req->areq.data) = original; + mpi_errno = MPIDI_CH4U_ack_get_acc(rreq); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + } + else { + MPIDI_CH4U_REQUEST(rreq, req->areq.data) = NULL; + mpi_errno = MPIDI_CH4U_ack_acc(rreq); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + } + MPIDI_CH4I_am_request_complete(rreq); + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_HANDLE_ACC_CMPL); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_acc_cmpl_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_acc_cmpl_handler(MPIR_Request * rreq) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_ACC_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_ACC_HANDLER); + + if (!MPIDI_CH4U_check_cmpl_order(rreq, MPIDI_CH4U_acc_cmpl_handler)) + return mpi_errno; + + mpi_errno = MPIDI_CH4U_handle_acc_cmpl(rreq, 0); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + MPIDI_CH4U_progress_cmpl_list(); + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_ACC_HANDLER); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_get_acc_cmpl_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_get_acc_cmpl_handler(MPIR_Request * rreq) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_GET_ACC_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_GET_ACC_HANDLER); + + if (!MPIDI_CH4U_check_cmpl_order(rreq, MPIDI_CH4U_get_acc_cmpl_handler)) + return mpi_errno; + + mpi_errno = MPIDI_CH4U_handle_acc_cmpl(rreq, 1); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + MPIDI_CH4U_progress_cmpl_list(); + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_GET_ACC_HANDLER); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_get_ack_cmpl_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_get_ack_cmpl_handler(MPIR_Request * rreq) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Request *greq; + MPIR_Win *win; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_GET_ACK_CMPL_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_GET_ACK_CMPL_HANDLER); + + if (!MPIDI_CH4U_check_cmpl_order(rreq, MPIDI_CH4U_get_ack_cmpl_handler)) + return mpi_errno; + + greq = (MPIR_Request *) MPIDI_CH4U_REQUEST(rreq, req->greq.greq_ptr); + if (MPIDI_CH4U_REQUEST(greq, req->status) & MPIDI_CH4U_REQ_RCV_NON_CONTIG) { + MPL_free(MPIDI_CH4U_REQUEST(greq, req->iov)); + } + + win = MPIDI_CH4U_REQUEST(greq, req->greq.win_ptr); + /* MPIDI_CS_ENTER(); */ + OPA_decr_int(&MPIDI_CH4U_WIN(win, outstanding_ops)); + /* MPIDI_CS_EXIT(); */ + + MPIDI_CH4I_am_request_complete(greq); + MPIDI_CH4I_am_request_complete(rreq); + MPIDI_CH4U_progress_cmpl_list(); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_GET_ACK_CMPL_HANDLER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_cswap_ack_cmpl_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_cswap_ack_cmpl_handler(MPIR_Request * rreq) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Win *win; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_CSWAP_ACK_CMPL_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_CSWAP_ACK_CMPL_HANDLER); + + if (!MPIDI_CH4U_check_cmpl_order(rreq, MPIDI_CH4U_cswap_ack_cmpl_handler)) + return mpi_errno; + + win = MPIDI_CH4U_REQUEST(rreq, req->creq.win_ptr); + /* MPIDI_CS_ENTER(); */ + OPA_decr_int(&MPIDI_CH4U_WIN(win, outstanding_ops)); + /* MPIDI_CS_EXIT(); */ + + MPL_free(MPIDI_CH4U_REQUEST(rreq, req->creq.data)); + MPIDI_CH4I_am_request_complete(rreq); + + MPIDI_CH4U_progress_cmpl_list(); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_CSWAP_ACK_CMPL_HANDLER); + return mpi_errno; +} + +/* Main body of the send target handler, commonly used by both eager send and rendezvous send */ +static inline int MPIDI_CH4I_do_send_target_handler(void **data, + size_t * p_data_sz, + int *is_contig, + MPIDI_NM_am_completion_handler_fn * + cmpl_handler_fn, MPIR_Request * rreq) +{ + int dt_contig, n_iov; + MPI_Aint dt_true_lb, last, num_iov; + MPIR_Datatype *dt_ptr; + MPID_Segment *segment_ptr; + size_t data_sz; + + *cmpl_handler_fn = MPIDI_CH4U_recv_cmpl_handler; + MPIDI_CH4U_REQUEST(rreq, req->seq_no) = OPA_fetch_and_add_int(&MPIDI_CH4_Global.nxt_seq_no, 1); + + MPIDI_Datatype_get_info(MPIDI_CH4U_REQUEST(rreq, count), + MPIDI_CH4U_REQUEST(rreq, datatype), + dt_contig, data_sz, dt_ptr, dt_true_lb); + *is_contig = dt_contig; + + if (dt_contig) { + *p_data_sz = data_sz; + *data = (char *) MPIDI_CH4U_REQUEST(rreq, buffer) + dt_true_lb; + } + else { + segment_ptr = MPIDU_Segment_alloc(); + MPIR_Assert(segment_ptr); + + MPIDU_Segment_init(MPIDI_CH4U_REQUEST(rreq, buffer), + MPIDI_CH4U_REQUEST(rreq, count), + MPIDI_CH4U_REQUEST(rreq, datatype), segment_ptr, 0); + + if (*p_data_sz > data_sz) { + rreq->status.MPI_ERROR = MPI_ERR_TRUNCATE; + *p_data_sz = data_sz; + } + last = data_sz; + MPIDU_Segment_count_contig_blocks(segment_ptr, 0, &last, &num_iov); + n_iov = (int) num_iov; + MPIR_Assert(n_iov > 0); + MPIDI_CH4U_REQUEST(rreq, req->iov) = + (struct iovec *) MPL_malloc(n_iov * sizeof(struct iovec)); + MPIR_Assert(MPIDI_CH4U_REQUEST(rreq, req->iov)); + + last = *p_data_sz; + MPIDU_Segment_pack_vector(segment_ptr, 0, &last, MPIDI_CH4U_REQUEST(rreq, req->iov), + &n_iov); + if (last != (MPI_Aint) * p_data_sz) { + rreq->status.MPI_ERROR = MPI_ERR_TYPE; + } + *data = MPIDI_CH4U_REQUEST(rreq, req->iov); + *p_data_sz = n_iov; + MPIDI_CH4U_REQUEST(rreq, req->status) |= MPIDI_CH4U_REQ_RCV_NON_CONTIG; + MPL_free(segment_ptr); + } + + return MPI_SUCCESS; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_send_target_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_send_target_handler(void *am_hdr, + void **data, + size_t * p_data_sz, + int *is_contig, + MPIDI_NM_am_completion_handler_fn * + cmpl_handler_fn, MPIR_Request ** req) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Request *rreq = NULL; + MPIR_Comm *root_comm; + MPIDI_CH4U_hdr_t *hdr = (MPIDI_CH4U_hdr_t *) am_hdr; + MPIR_Context_id_t context_id; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_SEND_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_SEND_HANDLER); + context_id = MPIDI_CH4U_get_context(hdr->msg_tag); + root_comm = MPIDI_CH4U_context_id_to_comm(context_id); + if (root_comm) { + /* MPIDI_CS_ENTER(); */ + rreq = MPIDI_CH4U_dequeue_posted(hdr->msg_tag, &MPIDI_CH4U_COMM(root_comm, posted_list)); + /* MPIDI_CS_EXIT(); */ + } + + if (rreq == NULL) { + rreq = MPIDI_CH4I_am_request_create(MPIR_REQUEST_KIND__RECV); + MPIDI_CH4U_REQUEST(rreq, buffer) = (char *) MPL_malloc(*p_data_sz); + MPIDI_CH4U_REQUEST(rreq, datatype) = MPI_BYTE; + MPIDI_CH4U_REQUEST(rreq, count) = *p_data_sz; + MPIDI_CH4U_REQUEST(rreq, tag) = hdr->msg_tag; + MPIDI_CH4U_REQUEST(rreq, src_rank) = hdr->src_rank; + MPIDI_CH4U_REQUEST(rreq, req->status) |= MPIDI_CH4U_REQ_BUSY; + MPIDI_CH4U_REQUEST(rreq, req->status) |= MPIDI_CH4U_REQ_UNEXPECTED; + /* MPIDI_CS_ENTER(); */ + if (root_comm) { + MPIR_Comm_add_ref(root_comm); + MPIDI_CH4U_enqueue_unexp(rreq, &MPIDI_CH4U_COMM(root_comm, unexp_list)); + } + else { + MPIDI_CH4U_enqueue_unexp(rreq, MPIDI_CH4U_context_id_to_uelist(context_id)); + } + /* MPIDI_CS_EXIT(); */ + } + else { + /* rreq != NULL <=> root_comm != NULL */ + MPIR_Assert(root_comm); + /* Decrement the refcnt when popping a request out from posted_list */ + MPIR_Comm_release(root_comm); + MPIDI_CH4U_REQUEST(rreq, src_rank) = hdr->src_rank; + MPIDI_CH4U_REQUEST(rreq, tag) = hdr->msg_tag; + } + + *req = rreq; + + mpi_errno = MPIDI_CH4I_do_send_target_handler(data, p_data_sz, is_contig, + cmpl_handler_fn, rreq); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_SEND_HANDLER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_send_long_req_target_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_send_long_req_target_handler(void *am_hdr, + void **data, + size_t * p_data_sz, + int *is_contig, + MPIDI_NM_am_completion_handler_fn * + cmpl_handler_fn, MPIR_Request ** req) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Request *rreq = NULL; + MPIR_Comm *root_comm; + MPIDI_CH4U_hdr_t *hdr = (MPIDI_CH4U_hdr_t *) am_hdr; + MPIDI_CH4U_send_long_req_msg_t *lreq_hdr = (MPIDI_CH4U_send_long_req_msg_t *) am_hdr; + MPIR_Context_id_t context_id; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_SEND_LONG_REQ_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_SEND_LONG_REQ_HANDLER); + + context_id = MPIDI_CH4U_get_context(hdr->msg_tag); + root_comm = MPIDI_CH4U_context_id_to_comm(context_id); + if (root_comm) { + /* MPIDI_CS_ENTER(); */ + rreq = MPIDI_CH4U_dequeue_posted(hdr->msg_tag, &MPIDI_CH4U_COMM(root_comm, posted_list)); + /* MPIDI_CS_EXIT(); */ + } + + if (rreq == NULL) { + rreq = MPIDI_CH4I_am_request_create(MPIR_REQUEST_KIND__RECV); + + MPIDI_CH4U_REQUEST(rreq, buffer) = NULL; + MPIDI_CH4U_REQUEST(rreq, datatype) = MPI_BYTE; + MPIDI_CH4U_REQUEST(rreq, count) = lreq_hdr->data_sz; + MPIDI_CH4U_REQUEST(rreq, req->status) |= MPIDI_CH4U_REQ_LONG_RTS; + MPIDI_CH4U_REQUEST(rreq, req->rreq.peer_req_ptr) = lreq_hdr->sreq_ptr; + MPIDI_CH4U_REQUEST(rreq, tag) = hdr->msg_tag; + MPIDI_CH4U_REQUEST(rreq, src_rank) = hdr->src_rank; + + /* MPIDI_CS_ENTER(); */ + if (root_comm) { + MPIR_Comm_add_ref(root_comm); + MPIDI_CH4U_enqueue_unexp(rreq, &MPIDI_CH4U_COMM(root_comm, unexp_list)); + } + else { + MPIDI_CH4U_enqueue_unexp(rreq, MPIDI_CH4U_context_id_to_uelist(context_id)); + } + /* MPIDI_CS_EXIT(); */ + } + else { + /* Matching receive was posted, tell the netmod */ + MPIR_Comm_release(root_comm); /* -1 for posted_list */ + MPIDI_CH4U_REQUEST(rreq, req->status) |= MPIDI_CH4U_REQ_LONG_RTS; + MPIDI_CH4U_REQUEST(rreq, req->rreq.peer_req_ptr) = lreq_hdr->sreq_ptr; + MPIDI_CH4U_REQUEST(rreq, tag) = hdr->msg_tag; + MPIDI_CH4U_REQUEST(rreq, src_rank) = hdr->src_rank; + mpi_errno = MPIDI_NM_am_recv(rreq); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + } + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_SEND_LONG_REQ_HANDLER); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_send_long_lmt_target_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_send_long_lmt_target_handler(void *am_hdr, + void **data, + size_t * p_data_sz, + int *is_contig, + MPIDI_NM_am_completion_handler_fn * + cmpl_handler_fn, MPIR_Request ** req) +{ + int mpi_errno; + MPIR_Request *rreq; + MPIDI_CH4U_send_long_lmt_msg_t *lmt_hdr = (MPIDI_CH4U_send_long_lmt_msg_t *) am_hdr; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_SEND_LONG_LMT_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_SEND_LONG_LMT_HANDLER); + + rreq = (MPIR_Request *) lmt_hdr->rreq_ptr; + MPIR_Assert(rreq); + mpi_errno = MPIDI_CH4I_do_send_target_handler(data, p_data_sz, is_contig, + cmpl_handler_fn, rreq); + *req = rreq; + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_SEND_LONG_LMT_HANDLER); + + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_ssend_target_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_ssend_target_handler(void *am_hdr, + void **data, + size_t * p_data_sz, + int *is_contig, + MPIDI_NM_am_completion_handler_fn * + cmpl_handler_fn, MPIR_Request ** req) +{ + int mpi_errno = MPI_SUCCESS; + + MPIDI_CH4U_ssend_req_msg_t *msg_hdr = (MPIDI_CH4U_ssend_req_msg_t *) am_hdr; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_SSEND_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_SSEND_HANDLER); + + mpi_errno = MPIDI_CH4U_send_target_handler(am_hdr, + data, p_data_sz, is_contig, cmpl_handler_fn, req); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + MPIR_Assert(req); + MPIDI_CH4U_REQUEST(*req, req->rreq.peer_req_ptr) = msg_hdr->sreq_ptr; + MPIDI_CH4U_REQUEST(*req, req->status) |= MPIDI_CH4U_REQ_PEER_SSEND; + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_SSEND_HANDLER); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_ssend_ack_target_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_ssend_ack_target_handler(void *am_hdr, + void **data, + size_t * p_data_sz, int *is_contig, + MPIDI_NM_am_completion_handler_fn * + cmpl_handler_fn, MPIR_Request ** req) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Request *sreq; + MPIDI_CH4U_ssend_ack_msg_t *msg_hdr = (MPIDI_CH4U_ssend_ack_msg_t *) am_hdr; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_SSEND_ACK_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_SSEND_ACK_HANDLER); + + sreq = (MPIR_Request *) msg_hdr->sreq_ptr; + MPIDI_CH4I_am_request_complete(sreq); + + if (req) + *req = NULL; + if (cmpl_handler_fn) + *cmpl_handler_fn = NULL; + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_SSEND_ACK_HANDLER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_send_long_ack_target_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_send_long_ack_target_handler(void *am_hdr, + void **data, + size_t * p_data_sz, int *is_contig, + MPIDI_NM_am_completion_handler_fn * + cmpl_handler_fn, MPIR_Request ** req) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Request *sreq; + MPIDI_CH4U_send_long_ack_msg_t *msg_hdr = (MPIDI_CH4U_send_long_ack_msg_t *) am_hdr; + MPIDI_CH4U_send_long_lmt_msg_t send_hdr; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_SEND_LONG_ACK_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_SEND_LONG_ACK_HANDLER); + + sreq = (MPIR_Request *) msg_hdr->sreq_ptr; + MPIR_Assert(sreq != NULL); + + /* Start the main data transfer */ + send_hdr.rreq_ptr = msg_hdr->rreq_ptr; + mpi_errno = + MPIDI_NM_send_am_reply(MPIDI_CH4U_get_context(MPIDI_CH4U_REQUEST(sreq, req->lreq).msg_tag), + MPIDI_CH4U_REQUEST(sreq, src_rank), MPIDI_CH4U_SEND_LONG_LMT, + &send_hdr, sizeof(send_hdr), MPIDI_CH4U_REQUEST(sreq, + req->lreq).src_buf, + MPIDI_CH4U_REQUEST(sreq, req->lreq).count, MPIDI_CH4U_REQUEST(sreq, + req-> + lreq). + datatype, sreq); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + if (cmpl_handler_fn) + *cmpl_handler_fn = MPIDI_CH4U_send_origin_cmpl_handler; + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_SEND_LONG_ACK_HANDLER); + + fn_exit: + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_put_ack_target_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_put_ack_target_handler(void *am_hdr, + void **data, + size_t * p_data_sz, int *is_contig, + MPIDI_NM_am_completion_handler_fn * + cmpl_handler_fn, MPIR_Request ** req) +{ + int mpi_errno = MPI_SUCCESS; + MPIDI_CH4U_put_ack_msg_t *msg_hdr = (MPIDI_CH4U_put_ack_msg_t *) am_hdr; + MPIR_Win *win; + MPIR_Request *preq; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_PUT_ACK_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_PUT_ACK_HANDLER); + + preq = (MPIR_Request *) msg_hdr->preq_ptr; + win = MPIDI_CH4U_REQUEST(preq, req->preq.win_ptr); + + if (MPIDI_CH4U_REQUEST(preq, req->preq.dt_iov)) { + MPL_free(MPIDI_CH4U_REQUEST(preq, req->preq.dt_iov)); + } + + MPIDI_CH4I_am_request_complete(preq); + + /* MPIDI_CS_ENTER(); */ + OPA_decr_int(&MPIDI_CH4U_WIN(win, outstanding_ops)); + /* MPIDI_CS_EXIT(); */ + + if (req) + *req = NULL; + if (cmpl_handler_fn) + *cmpl_handler_fn = NULL; + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_PUT_ACK_HANDLER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_acc_ack_target_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_acc_ack_target_handler(void *am_hdr, + void **data, + size_t * p_data_sz, int *is_contig, + MPIDI_NM_am_completion_handler_fn * + cmpl_handler_fn, MPIR_Request ** req) +{ + int mpi_errno = MPI_SUCCESS; + MPIDI_CH4U_acc_ack_msg_t *msg_hdr = (MPIDI_CH4U_acc_ack_msg_t *) am_hdr; + MPIR_Win *win; + MPIR_Request *areq; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_ACC_ACK_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_ACC_ACK_HANDLER); + + areq = (MPIR_Request *) msg_hdr->req_ptr; + win = MPIDI_CH4U_REQUEST(areq, req->areq.win_ptr); + + if (MPIDI_CH4U_REQUEST(areq, req->areq.dt_iov)) { + MPL_free(MPIDI_CH4U_REQUEST(areq, req->areq.dt_iov)); + } + + MPIDI_CH4I_am_request_complete(areq); + + /* MPIDI_CS_ENTER(); */ + OPA_decr_int(&MPIDI_CH4U_WIN(win, outstanding_ops)); + /* MPIDI_CS_EXIT(); */ + + if (req) + *req = NULL; + if (cmpl_handler_fn) + *cmpl_handler_fn = NULL; + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_ACC_ACK_HANDLER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_get_acc_ack_target_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_get_acc_ack_target_handler(void *am_hdr, + void **data, + size_t * p_data_sz, int *is_contig, + MPIDI_NM_am_completion_handler_fn * + cmpl_handler_fn, MPIR_Request ** req) +{ + int mpi_errno = MPI_SUCCESS; + MPIDI_CH4U_acc_ack_msg_t *msg_hdr = (MPIDI_CH4U_acc_ack_msg_t *) am_hdr; + MPIR_Request *areq; + + size_t data_sz; + int dt_contig, n_iov; + MPI_Aint dt_true_lb, last, num_iov; + MPIR_Datatype *dt_ptr; + MPID_Segment *segment_ptr; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_ACC_ACK_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_ACC_ACK_HANDLER); + + areq = (MPIR_Request *) msg_hdr->req_ptr; + + if (MPIDI_CH4U_REQUEST(areq, req->areq.dt_iov)) { + MPL_free(MPIDI_CH4U_REQUEST(areq, req->areq.dt_iov)); + } + + MPIDI_Datatype_get_info(MPIDI_CH4U_REQUEST(areq, req->areq.result_count), + MPIDI_CH4U_REQUEST(areq, req->areq.result_datatype), + dt_contig, data_sz, dt_ptr, dt_true_lb); + *is_contig = dt_contig; + + if (dt_contig) { + *p_data_sz = data_sz; + *data = (char *) MPIDI_CH4U_REQUEST(areq, req->areq.result_addr) + dt_true_lb; + } + else { + segment_ptr = MPIDU_Segment_alloc(); + MPIR_Assert(segment_ptr); + + MPIDU_Segment_init(MPIDI_CH4U_REQUEST(areq, req->areq.result_addr), + MPIDI_CH4U_REQUEST(areq, req->areq.result_count), + MPIDI_CH4U_REQUEST(areq, req->areq.result_datatype), segment_ptr, 0); + + last = data_sz; + MPIDU_Segment_count_contig_blocks(segment_ptr, 0, &last, &num_iov); + n_iov = (int) num_iov; + MPIR_Assert(n_iov > 0); + MPIDI_CH4U_REQUEST(areq, req->iov) = + (struct iovec *) MPL_malloc(n_iov * sizeof(struct iovec)); + MPIR_Assert(MPIDI_CH4U_REQUEST(areq, req->iov)); + + last = data_sz; + MPIDU_Segment_pack_vector(segment_ptr, 0, &last, MPIDI_CH4U_REQUEST(areq, req->iov), + &n_iov); + MPIR_Assert(last == (MPI_Aint) data_sz); + *data = MPIDI_CH4U_REQUEST(areq, req->iov); + *p_data_sz = n_iov; + MPIDI_CH4U_REQUEST(areq, req->status) |= MPIDI_CH4U_REQ_RCV_NON_CONTIG; + MPL_free(segment_ptr); + } + + *req = areq; + *cmpl_handler_fn = MPIDI_CH4U_get_acc_ack_cmpl_handler; + MPIDI_CH4U_REQUEST(areq, req->seq_no) = OPA_fetch_and_add_int(&MPIDI_CH4_Global.nxt_seq_no, 1); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_ACC_ACK_HANDLER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_cswap_ack_target_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_cswap_ack_target_handler(void *am_hdr, + void **data, + size_t * p_data_sz, int *is_contig, + MPIDI_NM_am_completion_handler_fn * + cmpl_handler_fn, MPIR_Request ** req) +{ + int mpi_errno = MPI_SUCCESS; + MPIDI_CH4U_cswap_ack_msg_t *msg_hdr = (MPIDI_CH4U_cswap_ack_msg_t *) am_hdr; + MPIR_Request *creq; + uint64_t data_sz; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_CSWAP_ACK_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_CSWAP_ACK_HANDLER); + + creq = (MPIR_Request *) msg_hdr->req_ptr; + MPIDI_Datatype_check_size(MPIDI_CH4U_REQUEST(creq, req->creq.datatype), 1, data_sz); + *data = MPIDI_CH4U_REQUEST(creq, req->creq.result_addr); + *p_data_sz = data_sz; + *is_contig = 1; + + *req = creq; + *cmpl_handler_fn = MPIDI_CH4U_cswap_ack_cmpl_handler; + MPIDI_CH4U_REQUEST(creq, req->seq_no) = OPA_fetch_and_add_int(&MPIDI_CH4_Global.nxt_seq_no, 1); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_CSWAP_ACK_HANDLER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_win_lock_advance +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_win_lock_advance(MPIR_Win * win) +{ + int mpi_errno = MPI_SUCCESS; + struct MPIDI_CH4U_win_sync_lock *slock = &MPIDI_CH4U_WIN(win, sync).lock; + struct MPIDI_CH4U_win_queue *q = &slock->local.requested; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_WIN_LOCK_ADVANCE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_WIN_LOCK_ADVANCE); + + if ((q->head != NULL) && ((slock->local.count == 0) || + ((slock->local.type == MPI_LOCK_SHARED) && + (q->head->type == MPI_LOCK_SHARED)))) { + struct MPIDI_CH4U_win_lock *lock = q->head; + q->head = lock->next; + + if (q->head == NULL) + q->tail = NULL; + + ++slock->local.count; + slock->local.type = lock->type; + + MPIDI_CH4U_win_cntrl_msg_t msg; + msg.win_id = MPIDI_CH4U_WIN(win, win_id); + msg.origin_rank = win->comm_ptr->rank; + + if (lock->mtype == MPIDI_CH4U_WIN_LOCK) + msg.type = MPIDI_CH4U_WIN_LOCK_ACK; + else if (lock->mtype == MPIDI_CH4U_WIN_LOCKALL) + msg.type = MPIDI_CH4U_WIN_LOCKALL_ACK; + else + MPIR_ERR_SETANDJUMP(mpi_errno, MPI_ERR_OTHER, "**rmasync"); + + mpi_errno = MPIDI_NM_inject_am_hdr_reply(MPIDI_CH4U_win_to_context(win), + lock->rank, + MPIDI_CH4U_WIN_CTRL, &msg, sizeof(msg)); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + MPL_free(lock); + + mpi_errno = MPIDI_CH4U_win_lock_advance(win); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + } + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_WIN_LOCK_ADVANCE); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_win_lock_req_proc +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline void MPIDI_CH4U_win_lock_req_proc(const MPIDI_CH4U_win_cntrl_msg_t * info, + MPIR_Win * win) +{ + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_WIN_LOCK_REQ_PROC); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_WIN_LOCK_REQ_PROC); + + struct MPIDI_CH4U_win_lock *lock = (struct MPIDI_CH4U_win_lock *) + MPL_calloc(1, sizeof(struct MPIDI_CH4U_win_lock)); + + if (info->type == MPIDI_CH4U_WIN_LOCK) + lock->mtype = MPIDI_CH4U_WIN_LOCK; + else if (info->type == MPIDI_CH4U_WIN_LOCKALL) + lock->mtype = MPIDI_CH4U_WIN_LOCKALL; + + lock->rank = info->origin_rank; + lock->type = info->lock_type; + struct MPIDI_CH4U_win_queue *q = &MPIDI_CH4U_WIN(win, sync).lock.local.requested; + MPIR_Assert((q->head != NULL) ^ (q->tail == NULL)); + + if (q->tail == NULL) + q->head = lock; + else + q->tail->next = lock; + + q->tail = lock; + + MPIDI_CH4U_win_lock_advance(win); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_WIN_LOCK_REQ_PROC); + return; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_win_lock_ack_proc +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline void MPIDI_CH4U_win_lock_ack_proc(const MPIDI_CH4U_win_cntrl_msg_t * info, + MPIR_Win * win) +{ + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_WIN_LOCK_ACK_PROC); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_WIN_LOCK_ACK_PROC); + + if (info->type == MPIDI_CH4U_WIN_LOCK_ACK) + MPIDI_CH4U_WIN(win, sync).lock.remote.locked += 1; + else if (info->type == MPIDI_CH4U_WIN_LOCKALL_ACK) + MPIDI_CH4U_WIN(win, sync).lock.remote.allLocked += 1; + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_WIN_LOCK_ACK_PROC); +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_win_unlock_ack_proc +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline void MPIDI_CH4U_win_unlock_proc(const MPIDI_CH4U_win_cntrl_msg_t * info, + MPIR_Win * win) +{ + + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_WIN_UNLOCK_ACK_PROC); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_WIN_UNLOCK_ACK_PROC); + + --MPIDI_CH4U_WIN(win, sync).lock.local.count; + MPIR_Assert((int) MPIDI_CH4U_WIN(win, sync).lock.local.count >= 0); + MPIDI_CH4U_win_lock_advance(win); + + MPIDI_CH4U_win_cntrl_msg_t msg; + msg.win_id = MPIDI_CH4U_WIN(win, win_id); + msg.origin_rank = win->comm_ptr->rank; + msg.type = MPIDI_CH4U_WIN_UNLOCK_ACK; + + mpi_errno = MPIDI_NM_inject_am_hdr_reply(MPIDI_CH4U_win_to_context(win), + info->origin_rank, + MPIDI_CH4U_WIN_CTRL, &msg, sizeof(msg)); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_WIN_UNLOCK_ACK_PROC); + return; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_win_complete_proc +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline void MPIDI_CH4U_win_complete_proc(const MPIDI_CH4U_win_cntrl_msg_t * info, + MPIR_Win * win) +{ + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_WIN_COMPLETE_PROC); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_WIN_COMPLETE_PROC); + + ++MPIDI_CH4U_WIN(win, sync).sc.count; + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_WIN_COMPLETE_PROC); +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_win_post_proc +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline void MPIDI_CH4U_win_post_proc(const MPIDI_CH4U_win_cntrl_msg_t * info, MPIR_Win * win) +{ + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_WIN_POST_PROC); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_WIN_POST_PROC); + + ++MPIDI_CH4U_WIN(win, sync).pw.count; + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_WIN_POST_PROC); +} + + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_win_unlock_done_cb +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline void MPIDI_CH4U_win_unlock_done_cb(const MPIDI_CH4U_win_cntrl_msg_t * info, + MPIR_Win * win) +{ + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_WIN_UNLOCK_DONE_CB); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_WIN_UNLOCK_DONE_CB); + + if (MPIDI_CH4U_WIN(win, sync).origin_epoch_type == MPIDI_CH4U_EPOTYPE_LOCK) { + MPIDI_CH4U_WIN(win, sync).lock.remote.locked--; + } + else if (MPIDI_CH4U_WIN(win, sync).origin_epoch_type == MPIDI_CH4U_EPOTYPE_LOCK_ALL) { + MPIR_Assert((int) MPIDI_CH4U_WIN(win, sync).lock.remote.allLocked > 0); + MPIDI_CH4U_WIN(win, sync).lock.remote.allLocked -= 1; + } + else { + MPIR_Assert(0); + } + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_WIN_UNLOCK_DONE_CB); +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_win_ctrl_target_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_win_ctrl_target_handler(void *am_hdr, + void **data, + size_t * p_data_sz, int *is_contig, + MPIDI_NM_am_completion_handler_fn * + cmpl_handler_fn, MPIR_Request ** req) +{ + int mpi_errno = MPI_SUCCESS; + MPIDI_CH4U_win_cntrl_msg_t *msg_hdr = (MPIDI_CH4U_win_cntrl_msg_t *) am_hdr; + MPIR_Win *win; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_WIN_CTRL_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_WIN_CTRL_HANDLER); + + MPL_HASH_FIND(dev.ch4u.hash_handle, MPIDI_CH4_Global.win_hash, + &msg_hdr->win_id, sizeof(uint64_t), win); + /* TODO: check output win ptr */ + + switch (msg_hdr->type) { + char buff[32]; + + case MPIDI_CH4U_WIN_LOCK: + case MPIDI_CH4U_WIN_LOCKALL: + MPIDI_CH4U_win_lock_req_proc(msg_hdr, win); + break; + + case MPIDI_CH4U_WIN_LOCK_ACK: + case MPIDI_CH4U_WIN_LOCKALL_ACK: + MPIDI_CH4U_win_lock_ack_proc(msg_hdr, win); + break; + + case MPIDI_CH4U_WIN_UNLOCK: + case MPIDI_CH4U_WIN_UNLOCKALL: + MPIDI_CH4U_win_unlock_proc(msg_hdr, win); + break; + + case MPIDI_CH4U_WIN_UNLOCK_ACK: + case MPIDI_CH4U_WIN_UNLOCKALL_ACK: + MPIDI_CH4U_win_unlock_done_cb(msg_hdr, win); + break; + + case MPIDI_CH4U_WIN_COMPLETE: + MPIDI_CH4U_win_complete_proc(msg_hdr, win); + break; + + case MPIDI_CH4U_WIN_POST: + MPIDI_CH4U_win_post_proc(msg_hdr, win); + break; + + default: + MPL_snprintf(buff, sizeof(buff), "Invalid message type: %d\n", msg_hdr->type); + MPID_Abort(NULL, MPI_ERR_INTERN, 1, buff); + } + + if (req) + *req = NULL; + if (cmpl_handler_fn) + *cmpl_handler_fn = NULL; + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_WIN_CTRL_HANDLER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_put_target_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_put_target_handler(void *am_hdr, + void **data, + size_t * p_data_sz, + int *is_contig, + MPIDI_NM_am_completion_handler_fn * + cmpl_handler_fn, MPIR_Request ** req) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Request *rreq = NULL; + size_t data_sz; + struct iovec *iov, *dt_iov; + uintptr_t base; /* Base address of the window */ + size_t offset; + + int dt_contig, n_iov; + MPI_Aint dt_true_lb, last, num_iov; + MPIR_Datatype *dt_ptr; + MPID_Segment *segment_ptr; + MPIR_Win *win; + MPIDI_CH4U_put_msg_t *msg_hdr = (MPIDI_CH4U_put_msg_t *) am_hdr; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_PUT_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_PUT_HANDLER); + + rreq = MPIDI_CH4I_am_win_request_create(); + MPIR_Assert(rreq); + rreq->kind = MPIR_REQUEST_KIND__RMA; + *req = rreq; + + MPIDI_CH4U_REQUEST(*req, req->preq.preq_ptr) = msg_hdr->preq_ptr; + MPIDI_CH4U_REQUEST(*req, src_rank) = msg_hdr->src_rank; + + MPL_HASH_FIND(dev.ch4u.hash_handle, MPIDI_CH4_Global.win_hash, + &msg_hdr->win_id, sizeof(uint64_t), win); + MPIR_Assert(win); + + base = MPIDI_CH4I_win_base_at_target(win); + + /* MPIDI_CS_ENTER(); */ + OPA_incr_int(&MPIDI_CH4U_WIN(win, outstanding_ops)); + /* MPIDI_CS_EXIT(); */ + MPIDI_CH4U_REQUEST(rreq, req->preq.win_ptr) = win; + + *cmpl_handler_fn = MPIDI_CH4U_put_cmpl_handler; + MPIDI_CH4U_REQUEST(rreq, req->seq_no) = OPA_fetch_and_add_int(&MPIDI_CH4_Global.nxt_seq_no, 1); + + offset = win->disp_unit * msg_hdr->target_disp; + if (msg_hdr->n_iov) { + int i; + dt_iov = (struct iovec *) MPL_malloc(sizeof(struct iovec) * msg_hdr->n_iov); + MPIR_Assert(dt_iov); + + iov = (struct iovec *) ((char *) am_hdr + sizeof(*msg_hdr)); + for (i = 0; i < msg_hdr->n_iov; i++) + iov[i].iov_base = (char *) iov[i].iov_base + base + offset; + MPIR_Memcpy(dt_iov, iov, sizeof(struct iovec) * msg_hdr->n_iov); + MPIDI_CH4U_REQUEST(rreq, req->preq.dt_iov) = dt_iov; + MPIDI_CH4U_REQUEST(rreq, req->preq.n_iov) = msg_hdr->n_iov; + *is_contig = 0; + *data = iov; + *p_data_sz = msg_hdr->n_iov; + goto fn_exit; + } + + MPIDI_CH4U_REQUEST(rreq, req->preq.dt_iov) = NULL; + MPIDI_Datatype_get_info(msg_hdr->count, msg_hdr->datatype, + dt_contig, data_sz, dt_ptr, dt_true_lb); + *is_contig = dt_contig; + + if (dt_contig) { + *p_data_sz = data_sz; + *data = (char *) (offset + base + dt_true_lb); + } + else { + segment_ptr = MPIDU_Segment_alloc(); + MPIR_Assert(segment_ptr); + + MPIDU_Segment_init((void *) (offset + base), msg_hdr->count, msg_hdr->datatype, + segment_ptr, 0); + last = data_sz; + MPIDU_Segment_count_contig_blocks(segment_ptr, 0, &last, &num_iov); + n_iov = (int) num_iov; + MPIR_Assert(n_iov > 0); + MPIDI_CH4U_REQUEST(rreq, req->iov) = + (struct iovec *) MPL_malloc(n_iov * sizeof(struct iovec)); + MPIR_Assert(MPIDI_CH4U_REQUEST(rreq, req->iov)); + + last = data_sz; + MPIDU_Segment_pack_vector(segment_ptr, 0, &last, MPIDI_CH4U_REQUEST(rreq, req->iov), + &n_iov); + MPIR_Assert(last == (MPI_Aint) data_sz); + *data = MPIDI_CH4U_REQUEST(rreq, req->iov); + *p_data_sz = n_iov; + MPIDI_CH4U_REQUEST(rreq, req->status) |= MPIDI_CH4U_REQ_RCV_NON_CONTIG; + MPL_free(segment_ptr); + } + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_PUT_HANDLER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_put_iov_target_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_put_iov_target_handler(void *am_hdr, + void **data, + size_t * p_data_sz, + int *is_contig, + MPIDI_NM_am_completion_handler_fn * + cmpl_handler_fn, MPIR_Request ** req) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Request *rreq = NULL; + struct iovec *dt_iov; + + MPIR_Win *win; + MPIDI_CH4U_put_msg_t *msg_hdr = (MPIDI_CH4U_put_msg_t *) am_hdr; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_PUT_IOV_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_PUT_IOV_HANDLER); + + rreq = MPIDI_CH4I_am_win_request_create(); + MPIR_Assert(rreq); + rreq->kind = MPIR_REQUEST_KIND__RMA; + *req = rreq; + + MPIDI_CH4U_REQUEST(*req, req->preq.preq_ptr) = msg_hdr->preq_ptr; + MPIDI_CH4U_REQUEST(*req, src_rank) = msg_hdr->src_rank; + + MPL_HASH_FIND(dev.ch4u.hash_handle, MPIDI_CH4_Global.win_hash, + &msg_hdr->win_id, sizeof(uint64_t), win); + MPIR_Assert(win); + + /* MPIDI_CS_ENTER(); */ + OPA_incr_int(&MPIDI_CH4U_WIN(win, outstanding_ops)); + /* MPIDI_CS_EXIT(); */ + MPIDI_CH4U_REQUEST(rreq, req->preq.win_ptr) = win; + + *cmpl_handler_fn = MPIDI_CH4U_put_iov_cmpl_handler; + MPIDI_CH4U_REQUEST(rreq, req->seq_no) = OPA_fetch_and_add_int(&MPIDI_CH4_Global.nxt_seq_no, 1); + + /* Base adjustment for iov will be done after we get the entire iovs, + * at MPIDI_CH4U_put_data_target_handler */ + MPIR_Assert(msg_hdr->n_iov); + dt_iov = (struct iovec *) MPL_malloc(sizeof(struct iovec) * msg_hdr->n_iov); + MPIR_Assert(dt_iov); + + MPIDI_CH4U_REQUEST(rreq, req->preq.dt_iov) = dt_iov; + MPIDI_CH4U_REQUEST(rreq, req->preq.n_iov) = msg_hdr->n_iov; + *is_contig = 1; + *data = dt_iov; + *p_data_sz = msg_hdr->n_iov * sizeof(struct iovec); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_PUT_IOV_HANDLER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_put_iov_ack_target_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_put_iov_ack_target_handler(void *am_hdr, + void **data, + size_t * p_data_sz, + int *is_contig, + MPIDI_NM_am_completion_handler_fn * + cmpl_handler_fn, MPIR_Request ** req) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Request *rreq, *origin_req; + MPIDI_CH4U_put_iov_ack_msg_t *msg_hdr = (MPIDI_CH4U_put_iov_ack_msg_t *) am_hdr; + MPIDI_CH4U_put_dat_msg_t dat_msg; + MPIR_Win *win; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_PUT_IOV_ACK_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_PUT_IOV_ACK_HANDLER); + + rreq = MPIDI_CH4I_am_win_request_create(); + MPIR_Assert(rreq); + rreq->kind = MPIR_REQUEST_KIND__RMA; + + origin_req = (MPIR_Request *) msg_hdr->origin_preq_ptr; + dat_msg.preq_ptr = msg_hdr->target_preq_ptr; + win = MPIDI_CH4U_REQUEST(origin_req, req->preq.win_ptr); + mpi_errno = MPIDI_NM_send_am_reply(MPIDI_CH4U_win_to_context(win), + MPIDI_CH4U_REQUEST(origin_req, src_rank), + MPIDI_CH4U_PUT_DAT_REQ, + &dat_msg, sizeof(dat_msg), + MPIDI_CH4U_REQUEST(origin_req, req->preq.origin_addr), + MPIDI_CH4U_REQUEST(origin_req, req->preq.origin_count), + MPIDI_CH4U_REQUEST(origin_req, req->preq.origin_datatype), + rreq); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + dtype_release_if_not_builtin(MPIDI_CH4U_REQUEST(origin_req, req->preq.origin_datatype)); + + *cmpl_handler_fn = NULL; + *req = NULL; + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_PUT_IOV_ACK_HANDLER); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_acc_iov_ack_target_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_acc_iov_ack_target_handler(void *am_hdr, + void **data, + size_t * p_data_sz, + int *is_contig, + MPIDI_NM_am_completion_handler_fn * + cmpl_handler_fn, MPIR_Request ** req) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Request *rreq, *origin_req; + MPIDI_CH4U_acc_iov_ack_msg_t *msg_hdr = (MPIDI_CH4U_acc_iov_ack_msg_t *) am_hdr; + MPIDI_CH4U_acc_dat_msg_t dat_msg; + MPIR_Win *win; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_ACC_IOV_ACK_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_ACC_IOV_ACK_HANDLER); + + rreq = MPIDI_CH4I_am_win_request_create(); + MPIR_Assert(rreq); + rreq->kind = MPIR_REQUEST_KIND__RMA; + + origin_req = (MPIR_Request *) msg_hdr->origin_preq_ptr; + dat_msg.preq_ptr = msg_hdr->target_preq_ptr; + win = MPIDI_CH4U_REQUEST(origin_req, req->areq.win_ptr); + mpi_errno = MPIDI_NM_send_am_reply(MPIDI_CH4U_win_to_context(win), + MPIDI_CH4U_REQUEST(origin_req, src_rank), + MPIDI_CH4U_ACC_DAT_REQ, + &dat_msg, sizeof(dat_msg), + MPIDI_CH4U_REQUEST(origin_req, req->areq.origin_addr), + MPIDI_CH4U_REQUEST(origin_req, req->areq.origin_count), + MPIDI_CH4U_REQUEST(origin_req, req->areq.origin_datatype), + rreq); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + dtype_release_if_not_builtin(MPIDI_CH4U_REQUEST(origin_req, req->areq.origin_datatype)); + + *cmpl_handler_fn = NULL; + *req = NULL; + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_ACC_IOV_ACK_HANDLER); + return mpi_errno; + fn_fail: + goto fn_exit; +} + + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_put_data_target_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_put_data_target_handler(void *am_hdr, + void **data, + size_t * p_data_sz, + int *is_contig, + MPIDI_NM_am_completion_handler_fn * + cmpl_handler_fn, MPIR_Request ** req) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Request *rreq; + MPIDI_CH4U_put_dat_msg_t *msg_hdr = (MPIDI_CH4U_put_dat_msg_t *) am_hdr; + MPIR_Win *win; + struct iovec *iov; + uintptr_t base; + int i; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_PUT_DATA_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_PUT_DATA_HANDLER); + + rreq = (MPIR_Request *) msg_hdr->preq_ptr; + win = MPIDI_CH4U_REQUEST(rreq, req->preq.win_ptr); + base = MPIDI_CH4I_win_base_at_target(win); + + /* Adjust the target addresses using the window base address */ + iov = (struct iovec *) MPIDI_CH4U_REQUEST(rreq, req->preq.dt_iov); + for (i = 0; i < MPIDI_CH4U_REQUEST(rreq, req->preq.n_iov); i++) + iov[i].iov_base = (char *) iov[i].iov_base + base; + + *data = MPIDI_CH4U_REQUEST(rreq, req->preq.dt_iov); + *is_contig = 0; + *p_data_sz = MPIDI_CH4U_REQUEST(rreq, req->preq.n_iov); + *req = rreq; + *cmpl_handler_fn = MPIDI_CH4U_put_cmpl_handler; + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_PUT_DATA_HANDLER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_acc_data_target_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_acc_data_target_handler(void *am_hdr, + void **data, + size_t * p_data_sz, + int *is_contig, + MPIDI_NM_am_completion_handler_fn * + cmpl_handler_fn, MPIR_Request ** req) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Request *rreq; + size_t data_sz; + void *p_data = NULL; + MPIDI_CH4U_acc_dat_msg_t *msg_hdr = (MPIDI_CH4U_acc_dat_msg_t *) am_hdr; + uintptr_t base; + int i; + struct iovec *iov; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_ACC_DATA_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_ACC_DATA_HANDLER); + + rreq = (MPIR_Request *) msg_hdr->preq_ptr; + base = (uintptr_t) MPIDI_CH4U_REQUEST(rreq, req->areq.target_addr); + + MPIDI_Datatype_check_size(MPIDI_CH4U_REQUEST(rreq, req->areq.origin_datatype), + MPIDI_CH4U_REQUEST(rreq, req->areq.origin_count), data_sz); + if (data_sz) { + p_data = MPL_malloc(data_sz); + MPIR_Assert(p_data); + } + + MPIDI_CH4U_REQUEST(rreq, req->areq.data) = p_data; + + /* Adjust the target addresses using the window base address */ + iov = (struct iovec *) MPIDI_CH4U_REQUEST(rreq, req->areq.dt_iov); + for (i = 0; i < MPIDI_CH4U_REQUEST(rreq, req->areq.n_iov); i++) + iov[i].iov_base = (char *) iov[i].iov_base + base; + + *data = p_data; + *is_contig = 1; + *p_data_sz = data_sz; + *req = rreq; + *cmpl_handler_fn = MPIDI_CH4U_REQUEST(rreq, req->areq.do_get) ? + MPIDI_CH4U_get_acc_cmpl_handler : MPIDI_CH4U_acc_cmpl_handler; + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_ACC_DATA_HANDLER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_cswap_target_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_cswap_target_handler(void *am_hdr, + void **data, + size_t * p_data_sz, + int *is_contig, + MPIDI_NM_am_completion_handler_fn * + cmpl_handler_fn, MPIR_Request ** req) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Request *rreq = NULL; + size_t data_sz; + MPIR_Win *win; + uintptr_t base; + size_t offset; + + int dt_contig; + void *p_data; + + MPIDI_CH4U_cswap_req_msg_t *msg_hdr = (MPIDI_CH4U_cswap_req_msg_t *) am_hdr; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_CSWAP_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_CSWAP_HANDLER); + + rreq = MPIDI_CH4I_am_win_request_create(); + MPIR_Assert(rreq); + rreq->kind = MPIR_REQUEST_KIND__RMA; + *req = rreq; + + *cmpl_handler_fn = MPIDI_CH4U_cswap_cmpl_handler; + MPIDI_CH4U_REQUEST(rreq, req->seq_no) = OPA_fetch_and_add_int(&MPIDI_CH4_Global.nxt_seq_no, 1); + + MPIDI_Datatype_check_contig_size(msg_hdr->datatype, 1, dt_contig, data_sz); + *is_contig = dt_contig; + + MPL_HASH_FIND(dev.ch4u.hash_handle, MPIDI_CH4_Global.win_hash, + &msg_hdr->win_id, sizeof(uint64_t), win); + MPIR_Assert(win); + + base = MPIDI_CH4I_win_base_at_target(win); + + /* MPIDI_CS_ENTER(); */ + OPA_incr_int(&MPIDI_CH4U_WIN(win, outstanding_ops)); + /* MPIDI_CS_EXIT(); */ + offset = win->disp_unit * msg_hdr->target_disp; + + MPIDI_CH4U_REQUEST(*req, req->creq.win_ptr) = win; + MPIDI_CH4U_REQUEST(*req, req->creq.creq_ptr) = msg_hdr->req_ptr; + MPIDI_CH4U_REQUEST(*req, req->creq.datatype) = msg_hdr->datatype; + MPIDI_CH4U_REQUEST(*req, req->creq.addr) = offset + base; + MPIDI_CH4U_REQUEST(*req, src_rank) = msg_hdr->src_rank; + + MPIR_Assert(dt_contig == 1); + p_data = MPL_malloc(data_sz * 2); + MPIR_Assert(p_data); + + *p_data_sz = data_sz * 2; + *data = p_data; + MPIDI_CH4U_REQUEST(*req, req->creq.data) = p_data; + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_CSWAP_HANDLER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_handle_acc_request +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_handle_acc_request(void *am_hdr, + void **data, + size_t * p_data_sz, + int *is_contig, + MPIDI_NM_am_completion_handler_fn * + cmpl_handler_fn, MPIR_Request ** req) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Request *rreq = NULL; + size_t data_sz; + void *p_data = NULL; + struct iovec *iov, *dt_iov; + MPIR_Win *win; + uintptr_t base; + size_t offset; + int i; + + MPIDI_CH4U_acc_req_msg_t *msg_hdr = (MPIDI_CH4U_acc_req_msg_t *) am_hdr; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_HANDLE_ACC_REQ); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_HANDLE_ACC_REQ); + + rreq = MPIDI_CH4I_am_win_request_create(); + MPIR_Assert(rreq); + rreq->kind = MPIR_REQUEST_KIND__RMA; + *req = rreq; + + MPIDI_Datatype_check_size(msg_hdr->origin_datatype, msg_hdr->origin_count, data_sz); + if (data_sz) { + p_data = MPL_malloc(data_sz); + MPIR_Assert(p_data); + } + + *cmpl_handler_fn = (msg_hdr->do_get) ? MPIDI_CH4U_get_acc_cmpl_handler : + MPIDI_CH4U_acc_cmpl_handler; + MPIDI_CH4U_REQUEST(rreq, req->seq_no) = OPA_fetch_and_add_int(&MPIDI_CH4_Global.nxt_seq_no, 1); + + *is_contig = 1; + *p_data_sz = data_sz; + *data = p_data; + + MPL_HASH_FIND(dev.ch4u.hash_handle, MPIDI_CH4_Global.win_hash, + &msg_hdr->win_id, sizeof(uint64_t), win); + MPIR_Assert(win); + + base = MPIDI_CH4I_win_base_at_target(win); + offset = win->disp_unit * msg_hdr->target_disp; + /* MPIDI_CS_ENTER(); */ + OPA_incr_int(&MPIDI_CH4U_WIN(win, outstanding_ops)); + /* MPIDI_CS_EXIT(); */ + + MPIDI_CH4U_REQUEST(*req, req->areq.win_ptr) = win; + MPIDI_CH4U_REQUEST(*req, req->areq.req_ptr) = msg_hdr->req_ptr; + MPIDI_CH4U_REQUEST(*req, req->areq.origin_datatype) = msg_hdr->origin_datatype; + MPIDI_CH4U_REQUEST(*req, req->areq.target_datatype) = msg_hdr->target_datatype; + MPIDI_CH4U_REQUEST(*req, req->areq.origin_count) = msg_hdr->origin_count; + MPIDI_CH4U_REQUEST(*req, req->areq.target_count) = msg_hdr->target_count; + MPIDI_CH4U_REQUEST(*req, req->areq.target_addr) = (void *) (offset + base); + MPIDI_CH4U_REQUEST(*req, req->areq.op) = msg_hdr->op; + MPIDI_CH4U_REQUEST(*req, req->areq.data) = p_data; + MPIDI_CH4U_REQUEST(*req, req->areq.n_iov) = msg_hdr->n_iov; + MPIDI_CH4U_REQUEST(*req, req->areq.data_sz) = msg_hdr->result_data_sz; + MPIDI_CH4U_REQUEST(*req, src_rank) = msg_hdr->src_rank; + + if (!msg_hdr->n_iov) { + MPIDI_CH4U_REQUEST(rreq, req->areq.dt_iov) = NULL; + goto fn_exit; + } + + dt_iov = (struct iovec *) MPL_malloc(sizeof(struct iovec) * msg_hdr->n_iov); + MPIR_Assert(dt_iov); + + iov = (struct iovec *) ((char *) msg_hdr + sizeof(*msg_hdr)); + for (i = 0; i < msg_hdr->n_iov; i++) { + dt_iov[i].iov_base = (char *) iov[i].iov_base + base + offset; + dt_iov[i].iov_len = iov[i].iov_len; + } + MPIDI_CH4U_REQUEST(rreq, req->areq.dt_iov) = dt_iov; + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_HANDLE_ACC_REQ); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_acc_iov_target_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_acc_iov_target_handler(void *am_hdr, + void **data, + size_t * p_data_sz, + int *is_contig, + MPIDI_NM_am_completion_handler_fn * + cmpl_handler_fn, MPIR_Request ** req) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Request *rreq = NULL; + struct iovec *dt_iov; + MPIR_Win *win; + uintptr_t base; + size_t offset; + + MPIDI_CH4U_acc_req_msg_t *msg_hdr = (MPIDI_CH4U_acc_req_msg_t *) am_hdr; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_HANDLE_ACC_IOV_REQ); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_HANDLE_ACC_IOV_REQ); + + rreq = MPIDI_CH4I_am_win_request_create(); + MPIR_Assert(rreq); + rreq->kind = MPIR_REQUEST_KIND__RMA; + *req = rreq; + + MPL_HASH_FIND(dev.ch4u.hash_handle, MPIDI_CH4_Global.win_hash, + &msg_hdr->win_id, sizeof(uint64_t), win); + MPIR_Assert(win); + + base = MPIDI_CH4I_win_base_at_target(win); + + offset = win->disp_unit * msg_hdr->target_disp; + /* MPIDI_CS_ENTER(); */ + OPA_incr_int(&MPIDI_CH4U_WIN(win, outstanding_ops)); + /* MPIDI_CS_EXIT(); */ + + MPIDI_CH4U_REQUEST(*req, req->areq.win_ptr) = win; + MPIDI_CH4U_REQUEST(*req, req->areq.req_ptr) = msg_hdr->req_ptr; + MPIDI_CH4U_REQUEST(*req, req->areq.origin_datatype) = msg_hdr->origin_datatype; + MPIDI_CH4U_REQUEST(*req, req->areq.target_datatype) = msg_hdr->target_datatype; + MPIDI_CH4U_REQUEST(*req, req->areq.origin_count) = msg_hdr->origin_count; + MPIDI_CH4U_REQUEST(*req, req->areq.target_count) = msg_hdr->target_count; + MPIDI_CH4U_REQUEST(*req, req->areq.target_addr) = (void *) (offset + base); + MPIDI_CH4U_REQUEST(*req, req->areq.op) = msg_hdr->op; + MPIDI_CH4U_REQUEST(*req, req->areq.n_iov) = msg_hdr->n_iov; + MPIDI_CH4U_REQUEST(*req, req->areq.data_sz) = msg_hdr->result_data_sz; + MPIDI_CH4U_REQUEST(*req, req->areq.do_get) = msg_hdr->do_get; + MPIDI_CH4U_REQUEST(*req, src_rank) = msg_hdr->src_rank; + + dt_iov = (struct iovec *) MPL_malloc(sizeof(struct iovec) * msg_hdr->n_iov); + MPIDI_CH4U_REQUEST(rreq, req->areq.dt_iov) = dt_iov; + MPIR_Assert(dt_iov); + + /* Base adjustment for iov will be done after we get the entire iovs, + * at MPIDI_CH4U_acc_data_target_handler */ + *is_contig = 1; + *p_data_sz = sizeof(struct iovec) * msg_hdr->n_iov; + *data = (void *) dt_iov; + + *cmpl_handler_fn = MPIDI_CH4U_acc_iov_cmpl_handler; + MPIDI_CH4U_REQUEST(rreq, req->seq_no) = OPA_fetch_and_add_int(&MPIDI_CH4_Global.nxt_seq_no, 1); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_HANDLE_ACC_IOV_REQ); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_get_target_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_get_target_handler(void *am_hdr, + void **data, + size_t * p_data_sz, + int *is_contig, + MPIDI_NM_am_completion_handler_fn * + cmpl_handler_fn, MPIR_Request ** req) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Request *rreq = NULL; + MPIDI_CH4U_get_req_msg_t *msg_hdr = (MPIDI_CH4U_get_req_msg_t *) am_hdr; + struct iovec *iov; + MPIR_Win *win; + uintptr_t base; + size_t offset; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_GET_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_GET_HANDLER); + + rreq = MPIDI_CH4I_am_win_request_create(); + MPIR_Assert(rreq); + rreq->kind = MPIR_REQUEST_KIND__RMA; + + *req = rreq; + *cmpl_handler_fn = MPIDI_CH4U_get_cmpl_handler; + MPIDI_CH4U_REQUEST(rreq, req->seq_no) = OPA_fetch_and_add_int(&MPIDI_CH4_Global.nxt_seq_no, 1); + + MPL_HASH_FIND(dev.ch4u.hash_handle, MPIDI_CH4_Global.win_hash, + &msg_hdr->win_id, sizeof(uint64_t), win); + MPIR_Assert(win); + + base = MPIDI_CH4I_win_base_at_target(win); + + /* MPIDI_CS_ENTER(); */ + OPA_incr_int(&MPIDI_CH4U_WIN(win, outstanding_ops)); + /* MPIDI_CS_EXIT(); */ + + offset = win->disp_unit * msg_hdr->target_disp; + MPIDI_CH4U_REQUEST(rreq, req->greq.win_ptr) = win; + MPIDI_CH4U_REQUEST(rreq, req->greq.n_iov) = msg_hdr->n_iov; + MPIDI_CH4U_REQUEST(rreq, req->greq.addr) = offset + base; + MPIDI_CH4U_REQUEST(rreq, req->greq.count) = msg_hdr->count; + MPIDI_CH4U_REQUEST(rreq, req->greq.datatype) = msg_hdr->datatype; + MPIDI_CH4U_REQUEST(rreq, req->greq.dt_iov) = NULL; + MPIDI_CH4U_REQUEST(rreq, req->greq.greq_ptr) = msg_hdr->greq_ptr; + MPIDI_CH4U_REQUEST(rreq, src_rank) = msg_hdr->src_rank; + + if (msg_hdr->n_iov) { + iov = (struct iovec *) MPL_malloc(msg_hdr->n_iov * sizeof(*iov)); + MPIR_Assert(iov); + + *data = (void *) iov; + *is_contig = 1; + *p_data_sz = msg_hdr->n_iov * sizeof(*iov); + MPIDI_CH4U_REQUEST(rreq, req->greq.dt_iov) = iov; + } + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_GET_HANDLER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_get_ack_target_handler +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_get_ack_target_handler(void *am_hdr, + void **data, + size_t * p_data_sz, + int *is_contig, + MPIDI_NM_am_completion_handler_fn * + cmpl_handler_fn, MPIR_Request ** req) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Request *rreq = NULL, *greq; + size_t data_sz; + + int dt_contig, n_iov; + MPI_Aint dt_true_lb, last, num_iov; + MPIR_Datatype *dt_ptr; + MPID_Segment *segment_ptr; + + MPIDI_CH4U_get_ack_msg_t *msg_hdr = (MPIDI_CH4U_get_ack_msg_t *) am_hdr; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_GET_ACK_HANDLER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_GET_ACK_HANDLER); + + greq = MPIDI_CH4I_am_win_request_create(); + MPIR_Assert(greq); + greq->kind = MPIR_REQUEST_KIND__RMA; + *req = greq; + + rreq = (MPIR_Request *) msg_hdr->greq_ptr; + MPIR_Assert(rreq->kind == MPIR_REQUEST_KIND__RMA); + MPIDI_CH4U_REQUEST(greq, req->greq.greq_ptr) = (uint64_t) rreq; + + if (MPIDI_CH4U_REQUEST(rreq, req->greq.dt_iov)) { + MPL_free(MPIDI_CH4U_REQUEST(rreq, req->greq.dt_iov)); + } + + *cmpl_handler_fn = MPIDI_CH4U_get_ack_cmpl_handler; + MPIDI_CH4U_REQUEST(greq, req->seq_no) = OPA_fetch_and_add_int(&MPIDI_CH4_Global.nxt_seq_no, 1); + + MPIDI_Datatype_get_info(MPIDI_CH4U_REQUEST(rreq, req->greq.count), + MPIDI_CH4U_REQUEST(rreq, req->greq.datatype), + dt_contig, data_sz, dt_ptr, dt_true_lb); + + *is_contig = dt_contig; + + if (dt_contig) { + *p_data_sz = data_sz; + *data = (char *) (MPIDI_CH4U_REQUEST(rreq, req->greq.addr) + dt_true_lb); + } + else { + segment_ptr = MPIDU_Segment_alloc(); + MPIR_Assert(segment_ptr); + + MPIDU_Segment_init((void *) MPIDI_CH4U_REQUEST(rreq, req->greq.addr), + MPIDI_CH4U_REQUEST(rreq, req->greq.count), + MPIDI_CH4U_REQUEST(rreq, req->greq.datatype), segment_ptr, 0); + last = data_sz; + MPIDU_Segment_count_contig_blocks(segment_ptr, 0, &last, &num_iov); + n_iov = (int) num_iov; + MPIR_Assert(n_iov > 0); + MPIDI_CH4U_REQUEST(rreq, req->iov) = + (struct iovec *) MPL_malloc(n_iov * sizeof(struct iovec)); + MPIR_Assert(MPIDI_CH4U_REQUEST(rreq, req->iov)); + + last = data_sz; + MPIDU_Segment_pack_vector(segment_ptr, 0, &last, MPIDI_CH4U_REQUEST(rreq, req->iov), + &n_iov); + MPIR_Assert(last == (MPI_Aint) data_sz); + *data = MPIDI_CH4U_REQUEST(rreq, req->iov); + *p_data_sz = n_iov; + MPIDI_CH4U_REQUEST(rreq, req->status) |= MPIDI_CH4U_REQ_RCV_NON_CONTIG; + MPL_free(segment_ptr); + } + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_GET_ACK_HANDLER); + return mpi_errno; +} + +#endif /* CH4R_CALLBACKS_H_INCLUDED */ diff --git a/src/mpid/ch4/src/ch4r_init.h b/src/mpid/ch4/src/ch4r_init.h new file mode 100644 index 0000000..124ac49 --- /dev/null +++ b/src/mpid/ch4/src/ch4r_init.h @@ -0,0 +1,333 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef CH4R_INIT_H_INCLUDED +#define CH4R_INIT_H_INCLUDED + +#include "ch4_impl.h" +#include "ch4i_util.h" +#include "ch4r_buf.h" +#include "ch4r_callbacks.h" +#include "mpl_uthash.h" + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_init_comm +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_CH4U_init_comm(MPIR_Comm * comm) +{ + int mpi_errno = MPI_SUCCESS, comm_idx, subcomm_type, is_localcomm; + MPIDI_CH4U_rreq_t **uelist; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_INIT_COMM); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_INIT_COMM); + + /* + * Prevents double initialization of some special communicators. + * + * comm_world and comm_self may exhibit this function twice, first during MPIDI_CH4U_init + * and the second during MPIR_Comm_commit in MPIDI_Init. + * If there is an early arrival of an unexpected message before the second visit, + * the following code will wipe out the unexpected queue andthe message is lost forever. + */ + if (unlikely(MPIDI_CH4_Global.is_ch4u_initialized && + (comm == MPIR_Process.comm_world || comm == MPIR_Process.comm_self))) + goto fn_exit; + + comm_idx = MPIDI_CH4U_get_context_index(comm->recvcontext_id); + subcomm_type = MPIR_CONTEXT_READ_FIELD(SUBCOMM, comm->recvcontext_id); + is_localcomm = MPIR_CONTEXT_READ_FIELD(IS_LOCALCOMM, comm->recvcontext_id); + + MPIR_Assert(subcomm_type <= 3); + MPIR_Assert(is_localcomm <= 1); + MPIDI_CH4_Global.comm_req_lists[comm_idx].comm[is_localcomm][subcomm_type] = comm; + MPIDI_CH4U_COMM(comm, posted_list) = NULL; + MPIDI_CH4U_COMM(comm, unexp_list) = NULL; + + uelist = MPIDI_CH4U_context_id_to_uelist(comm->context_id); + if (*uelist) { + MPIDI_CH4U_rreq_t *curr, *tmp; + MPL_DL_FOREACH_SAFE(*uelist, curr, tmp) { + MPL_DL_DELETE(*uelist, curr); + MPIR_Comm_add_ref(comm); /* +1 for each entry in unexp_list */ + MPL_DL_APPEND(MPIDI_CH4U_COMM(comm, unexp_list), curr); + } + *uelist = NULL; + } + + MPIDI_CH4U_COMM(comm, window_instance) = 0; + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_INIT_COMM); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_destroy_comm +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_CH4U_destroy_comm(MPIR_Comm * comm) +{ + int mpi_errno = MPI_SUCCESS, comm_idx, subcomm_type, is_localcomm; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_DESTROY_COMM); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_DESTROY_COMM); + + comm_idx = MPIDI_CH4U_get_context_index(comm->recvcontext_id); + subcomm_type = MPIR_CONTEXT_READ_FIELD(SUBCOMM, comm->recvcontext_id); + is_localcomm = MPIR_CONTEXT_READ_FIELD(IS_LOCALCOMM, comm->recvcontext_id); + + MPIR_Assert(subcomm_type <= 3); + MPIR_Assert(is_localcomm <= 1); + MPIR_Assert(MPIDI_CH4_Global.comm_req_lists[comm_idx].comm[is_localcomm][subcomm_type] != NULL); + + if (MPIDI_CH4_Global.comm_req_lists[comm_idx].comm[subcomm_type]) { + MPIR_Assert(MPIDI_CH4_Global.comm_req_lists[comm_idx].comm[is_localcomm][subcomm_type]->dev. + ch4.ch4u.posted_list == NULL); + MPIR_Assert(MPIDI_CH4_Global.comm_req_lists[comm_idx].comm[is_localcomm][subcomm_type]->dev. + ch4.ch4u.unexp_list == NULL); + } + MPIDI_CH4_Global.comm_req_lists[comm_idx].comm[is_localcomm][subcomm_type] = NULL; + + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_DESTROY_COMM); + return mpi_errno; +} + + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_init +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_CH4U_init(MPIR_Comm * comm_world, MPIR_Comm * comm_self, + int num_contexts, void **netmod_contexts) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_INIT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_INIT); + + MPIDI_CH4_Global.is_ch4u_initialized = 0; + + MPIDI_CH4_Global.comm_req_lists = (MPIDI_CH4U_comm_req_list_t *) + MPL_calloc(MPIR_MAX_CONTEXT_MASK * MPIR_CONTEXT_INT_BITS, + sizeof(MPIDI_CH4U_comm_req_list_t)); +#ifndef MPIDI_CH4U_USE_PER_COMM_QUEUE + MPIDI_CH4_Global.posted_list = NULL; + MPIDI_CH4_Global.unexp_list = NULL; +#endif + + MPIDI_CH4_Global.cmpl_list = NULL; + OPA_store_int(&MPIDI_CH4_Global.exp_seq_no, 0); + OPA_store_int(&MPIDI_CH4_Global.nxt_seq_no, 0); + + MPIDI_CH4_Global.buf_pool = MPIDI_CH4U_create_buf_pool(MPIDI_CH4I_BUF_POOL_NUM, + MPIDI_CH4I_BUF_POOL_SZ); + MPIR_Assert(MPIDI_CH4_Global.buf_pool); + + mpi_errno = MPIDI_NM_reg_hdr_handler(MPIDI_CH4U_SEND, + &MPIDI_CH4U_send_origin_cmpl_handler, + &MPIDI_CH4U_send_target_handler); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + mpi_errno = MPIDI_NM_reg_hdr_handler(MPIDI_CH4U_SEND_LONG_REQ, NULL /* Injection only */ , + &MPIDI_CH4U_send_long_req_target_handler); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + mpi_errno = MPIDI_NM_reg_hdr_handler(MPIDI_CH4U_SEND_LONG_ACK, NULL /* Injection only */ , + &MPIDI_CH4U_send_long_ack_target_handler); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + mpi_errno = MPIDI_NM_reg_hdr_handler(MPIDI_CH4U_SEND_LONG_LMT, + &MPIDI_CH4U_send_long_lmt_origin_cmpl_handler, + &MPIDI_CH4U_send_long_lmt_target_handler); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + mpi_errno = MPIDI_NM_reg_hdr_handler(MPIDI_CH4U_SSEND_REQ, + &MPIDI_CH4U_send_origin_cmpl_handler, + &MPIDI_CH4U_ssend_target_handler); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + mpi_errno = MPIDI_NM_reg_hdr_handler(MPIDI_CH4U_SSEND_ACK, + &MPIDI_CH4U_ssend_ack_origin_cmpl_handler, + &MPIDI_CH4U_ssend_ack_target_handler); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + mpi_errno = MPIDI_NM_reg_hdr_handler(MPIDI_CH4U_PUT_REQ, + &MPIDI_CH4U_put_origin_cmpl_handler, + &MPIDI_CH4U_put_target_handler); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + mpi_errno = MPIDI_NM_reg_hdr_handler(MPIDI_CH4U_PUT_ACK, + NULL, &MPIDI_CH4U_put_ack_target_handler); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + mpi_errno = MPIDI_NM_reg_hdr_handler(MPIDI_CH4U_GET_REQ, + &MPIDI_CH4U_get_origin_cmpl_handler, + &MPIDI_CH4U_get_target_handler); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + mpi_errno = MPIDI_NM_reg_hdr_handler(MPIDI_CH4U_GET_ACK, + &MPIDI_CH4U_get_ack_origin_cmpl_handler, + &MPIDI_CH4U_get_ack_target_handler); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + mpi_errno = MPIDI_NM_reg_hdr_handler(MPIDI_CH4U_CSWAP_REQ, + &MPIDI_CH4U_cswap_origin_cmpl_handler, + &MPIDI_CH4U_cswap_target_handler); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + mpi_errno = MPIDI_NM_reg_hdr_handler(MPIDI_CH4U_CSWAP_ACK, + &MPIDI_CH4U_cswap_ack_origin_cmpl_handler, + &MPIDI_CH4U_cswap_ack_target_handler); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + mpi_errno = MPIDI_NM_reg_hdr_handler(MPIDI_CH4U_ACC_REQ, + &MPIDI_CH4U_acc_origin_cmpl_handler, + &MPIDI_CH4U_handle_acc_request); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + mpi_errno = MPIDI_NM_reg_hdr_handler(MPIDI_CH4U_ACC_ACK, + NULL, &MPIDI_CH4U_acc_ack_target_handler); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + mpi_errno = MPIDI_NM_reg_hdr_handler(MPIDI_CH4U_GET_ACC_ACK, + &MPIDI_CH4U_get_acc_ack_origin_cmpl_handler, + &MPIDI_CH4U_get_acc_ack_target_handler); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + mpi_errno = MPIDI_NM_reg_hdr_handler(MPIDI_CH4U_WIN_CTRL, + NULL, &MPIDI_CH4U_win_ctrl_target_handler); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + + mpi_errno = MPIDI_NM_reg_hdr_handler(MPIDI_CH4U_PUT_IOV_REQ, + &MPIDI_CH4U_put_iov_origin_cmpl_handler, + &MPIDI_CH4U_put_iov_target_handler); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + mpi_errno = MPIDI_NM_reg_hdr_handler(MPIDI_CH4U_PUT_IOV_ACK, + NULL, &MPIDI_CH4U_put_iov_ack_target_handler); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + mpi_errno = MPIDI_NM_reg_hdr_handler(MPIDI_CH4U_PUT_DAT_REQ, + &MPIDI_CH4U_put_data_origin_cmpl_handler, + &MPIDI_CH4U_put_data_target_handler); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + mpi_errno = MPIDI_NM_reg_hdr_handler(MPIDI_CH4U_ACC_IOV_REQ, + &MPIDI_CH4U_acc_iov_origin_cmpl_handler, + &MPIDI_CH4U_acc_iov_target_handler); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + mpi_errno = MPIDI_NM_reg_hdr_handler(MPIDI_CH4U_ACC_IOV_ACK, + NULL, &MPIDI_CH4U_acc_iov_ack_target_handler); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + mpi_errno = MPIDI_NM_reg_hdr_handler(MPIDI_CH4U_ACC_DAT_REQ, + &MPIDI_CH4U_acc_data_origin_cmpl_handler, + &MPIDI_CH4U_acc_data_target_handler); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + mpi_errno = MPIDI_CH4U_init_comm(comm_world); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + mpi_errno = MPIDI_CH4U_init_comm(comm_self); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + MPIDI_CH4_Global.win_hash = NULL; + + MPIDI_CH4_Global.is_ch4u_initialized = 1; + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_INIT); + + fn_exit: + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_init +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ void MPIDI_CH4U_finalize() +{ + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_FINALIZE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_FINALIZE); + MPIDI_CH4_Global.is_ch4u_initialized = 0; + MPL_HASH_CLEAR(dev.ch4u.hash_handle, MPIDI_CH4_Global.win_hash); + MPIDI_CH4R_destroy_buf_pool(MPIDI_CH4_Global.buf_pool); + MPL_free(MPIDI_CH4_Global.comm_req_lists); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_FINALIZE); +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_alloc_mem +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ void *MPIDI_CH4U_alloc_mem(size_t size, MPIR_Info * info_ptr) +{ + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_ALLOC_MEM); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_ALLOC_MEM); + void *p; + p = MPL_malloc(size); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_ALLOC_MEM); + return p; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_free_mem +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_CH4U_free_mem(void *ptr) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_FREE_MEM); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_FREE_MEM); + MPL_free(ptr); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_FREE_MEM); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDIU_update_node_map +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDIU_update_node_map(int avtid, int size, MPID_Node_id_t node_map[]) +{ + int i; + for (i = 0; i < size; i++) { + MPIDI_CH4_Global.node_map[avtid][i] = node_map[i]; + } + return MPI_SUCCESS; +} + +#endif /* CH4R_INIT_H_INCLUDED */ diff --git a/src/mpid/ch4/src/ch4r_probe.h b/src/mpid/ch4/src/ch4r_probe.h new file mode 100644 index 0000000..20753db --- /dev/null +++ b/src/mpid/ch4/src/ch4r_probe.h @@ -0,0 +1,176 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef CH4R_PROBE_H_INCLUDED +#define CH4R_PROBE_H_INCLUDED + +#include "ch4_impl.h" + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_iprobe +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_CH4U_iprobe(int source, + int tag, + MPIR_Comm * comm, + int context_offset, int *flag, MPI_Status * status) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Comm *root_comm; + MPIR_Request *unexp_req; + uint64_t match_bits, mask_bits; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_IPROBE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_IPROBE); + + if (unlikely(source == MPI_PROC_NULL)) { + MPIR_Status_set_procnull(status); + *flag = true; + goto fn_exit; + } + + root_comm = MPIDI_CH4U_context_id_to_comm(comm->context_id); + match_bits = MPIDI_CH4U_init_recvtag(&mask_bits, root_comm->recvcontext_id + + context_offset, source, tag); + + /* MPIDI_CS_ENTER(); */ + unexp_req = MPIDI_CH4U_find_unexp(match_bits, mask_bits, + &MPIDI_CH4U_COMM(root_comm, unexp_list)); + + if (unexp_req) { + *flag = 1; + unexp_req->status.MPI_ERROR = MPI_SUCCESS; + unexp_req->status.MPI_SOURCE = MPIDI_CH4U_REQUEST(unexp_req, src_rank); + unexp_req->status.MPI_TAG = MPIDI_CH4U_get_tag(MPIDI_CH4U_REQUEST(unexp_req, tag)); + MPIR_STATUS_SET_COUNT(unexp_req->status, MPIDI_CH4U_REQUEST(unexp_req, count)); + + status->MPI_TAG = unexp_req->status.MPI_TAG; + status->MPI_SOURCE = unexp_req->status.MPI_SOURCE; + MPIR_STATUS_SET_COUNT(*status, MPIDI_CH4U_REQUEST(unexp_req, count)); + } + else { + *flag = 0; + MPIDI_Progress_test(); + } + /* MPIDI_CS_EXIT(); */ + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_IPROBE); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_Probe +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_CH4U_probe(int source, + int tag, + MPIR_Comm * comm, int context_offset, MPI_Status * status) +{ + int mpi_errno, flag = 0; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_PROBE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_PROBE); + + while (!flag) { + mpi_errno = MPIDI_CH4U_iprobe(source, tag, comm, context_offset, &flag, status); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + } + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_PROBE); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_improbe +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_CH4U_improbe(int source, + int tag, + MPIR_Comm * comm, + int context_offset, + int *flag, MPIR_Request ** message, MPI_Status * status) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Comm *root_comm; + MPIR_Request *unexp_req; + uint64_t match_bits, mask_bits; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_IMPROBE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_IMPROBE); + + if (unlikely(source == MPI_PROC_NULL)) { + MPIR_Status_set_procnull(status); + *flag = true; + goto fn_exit; + } + + root_comm = MPIDI_CH4U_context_id_to_comm(comm->context_id); + match_bits = MPIDI_CH4U_init_recvtag(&mask_bits, root_comm->recvcontext_id + + context_offset, source, tag); + + /* MPIDI_CS_ENTER(); */ + unexp_req = MPIDI_CH4U_dequeue_unexp(match_bits, mask_bits, + &MPIDI_CH4U_COMM(root_comm, unexp_list)); + + if (unexp_req) { + *flag = 1; + *message = unexp_req; + + (*message)->kind = MPIR_REQUEST_KIND__MPROBE; + (*message)->comm = comm; + /* Notes on refcounting comm: + * We intentionally do nothing here because what we are supposed to do here + * is -1 for dequeue(unexp_list) and +1 for (*message)->comm */ + + unexp_req->status.MPI_ERROR = MPI_SUCCESS; + unexp_req->status.MPI_SOURCE = MPIDI_CH4U_REQUEST(unexp_req, src_rank); + unexp_req->status.MPI_TAG = MPIDI_CH4U_get_tag(MPIDI_CH4U_REQUEST(unexp_req, tag)); + MPIR_STATUS_SET_COUNT(unexp_req->status, MPIDI_CH4U_REQUEST(unexp_req, count)); + MPIDI_CH4U_REQUEST(unexp_req, req->status) |= MPIDI_CH4U_REQ_UNEXP_DQUED; + + status->MPI_TAG = unexp_req->status.MPI_TAG; + status->MPI_SOURCE = unexp_req->status.MPI_SOURCE; + MPIR_STATUS_SET_COUNT(*status, MPIDI_CH4U_REQUEST(unexp_req, count)); + } + else { + *flag = 0; + MPIDI_Progress_test(); + } + /* MPIDI_CS_EXIT(); */ + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_IMPROBE); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_mprobe +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_CH4U_mprobe(int source, + int tag, + MPIR_Comm * comm, + int context_offset, + MPIR_Request ** message, MPI_Status * status) +{ + int mpi_errno, flag = 0; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_MPROBE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_MPROBE); + while (!flag) { + mpi_errno = MPIDI_CH4U_improbe(source, tag, comm, context_offset, &flag, message, status); + } + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_MPROBE); + return mpi_errno; +} + +#endif /* CH4R_WIN_H_INCLUDED */ diff --git a/src/mpid/ch4/src/ch4r_proc.h b/src/mpid/ch4/src/ch4r_proc.h new file mode 100644 index 0000000..65c75bf --- /dev/null +++ b/src/mpid/ch4/src/ch4r_proc.h @@ -0,0 +1,411 @@ +/* -*- Mode: C; c-basic-offset:4 ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef CH4R_PROC_H_INCLUDED +#define CH4R_PROC_H_INCLUDED + +#include "ch4_types.h" +#include "build_nodemap.h" + +#undef FUNCNAME +#define FUNCNAME MPIDIU_comm_rank_to_pid +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDIU_comm_rank_to_pid(MPIR_Comm * comm, int rank, int *index, int *avtid) +{ + switch (MPIDII_COMM(comm, map).mode) { + case MPIDII_RANK_MAP_DIRECT: + *avtid = MPIDII_COMM(comm, map).avtid; + *index = rank; + break; + case MPIDII_RANK_MAP_DIRECT_INTRA: + *index = rank; + break; + case MPIDII_RANK_MAP_OFFSET: + *avtid = MPIDII_COMM(comm, map).avtid; + *index = rank + MPIDII_COMM(comm, map).reg.offset; + break; + case MPIDII_RANK_MAP_OFFSET_INTRA: + *index = rank + MPIDII_COMM(comm, map).reg.offset; + break; + case MPIDII_RANK_MAP_STRIDE: + *avtid = MPIDII_COMM(comm, map).avtid; + *index = MPIDII_CALC_STRIDE_SIMPLE(rank, MPIDII_COMM(comm, map).reg.stride.stride, + MPIDII_COMM(comm, map).reg.stride.offset); + break; + case MPIDII_RANK_MAP_STRIDE_INTRA: + *index = MPIDII_CALC_STRIDE_SIMPLE(rank, MPIDII_COMM(comm, map).reg.stride.stride, + MPIDII_COMM(comm, map).reg.stride.offset); + break; + case MPIDII_RANK_MAP_STRIDE_BLOCK: + *avtid = MPIDII_COMM(comm, map).avtid; + *index = MPIDII_CALC_STRIDE(rank, MPIDII_COMM(comm, map).reg.stride.stride, + MPIDII_COMM(comm, map).reg.stride.blocksize, + MPIDII_COMM(comm, map).reg.stride.offset); + break; + case MPIDII_RANK_MAP_STRIDE_BLOCK_INTRA: + *index = MPIDII_CALC_STRIDE(rank, MPIDII_COMM(comm, map).reg.stride.stride, + MPIDII_COMM(comm, map).reg.stride.blocksize, + MPIDII_COMM(comm, map).reg.stride.offset); + break; + case MPIDII_RANK_MAP_LUT: + *avtid = MPIDII_COMM(comm, map).avtid; + *index = MPIDII_COMM(comm, map).irreg.lut.lpid[rank]; + break; + case MPIDII_RANK_MAP_LUT_INTRA: + *index = MPIDII_COMM(comm, map).irreg.lut.lpid[rank]; + break; + case MPIDII_RANK_MAP_MLUT: + *index = MPIDII_COMM(comm, map).irreg.mlut.gpid[rank].lpid; + *avtid = MPIDII_COMM(comm, map).irreg.mlut.gpid[rank].avtid; + break; + case MPIDII_RANK_MAP_NONE: + MPIR_Assert(0); + break; + } + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, + (MPL_DBG_FDEST, " rank=%d, index=%d", rank, *index)); + return *index; +} + +#undef FUNCNAME +#define FUNCNAME MPIDIU_comm_rank_to_av +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline MPIDII_av_entry_t *MPIDIU_comm_rank_to_av(MPIR_Comm * comm, int rank) +{ + switch (MPIDII_COMM(comm, map).mode) { + case MPIDII_RANK_MAP_DIRECT: + return &MPIDII_av_table[MPIDII_COMM(comm, map).avtid]->table[rank]; + case MPIDII_RANK_MAP_DIRECT_INTRA: + return &MPIDII_av_table0->table[rank]; + case MPIDII_RANK_MAP_OFFSET: + return &MPIDII_av_table[MPIDII_COMM(comm, map).avtid] + ->table[rank + MPIDII_COMM(comm, map).reg.offset]; + case MPIDII_RANK_MAP_OFFSET_INTRA: + return &MPIDII_av_table0->table[rank + MPIDII_COMM(comm, map).reg.offset]; + case MPIDII_RANK_MAP_STRIDE: + return &MPIDII_av_table[MPIDII_COMM(comm, map).avtid] + ->table[MPIDII_CALC_STRIDE_SIMPLE(rank, + MPIDII_COMM(comm, map).reg.stride.stride, + MPIDII_COMM(comm, map).reg.stride.offset)]; + case MPIDII_RANK_MAP_STRIDE_INTRA: + return &MPIDII_av_table0->table[MPIDII_CALC_STRIDE_SIMPLE(rank, + MPIDII_COMM(comm, + map).reg.stride. + stride, MPIDII_COMM(comm, + map).reg. + stride.offset)]; + case MPIDII_RANK_MAP_STRIDE_BLOCK: + return &MPIDII_av_table[MPIDII_COMM(comm, map).avtid] + ->table[MPIDII_CALC_STRIDE(rank, + MPIDII_COMM(comm, map).reg.stride.stride, + MPIDII_COMM(comm, map).reg.stride.blocksize, + MPIDII_COMM(comm, map).reg.stride.offset)]; + case MPIDII_RANK_MAP_STRIDE_BLOCK_INTRA: + return &MPIDII_av_table0->table[MPIDII_CALC_STRIDE(rank, + MPIDII_COMM(comm, map).reg.stride.stride, + MPIDII_COMM(comm, + map).reg.stride.blocksize, + MPIDII_COMM(comm, + map).reg.stride.offset)]; + case MPIDII_RANK_MAP_LUT: + return &MPIDII_av_table[MPIDII_COMM(comm, map).avtid] + ->table[MPIDII_COMM(comm, map).irreg.lut.lpid[rank]]; + case MPIDII_RANK_MAP_LUT_INTRA: + return &MPIDII_av_table0->table[MPIDII_COMM(comm, map).irreg.lut.lpid[rank]]; + case MPIDII_RANK_MAP_MLUT: + return &MPIDII_av_table[MPIDII_COMM(comm, map).irreg.mlut.gpid[rank].avtid] + ->table[MPIDII_COMM(comm, map).irreg.mlut.gpid[rank].lpid]; + case MPIDII_RANK_MAP_NONE: + MPIR_Assert(0); + return NULL; + } + return NULL; +} + +#undef FUNCNAME +#define FUNCNAME MPIDIU_comm_rank_to_pid_local +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDIU_comm_rank_to_pid_local(MPIR_Comm * comm, int rank, int *index, int *avtid) +{ + *avtid = MPIDII_COMM(comm, local_map).avtid; + switch (MPIDII_COMM(comm, local_map).mode) { + case MPIDII_RANK_MAP_DIRECT: + case MPIDII_RANK_MAP_DIRECT_INTRA: + *index = rank; + break; + case MPIDII_RANK_MAP_OFFSET: + case MPIDII_RANK_MAP_OFFSET_INTRA: + *index = rank + MPIDII_COMM(comm, local_map).reg.offset; + break; + case MPIDII_RANK_MAP_STRIDE: + case MPIDII_RANK_MAP_STRIDE_INTRA: + *index = MPIDII_CALC_STRIDE_SIMPLE(rank, MPIDII_COMM(comm, map).reg.stride.stride, + MPIDII_COMM(comm, map).reg.stride.offset); + break; + case MPIDII_RANK_MAP_STRIDE_BLOCK: + case MPIDII_RANK_MAP_STRIDE_BLOCK_INTRA: + *index = MPIDII_CALC_STRIDE(rank, MPIDII_COMM(comm, local_map).reg.stride.stride, + MPIDII_COMM(comm, local_map).reg.stride.blocksize, + MPIDII_COMM(comm, local_map).reg.stride.offset); + break; + case MPIDII_RANK_MAP_LUT: + case MPIDII_RANK_MAP_LUT_INTRA: + *index = MPIDII_COMM(comm, local_map).irreg.lut.lpid[rank]; + break; + case MPIDII_RANK_MAP_MLUT: + *index = MPIDII_COMM(comm, local_map).irreg.mlut.gpid[rank].lpid; + *avtid = MPIDII_COMM(comm, local_map).irreg.mlut.gpid[rank].avtid; + break; + case MPIDII_RANK_MAP_NONE: + MPIR_Assert(0); + break; + } + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, + (MPL_DBG_FDEST, " rank: rank=%d, index=%d", rank, *index)); + return *index; +} + +static inline int MPIDI_CH4U_rank_is_local(int rank, MPIR_Comm * comm) +{ + int ret; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDIU_RANK_IS_LOCAL); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDIU_RANK_IS_LOCAL); + +#ifdef MPIDI_BUILD_CH4_LOCALITY_INFO + if (comm->comm_kind == MPIR_COMM_KIND__INTERCOMM) { + ret = 0; + goto fn_exit; + } + ret = MPIDIU_comm_rank_to_av(comm, rank)->is_local; + MPL_DBG_MSG_FMT(MPIDI_CH4_DBG_MAP, VERBOSE, + (MPL_DBG_FDEST, " is_local=%d, rank=%d", ret, rank)); +#endif + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDIU_RANK_IS_LOCAL); + return ret; +} + + +static inline int MPIDI_CH4U_rank_to_lpid(int rank, MPIR_Comm * comm) +{ + int ret; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDIU_RANK_TO_LPID); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDIU_RANK_TO_LPID); + + int avtid = 0, lpid = 0; + MPIDIU_comm_rank_to_pid(comm, rank, &lpid, &avtid); + if (avtid == 0) { + ret = lpid; + } + else { + ret = -1; + } + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDIU_RANK_TO_LPID); + return ret; +} + +static inline int MPIDI_CH4U_get_node_id(MPIR_Comm * comm, int rank, MPID_Node_id_t * id_p) +{ + int mpi_errno = MPI_SUCCESS; + int avtid = 0, lpid = 0; + MPIDIU_comm_rank_to_pid(comm, rank, &lpid, &avtid); + *id_p = MPIDI_CH4_Global.node_map[avtid][lpid]; + + return mpi_errno; +} + +static inline int MPIDI_CH4U_get_max_node_id(MPIR_Comm * comm, MPID_Node_id_t * max_id_p) +{ + int mpi_errno = MPI_SUCCESS; + *max_id_p = MPIDI_CH4_Global.max_node_id; + + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_build_nodemap +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4U_build_nodemap(int myrank, + MPIR_Comm * comm, + int sz, + MPID_Node_id_t * out_nodemap, MPID_Node_id_t * sz_out) +{ + return MPIR_NODEMAP_build_nodemap(sz, myrank, out_nodemap, sz_out); +} + +static inline int MPIDIU_get_n_avts() +{ + return MPIDI_CH4_Global.avt_mgr.n_avts; +} + +static inline int MPIDIU_get_max_n_avts() +{ + return MPIDI_CH4_Global.avt_mgr.max_n_avts; +} + +static inline int MPIDIU_get_avt_size(int avtid) +{ + return MPIDII_av_table[avtid]->size; +} + +#undef FUNCNAME +#define FUNCNAME MPIDIU_alloc_globals_for_avtid +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDIU_alloc_globals_for_avtid(int avtid) +{ + int max_n_avts; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATER_ALLOC_GLOBALS_FOR_AVTID); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATER_ALLOC_GLOBALS_FOR_AVTID); + max_n_avts = MPIDIU_get_max_n_avts(); + if (max_n_avts > MPIDI_CH4_Global.allocated_max_n_avts) { + MPIDI_CH4_Global.node_map = (MPID_Node_id_t **) MPL_realloc(MPIDI_CH4_Global.node_map, + max_n_avts * + sizeof(MPID_Node_id_t *)); + } + + MPIDI_CH4_Global.node_map[avtid] = + (MPID_Node_id_t *) MPL_malloc(MPIDII_av_table[avtid]->size * sizeof(MPID_Node_id_t)); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATER_ALLOC_GLOBALS_FOR_AVTID); + return MPI_SUCCESS; +} + +#undef FUNCNAME +#define FUNCNAME MPIDIU_free_globals_for_avtid +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDIU_free_globals_for_avtid(int avtid) +{ + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATER_FREE_GLOBALS_FOR_AVTID); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATER_FREE_GLOBALS_FOR_AVTID); + MPL_free(MPIDI_CH4_Global.node_map[avtid]); + MPIDI_CH4_Global.node_map[avtid] = NULL; + MPIR_FUNC_VERBOSE_EXIT(MPID_STATER_FREE_GLOBALS_FOR_AVTID); + return MPI_SUCCESS; +} + +static inline int MPIDIU_get_next_avtid(int *avtid) +{ + if (MPIDI_CH4_Global.avt_mgr.next_avtid == -1) { /* out of free avtids */ + int old_max, new_max, i; + old_max = MPIDI_CH4_Global.avt_mgr.max_n_avts; + new_max = old_max + 1; + MPIDI_CH4_Global.avt_mgr.free_avtid = + (int *) MPL_realloc(MPIDI_CH4_Global.avt_mgr.free_avtid, new_max * sizeof(int)); + for (i = old_max; i < new_max - 1; i++) { + MPIDI_CH4_Global.avt_mgr.free_avtid[i] = i + 1; + } + MPIDI_CH4_Global.avt_mgr.free_avtid[new_max - 1] = -1; + MPIDI_CH4_Global.avt_mgr.max_n_avts = new_max; + MPIDI_CH4_Global.avt_mgr.next_avtid = old_max; + } + + *avtid = MPIDI_CH4_Global.avt_mgr.next_avtid; + MPIDI_CH4_Global.avt_mgr.next_avtid = MPIDI_CH4_Global.avt_mgr.free_avtid[*avtid]; + MPIDI_CH4_Global.avt_mgr.free_avtid[*avtid] = -1; + MPIDI_CH4_Global.avt_mgr.n_avts++; + MPIR_Assert(MPIDI_CH4_Global.avt_mgr.n_avts <= MPIDI_CH4_Global.avt_mgr.max_n_avts); + return *avtid; +} + +static inline int MPIDIU_free_avtid(int avtid) +{ + MPIR_Assert(MPIDI_CH4_Global.avt_mgr.n_avts > 0); + MPIDI_CH4_Global.avt_mgr.free_avtid[avtid] = MPIDI_CH4_Global.avt_mgr.next_avtid; + MPIDI_CH4_Global.avt_mgr.next_avtid = avtid; + MPIDI_CH4_Global.avt_mgr.n_avts--; + return 0; +} + +static inline int MPIDIU_new_avt(int size, int *avtid) +{ + int mpi_errno = MPI_SUCCESS; + int max_n_avts; + MPIDII_av_table_t *new_av_table; + MPIDIU_get_next_avtid(avtid); + + new_av_table = (MPIDII_av_table_t *) MPL_malloc(size * sizeof(MPIDII_av_entry_t) + + sizeof(MPIDII_av_table_t)); + max_n_avts = MPIDIU_get_max_n_avts(); + MPIDII_av_table = (MPIDII_av_table_t **) MPL_realloc(MPIDII_av_table, + max_n_avts * sizeof(MPIDII_av_table_t *)); + new_av_table->size = size; + MPIDII_av_table[*avtid] = new_av_table; + + MPIR_Object_set_ref(MPIDII_av_table[*avtid], 0); + + MPIDIU_alloc_globals_for_avtid(*avtid); + return mpi_errno; +} + +static inline int MPIDIU_free_avt(int avtid) +{ + int mpi_errno = MPI_SUCCESS; + MPIDIU_free_globals_for_avtid(avtid); + MPL_free(MPIDII_av_table[avtid]); + MPIDII_av_table[avtid] = NULL; + MPIDIU_free_avtid(avtid); + return mpi_errno; +} + +static inline int MPIDIU_avt_add_ref(int avtid) +{ + MPIR_Object_add_ref(MPIDII_av_table[avtid]); + return MPI_SUCCESS; +} + +static inline int MPIDIU_avt_release_ref(int avtid) +{ + int count; + MPIR_Object_release_ref(MPIDIU_get_av_table(avtid), &count); + if (count == 0) { + MPIDIU_free_avt(avtid); + MPIDIU_free_globals_for_avtid(avtid); + } + return MPI_SUCCESS; +} + +static inline int MPIDIU_avt_init() +{ + int i; + MPIDI_CH4_Global.avt_mgr.max_n_avts = 1; + MPIDI_CH4_Global.avt_mgr.next_avtid = 0; + MPIDI_CH4_Global.avt_mgr.n_avts = 0; + MPIDI_CH4_Global.avt_mgr.free_avtid = + (int *) MPL_malloc(MPIDI_CH4_Global.avt_mgr.max_n_avts * sizeof(int)); + + for (i = 0; i < MPIDI_CH4_Global.avt_mgr.max_n_avts - 1; i++) { + MPIDI_CH4_Global.avt_mgr.free_avtid[i] = i + 1; + } + MPIDI_CH4_Global.avt_mgr.free_avtid[MPIDI_CH4_Global.avt_mgr.max_n_avts - 1] = -1; + + return MPI_SUCCESS; +} + +static inline int MPIDIU_avt_destroy() +{ + MPL_free(MPIDI_CH4_Global.avt_mgr.free_avtid); + return MPI_SUCCESS; +} + +static inline int MPIDIU_build_nodemap_avtid(int myrank, MPIR_Comm * comm, int sz, int avtid) +{ + return MPIDI_CH4U_build_nodemap(myrank, comm, sz, + MPIDI_CH4_Global.node_map[avtid], + &MPIDI_CH4_Global.max_node_id); +} + +#endif /* CH4R_PROC_H_INCLUDED */ diff --git a/src/mpid/ch4/src/ch4r_recv.h b/src/mpid/ch4/src/ch4r_recv.h new file mode 100644 index 0000000..c77040c --- /dev/null +++ b/src/mpid/ch4/src/ch4r_recv.h @@ -0,0 +1,429 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef CH4R_RECV_H_INCLUDED +#define CH4R_RECV_H_INCLUDED + +#include "ch4_impl.h" +#include "ch4r_proc.h" + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4I_prepare_recv_req +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4I_prepare_recv_req(void *buf, int count, MPI_Datatype datatype, + MPIR_Request * rreq) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_PREPARE_RECV_BUFFER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_PREPARE_RECV_BUFFER); + + MPIDI_CH4U_REQUEST(rreq, datatype) = datatype; + MPIDI_CH4U_REQUEST(rreq, buffer) = (char *) buf; + MPIDI_CH4U_REQUEST(rreq, count) = count; + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_PREPARE_RECV_BUFFER); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4I_handle_unexpected +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4I_handle_unexpected(void *buf, + int count, + MPI_Datatype datatype, + MPIR_Comm * comm, + int context_offset, MPIR_Request * rreq) +{ + int mpi_errno = MPI_SUCCESS; + int dt_contig; + MPI_Aint dt_true_lb, last; + MPIR_Datatype *dt_ptr; + size_t in_data_sz, dt_sz, nbytes; + MPID_Segment *segment_ptr; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_HANDLE_UNEXPECTED); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_HANDLE_UNEXPECTED); + + in_data_sz = MPIDI_CH4U_REQUEST(rreq, count); + MPID_Datatype_get_size_macro(datatype, dt_sz); + + if (in_data_sz > dt_sz * count) { + rreq->status.MPI_ERROR = MPI_ERR_TRUNCATE; + nbytes = dt_sz * count; + } + else { + rreq->status.MPI_ERROR = MPI_SUCCESS; + nbytes = in_data_sz; + count = dt_sz ? nbytes / dt_sz : 0; + } + MPIR_STATUS_SET_COUNT(rreq->status, nbytes); + MPIDI_CH4U_REQUEST(rreq, datatype) = datatype; + MPIDI_CH4U_REQUEST(rreq, count) = nbytes; + + MPIDI_Datatype_get_info(count, datatype, dt_contig, dt_sz, dt_ptr, dt_true_lb); + + if (!dt_contig) { + segment_ptr = MPID_Segment_alloc(); + MPIR_ERR_CHKANDJUMP1(segment_ptr == NULL, mpi_errno, + MPI_ERR_OTHER, "**nomem", "**nomem %s", "Recv MPID_Segment_alloc"); + MPID_Segment_init(buf, count, datatype, segment_ptr, 0); + + last = nbytes; + MPID_Segment_unpack(segment_ptr, 0, &last, MPIDI_CH4U_REQUEST(rreq, buffer)); + MPID_Segment_free(segment_ptr); + if (last != (MPI_Aint) (nbytes)) { + mpi_errno = MPIR_Err_create_code(MPI_SUCCESS, MPIR_ERR_RECOVERABLE, + __FUNCTION__, __LINE__, + MPI_ERR_TYPE, "**dtypemismatch", 0); + rreq->status.MPI_ERROR = mpi_errno; + } + } + else { + MPIR_Memcpy((char *) buf + dt_true_lb, MPIDI_CH4U_REQUEST(rreq, buffer), nbytes); + } + + MPIDI_CH4U_REQUEST(rreq, req->status) &= ~MPIDI_CH4U_REQ_UNEXPECTED; + MPL_free(MPIDI_CH4U_REQUEST(rreq, buffer)); + + rreq->status.MPI_SOURCE = MPIDI_CH4U_REQUEST(rreq, src_rank); + rreq->status.MPI_TAG = MPIDI_CH4U_get_tag(MPIDI_CH4U_REQUEST(rreq, tag)); + + if (MPIDI_CH4U_REQUEST(rreq, req->status) & MPIDI_CH4U_REQ_PEER_SSEND) { + mpi_errno = MPIDI_CH4U_reply_ssend(rreq); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + } + MPIDI_CH4I_am_request_complete(rreq); + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_HANDLE_UNEXPECTED); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4I_do_irecv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4I_do_irecv(void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, + int context_offset, + MPIR_Request ** request, int alloc_req, uint64_t flags) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Request *rreq = NULL, *unexp_req = NULL; + uint64_t match_bits, mask_bits; + MPIR_Context_id_t context_id = comm->recvcontext_id + context_offset; + MPIR_Comm *root_comm; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_DO_IRECV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_DO_IRECV); + + match_bits = MPIDI_CH4U_init_recvtag(&mask_bits, context_id, rank, tag); + root_comm = MPIDI_CH4U_context_id_to_comm(comm->recvcontext_id); + unexp_req = MPIDI_CH4U_dequeue_unexp(match_bits, mask_bits, + &MPIDI_CH4U_COMM(root_comm, unexp_list)); + + if (unexp_req) { + MPIR_Comm_release(root_comm); /* -1 for removing from unexp_list */ + if (MPIDI_CH4U_REQUEST(unexp_req, req->status) & MPIDI_CH4U_REQ_BUSY) { + MPIDI_CH4U_REQUEST(unexp_req, req->status) |= MPIDI_CH4U_REQ_MATCHED; + } + else if (MPIDI_CH4U_REQUEST(unexp_req, req->status) & MPIDI_CH4U_REQ_LONG_RTS) { + /* Matching receive is now posted, tell the netmod */ + dtype_add_ref_if_not_builtin(datatype); + MPIDI_CH4U_REQUEST(unexp_req, datatype) = datatype; + MPIDI_CH4U_REQUEST(unexp_req, buffer) = (char *) buf; + MPIDI_CH4U_REQUEST(unexp_req, count) = count; + *request = unexp_req; + mpi_errno = MPIDI_NM_am_recv(unexp_req); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + goto fn_exit; + } + else { + *request = unexp_req; + mpi_errno = MPIDI_CH4I_handle_unexpected(buf, count, datatype, + root_comm, context_id, unexp_req); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + goto fn_exit; + } + } + + if (alloc_req) { + rreq = MPIDI_CH4I_am_request_create(MPIR_REQUEST_KIND__RECV); + } + else { + rreq = *request; + MPIR_Assert(0); + } + + *request = rreq; + if (unlikely(rank == MPI_PROC_NULL)) { + rreq->kind = MPIR_REQUEST_KIND__RECV; + rreq->status.MPI_ERROR = MPI_SUCCESS; + rreq->status.MPI_SOURCE = rank; + rreq->status.MPI_TAG = tag; + MPIDI_CH4I_am_request_complete(rreq); + goto fn_exit; + } + + dtype_add_ref_if_not_builtin(datatype); + MPIDI_CH4U_REQUEST(rreq, tag) = match_bits; + MPIDI_CH4U_REQUEST(rreq, req->rreq.ignore) = mask_bits; + MPIDI_CH4U_REQUEST(rreq, datatype) = datatype; + + mpi_errno = MPIDI_CH4I_prepare_recv_req(buf, count, datatype, rreq); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + + if (!unexp_req) { + /* MPIDI_CS_ENTER(); */ + /* Increment refcnt for comm before posting rreq to posted_list, + * to make sure comm is alive while holding an entry in the posted_list */ + MPIR_Comm_add_ref(root_comm); + MPIDI_CH4U_enqueue_posted(rreq, &MPIDI_CH4U_COMM(root_comm, posted_list)); + /* MPIDI_CS_EXIT(); */ + } + else { + MPIDI_CH4U_REQUEST(unexp_req, req->rreq.match_req) = (uint64_t) rreq; + } + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_DO_IRECV); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_recv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_CH4U_recv(void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, + int context_offset, MPI_Status * status, MPIR_Request ** request) +{ + int mpi_errno; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_RECV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_RECV); + + mpi_errno = MPIDI_CH4I_do_irecv(buf, count, datatype, rank, tag, + comm, context_offset, request, 1, 0ULL); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_RECV); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_recv_init +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_CH4U_recv_init(void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, + int context_offset, MPIR_Request ** request) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Request *rreq; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_RECV_INIT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_RECV_INIT); + + rreq = MPIDI_CH4I_am_request_create(MPIR_REQUEST_KIND__PREQUEST_RECV); + + *request = rreq; + rreq->comm = comm; + MPIR_Comm_add_ref(comm); + + MPIDI_CH4U_REQUEST(rreq, buffer) = (void *) buf; + MPIDI_CH4U_REQUEST(rreq, count) = count; + MPIDI_CH4U_REQUEST(rreq, datatype) = datatype; + MPIDI_CH4U_REQUEST(rreq, tag) = + MPIDI_CH4U_init_send_tag(comm->context_id + context_offset, rank, tag); + rreq->u.persist.real_request = NULL; + MPIDI_CH4I_am_request_complete(rreq); + MPIDI_CH4U_REQUEST(rreq, p_type) = MPIDI_PTYPE_RECV; + dtype_add_ref_if_not_builtin(datatype); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_RECV_INIT); + return mpi_errno; +} + + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_imrecv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_CH4U_imrecv(void *buf, + int count, + MPI_Datatype datatype, + MPIR_Request * message, MPIR_Request ** rreqp) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Request *rreq; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_IMRECV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_IMRECV); + + if (message == NULL) { + MPIDI_Request_create_null_rreq(rreq, mpi_errno, fn_fail); + *rreqp = rreq; + goto fn_exit; + } + + MPIR_Assert(message->kind == MPIR_REQUEST_KIND__MPROBE); + MPIDI_CH4U_REQUEST(message, req->rreq.mrcv_buffer) = buf; + MPIDI_CH4U_REQUEST(message, req->rreq.mrcv_count) = count; + MPIDI_CH4U_REQUEST(message, req->rreq.mrcv_datatype) = datatype; + *rreqp = message; + + /* MPIDI_CS_ENTER(); */ + if (MPIDI_CH4U_REQUEST(message, req->status) & MPIDI_CH4U_REQ_BUSY) { + MPIDI_CH4U_REQUEST(message, req->status) |= MPIDI_CH4U_REQ_UNEXP_CLAIMED; + } + else if (MPIDI_CH4U_REQUEST(message, req->status) & MPIDI_CH4U_REQ_LONG_RTS) { + /* Matching receive is now posted, tell the netmod */ + message->kind = MPIR_REQUEST_KIND__RECV; + dtype_add_ref_if_not_builtin(datatype); + MPIDI_CH4U_REQUEST(message, datatype) = datatype; + MPIDI_CH4U_REQUEST(message, buffer) = (char *) buf; + MPIDI_CH4U_REQUEST(message, count) = count; + mpi_errno = MPIDI_NM_am_recv(message); + } + else { + mpi_errno = MPIDI_CH4U_unexp_mrecv_cmpl_handler(message); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + } + /* MPIDI_CS_EXIT(); */ + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_IMRECV); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_mrecv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_CH4U_mrecv(void *buf, + int count, + MPI_Datatype datatype, + MPIR_Request * message, MPI_Status * status) +{ + int mpi_errno = MPI_SUCCESS, active_flag; + MPI_Request req_handle; + MPIR_Request *rreq = NULL; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_MRECV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_MRECV); + + mpi_errno = MPIDI_Imrecv(buf, count, datatype, message, &rreq); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + while (!MPIR_Request_is_complete(rreq)) { + MPIDI_Progress_test(); + } + + MPIR_Request_extract_status(rreq, status); + + mpi_errno = MPIR_Request_complete(&req_handle, rreq, status, &active_flag); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_MRECV); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_irecv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_CH4U_irecv(void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_IRECV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_IRECV); + + mpi_errno = MPIDI_CH4I_do_irecv(buf, count, datatype, rank, tag, + comm, context_offset, request, 1, 0ULL); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_IRECV); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_cancel_recv +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_CH4U_cancel_recv(MPIR_Request * rreq) +{ + int mpi_errno = MPI_SUCCESS, found; + MPIR_Comm *root_comm; + uint64_t msg_tag; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_CANCEL_RECV); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_CANCEL_RECV); + + msg_tag = MPIDI_CH4U_REQUEST(rreq, tag); + root_comm = MPIDI_CH4U_context_id_to_comm(MPIDI_CH4U_get_context(msg_tag)); + + /* MPIDI_CS_ENTER(); */ + found = + MPIDI_CH4U_delete_posted(&rreq->dev.ch4.ch4u.req->rreq, + &MPIDI_CH4U_COMM(root_comm, posted_list)); + /* MPIDI_CS_EXIT(); */ + + if (found) { + MPIR_STATUS_SET_CANCEL_BIT(rreq->status, TRUE); + MPIR_STATUS_SET_COUNT(rreq->status, 0); + MPIR_Comm_release(root_comm); /* -1 for posted_list */ + MPIDI_CH4I_am_request_complete(rreq); + } + else { + MPIR_STATUS_SET_CANCEL_BIT(rreq->status, FALSE); + } + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_CANCEL_RECV); + return mpi_errno; +} + +#endif /* CH4R_RECV_H_INCLUDED */ diff --git a/src/mpid/ch4/src/ch4r_recvq.h b/src/mpid/ch4/src/ch4r_recvq.h new file mode 100644 index 0000000..ff98e36 --- /dev/null +++ b/src/mpid/ch4/src/ch4r_recvq.h @@ -0,0 +1,336 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef CH4R_RECVQ_H_INCLUDED +#define CH4R_RECVQ_H_INCLUDED + +#include <mpidimpl.h> +#include "mpl_utlist.h" +#include "ch4_impl.h" + +#ifdef MPIDI_CH4U_USE_PER_COMM_QUEUE + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_enqueue_posted +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ void MPIDI_CH4U_enqueue_posted(MPIR_Request * req, MPIDI_CH4U_rreq_t ** list) +{ + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_ENQUEUE_POSTED); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_ENQUEUE_POSTED); + MPIDI_CH4U_REQUEST(req, req->rreq.request) = (uint64_t) req; + MPL_DL_APPEND(*list, &req->dev.ch4.ch4u.req->rreq); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_ENQUEUE_POSTED); +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_enqueue_unexp +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ void MPIDI_CH4U_enqueue_unexp(MPIR_Request * req, MPIDI_CH4U_rreq_t ** list) +{ + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_ENQUEUE_UNEXP); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_ENQUEUE_UNEXP); + MPIDI_CH4U_REQUEST(req, req->rreq.request) = (uint64_t) req; + MPL_DL_APPEND(*list, &req->dev.ch4.ch4u.req->rreq); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_ENQUEUE_UNEXP); +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_delete_unexp +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ void MPIDI_CH4U_delete_unexp(MPIR_Request * req, MPIDI_CH4U_rreq_t ** list) +{ + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_DELETE_UNEXP); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_DELETE_UNEXP); + MPL_DL_DELETE(*list, &req->dev.ch4.ch4u.req->rreq); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_DELETE_UNEXP); +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_dequeue_unexp_strict +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ MPIR_Request *MPIDI_CH4U_dequeue_unexp_strict(uint64_t tag, uint64_t ignore, + MPIDI_CH4U_rreq_t ** list) +{ + MPIDI_CH4U_rreq_t *curr, *tmp; + MPIR_Request *req = NULL; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_DEQUEUE_UNEXP_STRICT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_DEQUEUE_UNEXP_STRICT); + + MPL_DL_FOREACH_SAFE(*list, curr, tmp) { + req = (MPIR_Request *) curr->request; + if (!(MPIDI_CH4U_REQUEST(req, req->status) & MPIDI_CH4U_REQ_BUSY) && + ((tag & ~ignore) == (MPIDI_CH4U_REQUEST(req, tag) & ~ignore))) { + MPL_DL_DELETE(*list, curr); + break; + } + req = NULL; + } + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_DEQUEUE_UNEXP_STRICT); + return req; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_dequeue_unexp +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ MPIR_Request *MPIDI_CH4U_dequeue_unexp(uint64_t tag, uint64_t ignore, + MPIDI_CH4U_rreq_t ** list) +{ + MPIDI_CH4U_rreq_t *curr, *tmp; + MPIR_Request *req = NULL; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_DEQUEUE_UNEXP); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_DEQUEUE_UNEXP); + + MPL_DL_FOREACH_SAFE(*list, curr, tmp) { + req = (MPIR_Request *) curr->request; + if ((tag & ~ignore) == (MPIDI_CH4U_REQUEST(req, tag) & ~ignore)) { + MPL_DL_DELETE(*list, curr); + break; + } + req = NULL; + } + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_DEQUEUE_UNEXP); + return req; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_find_unexp +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ MPIR_Request *MPIDI_CH4U_find_unexp(uint64_t tag, uint64_t ignore, + MPIDI_CH4U_rreq_t ** list) +{ + MPIDI_CH4U_rreq_t *curr, *tmp; + MPIR_Request *req = NULL; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_FIND_UNEXP); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_FIND_UNEXP); + + MPL_DL_FOREACH_SAFE(*list, curr, tmp) { + req = (MPIR_Request *) curr->request; + if ((tag & ~ignore) == (MPIDI_CH4U_REQUEST(req, tag) & ~ignore)) { + break; + } + req = NULL; + } + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_FIND_UNEXP); + return req; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_dequeue_posted +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ MPIR_Request *MPIDI_CH4U_dequeue_posted(uint64_t tag, MPIDI_CH4U_rreq_t ** list) +{ + MPIR_Request *req = NULL; + MPIDI_CH4U_rreq_t *curr, *tmp; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_DEQUEUE_POSTED); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_DEQUEUE_POSTED); + + MPL_DL_FOREACH_SAFE(*list, curr, tmp) { + req = (MPIR_Request *) curr->request; + if ((tag & ~(MPIDI_CH4U_REQUEST(req, req->rreq.ignore))) == + (MPIDI_CH4U_REQUEST(req, tag) & ~(MPIDI_CH4U_REQUEST(req, req->rreq.ignore)))) { + MPL_DL_DELETE(*list, curr); + break; + } + req = NULL; + } + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_DEQUEUE_POSTED); + return req; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_delete_posted +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_CH4U_delete_posted(MPIDI_CH4U_rreq_t * req, MPIDI_CH4U_rreq_t ** list) +{ + int found = 0; + MPIDI_CH4U_rreq_t *curr, *tmp; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_DELETE_POSTED); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_DELETE_POSTED); + MPL_DL_FOREACH_SAFE(*list, curr, tmp) { + if (curr == req) { + MPL_DL_DELETE(*list, curr); + found = 1; + break; + } + } + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_DELETE_POSTED); + return found; +} + +#else /* #ifdef MPIDI_CH4U_USE_PER_COMM_QUEUE */ + +/* Use global queue */ + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_enqueue_posted +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ void MPIDI_CH4U_enqueue_posted(MPIR_Request * req, MPIDI_CH4U_rreq_t ** list) +{ + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_ENQUEUE_POSTED); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_ENQUEUE_POSTED); + MPIDI_CH4U_REQUEST(req, req->rreq.request) = (uint64_t) req; + MPL_DL_APPEND(MPIDI_CH4_Global.posted_list, &req->dev.ch4.ch4u.req->rreq); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_ENQUEUE_POSTED); +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_enqueue_unexp +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ void MPIDI_CH4U_enqueue_unexp(MPIR_Request * req, MPIDI_CH4U_rreq_t ** list) +{ + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_ENQUEUE_UNEXP); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_ENQUEUE_UNEXP); + MPIDI_CH4U_REQUEST(req, req->rreq.request) = (uint64_t) req; + MPL_DL_APPEND(MPIDI_CH4_Global.unexp_list, &req->dev.ch4.ch4u.req->rreq); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_ENQUEUE_UNEXP); +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_delete_unexp +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ void MPIDI_CH4U_delete_unexp(MPIR_Request * req, MPIDI_CH4U_rreq_t ** list) +{ + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_DELETE_UNEXP); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_DELETE_UNEXP); + MPL_DL_DELETE(MPIDI_CH4_Global.unexp_list, &req->dev.ch4.ch4u.req->rreq); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_DELETE_UNEXP); +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_dequeue_unexp_strict +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ MPIR_Request *MPIDI_CH4U_dequeue_unexp_strict(uint64_t tag, uint64_t ignore, + MPIDI_CH4U_rreq_t ** list) +{ + MPIDI_CH4U_rreq_t *curr, *tmp; + MPIR_Request *req = NULL; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_DEQUEUE_UNEXP_STRICT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_DEQUEUE_UNEXP_STRICT); + + MPL_DL_FOREACH_SAFE(MPIDI_CH4_Global.unexp_list, curr, tmp) { + req = (MPIR_Request *) curr->request; + if (!(MPIDI_CH4U_REQUEST(req, req->status) & MPIDI_CH4U_REQ_BUSY) && + ((tag & ~ignore) == (MPIDI_CH4U_REQUEST(req, tag) & ~ignore))) { + MPL_DL_DELETE(MPIDI_CH4_Global.unexp_list, curr); + break; + } + req = NULL; + } + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_DEQUEUE_UNEXP_STRICT); + return req; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_dequeue_unexp +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ MPIR_Request *MPIDI_CH4U_dequeue_unexp(uint64_t tag, uint64_t ignore, + MPIDI_CH4U_rreq_t ** list) +{ + MPIDI_CH4U_rreq_t *curr, *tmp; + MPIR_Request *req = NULL; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_DEQUEUE_UNEXP); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_DEQUEUE_UNEXP); + + MPL_DL_FOREACH_SAFE(MPIDI_CH4_Global.unexp_list, curr, tmp) { + req = (MPIR_Request *) curr->request; + if ((tag & ~ignore) == (MPIDI_CH4U_REQUEST(req, tag) & ~ignore)) { + MPL_DL_DELETE(MPIDI_CH4_Global.unexp_list, curr); + break; + } + req = NULL; + } + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_DEQUEUE_UNEXP); + return req; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_find_unexp +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ MPIR_Request *MPIDI_CH4U_find_unexp(uint64_t tag, uint64_t ignore, + MPIDI_CH4U_rreq_t ** list) +{ + MPIDI_CH4U_rreq_t *curr, *tmp; + MPIR_Request *req = NULL; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_FIND_UNEXP); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_FIND_UNEXP); + + MPL_DL_FOREACH_SAFE(MPIDI_CH4_Global.unexp_list, curr, tmp) { + req = (MPIR_Request *) curr->request; + if ((tag & ~ignore) == (MPIDI_CH4U_REQUEST(req, tag) & ~ignore)) { + break; + } + req = NULL; + } + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_FIND_UNEXP); + return req; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_dequeue_posted +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ MPIR_Request *MPIDI_CH4U_dequeue_posted(uint64_t tag, MPIDI_CH4U_rreq_t ** list) +{ + MPIR_Request *req = NULL; + MPIDI_CH4U_rreq_t *curr, *tmp; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_DEQUEUE_POSTED); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_DEQUEUE_POSTED); + + MPL_DL_FOREACH_SAFE(MPIDI_CH4_Global.posted_list, curr, tmp) { + req = (MPIR_Request *) curr->request; + if ((tag & ~MPIDI_CH4U_REQUEST(req, req->rreq.ignore)) == + (MPIDI_CH4U_REQUEST(req, tag) & ~MPIDI_CH4U_REQUEST(req, req->rreq.ignore))) { + MPL_DL_DELETE(MPIDI_CH4_Global.posted_list, curr); + break; + } + req = NULL; + } + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_DEQUEUE_POSTED); + return req; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_delete_posted +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_CH4U_delete_posted(MPIDI_CH4U_rreq_t * req, MPIDI_CH4U_rreq_t ** list) +{ + int found = 0; + MPIDI_CH4U_rreq_t *curr, *tmp; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_DELETE_POSTED); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_DELETE_POSTED); + MPL_DL_FOREACH_SAFE(MPIDI_CH4_Global.posted_list, curr, tmp) { + if (curr == req) { + MPL_DL_DELETE(MPIDI_CH4_Global.posted_list, curr); + found = 1; + break; + } + } + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_DELETE_POSTED); + return found; +} + +#endif /* MPIDI_CH4U_USE_PER_COMM_QUEUE */ + +#endif /* CH4R_RECVQ_H_INCLUDED */ diff --git a/src/mpid/ch4/src/ch4r_request.h b/src/mpid/ch4/src/ch4r_request.h new file mode 100644 index 0000000..68399c5 --- /dev/null +++ b/src/mpid/ch4/src/ch4r_request.h @@ -0,0 +1,126 @@ +/* -*- Mode: C; c-basic-offset:4 ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef CH4R_REQUEST_H_INCLUDED +#define CH4R_REQUEST_H_INCLUDED + +#include "ch4_types.h" +#include "ch4r_buf.h" + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4I_am_request_create +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline MPIR_Request *MPIDI_CH4I_am_request_create(MPIR_Request_kind_t kind) +{ + MPIR_Request *req; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4I_REQUEST_CREATE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4I_REQUEST_CREATE); + + req = MPIR_Request_create(kind); + MPIDI_NM_am_request_init(req); + MPIR_Request_add_ref(req); + + CH4_COMPILE_TIME_ASSERT(sizeof(MPIDI_CH4U_req_ext_t) <= MPIDI_CH4I_BUF_POOL_SZ); + MPIDI_CH4U_REQUEST(req, req) = + (MPIDI_CH4U_req_ext_t *) MPIDI_CH4R_get_buf(MPIDI_CH4_Global.buf_pool); + MPIR_Assert(MPIDI_CH4U_REQUEST(req, req)); + MPIDI_CH4U_REQUEST(req, req->status) = 0; + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4I_REQUEST_CREATE); + + return req; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4I_am_win_request_create +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline MPIR_Request *MPIDI_CH4I_am_win_request_create() +{ + MPIR_Request *req; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4I_WIN_REQUEST_CREATE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4I_WIN_REQUEST_CREATE); + + req = MPIR_Request_create(MPIR_REQUEST_KIND__UNDEFINED); + MPIDI_NM_am_request_init(req); + + CH4_COMPILE_TIME_ASSERT(sizeof(MPIDI_CH4U_req_ext_t) <= MPIDI_CH4I_BUF_POOL_SZ); + MPIDI_CH4U_REQUEST(req, req) = + (MPIDI_CH4U_req_ext_t *) MPIDI_CH4R_get_buf(MPIDI_CH4_Global.buf_pool); + MPIR_Assert(MPIDI_CH4U_REQUEST(req, req)); + MPIDI_CH4U_REQUEST(req, req->status) = 0; + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4I_WIN_REQUEST_CREATE); + return req; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4I_am_request_complete +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ void MPIDI_CH4I_am_request_complete(MPIR_Request * req) +{ + int incomplete; + MPIR_cc_decr(req->cc_ptr, &incomplete); + if (!incomplete) { + if (MPIDI_CH4U_REQUEST(req, req) && MPIR_cc_is_complete(&req->cc)) { + MPIDI_CH4R_release_buf(MPIDI_CH4U_REQUEST(req, req)); + MPIDI_CH4U_REQUEST(req, req) = NULL; + } + MPIDI_NM_am_request_finalize(req); + MPIDI_CH4U_request_release(req); + } +} + +/* This function should be called any time an anysource request is matched so + * the upper layer will have a chance to arbitrate who wins the race between + * the netmod and the shmod. This will cancel the request of the other side and + * take care of copying any relevant data. */ +static inline int MPIDI_CH4R_anysource_matched(MPIR_Request * rreq, int caller, + int *continue_matching) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPIDI_CH4R_ANYSOURCE_MATCHED); + MPIR_FUNC_VERBOSE_ENTER(MPIDI_CH4R_ANYSOURCE_MATCHED); + + MPIR_Assert(MPIDI_CH4R_NETMOD == caller || MPIDI_CH4R_SHM == caller); + + if (MPIDI_CH4R_NETMOD == caller) { +#ifdef MPIDI_BUILD_CH4_SHM + mpi_errno = MPIDI_SHM_cancel_recv(rreq); + + /* If the netmod is cancelling the request, then shared memory will + * just copy the status from the shared memory side because the netmod + * will always win the race condition here. */ + if (MPIR_STATUS_GET_CANCEL_BIT(rreq->status)) { + /* If the request is cancelled, copy the status object from the + * partner request */ + rreq->status = MPIDI_CH4I_REQUEST_ANYSOURCE_PARTNER(rreq)->status; + } +#endif + *continue_matching = 0; + } + else if (MPIDI_CH4R_SHM == caller) { + mpi_errno = MPIDI_NM_cancel_recv(rreq); + + /* If the netmod has already matched this request, shared memory will + * lose and should stop matching this request */ + *continue_matching = !MPIR_STATUS_GET_CANCEL_BIT(rreq->status); + } + + MPIR_FUNC_VERBOSE_EXIT(MPIDI_CH4R_ANYSOURCE_MATCHED); + return mpi_errno; +} + +#endif /* CH4R_REQUEST_H_INCLUDED */ diff --git a/src/mpid/ch4/src/ch4r_rma.h b/src/mpid/ch4/src/ch4r_rma.h new file mode 100644 index 0000000..b0411db --- /dev/null +++ b/src/mpid/ch4/src/ch4r_rma.h @@ -0,0 +1,761 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef CH4R_RMA_H_INCLUDED +#define CH4R_RMA_H_INCLUDED + +#include "ch4_impl.h" + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4I_do_put +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4I_do_put(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPIR_Win * win, MPIR_Request ** request) +{ + int mpi_errno = MPI_SUCCESS, n_iov, c; + MPIR_Request *sreq = NULL; + MPIDI_CH4U_put_msg_t am_hdr; + uint64_t offset; + size_t data_sz; + MPI_Aint last, num_iov; + MPID_Segment *segment_ptr; + struct iovec *dt_iov, am_iov[2]; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4I_DO_PUT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4I_DO_PUT); + + MPIDI_CH4U_EPOCH_CHECK_SYNC(win, mpi_errno, goto fn_fail); + + sreq = MPIDI_CH4I_am_win_request_create(); + MPIR_Assert(sreq); + sreq->kind = MPIR_REQUEST_KIND__RMA; + if (request) { + *request = sreq; + MPIDI_Request_add_ref(sreq); + } + + MPIDI_CH4U_REQUEST(sreq, req->preq.win_ptr) = win; + MPIDI_Datatype_check_size(origin_datatype, origin_count, data_sz); + if (data_sz == 0 || target_rank == MPI_PROC_NULL) { + MPIDI_CH4I_am_request_complete(sreq); + goto fn_exit; + } + + if (target_rank == win->comm_ptr->rank) { + offset = win->disp_unit * target_disp; + MPIDI_CH4I_am_request_complete(sreq); + return MPIR_Localcopy(origin_addr, + origin_count, + origin_datatype, + (char *) win->base + offset, target_count, target_datatype); + } + + MPIDI_CH4U_EPOCH_START_CHECK(win, mpi_errno, goto fn_fail); + MPIR_cc_incr(sreq->cc_ptr, &c); + am_hdr.src_rank = win->comm_ptr->rank; + am_hdr.target_disp = target_disp; + am_hdr.count = target_count; + am_hdr.datatype = target_datatype; + am_hdr.preq_ptr = (uint64_t) sreq; + am_hdr.win_id = MPIDI_CH4U_WIN(win, win_id); + + /* MPIDI_CS_ENTER(); */ + OPA_incr_int(&MPIDI_CH4U_WIN(win, outstanding_ops)); + /* MPIDI_CS_EXIT(); */ + + if (HANDLE_GET_KIND(target_datatype) == HANDLE_KIND_BUILTIN) { + am_hdr.n_iov = 0; + MPIDI_CH4U_REQUEST(sreq, req->preq.dt_iov) = NULL; + + mpi_errno = MPIDI_NM_send_am(target_rank, win->comm_ptr, MPIDI_CH4U_PUT_REQ, + &am_hdr, sizeof(am_hdr), origin_addr, + origin_count, origin_datatype, sreq, NULL); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + goto fn_exit; + } + + segment_ptr = MPIDU_Segment_alloc(); + MPIR_Assert(segment_ptr); + + MPIDU_Segment_init(NULL, target_count, target_datatype, segment_ptr, 0); + last = data_sz; + MPIDU_Segment_count_contig_blocks(segment_ptr, 0, &last, &num_iov); + n_iov = (int) num_iov; + MPIR_Assert(n_iov > 0); + am_hdr.n_iov = n_iov; + dt_iov = (struct iovec *) MPL_malloc(n_iov * sizeof(struct iovec)); + MPIR_Assert(dt_iov); + + last = data_sz; + MPIDU_Segment_pack_vector(segment_ptr, 0, &last, dt_iov, &n_iov); + MPIR_Assert(last == (MPI_Aint) data_sz); + MPL_free(segment_ptr); + + am_iov[0].iov_base = &am_hdr; + am_iov[0].iov_len = sizeof(am_hdr); + am_iov[1].iov_base = dt_iov; + am_iov[1].iov_len = sizeof(struct iovec) * am_hdr.n_iov; + + MPIDI_CH4U_REQUEST(sreq, req->preq.dt_iov) = dt_iov; + + if ((am_iov[0].iov_len + am_iov[1].iov_len) <= MPIDI_NM_am_hdr_max_sz()) { + mpi_errno = MPIDI_NM_send_amv(target_rank, win->comm_ptr, MPIDI_CH4U_PUT_REQ, + &am_iov[0], 2, origin_addr, origin_count, origin_datatype, + sreq, NULL); + } + else { + MPIDI_CH4U_REQUEST(sreq, req->preq.origin_addr) = (void *) origin_addr; + MPIDI_CH4U_REQUEST(sreq, req->preq.origin_count) = origin_count; + MPIDI_CH4U_REQUEST(sreq, req->preq.origin_datatype) = origin_datatype; + MPIDI_CH4U_REQUEST(sreq, src_rank) = target_rank; + dtype_add_ref_if_not_builtin(origin_datatype); + + mpi_errno = MPIDI_NM_send_am(target_rank, win->comm_ptr, MPIDI_CH4U_PUT_IOV_REQ, + &am_hdr, sizeof(am_hdr), am_iov[1].iov_base, + am_iov[1].iov_len, MPI_BYTE, sreq, NULL); + } + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4I_DO_PUT); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4I_do_get +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4I_do_get(void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPIR_Win * win, MPIR_Request ** request) +{ + int mpi_errno = MPI_SUCCESS, n_iov, c; + size_t offset; + MPIR_Request *sreq = NULL; + MPIDI_CH4U_get_req_msg_t am_hdr; + size_t data_sz; + MPI_Aint last, num_iov; + MPID_Segment *segment_ptr; + struct iovec *dt_iov; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_MPIDI_CH4I_DO_GET); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_MPIDI_CH4I_DO_GET); + + MPIDI_CH4U_EPOCH_CHECK_SYNC(win, mpi_errno, goto fn_fail); + + sreq = MPIDI_CH4I_am_win_request_create(); + MPIR_Assert(sreq); + sreq->kind = MPIR_REQUEST_KIND__RMA; + if (request) { + *request = sreq; + MPIDI_Request_add_ref(sreq); + } + + MPIDI_Datatype_check_size(origin_datatype, origin_count, data_sz); + if (data_sz == 0 || target_rank == MPI_PROC_NULL) { + MPIDI_CH4I_am_request_complete(sreq); + goto fn_exit; + } + + MPIDI_CH4U_REQUEST(sreq, req->greq.win_ptr) = win; + MPIDI_CH4U_REQUEST(sreq, req->greq.addr) = (uint64_t) ((char *) origin_addr); + MPIDI_CH4U_REQUEST(sreq, req->greq.count) = origin_count; + MPIDI_CH4U_REQUEST(sreq, req->greq.datatype) = origin_datatype; + + if (target_rank == win->comm_ptr->rank) { + MPIDI_CH4I_am_request_complete(sreq); + offset = win->disp_unit * target_disp; + return MPIR_Localcopy((char *) win->base + offset, + target_count, + target_datatype, origin_addr, origin_count, origin_datatype); + } + + MPIDI_CH4U_EPOCH_START_CHECK(win, mpi_errno, goto fn_fail); + MPIR_cc_incr(sreq->cc_ptr, &c); + am_hdr.target_disp = target_disp; + am_hdr.count = target_count; + am_hdr.datatype = target_datatype; + am_hdr.greq_ptr = (uint64_t) sreq; + am_hdr.win_id = MPIDI_CH4U_WIN(win, win_id); + am_hdr.src_rank = win->comm_ptr->rank; + + /* MPIDI_CS_ENTER(); */ + OPA_incr_int(&MPIDI_CH4U_WIN(win, outstanding_ops)); + /* MPIDI_CS_EXIT(); */ + + if (HANDLE_GET_KIND(target_datatype) == HANDLE_KIND_BUILTIN) { + am_hdr.n_iov = 0; + MPIDI_CH4U_REQUEST(sreq, req->greq.dt_iov) = NULL; + + mpi_errno = MPIDI_NM_send_am_hdr(target_rank, win->comm_ptr, + MPIDI_CH4U_GET_REQ, &am_hdr, sizeof(am_hdr), sreq, NULL); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + goto fn_exit; + } + + segment_ptr = MPIDU_Segment_alloc(); + MPIR_Assert(segment_ptr); + + MPIDU_Segment_init(NULL, target_count, target_datatype, segment_ptr, 0); + last = data_sz; + MPIDU_Segment_count_contig_blocks(segment_ptr, 0, &last, &num_iov); + n_iov = (int) num_iov; + MPIR_Assert(n_iov > 0); + am_hdr.n_iov = n_iov; + dt_iov = (struct iovec *) MPL_malloc(n_iov * sizeof(struct iovec)); + MPIR_Assert(dt_iov); + + last = data_sz; + MPIDU_Segment_pack_vector(segment_ptr, 0, &last, dt_iov, &n_iov); + MPIR_Assert(last == (MPI_Aint) data_sz); + MPL_free(segment_ptr); + + MPIDI_CH4U_REQUEST(sreq, req->greq.dt_iov) = dt_iov; + mpi_errno = MPIDI_NM_send_am(target_rank, win->comm_ptr, MPIDI_CH4U_GET_REQ, + &am_hdr, sizeof(am_hdr), dt_iov, + sizeof(struct iovec) * am_hdr.n_iov, MPI_BYTE, sreq, NULL); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_MPIDI_CH4I_DO_GET); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_put +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_CH4U_put(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, MPI_Datatype target_datatype, MPIR_Win * win) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_PUT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_PUT); + + mpi_errno = MPIDI_CH4I_do_put(origin_addr, origin_count, origin_datatype, + target_rank, target_disp, target_count, + target_datatype, win, NULL); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_PUT); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_rput +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_CH4U_rput(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPIR_Win * win, MPIR_Request ** request) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_RPUT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_RPUT); + + mpi_errno = MPIDI_CH4I_do_put(origin_addr, origin_count, origin_datatype, + target_rank, target_disp, target_count, + target_datatype, win, request); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_RPUT); + return mpi_errno; + fn_fail: + goto fn_exit; +} + + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_get +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_CH4U_get(void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, MPI_Datatype target_datatype, MPIR_Win * win) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_GET); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_GET); + + mpi_errno = MPIDI_CH4I_do_get(origin_addr, origin_count, origin_datatype, + target_rank, target_disp, target_count, + target_datatype, win, NULL); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_GET); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_rget +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_CH4U_rget(void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPIR_Win * win, MPIR_Request ** request) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_RGET); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_RGET); + + mpi_errno = MPIDI_CH4I_do_get(origin_addr, origin_count, origin_datatype, + target_rank, target_disp, target_count, + target_datatype, win, request); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_RGET); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4I_do_accumulate +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_CH4I_do_accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPI_Op op, MPIR_Win * win, + int do_get, MPIR_Request * sreq) +{ + int mpi_errno = MPI_SUCCESS, c, n_iov; + size_t basic_type_size; + MPIDI_CH4U_acc_req_msg_t am_hdr; + uint64_t data_sz, result_data_sz, target_data_sz; + MPI_Aint last, num_iov; + MPID_Segment *segment_ptr; + struct iovec *dt_iov, am_iov[2]; + MPIR_Datatype *dt_ptr; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_DO_ACCUMULATE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_DO_ACCUMULATE); + + MPIDI_CH4U_EPOCH_CHECK_SYNC(win, mpi_errno, goto fn_fail); + + MPIDI_Datatype_get_size_dt_ptr(origin_count, origin_datatype, data_sz, dt_ptr); + MPIDI_Datatype_check_size(target_datatype, target_count, target_data_sz); + + if ((data_sz == 0 && do_get == 0) || + target_rank == MPI_PROC_NULL || target_count == 0 || target_data_sz == 0 || + (do_get == 1 && origin_count == 0 && + MPIDI_CH4U_REQUEST(sreq, req->areq.result_count) == 0)) { + if (do_get) + dtype_release_if_not_builtin(MPIDI_CH4U_REQUEST(sreq, req->areq.result_datatype)); + MPIDI_CH4I_am_request_complete(sreq); + goto fn_exit; + } + + MPIDI_CH4U_REQUEST(sreq, req->areq.win_ptr) = win; + MPIDI_CH4U_EPOCH_START_CHECK(win, mpi_errno, goto fn_fail); + MPIR_cc_incr(sreq->cc_ptr, &c); + + am_hdr.req_ptr = (uint64_t) sreq; + am_hdr.origin_count = origin_count; + am_hdr.do_get = do_get; + + if (HANDLE_GET_KIND(origin_datatype) == HANDLE_KIND_BUILTIN) { + am_hdr.origin_datatype = origin_datatype; + } + else { + am_hdr.origin_datatype = (dt_ptr) ? dt_ptr->basic_type : MPI_DATATYPE_NULL; + MPID_Datatype_get_size_macro(am_hdr.origin_datatype, basic_type_size); + am_hdr.origin_count = (basic_type_size > 0) ? data_sz / basic_type_size : 0; + } + + am_hdr.target_count = target_count; + am_hdr.target_datatype = target_datatype; + am_hdr.target_disp = target_disp; + am_hdr.op = op; + am_hdr.win_id = MPIDI_CH4U_WIN(win, win_id); + am_hdr.src_rank = win->comm_ptr->rank; + + if (do_get) { + MPIDI_Datatype_check_size(MPIDI_CH4U_REQUEST(sreq, req->areq.result_datatype), + MPIDI_CH4U_REQUEST(sreq, req->areq.result_count), result_data_sz); + am_hdr.result_data_sz = result_data_sz; + } + + /* MPIDI_CS_ENTER(); */ + OPA_incr_int(&MPIDI_CH4U_WIN(win, outstanding_ops)); + /* MPIDI_CS_EXIT(); */ + + MPIDI_CH4U_REQUEST(sreq, req->areq.data_sz) = data_sz; + if (HANDLE_GET_KIND(target_datatype) == HANDLE_KIND_BUILTIN) { + am_hdr.n_iov = 0; + MPIDI_CH4U_REQUEST(sreq, req->areq.dt_iov) = NULL; + + mpi_errno = MPIDI_NM_send_am(target_rank, win->comm_ptr, MPIDI_CH4U_ACC_REQ, + &am_hdr, sizeof(am_hdr), origin_addr, + (op == MPI_NO_OP) ? 0 : origin_count, + origin_datatype, sreq, NULL); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + goto fn_exit; + } + + MPIDI_Datatype_get_size_dt_ptr(target_count, target_datatype, data_sz, dt_ptr); + am_hdr.target_datatype = dt_ptr->basic_type; + am_hdr.target_count = dt_ptr->n_builtin_elements; + + segment_ptr = MPIDU_Segment_alloc(); + MPIR_Assert(segment_ptr); + + + MPIDU_Segment_init(NULL, target_count, target_datatype, segment_ptr, 0); + last = data_sz; + MPIDU_Segment_count_contig_blocks(segment_ptr, 0, &last, &num_iov); + n_iov = (int) num_iov; + MPIR_Assert(n_iov > 0); + am_hdr.n_iov = n_iov; + dt_iov = (struct iovec *) MPL_malloc(n_iov * sizeof(struct iovec)); + MPIR_Assert(dt_iov); + + last = data_sz; + MPIDU_Segment_pack_vector(segment_ptr, 0, &last, dt_iov, &n_iov); + MPIR_Assert(last == (MPI_Aint) data_sz); + MPL_free(segment_ptr); + + am_iov[0].iov_base = &am_hdr; + am_iov[0].iov_len = sizeof(am_hdr); + am_iov[1].iov_base = dt_iov; + am_iov[1].iov_len = sizeof(struct iovec) * am_hdr.n_iov; + MPIDI_CH4U_REQUEST(sreq, req->areq.dt_iov) = dt_iov; + + if ((am_iov[0].iov_len + am_iov[1].iov_len) <= MPIDI_NM_am_hdr_max_sz()) { + mpi_errno = MPIDI_NM_send_amv(target_rank, win->comm_ptr, MPIDI_CH4U_ACC_REQ, + &am_iov[0], 2, origin_addr, + (op == MPI_NO_OP) ? 0 : origin_count, + origin_datatype, sreq, NULL); + } + else { + MPIDI_CH4U_REQUEST(sreq, req->areq.origin_addr) = (void *) origin_addr; + MPIDI_CH4U_REQUEST(sreq, req->areq.origin_count) = origin_count; + MPIDI_CH4U_REQUEST(sreq, req->areq.origin_datatype) = origin_datatype; + MPIDI_CH4U_REQUEST(sreq, src_rank) = target_rank; + dtype_add_ref_if_not_builtin(origin_datatype); + + mpi_errno = MPIDI_NM_send_am(target_rank, win->comm_ptr, MPIDI_CH4U_ACC_IOV_REQ, + &am_hdr, sizeof(am_hdr), am_iov[1].iov_base, + am_iov[1].iov_len, MPI_BYTE, sreq, NULL); + } + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_DO_ACCUMULATE); + fn_exit: + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_raccumulate +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_CH4U_raccumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPI_Op op, MPIR_Win * win, MPIR_Request ** request) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Request *sreq; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_RACCUMULATE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_RACCUMULATE); + + sreq = MPIDI_CH4I_am_win_request_create(); + MPIR_Assert(sreq); + sreq->kind = MPIR_REQUEST_KIND__RMA; + if (request) { + *request = sreq; + MPIDI_Request_add_ref(sreq); + } + + mpi_errno = MPIDI_CH4I_do_accumulate(origin_addr, + origin_count, + origin_datatype, + target_rank, + target_disp, + target_count, target_datatype, op, win, 0, sreq); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_RACCUMULATE); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_accumulate +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_CH4U_accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, MPI_Op op, MPIR_Win * win) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_ACCUMULATE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_ACCUMULATE); + + mpi_errno = MPIDI_CH4U_raccumulate(origin_addr, + origin_count, + origin_datatype, + target_rank, + target_disp, target_count, target_datatype, op, win, NULL); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_ACCUMULATE); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_rget_accumulate +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_CH4U_rget_accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + void *result_addr, + int result_count, + MPI_Datatype result_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPI_Op op, MPIR_Win * win, MPIR_Request ** request) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Request *sreq; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_RGET_ACCUMULATE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_RGET_ACCUMULATE); + + sreq = MPIDI_CH4I_am_win_request_create(); + MPIR_Assert(sreq); + sreq->kind = MPIR_REQUEST_KIND__RMA; + + MPIDI_CH4U_REQUEST(sreq, req->areq.result_addr) = result_addr; + MPIDI_CH4U_REQUEST(sreq, req->areq.result_count) = result_count; + MPIDI_CH4U_REQUEST(sreq, req->areq.result_datatype) = result_datatype; + dtype_add_ref_if_not_builtin(result_datatype); + + if (request) { + *request = sreq; + MPIDI_Request_add_ref(sreq); + } + + mpi_errno = MPIDI_CH4I_do_accumulate(origin_addr, + origin_count, + origin_datatype, + target_rank, + target_disp, + target_count, target_datatype, op, win, 1, sreq); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_RGET_ACCUMULATE); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_get_accumulate +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_CH4U_get_accumulate(const void *origin_addr, + int origin_count, + MPI_Datatype origin_datatype, + void *result_addr, + int result_count, + MPI_Datatype result_datatype, + int target_rank, + MPI_Aint target_disp, + int target_count, + MPI_Datatype target_datatype, + MPI_Op op, MPIR_Win * win) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_GET_ACCUMULATE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_GET_ACCUMULATE); + + mpi_errno = MPIDI_CH4U_rget_accumulate(origin_addr, + origin_count, + origin_datatype, + result_addr, + result_count, + result_datatype, + target_rank, + target_disp, + target_count, target_datatype, op, win, NULL); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_GET_ACCUMULATE); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_compare_and_swap +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_CH4U_compare_and_swap(const void *origin_addr, + const void *compare_addr, + void *result_addr, + MPI_Datatype datatype, + int target_rank, + MPI_Aint target_disp, MPIR_Win * win) +{ + int mpi_errno = MPI_SUCCESS, c; + MPIR_Request *sreq = NULL; + MPIDI_CH4U_cswap_req_msg_t am_hdr; + size_t data_sz; + void *p_data; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_COMPARE_AND_SWAP); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_COMPARE_AND_SWAP); + + MPIDI_CH4U_EPOCH_CHECK_SYNC(win, mpi_errno, goto fn_fail); + + sreq = MPIDI_CH4I_am_win_request_create(); + MPIR_Assert(sreq); + sreq->kind = MPIR_REQUEST_KIND__RMA; + + MPIDI_Datatype_check_size(datatype, 1, data_sz); + if (data_sz == 0 || target_rank == MPI_PROC_NULL) { + MPIDI_CH4I_am_request_complete(sreq); + goto fn_exit; + } + + p_data = MPL_malloc(data_sz * 2); + MPIR_Assert(p_data); + MPIR_Memcpy(p_data, (char *) origin_addr, data_sz); + MPIR_Memcpy((char *) p_data + data_sz, (char *) compare_addr, data_sz); + + MPIDI_CH4U_REQUEST(sreq, req->creq.win_ptr) = win; + MPIDI_CH4U_REQUEST(sreq, req->creq.addr) = (uint64_t) ((char *) result_addr); + MPIDI_CH4U_REQUEST(sreq, req->creq.datatype) = datatype; + MPIDI_CH4U_REQUEST(sreq, req->creq.result_addr) = result_addr; + MPIDI_CH4U_REQUEST(sreq, req->creq.data) = p_data; + + MPIDI_CH4U_EPOCH_START_CHECK(win, mpi_errno, goto fn_fail); + MPIR_cc_incr(sreq->cc_ptr, &c); + + am_hdr.target_disp = target_disp; + am_hdr.datatype = datatype; + am_hdr.req_ptr = (uint64_t) sreq; + am_hdr.win_id = MPIDI_CH4U_WIN(win, win_id); + am_hdr.src_rank = win->comm_ptr->rank; + + /* MPIDI_CS_ENTER(); */ + OPA_incr_int(&MPIDI_CH4U_WIN(win, outstanding_ops)); + /* MPIDI_CS_EXIT(); */ + + mpi_errno = MPIDI_NM_send_am(target_rank, win->comm_ptr, MPIDI_CH4U_CSWAP_REQ, + &am_hdr, sizeof(am_hdr), (char *) p_data, 2, datatype, sreq, NULL); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_COMPARE_AND_SWAP); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_fetch_and_op +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_CH4U_fetch_and_op(const void *origin_addr, + void *result_addr, + MPI_Datatype datatype, + int target_rank, + MPI_Aint target_disp, MPI_Op op, MPIR_Win * win) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_FETCH_AND_OP); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_FETCH_AND_OP); + + mpi_errno = MPIDI_CH4U_get_accumulate(origin_addr, 1, datatype, + result_addr, 1, datatype, + target_rank, target_disp, 1, datatype, op, win); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_FETCH_AND_OP); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#endif /* CH4R_RMA_H_INCLUDED */ diff --git a/src/mpid/ch4/src/ch4r_send.h b/src/mpid/ch4/src/ch4r_send.h new file mode 100644 index 0000000..84a67e3 --- /dev/null +++ b/src/mpid/ch4/src/ch4r_send.h @@ -0,0 +1,468 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef CH4R_SEND_H_INCLUDED +#define CH4R_SEND_H_INCLUDED + +#include "ch4_impl.h" + +#include <../mpi/pt2pt/bsendutil.h> + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4I_do_send +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4I_do_send(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, + int context_offset, MPIR_Request ** request, int type) +{ + int mpi_errno = MPI_SUCCESS, c; + MPIR_Request *sreq = NULL; + uint64_t match_bits; + MPIDI_CH4U_hdr_t am_hdr; + MPIDI_CH4U_ssend_req_msg_t ssend_req; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_DO_SEND); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_DO_SEND); + + sreq = MPIDI_CH4I_am_request_create(MPIR_REQUEST_KIND__SEND); + MPIR_Assert(sreq); + + *request = sreq; + match_bits = MPIDI_CH4U_init_send_tag(comm->context_id + context_offset, comm->rank, tag); + + am_hdr.msg_tag = match_bits; + am_hdr.src_rank = comm->rank; + if (type == MPIDI_CH4U_SSEND_REQ) { + ssend_req.hdr = am_hdr; + ssend_req.sreq_ptr = (uint64_t) sreq; + MPIR_cc_incr(sreq->cc_ptr, &c); + + mpi_errno = MPIDI_NM_send_am(rank, comm, MPIDI_CH4U_SSEND_REQ, + &ssend_req, sizeof(ssend_req), + buf, count, datatype, sreq, NULL); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + } + else { + mpi_errno = MPIDI_NM_send_am(rank, comm, MPIDI_CH4U_SEND, + &am_hdr, sizeof(am_hdr), buf, count, datatype, sreq, NULL); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + } + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_DO_SEND); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4I_send +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4I_send(const void *buf, int count, MPI_Datatype datatype, + int rank, int tag, MPIR_Comm * comm, int context_offset, + MPIR_Request ** request, int noreq, int type) +{ + int mpi_errno; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_NM_SEND); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_NM_SEND); + + if (unlikely(rank == MPI_PROC_NULL)) { + mpi_errno = MPI_SUCCESS; + if (!noreq) { + *request = MPIDI_CH4I_am_request_create(MPIR_REQUEST_KIND__SEND); + MPIDI_Request_complete((*request)); + } + goto fn_exit; + } + + mpi_errno = + MPIDI_CH4I_do_send(buf, count, datatype, rank, tag, comm, context_offset, request, type); + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_NM_SEND); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4I_psend +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4I_psend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + MPIR_Request *sreq; + uint64_t match_bits; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_NM_PSEND); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_NM_PSEND); + + sreq = MPIDI_CH4I_am_request_create(MPIR_REQUEST_KIND__PREQUEST_SEND); + *request = sreq; + + MPIR_Comm_add_ref(comm); + sreq->comm = comm; + match_bits = MPIDI_CH4U_init_send_tag(comm->context_id + context_offset, rank, tag); + + MPIDI_CH4U_REQUEST(sreq, buffer) = (void *) buf; + MPIDI_CH4U_REQUEST(sreq, count) = count; + MPIDI_CH4U_REQUEST(sreq, datatype) = datatype; + MPIDI_CH4U_REQUEST(sreq, tag) = match_bits; + MPIDI_CH4U_REQUEST(sreq, src_rank) = rank; + + sreq->u.persist.real_request = NULL; + MPIDI_Request_complete(sreq); + + dtype_add_ref_if_not_builtin(datatype); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_NETMOD_OFI_NM_PSEND); + return MPI_SUCCESS; +} + + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_send +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_CH4U_send(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_SEND); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_SEND); + mpi_errno = MPIDI_CH4I_send(buf, count, datatype, rank, tag, comm, + context_offset, request, 1, 0ULL); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_SEND); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_isend +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_CH4U_isend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_ISEND); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_ISEND); + mpi_errno = MPIDI_CH4I_send(buf, count, datatype, rank, tag, comm, + context_offset, request, 0, 0ULL); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_ISEND); + return mpi_errno; +} + + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_rsend +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_CH4U_rsend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_RSEND); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_RSEND); + mpi_errno = MPIDI_CH4I_send(buf, count, datatype, rank, tag, comm, + context_offset, request, 1, 0ULL); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_RSEND); + return mpi_errno; +} + + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_irsend +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_CH4U_irsend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_IRSEND); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_IRSEND); + mpi_errno = MPIDI_CH4I_send(buf, count, datatype, rank, tag, comm, + context_offset, request, 0, 0ULL); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_SEND); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_ssend +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_CH4U_ssend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_SSEND); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_SSEND); + mpi_errno = MPIDI_CH4I_send(buf, count, datatype, rank, tag, comm, + context_offset, request, 1, MPIDI_CH4U_SSEND_REQ); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_SSEND); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_issend +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_CH4U_issend(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, int context_offset, MPIR_Request ** request) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_ISSEND); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_ISSEND); + mpi_errno = MPIDI_CH4I_send(buf, count, datatype, rank, tag, comm, + context_offset, request, 0, MPIDI_CH4U_SSEND_REQ); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_ISSEND); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_startall +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_CH4U_startall(int count, MPIR_Request * requests[]) +{ + int mpi_errno = MPI_SUCCESS, i; + int rank, tag, context_offset; + MPI_Datatype datatype; + uint64_t msg_tag; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_STARTALL); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_STARTALL); + + for (i = 0; i < count; i++) { + MPIR_Request *const preq = requests[i]; + MPI_Request sreq_handle; + + msg_tag = MPIDI_CH4U_REQUEST(preq, tag); + datatype = MPIDI_CH4U_REQUEST(preq, datatype); + + tag = MPIDI_CH4U_get_tag(msg_tag); + rank = MPIDI_CH4U_REQUEST(preq, src_rank); + context_offset = MPIDI_CH4U_get_context(msg_tag) - preq->comm->context_id; + + switch (MPIDI_CH4U_REQUEST(preq, p_type)) { + + case MPIDI_PTYPE_RECV: +#ifdef MPIDI_BUILD_CH4_SHM + mpi_errno = MPIDI_NM_irecv(MPIDI_CH4U_REQUEST(preq, buffer), + MPIDI_CH4U_REQUEST(preq, count), + datatype, rank, tag, + preq->comm, context_offset, &preq->u.persist.real_request); +#else + mpi_errno = MPIDI_Irecv(MPIDI_CH4U_REQUEST(preq, buffer), + MPIDI_CH4U_REQUEST(preq, count), + datatype, rank, tag, + preq->comm, context_offset, &preq->u.persist.real_request); +#endif + break; + + case MPIDI_PTYPE_SEND: +#ifdef MPIDI_BUILD_CH4_SHM + mpi_errno = MPIDI_NM_isend(MPIDI_CH4U_REQUEST(preq, buffer), + MPIDI_CH4U_REQUEST(preq, count), + datatype, rank, tag, + preq->comm, context_offset, &preq->u.persist.real_request); +#else + mpi_errno = MPIDI_Isend(MPIDI_CH4U_REQUEST(preq, buffer), + MPIDI_CH4U_REQUEST(preq, count), + datatype, rank, tag, + preq->comm, context_offset, &preq->u.persist.real_request); +#endif + break; + + case MPIDI_PTYPE_SSEND: +#ifdef MPIDI_BUILD_CH4_SHM + mpi_errno = MPIDI_NM_issend(MPIDI_CH4U_REQUEST(preq, buffer), + MPIDI_CH4U_REQUEST(preq, count), + datatype, rank, tag, + preq->comm, context_offset, &preq->u.persist.real_request); +#else + mpi_errno = MPIDI_Issend(MPIDI_CH4U_REQUEST(preq, buffer), + MPIDI_CH4U_REQUEST(preq, count), + datatype, rank, tag, + preq->comm, context_offset, &preq->u.persist.real_request); +#endif + break; + + case MPIDI_PTYPE_BSEND: + mpi_errno = MPIR_Ibsend_impl(MPIDI_CH4U_REQUEST(preq, buffer), + MPIDI_CH4U_REQUEST(preq, count), + datatype, rank, tag, preq->comm, &sreq_handle); + if (mpi_errno == MPI_SUCCESS) + MPIR_Request_get_ptr(sreq_handle, preq->u.persist.real_request); + + break; + + default: + mpi_errno = MPIR_Err_create_code(MPI_SUCCESS, MPIR_ERR_FATAL, __FUNCTION__, + __LINE__, MPI_ERR_INTERN, "**ch3|badreqtype", + "**ch3|badreqtype %d", MPIDI_CH4U_REQUEST(preq, + p_type)); + } + + if (mpi_errno == MPI_SUCCESS) { + preq->status.MPI_ERROR = MPI_SUCCESS; + + if (MPIDI_CH4U_REQUEST(preq, p_type) == MPIDI_PTYPE_BSEND) { + preq->cc_ptr = &preq->cc; + MPIDI_Request_set_completed(preq); + } + else + preq->cc_ptr = &preq->u.persist.real_request->cc; + } + else { + preq->u.persist.real_request = NULL; + preq->status.MPI_ERROR = mpi_errno; + preq->cc_ptr = &preq->cc; + MPIDI_Request_set_completed(preq); + } + dtype_release_if_not_builtin(datatype); + } + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_STARTALL); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_send_init +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_CH4U_send_init(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, + int context_offset, MPIR_Request ** request) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_SEND_INIT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_SEND_INIT); + mpi_errno = MPIDI_CH4I_psend(buf, count, datatype, rank, tag, comm, context_offset, request); + MPIDI_CH4U_REQUEST((*request), p_type) = MPIDI_PTYPE_SEND; + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_SEND_INIT); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_ssend_init +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_CH4U_ssend_init(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, + int context_offset, MPIR_Request ** request) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_SSEND_INIT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_SSEND_INIT); + mpi_errno = MPIDI_CH4I_psend(buf, count, datatype, rank, tag, comm, context_offset, request); + MPIDI_CH4U_REQUEST((*request), p_type) = MPIDI_PTYPE_SSEND; + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_SSEND_INIT); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_bsend_init +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_CH4U_bsend_init(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, + int context_offset, MPIR_Request ** request) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_BSEND_INIT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_BSEND_INIT); + mpi_errno = MPIDI_CH4I_psend(buf, count, datatype, rank, tag, comm, context_offset, request); + MPIDI_CH4U_REQUEST((*request), p_type) = MPIDI_PTYPE_BSEND; + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_BSEND_INIT); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_rsend_init +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_CH4U_rsend_init(const void *buf, + int count, + MPI_Datatype datatype, + int rank, + int tag, + MPIR_Comm * comm, + int context_offset, MPIR_Request ** request) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_RSEND_INIT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_RSEND_INIT); + mpi_errno = MPIDI_CH4I_psend(buf, count, datatype, rank, tag, comm, context_offset, request); + MPIDI_CH4U_REQUEST((*request), p_type) = MPIDI_PTYPE_SEND; + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_RSEND_INIT); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4U_cancel_send +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +__CH4_INLINE__ int MPIDI_CH4U_cancel_send(MPIR_Request * sreq) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_CANCEL_SEND); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_CANCEL_SEND); + /* cannot cancel send */ + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_CANCEL_SEND); + return mpi_errno; +} + +#endif /* CH4R_SEND_H_INCLUDED */ diff --git a/src/mpid/ch4/src/ch4r_symheap.h b/src/mpid/ch4/src/ch4r_symheap.h new file mode 100644 index 0000000..d64db40 --- /dev/null +++ b/src/mpid/ch4/src/ch4r_symheap.h @@ -0,0 +1,258 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef CH4R_SYMHEAP_H_INCLUDED +#define CH4R_SYMHEAP_H_INCLUDED + +#include <mpichconf.h> + +#include <opa_primitives.h> +#ifdef HAVE_SYS_MMAN_H +#include <sys/mman.h> +#endif /* HAVE_SYS_MMAN_H */ +#ifdef HAVE_SYS_TIME_H +#include <sys/time.h> +#endif /* HAVE_SYS_TIME_H */ +#ifdef HAVE_SYS_STAT_H +#include <sys/stat.h> +#endif /* HAVE_SYS_STAT_H */ +#ifdef HAVE_FCNTL_H +#include <fcntl.h> +#endif /* HAVE_FCNTL_H */ +#ifdef HAVE_STDINT_H +#include <stdint.h> +#endif /* HAVE_STDINT_H */ + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4R_get_mapsize +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline size_t MPIDI_CH4R_get_mapsize(size_t size, size_t * psz) +{ + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4R_GET_MAPSIZE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4R_GET_MAPSIZE); + + long page_sz = sysconf(_SC_PAGESIZE); + size_t mapsize = (size + (page_sz - 1)) & (~(page_sz - 1)); + *psz = page_sz; + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4R_GET_MAPSIZE); + return mapsize; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4R_check_maprange_ok +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4R_check_maprange_ok(void *start, size_t size) +{ + int rc = 0; + int ret = 0; + size_t page_sz; + size_t mapsize = MPIDI_CH4R_get_mapsize(size, &page_sz); + size_t i, num_pages = mapsize / page_sz; + char *ptr = (char *) start; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4R_CHECK_MAPRANGE_OK); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4R_CHECK_MAPRANGE_OK); + + for (i = 0; i < num_pages; i++) { + rc = msync(ptr, page_sz, 0); + + if (rc == -1) { + MPIR_Assert(errno == ENOMEM); + ptr += page_sz; + } + else + goto fn_exit; + } + + ret = 1; + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4R_CHECK_MAPRANGE_OK); + return ret; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4R_generate_random_addr +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline void *MPIDI_CH4R_generate_random_addr(size_t size) +{ + /* starting position for pointer to map + * This is not generic, probably only works properly on Linux + * but it's not fatal since we bail after a fixed number of iterations + */ +#define MPIDI_CH4I_MAP_POINTER ((random_unsigned&((0x00006FFFFFFFFFFF&(~(page_sz-1)))|0x0000600000000000))) + uintptr_t map_pointer; +#ifdef USE_SYM_HEAP + char random_state[256]; + size_t page_sz; + uint64_t random_unsigned; + size_t mapsize = MPIDI_CH4R_get_mapsize(size, &page_sz); + struct timeval ts; + int iter = 100; + int32_t rh, rl; + struct random_data rbuf; +#endif + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4R_GENERATE_RANDOM_ADDR); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4R_GENERATE_RANDOM_ADDR); + +#ifndef USE_SYM_HEAP + map_pointer = -1ULL; + goto fn_exit; +#else + + /* rbuf must be zero-cleared otherwise it results in SIGSEGV in glibc + * (http://stackoverflow.com/questions/4167034/c-initstate-r-crashing) */ + memset(&rbuf, 0, sizeof(rbuf)); + + gettimeofday(&ts, NULL); + + initstate_r(ts.tv_usec, random_state, sizeof(random_state), &rbuf); + random_r(&rbuf, &rh); + random_r(&rbuf, &rl); + random_unsigned = ((uint64_t) rh) << 32 | (uint64_t) rl; + map_pointer = MPIDI_CH4I_MAP_POINTER; + + while (MPIDI_CH4R_check_maprange_ok((void *) map_pointer, mapsize) == 0) { + random_r(&rbuf, &rh); + random_r(&rbuf, &rl); + random_unsigned = ((uint64_t) rh) << 32 | (uint64_t) rl; + map_pointer = MPIDI_CH4I_MAP_POINTER; + iter--; + + if (iter == 0) { + map_pointer = -1ULL; + goto fn_exit; + } + } + +#endif + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4R_GENERATE_RANDOM_ADDR); + return (void *) map_pointer; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4R_get_symmetric_heap +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4R_get_symmetric_heap(MPI_Aint size, + MPIR_Comm * comm, void **base, MPIR_Win * win) +{ + int mpi_errno = MPI_SUCCESS; + int iter = 100; + void *baseP; + size_t mapsize; +#ifdef USE_SYM_HEAP + unsigned test, result; + MPIR_Errflag_t errflag = MPIR_ERR_NONE; + size_t page_sz; +#endif + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4R_GET_SYMMETRIC_HEAP); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4R_GET_SYMMETRIC_HEAP); + +#ifndef USE_SYM_HEAP + iter = 0; +#else + + mapsize = MPIDI_CH4R_get_mapsize(size, &page_sz); + + struct { + uint64_t sz; + int loc; + } maxloc, maxloc_result; + + maxloc.sz = size; + maxloc.loc = comm->rank; + mpi_errno = MPIR_Allreduce_impl(&maxloc, + &maxloc_result, 1, MPI_LONG_INT, MPI_MAXLOC, comm, &errflag); + + if (mpi_errno != MPI_SUCCESS) + goto fn_fail; + + if (maxloc_result.sz > 0) { + result = 0; + + while (!result && --iter != 0) { + uintptr_t map_pointer = 0ULL; + + baseP = (void *) -1ULL; + + if (comm->rank == maxloc_result.loc) { + map_pointer = (uintptr_t) MPIDI_CH4R_generate_random_addr(mapsize); + baseP = mmap((void *) map_pointer, + mapsize, + PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON | MAP_FIXED, -1, 0); + } + + mpi_errno = MPIR_Bcast_impl(&map_pointer, + 1, MPI_UNSIGNED_LONG, maxloc_result.loc, comm, &errflag); + + if (mpi_errno != MPI_SUCCESS) + goto fn_fail; + + if (comm->rank != maxloc_result.loc) { + int rc = MPIDI_CH4R_check_maprange_ok((void *) map_pointer, mapsize); + + if (rc) { + baseP = mmap((void *) map_pointer, + mapsize, + PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON | MAP_FIXED, -1, 0); + } + else + baseP = (void *) -1ULL; + } + + if (mapsize == 0) + baseP = (void *) map_pointer; + + test = ((uintptr_t) baseP != -1ULL) ? 1 : 0; + mpi_errno = MPIR_Allreduce_impl(&test, + &result, 1, MPI_UNSIGNED, MPI_BAND, comm, &errflag); + + if (mpi_errno != MPI_SUCCESS) + goto fn_fail; + + if (result == 0 && baseP != (void *) -1ULL) + munmap(baseP, mapsize); + } + } + else + baseP = NULL; +#endif + + if (iter == 0) { + MPL_DBG_MSG(MPIDI_CH4_DBG_GENERAL, VERBOSE, + "WARNING: Win_allocate: Unable to allocate symmetric heap\n"); + baseP = MPL_malloc(size); + MPIR_ERR_CHKANDJUMP((baseP == NULL), mpi_errno, MPI_ERR_BUFFER, "**bufnull"); + MPIDI_CH4U_WIN(win, mmap_sz) = -1ULL; + MPIDI_CH4U_WIN(win, mmap_addr) = NULL; + } + else { + MPIDI_CH4U_WIN(win, mmap_sz) = mapsize; + MPIDI_CH4U_WIN(win, mmap_addr) = baseP; + } + + *base = baseP; + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4R_GET_SYMMETRIC_HEAP); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#endif /* CH4R_SYMHEAP_H_INCLUDED */ diff --git a/src/mpid/ch4/src/ch4r_win.h b/src/mpid/ch4/src/ch4r_win.h new file mode 100644 index 0000000..840f890 --- /dev/null +++ b/src/mpid/ch4/src/ch4r_win.h @@ -0,0 +1,1250 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ +#ifndef CH4R_WIN_H_INCLUDED +#define CH4R_WIN_H_INCLUDED + +#include "ch4_impl.h" +#include "ch4i_util.h" +#include <opa_primitives.h> +#include "mpir_info.h" +#include "mpl_uthash.h" +#ifdef HAVE_SYS_MMAN_H +#include <sys/mman.h> +#endif /* HAVE_SYS_MMAN_H */ + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4R_win_set_info +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4R_win_set_info(MPIR_Win * win, MPIR_Info * info) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Errflag_t errflag = MPIR_ERR_NONE; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_NETMOD_CH4I_WIN_SET_INFO); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_NETMOD_CH4I_WIN_SET_INFO); + + MPIR_Info *curr_ptr; + char *value, *token, *savePtr; + uint save_ordering; + + curr_ptr = info->next; + + while (curr_ptr) { + if (!strcmp(curr_ptr->key, "no_locks")) { + if (!strcmp(curr_ptr->value, "true")) + MPIDI_CH4U_WIN(win, info_args).no_locks = 1; + else + MPIDI_CH4U_WIN(win, info_args).no_locks = 0; + } + else if (!strcmp(curr_ptr->key, "accumulate_ordering")) { + save_ordering = (uint) MPIDI_CH4U_WIN(win, info_args).accumulate_ordering; + MPIDI_CH4U_WIN(win, info_args).accumulate_ordering = 0; + value = curr_ptr->value; + token = (char *) strtok_r(value, ",", &savePtr); + + while (token) { + if (!memcmp(token, "rar", 3)) + MPIDI_CH4U_WIN(win, info_args).accumulate_ordering = + (MPIDI_CH4U_WIN(win, info_args). + accumulate_ordering | MPIDI_CH4I_ACCU_ORDER_RAR); + else if (!memcmp(token, "raw", 3)) + MPIDI_CH4U_WIN(win, info_args).accumulate_ordering = + (MPIDI_CH4U_WIN(win, info_args). + accumulate_ordering | MPIDI_CH4I_ACCU_ORDER_RAW); + else if (!memcmp(token, "war", 3)) + MPIDI_CH4U_WIN(win, info_args).accumulate_ordering = + (MPIDI_CH4U_WIN(win, info_args). + accumulate_ordering | MPIDI_CH4I_ACCU_ORDER_WAR); + else if (!memcmp(token, "waw", 3)) + MPIDI_CH4U_WIN(win, info_args).accumulate_ordering = + (MPIDI_CH4U_WIN(win, info_args). + accumulate_ordering | MPIDI_CH4I_ACCU_ORDER_WAW); + else + MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_ARG, goto fn_fail, "**info"); + + token = (char *) strtok_r(NULL, ",", &savePtr); + } + + if (MPIDI_CH4U_WIN(win, info_args).accumulate_ordering == 0) + MPIDI_CH4U_WIN(win, info_args).accumulate_ordering = save_ordering; + } + else if (!strcmp(curr_ptr->key, "accumulate_ops")) { + /* the default setting is MPIDI_ACCU_SAME_OP_NO_OP */ + if (!strcmp(curr_ptr->value, "same_op")) + MPIDI_CH4U_WIN(win, info_args).accumulate_ops = MPIDI_CH4I_ACCU_SAME_OP; + } + + curr_ptr = curr_ptr->next; + } + + mpi_errno = MPIR_Barrier_impl(win->comm_ptr, &errflag); + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4I_WIN_SET_INFO); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4R_win_init +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4R_win_init(MPI_Aint length, + int disp_unit, + MPIR_Win ** win_ptr, + MPIR_Info * info, + MPIR_Comm * comm_ptr, int create_flavor, int model) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_Win *win = (MPIR_Win *) MPIR_Handle_obj_alloc(&MPIR_Win_mem); + MPIR_ERR_CHKANDSTMT(win == NULL, mpi_errno, MPI_ERR_NO_MEM, goto fn_fail, "**nomem"); + *win_ptr = win; + + memset(&win->dev.ch4u, 0, sizeof(MPIDI_CH4U_win_t)); + win->comm_ptr = comm_ptr; + MPIR_Comm_add_ref(comm_ptr); + + win->errhandler = NULL; + win->base = NULL; + win->size = length; + win->disp_unit = disp_unit; + win->create_flavor = (MPIR_Win_flavor_t) create_flavor; + win->model = (MPIR_Win_model_t) model; + win->copyCreateFlavor = (MPIR_Win_flavor_t) 0; + win->copyModel = (MPIR_Win_model_t) 0; + win->attributes = NULL; + win->comm_ptr = comm_ptr; + win->copyDispUnit = 0; + win->copySize = 0; + MPIDI_CH4U_WIN(win, shared_table) = NULL; + if ((info != NULL) && ((int *) info != (int *) MPI_INFO_NULL)) { + mpi_errno = MPIDI_CH4R_win_set_info(win, info); + MPIR_Assert(mpi_errno == 0); + } + + /* Initialize the info (hint) flags per window */ + MPIDI_CH4U_WIN(win, info_args).no_locks = 0; + MPIDI_CH4U_WIN(win, info_args).accumulate_ordering = (MPIDI_CH4I_ACCU_ORDER_RAR | + MPIDI_CH4I_ACCU_ORDER_RAW | + MPIDI_CH4I_ACCU_ORDER_WAR | + MPIDI_CH4I_ACCU_ORDER_WAW); + MPIDI_CH4U_WIN(win, info_args).accumulate_ops = MPIDI_CH4I_ACCU_SAME_OP_NO_OP; + MPIDI_CH4U_WIN(win, info_args).same_size = 0; + MPIDI_CH4U_WIN(win, info_args).alloc_shared_noncontig = 0; + MPIDI_CH4U_WIN(win, mmap_sz) = 0; + MPIDI_CH4U_WIN(win, mmap_addr) = NULL; + + MPIDI_CH4U_WIN(win, win_id) = MPIDI_CH4U_generate_win_id(comm_ptr); + MPL_HASH_ADD(dev.ch4u.hash_handle, MPIDI_CH4_Global.win_hash, + dev.ch4u.win_id, sizeof(uint64_t), win); + + fn_exit: + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4I_fill_ranks_in_win_grp +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4I_fill_ranks_in_win_grp(MPIR_Win * win_ptr, MPIR_Group * group_ptr, + int *ranks_in_win_grp) +{ + int mpi_errno = MPI_SUCCESS; + int i, *ranks_in_grp; + MPIR_Group *win_grp_ptr; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4I_FILL_RANKS_IN_WIN_GRP); + MPIR_FUNC_VERBOSE_RMA_ENTER(MPID_STATE_CH4I_FILL_RANKS_IN_WIN_GRP); + + ranks_in_grp = (int *) MPL_malloc(group_ptr->size * sizeof(int)); + MPIR_Assert(ranks_in_grp); + for (i = 0; i < group_ptr->size; i++) + ranks_in_grp[i] = i; + + mpi_errno = MPIR_Comm_group_impl(win_ptr->comm_ptr, &win_grp_ptr); + if (mpi_errno != MPI_SUCCESS) + MPIR_ERR_POP(mpi_errno); + + mpi_errno = MPIR_Group_translate_ranks_impl(group_ptr, group_ptr->size, + ranks_in_grp, win_grp_ptr, ranks_in_win_grp); + if (mpi_errno != MPI_SUCCESS) + MPIR_ERR_POP(mpi_errno); + + mpi_errno = MPIR_Group_free_impl(win_grp_ptr); + if (mpi_errno != MPI_SUCCESS) + MPIR_ERR_POP(mpi_errno); + + MPL_free(ranks_in_grp); + + fn_exit: + MPIR_FUNC_VERBOSE_RMA_EXIT(MPID_STATE_CH4I_FILL_RANKS_IN_WIN_GRP); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4I_progress_win_fence +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4I_progress_win_fence(MPIR_Win * win) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4_PROGRESS_WIN_FENCE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4_PROGRESS_WIN_FENCE); + + do { + MPIDI_CH4R_PROGRESS(); + } while (OPA_load_int(&MPIDI_CH4U_WIN(win, outstanding_ops)) != 0); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4_PROGRESS_WIN_FENCE); + return mpi_errno; + fn_fail: + goto fn_exit; +} + + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4R_win_start +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4R_win_start(MPIR_Group * group, int assert, MPIR_Win * win) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_WIN_START); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_WIN_START); + + MPIDI_CH4U_EPOCH_CHECK_TYPE(win, mpi_errno, goto fn_fail); + + MPIR_Group_add_ref(group); + + MPIDI_CH4R_PROGRESS_WHILE(group->size != (int) MPIDI_CH4U_WIN(win, sync).pw.count); + MPIDI_CH4U_WIN(win, sync).pw.count = 0; + + MPIR_ERR_CHKANDJUMP((MPIDI_CH4U_WIN(win, sync).sc.group != NULL), + mpi_errno, MPI_ERR_GROUP, "**group"); + MPIDI_CH4U_WIN(win, sync).sc.group = group; + MPIDI_CH4U_WIN(win, sync).origin_epoch_type = MPIDI_CH4U_EPOTYPE_START; + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_WIN_START); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4R_win_complete +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4R_win_complete(MPIR_Win * win) +{ + int mpi_errno = MPI_SUCCESS; + MPIDI_CH4U_win_cntrl_msg_t msg; + int index, peer; + MPIR_Group *group; + int *ranks_in_win_grp; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_WIN_COMPLETE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_WIN_COMPLETE); + + MPIDI_CH4U_EPOCH_START_CHECK2(win, mpi_errno, goto fn_fail); + + mpi_errno = MPIDI_CH4I_progress_win_fence(win); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + group = MPIDI_CH4U_WIN(win, sync).sc.group; + MPIR_Assert(group != NULL); + + msg.win_id = MPIDI_CH4U_WIN(win, win_id); + msg.origin_rank = win->comm_ptr->rank; + msg.type = MPIDI_CH4U_WIN_COMPLETE; + + ranks_in_win_grp = (int *) MPL_malloc(sizeof(int) * group->size); + MPIR_Assert(ranks_in_win_grp); + + mpi_errno = MPIDI_CH4I_fill_ranks_in_win_grp(win, group, ranks_in_win_grp); + if (mpi_errno != MPI_SUCCESS) + MPIR_ERR_POP(mpi_errno); + + for (index = 0; index < group->size; ++index) { + peer = ranks_in_win_grp[index]; + mpi_errno = MPIDI_NM_inject_am_hdr(peer, win->comm_ptr, + MPIDI_CH4U_WIN_CTRL, &msg, sizeof(msg), NULL); + if (mpi_errno != MPI_SUCCESS) + MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, goto fn_fail, "**rmasync"); + } + + MPL_free(ranks_in_win_grp); + MPIDI_CH4U_EPOCH_TARGET_EVENT(win); + MPIR_Group_release(MPIDI_CH4U_WIN(win, sync).sc.group); + MPIDI_CH4U_WIN(win, sync).sc.group = NULL; + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_WIN_COMPLETE); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4R_win_post +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4R_win_post(MPIR_Group * group, int assert, MPIR_Win * win) +{ + int mpi_errno = MPI_SUCCESS; + MPIDI_CH4U_win_cntrl_msg_t msg; + int index, peer; + int *ranks_in_win_grp; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4U_WIN_POST); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4U_WIN_POST); + + MPIDI_CH4U_EPOCH_POST_CHECK(win, mpi_errno, goto fn_fail); + + MPIR_Group_add_ref(group); + MPIR_ERR_CHKANDJUMP((MPIDI_CH4U_WIN(win, sync).pw.group != NULL), + mpi_errno, MPI_ERR_GROUP, "**group"); + + MPIDI_CH4U_WIN(win, sync).pw.group = group; + MPIR_Assert(group != NULL); + + msg.win_id = MPIDI_CH4U_WIN(win, win_id); + msg.origin_rank = win->comm_ptr->rank; + msg.type = MPIDI_CH4U_WIN_POST; + + ranks_in_win_grp = (int *) MPL_malloc(sizeof(int) * group->size); + MPIR_Assert(ranks_in_win_grp); + + mpi_errno = MPIDI_CH4I_fill_ranks_in_win_grp(win, group, ranks_in_win_grp); + if (mpi_errno != MPI_SUCCESS) + MPIR_ERR_POP(mpi_errno); + + for (index = 0; index < group->size; ++index) { + peer = ranks_in_win_grp[index]; + mpi_errno = MPIDI_NM_inject_am_hdr(peer, win->comm_ptr, + MPIDI_CH4U_WIN_CTRL, &msg, sizeof(msg), NULL); + if (mpi_errno != MPI_SUCCESS) + MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, goto fn_fail, "**rmasync"); + } + + MPL_free(ranks_in_win_grp); + MPIDI_CH4U_WIN(win, sync).target_epoch_type = MPIDI_CH4U_EPOTYPE_POST; + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4U_WIN_POST); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4R_win_wait +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4R_win_wait(MPIR_Win * win) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Group *group; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4I_WIN_WAIT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4I_WIN_WAIT); + + MPIDI_CH4U_EPOCH_TARGET_CHECK(win, MPIDI_CH4U_EPOTYPE_POST, mpi_errno, goto fn_fail); + group = MPIDI_CH4U_WIN(win, sync).pw.group; + MPIDI_CH4R_PROGRESS_WHILE(group->size != (int) MPIDI_CH4U_WIN(win, sync).sc.count); + + MPIDI_CH4U_WIN(win, sync).sc.count = 0; + MPIDI_CH4U_WIN(win, sync).pw.group = NULL; + MPIR_Group_release(group); + MPIDI_CH4U_EPOCH_ORIGIN_EVENT(win); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4I_WIN_WAIT); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4R_win_test +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4R_win_test(MPIR_Win * win, int *flag) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4I_WIN_TEST); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4I_WIN_TEST); + + MPIDI_CH4U_EPOCH_TARGET_CHECK(win, MPIDI_CH4U_EPOTYPE_POST, mpi_errno, goto fn_fail); + + MPIR_Group *group; + group = MPIDI_CH4U_WIN(win, sync).pw.group; + + if (group->size == (int) MPIDI_CH4U_WIN(win, sync).sc.count) { + MPIDI_CH4U_WIN(win, sync).sc.count = 0; + MPIDI_CH4U_WIN(win, sync).pw.group = NULL; + *flag = 1; + MPIR_Group_release(group); + MPIDI_CH4U_EPOCH_ORIGIN_EVENT(win); + } + else { + MPIDI_CH4R_PROGRESS(); + *flag = 0; + } + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4I_WIN_TEST); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4R_win_lock +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4R_win_lock(int lock_type, int rank, int assert, MPIR_Win * win) +{ + int mpi_errno = MPI_SUCCESS; + unsigned locked; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4I_WIN_LOCK); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4I_WIN_LOCK); + + MPIDI_CH4U_win_sync_lock *slock = &MPIDI_CH4U_WIN(win, sync).lock; + if (rank == MPI_PROC_NULL) + goto fn_exit0; + + MPIDI_CH4U_EPOCH_CHECK_TYPE(win, mpi_errno, goto fn_fail); + + MPIDI_CH4U_win_cntrl_msg_t msg; + msg.win_id = MPIDI_CH4U_WIN(win, win_id); + msg.origin_rank = win->comm_ptr->rank; + msg.type = MPIDI_CH4U_WIN_LOCK; + msg.lock_type = lock_type; + + locked = slock->remote.locked + 1; + mpi_errno = MPIDI_NM_inject_am_hdr(rank, win->comm_ptr, + MPIDI_CH4U_WIN_CTRL, &msg, sizeof(msg), NULL); + if (mpi_errno != MPI_SUCCESS) + MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, goto fn_fail, "**rmasync"); + MPIDI_CH4R_PROGRESS_WHILE(slock->remote.locked != locked); + + fn_exit0: + MPIDI_CH4U_WIN(win, sync).origin_epoch_type = MPIDI_CH4U_EPOTYPE_LOCK; + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4I_WIN_LOCK); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4R_win_unlock +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4R_win_unlock(int rank, MPIR_Win * win) +{ + int mpi_errno = MPI_SUCCESS; + unsigned unlocked; + MPIDI_CH4U_win_cntrl_msg_t msg; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4I_WIN_UNLOCK); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4I_WIN_UNLOCK); + if (rank == MPI_PROC_NULL) + goto fn_exit0; + + MPIDI_CH4U_EPOCH_ORIGIN_CHECK(win, MPIDI_CH4U_EPOTYPE_LOCK, mpi_errno, return mpi_errno); + + mpi_errno = MPIDI_CH4I_progress_win_fence(win); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + msg.win_id = MPIDI_CH4U_WIN(win, win_id); + msg.origin_rank = win->comm_ptr->rank; + msg.type = MPIDI_CH4U_WIN_UNLOCK; + unlocked = MPIDI_CH4U_WIN(win, sync).lock.remote.locked - 1; + + mpi_errno = MPIDI_NM_inject_am_hdr(rank, win->comm_ptr, + MPIDI_CH4U_WIN_CTRL, &msg, sizeof(msg), NULL); + if (mpi_errno != MPI_SUCCESS) + MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, goto fn_fail, "**rmasync"); + + MPIDI_CH4R_PROGRESS_WHILE(MPIDI_CH4U_WIN(win, sync).lock.remote.locked != unlocked); + fn_exit0: + + if (!MPIDI_CH4U_WIN(win, sync).lock.remote.locked) { + MPIDI_CH4U_WIN(win, sync).origin_epoch_type = MPIDI_CH4U_EPOTYPE_NONE; + MPIDI_CH4U_WIN(win, sync).target_epoch_type = MPIDI_CH4U_EPOTYPE_NONE; + } + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4I_WIN_UNLOCK); + return mpi_errno; + fn_fail: + goto fn_exit; + +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4R_win_get_info +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4R_win_get_info(MPIR_Win * win, MPIR_Info ** info_p_p) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4I_WIN_GET_INFO); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4I_WIN_GET_INFO); + + mpi_errno = MPIR_Info_alloc(info_p_p); + MPIR_Assert(mpi_errno == MPI_SUCCESS); + + if (MPIDI_CH4U_WIN(win, info_args).no_locks) + mpi_errno = MPIR_Info_set_impl(*info_p_p, "no_locks", "true"); + else + mpi_errno = MPIR_Info_set_impl(*info_p_p, "no_locks", "false"); + + MPIR_Assert(mpi_errno == MPI_SUCCESS); + + { +#define BUFSIZE 32 + char buf[BUFSIZE]; + int c = 0; + + if (MPIDI_CH4U_WIN(win, info_args).accumulate_ordering & MPIDI_CH4I_ACCU_ORDER_RAR) + c += snprintf(buf + c, BUFSIZE - c, "%srar", (c > 0) ? "," : ""); + + if (MPIDI_CH4U_WIN(win, info_args).accumulate_ordering & MPIDI_CH4I_ACCU_ORDER_RAW) + c += snprintf(buf + c, BUFSIZE - c, "%sraw", (c > 0) ? "," : ""); + + if (MPIDI_CH4U_WIN(win, info_args).accumulate_ordering & MPIDI_CH4I_ACCU_ORDER_WAR) + c += snprintf(buf + c, BUFSIZE - c, "%swar", (c > 0) ? "," : ""); + + if (MPIDI_CH4U_WIN(win, info_args).accumulate_ordering & MPIDI_CH4I_ACCU_ORDER_WAW) + c += snprintf(buf + c, BUFSIZE - c, "%swaw", (c > 0) ? "," : ""); + + if (c == 0) { + memcpy(&buf[0], "not set ", 10); + } + + MPIR_Info_set_impl(*info_p_p, "accumulate_ordering", buf); + MPIR_Assert(mpi_errno == MPI_SUCCESS); +#undef BUFSIZE + } + + if (MPIDI_CH4U_WIN(win, info_args).accumulate_ops == MPIDI_CH4I_ACCU_SAME_OP) + mpi_errno = MPIR_Info_set_impl(*info_p_p, "accumulate_ops", "same_op"); + else + mpi_errno = MPIR_Info_set_impl(*info_p_p, "accumulate_ops", "same_op_no_op"); + + MPIR_Assert(mpi_errno == MPI_SUCCESS); + + if (win->create_flavor == MPI_WIN_FLAVOR_SHARED) { + if (MPIDI_CH4U_WIN(win, info_args).alloc_shared_noncontig) + mpi_errno = MPIR_Info_set_impl(*info_p_p, "alloc_shared_noncontig", "true"); + else + mpi_errno = MPIR_Info_set_impl(*info_p_p, "alloc_shared_noncontig", "false"); + + MPIR_Assert(mpi_errno == MPI_SUCCESS); + } + else if (win->create_flavor == MPI_WIN_FLAVOR_ALLOCATE) { + if (MPIDI_CH4U_WIN(win, info_args).same_size) + mpi_errno = MPIR_Info_set_impl(*info_p_p, "same_size", "true"); + else + mpi_errno = MPIR_Info_set_impl(*info_p_p, "same_size", "false"); + + MPIR_Assert(mpi_errno == MPI_SUCCESS); + } + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4I_WIN_GET_INFO); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4R_win_finalize +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4R_win_finalize(MPIR_Win ** win_ptr) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Win *win = *win_ptr; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4I_WIN_FINALIZE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4I_WIN_FINALIZE); + + if (win->create_flavor == MPI_WIN_FLAVOR_ALLOCATE && win->base) { + if (MPIDI_CH4U_WIN(win, mmap_sz) > 0) + munmap(MPIDI_CH4U_WIN(win, mmap_addr), MPIDI_CH4U_WIN(win, mmap_sz)); + else if (MPIDI_CH4U_WIN(win, mmap_sz) == -1) + MPL_free(win->base); + } + + if (win->create_flavor == MPI_WIN_FLAVOR_SHARED) { + if (MPIDI_CH4U_WIN(win, mmap_addr)) + munmap(MPIDI_CH4U_WIN(win, mmap_addr), MPIDI_CH4U_WIN(win, mmap_sz)); + MPL_free(MPIDI_CH4U_WIN(win, sizes)); + } + + if (MPIDI_CH4U_WIN(win, lockQ)) { + MPL_free(MPIDI_CH4U_WIN(win, lockQ)); + MPIDI_CH4U_WIN(win, lockQ) = NULL; + } + + MPL_HASH_DELETE(dev.ch4u.hash_handle, MPIDI_CH4_Global.win_hash, win); + + if (win->create_flavor == MPI_WIN_FLAVOR_SHARED) { + MPL_free(MPIDI_CH4U_WIN(win, shared_table)); + } + + MPIR_Comm_release(win->comm_ptr); + MPIR_Handle_obj_free(&MPIR_Win_mem, win); + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4I_WIN_FINALIZE); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4R_win_free +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4R_win_free(MPIR_Win ** win_ptr) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Errflag_t errflag = MPIR_ERR_NONE; + MPIR_Win *win = *win_ptr; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4I_WIN_FREE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4I_WIN_FREE); + + MPIDI_CH4U_EPOCH_FREE_CHECK(win, mpi_errno, goto fn_fail); + mpi_errno = MPIR_Barrier_impl(win->comm_ptr, &errflag); + if (mpi_errno != MPI_SUCCESS) + goto fn_fail; + + MPIDI_CH4R_win_finalize(win_ptr); + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4I_WIN_FREE); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4R_win_fence +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4R_win_fence(int massert, MPIR_Win * win) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Errflag_t errflag = MPIR_ERR_NONE; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4I_WIN_FENCE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4I_WIN_FENCE); + + MPIDI_CH4U_EPOCH_FENCE_CHECK(win, mpi_errno, goto fn_fail); + mpi_errno = MPIDI_CH4I_progress_win_fence(win); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + MPIDI_CH4U_EPOCH_FENCE_EVENT(win, massert); + + /* + * We always make a barrier even if MPI_MODE_NOPRECEDE is specified. + * This is necessary because we no longer defer executions of RMA ops + * until synchronization calls as CH3 did. Otherwise, the code like + * this won't work correctly (cf. f77/rma/wingetf) + * + * Rank 0 Rank 1 + * ---- ---- + * Store to local mem in window + * MPI_Win_fence(MODE_NOPRECEDE) MPI_Win_fence(MODE_NOPRECEDE) + * MPI_Get(from rank 1) + */ + mpi_errno = MPIR_Barrier_impl(win->comm_ptr, &errflag); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4I_WIN_FENCE); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4R_win_create +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4R_win_create(void *base, + MPI_Aint length, + int disp_unit, + MPIR_Info * info, MPIR_Comm * comm_ptr, MPIR_Win ** win_ptr) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Errflag_t errflag = MPIR_ERR_NONE; + MPIR_Win *win; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4I_WIN_CREATE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4I_WIN_CREATE); + + mpi_errno = MPIDI_CH4R_win_init(length, + disp_unit, + win_ptr, + info, comm_ptr, MPI_WIN_FLAVOR_CREATE, MPI_WIN_UNIFIED); + + if (mpi_errno != MPI_SUCCESS) + goto fn_fail; + + win = *win_ptr; + win->base = base; + + mpi_errno = MPIR_Barrier_impl(comm_ptr, &errflag); + + if (mpi_errno != MPI_SUCCESS) + goto fn_fail; + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4I_WIN_CREATE); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4R_win_attach +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4R_win_attach(MPIR_Win * win, void *base, MPI_Aint size) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4I_WIN_ATTACH); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4I_WIN_ATTACH); + + MPIR_ERR_CHKANDSTMT((win->create_flavor != MPI_WIN_FLAVOR_DYNAMIC), mpi_errno, + MPI_ERR_RMA_FLAVOR, goto fn_fail, "**rmaflavor"); + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4I_WIN_ATTACH); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4R_win_allocate_shared +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4R_win_allocate_shared(MPI_Aint size, + int disp_unit, + MPIR_Info * info_ptr, + MPIR_Comm * comm_ptr, + void **base_ptr, MPIR_Win ** win_ptr) +{ + int i = 0, fd = -1, rc, first = 0, mpi_errno = MPI_SUCCESS; + MPIR_Errflag_t errflag = MPIR_ERR_NONE; + void *baseP = NULL; + MPIR_Win *win = NULL; + ssize_t total_size = 0LL; + MPI_Aint size_out = 0; + MPIDI_CH4U_win_shared_info_t *shared_table = NULL; + char shm_key[64]; + void *map_ptr; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4I_WIN_ALLOCATE_SHARED); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4I_WIN_ALLOCATE_SHARED); + + mpi_errno = MPIDI_CH4R_win_init(size, disp_unit, win_ptr, info_ptr, comm_ptr, + MPI_WIN_FLAVOR_SHARED, MPI_WIN_UNIFIED); + + win = *win_ptr; + MPIDI_CH4U_WIN(win, shared_table) = + (MPIDI_CH4U_win_shared_info_t *) MPL_malloc(sizeof(MPIDI_CH4U_win_shared_info_t) * + comm_ptr->local_size); + shared_table = MPIDI_CH4U_WIN(win, shared_table); + shared_table[comm_ptr->rank].size = size; + shared_table[comm_ptr->rank].disp_unit = disp_unit; + + mpi_errno = MPIR_Allgather_impl(MPI_IN_PLACE, + 0, + MPI_DATATYPE_NULL, + shared_table, + sizeof(MPIDI_CH4U_win_shared_info_t), + MPI_BYTE, comm_ptr, &errflag); + if (mpi_errno != MPI_SUCCESS) + goto fn_fail; + + /* No allreduce here because this is a shared memory domain + * and should be a relatively small number of processes + * and a non performance sensitive API. + */ + for (i = 0; i < comm_ptr->local_size; i++) + total_size += shared_table[i].size; + + if (total_size == 0) + goto fn_zero; + + sprintf(shm_key, "/mpi-%X-%" PRIx64, MPIDI_CH4_Global.jobid, MPIDI_CH4U_WIN(win, win_id)); + + rc = shm_open(shm_key, O_CREAT | O_EXCL | O_RDWR, 0600); + first = (rc != -1); + + if (!first) { + rc = shm_open(shm_key, O_RDWR, 0); + + if (rc == -1) { + shm_unlink(shm_key); + MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_NO_MEM, goto fn_fail, "**nomem"); + } + } + + /* Make the addresses symmetric by using MAP_FIXED */ + size_t page_sz, mapsize; + + mapsize = MPIDI_CH4R_get_mapsize(total_size, &page_sz); + fd = rc; + rc = ftruncate(fd, mapsize); + + if (rc == -1) { + close(fd); + + if (first) + shm_unlink(shm_key); + + MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_NO_MEM, goto fn_fail, "**nomem"); + } + + if (comm_ptr->rank == 0) { + map_ptr = MPIDI_CH4R_generate_random_addr(mapsize); + map_ptr = mmap(map_ptr, mapsize, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_FIXED, fd, 0); + + if (map_ptr == NULL || map_ptr == MAP_FAILED) { + close(fd); + + if (first) + shm_unlink(shm_key); + + MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_NO_MEM, goto fn_fail, "**nomem"); + } + + mpi_errno = MPIR_Bcast_impl(&map_ptr, 1, MPI_UNSIGNED_LONG, 0, comm_ptr, &errflag); + + if (mpi_errno != MPI_SUCCESS) + goto fn_fail; + + MPIDI_CH4U_WIN(win, mmap_addr) = map_ptr; + MPIDI_CH4U_WIN(win, mmap_sz) = mapsize; + } + else { + mpi_errno = MPIR_Bcast_impl(&map_ptr, 1, MPI_UNSIGNED_LONG, 0, comm_ptr, &errflag); + + if (mpi_errno != MPI_SUCCESS) + goto fn_fail; + + rc = MPIDI_CH4R_check_maprange_ok(map_ptr, mapsize); + /* If we hit this assert, we need to iterate + * trying more addresses + */ + MPIR_Assert(rc == 1); + map_ptr = mmap(map_ptr, mapsize, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_FIXED, fd, 0); + MPIDI_CH4U_WIN(win, mmap_addr) = map_ptr; + MPIDI_CH4U_WIN(win, mmap_sz) = mapsize; + + if (map_ptr == NULL || map_ptr == MAP_FAILED) { + close(fd); + + if (first) + shm_unlink(shm_key); + + MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_NO_MEM, goto fn_fail, "**nomem"); + } + } + + /* Scan for my offset into the buffer */ + /* Could use exscan if this is expensive at scale */ + for (i = 0; i < comm_ptr->rank; i++) + size_out += shared_table[i].size; + + fn_zero: + + baseP = (size == 0) ? NULL : (void *) ((char *) map_ptr + size_out); + win->base = baseP; + win->size = size; + + *(void **) base_ptr = (void *) win->base; + mpi_errno = MPIR_Barrier_impl(comm_ptr, &errflag); + + if (fd >= 0) + close(fd); + + if (first) + shm_unlink(shm_key); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4I_WIN_ALLOCATE_SHARED); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4R_win_detach +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4R_win_detach(MPIR_Win * win, const void *base) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4I_WIN_DETACH); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4I_WIN_DETACH); + MPIR_ERR_CHKANDSTMT((win->create_flavor != MPI_WIN_FLAVOR_DYNAMIC), mpi_errno, + MPI_ERR_RMA_FLAVOR, goto fn_fail, "**rmaflavor"); + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4I_WIN_DETACH); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4R_win_shared_query +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4R_win_shared_query(MPIR_Win * win, + int rank, + MPI_Aint * size, int *disp_unit, void *baseptr) +{ + int mpi_errno = MPI_SUCCESS; + uintptr_t base = (uintptr_t) MPIDI_CH4U_WIN(win, mmap_addr); + int offset = rank, i; + MPIDI_CH4U_win_shared_info_t *shared_table = MPIDI_CH4U_WIN(win, shared_table); + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4I_WIN_SHARED_QUERY); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4I_WIN_SHARED_QUERY); + + if (rank < 0) + offset = 0; + *size = shared_table[offset].size; + *disp_unit = shared_table[offset].disp_unit; + if (*size > 0) { + for (i = 0; i < offset; i++) + base += shared_table[i].size; + *(void **) baseptr = (void *) base; + } + else + *(void **) baseptr = NULL; + + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4I_WIN_SHARED_QUERY); + return mpi_errno; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4R_win_allocate +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4R_win_allocate(MPI_Aint size, + int disp_unit, + MPIR_Info * info, + MPIR_Comm * comm, void *baseptr, MPIR_Win ** win_ptr) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_Errflag_t errflag = MPIR_ERR_NONE; + void *baseP; + MPIR_Win *win; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4I_WIN_ALLOCATE); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4I_WIN_ALLOCATE); + + mpi_errno = MPIDI_CH4R_win_init(size, disp_unit, win_ptr, info, comm, + MPI_WIN_FLAVOR_ALLOCATE, MPI_WIN_UNIFIED); + + if (mpi_errno != MPI_SUCCESS) + goto fn_fail; + + mpi_errno = MPIDI_CH4R_get_symmetric_heap(size, comm, &baseP, *win_ptr); + + if (mpi_errno != MPI_SUCCESS) + goto fn_fail; + + win = *win_ptr; + win->base = baseP; + + *(void **) baseptr = (void *) win->base; + mpi_errno = MPIR_Barrier_impl(comm, &errflag); + + if (mpi_errno != MPI_SUCCESS) + goto fn_fail; + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4I_WIN_ALLOCATE); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4R_win_flush +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4R_win_flush(int rank, MPIR_Win * win) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4I_WIN_FLUSH); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4I_WIN_FLUSH); + + MPIDI_CH4U_EPOCH_LOCK_CHECK(win, mpi_errno, goto fn_fail); + mpi_errno = MPIDI_CH4I_progress_win_fence(win); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4I_WIN_FLUSH); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4R_win_flush_local_all +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4R_win_flush_local_all(MPIR_Win * win) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4I_WIN_FLUSH_LOCAL_ALL); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4I_WIN_FLUSH_LOCAL_ALL); + + MPIDI_CH4U_EPOCH_LOCK_CHECK(win, mpi_errno, goto fn_fail); + + mpi_errno = MPIDI_CH4I_progress_win_fence(win); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4I_WIN_FLUSH_LOCAL_ALL); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4R_win_unlock_all +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4R_win_unlock_all(MPIR_Win * win) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4I_WIN_UNLOCK_ALL); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4I_WIN_UNLOCK_ALL); + int i; + MPIDI_CH4U_win_lock_info *lockQ; + + MPIDI_CH4U_EPOCH_ORIGIN_CHECK(win, MPIDI_CH4U_EPOTYPE_LOCK_ALL, mpi_errno, goto fn_exit); + + mpi_errno = MPIDI_CH4I_progress_win_fence(win); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + MPIR_Assert(MPIDI_CH4U_WIN(win, lockQ) != NULL); + lockQ = (MPIDI_CH4U_win_lock_info *) MPIDI_CH4U_WIN(win, lockQ); + + for (i = 0; i < win->comm_ptr->local_size; i++) { + + MPIDI_CH4U_win_cntrl_msg_t msg; + msg.win_id = MPIDI_CH4U_WIN(win, win_id); + msg.origin_rank = win->comm_ptr->rank; + msg.type = MPIDI_CH4U_WIN_UNLOCKALL; + + lockQ[i].done = 0; + lockQ[i].peer = i; + lockQ[i].win = win; + + mpi_errno = MPIDI_NM_inject_am_hdr(i, win->comm_ptr, + MPIDI_CH4U_WIN_CTRL, &msg, sizeof(msg), NULL); + if (mpi_errno != MPI_SUCCESS) + MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, goto fn_fail, "**rmasync"); + + if (MPIDI_CH4U_WIN(win, sync).lock.remote.allLocked == 1) + lockQ[i].done = 1; + } + + MPIDI_CH4R_PROGRESS_WHILE(MPIDI_CH4U_WIN(win, sync).lock.remote.allLocked); + + MPIDI_CH4U_WIN(win, sync).origin_epoch_type = MPIDI_CH4U_EPOTYPE_NONE; + MPIDI_CH4U_WIN(win, sync).target_epoch_type = MPIDI_CH4U_EPOTYPE_NONE; + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4I_WIN_UNLOCK_ALL); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4R_win_create_dynamic +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4R_win_create_dynamic(MPIR_Info * info, + MPIR_Comm * comm, MPIR_Win ** win_ptr) +{ + int mpi_errno = MPI_SUCCESS; + int rc = MPI_SUCCESS; + MPIR_Errflag_t errflag = MPIR_ERR_NONE; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4I_WIN_CREATE_DYNAMIC); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4I_WIN_CREATE_DYNAMIC); + + MPIR_Win *win; + + rc = MPIDI_CH4R_win_init(0, 1, win_ptr, info, comm, MPI_WIN_FLAVOR_DYNAMIC, MPI_WIN_UNIFIED); + + if (rc != MPI_SUCCESS) + goto fn_fail; + + win = *win_ptr; + win->base = MPI_BOTTOM; + + + if (rc != MPI_SUCCESS) + goto fn_fail; + + mpi_errno = MPIR_Barrier_impl(comm, &errflag); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4I_WIN_CREATE_DYNAMIC); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4R_win_flush_local +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4R_win_flush_local(int rank, MPIR_Win * win) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4I_WIN_FLUSH_LOCAL); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4I_WIN_FLUSH_LOCAL); + + MPIDI_CH4U_EPOCH_LOCK_CHECK(win, mpi_errno, goto fn_fail); + + mpi_errno = MPIDI_CH4I_progress_win_fence(win); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4I_WIN_FLUSH_LOCAL); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4R_win_sync +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4R_win_sync(MPIR_Win * win) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4I_WIN_SYNC); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4I_WIN_SYNC); + + MPIDI_CH4U_EPOCH_LOCK_CHECK(win, mpi_errno, goto fn_fail); + OPA_read_write_barrier(); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4I_WIN_SYNC); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4R_win_flush_all +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4R_win_flush_all(MPIR_Win * win) +{ + int mpi_errno = MPI_SUCCESS; + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4I_WIN_FLUSH_ALL); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4I_WIN_FLUSH_ALL); + + MPIDI_CH4U_EPOCH_LOCK_CHECK(win, mpi_errno, goto fn_fail); + + mpi_errno = MPIDI_CH4I_progress_win_fence(win); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4I_WIN_FLUSH_ALL); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#undef FUNCNAME +#define FUNCNAME MPIDI_CH4R_win_lock_all +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPIDI_CH4R_win_lock_all(int assert, MPIR_Win * win) +{ + int mpi_errno = MPI_SUCCESS; + + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CH4I_WIN_LOCK_ALL); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CH4I_WIN_LOCK_ALL); + + MPIDI_CH4U_EPOCH_CHECK_TYPE(win, mpi_errno, goto fn_fail); + + int size; + size = win->comm_ptr->local_size; + + if (!MPIDI_CH4U_WIN(win, lockQ)) { + MPIDI_CH4U_WIN(win, lockQ) = + (MPIDI_CH4U_win_lock_info *) MPL_calloc(size, sizeof(MPIDI_CH4U_win_lock_info)); + MPIR_Assert(MPIDI_CH4U_WIN(win, lockQ) != NULL); + } + + MPIDI_CH4U_win_lock_info *lockQ; + lockQ = (MPIDI_CH4U_win_lock_info *) MPIDI_CH4U_WIN(win, lockQ); + int i; + + for (i = 0; i < size; i++) { + MPIDI_CH4U_win_cntrl_msg_t msg; + msg.win_id = MPIDI_CH4U_WIN(win, win_id); + msg.origin_rank = win->comm_ptr->rank; + msg.type = MPIDI_CH4U_WIN_LOCKALL; + msg.lock_type = MPI_LOCK_SHARED; + + lockQ[i].done = 0; + lockQ[i].peer = i; + lockQ[i].win = win; + lockQ[i].lock_type = MPI_LOCK_SHARED; + + mpi_errno = MPIDI_NM_inject_am_hdr(i, win->comm_ptr, + MPIDI_CH4U_WIN_CTRL, &msg, sizeof(msg), NULL); + if (mpi_errno != MPI_SUCCESS) + MPIR_ERR_SETANDSTMT(mpi_errno, MPI_ERR_RMA_SYNC, goto fn_fail, "**rmasync"); + + if (MPIDI_CH4U_WIN(win, sync).lock.remote.allLocked == 1) + lockQ[i].done = 1; + } + + MPIDI_CH4R_PROGRESS_WHILE(size != (int) MPIDI_CH4U_WIN(win, sync).lock.remote.allLocked); + MPIDI_CH4U_WIN(win, sync).origin_epoch_type = MPIDI_CH4U_EPOTYPE_LOCK_ALL; + + fn_exit: + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CH4I_WIN_LOCK_ALL); + return mpi_errno; + fn_fail: + goto fn_exit; +} + +#endif /* CH4R_WIN_H_INCLUDED */ diff --git a/src/mpid/ch4/src/mpid_ch4_net_array.c.in b/src/mpid/ch4/src/mpid_ch4_net_array.c.in new file mode 100644 index 0000000..8765f39 --- /dev/null +++ b/src/mpid/ch4/src/mpid_ch4_net_array.c.in @@ -0,0 +1,28 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ + +#include <mpidimpl.h> + +/* forward declaration of funcs structs defined in network modules */ +extern MPIDI_NM_funcs_t @ch4_nets_func_decl@; +extern MPIDI_NM_native_funcs_t @ch4_nets_native_func_decl@; + +#ifndef NETMOD_DIRECT +MPIDI_NM_funcs_t *MPIDI_NM_funcs[@ch4_nets_array_sz@] = { @ch4_nets_func_array@ }; +MPIDI_NM_native_funcs_t *MPIDI_NM_native_funcs[@ch4_nets_array_sz@] = + { @ch4_nets_native_func_array@ }; +#else +MPIDI_NM_funcs_t *MPIDI_NM_funcs[@ch4_nets_array_sz@] = { 0 }; +MPIDI_NM_native_funcs_t *MPIDI_NM_native_funcs[@ch4_nets_array_sz@] = { 0 }; +#endif +int MPIDI_num_netmods = @ch4_nets_array_sz@; +char MPIDI_NM_strings[@ch4_nets_array_sz@][MPIDI_MAX_NETMOD_STRING_LEN] = + { @ch4_nets_strings@ }; diff --git a/src/mpid/ch4/src/mpid_ch4_shm_array.c.in b/src/mpid/ch4/src/mpid_ch4_shm_array.c.in new file mode 100644 index 0000000..6bce704 --- /dev/null +++ b/src/mpid/ch4/src/mpid_ch4_shm_array.c.in @@ -0,0 +1,32 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + * + * Portions of this code were written by Intel Corporation. + * Copyright (C) 2011-2016 Intel Corporation. Intel provides this material + * to Argonne National Laboratory subject to Software Grant and Corporate + * Contributor License Agreement dated February 8, 2012. + */ + +#include <mpidimpl.h> + +#ifdef MPIDI_BUILD_CH4_SHM + +/* forward declaration of funcs structs defined in network modules */ +extern MPIDI_SHM_funcs_t @ch4_shm_func_decl@; +extern MPIDI_SHM_native_funcs_t @ch4_shm_native_func_decl@; + +#ifndef SHM_DIRECT +MPIDI_SHM_funcs_t *MPIDI_SHM_funcs[@ch4_shm_array_sz@] = { @ch4_shm_func_array@ }; +MPIDI_SHM_native_funcs_t *MPIDI_SHM_native_funcs[@ch4_shm_array_sz@] = + { @ch4_shm_native_func_array@ }; +#else +MPIDI_SHM_funcs_t *MPIDI_SHM_funcs[@ch4_shm_array_sz@] = { 0 }; +MPIDI_SHM_native_funcs_t *MPIDI_SHM_native_funcs[@ch4_shm_array_sz@] = { 0 }; +#endif +int MPIDI_num_shms = @ch4_shm_array_sz@; +char MPIDI_SHM_strings[@ch4_shm_array_sz@][MPIDI_MAX_SHM_STRING_LEN] = + { @ch4_shm_strings@ }; + +#endif diff --git a/src/mpid/ch4/subconfigure.m4 b/src/mpid/ch4/subconfigure.m4 new file mode 100644 index 0000000..f4ca5c5 --- /dev/null +++ b/src/mpid/ch4/subconfigure.m4 @@ -0,0 +1,391 @@ +[#] start of __file__ +dnl MPICH_SUBCFG_BEFORE=src/mpid/common/sched +dnl MPICH_SUBCFG_BEFORE=src/mpid/common/datatype +dnl MPICH_SUBCFG_BEFORE=src/mpid/common/thread + +dnl _PREREQ handles the former role of mpichprereq, setup_device, etc +[#] expansion is: PAC_SUBCFG_PREREQ_[]PAC_SUBCFG_AUTO_SUFFIX +AC_DEFUN([PAC_SUBCFG_PREREQ_]PAC_SUBCFG_AUTO_SUFFIX,[ +AM_CONDITIONAL([BUILD_CH4],[test "$device_name" = "ch4"]) + +# the CH4 device depends on the common NBC scheduler code +build_mpid_common_sched=yes +build_mpid_common_datatype=yes +build_mpid_common_thread=yes + +MPID_MAX_THREAD_LEVEL=MPI_THREAD_MULTIPLE +MPID_MAX_PROCESSOR_NAME=128 +MPID_MAX_ERROR_STRING=512 + +AM_COND_IF([BUILD_CH4],[ +AC_MSG_NOTICE([RUNNING PREREQ FOR CH4 DEVICE]) + +# $device_args - contains the netmods +if test -z "${device_args}" ; then + ch4_netmods="ofi" +else + ch4_netmods=`echo ${device_args} | sed -e 's/,/ /g'` +fi +export ch4_netmods + +# +# reset DEVICE so that it (a) always includes the channel name, and (b) does not include channel options +# +DEVICE="${device_name}:${ch4_netmods}" + +ch4_nets_func_decl="" +ch4_nets_native_func_decl="" +ch4_nets_func_array="" +ch4_nets_native_func_array="" +ch4_nets_strings="" +net_index=0 +for net in $ch4_netmods ; do + if test ! -d $srcdir/src/mpid/ch4/netmod/${net} ; then + AC_MSG_ERROR([Network module ${net} is unknown "$srcdir/src/mpid/ch4/netmod/${net}"]) + fi + net_macro=`echo $net | tr 'abcdefghijklmnopqrstuvwxyz' 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'` + net_macro="MPIDI_CH4_${net_macro}" + + if test -z "$ch4_nets_array" ; then + ch4_nets_array="$net_macro" + else + ch4_nets_array="$ch4_nets_array, $net_macro" + fi + + if test -z "$ch4_nets_func_decl" ; then + ch4_nets_func_decl="MPIDI_NM_${net}_funcs" + else + ch4_nets_func_decl="${ch4_nets_func_decl}, MPIDI_NM_${net}_funcs" + fi + + if test -z "$ch4_nets_native_func_decl" ; then + ch4_nets_native_func_decl="MPIDI_NM_native_${net}_funcs" + else + ch4_nets_native_func_decl="${ch4_nets_native_func_decl}, MPIDI_NM_native_${net}_funcs" + fi + + if test -z "$ch4_nets_func_array" ; then + ch4_nets_func_array="&MPIDI_NM_${net}_funcs" + else + ch4_nets_func_array="${ch4_nets_func_array}, &MPIDI_NM_${net}_funcs" + fi + + if test -z "$ch4_nets_native_func_array" ; then + ch4_nets_native_func_array="&MPIDI_NM_native_${net}_funcs" + else + ch4_nets_native_func_array="${ch4_nets_native_func_array}, &MPIDI_NM_native_${net}_funcs" + fi + + if test -z "$ch4_nets_strings" ; then + ch4_nets_strings="\"$net\"" + else + ch4_nets_strings="$ch4_nets_strings, \"$net\"" + fi + + if test -z "$ch4_netmod_pre_include" ; then + ch4_netmod_pre_include="#include \"../netmod/${net}/${net}_pre.h\"" + else + ch4_netmod_pre_include="${ch4_netmod_pre_include} +#include \"../netmod/${net}/${net}_pre.h\"" + fi + + net_upper=`echo ${net} | tr 'abcdefghijklmnopqrstuvwxyz' 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'` + if test -z "$ch4_netmod_amrequest_decl" ; then + ch4_netmod_amrequest_decl="MPIDI_${net_upper}_am_request_t ${net};" + else + ch4_netmod_amrequest_decl="${ch4_netmod_amrequest_decl} \\ +MPIDI_${net_upper}_am_request_t ${net};" + fi + + if test -z "$ch4_netmod_request_decl" ; then + ch4_netmod_request_decl="MPIDI_${net_upper}_request_t ${net};" + else + ch4_netmod_request_decl="${ch4_netmod_request_decl} \\ +MPIDI_${net_upper}_request_t ${net};" + fi + + if test -z "$ch4_netmod_comm_decl" ; then + ch4_netmod_comm_decl="MPIDI_${net_upper}_comm_t ${net};" + else + ch4_netmod_comm_decl="${ch4_netmod_comm_decl} \\ +MPIDI_${net_upper}_comm_t ${net};" + fi + if test -z "$ch4_netmod_dt_decl" ; then + ch4_netmod_dt_decl="MPIDI_${net_upper}_dt_t ${net};" + else + ch4_netmod_dt_decl="${ch4_netmod_dt_decl} \\ +MPIDI_${net_upper}_dt_t ${net};" + fi + if test -z "$ch4_netmod_op_decl" ; then + ch4_netmod_op_decl="MPIDI_${net_upper}_op_t ${net};" + else + ch4_netmod_op_decl="${ch4_netmod_op_decl} \\ +MPIDI_${net_upper}_op_t ${net};" + fi + + if test -z "$ch4_netmod_win_decl" ; then + ch4_netmod_win_decl="MPIDI_${net_upper}_win_t ${net};" + else + ch4_netmod_win_decl="${ch4_netmod_win_decl} \\ +MPIDI_${net_upper}_win_t ${net};" + fi + if test -z "$ch4_netmod_gpid_decl" ; then + ch4_netmod_gpid_decl="MPIDI_${net_upper}_gpid_t ${net};" + else + ch4_netmod_gpid_decl="${ch4_netmod_gpid_decl} \\ +MPIDI_${net_upper}_gpid_t ${net};" + fi + if test -z "$ch4_netmod_addr_decl" ; then + ch4_netmod_addr_decl="MPIDI_${net_upper}_addr_t ${net};" + else + ch4_netmod_addr_decl="${ch4_netmod_addr_decl} \\ +MPIDI_${net_upper}_addr_t ${net};" + fi + + + + +net_index=`expr $net_index + 1` +done +ch4_nets_array_sz=$net_index + +AC_SUBST(device_name) +AC_SUBST(ch4_netmods) +AC_SUBST(ch4_nets_array) +AC_SUBST(ch4_nets_array_sz) +AC_SUBST(ch4_nets_func_decl) +AC_SUBST(ch4_nets_native_func_decl) +AC_SUBST(ch4_nets_func_array) +AC_SUBST(ch4_nets_native_func_array) +AC_SUBST(ch4_nets_strings) +AC_SUBST(ch4_netmod_pre_include) +AC_SUBST(ch4_netmod_amrequest_decl) +AC_SUBST(ch4_netmod_request_decl) +AC_SUBST(ch4_netmod_comm_decl) +AC_SUBST(ch4_netmod_dt_decl) +AC_SUBST(ch4_netmod_win_decl) +AC_SUBST(ch4_netmod_gpid_decl) +AC_SUBST(ch4_netmod_addr_decl) +AC_SUBST(ch4_netmod_op_decl) +AM_SUBST_NOTMAKE(ch4_netmod_pre_include) +AM_SUBST_NOTMAKE(ch4_netmod_amrequest_decl) +AM_SUBST_NOTMAKE(ch4_netmod_request_decl) +AM_SUBST_NOTMAKE(ch4_netmod_comm_decl) +AM_SUBST_NOTMAKE(ch4_netmod_dt_decl) +AM_SUBST_NOTMAKE(ch4_netmod_win_decl) +AM_SUBST_NOTMAKE(ch4_netmod_gpid_decl) +AM_SUBST_NOTMAKE(ch4_netmod_addr_decl) +AM_SUBST_NOTMAKE(ch4_netmod_op_decl) + +AC_ARG_ENABLE(ch4-netmod-direct, + [--enable-ch4-netmod-direct + Enables inlined netmod build when a single netmod is used + level: + yes - Enabled (default) + no - Disabled (may improve build times and code size) + ],,enable_ch4_netmod_direct=yes) + + +if test "$ch4_nets_array_sz" = "1" && test "$enable_ch4_netmod_direct" = "yes" ; then + PAC_APPEND_FLAG([-DNETMOD_DIRECT=__netmod_direct_${ch4_netmods}__], [CPPFLAGS]) +fi + + +AC_ARG_ENABLE(ch4-shm, + [--enable-ch4-shm=level:module + Control whether CH4 shared memory is built and/or used. + level: + no - Do not build or use CH4 shared memory. + yes - Build CH4 shared memory, but do not use it by default (Your chosen netmod must provide it). + exclusive - Build and exclusively use CH4 shared memory. (Default) + module-list(optional). comma separated list of shared memory modules: + posix - POSIX shared memory implementation + ],,enable_ch4_shm=exclusive:posix) + +AC_ARG_ENABLE(ch4-shm-direct, + [--enable-ch4-shm-direct + Enables inlined shared memory build when a single shared memory module is used + level: + yes - Enabled (default) + no - Disabled (may improve build times and code size) + ],,enable_ch4_shm_direct=yes) + +ch4_shm_level=`echo $enable_ch4_shm | sed -e 's/:.*$//'` +changequote(<<,>>) +ch4_shm=`echo $enable_ch4_shm | sed -e 's/^[^:]*//' -e 's/^://'` +changequote([,]) + +if test "$ch4_shm_level" != "no" ; then + AC_DEFINE([MPIDI_BUILD_CH4_SHM], [1], + [Define if CH4 will build the default shared memory implementation as opposed to only using a netmod implementation]) +fi + +if test "$ch4_shm_level" = "exclusive" ; then + # This variable is set only when the user wants CH4 to handle all shared memory operations + AC_DEFINE(MPIDI_CH4_EXCLUSIVE_SHM, 1, [Define if CH4 will be providing the exclusive implementation of shared memory]) + + # This variable can be set either when the user asks for CH4 exclusive shared memory + # or when the netmod doesn't want to implement its own locality information + AC_DEFINE(MPIDI_BUILD_CH4_LOCALITY_INFO, 1, [CH4 should build locality info]) +fi + +# $ch4_shm - contains the shm mods +if test -z "${ch4_shm}" ; then + if test "$ch4_shm_level" != "no" ; then + ch4_shm="posix" + fi +else + ch4_shm=`echo ${ch4_shm} | sed -e 's/,/ /g'` +fi +export ch4_shm + +ch4_shm_func_decl="" +ch4_shm_native_func_decl="" +ch4_shm_func_array="" +ch4_shm_native_func_array="" +ch4_shm_strings="" +shm_index=0 +for shm in $ch4_shm ; do + if test ! -d $srcdir/src/mpid/ch4/shm/${shm} ; then + AC_MSG_ERROR([Shared memory module ${shm} is unknown "$srcdir/src/mpid/ch4/shm/${shm}"]) + fi + shm_macro=`echo $shm | tr 'abcdefghijklmnopqrstuvwxyz' 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'` + shm_macro="MPIDI_SHM_${shm_macro}" + + if test -z "$ch4_shm_array" ; then + ch4_shm_array="$shm_macro" + else + ch4_shm_array="$ch4_shm_array, $shm_macro" + fi + + if test -z "$ch4_shm_func_decl" ; then + ch4_shm_func_decl="MPIDI_SHM_${shm}_funcs" + else + ch4_shm_func_decl="${ch4_shm_func_decl}, MPIDI_SHM_${shm}_funcs" + fi + + if test -z "$ch4_shm_native_func_decl" ; then + ch4_shm_native_func_decl="MPIDI_SHM_native_${shm}_funcs" + else + ch4_shm_native_func_decl="${ch4_shm_native_func_decl}, MPIDI_SHM_native_${shm}_funcs" + fi + + if test -z "$ch4_shm_func_array" ; then + ch4_shm_func_array="&MPIDI_SHM_${shm}_funcs" + else + ch4_shm_func_array="${ch4_shm_func_array}, &MPIDI_SHM_${shm}_funcs" + fi + + if test -z "$ch4_shm_native_func_array" ; then + ch4_shm_native_func_array="&MPIDI_SHM_native_${shm}_funcs" + else + ch4_shm_native_func_array="${ch4_shm_native_func_array}, &MPIDI_SHM_native_${shm}_funcs" + fi + + if test -z "$ch4_shm_strings" ; then + ch4_shm_strings="\"$shm\"" + else + ch4_shm_strings="$ch4_shm_strings, \"$shm\"" + fi + + if test -z "$ch4_shm_pre_include" ; then + ch4_shm_pre_include="#include \"../shm/${shm}/${shm}_pre.h\"" + else + ch4_shm_pre_include="${ch4_shm_pre_include} +#include \"../shm/${shm}/${shm}_pre.h\"" + fi + + shm_upper=`echo ${shm} | tr 'abcdefghijklmnopqrstuvwxyz' 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'` + if test -z "$ch4_shm_request_decl" ; then + ch4_shm_request_decl="MPIDI_${shm_upper}_request_t ${shm};" + else + ch4_shm_request_decl="${ch4_shm_request_decl} \\ +MPIDI_${shm_upper}_request_t ${shm};" + fi + + if test -z "$ch4_shm_comm_decl" ; then + ch4_shm_comm_decl="MPIDI_${shm_upper}_comm_t ${shm};" + else + ch4_shm_comm_decl="${ch4_shm_comm_decl} \\ +MPIDI_${shm_upper}_comm_t ${shm};" + fi + + + shm_index=`expr $shm_index + 1` +done +ch4_shm_array_sz=$shm_index + +AC_SUBST(ch4_shm) +AC_SUBST(ch4_shm_array) +AC_SUBST(ch4_shm_array_sz) +AC_SUBST(ch4_shm_func_decl) +AC_SUBST(ch4_shm_native_func_decl) +AC_SUBST(ch4_shm_func_array) +AC_SUBST(ch4_shm_native_func_array) +AC_SUBST(ch4_shm_strings) +AC_SUBST(ch4_shm_pre_include) +AC_SUBST(ch4_shm_request_decl) +AC_SUBST(ch4_shm_comm_decl) +AM_SUBST_NOTMAKE(ch4_shm_pre_include) +AM_SUBST_NOTMAKE(ch4_shm_request_decl) +AM_SUBST_NOTMAKE(ch4_shm_comm_decl) + +if test "$ch4_shm_array_sz" = "1" && test "$enable_ch4_shm_direct" = "yes" ; then + PAC_APPEND_FLAG([-DSHM_DIRECT=__shm_direct_${ch4_shm}__], [CPPFLAGS]) +fi + +])dnl end AM_COND_IF(BUILD_CH4,...) +])dnl end PREREQ + +AC_DEFUN([PAC_SUBCFG_BODY_]PAC_SUBCFG_AUTO_SUFFIX,[ +AM_COND_IF([BUILD_CH4],[ +AC_MSG_NOTICE([RUNNING CONFIGURE FOR CH4 DEVICE]) + +AC_ARG_WITH(ch4-rank-bits, [--with-ch4-rank-bits=16/32 Number of bits allocated to the rank field (16 or 32)], + [ rankbits=$withval ], + [ rankbits=32 ]) +if test "$rankbits" != "16" -a "$rankbits" != "32" ; then + AC_MSG_ERROR(Only 16 or 32-bit ranks are supported) +fi +AC_DEFINE_UNQUOTED(CH4_RANK_BITS,$rankbits,[Define the number of CH4_RANK_BITS]) + +AC_ARG_ENABLE(ch4r-per-comm-msg-queue, + [--enable-ch4r-per-comm-msg-queue=option + Enable use of per-communicator message queues for posted recvs/unexpected messages + yes - Use per-communicator message queue. (Default) + no - Use global queue for posted recvs/unexpected messages. + ],,enable_ch4r_per_comm_msg_queue=yes) + +if test "$enable_ch4r_per_comm_msg_queue" = "yes" ; then + AC_DEFINE([MPIDI_CH4U_USE_PER_COMM_QUEUE], [1], + [Define if CH4U will use per-communicator message queues]) +fi + +PAC_ARG_SHARED_MEMORY + +AC_CONFIG_FILES([ +src/mpid/ch4/src/mpid_ch4_net_array.c +src/mpid/ch4/src/mpid_ch4_shm_array.c +src/mpid/ch4/include/netmodpre.h +src/mpid/ch4/include/shmpre.h +]) +PAC_ARG_SHARED_MEMORY +])dnl end AM_COND_IF(BUILD_CH4,...) + +AM_CONDITIONAL([BUILD_CH4_SHM],[test "$ch4_shm_level" = "yes" -o "$ch4_shm_level" = "exclusive"]) + +AC_CHECK_HEADERS(sys/mman.h sys/stat.h fcntl.h) + +found_rand_funcs=no +AC_CHECK_FUNCS(random_r initstate_r, found_rand_funcs=yes) +if test "$found_rand_funcs" = yes ; then + AC_DEFINE(USE_SYM_HEAP,1,[Define if we can use a symmetric heap]) + AC_MSG_NOTICE([Using a symmetric heap]) +else + AC_MSG_NOTICE([Using a non-symmetric heap]) +fi + + +])dnl end _BODY + +[#] end of __file__ ----------------------------------------------------------------------- Summary of changes: .gitignore | 3 + src/mpid/Makefile.mk | 1 + src/mpid/ch4/.gitignore | 3 + src/mpid/ch4/Makefile.mk | 20 + src/mpid/ch4/cross/gcc-linux-x86-8 | 13 + src/mpid/ch4/cross/icc-linux-x86-8 | 13 + src/mpid/ch4/errnames.txt | 47 + src/mpid/ch4/include/Makefile.mk | 23 + src/mpid/ch4/include/mpid_sched.h | 30 + src/mpid/ch4/include/mpid_thread.h | 68 + src/mpid/ch4/include/mpid_ticketlock.h | 159 ++ src/mpid/ch4/include/mpidch4.h | 435 ++++ src/mpid/ch4/include/mpidch4r.h | 26 + src/mpid/ch4/include/mpidimpl.h | 30 + src/mpid/ch4/include/mpidpost.h | 30 + src/mpid/ch4/include/mpidpre.h | 465 +++++ src/mpid/ch4/include/netmodpre.h.in | 26 + src/mpid/ch4/include/shmpre.h.in | 19 + src/mpid/ch4/netmod/Makefile.mk | 22 + src/mpid/ch4/netmod/include/netmod.h | 1133 ++++++++++ src/mpid/ch4/netmod/include/netmod_impl.h | 1083 ++++++++++ src/mpid/ch4/netmod/ofi/Makefile.mk | 23 + src/mpid/ch4/netmod/ofi/catalog.c | 61 + src/mpid/ch4/netmod/ofi/errnames.txt | 97 + src/mpid/ch4/netmod/ofi/fi_list.h | 196 ++ src/mpid/ch4/netmod/ofi/func_table.c | 157 ++ src/mpid/ch4/netmod/ofi/globals.c | 13 + src/mpid/ch4/netmod/ofi/netmod_direct.h | 41 + src/mpid/ch4/netmod/ofi/ofi_am.h | 383 ++++ src/mpid/ch4/netmod/ofi/ofi_am_events.h | 414 ++++ src/mpid/ch4/netmod/ofi/ofi_am_impl.h | 565 +++++ src/mpid/ch4/netmod/ofi/ofi_am_probe.h | 39 + src/mpid/ch4/netmod/ofi/ofi_am_recv.h | 61 + src/mpid/ch4/netmod/ofi/ofi_am_rma.h | 148 ++ src/mpid/ch4/netmod/ofi/ofi_am_send.h | 128 ++ src/mpid/ch4/netmod/ofi/ofi_am_spawn.h | 50 + src/mpid/ch4/netmod/ofi/ofi_am_win.h | 160 ++ src/mpid/ch4/netmod/ofi/ofi_coll.h | 869 ++++++++ src/mpid/ch4/netmod/ofi/ofi_comm.h | 65 + src/mpid/ch4/netmod/ofi/ofi_control.h | 72 + src/mpid/ch4/netmod/ofi/ofi_datatype.h | 31 + src/mpid/ch4/netmod/ofi/ofi_events.h | 804 ++++++++ src/mpid/ch4/netmod/ofi/ofi_impl.h | 484 +++++ src/mpid/ch4/netmod/ofi/ofi_init.h | 896 ++++++++ src/mpid/ch4/netmod/ofi/ofi_iovec_util.h | 391 ++++ src/mpid/ch4/netmod/ofi/ofi_op.h | 27 + src/mpid/ch4/netmod/ofi/ofi_pre.h | 190 ++ src/mpid/ch4/netmod/ofi/ofi_probe.h | 168 ++ src/mpid/ch4/netmod/ofi/ofi_proc.h | 29 + src/mpid/ch4/netmod/ofi/ofi_progress.h | 63 + src/mpid/ch4/netmod/ofi/ofi_recv.h | 280 +++ src/mpid/ch4/netmod/ofi/ofi_rma.h | 1298 ++++++++++++ src/mpid/ch4/netmod/ofi/ofi_send.h | 566 +++++ src/mpid/ch4/netmod/ofi/ofi_spawn.h | 561 +++++ src/mpid/ch4/netmod/ofi/ofi_types.h | 545 +++++ src/mpid/ch4/netmod/ofi/ofi_unimpl.h | 19 + src/mpid/ch4/netmod/ofi/ofi_win.h | 1254 ++++++++++++ src/mpid/ch4/netmod/ofi/subconfigure.m4 | 153 ++ src/mpid/ch4/netmod/ofi/util.c | 822 ++++++++ src/mpid/ch4/netmod/portals4/Makefile.mk | 7 + src/mpid/ch4/netmod/portals4/errnames.txt | 10 + src/mpid/ch4/netmod/portals4/func_table.c | 157 ++ src/mpid/ch4/netmod/portals4/globals.c | 14 + src/mpid/ch4/netmod/portals4/netmod_direct.h | 30 + src/mpid/ch4/netmod/portals4/portals4_pre.h | 51 + src/mpid/ch4/netmod/portals4/ptl_am.h | 370 ++++ src/mpid/ch4/netmod/portals4/ptl_coll.h | 871 ++++++++ src/mpid/ch4/netmod/portals4/ptl_comm.h | 39 + src/mpid/ch4/netmod/portals4/ptl_datatype.h | 28 + src/mpid/ch4/netmod/portals4/ptl_impl.h | 58 + src/mpid/ch4/netmod/portals4/ptl_init.h | 271 +++ src/mpid/ch4/netmod/portals4/ptl_op.h | 24 + src/mpid/ch4/netmod/portals4/ptl_probe.h | 39 + src/mpid/ch4/netmod/portals4/ptl_proc.h | 29 + src/mpid/ch4/netmod/portals4/ptl_progress.h | 185 ++ src/mpid/ch4/netmod/portals4/ptl_recv.h | 63 + src/mpid/ch4/netmod/portals4/ptl_request.h | 32 + src/mpid/ch4/netmod/portals4/ptl_rma.h | 148 ++ src/mpid/ch4/netmod/portals4/ptl_send.h | 128 ++ src/mpid/ch4/netmod/portals4/ptl_spawn.h | 50 + src/mpid/ch4/netmod/portals4/ptl_types.h | 76 + src/mpid/ch4/netmod/portals4/ptl_unimpl.h | 19 + src/mpid/ch4/netmod/portals4/ptl_win.h | 160 ++ src/mpid/ch4/netmod/portals4/subconfigure.m4 | 33 + src/mpid/ch4/netmod/stubnm/Makefile.mk | 6 + src/mpid/ch4/netmod/stubnm/globals.c | 157 ++ src/mpid/ch4/netmod/stubnm/netmod_direct.h | 29 + src/mpid/ch4/netmod/stubnm/stubnm_am.h | 138 ++ src/mpid/ch4/netmod/stubnm/stubnm_coll.h | 871 ++++++++ src/mpid/ch4/netmod/stubnm/stubnm_comm.h | 39 + src/mpid/ch4/netmod/stubnm/stubnm_datatype.h | 35 + src/mpid/ch4/netmod/stubnm/stubnm_impl.h | 17 + src/mpid/ch4/netmod/stubnm/stubnm_init.h | 95 + src/mpid/ch4/netmod/stubnm/stubnm_op.h | 29 + src/mpid/ch4/netmod/stubnm/stubnm_pre.h | 47 + src/mpid/ch4/netmod/stubnm/stubnm_probe.h | 39 + src/mpid/ch4/netmod/stubnm/stubnm_proc.h | 28 + src/mpid/ch4/netmod/stubnm/stubnm_progress.h | 76 + src/mpid/ch4/netmod/stubnm/stubnm_recv.h | 67 + src/mpid/ch4/netmod/stubnm/stubnm_request.h | 26 + src/mpid/ch4/netmod/stubnm/stubnm_rma.h | 148 ++ src/mpid/ch4/netmod/stubnm/stubnm_send.h | 128 ++ src/mpid/ch4/netmod/stubnm/stubnm_spawn.h | 50 + src/mpid/ch4/netmod/stubnm/stubnm_unimpl.h | 19 + src/mpid/ch4/netmod/stubnm/stubnm_win.h | 160 ++ src/mpid/ch4/netmod/stubnm/subconfigure.m4 | 21 + src/mpid/ch4/netmod/ucx/Makefile.mk | 15 + src/mpid/ch4/netmod/ucx/errnames.txt | 16 + src/mpid/ch4/netmod/ucx/func_table.c | 155 ++ src/mpid/ch4/netmod/ucx/globals.c | 13 + src/mpid/ch4/netmod/ucx/netmod_direct.h | 34 + src/mpid/ch4/netmod/ucx/subconfigure.m4 | 61 + src/mpid/ch4/netmod/ucx/ucx_am.h | 615 ++++++ src/mpid/ch4/netmod/ucx/ucx_am_recv.h | 60 + src/mpid/ch4/netmod/ucx/ucx_am_rma.h | 149 ++ src/mpid/ch4/netmod/ucx/ucx_am_send.h | 128 ++ src/mpid/ch4/netmod/ucx/ucx_am_win.h | 160 ++ src/mpid/ch4/netmod/ucx/ucx_coll.h | 867 ++++++++ src/mpid/ch4/netmod/ucx/ucx_comm.h | 48 + src/mpid/ch4/netmod/ucx/ucx_datatype.h | 150 ++ src/mpid/ch4/netmod/ucx/ucx_impl.h | 151 ++ src/mpid/ch4/netmod/ucx/ucx_init.h | 330 +++ src/mpid/ch4/netmod/ucx/ucx_op.h | 24 + src/mpid/ch4/netmod/ucx/ucx_pre.h | 75 + src/mpid/ch4/netmod/ucx/ucx_probe.h | 99 + src/mpid/ch4/netmod/ucx/ucx_proc.h | 26 + src/mpid/ch4/netmod/ucx/ucx_progress.h | 147 ++ src/mpid/ch4/netmod/ucx/ucx_recv.h | 244 +++ src/mpid/ch4/netmod/ucx/ucx_request.h | 116 ++ src/mpid/ch4/netmod/ucx/ucx_rma.h | 299 +++ src/mpid/ch4/netmod/ucx/ucx_send.h | 488 +++++ src/mpid/ch4/netmod/ucx/ucx_spawn.h | 104 + src/mpid/ch4/netmod/ucx/ucx_types.h | 66 + src/mpid/ch4/netmod/ucx/ucx_win.h | 503 +++++ src/mpid/ch4/shm/Makefile.mk | 24 + src/mpid/ch4/shm/include/shm.h | 1192 +++++++++++ src/mpid/ch4/shm/include/shm_impl.h | 1109 ++++++++++ src/mpid/ch4/shm/posix/Makefile.mk | 38 + src/mpid/ch4/shm/posix/barrier.c | 42 + src/mpid/ch4/shm/posix/func_table.c | 152 ++ src/mpid/ch4/shm/posix/globals.c | 22 + src/mpid/ch4/shm/posix/posix_am.h | 173 ++ src/mpid/ch4/shm/posix/posix_coll.h | 876 ++++++++ src/mpid/ch4/shm/posix/posix_comm.h | 45 + src/mpid/ch4/shm/posix/posix_datatypes.h | 185 ++ src/mpid/ch4/shm/posix/posix_defs.h | 115 ++ src/mpid/ch4/shm/posix/posix_impl.h | 174 ++ src/mpid/ch4/shm/posix/posix_init.h | 324 +++ src/mpid/ch4/shm/posix/posix_pre.h | 41 + src/mpid/ch4/shm/posix/posix_probe.h | 146 ++ src/mpid/ch4/shm/posix/posix_progress.h | 454 ++++ src/mpid/ch4/shm/posix/posix_queue.h | 343 ++++ src/mpid/ch4/shm/posix/posix_recv.h | 355 ++++ src/mpid/ch4/shm/posix/posix_request.h | 26 + src/mpid/ch4/shm/posix/posix_rma.h | 143 ++ src/mpid/ch4/shm/posix/posix_send.h | 462 +++++ src/mpid/ch4/shm/posix/posix_spawn.h | 50 + src/mpid/ch4/shm/posix/posix_unimpl.h | 19 + src/mpid/ch4/shm/posix/posix_win.h | 185 ++ src/mpid/ch4/shm/posix/shm_direct.h | 27 + src/mpid/ch4/shm/posix/subconfigure.m4 | 53 + src/mpid/ch4/shm/stubshm/Makefile.mk | 7 + src/mpid/ch4/shm/stubshm/func_table.c | 152 ++ src/mpid/ch4/shm/stubshm/globals.c | 13 + src/mpid/ch4/shm/stubshm/shm_direct.h | 27 + src/mpid/ch4/shm/stubshm/stubshm_am.h | 173 ++ src/mpid/ch4/shm/stubshm/stubshm_coll.h | 695 +++++++ src/mpid/ch4/shm/stubshm/stubshm_comm.h | 29 + src/mpid/ch4/shm/stubshm/stubshm_impl.h | 14 + src/mpid/ch4/shm/stubshm/stubshm_init.h | 85 + src/mpid/ch4/shm/stubshm/stubshm_pre.h | 27 + src/mpid/ch4/shm/stubshm/stubshm_probe.h | 36 + src/mpid/ch4/shm/stubshm/stubshm_proc.h | 21 + src/mpid/ch4/shm/stubshm/stubshm_progress.h | 88 + src/mpid/ch4/shm/stubshm/stubshm_recv.h | 70 + src/mpid/ch4/shm/stubshm/stubshm_request.h | 26 + src/mpid/ch4/shm/stubshm/stubshm_rma.h | 143 ++ src/mpid/ch4/shm/stubshm/stubshm_send.h | 140 ++ src/mpid/ch4/shm/stubshm/stubshm_spawn.h | 50 + src/mpid/ch4/shm/stubshm/stubshm_unimpl.h | 19 + src/mpid/ch4/shm/stubshm/stubshm_win.h | 185 ++ src/mpid/ch4/shm/stubshm/subconfigure.m4 | 19 + src/mpid/ch4/src/Makefile.mk | 44 + src/mpid/ch4/src/ch4_coll.h | 378 ++++ src/mpid/ch4/src/ch4_comm.h | 249 +++ src/mpid/ch4/src/ch4_globals.c | 62 + src/mpid/ch4/src/ch4_impl.h | 553 +++++ src/mpid/ch4/src/ch4_init.h | 751 +++++++ src/mpid/ch4/src/ch4_probe.h | 220 ++ src/mpid/ch4/src/ch4_proc.h | 37 + src/mpid/ch4/src/ch4_progress.h | 185 ++ src/mpid/ch4/src/ch4_recv.h | 411 ++++ src/mpid/ch4/src/ch4_request.h | 91 + src/mpid/ch4/src/ch4_rma.h | 297 +++ src/mpid/ch4/src/ch4_send.h | 515 +++++ src/mpid/ch4/src/ch4_spawn.h | 319 +++ src/mpid/ch4/src/ch4_types.h | 285 +++ src/mpid/ch4/src/ch4_win.h | 530 +++++ src/mpid/ch4/src/ch4i_comm.h | 1161 +++++++++++ src/mpid/ch4/src/ch4i_util.h | 20 + src/mpid/ch4/src/ch4r_buf.h | 168 ++ src/mpid/ch4/src/ch4r_callbacks.h | 2836 ++++++++++++++++++++++++++ src/mpid/ch4/src/ch4r_init.h | 333 +++ src/mpid/ch4/src/ch4r_probe.h | 176 ++ src/mpid/ch4/src/ch4r_proc.h | 411 ++++ src/mpid/ch4/src/ch4r_recv.h | 429 ++++ src/mpid/ch4/src/ch4r_recvq.h | 336 +++ src/mpid/ch4/src/ch4r_request.h | 126 ++ src/mpid/ch4/src/ch4r_rma.h | 761 +++++++ src/mpid/ch4/src/ch4r_send.h | 468 +++++ src/mpid/ch4/src/ch4r_symheap.h | 258 +++ src/mpid/ch4/src/ch4r_win.h | 1250 ++++++++++++ src/mpid/ch4/src/mpid_ch4_net_array.c.in | 28 + src/mpid/ch4/src/mpid_ch4_shm_array.c.in | 32 + src/mpid/ch4/subconfigure.m4 | 391 ++++ 215 files changed, 48937 insertions(+), 0 deletions(-) create mode 100644 src/mpid/ch4/.gitignore create mode 100644 src/mpid/ch4/Makefile.mk create mode 100644 src/mpid/ch4/cross/gcc-linux-x86-8 create mode 100644 src/mpid/ch4/cross/icc-linux-x86-8 create mode 100644 src/mpid/ch4/errnames.txt create mode 100644 src/mpid/ch4/include/Makefile.mk create mode 100644 src/mpid/ch4/include/mpid_sched.h create mode 100644 src/mpid/ch4/include/mpid_thread.h create mode 100644 src/mpid/ch4/include/mpid_ticketlock.h create mode 100644 src/mpid/ch4/include/mpidch4.h create mode 100644 src/mpid/ch4/include/mpidch4r.h create mode 100644 src/mpid/ch4/include/mpidimpl.h create mode 100644 src/mpid/ch4/include/mpidpost.h create mode 100644 src/mpid/ch4/include/mpidpre.h create mode 100644 src/mpid/ch4/include/netmodpre.h.in create mode 100644 src/mpid/ch4/include/shmpre.h.in create mode 100644 src/mpid/ch4/netmod/Makefile.mk create mode 100644 src/mpid/ch4/netmod/include/netmod.h create mode 100644 src/mpid/ch4/netmod/include/netmod_impl.h create mode 100644 src/mpid/ch4/netmod/ofi/Makefile.mk create mode 100644 src/mpid/ch4/netmod/ofi/catalog.c create mode 100644 src/mpid/ch4/netmod/ofi/errnames.txt create mode 100644 src/mpid/ch4/netmod/ofi/fi_list.h create mode 100644 src/mpid/ch4/netmod/ofi/func_table.c create mode 100644 src/mpid/ch4/netmod/ofi/globals.c create mode 100644 src/mpid/ch4/netmod/ofi/netmod_direct.h create mode 100644 src/mpid/ch4/netmod/ofi/ofi_am.h create mode 100644 src/mpid/ch4/netmod/ofi/ofi_am_events.h create mode 100644 src/mpid/ch4/netmod/ofi/ofi_am_impl.h create mode 100644 src/mpid/ch4/netmod/ofi/ofi_am_probe.h create mode 100644 src/mpid/ch4/netmod/ofi/ofi_am_recv.h create mode 100644 src/mpid/ch4/netmod/ofi/ofi_am_rma.h create mode 100644 src/mpid/ch4/netmod/ofi/ofi_am_send.h create mode 100644 src/mpid/ch4/netmod/ofi/ofi_am_spawn.h create mode 100644 src/mpid/ch4/netmod/ofi/ofi_am_win.h create mode 100644 src/mpid/ch4/netmod/ofi/ofi_coll.h create mode 100644 src/mpid/ch4/netmod/ofi/ofi_comm.h create mode 100644 src/mpid/ch4/netmod/ofi/ofi_control.h create mode 100644 src/mpid/ch4/netmod/ofi/ofi_datatype.h create mode 100644 src/mpid/ch4/netmod/ofi/ofi_events.h create mode 100644 src/mpid/ch4/netmod/ofi/ofi_impl.h create mode 100644 src/mpid/ch4/netmod/ofi/ofi_init.h create mode 100644 src/mpid/ch4/netmod/ofi/ofi_iovec_util.h create mode 100644 src/mpid/ch4/netmod/ofi/ofi_op.h create mode 100644 src/mpid/ch4/netmod/ofi/ofi_pre.h create mode 100644 src/mpid/ch4/netmod/ofi/ofi_probe.h create mode 100644 src/mpid/ch4/netmod/ofi/ofi_proc.h create mode 100644 src/mpid/ch4/netmod/ofi/ofi_progress.h create mode 100644 src/mpid/ch4/netmod/ofi/ofi_recv.h create mode 100644 src/mpid/ch4/netmod/ofi/ofi_rma.h create mode 100644 src/mpid/ch4/netmod/ofi/ofi_send.h create mode 100644 src/mpid/ch4/netmod/ofi/ofi_spawn.h create mode 100644 src/mpid/ch4/netmod/ofi/ofi_types.h create mode 100644 src/mpid/ch4/netmod/ofi/ofi_unimpl.h create mode 100644 src/mpid/ch4/netmod/ofi/ofi_win.h create mode 100644 src/mpid/ch4/netmod/ofi/subconfigure.m4 create mode 100644 src/mpid/ch4/netmod/ofi/util.c create mode 100644 src/mpid/ch4/netmod/portals4/Makefile.mk create mode 100644 src/mpid/ch4/netmod/portals4/errnames.txt create mode 100644 src/mpid/ch4/netmod/portals4/func_table.c create mode 100644 src/mpid/ch4/netmod/portals4/globals.c create mode 100644 src/mpid/ch4/netmod/portals4/netmod_direct.h create mode 100644 src/mpid/ch4/netmod/portals4/portals4_pre.h create mode 100644 src/mpid/ch4/netmod/portals4/ptl_am.h create mode 100644 src/mpid/ch4/netmod/portals4/ptl_coll.h create mode 100644 src/mpid/ch4/netmod/portals4/ptl_comm.h create mode 100644 src/mpid/ch4/netmod/portals4/ptl_datatype.h create mode 100644 src/mpid/ch4/netmod/portals4/ptl_impl.h create mode 100644 src/mpid/ch4/netmod/portals4/ptl_init.h create mode 100644 src/mpid/ch4/netmod/portals4/ptl_op.h create mode 100644 src/mpid/ch4/netmod/portals4/ptl_probe.h create mode 100644 src/mpid/ch4/netmod/portals4/ptl_proc.h create mode 100644 src/mpid/ch4/netmod/portals4/ptl_progress.h create mode 100644 src/mpid/ch4/netmod/portals4/ptl_recv.h create mode 100644 src/mpid/ch4/netmod/portals4/ptl_request.h create mode 100644 src/mpid/ch4/netmod/portals4/ptl_rma.h create mode 100644 src/mpid/ch4/netmod/portals4/ptl_send.h create mode 100644 src/mpid/ch4/netmod/portals4/ptl_spawn.h create mode 100644 src/mpid/ch4/netmod/portals4/ptl_types.h create mode 100644 src/mpid/ch4/netmod/portals4/ptl_unimpl.h create mode 100644 src/mpid/ch4/netmod/portals4/ptl_win.h create mode 100644 src/mpid/ch4/netmod/portals4/subconfigure.m4 create mode 100644 src/mpid/ch4/netmod/stubnm/Makefile.mk create mode 100644 src/mpid/ch4/netmod/stubnm/globals.c create mode 100644 src/mpid/ch4/netmod/stubnm/netmod_direct.h create mode 100644 src/mpid/ch4/netmod/stubnm/stubnm_am.h create mode 100644 src/mpid/ch4/netmod/stubnm/stubnm_coll.h create mode 100644 src/mpid/ch4/netmod/stubnm/stubnm_comm.h create mode 100644 src/mpid/ch4/netmod/stubnm/stubnm_datatype.h create mode 100644 src/mpid/ch4/netmod/stubnm/stubnm_impl.h create mode 100644 src/mpid/ch4/netmod/stubnm/stubnm_init.h create mode 100644 src/mpid/ch4/netmod/stubnm/stubnm_op.h create mode 100644 src/mpid/ch4/netmod/stubnm/stubnm_pre.h create mode 100644 src/mpid/ch4/netmod/stubnm/stubnm_probe.h create mode 100644 src/mpid/ch4/netmod/stubnm/stubnm_proc.h create mode 100644 src/mpid/ch4/netmod/stubnm/stubnm_progress.h create mode 100644 src/mpid/ch4/netmod/stubnm/stubnm_recv.h create mode 100644 src/mpid/ch4/netmod/stubnm/stubnm_request.h create mode 100644 src/mpid/ch4/netmod/stubnm/stubnm_rma.h create mode 100644 src/mpid/ch4/netmod/stubnm/stubnm_send.h create mode 100644 src/mpid/ch4/netmod/stubnm/stubnm_spawn.h create mode 100644 src/mpid/ch4/netmod/stubnm/stubnm_unimpl.h create mode 100644 src/mpid/ch4/netmod/stubnm/stubnm_win.h create mode 100644 src/mpid/ch4/netmod/stubnm/subconfigure.m4 create mode 100644 src/mpid/ch4/netmod/ucx/Makefile.mk create mode 100644 src/mpid/ch4/netmod/ucx/errnames.txt create mode 100644 src/mpid/ch4/netmod/ucx/func_table.c create mode 100644 src/mpid/ch4/netmod/ucx/globals.c create mode 100644 src/mpid/ch4/netmod/ucx/netmod_direct.h create mode 100644 src/mpid/ch4/netmod/ucx/subconfigure.m4 create mode 100644 src/mpid/ch4/netmod/ucx/ucx_am.h create mode 100644 src/mpid/ch4/netmod/ucx/ucx_am_recv.h create mode 100644 src/mpid/ch4/netmod/ucx/ucx_am_rma.h create mode 100644 src/mpid/ch4/netmod/ucx/ucx_am_send.h create mode 100644 src/mpid/ch4/netmod/ucx/ucx_am_win.h create mode 100644 src/mpid/ch4/netmod/ucx/ucx_coll.h create mode 100644 src/mpid/ch4/netmod/ucx/ucx_comm.h create mode 100644 src/mpid/ch4/netmod/ucx/ucx_datatype.h create mode 100644 src/mpid/ch4/netmod/ucx/ucx_impl.h create mode 100644 src/mpid/ch4/netmod/ucx/ucx_init.h create mode 100644 src/mpid/ch4/netmod/ucx/ucx_op.h create mode 100644 src/mpid/ch4/netmod/ucx/ucx_pre.h create mode 100644 src/mpid/ch4/netmod/ucx/ucx_probe.h create mode 100644 src/mpid/ch4/netmod/ucx/ucx_proc.h create mode 100644 src/mpid/ch4/netmod/ucx/ucx_progress.h create mode 100644 src/mpid/ch4/netmod/ucx/ucx_recv.h create mode 100644 src/mpid/ch4/netmod/ucx/ucx_request.h create mode 100644 src/mpid/ch4/netmod/ucx/ucx_rma.h create mode 100644 src/mpid/ch4/netmod/ucx/ucx_send.h create mode 100644 src/mpid/ch4/netmod/ucx/ucx_spawn.h create mode 100644 src/mpid/ch4/netmod/ucx/ucx_types.h create mode 100644 src/mpid/ch4/netmod/ucx/ucx_win.h create mode 100644 src/mpid/ch4/shm/Makefile.mk create mode 100644 src/mpid/ch4/shm/include/shm.h create mode 100644 src/mpid/ch4/shm/include/shm_impl.h create mode 100644 src/mpid/ch4/shm/posix/Makefile.mk create mode 100644 src/mpid/ch4/shm/posix/barrier.c create mode 100644 src/mpid/ch4/shm/posix/func_table.c create mode 100644 src/mpid/ch4/shm/posix/globals.c create mode 100644 src/mpid/ch4/shm/posix/posix_am.h create mode 100644 src/mpid/ch4/shm/posix/posix_coll.h create mode 100644 src/mpid/ch4/shm/posix/posix_comm.h create mode 100644 src/mpid/ch4/shm/posix/posix_datatypes.h create mode 100644 src/mpid/ch4/shm/posix/posix_defs.h create mode 100644 src/mpid/ch4/shm/posix/posix_impl.h create mode 100644 src/mpid/ch4/shm/posix/posix_init.h create mode 100644 src/mpid/ch4/shm/posix/posix_pre.h create mode 100644 src/mpid/ch4/shm/posix/posix_probe.h create mode 100644 src/mpid/ch4/shm/posix/posix_progress.h create mode 100644 src/mpid/ch4/shm/posix/posix_queue.h create mode 100644 src/mpid/ch4/shm/posix/posix_recv.h create mode 100644 src/mpid/ch4/shm/posix/posix_request.h create mode 100644 src/mpid/ch4/shm/posix/posix_rma.h create mode 100644 src/mpid/ch4/shm/posix/posix_send.h create mode 100644 src/mpid/ch4/shm/posix/posix_spawn.h create mode 100644 src/mpid/ch4/shm/posix/posix_unimpl.h create mode 100644 src/mpid/ch4/shm/posix/posix_win.h create mode 100644 src/mpid/ch4/shm/posix/shm_direct.h create mode 100644 src/mpid/ch4/shm/posix/subconfigure.m4 create mode 100644 src/mpid/ch4/shm/stubshm/Makefile.mk create mode 100644 src/mpid/ch4/shm/stubshm/func_table.c create mode 100644 src/mpid/ch4/shm/stubshm/globals.c create mode 100644 src/mpid/ch4/shm/stubshm/shm_direct.h create mode 100644 src/mpid/ch4/shm/stubshm/stubshm_am.h create mode 100644 src/mpid/ch4/shm/stubshm/stubshm_coll.h create mode 100644 src/mpid/ch4/shm/stubshm/stubshm_comm.h create mode 100644 src/mpid/ch4/shm/stubshm/stubshm_impl.h create mode 100644 src/mpid/ch4/shm/stubshm/stubshm_init.h create mode 100644 src/mpid/ch4/shm/stubshm/stubshm_pre.h create mode 100644 src/mpid/ch4/shm/stubshm/stubshm_probe.h create mode 100644 src/mpid/ch4/shm/stubshm/stubshm_proc.h create mode 100644 src/mpid/ch4/shm/stubshm/stubshm_progress.h create mode 100644 src/mpid/ch4/shm/stubshm/stubshm_recv.h create mode 100644 src/mpid/ch4/shm/stubshm/stubshm_request.h create mode 100644 src/mpid/ch4/shm/stubshm/stubshm_rma.h create mode 100644 src/mpid/ch4/shm/stubshm/stubshm_send.h create mode 100644 src/mpid/ch4/shm/stubshm/stubshm_spawn.h create mode 100644 src/mpid/ch4/shm/stubshm/stubshm_unimpl.h create mode 100644 src/mpid/ch4/shm/stubshm/stubshm_win.h create mode 100644 src/mpid/ch4/shm/stubshm/subconfigure.m4 create mode 100644 src/mpid/ch4/src/Makefile.mk create mode 100644 src/mpid/ch4/src/ch4_coll.h create mode 100644 src/mpid/ch4/src/ch4_comm.h create mode 100644 src/mpid/ch4/src/ch4_globals.c create mode 100644 src/mpid/ch4/src/ch4_impl.h create mode 100644 src/mpid/ch4/src/ch4_init.h create mode 100644 src/mpid/ch4/src/ch4_probe.h create mode 100644 src/mpid/ch4/src/ch4_proc.h create mode 100644 src/mpid/ch4/src/ch4_progress.h create mode 100644 src/mpid/ch4/src/ch4_recv.h create mode 100644 src/mpid/ch4/src/ch4_request.h create mode 100644 src/mpid/ch4/src/ch4_rma.h create mode 100644 src/mpid/ch4/src/ch4_send.h create mode 100644 src/mpid/ch4/src/ch4_spawn.h create mode 100644 src/mpid/ch4/src/ch4_types.h create mode 100644 src/mpid/ch4/src/ch4_win.h create mode 100644 src/mpid/ch4/src/ch4i_comm.h create mode 100644 src/mpid/ch4/src/ch4i_util.h create mode 100644 src/mpid/ch4/src/ch4r_buf.h create mode 100644 src/mpid/ch4/src/ch4r_callbacks.h create mode 100644 src/mpid/ch4/src/ch4r_init.h create mode 100644 src/mpid/ch4/src/ch4r_probe.h create mode 100644 src/mpid/ch4/src/ch4r_proc.h create mode 100644 src/mpid/ch4/src/ch4r_recv.h create mode 100644 src/mpid/ch4/src/ch4r_recvq.h create mode 100644 src/mpid/ch4/src/ch4r_request.h create mode 100644 src/mpid/ch4/src/ch4r_rma.h create mode 100644 src/mpid/ch4/src/ch4r_send.h create mode 100644 src/mpid/ch4/src/ch4r_symheap.h create mode 100644 src/mpid/ch4/src/ch4r_win.h create mode 100644 src/mpid/ch4/src/mpid_ch4_net_array.c.in create mode 100644 src/mpid/ch4/src/mpid_ch4_shm_array.c.in create mode 100644 src/mpid/ch4/subconfigure.m4 hooks/post-receive -- MPICH primary repository

1 0

[mpich] MPICH primary repository branch, master, updated. v3.2-384-g1494e0f
by noreply＠mpich.org 21 Aug '16

21 Aug '16

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "MPICH primary repository". The branch, master has been updated via 1494e0f44808516588ab7f57d0754cf27556f43c (commit) from 1fef7c7a94f19039b8642792dec79d69654a114b (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- http://git.mpich.org/mpich.git/commitdiff/1494e0f44808516588ab7f57d0754cf27… commit 1494e0f44808516588ab7f57d0754cf27556f43c Author: Ken Raffenetti <raffenet(a)mcs.anl.gov> Date: Sun Aug 21 17:42:43 2016 -0500 Revert "test/mpi/configure.ac: Enable all Fortran by default" This reverts commit [1fef7c7a94f1], which unconditionally enabled the f08 bindings. A better default would be to test for basic compile support and enable when found, like the main configure script. diff --git a/test/mpi/configure.ac b/test/mpi/configure.ac index b657070..6cc32a1 100644 --- a/test/mpi/configure.ac +++ b/test/mpi/configure.ac @@ -99,7 +99,7 @@ AC_ARG_ENABLE(fortran, f90 - Enable Fortran 90 support f08 - Enable Fortran 2008 support no|none - No Fortran support -],,[enable_fortran=all]) +],,[enable_fortran=f77,f90]) save_IFS="$IFS" IFS="," ----------------------------------------------------------------------- Summary of changes: test/mpi/configure.ac | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) hooks/post-receive -- MPICH primary repository

1 0

[mpich] MPICH primary repository branch, 3.2.x, created. v3.2-5-g88557f9
by noreply＠mpich.org 19 Aug '16

19 Aug '16

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "MPICH primary repository". The branch, 3.2.x has been created at 88557f903875b4f2a44283a343eee4ac045aeb2a (commit) - Log ----------------------------------------------------------------- http://git.mpich.org/mpich.git/commitdiff/88557f903875b4f2a44283a343eee4ac0… commit 88557f903875b4f2a44283a343eee4ac045aeb2a Author: Ken Raffenetti <raffenet(a)mcs.anl.gov> Date: Thu Aug 18 12:04:51 2016 -0500 hcoll: Add reference to internal requests Signed-off-by: Tomislav Janjusic <tomislavj(a)mellanox.com> diff --git a/src/mpid/common/hcoll/hcoll_rte.c b/src/mpid/common/hcoll/hcoll_rte.c index e0d2136..123a71f 100644 --- a/src/mpid/common/hcoll/hcoll_rte.c +++ b/src/mpid/common/hcoll/hcoll_rte.c @@ -364,6 +364,7 @@ static void *get_coll_handle(void) { MPID_Request *req; req = MPID_Request_create(); + MPIR_Request_add_ref(req); req->kind = MPID_COLL_REQUEST; return (void *) req; } http://git.mpich.org/mpich.git/commitdiff/ca5e6a6befeeffbc8347da7b32058d521… commit ca5e6a6befeeffbc8347da7b32058d5210911161 Author: Ken Raffenetti <raffenet(a)mcs.anl.gov> Date: Thu Aug 18 12:03:51 2016 -0500 hcoll: Use MPIU_Assert Signed-off-by: Tomislav Janjusic <tomislavj(a)mellanox.com> diff --git a/src/mpid/common/hcoll/hcoll_rte.c b/src/mpid/common/hcoll/hcoll_rte.c index 996cf7f..e0d2136 100644 --- a/src/mpid/common/hcoll/hcoll_rte.c +++ b/src/mpid/common/hcoll/hcoll_rte.c @@ -1,6 +1,5 @@ #include "hcoll.h" #include "hcoll/api/hcoll_dte.h" -#include <assert.h> static int recv_nb(dte_data_representation_t data, uint32_t count, @@ -54,7 +53,7 @@ static void progress(void) * error codes. The progress function pointer right now * expects that the function returns void. */ ret = hcoll_do_progress(&made_progress); - assert(ret == MPI_SUCCESS); + MPIU_Assert(ret == MPI_SUCCESS); } } http://git.mpich.org/mpich.git/commitdiff/5fe0a4918ac41e7e5f9035ac561e70349… commit 5fe0a4918ac41e7e5f9035ac561e7034902fefb0 Author: Ken Raffenetti <raffenet(a)mcs.anl.gov> Date: Tue Aug 16 15:53:58 2016 -0500 hcoll: Fix usage of progress hook MPICH progress hooks now take a made_progress parameter. We just set it to true for now for correctness purposes. Signed-off-by: Tomislav Janjusic <tomislavj(a)mellanox.com> diff --git a/src/mpid/common/hcoll/hcoll_init.c b/src/mpid/common/hcoll/hcoll_init.c index c04704a..daa0caa 100644 --- a/src/mpid/common/hcoll/hcoll_init.c +++ b/src/mpid/common/hcoll/hcoll_init.c @@ -226,8 +226,7 @@ int hcoll_comm_destroy(MPID_Comm * comm_ptr, void *param) int hcoll_do_progress(int *made_progress) { - if (made_progress) - *made_progress = 0; + *made_progress = 1; hcoll_progress_fn(); return MPI_SUCCESS; diff --git a/src/mpid/common/hcoll/hcoll_rte.c b/src/mpid/common/hcoll/hcoll_rte.c index 82fa961..996cf7f 100644 --- a/src/mpid/common/hcoll/hcoll_rte.c +++ b/src/mpid/common/hcoll/hcoll_rte.c @@ -44,6 +44,7 @@ static int world_rank(rte_grp_handle_t grp_h, rte_ec_handle_t ec); static void progress(void) { int ret; + int made_progress; if (0 == world_comm_destroying) { MPID_Progress_test(); @@ -52,7 +53,7 @@ static void progress(void) /* FIXME: The hcoll library needs to be updated to return * error codes. The progress function pointer right now * expects that the function returns void. */ - ret = hcoll_do_progress(); + ret = hcoll_do_progress(&made_progress); assert(ret == MPI_SUCCESS); } } http://git.mpich.org/mpich.git/commitdiff/d20930987ec9d4aee5830fbf89f72af6e… commit d20930987ec9d4aee5830fbf89f72af6e02807d3 Author: Ken Raffenetti <raffenet(a)mcs.anl.gov> Date: Thu Aug 18 11:43:09 2016 -0500 hcoll: Remove unused function Signed-off-by: Tomislav Janjusic <tomislavj(a)mellanox.com> diff --git a/src/mpid/common/hcoll/hcoll_rte.c b/src/mpid/common/hcoll/hcoll_rte.c index e7288a4..82fa961 100644 --- a/src/mpid/common/hcoll/hcoll_rte.c +++ b/src/mpid/common/hcoll/hcoll_rte.c @@ -23,8 +23,6 @@ static int ec_handle_compare(rte_ec_handle_t handle_1, static int get_ec_handles(int num_ec, int *ec_indexes, rte_grp_handle_t, rte_ec_handle_t * ec_handles); -static int get_my_ec(rte_grp_handle_t, rte_ec_handle_t * ec_handle); - static int group_size(rte_grp_handle_t group); static int my_rank(rte_grp_handle_t grp_h); static int ec_on_local_node(rte_ec_handle_t ec, rte_grp_handle_t group); @@ -294,21 +292,6 @@ static int get_ec_handles(int num_ec, } #undef FUNCNAME -#define FUNCNAME get_my_ec -#undef FCNAME -#define FCNAME MPL_QUOTE(FUNCNAME) -static int get_my_ec(rte_grp_handle_t grp_h, rte_ec_handle_t * ec_handle) -{ - MPID_Comm *comm; - comm = (MPID_Comm *) grp_h; - int my_rank = MPIR_Comm_rank(comm); - ec_handle->handle = (void *) (comm->vcrt->vcr_table[my_rank]); - ec_handle->rank = my_rank; - return HCOLL_SUCCESS; -} - - -#undef FUNCNAME #define FUNCNAME group_size #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) http://git.mpich.org/mpich.git/commitdiff/1db759cd3878fbf0b531a7ce630093ba7… commit 1db759cd3878fbf0b531a7ce630093ba789e9bd6 Author: Ken Raffenetti <raffenet(a)mcs.anl.gov> Date: Thu Aug 18 11:41:44 2016 -0500 hcoll: Fix ec_handles The glue code used to rely on access to VCs, which are no longer exposed to the upper layers. This commit hacks in access to them again to get hcoll working. We will add a new device interface to not be CH3 specific. Signed-off-by: Tomislav Janjusic <tomislavj(a)mellanox.com> diff --git a/src/mpid/common/hcoll/hcoll_rte.c b/src/mpid/common/hcoll/hcoll_rte.c index ea543fe..e7288a4 100644 --- a/src/mpid/common/hcoll/hcoll_rte.c +++ b/src/mpid/common/hcoll/hcoll_rte.c @@ -288,7 +288,7 @@ static int get_ec_handles(int num_ec, comm = (MPID_Comm *) grp_h; for (i = 0; i < num_ec; i++) { ec_handles[i].rank = ec_indexes[i]; - ec_handles[i].handle = (void *) (comm->vcrt->vcr_table[ec_indexes[i]]); + ec_handles[i].handle = (void *) (comm->dev.vcrt->vcr_table[ec_indexes[i]]); } return HCOLL_SUCCESS; } @@ -430,5 +430,5 @@ static void coll_handle_complete(void *handle) #define FCNAME MPL_QUOTE(FUNCNAME) static int world_rank(rte_grp_handle_t grp_h, rte_ec_handle_t ec) { - return (MPIR_Process.comm_world->rank); + return ((struct MPIDI_VC *)ec.handle)->pg_rank; } ----------------------------------------------------------------------- hooks/post-receive -- MPICH primary repository

1 0

[mpich] MPICH primary repository branch, master, updated. v3.2-383-g1fef7c7
by noreply＠mpich.org 19 Aug '16

19 Aug '16

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "MPICH primary repository". The branch, master has been updated via 1fef7c7a94f19039b8642792dec79d69654a114b (commit) from 1d8609c51cdc57c22c5de858ca667136e3c882b6 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- http://git.mpich.org/mpich.git/commitdiff/1fef7c7a94f19039b8642792dec79d696… commit 1fef7c7a94f19039b8642792dec79d69654a114b Author: Ken Raffenetti <raffenet(a)mcs.anl.gov> Date: Fri Aug 19 11:07:09 2016 -0500 test/mpi/configure.ac: Enable all Fortran by default The previous commit fixed the --enable-fortran=all option, but neglected to make it the default. No reviewer. diff --git a/test/mpi/configure.ac b/test/mpi/configure.ac index 6cc32a1..b657070 100644 --- a/test/mpi/configure.ac +++ b/test/mpi/configure.ac @@ -99,7 +99,7 @@ AC_ARG_ENABLE(fortran, f90 - Enable Fortran 90 support f08 - Enable Fortran 2008 support no|none - No Fortran support -],,[enable_fortran=f77,f90]) +],,[enable_fortran=all]) save_IFS="$IFS" IFS="," ----------------------------------------------------------------------- Summary of changes: test/mpi/configure.ac | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) hooks/post-receive -- MPICH primary repository

1 0

[mpich] MPICH primary repository branch, master, updated. v3.2-382-g1d8609c
by noreply＠mpich.org 18 Aug '16

18 Aug '16

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "MPICH primary repository". The branch, master has been updated via 1d8609c51cdc57c22c5de858ca667136e3c882b6 (commit) from ed1c7a11afac5e9291a0955ffb92269a5b6aa25b (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- http://git.mpich.org/mpich.git/commitdiff/1d8609c51cdc57c22c5de858ca667136e… commit 1d8609c51cdc57c22c5de858ca667136e3c882b6 Author: Ken Raffenetti <raffenet(a)mcs.anl.gov> Date: Thu Aug 18 17:43:28 2016 -0500 test/mpi/configure.ac: Modify f08 settings Enabling all (or none) of the Fortran bindings did not modify the f08 setting for the testsuite. The main configure script addressed this in [698d7f7203aa]. No reviewer. diff --git a/test/mpi/configure.ac b/test/mpi/configure.ac index 785423e..6cc32a1 100644 --- a/test/mpi/configure.ac +++ b/test/mpi/configure.ac @@ -111,10 +111,12 @@ for option in $enable_fortran ; do yes|all) enable_f77=yes enable_fc=yes + enable_f08=yes ;; no|none) enable_f77=no enable_fc=no + enable_f08=no ;; f77) enable_f77=yes ----------------------------------------------------------------------- Summary of changes: test/mpi/configure.ac | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) hooks/post-receive -- MPICH primary repository

1 0