May 2016 - commits - mailman3.cels.anl.gov

[mpich] MPICH primary repository branch, master, updated. v3.2-328-gafe418e
by noreply＠mpich.org 31 May '16

31 May '16

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "MPICH primary repository". The branch, master has been updated via afe418ebea58a752c1c013276dbe85221916399f (commit) from a145fd73a1bb7dd4f06c8c6b7dff5f6c3695f6f6 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- http://git.mpich.org/mpich.git/commitdiff/afe418ebea58a752c1c013276dbe85221… commit afe418ebea58a752c1c013276dbe85221916399f Author: Ken Raffenetti <raffenet(a)mcs.anl.gov> Date: Tue May 31 09:52:35 2016 -0500 test/mpi/rma: fix printf arg order in selfrma.c Reported-by: Clément Foyer <clement.foyer(a)gmail.com> Signed-off-by: Halim Amer <aamer(a)anl.gov> diff --git a/test/mpi/rma/selfrma.c b/test/mpi/rma/selfrma.c index 701fcea..6a8b4f0 100644 --- a/test/mpi/rma/selfrma.c +++ b/test/mpi/rma/selfrma.c @@ -63,7 +63,7 @@ int main(int argc, char *argv[]) for (i = 0; i < vcount; i++) { if (winbuf[j] != sbuf[j]) { errs++; - fprintf(stderr, "VecPut: winbuf[%d] = %d, should = %d\n", winbuf[j], j, sbuf[j]); + fprintf(stderr, "VecPut: winbuf[%d] = %d, should = %d\n", j, winbuf[j], sbuf[j]); } j += 2; } @@ -77,7 +77,7 @@ int main(int argc, char *argv[]) for (i = 0; i < vcount; i++) { if (winbuf[j] != sbuf[j]) { errs++; - fprintf(stderr, "VecAcc: winbuf[%d] = %d, should = %d\n", winbuf[j], j, sbuf[j]); + fprintf(stderr, "VecAcc: winbuf[%d] = %d, should = %d\n", j, winbuf[j], sbuf[j]); } j += 2; } @@ -92,7 +92,7 @@ int main(int argc, char *argv[]) for (i = 0; i < vcount; i++) { if (winbuf[j] != sbuf[j]) { errs++; - fprintf(stderr, "VecAcc: winbuf[%d] = %d, should = %d\n", winbuf[j], j, sbuf[j]); + fprintf(stderr, "VecAcc: winbuf[%d] = %d, should = %d\n", j, winbuf[j], sbuf[j]); } j += 2; } ----------------------------------------------------------------------- Summary of changes: test/mpi/rma/selfrma.c | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) hooks/post-receive -- MPICH primary repository

1 0

[mpich] MPICH primary repository branch, master, updated. v3.2-327-ga145fd7
by noreply＠mpich.org 26 May '16

26 May '16

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "MPICH primary repository". The branch, master has been updated via a145fd73a1bb7dd4f06c8c6b7dff5f6c3695f6f6 (commit) from dc4891a0ee8e9262f4b094871d0e18039b704eae (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- http://git.mpich.org/mpich.git/commitdiff/a145fd73a1bb7dd4f06c8c6b7dff5f6c3… commit a145fd73a1bb7dd4f06c8c6b7dff5f6c3695f6f6 Author: Ken Raffenetti <raffenet(a)mcs.anl.gov> Date: Tue May 24 13:45:59 2016 -0500 pm/hydra: do binding with HYDRA_TOPO_DEBUG set Users were understandably confused that enabling topology debugging resulted in processes not actually being bound at launch. This commit enables binding when debugging is turned on. No reviewer. diff --git a/src/pm/hydra/tools/topo/hwloc/topo_hwloc.c b/src/pm/hydra/tools/topo/hwloc/topo_hwloc.c index 013f1e7..8c97431 100644 --- a/src/pm/hydra/tools/topo/hwloc/topo_hwloc.c +++ b/src/pm/hydra/tools/topo/hwloc/topo_hwloc.c @@ -455,8 +455,8 @@ HYD_status HYDT_topo_hwloc_bind(int idx) if (!HYDT_topo_hwloc_info.user_binding || (idx < HYDT_topo_hwloc_info.num_bitmaps)) { id = idx % HYDT_topo_hwloc_info.num_bitmaps; - /* For debugging, print the binding bitmaps but don't actually bind. */ if (HYDT_topo_info.debug) { + /* Print the binding bitmaps for debugging. */ int i; char *binding; @@ -473,11 +473,9 @@ HYD_status HYDT_topo_hwloc_bind(int idx) HYDU_dump_noprefix(stdout, "process %d binding: %s\n", idx, binding); MPL_free(binding); } - else { - hwloc_set_cpubind(topology, HYDT_topo_hwloc_info.bitmap[id], 0); - hwloc_set_membind(topology, HYDT_topo_hwloc_info.bitmap[id], - HYDT_topo_hwloc_info.membind, 0); - } + hwloc_set_cpubind(topology, HYDT_topo_hwloc_info.bitmap[id], 0); + hwloc_set_membind(topology, HYDT_topo_hwloc_info.bitmap[id], + HYDT_topo_hwloc_info.membind, 0); } fn_exit: ----------------------------------------------------------------------- Summary of changes: src/pm/hydra/tools/topo/hwloc/topo_hwloc.c | 10 ++++------ 1 files changed, 4 insertions(+), 6 deletions(-) hooks/post-receive -- MPICH primary repository

1 0

[mpich] MPICH primary repository branch, master, updated. v3.2-326-gdc4891a
by noreply＠mpich.org 25 May '16

25 May '16

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "MPICH primary repository". The branch, master has been updated via dc4891a0ee8e9262f4b094871d0e18039b704eae (commit) from 95582e64c7c5abd99f9338667a7ef45228d27f29 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- http://git.mpich.org/mpich.git/commitdiff/dc4891a0ee8e9262f4b094871d0e18039… commit dc4891a0ee8e9262f4b094871d0e18039b704eae Author: Rob Latham <robl(a)mcs.anl.gov> Date: Tue Apr 26 10:59:19 2016 -0500 mainline bugfix: *fs.c is not a generated file causes some real problems with ROMIO (e.g logfs.c) No Reviewer diff --git a/.gitignore b/.gitignore index ad7bd8c..6ad5751 100644 --- a/.gitignore +++ b/.gitignore @@ -72,7 +72,6 @@ Makefile *.vfproj *conf.h *conf.h.in -*fs.c *old*.pdf .*-cache .deps ----------------------------------------------------------------------- Summary of changes: .gitignore | 1 - 1 files changed, 0 insertions(+), 1 deletions(-) hooks/post-receive -- MPICH primary repository

1 0

[mpich] MPICH primary repository branch, master, updated. v3.2-325-g95582e6
by noreply＠mpich.org 25 May '16

25 May '16

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "MPICH primary repository". The branch, master has been updated via 95582e64c7c5abd99f9338667a7ef45228d27f29 (commit) from cd51b6f0bd2f053ef1bf4dc9339ca40c5ef48cfa (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- http://git.mpich.org/mpich.git/commitdiff/95582e64c7c5abd99f9338667a7ef4522… commit 95582e64c7c5abd99f9338667a7ef45228d27f29 Author: Rob Latham <robl(a)mcs.anl.gov> Date: Wed May 25 13:55:39 2016 -0500 comm_split_type: fix directory-not-shared case if another process cannot find the file that's OK: it tells us something about the topology. do not report "cannot find file" up to the calling routines diff --git a/src/mpi/romio/mpi-io/mpir_cst_filesys.c b/src/mpi/romio/mpi-io/mpir_cst_filesys.c index 3ef9429..2ef7092 100644 --- a/src/mpi/romio/mpi-io/mpir_cst_filesys.c +++ b/src/mpi/romio/mpi-io/mpir_cst_filesys.c @@ -144,11 +144,15 @@ int MPIR_Comm_split_filesystem(MPI_Comm comm, int key, const char *dirname, MPI_ * open/close the file instead */ mpi_errno = MPI_File_open(MPI_COMM_SELF, filename, MPI_MODE_RDONLY, MPI_INFO_NULL, &fh); - if (mpi_errno != MPI_SUCCESS) - goto fn_exit; - if (mpi_errno == MPI_SUCCESS) + if (mpi_errno == MPI_SUCCESS) { globally_visible = 1; - MPI_File_close(&fh); + MPI_File_close(&fh); + } else { + /* do not report error up to caller. we are merely testing the + * presence of the file */ + mpi_errno = MPI_SUCCESS; + globally_visible = 0; + } } MPI_Bcast(&globally_visible, 1, MPI_INT, challenge_rank, comm); ----------------------------------------------------------------------- Summary of changes: src/mpi/romio/mpi-io/mpir_cst_filesys.c | 12 ++++++++---- 1 files changed, 8 insertions(+), 4 deletions(-) hooks/post-receive -- MPICH primary repository

1 0

[mpich] MPICH primary repository branch, master, updated. v3.2-324-gcd51b6f
by noreply＠mpich.org 25 May '16

25 May '16

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "MPICH primary repository". The branch, master has been updated via cd51b6f0bd2f053ef1bf4dc9339ca40c5ef48cfa (commit) via 9220062cd25625b64692005378f3003cb075d5fd (commit) via 08de58ea0f65b5b201eee3dd448f1814093a9f7a (commit) via 9559fd65c08e179947cff6981dceb8ac35c04672 (commit) from a131e534d26ea79ba028435b2a162e14b5c7764f (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- http://git.mpich.org/mpich.git/commitdiff/cd51b6f0bd2f053ef1bf4dc9339ca40c5… commit cd51b6f0bd2f053ef1bf4dc9339ca40c5ef48cfa Author: Rob Latham <robl(a)mcs.anl.gov> Date: Tue May 24 15:47:26 2016 -0500 promote types for 64-to-32 warnings 'long' should be large enough to hold all MPL_time_t cases No Reviewer diff --git a/src/mpl/src/timer/mpl_timer_clock_gettime.c b/src/mpl/src/timer/mpl_timer_clock_gettime.c index e42431d..6c081fa 100644 --- a/src/mpl/src/timer/mpl_timer_clock_gettime.c +++ b/src/mpl/src/timer/mpl_timer_clock_gettime.c @@ -34,7 +34,7 @@ int MPL_wtime_todouble(MPL_time_t * t, double *val) int MPL_wtime_acc(MPL_time_t * t1, MPL_time_t * t2, MPL_time_t * t3) { - int nsec, sec; + long nsec, sec; nsec = t2->tv_nsec - t1->tv_nsec; sec = t2->tv_sec - t1->tv_sec; http://git.mpich.org/mpich.git/commitdiff/9220062cd25625b64692005378f3003cb… commit 9220062cd25625b64692005378f3003cb075d5fd Author: Rob Latham <robl(a)mcs.anl.gov> Date: Tue May 24 15:45:21 2016 -0500 srand and 64-to-32 warnings time returns long, but srand takes an int. make sure we are explicitly taking the most relvant (lower) 32 bits from time. diff --git a/src/mpi/romio/adio/common/shfp_fname.c b/src/mpi/romio/adio/common/shfp_fname.c index 5c84381..1e76885 100644 --- a/src/mpi/romio/adio/common/shfp_fname.c +++ b/src/mpi/romio/adio/common/shfp_fname.c @@ -39,7 +39,9 @@ void ADIOI_Shfp_fname(ADIO_File fd, int rank, int *error_code) fd->shared_fp_fname = (char *) ADIOI_Malloc(PATH_MAX); if (!rank) { - srand(time(NULL)); + /* srand takes int but time returns long; keep the lower and most + * significant 32 bits */ + srand(time(NULL)& 0xffffffff); i = rand(); pid = (int)getpid(); diff --git a/src/mpi/romio/mpi-io/mpir_cst_filesys.c b/src/mpi/romio/mpi-io/mpir_cst_filesys.c index c6c5ca6..3ef9429 100644 --- a/src/mpi/romio/mpi-io/mpir_cst_filesys.c +++ b/src/mpi/romio/mpi-io/mpir_cst_filesys.c @@ -108,7 +108,7 @@ int MPIR_Comm_split_filesystem(MPI_Comm comm, int key, const char *dirname, MPI_ int r, pid; /* same algorithim as shared file pointer name */ - srand(time(NULL)); + srand(time(NULL) & 0xffffffff); r = rand(); pid = (int) getpid(); http://git.mpich.org/mpich.git/commitdiff/08de58ea0f65b5b201eee3dd448f18140… commit 08de58ea0f65b5b201eee3dd448f1814093a9f7a Author: Gilles Gouaillardet <gilles.gouaillardet(a)iferc.org> Date: Thu Jan 30 17:02:54 2014 +0900 Add wrap retry logic into romio/adio/ad_lustre. Since [p](read|write) can return with less data than requested, some wrap retry logic has been added in ad_lustre. For example, in my test environment, a 4GB write requires 3 calls to write ( write returns ~2GB, ~2GB and 8kB). read/write were replaced with pread/pwrite for thread safety. "obviously short" pread/pwrite are not wrapped yet. Fixes #2012 Signed-off-by: Rob Latham <robl(a)mcs.anl.gov> diff --git a/src/mpi/romio/adio/ad_lustre/ad_lustre_rwcontig.c b/src/mpi/romio/adio/ad_lustre/ad_lustre_rwcontig.c index 31c86be..ffd096a 100644 --- a/src/mpi/romio/adio/ad_lustre/ad_lustre_rwcontig.c +++ b/src/mpi/romio/adio/ad_lustre/ad_lustre_rwcontig.c @@ -16,12 +16,12 @@ #define LUSTRE_MEMALIGN (1<<12) /* to use page_shift */ -static void ADIOI_LUSTRE_Aligned_Mem_File_Write(ADIO_File fd, const void *buf, int len, - ADIO_Offset offset, int *err); -static void ADIOI_LUSTRE_Aligned_Mem_File_Write(ADIO_File fd, const void *buf, int len, - ADIO_Offset offset, int *err) +static void ADIOI_LUSTRE_Aligned_Mem_File_Write(ADIO_File fd, const void *buf, MPI_Count len, + ADIO_Offset offset, ssize_t *err); +static void ADIOI_LUSTRE_Aligned_Mem_File_Write(ADIO_File fd, const void *buf, MPI_Count len, + ADIO_Offset offset, ssize_t *err) { - int rem, size, nbytes; + ssize_t rem, size, nbytes; if (!(len % fd->d_miniosz) && (len >= fd->d_miniosz)) { *err = pwrite(fd->fd_direct, buf, len, offset); } else if (len < fd->d_miniosz) { @@ -29,18 +29,22 @@ static void ADIOI_LUSTRE_Aligned_Mem_File_Write(ADIO_File fd, const void *buf, i } else { rem = len % fd->d_miniosz; size = len - rem; - nbytes = pwrite(fd->fd_direct, buf, size, offset); - nbytes += pwrite(fd->fd_sys, ((char *)buf) + size, rem, offset+size); - *err = nbytes; + *err = pwrite(fd->fd_direct, buf, size, offset); + if (*err == -1) return; + nbytes = *err; + *err = pwrite(fd->fd_sys, ((char *)buf) + size, rem, offset+size); + if (*err == -1) return; + *err += nbytes; } } -static void ADIOI_LUSTRE_Aligned_Mem_File_Read(ADIO_File fd, const void *buf, int len, - ADIO_Offset offset, int *err); -static void ADIOI_LUSTRE_Aligned_Mem_File_Read(ADIO_File fd, const void *buf, int len, - ADIO_Offset offset, int *err) +static void ADIOI_LUSTRE_Aligned_Mem_File_Read(ADIO_File fd, const void *buf, MPI_Count len, + ADIO_Offset offset, ssize_t *err); +static void ADIOI_LUSTRE_Aligned_Mem_File_Read(ADIO_File fd, const void *buf, MPI_Count len, + ADIO_Offset offset, ssize_t *err) { - int rem, size, nbytes; + MPI_Count rem, size; + ssize_t nbytes; if (!(len % fd->d_miniosz) && (len >= fd->d_miniosz)) *err = pread(fd->fd_direct, (void *)buf, len, offset); else if (len < fd->d_miniosz) @@ -48,19 +52,22 @@ static void ADIOI_LUSTRE_Aligned_Mem_File_Read(ADIO_File fd, const void *buf, in else { rem = len % fd->d_miniosz; size = len - rem; - nbytes = pread(fd->fd_direct, (void *)buf, size, offset); - nbytes += pread(fd->fd_sys, ((char *)buf) + size, rem, offset+size); - *err = nbytes; + *err = pread(fd->fd_direct, (void *)buf, size, offset); + if (*err == -1) return; + nbytes = *err; + *err = pread(fd->fd_sys, ((char *)buf) + size, rem, offset+size); + if (*err == -1) return; + *err += nbytes; } } - -static int ADIOI_LUSTRE_Directio(ADIO_File fd, const void *buf, int len, +static ssize_t ADIOI_LUSTRE_Directio(ADIO_File fd, const void *buf, MPI_Count len, off_t offset, int rw); -static int ADIOI_LUSTRE_Directio(ADIO_File fd, const void *buf, int len, +static ssize_t ADIOI_LUSTRE_Directio(ADIO_File fd, const void *buf, MPI_Count len, off_t offset, int rw) { - int err=-1, diff, size=len, nbytes = 0; + ssize_t err=-1, diff, nbytes = 0; + MPI_Count size=len; void *newbuf; if (offset % fd->d_miniosz) { @@ -70,47 +77,58 @@ static int ADIOI_LUSTRE_Directio(ADIO_File fd, const void *buf, int len, nbytes = pwrite(fd->fd_sys, (void *)buf, diff, offset); else nbytes = pread(fd->fd_sys, (void *)buf, diff, offset); + if (nbytes == -1) return -1; buf = ((char *) buf) + diff; offset += diff; size = len - diff; } if (!size) { - return diff; + return nbytes; } if (rw) { /* direct I/O enabled */ if (!(((long) buf) % fd->d_mem)) { ADIOI_LUSTRE_Aligned_Mem_File_Write(fd, buf, size, offset, &err); + if (err == -1) return -1; nbytes += err; } else { newbuf = (void *) memalign(LUSTRE_MEMALIGN, size); if (newbuf) { memcpy(newbuf, buf, size); ADIOI_LUSTRE_Aligned_Mem_File_Write(fd, newbuf, size, offset, &err); + if (err == -1) return -1; nbytes += err; ADIOI_Free(newbuf); } - else nbytes += pwrite(fd->fd_sys, buf, size, offset); + else { + err = pwrite(fd->fd_sys, buf, size, offset); + if (err == -1) return -1; + nbytes += err; + } } - err = nbytes; } else { if (!(((long) buf) % fd->d_mem)) { ADIOI_LUSTRE_Aligned_Mem_File_Read(fd, buf, size, offset, &err); + if (err == -1) return -1; nbytes += err; } else { newbuf = (void *) memalign(LUSTRE_MEMALIGN, size); if (newbuf) { ADIOI_LUSTRE_Aligned_Mem_File_Read(fd, newbuf, size, offset, &err); + if (err == -1) return -1; if (err > 0) memcpy((void *)buf, newbuf, err); nbytes += err; ADIOI_Free(newbuf); } - else nbytes += pread(fd->fd_sys, (void *)buf, size, offset); + else { + err = pread(fd->fd_sys, (void *)buf, size, offset); + if (err == -1) return -1; + nbytes += err; + } } - err = nbytes; } - return err; + return nbytes; } static void ADIOI_LUSTRE_IOContig(ADIO_File fd, const void *buf, int count, @@ -122,9 +140,12 @@ static void ADIOI_LUSTRE_IOContig(ADIO_File fd, const void *buf, int count, ADIO_Offset offset, ADIO_Status *status, int io_mode, int *error_code) { - int err=-1; + ssize_t err=-1; + size_t rw_count; + ADIO_Offset bytes_xfered=0; MPI_Count datatype_size, len; static char myname[] = "ADIOI_LUSTRE_IOCONTIG"; + char *p; MPI_Type_size_x(datatype, &datatype_size); len = datatype_size * count; @@ -139,36 +160,52 @@ static void ADIOI_LUSTRE_IOContig(ADIO_File fd, const void *buf, int count, if (err == -1) goto ioerr; } + p = (char *)buf; if (io_mode) { + while (bytes_xfered < len) { #ifdef ADIOI_MPE_LOGGING - MPE_Log_event(ADIOI_MPE_write_a, 0, NULL); + MPE_Log_event(ADIOI_MPE_write_a, 0, NULL); #endif - err = write(fd->fd_sys, buf, len); + rw_count = len - bytes_xfered; + err = pwrite(fd->fd_sys, p, rw_count, offset+bytes_xfered); + if (err == -1) goto ioerr; #ifdef ADIOI_MPE_LOGGING - MPE_Log_event(ADIOI_MPE_write_b, 0, NULL); + MPE_Log_event(ADIOI_MPE_write_b, 0, NULL); #endif + if (err == 0) break; + bytes_xfered += err; + p += err; + } } else { + while (bytes_xfered < len) { #ifdef ADIOI_MPE_LOGGING - MPE_Log_event(ADIOI_MPE_read_a, 0, NULL); + MPE_Log_event(ADIOI_MPE_read_a, 0, NULL); #endif - err = read(fd->fd_sys, (void *)buf, len); + rw_count = len - bytes_xfered; + err = pread(fd->fd_sys, p, rw_count, offset+bytes_xfered); + if (err == -1) goto ioerr; #ifdef ADIOI_MPE_LOGGING - MPE_Log_event(ADIOI_MPE_read_b, 0, NULL); + MPE_Log_event(ADIOI_MPE_read_b, 0, NULL); #endif + if (err == 0) break; + bytes_xfered += err; + p += err; + } } } else { err = ADIOI_LUSTRE_Directio(fd, buf, len, offset, io_mode); + if (err == -1) goto ioerr; + bytes_xfered = err; } - if (err == -1) goto ioerr; - fd->fp_sys_posn = offset + err; + fd->fp_sys_posn = offset + bytes_xfered; if (file_ptr_type == ADIO_INDIVIDUAL) { - fd->fp_ind += err; + fd->fp_ind += bytes_xfered; } #ifdef HAVE_STATUS_SET_BYTES - if (status) MPIR_Status_set_bytes(status, datatype, err); + if (status) MPIR_Status_set_bytes(status, datatype, bytes_xfered); #endif *error_code = MPI_SUCCESS; http://git.mpich.org/mpich.git/commitdiff/9559fd65c08e179947cff6981dceb8ac3… commit 9559fd65c08e179947cff6981dceb8ac35c04672 Author: Gilles Gouaillardet <gilles.gouaillardet(a)iferc.org> Date: Thu Jan 30 15:05:57 2014 +0900 Fix integer overflow in romio/adio/ad_lustre. Some integer overflow prevented MPI_File_write_all to write more than 2GB at a time on lustre. This patch does the minimum. A fuller audit likely needed. (Michael Raymond at SGI complained to RobL about this bug, too) Fix #2011. Signed-off-by: Rob Latham <robl(a)mcs.anl.gov> diff --git a/src/mpi/romio/adio/ad_lustre/ad_lustre.h b/src/mpi/romio/adio/ad_lustre/ad_lustre.h index edd2bd3..5e8f017 100644 --- a/src/mpi/romio/adio/ad_lustre/ad_lustre.h +++ b/src/mpi/romio/adio/ad_lustre/ad_lustre.h @@ -88,7 +88,7 @@ void ADIOI_LUSTRE_Calc_my_req(ADIO_File fd, ADIO_Offset *offset_list, int *count_my_req_procs_ptr, int **count_my_req_per_proc_ptr, ADIOI_Access **my_req_ptr, - int ***buf_idx_ptr); + ADIO_Offset ***buf_idx_ptr); int ADIOI_LUSTRE_Calc_aggregator(ADIO_File fd, ADIO_Offset off, ADIO_Offset *len, int *striping_info); diff --git a/src/mpi/romio/adio/ad_lustre/ad_lustre_aggregate.c b/src/mpi/romio/adio/ad_lustre/ad_lustre_aggregate.c index 39f158d..a9840cc 100644 --- a/src/mpi/romio/adio/ad_lustre/ad_lustre_aggregate.c +++ b/src/mpi/romio/adio/ad_lustre/ad_lustre_aggregate.c @@ -140,13 +140,13 @@ void ADIOI_LUSTRE_Calc_my_req(ADIO_File fd, ADIO_Offset *offset_list, int *count_my_req_procs_ptr, int **count_my_req_per_proc_ptr, ADIOI_Access **my_req_ptr, - int ***buf_idx_ptr) + ADIO_Offset ***buf_idx_ptr) { /* Nothing different from ADIOI_Calc_my_req(), except calling * ADIOI_Lustre_Calc_aggregator() instead of the old one */ - int *count_my_req_per_proc, count_my_req_procs, **buf_idx; + int *count_my_req_per_proc, count_my_req_procs; int i, l, proc; - ADIO_Offset avail_len, rem_len, curr_idx, off; + ADIO_Offset avail_len, rem_len, curr_idx, off, **buf_idx; ADIOI_Access *my_req; *count_my_req_per_proc_ptr = (int *) ADIOI_Calloc(nprocs, sizeof(int)); @@ -157,7 +157,7 @@ void ADIOI_LUSTRE_Calc_my_req(ADIO_File fd, ADIO_Offset *offset_list, * MPI_Alltoall later on. */ - buf_idx = (int **) ADIOI_Malloc(nprocs * sizeof(int*)); + buf_idx = (ADIO_Offset **) ADIOI_Malloc(nprocs * sizeof(ADIO_Offset *)); /* one pass just to calculate how much space to allocate for my_req; * contig_access_count was calculated way back in ADIOI_Calc_my_off_len() @@ -201,8 +201,8 @@ void ADIOI_LUSTRE_Calc_my_req(ADIO_File fd, ADIO_Offset *offset_list, /* initialize buf_idx vectors */ for (i = 0; i < nprocs; i++) { /* add one to count_my_req_per_proc[i] to avoid zero size malloc */ - buf_idx[i] = (int *) ADIOI_Malloc((count_my_req_per_proc[i] + 1) - * sizeof(int)); + buf_idx[i] = (ADIO_Offset *) ADIOI_Malloc((count_my_req_per_proc[i] + 1) + * sizeof(ADIO_Offset)); } /* now allocate space for my_req, offset, and len */ @@ -235,9 +235,8 @@ void ADIOI_LUSTRE_Calc_my_req(ADIO_File fd, ADIO_Offset *offset_list, l = my_req[proc].count; - ADIOI_Assert(curr_idx == (int) curr_idx); ADIOI_Assert(l < count_my_req_per_proc[proc]); - buf_idx[proc][l] = (int) curr_idx; + buf_idx[proc][l] = curr_idx; curr_idx += avail_len; rem_len = len_list[i] - avail_len; @@ -259,9 +258,8 @@ void ADIOI_LUSTRE_Calc_my_req(ADIO_File fd, ADIO_Offset *offset_list, striping_info); l = my_req[proc].count; - ADIOI_Assert(curr_idx == (int) curr_idx); ADIOI_Assert(l < count_my_req_per_proc[proc]); - buf_idx[proc][l] = (int) curr_idx; + buf_idx[proc][l] = curr_idx; curr_idx += avail_len; rem_len -= avail_len; diff --git a/src/mpi/romio/adio/ad_lustre/ad_lustre_wrcoll.c b/src/mpi/romio/adio/ad_lustre/ad_lustre_wrcoll.c index 36f5f28..e87f927 100644 --- a/src/mpi/romio/adio/ad_lustre/ad_lustre_wrcoll.c +++ b/src/mpi/romio/adio/ad_lustre/ad_lustre_wrcoll.c @@ -21,7 +21,7 @@ static void ADIOI_LUSTRE_Exch_and_write(ADIO_File fd, const void *buf, ADIO_Offset *len_list, int contig_access_count, int *striping_info, - int **buf_idx, int *error_code); + ADIO_Offset **buf_idx, int *error_code); static void ADIOI_LUSTRE_Fill_send_buffer(ADIO_File fd, const void *buf, ADIOI_Flatlist_node *flat_buf, char **send_buf, @@ -31,7 +31,7 @@ static void ADIOI_LUSTRE_Fill_send_buffer(ADIO_File fd, const void *buf, int *sent_to_proc, int nprocs, int myrank, int contig_access_count, int *striping_info, - int *send_buf_idx, + ADIO_Offset *send_buf_idx, int *curr_to_proc, int *done_to_proc, int iter, MPI_Aint buftype_extent); @@ -48,11 +48,11 @@ static void ADIOI_LUSTRE_W_Exchange_data(ADIO_File fd, const void *buf, int contig_access_count, int *striping_info, ADIOI_Access *others_req, - int *send_buf_idx, + ADIO_Offset *send_buf_idx, int *curr_to_proc, int *done_to_proc, int *hole, int iter, MPI_Aint buftype_extent, - int *buf_idx, + ADIO_Offset *buf_idx, ADIO_Offset **srt_off, int **srt_len, int *srt_num, int *error_code); void ADIOI_Heap_merge(ADIOI_Access *others_req, int *count, @@ -85,7 +85,8 @@ void ADIOI_LUSTRE_WriteStridedColl(ADIO_File fd, const void *buf, int count, ADIO_Offset orig_fp, start_offset, end_offset, off; ADIO_Offset *offset_list = NULL, *st_offsets = NULL, *end_offsets = NULL; ADIO_Offset *len_list = NULL; - int **buf_idx = NULL, *striping_info = NULL; + int *striping_info = NULL; + ADIO_Offset **buf_idx = NULL; int old_error, tmp_error; MPI_Comm_size(fd->comm, &nprocs); @@ -287,7 +288,7 @@ static void ADIOI_LUSTRE_Exch_and_write(ADIO_File fd, const void *buf, ADIO_Offset *offset_list, ADIO_Offset *len_list, int contig_access_count, - int *striping_info, int **buf_idx, + int *striping_info, ADIO_Offset **buf_idx, int *error_code) { /* Send data to appropriate processes and write in sizes of no more @@ -308,8 +309,8 @@ static void ADIOI_LUSTRE_Exch_and_write(ADIO_File fd, const void *buf, int *recv_curr_offlen_ptr, *recv_count, *recv_size; int *send_curr_offlen_ptr, *send_size; int *sent_to_proc, *recv_start_pos; - int *send_buf_idx, *curr_to_proc, *done_to_proc; - int *this_buf_idx; + int *curr_to_proc, *done_to_proc; + ADIO_Offset *send_buf_idx, *this_buf_idx; char *write_buf = NULL; MPI_Status status; ADIOI_Flatlist_node *flat_buf = NULL; @@ -399,12 +400,12 @@ static void ADIOI_LUSTRE_Exch_and_write(ADIO_File fd, const void *buf, /* amount of data sent to each proc so far. Used in ADIOI_Fill_send_buffer. initialized to 0 here. */ - send_buf_idx = (int *) ADIOI_Malloc(nprocs * sizeof(int)); + send_buf_idx = (ADIO_Offset *) ADIOI_Malloc(nprocs * sizeof(ADIO_Offset)); curr_to_proc = (int *) ADIOI_Malloc(nprocs * sizeof(int)); done_to_proc = (int *) ADIOI_Malloc(nprocs * sizeof(int)); /* Above three are used in ADIOI_Fill_send_buffer */ - this_buf_idx = (int *) ADIOI_Malloc(nprocs * sizeof(int)); + this_buf_idx = (ADIO_Offset *) ADIOI_Malloc(nprocs * sizeof(ADIO_Offset)); recv_start_pos = (int *) ADIOI_Malloc(nprocs * sizeof(int)); /* used to store the starting value of recv_curr_offlen_ptr[i] in @@ -620,11 +621,11 @@ static void ADIOI_LUSTRE_W_Exchange_data(ADIO_File fd, const void *buf, int contig_access_count, int *striping_info, ADIOI_Access *others_req, - int *send_buf_idx, + ADIO_Offset *send_buf_idx, int *curr_to_proc, int *done_to_proc, int *hole, int iter, MPI_Aint buftype_extent, - int *buf_idx, + ADIO_Offset *buf_idx, ADIO_Offset **srt_off, int **srt_len, int *srt_num, int *error_code) { @@ -889,7 +890,7 @@ static void ADIOI_LUSTRE_Fill_send_buffer(ADIO_File fd, const void *buf, int myrank, int contig_access_count, int *striping_info, - int *send_buf_idx, + ADIO_Offset *send_buf_idx, int *curr_to_proc, int *done_to_proc, int iter, MPI_Aint buftype_extent) ----------------------------------------------------------------------- Summary of changes: src/mpi/romio/adio/ad_lustre/ad_lustre.h | 2 +- src/mpi/romio/adio/ad_lustre/ad_lustre_aggregate.c | 18 ++-- src/mpi/romio/adio/ad_lustre/ad_lustre_rwcontig.c | 111 +++++++++++++------- src/mpi/romio/adio/ad_lustre/ad_lustre_wrcoll.c | 27 +++--- src/mpi/romio/adio/common/shfp_fname.c | 4 +- src/mpi/romio/mpi-io/mpir_cst_filesys.c | 2 +- src/mpl/src/timer/mpl_timer_clock_gettime.c | 2 +- 7 files changed, 102 insertions(+), 64 deletions(-) hooks/post-receive -- MPICH primary repository

1 0

[mpich] MPICH primary repository branch, master, updated. v3.2-320-ga131e53
by noreply＠mpich.org 24 May '16

24 May '16

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "MPICH primary repository". The branch, master has been updated via a131e534d26ea79ba028435b2a162e14b5c7764f (commit) from 8a5e24404f4ed060c083008cd7a5b8ba4c216c73 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- http://git.mpich.org/mpich.git/commitdiff/a131e534d26ea79ba028435b2a162e14b… commit a131e534d26ea79ba028435b2a162e14b5c7764f Author: Ken Raffenetti <raffenet(a)mcs.anl.gov> Date: Tue May 24 09:36:12 2016 -0500 pm/hydra: fix bug in default binding Setting a default binding of "none" meant that the HYDRA_BINDING environment variable was always ignored. Using NULL is actually what we want in this case. No reviewer. diff --git a/src/pm/hydra/pm/pmiserv/pmip_cb.c b/src/pm/hydra/pm/pmiserv/pmip_cb.c index a66916b..50cf38e 100644 --- a/src/pm/hydra/pm/pmiserv/pmip_cb.c +++ b/src/pm/hydra/pm/pmiserv/pmip_cb.c @@ -803,9 +803,6 @@ static HYD_status parse_exec_params(char **t_argv) "no executable given and doesn't look like a restart either\n"); /* Set default values */ - if (HYD_pmcd_pmip.user_global.binding == NULL) - HYD_pmcd_pmip.user_global.binding = MPL_strdup("none"); - if (HYD_pmcd_pmip.user_global.topolib == NULL && HYDRA_DEFAULT_TOPOLIB) HYD_pmcd_pmip.user_global.topolib = MPL_strdup(HYDRA_DEFAULT_TOPOLIB); ----------------------------------------------------------------------- Summary of changes: src/pm/hydra/pm/pmiserv/pmip_cb.c | 3 --- 1 files changed, 0 insertions(+), 3 deletions(-) hooks/post-receive -- MPICH primary repository

1 0

[mpich] MPICH primary repository branch, master, updated. v3.2-319-g8a5e244
by noreply＠mpich.org 24 May '16

24 May '16

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "MPICH primary repository". The branch, master has been updated via 8a5e24404f4ed060c083008cd7a5b8ba4c216c73 (commit) from 096302d9f9c7b06439185feb82c3ea497909281f (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- http://git.mpich.org/mpich.git/commitdiff/8a5e24404f4ed060c083008cd7a5b8ba4… commit 8a5e24404f4ed060c083008cd7a5b8ba4c216c73 Author: Rob Latham <robl(a)mcs.anl.gov> Date: Mon Feb 29 13:23:05 2016 -0600 mainline bug fix: struct and zero-length elements if items with a zero-length block length are given, update the flat->count accordingly (as is already done in indexed and hindexed cases) No Reviewer diff --git a/src/mpi/romio/adio/common/flatten.c b/src/mpi/romio/adio/common/flatten.c index d6b1013..398e7ab 100644 --- a/src/mpi/romio/adio/common/flatten.c +++ b/src/mpi/romio/adio/common/flatten.c @@ -717,6 +717,9 @@ void ADIOI_Flatten(MPI_Datatype datatype, ADIOI_Flatlist_node *flat, DBG_FPRINTF(stderr,"ADIOI_Flatten:: simple adds[%#X] "MPI_AINT_FMT_HEX_SPEC", flat->indices[%#llX] %#llX, flat->blocklens[%#llX] %#llX\n",n,adds[n],j, flat->indices[j], j, flat->blocklens[j]); #endif (*curr_index)++; + } else { + flat->count--; /* don't count/consider any zero-length + blocklens */ } } else { ----------------------------------------------------------------------- Summary of changes: src/mpi/romio/adio/common/flatten.c | 3 +++ 1 files changed, 3 insertions(+), 0 deletions(-) hooks/post-receive -- MPICH primary repository

1 0

[mpich] MPICH primary repository branch, master, updated. v3.2-318-g096302d
by noreply＠mpich.org 24 May '16

24 May '16

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "MPICH primary repository". The branch, master has been updated via 096302d9f9c7b06439185feb82c3ea497909281f (commit) from 1bcb945c1f1636d838ba40d5b2f6980218d80802 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- http://git.mpich.org/mpich.git/commitdiff/096302d9f9c7b06439185feb82c3ea497… commit 096302d9f9c7b06439185feb82c3ea497909281f Author: Gilles Gouaillardet <gilles(a)rist.or.jp> Date: Mon May 9 13:54:46 2016 +0900 io/romio: fix filesystem type check on OpenBSD 5.7 check the existence of the f_type field in struct statfs Thanks Paul Hargrove for the report Signed-off-by: Rob Latham <robl(a)mcs.anl.gov> diff --git a/src/mpi/romio/adio/common/ad_fstype.c b/src/mpi/romio/adio/common/ad_fstype.c index 51893c0..94451c6 100644 --- a/src/mpi/romio/adio/common/ad_fstype.c +++ b/src/mpi/romio/adio/common/ad_fstype.c @@ -338,7 +338,7 @@ static void ADIO_FileSysType_fncall(const char *filename, int *fstype, int *erro /* --END ERROR HANDLING-- */ # ifdef ROMIO_HAVE_STRUCT_STATFS_WITH_F_FSTYPENAME - /* uncommon: maybe only on Darwin ? */ + /* less common: Darwin and OpenBSD */ if ( !strncmp("nfs",fsbuf.f_fstypename,3) ) { *fstype = ADIO_NFS; return; @@ -346,6 +346,8 @@ static void ADIO_FileSysType_fncall(const char *filename, int *fstype, int *erro # endif +# ifdef ROMIO_HAVE_STRUCT_STATFS_WITH_F_TYPE + #ifdef ROMIO_GPFS if (fsbuf.f_type == GPFS_SUPER_MAGIC) { *fstype = ADIO_GPFS; @@ -413,6 +415,8 @@ static void ADIO_FileSysType_fncall(const char *filename, int *fstype, int *erro } # endif +# endif /*ROMIO_HAVE_STRUCT_STATFS_WITH_F_TYPE */ + # ifdef ROMIO_UFS /* if UFS support is enabled, default to that */ *fstype = ADIO_UFS; diff --git a/src/mpi/romio/configure.ac b/src/mpi/romio/configure.ac index 250391d..67732c4 100644 --- a/src/mpi/romio/configure.ac +++ b/src/mpi/romio/configure.ac @@ -1177,12 +1177,54 @@ AC_MSG_RESULT($pac_cv_have_statfs) if test "$pac_cv_have_statfs" = yes ; then AC_DEFINE(HAVE_STRUCT_STATFS,1,[Define if struct statfs can be compiled]) fi - +AC_MSG_CHECKING([for f_type member of statfs structure]) +AC_TRY_COMPILE([ +#ifdef HAVE_SYS_VFS_H +#include <sys/vfs.h> +#endif +#ifdef HAVE_SYS_STATVFS_H +#include <sys/statvfs.h> +#endif +#ifdef HAVE_SYS_PARAM_H +#include <sys/param.h> +#endif +#ifdef HAVE_SYS_MOUNT_H +#include <sys/mount.h> +#endif +#ifdef HAVE_STRING_H +#include <string.h> +#endif + ],[ + struct statfs f; + memset(&f, 0, sizeof(f)); + f.f_type = 0; + ], + pac_cv_have_statfs_f_type=yes, + pac_cv_have_statfs_f_type=no +) +AC_MSG_RESULT($pac_cv_have_statfs_f_type) +if test $pac_cv_have_statfs_f_type = yes ; then + AC_DEFINE(ROMIO_HAVE_STRUCT_STATFS_WITH_F_TYPE, 1,[Define if statfs has f_type]) +fi + + AC_MSG_CHECKING([for f_fstypename member of statfs structure]) AC_TRY_COMPILE([ +#ifdef HAVE_SYS_VFS_H +#include <sys/vfs.h> +#endif +#ifdef HAVE_SYS_STATVFS_H +#include <sys/statvfs.h> +#endif +#ifdef HAVE_SYS_PARAM_H #include <sys/param.h> +#endif +#ifdef HAVE_SYS_MOUNT_H #include <sys/mount.h> +#endif +#ifdef HAVE_STRING_H #include <string.h> +#endif ],[ struct statfs f; memset(&f, 0, sizeof(f)); ----------------------------------------------------------------------- Summary of changes: src/mpi/romio/adio/common/ad_fstype.c | 6 +++- src/mpi/romio/configure.ac | 44 ++++++++++++++++++++++++++++++++- 2 files changed, 48 insertions(+), 2 deletions(-) hooks/post-receive -- MPICH primary repository

1 0

[mpich] MPICH primary repository branch, master, updated. v3.2-317-g1bcb945
by noreply＠mpich.org 20 May '16

20 May '16

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "MPICH primary repository". The branch, master has been updated via 1bcb945c1f1636d838ba40d5b2f6980218d80802 (commit) via 16b696194e26ea1517a0a4cd4ff1ce73ab1c9ee3 (commit) via 79a2a5aa19355ab17253d28376564a07a238b191 (commit) from b6e85ed8e711f352d15b9b98b7ea2e7f84c912f0 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- http://git.mpich.org/mpich.git/commitdiff/1bcb945c1f1636d838ba40d5b2f698021… commit 1bcb945c1f1636d838ba40d5b2f6980218d80802 Author: Ken Raffenetti <raffenet(a)mcs.anl.gov> Date: Fri May 20 09:25:06 2016 -0500 test/mpi/.gitignore: add missing f08 test artifacts No reviewer. diff --git a/test/mpi/.gitignore b/test/mpi/.gitignore index 4466ece..2a9d871 100644 --- a/test/mpi/.gitignore +++ b/test/mpi/.gitignore @@ -464,6 +464,105 @@ /f08/io/iwriteatallf90 /f08/io/i_fcoll_test /f08/io/testlist +/f08/attr/attrlangf08 +/f08/attr/attrmpi1f08 +/f08/attr/baseattr2f08 +/f08/attr/baseattr3f08 +/f08/attr/baseattrf08 +/f08/attr/commattr2f08 +/f08/attr/commattr3f08 +/f08/attr/commattrf08 +/f08/attr/fandcattrf08 +/f08/attr/typeattr2f08 +/f08/attr/typeattr3f08 +/f08/attr/typeattrf08 +/f08/comm/commerrf08 +/f08/comm/commnamef08 +/f08/ext/c2f2cf90 +/f08/ext/c2f902c.c +/f08/ext/c2f90mult +/f08/ext/ctypesfromc.c +/f08/ext/ctypesinf90 +/f08/info/infotest2f90 +/f08/info/infotestf90 +/f08/init/baseenvf90 +/f08/io/atomicityf90 +/f08/io/c2f2ciof90 +/f08/io/c2f90multio +/f08/io/fileerrf90 +/f08/io/fileinfof90 +/f08/io/iwriteatf90 +/f08/io/iwriteatf90.f90 +/f08/io/iwritef90 +/f08/io/iwritef90.f90 +/f08/io/iwriteshf90 +/f08/io/iwriteshf90.f90 +/f08/io/miscfilef90 +/f08/io/setviewcurf90 +/f08/io/shpositionf90 +/f08/io/writeallbef90 +/f08/io/writeallbef90.f90 +/f08/io/writeallf90 +/f08/io/writeallf90.f90 +/f08/io/writeatallbef90 +/f08/io/writeatallbef90.f90 +/f08/io/writeatallf90 +/f08/io/writeatallf90.f90 +/f08/io/writeatf90 +/f08/io/writeatf90.f90 +/f08/io/writef90 +/f08/io/writef90.f90 +/f08/io/writeordbef90 +/f08/io/writeordbef90.f90 +/f08/io/writeordf90 +/f08/io/writeordf90.f90 +/f08/io/writeshf90 +/f08/io/writeshf90.f90 +/f08/misc/hello +/f08/misc/ring +/f08/misc/sizeof2 +/f08/profile/profile1f90 +/f08/rma/aintf08 +/f08/rma/baseattrwinf08 +/f08/rma/c2f2cwinf08 +/f08/rma/winaccf08 +/f08/rma/winattr2f08 +/f08/rma/winattrf08 +/f08/rma/winerrf08 +/f08/rma/winfencef08 +/f08/rma/wingetf08 +/f08/rma/wingroupf08 +/f08/rma/winnamef08 +/f08/rma/winscale1f08 +/f08/rma/winscale2f08 +/f08/spawn/connaccf90 +/f08/spawn/namepubf90 +/f08/spawn/spawnargvf03 +/f08/spawn/spawnargvf90 +/f08/spawn/spawnf90 +/f08/spawn/spawnmult2f90 +/f08/spawn/spawnmultf03 +/f08/spawn/spawnmultf90 +/f08/subarray/test1 +/f08/subarray/test10 +/f08/subarray/test11 +/f08/subarray/test12 +/f08/subarray/test13 +/f08/subarray/test14 +/f08/subarray/test15 +/f08/subarray/test2 +/f08/subarray/test3 +/f08/subarray/test4 +/f08/subarray/test5 +/f08/subarray/test6 +/f08/subarray/test7 +/f08/subarray/test8 +/f08/subarray/test9 +/f08/timer/wtimef90 +/f08/topo/cart_subf90 +/f08/topo/cartcrf90 +/f08/topo/dgraph_unwgtf90 +/f08/topo/dgraph_wgtf90 /group/errstring /group/grouptest /group/groupcreate http://git.mpich.org/mpich.git/commitdiff/16b696194e26ea1517a0a4cd4ff1ce73a… commit 16b696194e26ea1517a0a4cd4ff1ce73ab1c9ee3 Author: Artem Yalozo <artem.v.yalozo(a)intel.com> Date: Thu May 19 16:34:22 2016 -0500 test/mpi/f08: add test for MPI_CART_SUB Add a simple regression test to ensure MPI_CART_SUB returns a valid communicator when using the F08 bindings. Signed-off-by: Ken Raffenetti <raffenet(a)mcs.anl.gov> diff --git a/test/mpi/f08/topo/Makefile.am b/test/mpi/f08/topo/Makefile.am index 78dce0a..7872054 100644 --- a/test/mpi/f08/topo/Makefile.am +++ b/test/mpi/f08/topo/Makefile.am @@ -11,4 +11,4 @@ EXTRA_DIST = testlist AM_DEFAULT_SOURCE_EXT = .f90 -noinst_PROGRAMS = cartcrf90 dgraph_wgtf90 dgraph_unwgtf90 +noinst_PROGRAMS = cartcrf90 dgraph_wgtf90 dgraph_unwgtf90 cart_subf90 diff --git a/test/mpi/f08/topo/cart_subf90.f90 b/test/mpi/f08/topo/cart_subf90.f90 new file mode 100644 index 0000000..3bd4344 --- /dev/null +++ b/test/mpi/f08/topo/cart_subf90.f90 @@ -0,0 +1,42 @@ +! -*- Mode: Fortran; -*- +! +! (C) 2016 by Argonne National Laboratory. +! See COPYRIGHT in top-level directory. +! +! Portions of this code were written by Intel Corporation. +! Copyright (C) 2011-2016 Intel Corporation. Intel provides this material +! to Argonne National Laboratory subject to Software Grant and Corporate +! Contributor License Agreement dated February 8, 2012. + +program main +use mpi_f08 +implicit none +integer rank, size + +type(MPI_Comm) :: comm_cart, comm_new +integer dims(2), coords(2) +logical periods(2), reorder, remain_dims(2) +integer errs + +dims(1:2) = 0 +periods(1) = .TRUE. +periods(2) = .FALSE. +reorder = .TRUE. +remain_dims(1) = .TRUE. +remain_dims(2) = .FALSE. +errs = 0 + +call MTEST_Init() +call MPI_Comm_rank(MPI_COMM_WORLD, rank) +call MPI_Comm_size(MPI_COMM_WORLD, size) +call MPI_Dims_create(size, 2, dims) +call MPI_Cart_create(MPI_COMM_WORLD, 2, dims, periods, reorder, comm_cart) +call MPI_Comm_rank(comm_cart, rank) +call MPI_Cart_coords(comm_cart, rank, 2, coords) + +call MPI_Cart_sub(comm_cart, remain_dims, comm_new) +call MPI_Comm_size(comm_new, size) + +call MTEST_Finalize(errs) + +end program diff --git a/test/mpi/f08/topo/testlist b/test/mpi/f08/topo/testlist index 9836414..9b9685e 100644 --- a/test/mpi/f08/topo/testlist +++ b/test/mpi/f08/topo/testlist @@ -1,3 +1,4 @@ cartcrf90 4 dgraph_wgtf90 4 dgraph_unwgtf90 4 +cart_subf90 4 http://git.mpich.org/mpich.git/commitdiff/79a2a5aa19355ab17253d28376564a07a… commit 79a2a5aa19355ab17253d28376564a07a238b191 Author: Artem Yalozo <artem.v.yalozo(a)intel.com> Date: Wed May 18 16:45:43 2016 +0300 binding/f08: fix bug in MPI_CART_SUB wrapper Fixes inverted logic in Fortran wrapper for MPI_CART_SUB. Change-Id: I73605eefe65364b6d10f7f35d43388aa1fbf1214 Signed-off-by: Ken Raffenetti <raffenet(a)mcs.anl.gov> diff --git a/src/binding/fortran/use_mpi_f08/wrappers_f/cart_sub_f08ts.f90 b/src/binding/fortran/use_mpi_f08/wrappers_f/cart_sub_f08ts.f90 index aaea10d..3487de2 100644 --- a/src/binding/fortran/use_mpi_f08/wrappers_f/cart_sub_f08ts.f90 +++ b/src/binding/fortran/use_mpi_f08/wrappers_f/cart_sub_f08ts.f90 @@ -25,7 +25,7 @@ subroutine MPI_Cart_sub_f08(comm, remain_dims, newcomm, ierror) comm_c = comm%MPI_VAL ierror_c = MPIR_Cartdim_get_c(comm_c, ndims) - if (ierror_c /= MPI_SUCCESS) then + if (ierror_c == MPI_SUCCESS) then remain_dims_c = merge(1, 0, remain_dims(1:ndims)) ierror_c = MPIR_Cart_sub_c(comm_c, remain_dims_c, newcomm_c) newcomm%MPI_VAL = newcomm_c ----------------------------------------------------------------------- Summary of changes: .../use_mpi_f08/wrappers_f/cart_sub_f08ts.f90 | 2 +- test/mpi/.gitignore | 99 ++++++++++++++++++++ test/mpi/f08/topo/Makefile.am | 2 +- test/mpi/f08/topo/cart_subf90.f90 | 42 ++++++++ test/mpi/f08/topo/testlist | 1 + 5 files changed, 144 insertions(+), 2 deletions(-) create mode 100644 test/mpi/f08/topo/cart_subf90.f90 hooks/post-receive -- MPICH primary repository

1 0

[mpich] MPICH primary repository branch, master, updated. v3.2-314-gb6e85ed
by noreply＠mpich.org 19 May '16

19 May '16

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "MPICH primary repository". The branch, master has been updated via b6e85ed8e711f352d15b9b98b7ea2e7f84c912f0 (commit) via 820a98f8289d0e18a77144b7036ca1da3848461a (commit) via 2de8f043d9992e338084c49887710794809b2b46 (commit) via 998a5d42ee48ad5e5a7048215edeae01bf5f27a2 (commit) via bb0e0b7a1e437ad95ee354bfa68b5bd15875cb8c (commit) via e01832a6f01292f4a1ce0fd06c67c43cfccde646 (commit) via 9c05f2563d632f9a855c9a78f86aa913292f1bd2 (commit) via 743a5ca7420f918503ff46f2737e92833aeeee4a (commit) via d0ce2eae76bc1eb3ceeb865886632ccaf6937d4a (commit) via 44388f15b7f13f291d40d7d4e71dfb2c419de6bd (commit) via 5a53eb25a721a0f46da5559d9e10e3cd16ce2911 (commit) from 86bb0020b7e66b75cf0370d2dbe31bd31b3bb287 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- http://git.mpich.org/mpich.git/commitdiff/b6e85ed8e711f352d15b9b98b7ea2e7f8… commit b6e85ed8e711f352d15b9b98b7ea2e7f84c912f0 Author: Yanfei Guo <yguo(a)anl.gov> Date: Mon May 2 10:09:34 2016 -0500 mpl/shm: move shm support files to MPL layer Moved shm wrappers to MPL, changed namespace accordingly to MPL_shm. Removed internal error checking code, instead just return errors to the caller. The implementation of SYSV, MMAP, and NT shm are seperated into different files in MPL. This patch is based on the original SHM cleanup patch from Gengbin Zheng <gengbin.zheng(a)intel.com> Signed-off-by: Ken Raffenetti <raffenet(a)mcs.anl.gov> diff --git a/src/mpid/ch3/channels/nemesis/include/mpid_nem_defs.h b/src/mpid/ch3/channels/nemesis/include/mpid_nem_defs.h index f3fee81..51d7676 100644 --- a/src/mpid/ch3/channels/nemesis/include/mpid_nem_defs.h +++ b/src/mpid/ch3/channels/nemesis/include/mpid_nem_defs.h @@ -10,7 +10,6 @@ #include "mpid_nem_datatypes.h" #include "mpi.h" #include "mpiimpl.h" -#include "mpiu_os_wrappers_pre.h" #include "mpidu_shm.h" /* FIXME: This definition should be gotten from mpidi_ch3_impl.h */ diff --git a/src/mpid/ch3/channels/nemesis/include/mpid_nem_pre.h b/src/mpid/ch3/channels/nemesis/include/mpid_nem_pre.h index 1aaa4ce..908a1a2 100644 --- a/src/mpid/ch3/channels/nemesis/include/mpid_nem_pre.h +++ b/src/mpid/ch3/channels/nemesis/include/mpid_nem_pre.h @@ -19,14 +19,14 @@ typedef pthread_mutex_t MPIDI_CH3I_SHM_MUTEX; #define MPIDI_CH3_WIN_DECL \ void *shm_base_addr; /* base address of shared memory region */ \ MPI_Aint shm_segment_len; /* size of shared memory region */ \ - MPIU_SHMW_Hnd_t shm_segment_handle; /* handle to shared memory region */ \ + MPL_shm_hnd_t shm_segment_handle; /* handle to shared memory region */ \ MPIDI_CH3I_SHM_MUTEX *shm_mutex; /* shared memory windows -- lock for \ accumulate/atomic operations */ \ - MPIU_SHMW_Hnd_t shm_mutex_segment_handle; /* handle to interprocess mutex memory \ + MPL_shm_hnd_t shm_mutex_segment_handle; /* handle to interprocess mutex memory \ region */ \ \ void *info_shm_base_addr; /* base address of shared memory region for window info */ \ MPI_Aint info_shm_segment_len; /* size of shared memory region for window info */ \ - MPIU_SHMW_Hnd_t info_shm_segment_handle; /* handle to shared memory region for window info */ \ + MPL_shm_hnd_t info_shm_segment_handle; /* handle to shared memory region for window info */ \ #endif /* MPID_NEM_PRE_H */ diff --git a/src/mpid/ch3/channels/nemesis/include/mpidi_ch3_impl.h b/src/mpid/ch3/channels/nemesis/include/mpidi_ch3_impl.h index 92edc88..7719563 100644 --- a/src/mpid/ch3/channels/nemesis/include/mpidi_ch3_impl.h +++ b/src/mpid/ch3/channels/nemesis/include/mpidi_ch3_impl.h @@ -8,7 +8,6 @@ #define MPICH_MPIDI_CH3_IMPL_H_INCLUDED #include "mpidimpl.h" -#include "mpiu_os_wrappers.h" #include "mpidu_generic_queue.h" #include "mpl_utlist.h" diff --git a/src/mpid/ch3/channels/nemesis/include/mpidi_ch3_pre.h b/src/mpid/ch3/channels/nemesis/include/mpidi_ch3_pre.h index 44aedc8..a9f05be 100644 --- a/src/mpid/ch3/channels/nemesis/include/mpidi_ch3_pre.h +++ b/src/mpid/ch3/channels/nemesis/include/mpidi_ch3_pre.h @@ -128,8 +128,8 @@ typedef struct MPIDI_CH3I_VC /* LMT shared memory copy-buffer ptr */ struct MPID_nem_copy_buf *lmt_copy_buf; - MPIU_SHMW_Hnd_t lmt_copy_buf_handle; - MPIU_SHMW_Hnd_t lmt_recv_copy_buf_handle; + MPL_shm_hnd_t lmt_copy_buf_handle; + MPL_shm_hnd_t lmt_recv_copy_buf_handle; int lmt_buf_num; intptr_t lmt_surfeit; struct {struct MPID_nem_lmt_shm_wait_element *head, *tail;} lmt_queue; diff --git a/src/mpid/ch3/channels/nemesis/src/ch3_progress.c b/src/mpid/ch3/channels/nemesis/src/ch3_progress.c index 35b03b3..e363566 100644 --- a/src/mpid/ch3/channels/nemesis/src/ch3_progress.c +++ b/src/mpid/ch3/channels/nemesis/src/ch3_progress.c @@ -1045,9 +1045,9 @@ static int shm_connection_terminated(MPIDI_VC_t * vc) if (mpi_errno) MPIR_ERR_POP(mpi_errno); } - mpi_errno = MPIU_SHMW_Hnd_finalize(&(vc->ch.lmt_copy_buf_handle)); + mpi_errno = MPL_shm_hnd_finalize(&(vc->ch.lmt_copy_buf_handle)); if(mpi_errno != MPI_SUCCESS) { MPIR_ERR_POP(mpi_errno); } - mpi_errno = MPIU_SHMW_Hnd_finalize(&(vc->ch.lmt_recv_copy_buf_handle)); + mpi_errno = MPL_shm_hnd_finalize(&(vc->ch.lmt_recv_copy_buf_handle)); if(mpi_errno != MPI_SUCCESS) { MPIR_ERR_POP(mpi_errno); } mpi_errno = MPIDI_CH3U_Handle_connection(vc, MPIDI_VC_EVENT_TERMINATED); diff --git a/src/mpid/ch3/channels/nemesis/src/ch3_rma_shm.c b/src/mpid/ch3/channels/nemesis/src/ch3_rma_shm.c index ff5eeed..28086e1 100644 --- a/src/mpid/ch3/channels/nemesis/src/ch3_rma_shm.c +++ b/src/mpid/ch3/channels/nemesis/src/ch3_rma_shm.c @@ -97,13 +97,13 @@ int MPIDI_CH3_SHM_Win_free(MPIR_Win ** win_ptr) (*win_ptr)->shm_segment_len > 0) { /* detach from shared memory segment */ mpi_errno = - MPIU_SHMW_Seg_detach((*win_ptr)->shm_segment_handle, + MPL_shm_seg_detach((*win_ptr)->shm_segment_handle, (char **) &(*win_ptr)->shm_base_addr, (*win_ptr)->shm_segment_len); if (mpi_errno) MPIR_ERR_POP(mpi_errno); - MPIU_SHMW_Hnd_finalize(&(*win_ptr)->shm_segment_handle); + MPL_shm_hnd_finalize(&(*win_ptr)->shm_segment_handle); } } @@ -128,23 +128,23 @@ int MPIDI_CH3_SHM_Win_free(MPIR_Win ** win_ptr) /* detach from shared memory segment */ mpi_errno = - MPIU_SHMW_Seg_detach((*win_ptr)->shm_mutex_segment_handle, + MPL_shm_seg_detach((*win_ptr)->shm_mutex_segment_handle, (char **) &(*win_ptr)->shm_mutex, sizeof(MPIDI_CH3I_SHM_MUTEX)); if (mpi_errno) MPIR_ERR_POP(mpi_errno); - MPIU_SHMW_Hnd_finalize(&(*win_ptr)->shm_mutex_segment_handle); + MPL_shm_hnd_finalize(&(*win_ptr)->shm_mutex_segment_handle); } /* Free shared memory region for window info */ if ((*win_ptr)->info_shm_base_addr != NULL) { - mpi_errno = MPIU_SHMW_Seg_detach((*win_ptr)->info_shm_segment_handle, + mpi_errno = MPL_shm_seg_detach((*win_ptr)->info_shm_segment_handle, (char **) &(*win_ptr)->info_shm_base_addr, (*win_ptr)->info_shm_segment_len); if (mpi_errno != MPI_SUCCESS) MPIR_ERR_POP(mpi_errno); - MPIU_SHMW_Hnd_finalize(&(*win_ptr)->info_shm_segment_handle); + MPL_shm_hnd_finalize(&(*win_ptr)->info_shm_segment_handle); (*win_ptr)->basic_info_table = NULL; } diff --git a/src/mpid/ch3/channels/nemesis/src/ch3_win_fns.c b/src/mpid/ch3/channels/nemesis/src/ch3_win_fns.c index 6e5937a..eec9c7c 100644 --- a/src/mpid/ch3/channels/nemesis/src/ch3_win_fns.c +++ b/src/mpid/ch3/channels/nemesis/src/ch3_win_fns.c @@ -376,7 +376,7 @@ static int MPIDI_CH3I_Win_gather_info(void *base, MPI_Aint size, int disp_unit, (*win_ptr)->info_shm_segment_len = comm_size * sizeof(MPIDI_Win_basic_info_t); - mpi_errno = MPIU_SHMW_Hnd_init(&(*win_ptr)->info_shm_segment_handle); + mpi_errno = MPL_shm_hnd_init(&(*win_ptr)->info_shm_segment_handle); if (mpi_errno != MPI_SUCCESS) MPIR_ERR_POP(mpi_errno); @@ -384,7 +384,7 @@ static int MPIDI_CH3I_Win_gather_info(void *base, MPI_Aint size, int disp_unit, char *serialized_hnd_ptr = NULL; /* create shared memory region for all processes in win and map. */ - mpi_errno = MPIU_SHMW_Seg_create_and_attach((*win_ptr)->info_shm_segment_handle, + mpi_errno = MPL_shm_seg_create_and_attach((*win_ptr)->info_shm_segment_handle, (*win_ptr)->info_shm_segment_len, (char **) &(*win_ptr)->info_shm_base_addr, 0); if (mpi_errno) @@ -392,13 +392,13 @@ static int MPIDI_CH3I_Win_gather_info(void *base, MPI_Aint size, int disp_unit, /* serialize handle and broadcast it to the other processes in win */ mpi_errno = - MPIU_SHMW_Hnd_get_serialized_by_ref((*win_ptr)->info_shm_segment_handle, + MPL_shm_hnd_get_serialized_by_ref((*win_ptr)->info_shm_segment_handle, &serialized_hnd_ptr); if (mpi_errno) MPIR_ERR_POP(mpi_errno); mpi_errno = - MPIR_Bcast_impl(serialized_hnd_ptr, MPIU_SHMW_GHND_SZ, MPI_CHAR, 0, node_comm_ptr, + MPIR_Bcast_impl(serialized_hnd_ptr, MPL_SHM_GHND_SZ, MPI_CHAR, 0, node_comm_ptr, &errflag); if (mpi_errno) MPIR_ERR_POP(mpi_errno); @@ -411,29 +411,29 @@ static int MPIDI_CH3I_Win_gather_info(void *base, MPI_Aint size, int disp_unit, MPIR_ERR_CHKANDJUMP(errflag, mpi_errno, MPI_ERR_OTHER, "**coll_fail"); /* unlink shared memory region so it gets deleted when all processes exit */ - mpi_errno = MPIU_SHMW_Seg_remove((*win_ptr)->info_shm_segment_handle); + mpi_errno = MPL_shm_seg_remove((*win_ptr)->info_shm_segment_handle); if (mpi_errno) MPIR_ERR_POP(mpi_errno); } else { - char serialized_hnd[MPIU_SHMW_GHND_SZ] = { 0 }; + char serialized_hnd[MPL_SHM_GHND_SZ] = { 0 }; /* get serialized handle from rank 0 and deserialize it */ mpi_errno = - MPIR_Bcast_impl(serialized_hnd, MPIU_SHMW_GHND_SZ, MPI_CHAR, 0, node_comm_ptr, + MPIR_Bcast_impl(serialized_hnd, MPL_SHM_GHND_SZ, MPI_CHAR, 0, node_comm_ptr, &errflag); if (mpi_errno) MPIR_ERR_POP(mpi_errno); MPIR_ERR_CHKANDJUMP(errflag, mpi_errno, MPI_ERR_OTHER, "**coll_fail"); - mpi_errno = MPIU_SHMW_Hnd_deserialize((*win_ptr)->info_shm_segment_handle, serialized_hnd, + mpi_errno = MPL_shm_hnd_deserialize((*win_ptr)->info_shm_segment_handle, serialized_hnd, strlen(serialized_hnd)); if (mpi_errno) MPIR_ERR_POP(mpi_errno); /* attach to shared memory region created by rank 0 */ mpi_errno = - MPIU_SHMW_Seg_attach((*win_ptr)->info_shm_segment_handle, + MPL_shm_seg_attach((*win_ptr)->info_shm_segment_handle, (*win_ptr)->info_shm_segment_len, (char **) &(*win_ptr)->info_shm_base_addr, 0); if (mpi_errno) @@ -563,7 +563,7 @@ static int MPIDI_CH3I_Win_allocate_shm(MPI_Aint size, int disp_unit, MPIR_Info * } else { - mpi_errno = MPIU_SHMW_Hnd_init(&(*win_ptr)->shm_segment_handle); + mpi_errno = MPL_shm_hnd_init(&(*win_ptr)->shm_segment_handle); if (mpi_errno) MPIR_ERR_POP(mpi_errno); @@ -572,7 +572,7 @@ static int MPIDI_CH3I_Win_allocate_shm(MPI_Aint size, int disp_unit, MPIR_Info * /* create shared memory region for all processes in win and map */ mpi_errno = - MPIU_SHMW_Seg_create_and_attach((*win_ptr)->shm_segment_handle, + MPL_shm_seg_create_and_attach((*win_ptr)->shm_segment_handle, (*win_ptr)->shm_segment_len, (char **) &(*win_ptr)->shm_base_addr, 0); if (mpi_errno) @@ -580,13 +580,13 @@ static int MPIDI_CH3I_Win_allocate_shm(MPI_Aint size, int disp_unit, MPIR_Info * /* serialize handle and broadcast it to the other processes in win */ mpi_errno = - MPIU_SHMW_Hnd_get_serialized_by_ref((*win_ptr)->shm_segment_handle, + MPL_shm_hnd_get_serialized_by_ref((*win_ptr)->shm_segment_handle, &serialized_hnd_ptr); if (mpi_errno) MPIR_ERR_POP(mpi_errno); mpi_errno = - MPIR_Bcast_impl(serialized_hnd_ptr, MPIU_SHMW_GHND_SZ, MPI_CHAR, 0, node_comm_ptr, + MPIR_Bcast_impl(serialized_hnd_ptr, MPL_SHM_GHND_SZ, MPI_CHAR, 0, node_comm_ptr, &errflag); if (mpi_errno) MPIR_ERR_POP(mpi_errno); @@ -599,31 +599,31 @@ static int MPIDI_CH3I_Win_allocate_shm(MPI_Aint size, int disp_unit, MPIR_Info * MPIR_ERR_CHKANDJUMP(errflag, mpi_errno, MPI_ERR_OTHER, "**coll_fail"); /* unlink shared memory region so it gets deleted when all processes exit */ - mpi_errno = MPIU_SHMW_Seg_remove((*win_ptr)->shm_segment_handle); + mpi_errno = MPL_shm_seg_remove((*win_ptr)->shm_segment_handle); if (mpi_errno) MPIR_ERR_POP(mpi_errno); } else { - char serialized_hnd[MPIU_SHMW_GHND_SZ] = { 0 }; + char serialized_hnd[MPL_SHM_GHND_SZ] = { 0 }; /* get serialized handle from rank 0 and deserialize it */ mpi_errno = - MPIR_Bcast_impl(serialized_hnd, MPIU_SHMW_GHND_SZ, MPI_CHAR, 0, node_comm_ptr, + MPIR_Bcast_impl(serialized_hnd, MPL_SHM_GHND_SZ, MPI_CHAR, 0, node_comm_ptr, &errflag); if (mpi_errno) MPIR_ERR_POP(mpi_errno); MPIR_ERR_CHKANDJUMP(errflag, mpi_errno, MPI_ERR_OTHER, "**coll_fail"); mpi_errno = - MPIU_SHMW_Hnd_deserialize((*win_ptr)->shm_segment_handle, serialized_hnd, + MPL_shm_hnd_deserialize((*win_ptr)->shm_segment_handle, serialized_hnd, strlen(serialized_hnd)); if (mpi_errno) MPIR_ERR_POP(mpi_errno); /* attach to shared memory region created by rank 0 */ mpi_errno = - MPIU_SHMW_Seg_attach((*win_ptr)->shm_segment_handle, (*win_ptr)->shm_segment_len, + MPL_shm_seg_attach((*win_ptr)->shm_segment_handle, (*win_ptr)->shm_segment_len, (char **) &(*win_ptr)->shm_base_addr, 0); if (mpi_errno) MPIR_ERR_POP(mpi_errno); @@ -635,7 +635,7 @@ static int MPIDI_CH3I_Win_allocate_shm(MPI_Aint size, int disp_unit, MPIR_Info * } /* Allocated the interprocess mutex segment. */ - mpi_errno = MPIU_SHMW_Hnd_init(&(*win_ptr)->shm_mutex_segment_handle); + mpi_errno = MPL_shm_hnd_init(&(*win_ptr)->shm_mutex_segment_handle); if (mpi_errno) MPIR_ERR_POP(mpi_errno); @@ -644,7 +644,7 @@ static int MPIDI_CH3I_Win_allocate_shm(MPI_Aint size, int disp_unit, MPIR_Info * /* create shared memory region for all processes in win and map */ mpi_errno = - MPIU_SHMW_Seg_create_and_attach((*win_ptr)->shm_mutex_segment_handle, + MPL_shm_seg_create_and_attach((*win_ptr)->shm_mutex_segment_handle, sizeof(MPIDI_CH3I_SHM_MUTEX), (char **) &(*win_ptr)->shm_mutex, 0); if (mpi_errno) @@ -654,13 +654,13 @@ static int MPIDI_CH3I_Win_allocate_shm(MPI_Aint size, int disp_unit, MPIR_Info * /* serialize handle and broadcast it to the other processes in win */ mpi_errno = - MPIU_SHMW_Hnd_get_serialized_by_ref((*win_ptr)->shm_mutex_segment_handle, + MPL_shm_hnd_get_serialized_by_ref((*win_ptr)->shm_mutex_segment_handle, &serialized_hnd_ptr); if (mpi_errno) MPIR_ERR_POP(mpi_errno); mpi_errno = - MPIR_Bcast_impl(serialized_hnd_ptr, MPIU_SHMW_GHND_SZ, MPI_CHAR, 0, node_comm_ptr, + MPIR_Bcast_impl(serialized_hnd_ptr, MPL_SHM_GHND_SZ, MPI_CHAR, 0, node_comm_ptr, &errflag); if (mpi_errno) MPIR_ERR_POP(mpi_errno); @@ -673,30 +673,30 @@ static int MPIDI_CH3I_Win_allocate_shm(MPI_Aint size, int disp_unit, MPIR_Info * MPIR_ERR_CHKANDJUMP(errflag, mpi_errno, MPI_ERR_OTHER, "**coll_fail"); /* unlink shared memory region so it gets deleted when all processes exit */ - mpi_errno = MPIU_SHMW_Seg_remove((*win_ptr)->shm_mutex_segment_handle); + mpi_errno = MPL_shm_seg_remove((*win_ptr)->shm_mutex_segment_handle); if (mpi_errno) MPIR_ERR_POP(mpi_errno); } else { - char serialized_hnd[MPIU_SHMW_GHND_SZ] = { 0 }; + char serialized_hnd[MPL_SHM_GHND_SZ] = { 0 }; /* get serialized handle from rank 0 and deserialize it */ mpi_errno = - MPIR_Bcast_impl(serialized_hnd, MPIU_SHMW_GHND_SZ, MPI_CHAR, 0, node_comm_ptr, + MPIR_Bcast_impl(serialized_hnd, MPL_SHM_GHND_SZ, MPI_CHAR, 0, node_comm_ptr, &errflag); if (mpi_errno) MPIR_ERR_POP(mpi_errno); MPIR_ERR_CHKANDJUMP(errflag, mpi_errno, MPI_ERR_OTHER, "**coll_fail"); mpi_errno = - MPIU_SHMW_Hnd_deserialize((*win_ptr)->shm_mutex_segment_handle, serialized_hnd, + MPL_shm_hnd_deserialize((*win_ptr)->shm_mutex_segment_handle, serialized_hnd, strlen(serialized_hnd)); if (mpi_errno) MPIR_ERR_POP(mpi_errno); /* attach to shared memory region created by rank 0 */ mpi_errno = - MPIU_SHMW_Seg_attach((*win_ptr)->shm_mutex_segment_handle, + MPL_shm_seg_attach((*win_ptr)->shm_mutex_segment_handle, sizeof(MPIDI_CH3I_SHM_MUTEX), (char **) &(*win_ptr)->shm_mutex, 0); if (mpi_errno) diff --git a/src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c b/src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c index c335461..5f445bb 100644 --- a/src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c +++ b/src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c @@ -206,33 +206,33 @@ MPID_nem_init(int pg_rank, MPIDI_PG_t *pg_p, int has_parent ATTRIBUTE((unused))) communication is allocated it will probably be mapped at a different location for each process */ - MPIU_SHMW_Hnd_t handle; + MPL_shm_hnd_t handle; int size = (local_rank * 65536) + 65536; char *base_addr; - mpi_errno = MPIU_SHMW_Hnd_init(&handle); + mpi_errno = MPL_shm_hnd_init(&handle); if(mpi_errno != MPI_SUCCESS) { MPIR_ERR_POP(mpi_errno); } - mpi_errno = MPIU_SHMW_Seg_create_and_attach(handle, size, &base_addr, 0); + mpi_errno = MPL_shm_seg_create_and_attach(handle, size, &base_addr, 0); /* --BEGIN ERROR HANDLING-- */ if (mpi_errno) { - MPIU_SHMW_Seg_remove(handle); - MPIU_SHMW_Hnd_finalize(&handle); + MPL_shm_seg_remove(handle); + MPL_shm_hnd_finalize(&handle); MPIR_ERR_POP (mpi_errno); } /* --END ERROR HANDLING-- */ - mpi_errno = MPIU_SHMW_Seg_remove(handle); + mpi_errno = MPL_shm_seg_remove(handle); /* --BEGIN ERROR HANDLING-- */ if (mpi_errno) { - MPIU_SHMW_Hnd_finalize(&handle); + MPL_shm_hnd_finalize(&handle); MPIR_ERR_POP (mpi_errno); } /* --END ERROR HANDLING-- */ - MPIU_SHMW_Hnd_finalize(&handle); + MPL_shm_hnd_finalize(&handle); } /*fprintf(stderr,"[%i] -- address shift ok \n",pg_rank); */ #endif /*FORCE_ASYM */ @@ -540,9 +540,9 @@ MPID_nem_vc_init (MPIDI_VC_t *vc) #endif vc_ch->lmt_copy_buf = NULL; - mpi_errno = MPIU_SHMW_Hnd_init(&(vc_ch->lmt_copy_buf_handle)); + mpi_errno = MPL_shm_hnd_init(&(vc_ch->lmt_copy_buf_handle)); if(mpi_errno != MPI_SUCCESS) { MPIR_ERR_POP(mpi_errno); } - mpi_errno = MPIU_SHMW_Hnd_init(&(vc_ch->lmt_recv_copy_buf_handle)); + mpi_errno = MPL_shm_hnd_init(&(vc_ch->lmt_recv_copy_buf_handle)); if(mpi_errno != MPI_SUCCESS) { MPIR_ERR_POP(mpi_errno); } vc_ch->lmt_queue.head = NULL; vc_ch->lmt_queue.tail = NULL; diff --git a/src/mpid/ch3/channels/nemesis/src/mpid_nem_lmt_shm.c b/src/mpid/ch3/channels/nemesis/src/mpid_nem_lmt_shm.c index acba48f..38b8eb2 100644 --- a/src/mpid/ch3/channels/nemesis/src/mpid_nem_lmt_shm.c +++ b/src/mpid/ch3/channels/nemesis/src/mpid_nem_lmt_shm.c @@ -8,7 +8,6 @@ #include "mpid_nem_inline.h" #include "mpid_nem_datatypes.h" -#include "mpiu_os_wrappers.h" #if defined(MPL_USE_DBG_LOGGING) && 0 #define DBG_LMT(x) x #else @@ -108,10 +107,10 @@ typedef struct MPID_nem_copy_buf static inline int lmt_shm_progress_vc(MPIDI_VC_t *vc, int *done); static int lmt_shm_send_progress(MPIDI_VC_t *vc, MPIR_Request *req, int *done); static int lmt_shm_recv_progress(MPIDI_VC_t *vc, MPIR_Request *req, int *done); -static int MPID_nem_allocate_shm_region(MPID_nem_copy_buf_t **buf_p, MPIU_SHMW_Hnd_t handle); -static int MPID_nem_attach_shm_region(MPID_nem_copy_buf_t **buf_p, MPIU_SHMW_Hnd_t handle); -static int MPID_nem_detach_shm_region(MPID_nem_copy_buf_t **buf, MPIU_SHMW_Hnd_t handle); -static int MPID_nem_delete_shm_region(MPID_nem_copy_buf_t **buf, MPIU_SHMW_Hnd_t *handle_p); +static int MPID_nem_allocate_shm_region(MPID_nem_copy_buf_t **buf_p, MPL_shm_hnd_t handle); +static int MPID_nem_attach_shm_region(MPID_nem_copy_buf_t **buf_p, MPL_shm_hnd_t handle); +static int MPID_nem_detach_shm_region(MPID_nem_copy_buf_t **buf, MPL_shm_hnd_t handle); +static int MPID_nem_delete_shm_region(MPID_nem_copy_buf_t **buf, MPL_shm_hnd_t *handle_p); /* number of iterations to wait for the other side to process a buffer */ #define LMT_POLLS_BEFORE_GIVING_UP 5000 @@ -192,7 +191,7 @@ int MPID_nem_lmt_shm_start_recv(MPIDI_VC_t *vc, MPIR_Request *req, MPL_IOV s_coo } /* send CTS with handle for copy buffer */ - mpi_errno = MPIU_SHMW_Hnd_get_serialized_by_ref((vc_ch->lmt_copy_buf_handle), &ser_lmt_copy_buf_handle); + mpi_errno = MPL_shm_hnd_get_serialized_by_ref((vc_ch->lmt_copy_buf_handle), &ser_lmt_copy_buf_handle); if(mpi_errno != MPI_SUCCESS) { MPIR_ERR_POP(mpi_errno); } MPID_nem_lmt_send_CTS(vc, req, ser_lmt_copy_buf_handle, (int)strlen(ser_lmt_copy_buf_handle) + 1); @@ -254,7 +253,7 @@ int MPID_nem_lmt_shm_start_send(MPIDI_VC_t *vc, MPIR_Request *req, MPL_IOV r_coo MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPID_NEM_LMT_SHM_START_SEND); if (vc_ch->lmt_copy_buf == NULL){ - mpi_errno = MPIU_SHMW_Hnd_deserialize(vc_ch->lmt_copy_buf_handle, r_cookie.MPL_IOV_BUF, strlen(r_cookie.MPL_IOV_BUF)); + mpi_errno = MPL_shm_hnd_deserialize(vc_ch->lmt_copy_buf_handle, r_cookie.MPL_IOV_BUF, strlen(r_cookie.MPL_IOV_BUF)); if(mpi_errno != MPI_SUCCESS) { MPIR_ERR_POP(mpi_errno); } mpi_errno = MPID_nem_attach_shm_region(&vc_ch->lmt_copy_buf, vc_ch->lmt_copy_buf_handle); @@ -263,7 +262,7 @@ int MPID_nem_lmt_shm_start_send(MPIDI_VC_t *vc, MPIR_Request *req, MPL_IOV r_coo } else{ char *ser_lmt_copy_buf_handle=NULL; - mpi_errno = MPIU_SHMW_Hnd_get_serialized_by_ref(vc_ch->lmt_copy_buf_handle, &ser_lmt_copy_buf_handle); + mpi_errno = MPL_shm_hnd_get_serialized_by_ref(vc_ch->lmt_copy_buf_handle, &ser_lmt_copy_buf_handle); if(mpi_errno != MPI_SUCCESS) { MPIR_ERR_POP(mpi_errno); } if (strncmp(ser_lmt_copy_buf_handle, r_cookie.MPL_IOV_BUF, r_cookie.MPL_IOV_LEN) < 0){ /* Each side allocated its own buffer, lexicographically lower valued buffer handle is deleted */ @@ -273,10 +272,10 @@ int MPID_nem_lmt_shm_start_send(MPIDI_VC_t *vc, MPIR_Request *req, MPL_IOV r_coo vc_ch->lmt_copy_buf = NULL; /* The shared memory handle is not valid any more -- so get a new shm handle */ - mpi_errno = MPIU_SHMW_Hnd_init(&vc_ch->lmt_copy_buf_handle); + mpi_errno = MPL_shm_hnd_init(&vc_ch->lmt_copy_buf_handle); if(mpi_errno != MPI_SUCCESS) { MPIR_ERR_POP(mpi_errno); } - mpi_errno = MPIU_SHMW_Hnd_deserialize(vc_ch->lmt_copy_buf_handle, r_cookie.MPL_IOV_BUF, strlen(r_cookie.MPL_IOV_BUF)); + mpi_errno = MPL_shm_hnd_deserialize(vc_ch->lmt_copy_buf_handle, r_cookie.MPL_IOV_BUF, strlen(r_cookie.MPL_IOV_BUF)); if(mpi_errno != MPI_SUCCESS) { MPIR_ERR_POP(mpi_errno); } mpi_errno = MPID_nem_attach_shm_region(&vc_ch->lmt_copy_buf, vc_ch->lmt_copy_buf_handle); @@ -869,7 +868,7 @@ int MPID_nem_lmt_shm_vc_terminated(MPIDI_VC_t *vc) #define FUNCNAME MPID_nem_allocate_shm_region #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static int MPID_nem_allocate_shm_region(MPID_nem_copy_buf_t **buf_p, MPIU_SHMW_Hnd_t handle) +static int MPID_nem_allocate_shm_region(MPID_nem_copy_buf_t **buf_p, MPL_shm_hnd_t handle) { int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPID_NEM_ALLOCATE_SHM_REGION); @@ -882,7 +881,7 @@ static int MPID_nem_allocate_shm_region(MPID_nem_copy_buf_t **buf_p, MPIU_SHMW_H goto fn_exit; } - mpi_errno = MPIU_SHMW_Seg_create_and_attach(handle, sizeof(MPID_nem_copy_buf_t), (char **)buf_p, 0); + mpi_errno = MPL_shm_seg_create_and_attach(handle, sizeof(MPID_nem_copy_buf_t), (char **)buf_p, 0); if (mpi_errno) MPIR_ERR_POP(mpi_errno); fn_exit: @@ -896,7 +895,7 @@ static int MPID_nem_allocate_shm_region(MPID_nem_copy_buf_t **buf_p, MPIU_SHMW_H #define FUNCNAME MPID_nem_attach_shm_region #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static int MPID_nem_attach_shm_region(MPID_nem_copy_buf_t **buf_p, MPIU_SHMW_Hnd_t handle) +static int MPID_nem_attach_shm_region(MPID_nem_copy_buf_t **buf_p, MPL_shm_hnd_t handle) { int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPID_NEM_ATTACH_SHM_REGION); @@ -909,10 +908,10 @@ static int MPID_nem_attach_shm_region(MPID_nem_copy_buf_t **buf_p, MPIU_SHMW_Hnd goto fn_exit; } - mpi_errno = MPIU_SHMW_Seg_attach(handle, sizeof(MPID_nem_copy_buf_t), (char **)buf_p, 0); + mpi_errno = MPL_shm_seg_attach(handle, sizeof(MPID_nem_copy_buf_t), (char **)buf_p, 0); if (mpi_errno) MPIR_ERR_POP(mpi_errno); - mpi_errno = MPIU_SHMW_Seg_remove(handle); + mpi_errno = MPL_shm_seg_remove(handle); if (mpi_errno) MPIR_ERR_POP(mpi_errno); fn_exit: @@ -926,14 +925,14 @@ static int MPID_nem_attach_shm_region(MPID_nem_copy_buf_t **buf_p, MPIU_SHMW_Hnd #define FUNCNAME MPID_nem_detach_shm_region #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static int MPID_nem_detach_shm_region(MPID_nem_copy_buf_t **buf_p, MPIU_SHMW_Hnd_t handle) +static int MPID_nem_detach_shm_region(MPID_nem_copy_buf_t **buf_p, MPL_shm_hnd_t handle) { int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPID_NEM_DETACH_SHM_REGION); MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPID_NEM_DETACH_SHM_REGION); - mpi_errno = MPIU_SHMW_Seg_detach(handle, (char **)buf_p, sizeof(MPID_nem_copy_buf_t)); + mpi_errno = MPL_shm_seg_detach(handle, (char **)buf_p, sizeof(MPID_nem_copy_buf_t)); if (mpi_errno) MPIR_ERR_POP(mpi_errno); fn_exit: @@ -947,20 +946,20 @@ static int MPID_nem_detach_shm_region(MPID_nem_copy_buf_t **buf_p, MPIU_SHMW_Hnd #define FUNCNAME MPID_nem_delete_shm_region #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static int MPID_nem_delete_shm_region(MPID_nem_copy_buf_t **buf_p, MPIU_SHMW_Hnd_t *handle_p) +static int MPID_nem_delete_shm_region(MPID_nem_copy_buf_t **buf_p, MPL_shm_hnd_t *handle_p) { int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPID_NEM_DELETE_SHM_REGION); MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPID_NEM_DELETE_SHM_REGION); - mpi_errno = MPIU_SHMW_Seg_remove(*handle_p); + mpi_errno = MPL_shm_seg_remove(*handle_p); if (mpi_errno != MPI_SUCCESS) { MPIR_ERR_POP(mpi_errno); } mpi_errno = MPID_nem_detach_shm_region(buf_p, *handle_p); if (mpi_errno != MPI_SUCCESS) { MPIR_ERR_POP(mpi_errno); } - mpi_errno = MPIU_SHMW_Hnd_finalize(handle_p); + mpi_errno = MPL_shm_hnd_finalize(handle_p); if(mpi_errno != MPI_SUCCESS) { MPIR_ERR_POP(mpi_errno); } fn_exit: diff --git a/src/mpid/ch3/channels/nemesis/subconfigure.m4 b/src/mpid/ch3/channels/nemesis/subconfigure.m4 index 3a7c13b..2ab9cda 100644 --- a/src/mpid/ch3/channels/nemesis/subconfigure.m4 +++ b/src/mpid/ch3/channels/nemesis/subconfigure.m4 @@ -184,10 +184,10 @@ AC_CHECK_FUNCS(rand) AC_CHECK_FUNCS(srand) # Check for available shared memory functions -PAC_ARG_SHARED_MEMORY -if test "$with_shared_memory" != "mmap" -a "$with_shared_memory" != "sysv"; then - AC_MSG_ERROR([cannot support shared memory: need either sysv shared memory functions or mmap in order to support shared memory]) -fi +#PAC_ARG_SHARED_MEMORY +#if test "$with_shared_memory" != "mmap" -a "$with_shared_memory" != "sysv"; then +# AC_MSG_ERROR([cannot support shared memory: need either sysv shared memory functions or mmap in order to support shared memory]) +#fi AC_ARG_ENABLE(nemesis-shm-collectives, [--enable-nemesis-shm-collectives - enables use of shared memory for collective comunication within a node], AC_DEFINE(ENABLED_SHM_COLLECTIVES, 1, [Define to enable shared-memory collectives])) diff --git a/src/mpid/common/shm/mpidu_shm.h b/src/mpid/common/shm/mpidu_shm.h index 8558a5f..2060cd6 100644 --- a/src/mpid/common/shm/mpidu_shm.h +++ b/src/mpid/common/shm/mpidu_shm.h @@ -2,7 +2,7 @@ #ifndef MPIDU_SHM_H #define MPIDU_SHM_H -#include "mpiutil.h" +#include "mpiimpl.h" #define MPIDU_SHM_MAX_FNAME_LEN 256 #define MPIDU_SHM_CACHE_LINE_LEN 64 @@ -15,9 +15,9 @@ typedef struct MPIDU_shm_barrier typedef struct MPIDU_shm_seg { - MPIU_Size_t segment_len; + size_t segment_len; /* Handle to shm seg */ - MPIU_SHMW_Hnd_t hnd; + MPL_shm_hnd_t hnd; /* Pointers */ char *base_addr; /* Misc */ diff --git a/src/mpid/common/shm/mpidu_shm_alloc.c b/src/mpid/common/shm/mpidu_shm_alloc.c index 7db7607..c938fab 100644 --- a/src/mpid/common/shm/mpidu_shm_alloc.c +++ b/src/mpid/common/shm/mpidu_shm_alloc.c @@ -4,7 +4,7 @@ * See COPYRIGHT in top-level directory. */ #include <mpidimpl.h> -#include "mpiu_shm_wrappers.h" +#include "mpl_shm.h" #ifdef USE_PMI2_API #include "pmi2.h" @@ -78,19 +78,19 @@ int MPIDU_shm_seg_alloc(size_t len, void **ptr_p) { int mpi_errno = MPI_SUCCESS; alloc_elem_t *ep; - MPIU_CHKPMEM_DECL(1); - MPIDI_STATE_DECL(MPID_STATE_MPIDU_SHM_SEG_ALLOC); + MPIR_CHKPMEM_DECL(1); + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDU_SHM_SEG_ALLOC); - MPIDI_FUNC_ENTER(MPID_STATE_MPIDU_SHM_SEG_ALLOC); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDU_SHM_SEG_ALLOC); /* round up to multiple of 8 to ensure the start of the next region is 64-bit aligned. */ len = ROUND_UP_8(len); - MPIU_Assert(len); - MPIU_Assert(ptr_p); + MPIR_Assert(len); + MPIR_Assert(ptr_p); - MPIU_CHKPMEM_MALLOC(ep, alloc_elem_t *, sizeof(alloc_elem_t), mpi_errno, "el"); + MPIR_CHKPMEM_MALLOC(ep, alloc_elem_t *, sizeof(alloc_elem_t), mpi_errno, "el"); ep->ptr_p = ptr_p; ep->len = len; @@ -100,11 +100,11 @@ int MPIDU_shm_seg_alloc(size_t len, void **ptr_p) segment_len += len; fn_exit: - MPIU_CHKPMEM_COMMIT(); - MPIDI_FUNC_EXIT(MPID_STATE_MPIDU_SHM_SEG_ALLOC); + MPIR_CHKPMEM_COMMIT(); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDU_SHM_SEG_ALLOC); return mpi_errno; fn_fail: - MPIU_CHKPMEM_REAP(); + MPIR_CHKPMEM_REAP(); goto fn_exit; } @@ -146,21 +146,21 @@ int MPIDU_shm_seg_commit(MPIDU_shm_seg_ptr_t memory, MPIDU_shm_barrier_ptr_t *ba void *current_addr; void *start_addr ATTRIBUTE((unused)); size_t size_left; - MPIU_CHKPMEM_DECL(1); - MPIU_CHKLMEM_DECL(2); - MPIDI_STATE_DECL(MPID_STATE_MPIDU_SHM_SEG_COMMIT); + MPIR_CHKPMEM_DECL(1); + MPIR_CHKLMEM_DECL(2); + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDU_SHM_SEG_COMMIT); - MPIDI_FUNC_ENTER(MPID_STATE_MPIDU_SHM_SEG_COMMIT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDU_SHM_SEG_COMMIT); /* MPIDU_SHM_Seg_alloc() needs to have been called before this function */ - MPIU_Assert(!ALLOCQ_EMPTY()); - MPIU_Assert(segment_len > 0); + MPIR_Assert(!ALLOCQ_EMPTY()); + MPIR_Assert(segment_len > 0); /* allocate an area to check if the segment was allocated symmetrically */ mpi_errno = MPIDU_shm_seg_alloc(sizeof(asym_check_region), (void **) &asym_check_region_p); if (mpi_errno) MPIR_ERR_POP(mpi_errno); - mpi_errno = MPIU_SHMW_Hnd_init(&(memory->hnd)); + mpi_errno = MPL_shm_hnd_init(&(memory->hnd)); if (mpi_errno != MPI_SUCCESS) MPIR_ERR_POP (mpi_errno); /* Shared memory barrier variables are in the front of the shared @@ -174,7 +174,7 @@ int MPIDU_shm_seg_commit(MPIDU_shm_seg_ptr_t memory, MPIDU_shm_barrier_ptr_t *ba region containing the barrier vars. */ /* add space for local barrier region. Use a whole cacheline. */ - MPIU_Assert(MPIDU_SHM_CACHE_LINE_LEN >= sizeof(MPIDU_shm_barrier_t)); + MPIR_Assert(MPIDU_SHM_CACHE_LINE_LEN >= sizeof(MPIDU_shm_barrier_t)); segment_len += MPIDU_SHM_CACHE_LINE_LEN; #ifdef OPA_USE_LOCK_BASED_PRIMITIVES @@ -187,7 +187,7 @@ int MPIDU_shm_seg_commit(MPIDU_shm_seg_ptr_t memory, MPIDU_shm_barrier_ptr_t *ba /* offset from memory->base_addr to the start of ipc_lock */ ipc_lock_offset = MPIDU_SHM_CACHE_LINE_LEN; - MPIU_Assert(ipc_lock_offset >= sizeof(OPA_emulation_ipl_t)); + MPIR_Assert(ipc_lock_offset >= sizeof(OPA_emulation_ipl_t)); segment_len += MPIDU_SHM_CACHE_LINE_LEN; #endif @@ -199,12 +199,12 @@ int MPIDU_shm_seg_commit(MPIDU_shm_seg_ptr_t memory, MPIDU_shm_barrier_ptr_t *ba { char *addr; - MPIU_CHKPMEM_MALLOC(addr, char *, segment_len + MPIDU_SHM_CACHE_LINE_LEN, mpi_errno, "segment"); + MPIR_CHKPMEM_MALLOC(addr, char *, segment_len + MPIDU_SHM_CACHE_LINE_LEN, mpi_errno, "segment"); memory->base_addr = addr; current_addr = - (char *) (((MPIU_Upint) addr + (MPIU_Upint) MPIDU_SHM_CACHE_LINE_LEN - 1) & - (~((MPIU_Upint) MPIDU_SHM_CACHE_LINE_LEN - 1))); + (char *) (((uintptr_t) addr + (uintptr_t) MPIDU_SHM_CACHE_LINE_LEN - 1) & + (~((uintptr_t) MPIDU_SHM_CACHE_LINE_LEN - 1))); memory->symmetrical = 0; /* must come before barrier_init since we use OPA in that function */ @@ -220,13 +220,13 @@ int MPIDU_shm_seg_commit(MPIDU_shm_seg_ptr_t memory, MPIDU_shm_barrier_ptr_t *ba else { if (local_rank == 0) { - mpi_errno = MPIU_SHMW_Seg_create_and_attach(memory->hnd, memory->segment_len, &(memory->base_addr), 0); + mpi_errno = MPL_shm_seg_create_and_attach(memory->hnd, memory->segment_len, &(memory->base_addr), 0); if (mpi_errno) MPIR_ERR_POP(mpi_errno); /* post name of shared file */ - MPIU_Assert(local_procs_0 == rank); + MPIR_Assert(local_procs_0 == rank); - mpi_errno = MPIU_SHMW_Hnd_get_serialized_by_ref(memory->hnd, &serialized_hnd); + mpi_errno = MPL_shm_hnd_get_serialized_by_ref(memory->hnd, &serialized_hnd); if (mpi_errno) MPIR_ERR_POP(mpi_errno); /* must come before barrier_init since we use OPA in that function */ @@ -250,17 +250,17 @@ int MPIDU_shm_seg_commit(MPIDU_shm_seg_ptr_t memory, MPIDU_shm_barrier_ptr_t *ba int found = FALSE; /* Allocate space for pmi key and val */ - MPIU_CHKLMEM_MALLOC(val, char *, PMI2_MAX_VALLEN, mpi_errno, "val"); + MPIR_CHKLMEM_MALLOC(val, char *, PMI2_MAX_VALLEN, mpi_errno, "val"); /* get name of shared file */ mpi_errno = PMI2_Info_GetNodeAttr("sharedFilename", val, PMI2_MAX_VALLEN, &found, TRUE); if (mpi_errno) MPIR_ERR_POP(mpi_errno); MPIR_ERR_CHKINTERNAL(!found, mpi_errno, "nodeattr not found"); - mpi_errno = MPIU_SHMW_Hnd_deserialize(memory->hnd, val, strlen(val)); + mpi_errno = MPL_shm_hnd_deserialize(memory->hnd, val, strlen(val)); if (mpi_errno) MPIR_ERR_POP(mpi_errno); - mpi_errno = MPIU_SHMW_Seg_attach(memory->hnd, memory->segment_len, (char **)&memory->base_addr, 0); + mpi_errno = MPL_shm_seg_attach(memory->hnd, memory->segment_len, (char **)&memory->base_addr, 0); if (mpi_errno) MPIR_ERR_POP(mpi_errno); /* must come before barrier_init since we use OPA in that function */ @@ -285,7 +285,7 @@ int MPIDU_shm_seg_commit(MPIDU_shm_seg_ptr_t memory, MPIDU_shm_barrier_ptr_t *ba if (mpi_errno) MPIR_ERR_POP(mpi_errno); if (local_rank == 0) { - mpi_errno = MPIU_SHMW_Seg_remove(memory->hnd); + mpi_errno = MPL_shm_seg_remove(memory->hnd); if (mpi_errno) MPIR_ERR_POP(mpi_errno); } current_addr = memory->base_addr; @@ -297,12 +297,12 @@ int MPIDU_shm_seg_commit(MPIDU_shm_seg_ptr_t memory, MPIDU_shm_barrier_ptr_t *ba { char *addr; - MPIU_CHKPMEM_MALLOC(addr, char *, segment_len + MPIDU_SHM_CACHE_LINE_LEN, mpi_errno, "segment"); + MPIR_CHKPMEM_MALLOC(addr, char *, segment_len + MPIDU_SHM_CACHE_LINE_LEN, mpi_errno, "segment"); memory->base_addr = addr; current_addr = - (char *) (((MPIU_Upint) addr + (MPIU_Upint) MPIDU_SHM_CACHE_LINE_LEN - 1) & - (~((MPIU_Upint) MPIDU_SHM_CACHE_LINE_LEN - 1))); + (char *) (((uintptr_t) addr + (uintptr_t) MPIDU_SHM_CACHE_LINE_LEN - 1) & + (~((uintptr_t) MPIDU_SHM_CACHE_LINE_LEN - 1))); memory->symmetrical = 0; /* we still need to call barrier */ @@ -322,24 +322,24 @@ int MPIDU_shm_seg_commit(MPIDU_shm_seg_ptr_t memory, MPIDU_shm_barrier_ptr_t *ba /* Allocate space for pmi key and val */ pmi_errno = PMI_KVS_Get_key_length_max(&key_max_sz); MPIR_ERR_CHKANDJUMP1(pmi_errno, mpi_errno, MPI_ERR_OTHER, "**fail", "**fail %d", pmi_errno); - MPIU_CHKLMEM_MALLOC(key, char *, key_max_sz, mpi_errno, "key"); + MPIR_CHKLMEM_MALLOC(key, char *, key_max_sz, mpi_errno, "key"); pmi_errno = PMI_KVS_Get_value_length_max(&val_max_sz); MPIR_ERR_CHKANDJUMP1(pmi_errno, mpi_errno, MPI_ERR_OTHER, "**fail", "**fail %d", pmi_errno); - MPIU_CHKLMEM_MALLOC(val, char *, val_max_sz, mpi_errno, "val"); + MPIR_CHKLMEM_MALLOC(val, char *, val_max_sz, mpi_errno, "val"); mpi_errno = PMI_KVS_Get_my_name(kvs_name, 256); if (mpi_errno) MPIR_ERR_POP (mpi_errno); if (local_rank == 0){ - mpi_errno = MPIU_SHMW_Seg_create_and_attach(memory->hnd, memory->segment_len, &(memory->base_addr), 0); + mpi_errno = MPL_shm_seg_create_and_attach(memory->hnd, memory->segment_len, &(memory->base_addr), 0); if (mpi_errno != MPI_SUCCESS) MPIR_ERR_POP (mpi_errno); /* post name of shared file */ - MPIU_Assert(local_procs_0 == rank); + MPIR_Assert(local_procs_0 == rank); MPL_snprintf(key, key_max_sz, "sharedFilename[%i]", rank); - mpi_errno = MPIU_SHMW_Hnd_get_serialized_by_ref(memory->hnd, &serialized_hnd); + mpi_errno = MPL_shm_hnd_get_serialized_by_ref(memory->hnd, &serialized_hnd); if (mpi_errno != MPI_SUCCESS) MPIR_ERR_POP (mpi_errno); pmi_errno = PMI_KVS_Put (kvs_name, key, serialized_hnd); @@ -372,10 +372,10 @@ int MPIDU_shm_seg_commit(MPIDU_shm_seg_ptr_t memory, MPIDU_shm_barrier_ptr_t *ba MPIR_ERR_CHKANDJUMP1(pmi_errno != PMI_SUCCESS, mpi_errno, MPI_ERR_OTHER, "**pmi_kvs_get", "**pmi_kvs_get %d", pmi_errno); - mpi_errno = MPIU_SHMW_Hnd_deserialize(memory->hnd, val, strlen(val)); + mpi_errno = MPL_shm_hnd_deserialize(memory->hnd, val, strlen(val)); if(mpi_errno != MPI_SUCCESS) MPIR_ERR_POP(mpi_errno); - mpi_errno = MPIU_SHMW_Seg_attach(memory->hnd, memory->segment_len, (char **)&memory->base_addr, 0); + mpi_errno = MPL_shm_seg_attach(memory->hnd, memory->segment_len, (char **)&memory->base_addr, 0); if (mpi_errno) MPIR_ERR_POP (mpi_errno); /* must come before barrier_init since we use OPA in that function */ @@ -394,7 +394,7 @@ int MPIDU_shm_seg_commit(MPIDU_shm_seg_ptr_t memory, MPIDU_shm_barrier_ptr_t *ba if (local_rank == 0) { - mpi_errno = MPIU_SHMW_Seg_remove(memory->hnd); + mpi_errno = MPL_shm_seg_remove(memory->hnd); if (mpi_errno != MPI_SUCCESS) MPIR_ERR_POP (mpi_errno); } current_addr = memory->base_addr; @@ -408,13 +408,13 @@ int MPIDU_shm_seg_commit(MPIDU_shm_seg_ptr_t memory, MPIDU_shm_barrier_ptr_t *ba /* reserve room for shared mem barrier (We used a whole cacheline) */ current_addr = (char *) current_addr + MPIDU_SHM_CACHE_LINE_LEN; - MPIU_Assert(size_left >= MPIDU_SHM_CACHE_LINE_LEN); + MPIR_Assert(size_left >= MPIDU_SHM_CACHE_LINE_LEN); size_left -= MPIDU_SHM_CACHE_LINE_LEN; #ifdef OPA_USE_LOCK_BASED_PRIMITIVES /* reserve room for the opa emulation lock */ current_addr = (char *) current_addr + MPIDU_SHM_CACHE_LINE_LEN; - MPIU_Assert(size_left >= MPIDU_SHM_CACHE_LINE_LEN); + MPIR_Assert(size_left >= MPIDU_SHM_CACHE_LINE_LEN); size_left -= MPIDU_SHM_CACHE_LINE_LEN; #endif @@ -425,29 +425,29 @@ int MPIDU_shm_seg_commit(MPIDU_shm_seg_ptr_t memory, MPIDU_shm_barrier_ptr_t *ba ALLOCQ_DEQUEUE(&ep); *(ep->ptr_p) = current_addr; - MPIU_Assert(size_left >= ep->len); + MPIR_Assert(size_left >= ep->len); size_left -= ep->len; current_addr = (char *)current_addr + ep->len; MPL_free(ep); - MPIU_Assert((char *)current_addr <= (char *)start_addr + segment_len); + MPIR_Assert((char *)current_addr <= (char *)start_addr + segment_len); } while (!ALLOCQ_EMPTY()); mpi_errno = check_alloc(memory, *barrier, num_local, local_rank); if (mpi_errno) MPIR_ERR_POP(mpi_errno); - MPIU_CHKPMEM_COMMIT(); + MPIR_CHKPMEM_COMMIT(); fn_exit: - MPIU_CHKLMEM_FREEALL(); - MPIDI_FUNC_EXIT(MPID_STATE_MPIDU_SHM_SEG_COMMIT); + MPIR_CHKLMEM_FREEALL(); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDU_SHM_SEG_COMMIT); return mpi_errno; fn_fail: /* --BEGIN ERROR HANDLING-- */ - MPIU_SHMW_Seg_remove(memory->hnd); - MPIU_SHMW_Hnd_finalize(&(memory->hnd)); - MPIU_CHKPMEM_REAP(); + MPL_shm_seg_remove(memory->hnd); + MPL_shm_hnd_finalize(&(memory->hnd)); + MPIR_CHKPMEM_REAP(); goto fn_exit; /* --END ERROR HANDLING-- */ } @@ -460,22 +460,22 @@ int MPIDU_shm_seg_commit(MPIDU_shm_seg_ptr_t memory, MPIDU_shm_barrier_ptr_t *ba int MPIDU_shm_seg_destroy(MPIDU_shm_seg_ptr_t memory, int num_local) { int mpi_errno = MPI_SUCCESS; - MPIDI_STATE_DECL(MPID_STATE_MPIDU_SHM_SEG_DESTROY); + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDU_SHM_SEG_DESTROY); - MPIDI_FUNC_ENTER(MPID_STATE_MPIDU_SHM_SEG_DESTROY); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDU_SHM_SEG_DESTROY); if (num_local == 1) MPL_free(memory->base_addr); else { - mpi_errno = MPIU_SHMW_Seg_detach(memory->hnd, + mpi_errno = MPL_shm_seg_detach(memory->hnd, &(memory->base_addr), memory->segment_len); if (mpi_errno) MPIR_ERR_POP (mpi_errno); } fn_exit: - MPIU_SHMW_Hnd_finalize(&(memory->hnd)); - MPIDI_FUNC_EXIT(MPID_STATE_MPIDU_SHM_SEG_DESTROY); + MPL_shm_hnd_finalize(&(memory->hnd)); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDU_SHM_SEG_DESTROY); return mpi_errno; fn_fail: goto fn_exit; @@ -492,9 +492,9 @@ static int check_alloc(MPIDU_shm_seg_ptr_t memory, MPIDU_shm_barrier_ptr_t barri int num_local, int local_rank) { int mpi_errno = MPI_SUCCESS; - MPIDI_STATE_DECL(MPID_STATE_MPIDU_SHM_CHECK_ALLOC); + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDU_SHM_CHECK_ALLOC); - MPIDI_FUNC_ENTER(MPID_STATE_MPIDU_SHM_CHECK_ALLOC); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDU_SHM_CHECK_ALLOC); if (local_rank == 0) { asym_check_region_p->base_ptr = memory->base_addr; @@ -519,7 +519,7 @@ static int check_alloc(MPIDU_shm_seg_ptr_t memory, MPIDU_shm_barrier_ptr_t barri } fn_exit: - MPIDI_FUNC_EXIT(MPID_STATE_MPIDU_SHM_CHECK_ALLOC); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDU_SHM_CHECK_ALLOC); return mpi_errno; fn_fail: goto fn_exit; diff --git a/src/mpid/common/shm/mpidu_shm_impl.h b/src/mpid/common/shm/mpidu_shm_impl.h index b822101..2b6877a 100644 --- a/src/mpid/common/shm/mpidu_shm_impl.h +++ b/src/mpid/common/shm/mpidu_shm_impl.h @@ -7,7 +7,6 @@ #define MPIDU_SHM_IMPL_H #include "mpidimpl.h" -#include "mpiu_os_wrappers.h" #include "mpidu_shm.h" diff --git a/src/mpl/Makefile.am b/src/mpl/Makefile.am index 1985d44..eb73cca 100644 --- a/src/mpl/Makefile.am +++ b/src/mpl/Makefile.am @@ -30,7 +30,8 @@ mpl_headers = \ include/mpl_msg.h \ include/mpl_sock.h \ include/mpl_iov.h \ - include/mpl_bt.h + include/mpl_bt.h \ + include/mpl_shm.h if MPL_EMBEDDED_MODE noinst_HEADERS = $(mpl_headers) diff --git a/src/mpl/configure.ac b/src/mpl/configure.ac index 42d8d70..e07f2db 100644 --- a/src/mpl/configure.ac +++ b/src/mpl/configure.ac @@ -738,6 +738,31 @@ AC_CHECK_FUNCS(getpid) ## END OF DBG CODE ####################################################################### +####################################################################### +## START OF SHM CODE +####################################################################### + +dnl Check for shm +PAC_ARG_SHARED_MEMORY + +case $with_shared_memory in + sysv) + AC_DEFINE(MPL_USE_SYSV_SHM,1,[Define if use SYSV shared memory]) + ;; + mmap) + AC_DEFINE(MPL_USE_MMAP_SHM,1,[Define if use MMAP shared memory]) + ;; + nt) + AC_DEFINE(MPL_USE_NT_SHM,1,[Define if use Windows shared memory]) + ;; + *) + AC_MSG_ERROR([cannot support shared memory: need either sysv shared memory functions or mmap in order to support shared memory]) +esac + + +####################################################################### +## END OF SHM CODE +####################################################################### ## Enable creation of libtool-style versioning or no versioning AC_ARG_ENABLE(versioning, diff --git a/src/mpl/include/mpl.h b/src/mpl/include/mpl.h index d53e033..d9d6dd1 100644 --- a/src/mpl/include/mpl.h +++ b/src/mpl/include/mpl.h @@ -22,5 +22,6 @@ #include "mpl_timer.h" #include "mpl_yield.h" #include "mpl_dbg.h" +#include "mpl_shm.h" #endif /* !defined(MPL_H_INCLUDED) */ diff --git a/src/mpl/include/mpl_shm.h b/src/mpl/include/mpl_shm.h new file mode 100644 index 0000000..dd40ad6 --- /dev/null +++ b/src/mpl/include/mpl_shm.h @@ -0,0 +1,160 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* vim: set ft=c.mpich : */ +/* + * (C) 2010 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + */ + +/* This file contains "pre" definitions and declarations for the OS wrappers. + * That is, things that shouldn't depend on much more than the mpichconf.h + * values. */ + +#ifndef MPL_SHM_H_INCLUDED +#define MPL_SHM_H_INCLUDED + +#include "mplconfig.h" + +#ifdef MPL_USE_SYSV_SHM +#include "mpl_shm_sysv.h" +#elif defined MPL_USE_MMAP_SHM +#include "mpl_shm_mmap.h" +#elif defined MPL_USE_NT_SHM +#include "mpl_shm_win.h" +#endif + +#define MPLI_SHM_FLAG_CLR 0x0 +#define MPLI_SHM_FLAG_SHM_CREATE 0x1 +#define MPLI_SHM_FLAG_SHM_ATTACH 0x10 +#define MPLI_SHM_FLAG_GHND_STATIC 0x100 + +#define MPL_SHM_HND_INVALID NULL +#define MPLI_SHM_GHND_INVALID NULL +#define MPLI_SHM_GHND_INIT_VAL '\0' +#define MPL_SHM_HND_SZ (sizeof(MPLI_shm_lghnd_t)) +#define MPL_SHM_GHND_SZ MPLI_SHM_GHND_SZ + +/* A Handle is valid if it is initialized/init and has a value + * different from the default/invalid value assigned during init + */ +#define MPLI_shm_hnd_is_valid(hnd) ( \ + ((hnd) && \ + MPLI_shm_lhnd_is_valid(hnd) && \ + MPLI_shm_ghnd_is_valid(hnd)) \ +) + +/* With MMAP_SHM, NT_SHM & SYSV_SHM local handle is always init'ed */ +#define MPLI_shm_hnd_is_init(hnd) ( \ + ((hnd) && /* MPL_shm_lhnd_is_init(hnd) && */ \ + MPLI_shm_ghnd_is_init(hnd)) \ +) + +/* These macros are the setters/getters for the shm handle */ +#define MPLI_shm_lhnd_get(hnd) ((hnd)->lhnd) +#define MPLI_shm_lhnd_set(hnd, val) ((hnd)->lhnd=val) +#define MPLI_shm_lhnd_is_valid(hnd) (((hnd)->lhnd != MPLI_SHM_LHND_INVALID)) +#define MPLI_shm_lhnd_is_init(hnd) 1 + +/* Allocate mem for references within the handle */ +/* Returns 0 on success, -1 on error */ +#define MPL_shm_hnd_ref_alloc(hnd)( \ + ((hnd)->ghnd = (MPLI_shm_ghnd_t) \ + MPL_malloc(MPLI_SHM_GHND_SZ)) ? 0 : -1 \ +) + + +/* These macros are the setters/getters for the shm handle */ +#define MPLI_shm_ghnd_get_by_ref(hnd) ((hnd)->ghnd) + +/* Returns -1 on error, 0 on success */ +#define MPLI_shm_ghnd_get_by_val(hnd, str, strlen) ( \ + (MPL_snprintf(str, strlen, "%s", \ + MPLI_shm_ghnd_get_by_ref(hnd))) ? 0 : -1 \ +) +#define MPLI_shm_ghnd_set_by_ref(hnd, val) ((hnd)->ghnd = val) +/* Returns -1 on error, 0 on success */ +/* FIXME: What if val is a non-null terminated string ? */ +#define MPLI_shm_ghnd_set_by_val(hnd, fmt, val) ( \ + (MPL_snprintf(MPLI_shm_ghnd_get_by_ref(hnd), \ + MPLI_SHM_GHND_SZ, fmt, val)) ? 0 : -1 \ +) + +#define MPLI_shm_ghnd_is_valid(hnd) ( \ + (((hnd)->ghnd == MPLI_SHM_GHND_INVALID) || \ + (strlen((hnd)->ghnd) == 0)) ? 0 : 1 \ +) +#define MPLI_shm_ghnd_is_init(hnd) ( \ + ((hnd)->flag & MPLI_SHM_FLAG_GHND_STATIC) ? \ + 1 : \ + (((hnd)->ghnd != MPLI_SHM_GHND_INVALID) ? 1 : 0) \ +) + +/* Allocate mem for global handle. + * Returns 0 on success, -1 on failure + */ +static inline int MPLI_shm_ghnd_alloc(MPL_shm_hnd_t hnd) +{ + if(!(hnd->ghnd)){ + hnd->ghnd = (MPLI_shm_ghnd_t)MPL_malloc(MPLI_SHM_GHND_SZ); + if(!(hnd->ghnd)){ return -1; } + } + /* Global handle is no longer static */ + hnd->flag &= ~MPLI_SHM_FLAG_GHND_STATIC; + return 0; +} + + +/* Allocate mem for handle. Lazy allocation for global handle */ +/* Returns 0 on success, -1 on error */ +static inline int MPLI_shm_hnd_alloc(MPL_shm_hnd_t *hnd_ptr) +{ + *hnd_ptr = (MPL_shm_hnd_t) MPL_malloc(MPL_SHM_HND_SZ); + if(*hnd_ptr){ + (*hnd_ptr)->flag = MPLI_SHM_FLAG_GHND_STATIC; + } + else{ + return -1; + } + return 0; +} + +/* Close Handle */ +#define MPLI_shm_hnd_close(hnd) MPLI_shm_lhnd_close(hnd) + +static inline void MPLI_shm_hnd_reset_val(MPL_shm_hnd_t hnd) +{ + MPLI_shm_lhnd_set(hnd, MPLI_SHM_LHND_INIT_VAL); + if(hnd->flag & MPLI_SHM_FLAG_GHND_STATIC){ + hnd->ghnd = MPLI_SHM_GHND_INVALID; + } + else{ + (hnd->ghnd)[0] = MPLI_SHM_GHND_INIT_VAL; + } +} + +static inline void MPLI_shm_hnd_free(MPL_shm_hnd_t hnd) +{ + if(MPLI_shm_hnd_is_init(hnd)){ + if(!(hnd->flag & MPLI_SHM_FLAG_GHND_STATIC)){ + MPL_free(hnd->ghnd); + } + MPL_free(hnd); + } +} + +/* interfaces */ +int MPL_shm_hnd_serialize(char *str, MPL_shm_hnd_t hnd, int str_len); +int MPL_shm_hnd_deserialize(MPL_shm_hnd_t hnd, const char *str_hnd, size_t str_hnd_len); +int MPL_shm_hnd_get_serialized_by_ref(MPL_shm_hnd_t hnd, char **str_ptr); +int MPL_shm_hnd_deserialize_by_ref(MPL_shm_hnd_t hnd, char **ser_hnd_ptr); +int MPL_shm_hnd_init(MPL_shm_hnd_t *hnd_ptr); +int MPL_shm_hnd_finalize(MPL_shm_hnd_t *hnd_ptr); +int MPL_shm_seg_create(MPL_shm_hnd_t hnd, intptr_t seg_sz); +int MPL_shm_seg_open(MPL_shm_hnd_t hnd, intptr_t seg_sz); +int MPL_shm_seg_create_and_attach(MPL_shm_hnd_t hnd, intptr_t seg_sz, + char **shm_addr_ptr, int offset); +int MPL_shm_seg_attach(MPL_shm_hnd_t hnd, intptr_t seg_sz, char **shm_addr_ptr, + int offset); +int MPL_shm_seg_detach(MPL_shm_hnd_t hnd, char **shm_addr_ptr, intptr_t seg_sz); +int MPL_shm_seg_remove(MPL_shm_hnd_t hnd); + +#endif /* MPL_SHM_H_INCLUDED */ diff --git a/src/mpl/include/mpl_shm_mmap.h b/src/mpl/include/mpl_shm_mmap.h new file mode 100644 index 0000000..a2a91c9 --- /dev/null +++ b/src/mpl/include/mpl_shm_mmap.h @@ -0,0 +1,36 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* vim: set ft=c.mpich : */ +/* + * (C) 2016 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + */ +#ifndef MPL_SHM_MMAP_H_INCLUDED +#define MPL_SHM_MMAP_H_INCLUDED + +typedef intptr_t MPLI_shm_lhnd_t; + +typedef char * MPLI_shm_ghnd_t; +/* The local handle, lhnd, is valid only for the current process, + * The global handle, ghnd, is valid across multiple processes + * The handle flag, flag, is used to set various attributes of the + * handle. + */ +typedef struct MPLI_shm_lghnd_t { + MPLI_shm_lhnd_t lhnd; + MPLI_shm_ghnd_t ghnd; + int flag; +} MPLI_shm_lghnd_t; + +typedef MPLI_shm_lghnd_t *MPL_shm_hnd_t; + +#define MPL_SHM_FNAME_LEN 50 +#define MPLI_SHM_GHND_SZ MPL_SHM_FNAME_LEN +#define MPLI_SHM_LHND_INVALID (-1) +#define MPLI_SHM_LHND_INIT_VAL (-1) + +#define MPLI_SHM_SEG_ALREADY_EXISTS EEXIST + +/* Returns 0 on success, -1 on error */ +int MPLI_shm_lhnd_close(MPL_shm_hnd_t hnd); + +#endif /* ifndef MPL_SHM_MMAP_H_INCLUDED */ diff --git a/src/mpl/include/mpl_shm_sysv.h b/src/mpl/include/mpl_shm_sysv.h new file mode 100644 index 0000000..babae48 --- /dev/null +++ b/src/mpl/include/mpl_shm_sysv.h @@ -0,0 +1,38 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* vim: set ft=c.mpich : */ +/* + * (C) 2016 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + */ +#ifndef MPL_SHM_SYSV_H_INCLUDED +#define MPL_SHM_SYSV_H_INCLUDED + +typedef int MPLI_shm_lhnd_t; + +typedef char * MPLI_shm_ghnd_t; +/* The local handle, lhnd, is valid only for the current process, + * The global handle, ghnd, is valid across multiple processes + * The handle flag, flag, is used to set various attributes of the + * handle. + */ +typedef struct MPLI_shm_lghnd_t { + MPLI_shm_lhnd_t lhnd; + MPLI_shm_ghnd_t ghnd; + int flag; +} MPLI_shm_lghnd_t; + +typedef MPLI_shm_lghnd_t *MPL_shm_hnd_t; + +#define MPL_SHM_MAX_INT_STR_LEN 20 +#define MPL_SHM_SEG_KEY_LEN MPL_SHM_MAX_INT_STR_LEN +#define MPLI_SHM_GHND_SZ MPL_SHM_SEG_KEY_LEN +#define MPLI_SHM_LHND_INVALID (-1) +#define MPLI_SHM_LHND_INIT_VAL (-1) +#define MPL_SHM_SER_HND_SZ MPLI_SHM_GHND_SZ + +#define MPL_SHM_SEG_ALREADY_EXISTS EEXIST + +/* Nothing to be done at close */ +#define MPLI_shm_lhnd_close(hnd) 0 + +#endif /* ifndef MPL_SHM_SYSV_H_INCLUDED */ diff --git a/src/mpl/include/mpl_shm_win.h b/src/mpl/include/mpl_shm_win.h new file mode 100644 index 0000000..02bfaf6 --- /dev/null +++ b/src/mpl/include/mpl_shm_win.h @@ -0,0 +1,70 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* vim: set ft=c.mpich : */ +/* + * (C) 2016 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + */ +#ifndef MPL_SHM_WIN_INCLUDED +#define MPL_SHM_WIN_INCLUDED + +typedef HANDLE MPL_shm_lhnd_t; + +typedef char * MPLI_shm_ghnd_t; +/* The local handle, lhnd, is valid only for the current process, + * The global handle, ghnd, is valid across multiple processes + * The handle flag, flag, is used to set various attributes of the + * handle. + */ +typedef struct MPLI_shm_lghnd_t { + MPLI_shm_lhnd_t lhnd; + MPLI_shm_ghnd_t ghnd; + int flag; +} MPLI_shm_lghnd_t; + +typedef MPLI_shm_lghnd_t *MPL_shm_hnd_t; + +#define MPL_SHM_SEG_NAME_LEN 70 +#define MPLI_SHM_GHND_SZ MPL_SHM_SEG_NAME_LEN +#define MPLI_SHM_LHND_INVALID INVALID_HANDLE_VALUE +#define MPLI_SHM_LHND_INIT_VAL INVALID_HANDLE_VALUE + +#define MPL_shm_SEG_ALREADY_EXISTS ERROR_ALREADY_EXISTS + +/* Returns 0 on success, -1 on error */ +#define MPLI_shm_lhnd_close(hnd)( \ + (CloseHandle(MPLI_shm_lhnd_get(hnd)) != 0) ? 0 : -1 \ +) + +#if defined (HAVE_QUERYPERFORMANCECOUNTER) +/* + * Returns size of uniqStr, 0 on error + */ +static inline int MPL_shm_get_uniq_str(char *str, int strlen) +{ + LARGE_INTEGER perfCnt; + QueryPerformanceCounter(&perfCnt); + return(MPL_snprintf(str, strlen, "MPICH_NEM_%d_%I64d", + GetCurrentThreadId(), (perfCnt.QuadPart))); +} +#endif + +/* Returns 0 on success, -1 on error */ +static inline int MPLI_shm_ghnd_set_uniq(MPL_shm_hnd_t hnd) +{ + if (MPL_shm_hnd_ref_alloc(hnd) == 0) { + if (MPLI_shm_get_uniq_str(hnd->ghnd, MPLI_SHM_GHND_SZ) != 0) { + return -1; + } + } else { + return -1; + } + return 0; +} + +/* Nothing to be done when removing an SHM segment */ +static inline int MPL_shm_seg_remove(MPL_shm_hnd_t hnd) +{ + return MPI_SUCCESS; +} + +#endif /* ifndef MPL_SHM_WIN_INCLUDED */ diff --git a/src/mpl/src/Makefile.mk b/src/mpl/src/Makefile.mk index 4ae620e..7710d24 100644 --- a/src/mpl/src/Makefile.mk +++ b/src/mpl/src/Makefile.mk @@ -13,3 +13,4 @@ include src/sock/Makefile.mk include src/str/Makefile.mk include src/thread/Makefile.mk include src/timer/Makefile.mk +include src/shm/Makefile.mk diff --git a/src/mpl/src/shm/Makefile.mk b/src/mpl/src/shm/Makefile.mk new file mode 100644 index 0000000..5110c35 --- /dev/null +++ b/src/mpl/src/shm/Makefile.mk @@ -0,0 +1,12 @@ +## -*- Mode: Makefile; -*- +## vim: set ft=automake : +## +## (C) 2011 by Argonne National Laboratory. +## See COPYRIGHT in top-level directory. +## + +lib@MPLLIBNAME@_la_SOURCES += \ + src/shm/mpl_shm.c \ + src/shm/mpl_shm_sysv.c \ + src/shm/mpl_shm_mmap.c \ + src/shm/mpl_shm_win.c diff --git a/src/mpl/src/shm/mpl_shm.c b/src/mpl/src/shm/mpl_shm.c new file mode 100644 index 0000000..0b18921 --- /dev/null +++ b/src/mpl/src/shm/mpl_shm.c @@ -0,0 +1,112 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* vim: set ft=c.mpich : */ +/* + * (C) 2006 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + */ + +#include "mpl.h" + +/* Serialize a handle. A serialized handle is a string of + * characters that can be persisted by the caller. The serialized + * handle can be used to create another ref to the shared mem seg + * by deserializing it. + * str : A string of chars of len, str_len. + * If the function succeeds the serialized handle is copied + * into this user buffer + * hnd : Handle to shared memory + */ +int MPL_shm_hnd_serialize(char *str, MPL_shm_hnd_t hnd, int str_len) +{ + int rc = -1; + rc = MPLI_shm_ghnd_get_by_val(hnd, str, str_len); + return rc; +} + +/* Deserialize a handle. + * str_hnd : A null-terminated string of len str_hnd_len that + * contains the serialized handle. + * hnd : If the call succeeds the user gets back a handle,hnd, to + * shared mem - deserialized from strhnd. This handle + * will refer to the shm seg referred by the serialized + * handle. + */ +int MPL_shm_hnd_deserialize(MPL_shm_hnd_t hnd, const char *str_hnd, size_t str_hnd_len) +{ + int rc = -1; + MPLI_shm_hnd_reset_val(hnd); + rc = MPLI_shm_ghnd_alloc(hnd); + rc = MPLI_shm_ghnd_set_by_val(hnd, "%s", str_hnd); + rc = MPL_shm_seg_open(hnd, 0); + return rc; +} + +/* Get a serialized handle by reference. + * Rationale: The user might only want to read the serialized view + * of the handle & hence not want to allocate a buffer for the ser view + * of the handle. + * str_ptr : Pointer to a string of chars to hold the serialized handle + * If the function succeeds, the pointer points to a + * serialized view of the handle. + * hnd : Handle to shm seg which has to be serialized + */ + +int MPL_shm_hnd_get_serialized_by_ref(MPL_shm_hnd_t hnd, char **str_ptr) +{ + int mpi_errno = 0; + *str_ptr = (char *)MPLI_shm_ghnd_get_by_ref(hnd); + return mpi_errno; +} + +/* Deserialize a handle by reference. + * Rationale : The user already has a serialized view of the handle. + * The user does not want to manage the view buffer any more + * & also needs to deserialize from the buffer. + * ser_hnd_ptr : Pointer to a serialized view of the handle. The user + * no longer needs to take care of this buffer. + * hnd : If the function succeeds this points to the deserialized + * handle. + */ +int MPL_shm_hnd_deserialize_by_ref(MPL_shm_hnd_t hnd, char **ser_hnd_ptr) +{ + int mpi_errno = 0; + + MPLI_shm_hnd_reset_val(hnd); + MPLI_shm_ghnd_set_by_ref(hnd, *ser_hnd_ptr); + + mpi_errno = MPL_shm_seg_open(hnd, 0); + + return mpi_errno; +} + +/* Initialize a shared memory handle + * hnd_ptr : A pointer to the shared memory handle + */ + +int MPL_shm_hnd_init(MPL_shm_hnd_t *hnd_ptr) +{ + int rc = -1; + rc = MPLI_shm_hnd_alloc(hnd_ptr); + MPLI_shm_hnd_reset_val(*hnd_ptr); + return rc; +} + +/* Finalize a shared memory handle. + * hnd_ptr : A pointer to the shm handle to be finalized. + * Any handle that is init has to be finalized. + */ +int MPL_shm_hnd_finalize(MPL_shm_hnd_t *hnd_ptr) +{ + int mpi_errno = 0; + + /* A finalize can/should be called on an invalid handle + * Don't assert if we fail here ... + */ + MPLI_shm_hnd_close(*hnd_ptr); + MPLI_shm_hnd_free(*hnd_ptr); + + *hnd_ptr = MPL_SHM_HND_INVALID; + + return mpi_errno; +} + diff --git a/src/mpl/src/shm/mpl_shm_mmap.c b/src/mpl/src/shm/mpl_shm_mmap.c new file mode 100644 index 0000000..edefa1a --- /dev/null +++ b/src/mpl/src/shm/mpl_shm_mmap.c @@ -0,0 +1,188 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* vim: set ft=c.mpich : */ +/* + * (C) 2016 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + */ + +#include "mpl.h" + +#ifdef MPL_USE_MMAP_SHM + +#include <fcntl.h> +#include <sys/mman.h> +#if defined (MPL_HAVE_MKSTEMP) && defined (MPL_NEEDS_MKSTEMP_DECL) +extern int mkstemp(char *template); +#endif + +inline int MPLI_shm_lhnd_close(MPL_shm_hnd_t hnd) +{ + MPLI_shm_lhnd_t lhnd = MPLI_SHM_LHND_INVALID; + lhnd = MPLI_shm_lhnd_get(hnd); + if (lhnd != MPLI_SHM_LHND_INVALID) { + if (close(lhnd) == 0) { + MPLI_shm_lhnd_set(hnd, MPLI_SHM_LHND_INIT_VAL); + } else { + /* close() failed */ + return -1; + } + } + return 0; +} + +/* A template function which creates/attaches shm seg handle + * to the shared memory. Used by user-exposed functions below + */ +/* FIXME: Pass (void **)shm_addr_ptr instead of (char **) shm_addr_ptr + * since a util func should be generic + * Currently not passing (void **) to reduce warning in nemesis + * code which passes (char **) ptrs to be attached to a seg + */ + +#undef FUNCNAME +#define FUNCNAME MPL_shm_seg_create_attach_templ +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPL_shm_seg_create_attach_templ( + MPL_shm_hnd_t hnd, intptr_t seg_sz, char **shm_addr_ptr, + int offset, int flag) +{ + MPLI_shm_lhnd_t lhnd = -1, rc = -1; + + if(flag & MPLI_SHM_FLAG_SHM_CREATE){ + char dev_shm_fname[] = "/dev/shm/mpich_shar_tmpXXXXXX"; + char tmp_fname[] = "/tmp/mpich_shar_tmpXXXXXX"; + char *chosen_fname = NULL; + + chosen_fname = dev_shm_fname; + lhnd = mkstemp(chosen_fname); + if(lhnd == -1){ + chosen_fname = tmp_fname; + lhnd = mkstemp(chosen_fname); + } + + MPLI_shm_lhnd_set(hnd, lhnd); + rc = (MPLI_shm_lhnd_t)lseek(lhnd, seg_sz - 1, SEEK_SET); + do{ + rc = (int) write(lhnd, "", 1); + }while((rc == -1) && (errno == EINTR)); + + rc = MPLI_shm_ghnd_alloc(hnd); + rc = MPLI_shm_ghnd_set_by_val(hnd, "%s", chosen_fname); + } + else{ + /* Open an existing shared memory seg */ + if(!MPLI_shm_lhnd_is_valid(hnd)){ + lhnd = open(MPLI_shm_ghnd_get_by_ref(hnd), O_RDWR); + MPLI_shm_lhnd_set(hnd, lhnd); + } + } + + if(flag & MPLI_SHM_FLAG_SHM_ATTACH){ + void *buf_ptr = NULL; + buf_ptr = mmap(NULL, seg_sz, PROT_READ | PROT_WRITE, + MAP_SHARED, MPLI_shm_lhnd_get(hnd), 0); + *shm_addr_ptr = (char*)buf_ptr; + } + +fn_exit: + /* FIXME: Close local handle only when closing the shm handle */ + if(MPLI_shm_lhnd_is_valid(hnd)){ + rc = MPLI_shm_lhnd_close(hnd); + } + return rc; +fn_fail: + goto fn_exit; +} + +/* Create new SHM segment + * hnd : A "init"ed shared memory handle + * seg_sz : Size of shared memory segment to be created + */ +int MPL_shm_seg_create(MPL_shm_hnd_t hnd, intptr_t seg_sz) +{ + int rc = -1; + rc = MPL_shm_seg_create_attach_templ(hnd, seg_sz, NULL, 0, + MPLI_SHM_FLAG_SHM_CREATE); + return rc; +} + +/* Open an existing SHM segment + * hnd : A shm handle with a valid global handle + * seg_sz : Size of shared memory segment to open + * Currently only using internally within wrapper funcs + */ +int MPL_shm_seg_open(MPL_shm_hnd_t hnd, intptr_t seg_sz) +{ + int rc = -1; + rc = MPL_shm_seg_create_attach_templ(hnd, seg_sz, NULL, 0, MPLI_SHM_FLAG_CLR); + return rc; +} + +/* Create new SHM segment and attach to it + * hnd : A "init"ed shared mem handle + * seg_sz: Size of shared mem segment + * shm_addr_ptr : Pointer to shared memory address to attach + * the shared mem segment + * offset : Offset to attach the shared memory address to + */ +int MPL_shm_seg_create_and_attach(MPL_shm_hnd_t hnd, intptr_t seg_sz, + char **shm_addr_ptr, int offset) +{ + int rc = 0; + rc = MPL_shm_seg_create_attach_templ(hnd, seg_sz, shm_addr_ptr, offset, + MPLI_SHM_FLAG_SHM_CREATE | MPLI_SHM_FLAG_SHM_ATTACH); + return rc; +} + +/* Attach to an existing SHM segment + * hnd : A "init"ed shared mem handle + * seg_sz: Size of shared mem segment + * shm_addr_ptr : Pointer to shared memory address to attach + * the shared mem segment + * offset : Offset to attach the shared memory address to + */ +int MPL_shm_seg_attach(MPL_shm_hnd_t hnd, intptr_t seg_sz, char **shm_addr_ptr, + int offset) +{ + int rc = 0; + rc = MPL_shm_seg_create_attach_templ(hnd, seg_sz, shm_addr_ptr, offset, + MPLI_SHM_FLAG_SHM_ATTACH); + return rc; +} +/* Detach from an attached SHM segment */ +#undef FUNCNAME +#define FUNCNAME MPL_shm_seg_detach +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +int MPL_shm_seg_detach(MPL_shm_hnd_t hnd, char **shm_addr_ptr, intptr_t seg_sz) +{ + int rc = -1; + + rc = munmap(*shm_addr_ptr, seg_sz); + *shm_addr_ptr = NULL; + +fn_exit: + return rc; +fn_fail: + goto fn_exit; +} + +/* Remove an existing SHM segment */ +#undef FUNCNAME +#define FUNCNAME MPL_shm_seg_remove +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +int MPL_shm_seg_remove(MPL_shm_hnd_t hnd) +{ + int rc = -1; + + rc = unlink(MPLI_shm_ghnd_get_by_ref(hnd)); + +fn_exit: + return rc; +fn_fail: + goto fn_exit; +} + +#endif /* MPL_USE_MMAP_SHM */ diff --git a/src/mpl/src/shm/mpl_shm_sysv.c b/src/mpl/src/shm/mpl_shm_sysv.c new file mode 100644 index 0000000..ef147e6 --- /dev/null +++ b/src/mpl/src/shm/mpl_shm_sysv.c @@ -0,0 +1,152 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* vim: set ft=c.mpich : */ +/* + * (C) 2016 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + */ + +#include "mpl.h" + +#ifdef MPL_USE_SYSV_SHM + +#include <sys/stat.h> +#include <sys/ipc.h> +#include <sys/shm.h> + +/* A template function which creates/attaches shm seg handle + * to the shared memory. Used by user-exposed functions below + */ +#undef FUNCNAME +#define FUNCNAME MPL_shm_seg_create_attach_templ +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPL_shm_seg_create_attach_templ( + MPL_shm_hnd_t hnd, intptr_t seg_sz, char **shm_addr_ptr, + int offset, int flag) +{ + int rc = -1; + int lhnd = -1; + + if(flag & MPLI_SHM_FLAG_SHM_CREATE){ + lhnd = shmget(IPC_PRIVATE, seg_sz, IPC_CREAT | S_IRWXU); + MPL_shm_lhnd_set(hnd, lhnd); + rc = MPLI_shm_ghnd_alloc(hnd); + rc = MPLI_shm_ghnd_set_by_val(hnd, "%d", lhnd); + } + else{ + /* Open an existing shared memory seg */ + if(!MPLI_shm_lhnd_is_valid(hnd)){ + lhnd = atoi(MPLI_shm_ghnd_get_by_ref(hnd)); + MPLI_shm_lhnd_set(hnd, lhnd); + } + } + + if(flag & MPLI_SHM_FLAG_SHM_ATTACH){ + /* Attach to shared mem seg */ + *shm_addr_ptr = shmat(MPLI_shm_lhnd_get(hnd), NULL, 0x0); + } + +fn_exit: + return rc; +fn_fail: + goto fn_exit; +} + +/* Create new SHM segment + * hnd : A "init"ed shared memory handle + * seg_sz : Size of shared memory segment to be created + */ +int MPL_shm_seg_create(MPL_shm_hnd_t hnd, intptr_t seg_sz) +{ + int rc = -1; + rc = MPL_shm_seg_create_attach_templ(hnd, seg_sz, NULL, 0, + MPLI_SHM_FLAG_SHM_CREATE); + return rc; +} + +/* Open an existing SHM segment + * hnd : A shm handle with a valid global handle + * seg_sz : Size of shared memory segment to open + * Currently only using internally within wrapper funcs + */ +int MPL_shm_seg_open(MPL_shm_hnd_t hnd, intptr_t seg_sz) +{ + int rc = -1; + rc = MPL_shm_seg_create_attach_templ(hnd, seg_sz, NULL, 0, MPLI_SHM_FLAG_CLR); + return rc; +} + +/* Create new SHM segment and attach to it + * hnd : A "init"ed shared mem handle + * seg_sz: Size of shared mem segment + * shm_addr_ptr : Pointer to shared memory address to attach + * the shared mem segment + * offset : Offset to attach the shared memory address to + */ +int MPL_shm_seg_create_and_attach(MPL_shm_hnd_t hnd, intptr_t seg_sz, + char **shm_addr_ptr, int offset) +{ + int rc = 0; + rc = MPL_shm_seg_create_attach_templ(hnd, seg_sz, shm_addr_ptr, offset, + MPLI_SHM_FLAG_SHM_CREATE | MPLI_SHM_FLAG_SHM_ATTACH); + return rc; +} + +/* Attach to an existing SHM segment + * hnd : A "init"ed shared mem handle + * seg_sz: Size of shared mem segment + * shm_addr_ptr : Pointer to shared memory address to attach + * the shared mem segment + * offset : Offset to attach the shared memory address to + */ +int MPL_shm_seg_attach(MPL_shm_hnd_t hnd, intptr_t seg_sz, char **shm_addr_ptr, + int offset) +{ + int rc = 0; + rc = MPL_shm_seg_create_attach_templ(hnd, seg_sz, shm_addr_ptr, offset, + MPLI_SHM_FLAG_SHM_ATTACH); + return rc; +} +/* Detach from an attached SHM segment + * hnd : Handle to the shm segment + * shm_addr_ptr : Pointer to the shm address to detach + * seg_sz : Size of shm segment + */ +#undef FUNCNAME +#define FUNCNAME MPL_shm_seg_detach +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +int MPL_shm_seg_detach(MPL_shm_hnd_t hnd, char **shm_addr_ptr, intptr_t seg_sz) +{ + int rc = -1; + + rc = shmdt(*shm_addr_ptr); + *shm_addr_ptr = NULL; + +fn_exit: + return rc; +fn_fail: + goto fn_exit; +} + +/* Remove a shared memory segment + * hnd : Handle to the shared memory segment to be removed + */ +#undef FUNCNAME +#define FUNCNAME MPL_shm_seg_remove +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +int MPL_shm_seg_remove(MPL_shm_hnd_t hnd) +{ + struct shmid_ds ds; + int rc = -1; + + rc = shmctl(MPL_shm_lhnd_get(hnd), IPC_RMID, &ds); + +fn_exit: + return rc; +fn_fail: + goto fn_exit; +} + +#endif /* MPL_USE_SYSV_SHM */ diff --git a/src/mpl/src/shm/mpl_shm_win.c b/src/mpl/src/shm/mpl_shm_win.c new file mode 100644 index 0000000..a2b7a87 --- /dev/null +++ b/src/mpl/src/shm/mpl_shm_win.c @@ -0,0 +1,137 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* vim: set ft=c.mpich : */ +/* + * (C) 2016 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + */ + +#include "mpl.h" + +#ifdef MPL_USE_NT_SHM + +#include<winsock2.h> +#include<windows.h> + +/* A template function which creates/attaches shm seg handle + * to the shared memory. Used by user-exposed functions below + */ +#undef FUNCNAME +#define FUNCNAME MPL_shm_seg_create_attach_templ +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPL_shm_seg_create_attach_templ( + MPL_shm_hnd_t hnd, intptr_t seg_sz, char **shm_addr_ptr, + int offset, int flag) +{ + HANDLE lhnd = INVALID_HANDLE_VALUE; + int rc = -1; + ULARGE_INTEGER seg_sz_large; + seg_sz_large.QuadPart = seg_sz; + + if(!MPLI_shm_ghnd_is_valid(hnd)){ + rc = MPLI_shm_ghnd_set_uniq(hnd); + } + + if(flag & MPLI_SHM_FLAG_SHM_CREATE){ + lhnd = CreateFileMapping(INVALID_HANDLE_VALUE, NULL, + PAGE_READWRITE, seg_sz_large.HighPart, seg_sz_large.LowPart, + MPLI_shm_ghnd_get_by_ref(hnd)); + MPLI_shm_lhnd_set(hnd, lhnd); + } + else{ + if(!MPLI_shm_lhnd_is_valid(hnd)){ + /* Strangely OpenFileMapping() returns NULL on error! */ + lhnd = OpenFileMapping(FILE_MAP_WRITE, FALSE, + MPLI_shm_ghnd_get_by_ref(hnd)); + + MPLI_shm_lhnd_set(hnd, lhnd); + } + } + + if(flag & MPLI_SHM_FLAG_SHM_ATTACH){ + *shm_addr_ptr = (char *)MapViewOfFile(MPLI_shm_lhnd_get(hnd), + FILE_MAP_WRITE, 0, offset, 0); + } + +fn_exit: + return rc; +fn_fail: + goto fn_exit; +} + +/* Create new SHM segment + * hnd : A "init"ed shared memory handle + * seg_sz : Size of shared memory segment to be created + */ +int MPL_shm_seg_create(MPL_shm_hnd_t hnd, intptr_t seg_sz) +{ + int rc = -1; + rc = MPL_shm_seg_create_attach_templ(hnd, seg_sz, NULL, 0, + MPLI_SHM_FLAG_SHM_CREATE); + return rc; +} + +/* Open an existing SHM segment + * hnd : A shm handle with a valid global handle + * seg_sz : Size of shared memory segment to open + * Currently only using internally within wrapper funcs + */ +int MPL_shm_seg_open(MPL_shm_hnd_t hnd, intptr_t seg_sz) +{ + int rc = -1; + rc = MPL_shm_seg_create_attach_templ(hnd, seg_sz, NULL, 0, MPLI_SHM_FLAG_CLR); + return rc; +} + +/* Create new SHM segment and attach to it + * hnd : A "init"ed shared mem handle + * seg_sz: Size of shared mem segment + * shm_addr_ptr : Pointer to shared memory address to attach + * the shared mem segment + * offset : Offset to attach the shared memory address to + */ +int MPL_shm_seg_create_and_attach(MPL_shm_hnd_t hnd, intptr_t seg_sz, + char **shm_addr_ptr, int offset) +{ + int rc = 0; + rc = MPL_shm_seg_create_attach_templ(hnd, seg_sz, shm_addr_ptr, offset, + MPLI_SHM_FLAG_SHM_CREATE | MPLI_SHM_FLAG_SHM_ATTACH); + return rc; +} + +/* Attach to an existing SHM segment + * hnd : A "init"ed shared mem handle + * seg_sz: Size of shared mem segment + * shm_addr_ptr : Pointer to shared memory address to attach + * the shared mem segment + * offset : Offset to attach the shared memory address to + */ +int MPL_shm_seg_attach(MPL_shm_hnd_t hnd, intptr_t seg_sz, char **shm_addr_ptr, + int offset) +{ + int rc = 0; + rc = MPL_shm_seg_create_attach_templ(hnd, seg_sz, shm_addr_ptr, offset, + MPLI_SHM_FLAG_SHM_ATTACH); + return rc; +} +/* Detach from an attached SHM segment */ +#undef FUNCNAME +#define FUNCNAME MPL_shm_seg_detach +#undef FCNAME +#define FCNAME MPL_QUOTE(FUNCNAME) +static inline int MPL_shm_seg_detach( + MPL_shm_hnd_t hnd, char **shm_addr_ptr, intptr_t seg_sz) +{ + int rc = -1; + + rc = UnmapViewOfFile(*shm_addr_ptr); + *shm_addr_ptr = NULL; + +fn_exit: + return rc; +fn_fail: + goto fn_exit; +} + + +#endif /* MPL_USE_NT_SHM */ diff --git a/src/util/wrappers/Makefile.mk b/src/util/wrappers/Makefile.mk index 3d21618..6cfe1ee 100644 --- a/src/util/wrappers/Makefile.mk +++ b/src/util/wrappers/Makefile.mk @@ -9,9 +9,6 @@ AM_CPPFLAGS += -I$(top_srcdir)/src/util/wrappers -I$(top_builddir)/src/util/wrappers noinst_HEADERS += \ - src/util/wrappers/mpiu_os_wrappers.h \ - src/util/wrappers/mpiu_os_wrappers_pre.h \ - src/util/wrappers/mpiu_shm_wrappers.h \ src/util/wrappers/mpiu_sock_wrappers.h \ src/util/wrappers/mpiu_util_wrappers.h diff --git a/src/util/wrappers/mpiu_os_wrappers.h b/src/util/wrappers/mpiu_os_wrappers.h deleted file mode 100644 index 9ac6c18..0000000 --- a/src/util/wrappers/mpiu_os_wrappers.h +++ /dev/null @@ -1,13 +0,0 @@ -/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ -/* - * (C) 2006 by Argonne National Laboratory. - * See COPYRIGHT in top-level directory. - */ -#ifndef MPIU_OS_WRAPPERS_H_INCLUDED -#define MPIU_OS_WRAPPERS_H_INCLUDED -/* #include "mpiu_os_wrappers_conf.h" */ -#include "mpiu_util_wrappers.h" -#include "mpl.h" -#include "mpiu_shm_wrappers.h" -#include "mpiu_sock_wrappers.h" -#endif /* MPIU_OS_WRAPPERS_H_INCLUDED */ diff --git a/src/util/wrappers/mpiu_os_wrappers_pre.h b/src/util/wrappers/mpiu_os_wrappers_pre.h deleted file mode 100644 index 44858fe..0000000 --- a/src/util/wrappers/mpiu_os_wrappers_pre.h +++ /dev/null @@ -1,61 +0,0 @@ -/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ -/* - * (C) 2010 by Argonne National Laboratory. - * See COPYRIGHT in top-level directory. - */ - -/* This file contains "pre" definitions and declarations for the OS wrappers. - * That is, things that shouldn't depend on much more than the mpichconf.h - * values. */ - -#ifndef MPIU_OS_WRAPPERS_PRE_H_INCLUDED -#define MPIU_OS_WRAPPERS_PRE_H_INCLUDED - -/* ------------------------------------------------------------------------ */ -/* util wrappers */ -/* TODO port defs/decls here as necessary */ - -/* ------------------------------------------------------------------------ */ -/* process wrappers */ -/* TODO port defs/decls here as necessary */ - -/* ------------------------------------------------------------------------ */ -/* shm wrappers */ - -#define MPIU_SHMW_FLAG_CLR 0x0 -#define MPIU_SHMW_FLAG_SHM_CREATE 0x1 -#define MPIU_SHMW_FLAG_SHM_ATTACH 0x10 -#define MPIU_SHMW_FLAG_GHND_STATIC 0x100 - -#ifdef USE_SYSV_SHM -typedef int MPIU_SHMW_Lhnd_t; -#elif defined USE_MMAP_SHM -typedef MPI_Aint MPIU_SHMW_Lhnd_t; -#elif defined USE_NT_SHM -typedef HANDLE MPIU_SHMW_Lhnd_t; -#endif - -typedef char * MPIU_SHMW_Ghnd_t; -/* The local handle, lhnd, is valid only for the current process, - * The global handle, ghnd, is valid across multiple processes - * The handle flag, flag, is used to set various attributes of the - * handle. - */ -typedef struct{ - MPIU_SHMW_Lhnd_t lhnd; - MPIU_SHMW_Ghnd_t ghnd; - int flag; -} MPIU_SHMW_LGhnd_t; - -typedef MPIU_SHMW_LGhnd_t * MPIU_SHMW_Hnd_t; - -#define MPIU_SHMW_HND_INVALID NULL -#define MPIU_SHMW_GHND_INVALID NULL -#define MPIU_SHMW_GHND_INIT_VAL '\0' -/* TODO port additional defs/decls here as necessary */ - -/* ------------------------------------------------------------------------ */ -/* sock wrappers */ -/* TODO port defs/decls here as necessary */ - -#endif /* MPIU_OS_WRAPPERS_PRE_H_INCLUDED */ diff --git a/src/util/wrappers/mpiu_shm_wrappers.h b/src/util/wrappers/mpiu_shm_wrappers.h deleted file mode 100644 index 8ea3575..0000000 --- a/src/util/wrappers/mpiu_shm_wrappers.h +++ /dev/null @@ -1,938 +0,0 @@ -/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ -/* - * (C) 2006 by Argonne National Laboratory. - * See COPYRIGHT in top-level directory. - */ -#ifndef MPIU_SHM_WRAPPERS_H_INCLUDED -#define MPIU_SHM_WRAPPERS_H_INCLUDED -/* FIXME: Add underscore to the end of funcs/macro names not to - * be exposed to user - */ -/* SHM Wrapper funcs defined in this header file */ - -#include "mpichconfconst.h" -#include "mpichconf.h" -#ifdef HAVE_UNISTD_H -#include <unistd.h> -#endif -#ifdef HAVE_ERRNO_H -#include <errno.h> -#endif -#ifdef HAVE_STDLIB_H -#include <stdlib.h> -#endif - -#ifdef USE_SYSV_SHM - #include <sys/stat.h> - #include <sys/ipc.h> - #include <sys/shm.h> -#elif defined USE_MMAP_SHM - #include <fcntl.h> - #include <sys/mman.h> - #if defined (HAVE_MKSTEMP) && defined (NEEDS_MKSTEMP_DECL) - extern int mkstemp(char *template); - #endif -#elif defined USE_NT_SHM - #include<winsock2.h> - #include<windows.h> -#else -# error "No shared memory mechanism specified" -#endif - -#include "mpiu_os_wrappers_pre.h" -#include "mpiu_util_wrappers.h" - -#if !defined(MPIIMPL_H_INCLUDED) -#error "this header must be included after mpiimpl.h" -#endif - -/* FIXME: Reduce the conditional checks for wrapper-internal - * utility funcs/macros. - */ - -/* Allocate mem for references within the handle */ -/* Returns 0 on success, -1 on error */ -#define MPIU_SHMW_Hnd_ref_alloc(hnd)( \ - ((hnd)->ghnd = (MPIU_SHMW_Ghnd_t) \ - MPL_malloc(MPIU_SHMW_GHND_SZ)) ? 0 : -1 \ -) - -#ifdef USE_SYSV_SHM -/************************** Sys V shm **************************/ -#define MPIU_SHMW_MAX_INT_STR_LEN 20 -#define MPIU_SHMW_SEG_KEY_LEN MPIU_SHMW_MAX_INT_STR_LEN -#define MPIU_SHMW_GHND_SZ MPIU_SHMW_SEG_KEY_LEN -#define MPIU_SHMW_LHND_INVALID -1 -#define MPIU_SHMW_LHND_INIT_VAL -1 - -#define MPIU_SHMW_SEG_ALREADY_EXISTS EEXIST - -/* These macros are the setters/getters for the shm handle */ -#define MPIU_SHMW_Lhnd_get(hnd) ((hnd)->lhnd) -#define MPIU_SHMW_Lhnd_set(hnd, val) ((hnd)->lhnd=val) - -#define MPIU_SHMW_Lhnd_is_valid(hnd) ( \ - ((hnd)->lhnd != MPIU_SHMW_LHND_INVALID) \ -) -#define MPIU_SHMW_Lhnd_is_init(hnd) 1 -/* Nothing to be done at close */ -#define MPIU_SHMW_Lhnd_close(hnd) 0 - -#elif defined USE_MMAP_SHM -/************************** MMAP shm **************************/ -#define MPIU_SHMW_FNAME_LEN 50 -#define MPIU_SHMW_GHND_SZ MPIU_SHMW_FNAME_LEN -#define MPIU_SHMW_LHND_INVALID -1 -#define MPIU_SHMW_LHND_INIT_VAL -1 - -#define MPIU_SHMW_SEG_ALREADY_EXISTS EEXIST - -/* These macros are the setters/getters for the shm handle */ -#define MPIU_SHMW_Lhnd_get(hnd) ((hnd)->lhnd) -#define MPIU_SHMW_Lhnd_set(hnd, val) ((hnd)->lhnd=val) - -#define MPIU_SHMW_Lhnd_is_valid(hnd) ( \ - ((hnd)->lhnd != MPIU_SHMW_LHND_INVALID) \ -) -#define MPIU_SHMW_Lhnd_is_init(hnd) 1 - -/* Returns 0 on success, -1 on error */ -static inline int MPIU_SHMW_Lhnd_close(MPIU_SHMW_Hnd_t hnd) -{ - MPIU_SHMW_Lhnd_t lhnd = MPIU_SHMW_LHND_INVALID; - lhnd = MPIU_SHMW_Lhnd_get(hnd); - if(lhnd != MPIU_SHMW_LHND_INVALID) { - if(close(lhnd) == 0){ - MPIU_SHMW_Lhnd_set(hnd, MPIU_SHMW_LHND_INIT_VAL); - } - else{ - /* close() failed */ - return -1; - } - } - return 0; -} - -#elif defined USE_NT_SHM -/************************** NT shm **************************/ -#define MPIU_SHMW_SEG_NAME_LEN 70 -#define MPIU_SHMW_GHND_SZ MPIU_SHMW_SEG_NAME_LEN -#define MPIU_SHMW_LHND_INVALID INVALID_HANDLE_VALUE -#define MPIU_SHMW_LHND_INIT_VAL INVALID_HANDLE_VALUE - -#define MPIU_SHMW_SEG_ALREADY_EXISTS ERROR_ALREADY_EXISTS - -/* These macros are the setters/getters for the shm handle */ -#define MPIU_SHMW_Lhnd_get(hnd) ((hnd)->lhnd) -#define MPIU_SHMW_Lhnd_set(hnd, val) ((hnd)->lhnd=val) - -#define MPIU_SHMW_Lhnd_is_valid(hnd) ( \ - ((hnd)->lhnd != MPIU_SHMW_LHND_INVALID) \ -) -#define MPIU_SHMW_Lhnd_is_init(hnd) 1 - -/* Returns 0 on success, -1 on error */ -#define MPIU_SHMW_Lhnd_close(hnd)( \ - (CloseHandle(MPIU_SHMW_Lhnd_get(hnd)) != 0) ? 0 : -1 \ -) -/* Returns 0 on success, -1 on error */ -static inline int MPIU_SHMW_Ghnd_set_uniq(MPIU_SHMW_Hnd_t hnd) -{ - if(MPIU_SHMW_Hnd_ref_alloc(hnd) == 0){ - if(MPIU_OSW_Get_uniq_str(hnd->ghnd, MPIU_SHMW_GHND_SZ) != 0){ - return -1; - } - } - else{ - return -1; - } - return 0; -} -#endif /* USE_NT_SHM */ - -#define MPIU_SHMW_HND_SZ (sizeof(MPIU_SHMW_LGhnd_t)) -#define MPIU_SHMW_SER_HND_SZ MPIU_SHMW_GHND_SZ - -/* These macros are the setters/getters for the shm handle */ -#define MPIU_SHMW_Ghnd_get_by_ref(hnd) ((hnd)->ghnd) - -/* Returns -1 on error, 0 on success */ -#define MPIU_SHMW_Ghnd_get_by_val(hnd, str, strlen) ( \ - (MPL_snprintf(str, strlen, "%s", \ - MPIU_SHMW_Ghnd_get_by_ref(hnd))) ? 0 : -1 \ -) -#define MPIU_SHMW_Ghnd_set_by_ref(hnd, val) ((hnd)->ghnd = val) -/* Returns -1 on error, 0 on success */ -/* FIXME: What if val is a non-null terminated string ? */ -#define MPIU_SHMW_Ghnd_set_by_val(hnd, fmt, val) ( \ - (MPL_snprintf(MPIU_SHMW_Ghnd_get_by_ref(hnd), \ - MPIU_SHMW_GHND_SZ, fmt, val)) ? 0 : -1 \ -) - -#define MPIU_SHMW_Ghnd_is_valid(hnd) ( \ - (((hnd)->ghnd == MPIU_SHMW_GHND_INVALID) || \ - (strlen((hnd)->ghnd) == 0)) ? 0 : 1 \ -) -#define MPIU_SHMW_Ghnd_is_init(hnd) ( \ - ((hnd)->flag & MPIU_SHMW_FLAG_GHND_STATIC) ? \ - 1 : \ - (((hnd)->ghnd != MPIU_SHMW_GHND_INVALID) ? 1 : 0) \ -) - -/* Allocate mem for global handle. - * Returns 0 on success, -1 on failure - */ -static inline int MPIU_SHMW_Ghnd_alloc(MPIU_SHMW_Hnd_t hnd) -{ - if(!(hnd->ghnd)){ - hnd->ghnd = (MPIU_SHMW_Ghnd_t)MPL_malloc(MPIU_SHMW_GHND_SZ); - if(!(hnd->ghnd)){ return -1; } - } - /* Global handle is no longer static */ - hnd->flag &= ~MPIU_SHMW_FLAG_GHND_STATIC; - return 0; -} - -/* A Handle is valid if it is initialized/init and has a value - * different from the default/invalid value assigned during init - */ -#define MPIU_SHMW_Hnd_is_valid(hnd) ( \ - ((hnd) && \ - MPIU_SHMW_Lhnd_is_valid(hnd) && \ - MPIU_SHMW_Ghnd_is_valid(hnd)) \ -) - -/* With MMAP_SHM, NT_SHM & SYSV_SHM local handle is always init'ed */ -#define MPIU_SHMW_Hnd_is_init(hnd) ( \ - ((hnd) && /* MPIU_SHMW_Lhnd_is_init(hnd) && */ \ - MPIU_SHMW_Ghnd_is_init(hnd)) \ -) - - -/* Allocate mem for handle. Lazy allocation for global handle */ -/* Returns 0 on success, -1 on error */ -static inline int MPIU_SHMW_Hnd_alloc(MPIU_SHMW_Hnd_t *hnd_ptr) -{ - MPIR_Assert(hnd_ptr); - *hnd_ptr = (MPIU_SHMW_Hnd_t) MPL_malloc(MPIU_SHMW_HND_SZ); - if(*hnd_ptr){ - (*hnd_ptr)->flag = MPIU_SHMW_FLAG_GHND_STATIC; - } - else{ - return -1; - } - return 0; -} - -/* Close Handle */ -#define MPIU_SHMW_Hnd_close(hnd) MPIU_SHMW_Lhnd_close(hnd) - -static inline void MPIU_SHMW_Hnd_reset_val(MPIU_SHMW_Hnd_t hnd) -{ - MPIR_Assert(hnd); - MPIU_SHMW_Lhnd_set(hnd, MPIU_SHMW_LHND_INIT_VAL); - if(hnd->flag & MPIU_SHMW_FLAG_GHND_STATIC){ - hnd->ghnd = MPIU_SHMW_GHND_INVALID; - } - else{ - MPIR_Assert(hnd->ghnd); - (hnd->ghnd)[0] = MPIU_SHMW_GHND_INIT_VAL; - } -} - -static inline void MPIU_SHMW_Hnd_free(MPIU_SHMW_Hnd_t hnd) -{ - if(MPIU_SHMW_Hnd_is_init(hnd)){ - if(!(hnd->flag & MPIU_SHMW_FLAG_GHND_STATIC)){ - MPL_free(hnd->ghnd); - } - MPL_free(hnd); - } -} - -static inline int MPIU_SHMW_Seg_open(MPIU_SHMW_Hnd_t hnd, size_t seg_sz); -static inline int MPIU_SHMW_Hnd_deserialize_by_ref(MPIU_SHMW_Hnd_t hnd, char **ser_hnd_ptr); - -/* FIXME : Don't print ENGLISH strings on error. Define the error - * strings in errnames.txt - */ -/* Serialize a handle. A serialized handle is a string of - * characters that can be persisted by the caller. The serialized - * handle can be used to create another ref to the shared mem seg - * by deserializing it. - * str : A string of chars of len, str_len. - * If the function succeeds the serialized handle is copied - * into this user buffer - * hnd : Handle to shared memory - */ -#undef FUNCNAME -#define FUNCNAME MPIU_SHMW_Hnd_serialize -#undef FCNAME -#define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIU_SHMW_Hnd_serialize(char *str, - MPIU_SHMW_Hnd_t hnd, int str_len) -{ - int mpi_errno = MPI_SUCCESS; - int rc = -1; - - MPIR_Assert(MPIU_SHMW_Hnd_is_init(hnd)); - MPIR_Assert(str); - MPIR_Assert(str_len >= MPIU_SHMW_GHND_SZ); - - rc = MPIU_SHMW_Ghnd_get_by_val(hnd, str, str_len); - MPIR_ERR_CHKANDJUMP(rc != 0, mpi_errno, MPI_ERR_OTHER, "**shmw_gethnd"); - -fn_exit: - return mpi_errno; -fn_fail: - goto fn_exit; -} - -/* Deserialize a handle. - * str_hnd : A null-terminated string of len str_hnd_len that - * contains the serialized handle. - * hnd : If the call succeeds the user gets back a handle,hnd, to - * shared mem - deserialized from strHnd. This handle - * will refer to the shm seg referred by the serialized - * handle. - */ -#undef FUNCNAME -#define FUNCNAME MPIU_SHMW_Hnd_deserialize -#undef FCNAME -#define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIU_SHMW_Hnd_deserialize( - MPIU_SHMW_Hnd_t hnd, const char *str_hnd, size_t str_hnd_len) -{ - int mpi_errno = MPI_SUCCESS; - int rc = -1; - - MPIR_Assert(MPIU_SHMW_Hnd_is_init(hnd)); - MPIR_ERR_CHKINTERNAL(!str_hnd, mpi_errno, "ser hnd is null"); - MPIR_ERR_CHKANDJUMP(str_hnd_len>=MPIU_SHMW_GHND_SZ, - mpi_errno, MPI_ERR_OTHER, "**shmw_deserbufbig"); - - MPIU_SHMW_Hnd_reset_val(hnd); - - rc = MPIU_SHMW_Ghnd_alloc(hnd); - MPIR_ERR_CHKANDJUMP1((rc != 0), mpi_errno, MPI_ERR_OTHER, - "**nomem", "**nomem %s", "shared mem global handle"); - - rc = MPIU_SHMW_Ghnd_set_by_val(hnd, "%s", str_hnd); - MPIR_Assert(rc == 0); - - mpi_errno = MPIU_SHMW_Seg_open(hnd, 0); - if(mpi_errno != MPI_SUCCESS) { MPIR_ERR_POP(mpi_errno); } - -fn_exit: - return mpi_errno; -fn_fail: - goto fn_exit; -} - -/* Get a serialized handle by reference. - * Rationale: The user might only want to read the serialized view - * of the handle & hence not want to allocate a buffer for the ser view - * of the handle. - * str_ptr : Pointer to a string of chars to hold the serialized handle - * If the function succeeds, the pointer points to a - * serialized view of the handle. - * hnd : Handle to shm seg which has to be serialized - */ - -#undef FUNCNAME -#define FUNCNAME MPIU_SHMW_Hnd_get_serialized_by_ref -#undef FCNAME -#define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIU_SHMW_Hnd_get_serialized_by_ref( - MPIU_SHMW_Hnd_t hnd, char **str_ptr) -{ - int mpi_errno = MPI_SUCCESS; - - MPIR_Assert(MPIU_SHMW_Hnd_is_init(hnd)); - MPIR_Assert(str_ptr); - - *str_ptr = (char *)MPIU_SHMW_Ghnd_get_by_ref(hnd); - MPIR_Assert(*str_ptr); - - return mpi_errno; -} - -/* Deserialize a handle by reference. - * Rationale : The user already has a serialized view of the handle. - * The user does not want to manage the view buffer any more - * & also needs to deserialize from the buffer. - * ser_hnd_ptr : Pointer to a serialized view of the handle. The user - * no longer needs to take care of this buffer. - * hnd : If the function succeeds this points to the deserialized - * handle. - */ -#undef FUNCNAME -#define FUNCNAME MPIU_SHMW_Hnd_deserialize_by_ref -#undef FCNAME -#define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIU_SHMW_Hnd_deserialize_by_ref( - MPIU_SHMW_Hnd_t hnd, char **ser_hnd_ptr) -{ - int mpi_errno = MPI_SUCCESS; - - MPIR_Assert(MPIU_SHMW_Hnd_is_init(hnd)); - MPIR_Assert(ser_hnd_ptr); - - MPIR_ERR_CHKINTERNAL(!(*ser_hnd_ptr), mpi_errno, "ser hnd is null"); - - MPIU_SHMW_Hnd_reset_val(hnd); - MPIU_SHMW_Ghnd_set_by_ref(hnd, *ser_hnd_ptr); - - mpi_errno = MPIU_SHMW_Seg_open(hnd, 0); - if(mpi_errno != MPI_SUCCESS) { MPIR_ERR_POP(mpi_errno); } - -fn_exit: - return mpi_errno; -fn_fail: - goto fn_exit; -} - -/* Initialize a shared memory handle - * hnd_ptr : A pointer to the shared memory handle - */ - -#undef FUNCNAME -#define FUNCNAME MPIU_SHMW_Hnd_init -#undef FCNAME -#define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIU_SHMW_Hnd_init( - MPIU_SHMW_Hnd_t *hnd_ptr) -{ - int mpi_errno = MPI_SUCCESS; - int rc = -1; - - MPIR_Assert(hnd_ptr); - - rc = MPIU_SHMW_Hnd_alloc(hnd_ptr); - MPIR_ERR_CHKANDJUMP1((rc != 0), mpi_errno, MPI_ERR_OTHER, - "**nomem", "**nomem %s", "shared mem handle"); - - MPIU_SHMW_Hnd_reset_val(*hnd_ptr); - -fn_exit: - return mpi_errno; -fn_fail: - goto fn_exit; -} - -/* Finalize a shared memory handle. - * hnd_ptr : A pointer to the shm handle to be finalized. - * Any handle that is init has to be finalized. - */ -#undef FUNCNAME -#define FUNCNAME MPIU_SHMW_Hnd_finalize -#undef FCNAME -#define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIU_SHMW_Hnd_finalize( - MPIU_SHMW_Hnd_t *hnd_ptr) -{ - int mpi_errno = MPI_SUCCESS; - - MPIR_Assert(hnd_ptr); - MPIR_Assert(*hnd_ptr); - - /* A finalize can/should be called on an invalid handle - * Don't assert if we fail here ... - */ - MPIU_SHMW_Hnd_close(*hnd_ptr); - MPIU_SHMW_Hnd_free(*hnd_ptr); - - *hnd_ptr = MPIU_SHMW_HND_INVALID; - - return mpi_errno; -} - -#ifdef USE_SYSV_SHM -/************************** Sys V shm **************************/ -/* A template function which creates/attaches shm seg handle - * to the shared memory. Used by user-exposed functions below - */ -#undef FUNCNAME -#define FUNCNAME MPIU_SHMW_Seg_create_attach_templ -#undef FCNAME -#define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIU_SHMW_Seg_create_attach_templ( - MPIU_SHMW_Hnd_t hnd, size_t seg_sz, char **shm_addr_ptr, - int offset, int flag) -{ - int mpi_errno = MPI_SUCCESS; - int rc = -1; - int lhnd = -1; - - if(flag & MPIU_SHMW_FLAG_SHM_CREATE){ - lhnd = shmget(IPC_PRIVATE, seg_sz, IPC_CREAT | S_IRWXU); - /* Return error if SHM seg already exists or create fails */ - MPIR_ERR_CHKANDJUMP2((lhnd == -1) || - (MPIU_OSW_Get_errno() == MPIU_SHMW_SEG_ALREADY_EXISTS), - mpi_errno, MPI_ERR_OTHER, "**alloc_shar_mem", - "**alloc_shar_mem %s %s", "shmget", - MPIU_OSW_Strerror(MPIU_OSW_Get_errno())); - - MPIU_SHMW_Lhnd_set(hnd, lhnd); - - rc = MPIU_SHMW_Ghnd_alloc(hnd); - MPIR_ERR_CHKANDJUMP1((rc != 0), mpi_errno, MPI_ERR_OTHER, - "**nomem", "**nomem %s", "shared mem global handle"); - - rc = MPIU_SHMW_Ghnd_set_by_val(hnd, "%d", lhnd); - MPIR_Assert(rc == 0); - } - else{ - /* Open an existing shared memory seg */ - MPIR_Assert(MPIU_SHMW_Ghnd_is_valid(hnd)); - - if(!MPIU_SHMW_Lhnd_is_valid(hnd)){ - lhnd = atoi(MPIU_SHMW_Ghnd_get_by_ref(hnd)); - MPIR_ERR_CHKANDJUMP((lhnd == -1), mpi_errno, - MPI_ERR_OTHER, "**shmw_badhnd"); - - MPIU_SHMW_Lhnd_set(hnd, lhnd); - } - } - - if(flag & MPIU_SHMW_FLAG_SHM_ATTACH){ - /* Attach to shared mem seg */ - MPIR_Assert(shm_addr_ptr); - - *shm_addr_ptr = shmat(MPIU_SHMW_Lhnd_get(hnd), NULL, 0x0); - - MPIR_ERR_CHKANDJUMP2((*shm_addr_ptr == (void*)(-1)), mpi_errno, MPI_ERR_OTHER, - "**attach_shar_mem", "**attach_shar_mem %s %s", - "shmat", MPIU_OSW_Strerror(MPIU_OSW_Get_errno())); - } - -fn_exit: - return mpi_errno; -fn_fail: - goto fn_exit; -} - -/* Detach from an attached SHM segment - * hnd : Handle to the shm segment - * shm_addr_ptr : Pointer to the shm address to detach - * seg_sz : Size of shm segment - */ -#undef FUNCNAME -#define FUNCNAME MPIU_SHMW_Seg_detach -#undef FCNAME -#define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIU_SHMW_Seg_detach( - MPIU_SHMW_Hnd_t hnd, char **shm_addr_ptr, size_t seg_sz) -{ - int mpi_errno = MPI_SUCCESS; - int rc = -1; - - MPIR_ERR_CHKANDJUMP(!MPIU_SHMW_Hnd_is_valid(hnd), - mpi_errno, MPI_ERR_OTHER, "**shmw_badhnd"); - MPIR_Assert(shm_addr_ptr); - MPIR_ERR_CHKINTERNAL(!(*shm_addr_ptr), mpi_errno, "shm address is null"); - - rc = shmdt(*shm_addr_ptr); - MPIR_ERR_CHKANDJUMP2((rc != 0), mpi_errno, MPI_ERR_OTHER, - "**detach_shar_mem","**detach_shar_mem %s %s", - "shmdt", MPIU_OSW_Strerror(MPIU_OSW_Get_errno())); - *shm_addr_ptr = NULL; - -fn_exit: - return mpi_errno; -fn_fail: - goto fn_exit; -} - -/* Remove a shared memory segment - * hnd : Handle to the shared memory segment to be removed - */ - -#undef FUNCNAME -#define FUNCNAME MPIU_SHMW_Seg_remove -#undef FCNAME -#define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIU_SHMW_Seg_remove(MPIU_SHMW_Hnd_t hnd) -{ - int mpi_errno = MPI_SUCCESS; - struct shmid_ds ds; - int rc = -1; - - MPIR_ERR_CHKANDJUMP(!MPIU_SHMW_Hnd_is_valid(hnd), - mpi_errno, MPI_ERR_OTHER, "**shmw_badhnd"); - - rc = shmctl(MPIU_SHMW_Lhnd_get(hnd), IPC_RMID, &ds); - MPIR_ERR_CHKANDJUMP2((rc != 0), mpi_errno, MPI_ERR_OTHER, - "**remove_shar_mem", "**remove_shar_mem %s %s","shmctl", - MPIU_OSW_Strerror(MPIU_OSW_Get_errno())); - -fn_exit: - return mpi_errno; -fn_fail: - goto fn_exit; -} - -#elif USE_MMAP_SHM -/************************** MMAP shm **************************/ -/* A template function which creates/attaches shm seg handle - * to the shared memory. Used by user-exposed functions below - */ -/* FIXME: Pass (void **)shm_addr_ptr instead of (char **) shm_addr_ptr - * since a util func should be generic - * Currently not passing (void **) to reduce warning in nemesis - * code which passes (char **) ptrs to be attached to a seg - */ - -#undef FUNCNAME -#define FUNCNAME MPIU_SHMW_Seg_create_attach_templ -#undef FCNAME -#define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIU_SHMW_Seg_create_attach_templ( - MPIU_SHMW_Hnd_t hnd, size_t seg_sz, char **shm_addr_ptr, - int offset, int flag) -{ - int mpi_errno = MPI_SUCCESS; - MPIU_SHMW_Lhnd_t lhnd = -1, rc = -1; - - if(flag & MPIU_SHMW_FLAG_SHM_CREATE){ - char dev_shm_fname[] = "/dev/shm/mpich_shar_tmpXXXXXX"; - char tmp_fname[] = "/tmp/mpich_shar_tmpXXXXXX"; - char *chosen_fname = NULL; - - chosen_fname = dev_shm_fname; - lhnd = mkstemp(chosen_fname); - if(lhnd == -1){ - chosen_fname = tmp_fname; - lhnd = mkstemp(chosen_fname); - } - MPIR_ERR_CHKANDJUMP1((lhnd == -1), mpi_errno, - MPI_ERR_OTHER,"**mkstemp","**mkstemp %s", - MPIU_OSW_Strerror(MPIU_OSW_Get_errno())); - - MPIU_SHMW_Lhnd_set(hnd, lhnd); - rc = (MPIU_SHMW_Lhnd_t)lseek(lhnd, seg_sz - 1, SEEK_SET); - MPIR_ERR_CHKANDJUMP1((rc == -1), mpi_errno, - MPI_ERR_OTHER, "**lseek", "**lseek %s", - MPIU_OSW_Strerror(MPIU_OSW_Get_errno())); - - MPIU_OSW_RETRYON_INTR((rc == -1), (rc = (int)write(lhnd, "", 1))); - MPIR_ERR_CHKANDJUMP((rc == -1), mpi_errno, MPI_ERR_OTHER, - "**write"); - - rc = MPIU_SHMW_Ghnd_alloc(hnd); - MPIR_ERR_CHKANDJUMP1((rc != 0),mpi_errno, MPI_ERR_OTHER, - "**nomem", "**nomem %s", "shared memory global handle"); - - rc = MPIU_SHMW_Ghnd_set_by_val(hnd, "%s", chosen_fname); - MPIR_Assert(rc == 0); - } - else{ - /* Open an existing shared memory seg */ - MPIR_Assert(MPIU_SHMW_Ghnd_is_valid(hnd)); - - if(!MPIU_SHMW_Lhnd_is_valid(hnd)){ - lhnd = open(MPIU_SHMW_Ghnd_get_by_ref(hnd), O_RDWR); - MPIR_ERR_CHKANDJUMP1((lhnd == -1), mpi_errno, - MPI_ERR_OTHER, "**open", "**open %s", - MPIU_OSW_Strerror(MPIU_OSW_Get_errno())); - - MPIU_SHMW_Lhnd_set(hnd, lhnd); - } - } - - if(flag & MPIU_SHMW_FLAG_SHM_ATTACH){ - void *buf_ptr = NULL; - - MPIR_Assert(shm_addr_ptr); - - buf_ptr = mmap(NULL, seg_sz, PROT_READ | PROT_WRITE, - MAP_SHARED, MPIU_SHMW_Lhnd_get(hnd), 0); - MPIR_ERR_CHKANDJUMP2((buf_ptr == MAP_FAILED), - mpi_errno, MPI_ERR_OTHER, "**alloc_shar_mem", - "**alloc_shar_mem %s %s", "mmap", - MPIU_OSW_Strerror(MPIU_OSW_Get_errno())); - - *shm_addr_ptr = buf_ptr; - } - -fn_exit: - /* FIXME: Close local handle only when closing the shm handle */ - if(MPIU_SHMW_Lhnd_is_valid(hnd)){ - rc = MPIU_SHMW_Lhnd_close(hnd); - MPIR_Assert(rc == 0); - } - return mpi_errno; -fn_fail: - goto fn_exit; -} - -/* Detach from an attached SHM segment */ -#undef FUNCNAME -#define FUNCNAME MPIU_SHMW_Seg_detach -#undef FCNAME -#define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIU_SHMW_Seg_detach( - MPIU_SHMW_Hnd_t hnd, char **shm_addr_ptr, size_t seg_sz) -{ - int mpi_errno = MPI_SUCCESS; - int rc = -1; - - MPIR_Assert(shm_addr_ptr); - MPIR_ERR_CHKINTERNAL(!(*shm_addr_ptr), mpi_errno, "shm address is null"); - - rc = munmap(*shm_addr_ptr, seg_sz); - MPIR_ERR_CHKANDJUMP2((rc != 0), mpi_errno, MPI_ERR_OTHER, - "**detach_shar_mem", "**detach_shar_mem %s %s","munmap", - MPIU_OSW_Strerror(MPIU_OSW_Get_errno())); - *shm_addr_ptr = NULL; - -fn_exit: - return mpi_errno; -fn_fail: - goto fn_exit; -} - -/* Remove an existing SHM segment */ -static inline int MPIU_SHMW_Seg_remove(MPIU_SHMW_Hnd_t hnd) -{ - int mpi_errno = MPI_SUCCESS; - int rc = -1; - - MPIR_ERR_CHKANDJUMP(!MPIU_SHMW_Ghnd_is_valid(hnd), - mpi_errno, MPI_ERR_OTHER, "**shmw_badhnd"); - - rc = unlink(MPIU_SHMW_Ghnd_get_by_ref(hnd)); - MPIR_ERR_CHKANDJUMP2((rc != 0), mpi_errno, MPI_ERR_OTHER, - "**remove_shar_mem", "**remove_shar_mem %s %s","unlink", - MPIU_OSW_Strerror(MPIU_OSW_Get_errno())); - -fn_exit: - return mpi_errno; -fn_fail: - goto fn_exit; -} - -#elif defined (USE_NT_SHM) -/************************** NT shm **************************/ -/* A template function which creates/attaches shm seg handle - * to the shared memory. Used by user-exposed functions below - */ -#undef FUNCNAME -#define FUNCNAME MPIU_SHMW_Seg_create_attach_templ -#undef FCNAME -#define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIU_SHMW_Seg_create_attach_templ( - MPIU_SHMW_Hnd_t hnd, size_t seg_sz, char **shm_addr_ptr, - int offset, int flag) -{ - int mpi_errno = MPI_SUCCESS; - HANDLE lhnd = INVALID_HANDLE_VALUE; - int rc = -1; - ULARGE_INTEGER seg_sz_large; - seg_sz_large.QuadPart = seg_sz; - - if(!MPIU_SHMW_Ghnd_is_valid(hnd)){ - MPIR_Assert(flag & MPIU_SHMW_FLAG_SHM_CREATE); - - rc = MPIU_SHMW_Ghnd_set_uniq(hnd); - MPIR_ERR_CHKANDJUMP((rc == 0), mpi_errno, MPI_ERR_OTHER, - "**shmw_sethnd"); - } - - if(flag & MPIU_SHMW_FLAG_SHM_CREATE){ - lhnd = CreateFileMapping(INVALID_HANDLE_VALUE, NULL, - PAGE_READWRITE, seg_sz_large.HighPart, seg_sz_large.LowPart, - MPIU_SHMW_Ghnd_get_by_ref(hnd)); - /* Return error if SHM seg already exists or create fails */ - MPIR_ERR_CHKANDJUMP2((lhnd == INVALID_HANDLE_VALUE) || - (MPIU_OSW_Get_errno() == MPIU_SHMW_SEG_ALREADY_EXISTS), - mpi_errno, MPI_ERR_OTHER, "**alloc_shar_mem", - "**alloc_shar_mem %s %s", "CreateFileMapping", - MPIU_OSW_Strerror(MPIU_OSW_Get_errno())); - - MPIU_SHMW_Lhnd_set(hnd, lhnd); - } - else{ - MPIR_Assert(MPIU_SHMW_Ghnd_is_valid(hnd)); - - if(!MPIU_SHMW_Lhnd_is_valid(hnd)){ - /* Strangely OpenFileMapping() returns NULL on error! */ - lhnd = OpenFileMapping(FILE_MAP_WRITE, FALSE, - MPIU_SHMW_Ghnd_get_by_ref(hnd)); - MPIR_ERR_CHKANDJUMP2((lhnd == NULL), mpi_errno, - MPI_ERR_OTHER, "**alloc_shar_mem", - "**alloc_shar_mem %s %s", "OpenFileMapping", - MPIU_OSW_Strerror(MPIU_OSW_Get_errno())); - - MPIU_SHMW_Lhnd_set(hnd, lhnd); - } - } - - if(flag & MPIU_SHMW_FLAG_SHM_ATTACH){ - MPIR_Assert(shm_addr_ptr); - - *shm_addr_ptr = (char *)MapViewOfFile(MPIU_SHMW_Lhnd_get(hnd), - FILE_MAP_WRITE, 0, offset, 0); - MPIR_ERR_CHKANDJUMP2(!(*shm_addr_ptr), mpi_errno, MPI_ERR_OTHER, - "**attach_shar_mem", "**attach_shar_mem %s %s", - "MapViewOfFile", MPIU_OSW_Strerror(MPIU_OSW_Get_errno())); - } - -fn_exit: - return mpi_errno; -fn_fail: - goto fn_exit; -} - -/* Detach from an attached SHM segment */ -#undef FUNCNAME -#define FUNCNAME MPIU_SHMW_Seg_detach -#undef FCNAME -#define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIU_SHMW_Seg_detach( - MPIU_SHMW_Hnd_t hnd, char **shm_addr_ptr, size_t seg_sz) -{ - int mpi_errno = MPI_SUCCESS; - int rc = -1; - - MPL_UNREFERENCED_ARG(seg_sz); - MPIR_ERR_CHKANDJUMP(!MPIU_SHMW_Hnd_is_valid(hnd), - mpi_errno, MPI_ERR_OTHER, "**shmw_badhnd"); - MPIR_Assert(shm_addr_ptr); - MPIR_ERR_CHKINTERNAL(!(*shm_addr_ptr), mpi_errno, "shm address is null"); - - rc = UnmapViewOfFile(*shm_addr_ptr); - MPIR_ERR_CHKANDJUMP2((rc == 0), mpi_errno, MPI_ERR_OTHER, - "**detach_shar_mem", "**detach_shar_mem %s %s","UnmapViewOfFile", - MPIU_OSW_Strerror(MPIU_OSW_Get_errno())); - *shm_addr_ptr = NULL; - -fn_exit: - return mpi_errno; -fn_fail: - goto fn_exit; -} - -/* Nothing to be done when removing an SHM segment */ -#define MPIU_SHMW_Seg_remove(hnd) MPI_SUCCESS - -#endif /* USE_NT_SHM */ - -/* Create new SHM segment - * hnd : A "init"ed shared memory handle - * seg_sz : Size of shared memory segment to be created - */ -#undef FUNCNAME -#define FUNCNAME MPIU_SHMW_Seg_create -#undef FCNAME -#define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIU_SHMW_Seg_create( - MPIU_SHMW_Hnd_t hnd, size_t seg_sz) -{ - int mpi_errno = MPI_SUCCESS; - - MPIR_Assert(MPIU_SHMW_Hnd_is_init(hnd)); - MPIR_Assert(seg_sz > 0); - - mpi_errno = MPIU_SHMW_Seg_create_attach_templ(hnd, - seg_sz, NULL, 0, MPIU_SHMW_FLAG_SHM_CREATE); - if(mpi_errno != MPI_SUCCESS) { MPIR_ERR_POP(mpi_errno); } - -fn_exit: - return mpi_errno; -fn_fail: - goto fn_exit; -} - -/* Open an existing SHM segment - * hnd : A shm handle with a valid global handle - * seg_sz : Size of shared memory segment to open - * Currently only using internally within wrapper funcs - */ -#undef FUNCNAME -#define FUNCNAME MPIU_SHMW_Seg_open -#undef FCNAME -#define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIU_SHMW_Seg_open( - MPIU_SHMW_Hnd_t hnd, size_t seg_sz) -{ - int mpi_errno = MPI_SUCCESS; - - MPIR_Assert(MPIU_SHMW_Hnd_is_init(hnd)); - - mpi_errno = MPIU_SHMW_Seg_create_attach_templ(hnd, seg_sz, - NULL, 0, MPIU_SHMW_FLAG_CLR); - if(mpi_errno != MPI_SUCCESS) { MPIR_ERR_POP(mpi_errno); } - -fn_exit: - return mpi_errno; -fn_fail: - goto fn_exit; -} - -/* Create new SHM segment and attach to it - * hnd : A "init"ed shared mem handle - * seg_sz: Size of shared mem segment - * shm_addr_ptr : Pointer to shared memory address to attach - * the shared mem segment - * offset : Offset to attach the shared memory address to - */ -#undef FUNCNAME -#define FUNCNAME MPIU_SHMW_Seg_create_and_attach -#undef FCNAME -#define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIU_SHMW_Seg_create_and_attach( - MPIU_SHMW_Hnd_t hnd, size_t seg_sz, char **shm_addr_ptr, - int offset) -{ - int mpi_errno = MPI_SUCCESS; - - MPIR_Assert(MPIU_SHMW_Hnd_is_init(hnd)); - MPIR_Assert(seg_sz > 0); - MPIR_Assert(shm_addr_ptr); - - mpi_errno = MPIU_SHMW_Seg_create_attach_templ(hnd, seg_sz, - shm_addr_ptr, offset, MPIU_SHMW_FLAG_SHM_CREATE | - MPIU_SHMW_FLAG_SHM_ATTACH); - if(mpi_errno != MPI_SUCCESS) { MPIR_ERR_POP(mpi_errno); } - -fn_exit: - return mpi_errno; -fn_fail: - goto fn_exit; -} - -/* Attach to an existing SHM segment - * hnd : A "init"ed shared mem handle - * seg_sz: Size of shared mem segment - * shm_addr_ptr : Pointer to shared memory address to attach - * the shared mem segment - * offset : Offset to attach the shared memory address to - */ -#undef FUNCNAME -#define FUNCNAME MPIU_SHMW_Seg_attach -#undef FCNAME -#define FCNAME MPL_QUOTE(FUNCNAME) -static inline int MPIU_SHMW_Seg_attach( - MPIU_SHMW_Hnd_t hnd, size_t seg_sz, char **shm_addr_ptr, - int offset) -{ - int mpi_errno = MPI_SUCCESS; - - MPIR_Assert(MPIU_SHMW_Hnd_is_init(hnd)); - MPIR_Assert(shm_addr_ptr); - - mpi_errno = MPIU_SHMW_Seg_create_attach_templ(hnd, seg_sz, - shm_addr_ptr, offset, MPIU_SHMW_FLAG_SHM_ATTACH); - if(mpi_errno != MPI_SUCCESS) { MPIR_ERR_POP(mpi_errno); } - -fn_exit: - return mpi_errno; -fn_fail: - goto fn_exit; -} - - -#undef FUNCNAME -#undef FCNAME -#endif /* MPIU_SHM_WRAPPERS_H_INCLUDED */ diff --git a/src/util/wrappers/mpiu_util_wrappers.h b/src/util/wrappers/mpiu_util_wrappers.h index 571da6f..04e9a03 100644 --- a/src/util/wrappers/mpiu_util_wrappers.h +++ b/src/util/wrappers/mpiu_util_wrappers.h @@ -40,19 +40,6 @@ # define MPIU_OSW_Strerror(errno) strerror(errno) #endif -#if defined (HAVE_QUERYPERFORMANCECOUNTER) -/* - * Returns size of uniqStr, 0 on error - */ -static inline int MPIU_OSW_Get_uniq_str(char *str, int strlen) -{ - LARGE_INTEGER perfCnt; - QueryPerformanceCounter(&perfCnt); - return(MPL_snprintf(str, strlen, "MPICH_NEM_%d_%I64d", - GetCurrentThreadId(), (perfCnt.QuadPart))); -} -#endif - #ifdef HAVE_WINDOWS_H # define MPIU_OSW_EINTR WSAEINTR # define MPIU_OSW_ENOBUFS WSAENOBUFS http://git.mpich.org/mpich.git/commitdiff/820a98f8289d0e18a77144b7036ca1da3… commit 820a98f8289d0e18a77144b7036ca1da3848461a Author: Yanfei Guo <yguo(a)anl.gov> Date: Thu Apr 14 16:26:33 2016 -0500 fix get kvs name in shm diff --git a/src/mpid/common/shm/mpidu_shm_alloc.c b/src/mpid/common/shm/mpidu_shm_alloc.c index 04f4971..7db7607 100644 --- a/src/mpid/common/shm/mpidu_shm_alloc.c +++ b/src/mpid/common/shm/mpidu_shm_alloc.c @@ -141,7 +141,7 @@ int MPIDU_shm_seg_commit(MPIDU_shm_seg_ptr_t memory, MPIDU_shm_barrier_ptr_t *ba int val_max_sz; char *key; char *val; - char *kvs_name; + char kvs_name[256]; char *serialized_hnd = NULL; void *current_addr; void *start_addr ATTRIBUTE((unused)); @@ -328,8 +328,7 @@ int MPIDU_shm_seg_commit(MPIDU_shm_seg_ptr_t memory, MPIDU_shm_barrier_ptr_t *ba MPIR_ERR_CHKANDJUMP1(pmi_errno, mpi_errno, MPI_ERR_OTHER, "**fail", "**fail %d", pmi_errno); MPIU_CHKLMEM_MALLOC(val, char *, val_max_sz, mpi_errno, "val"); - //mpi_errno = MPIDI_PG_GetConnKVSname (&kvs_name); - kvs_name = "my_kvs_name"; + mpi_errno = PMI_KVS_Get_my_name(kvs_name, 256); if (mpi_errno) MPIR_ERR_POP (mpi_errno); if (local_rank == 0){ http://git.mpich.org/mpich.git/commitdiff/2de8f043d9992e338084c498877107948… commit 2de8f043d9992e338084c49887710794809b2b46 Author: Yanfei Guo <yguo(a)anl.gov> Date: Tue Feb 2 00:48:23 2016 -0600 CH4: use mpid/common/shm The data structures and functions for barrier and SHM seg are provided by mpid/common/shm now. The original definitions in CH4 SHM is removed. diff --git a/src/mpid/common/shm/mpidu_shm_alloc.c b/src/mpid/common/shm/mpidu_shm_alloc.c index d85a40d..04f4971 100644 --- a/src/mpid/common/shm/mpidu_shm_alloc.c +++ b/src/mpid/common/shm/mpidu_shm_alloc.c @@ -3,7 +3,6 @@ * (C) 2006 by Argonne National Laboratory. * See COPYRIGHT in top-level directory. */ - #include <mpidimpl.h> #include "mpiu_shm_wrappers.h" @@ -329,7 +328,8 @@ int MPIDU_shm_seg_commit(MPIDU_shm_seg_ptr_t memory, MPIDU_shm_barrier_ptr_t *ba MPIR_ERR_CHKANDJUMP1(pmi_errno, mpi_errno, MPI_ERR_OTHER, "**fail", "**fail %d", pmi_errno); MPIU_CHKLMEM_MALLOC(val, char *, val_max_sz, mpi_errno, "val"); - mpi_errno = MPIDI_PG_GetConnKVSname (&kvs_name); + //mpi_errno = MPIDI_PG_GetConnKVSname (&kvs_name); + kvs_name = "my_kvs_name"; if (mpi_errno) MPIR_ERR_POP (mpi_errno); if (local_rank == 0){ http://git.mpich.org/mpich.git/commitdiff/998a5d42ee48ad5e5a7048215edeae01b… commit 998a5d42ee48ad5e5a7048215edeae01bf5f27a2 Author: Yanfei Guo <yguo(a)anl.gov> Date: Tue Feb 2 00:27:52 2016 -0600 autotools: add mpid/common/shm subconfigure http://git.mpich.org/mpich.git/commitdiff/bb0e0b7a1e437ad95ee354bfa68b5bd15… commit bb0e0b7a1e437ad95ee354bfa68b5bd15875cb8c Author: Yanfei Guo <yguo(a)anl.gov> Date: Mon Feb 1 20:13:29 2016 -0600 mpid/common/shm: use MPL_sched_yield diff --git a/src/mpid/common/shm/mpidu_shm_barrier.c b/src/mpid/common/shm/mpidu_shm_barrier.c index bfefb26..01d5e50 100644 --- a/src/mpid/common/shm/mpidu_shm_barrier.c +++ b/src/mpid/common/shm/mpidu_shm_barrier.c @@ -61,7 +61,7 @@ int MPIDU_shm_barrier(MPIDU_shm_barrier_t *barrier, int num_local) { /* wait */ while (OPA_load_int(&barrier->wait) == sense) - MPIU_PW_Sched_yield(); /* skip */ + MPL_sched_yield(); /* skip */ } sense = 1 - sense; http://git.mpich.org/mpich.git/commitdiff/e01832a6f01292f4a1ce0fd06c67c43cf… commit e01832a6f01292f4a1ce0fd06c67c43cfccde646 Author: Yanfei Guo <yguo(a)anl.gov> Date: Mon Feb 1 18:27:36 2016 -0600 mpid/common/shm: add subconfigure for shm mpid/common/shm is added as prerequisite of nemesis channel and pamid device. diff --git a/src/mpid/ch3/channels/nemesis/subconfigure.m4 b/src/mpid/ch3/channels/nemesis/subconfigure.m4 index 062698f..3a7c13b 100644 --- a/src/mpid/ch3/channels/nemesis/subconfigure.m4 +++ b/src/mpid/ch3/channels/nemesis/subconfigure.m4 @@ -1,4 +1,5 @@ [#] start of __file__ +dnl MPICH_SUBCFG_BEFORE=src/mpid/common/shm dnl dnl _PREREQ handles the former role of mpichprereq, setup_device, etc AC_DEFUN([PAC_SUBCFG_PREREQ_]PAC_SUBCFG_AUTO_SUFFIX,[ @@ -24,6 +25,8 @@ else fi export nemesis_networks +# the nemesis channel depends on the common shm code +build_mpid_common_shm=yes ])dnl end AM_COND_IF(BUILD_CH3_NEMESIS,...) ])dnl dnl diff --git a/src/mpid/common/shm/Makefile.mk b/src/mpid/common/shm/Makefile.mk index d897eb3..8c044fd 100644 --- a/src/mpid/common/shm/Makefile.mk +++ b/src/mpid/common/shm/Makefile.mk @@ -5,6 +5,8 @@ ## See COPYRIGHT in top-level directory. ## +if BUILD_MPID_COMMON_SHM + mpi_core_sources += \ src/mpid/common/shm/mpidu_shm_alloc.c \ src/mpid/common/shm/mpidu_shm_barrier.c @@ -17,4 +19,4 @@ noinst_HEADERS += \ src/mpid/common/shm/mpidu_shm_impl.h \ src/mpid/common/shm/mpidu_shm.h - +endif BUILD_MPID_COMMON_SHM diff --git a/src/mpid/common/shm/subconfigure.m4 b/src/mpid/common/shm/subconfigure.m4 new file mode 100644 index 0000000..e2fe593 --- /dev/null +++ b/src/mpid/common/shm/subconfigure.m4 @@ -0,0 +1,21 @@ +[#] start of __file__ + +dnl _PREREQ handles the former role of mpichprereq, setup_device, etc +AC_DEFUN([PAC_SUBCFG_PREREQ_]PAC_SUBCFG_AUTO_SUFFIX,[ + AM_CONDITIONAL([BUILD_MPID_COMMON_SHM],[test "X$build_mpid_common_shm" = "Xyes"]) +]) + +dnl _BODY handles the former role of configure in the subsystem +AC_DEFUN([PAC_SUBCFG_BODY_]PAC_SUBCFG_AUTO_SUFFIX,[ + +dnl the code below is currently commented out because the common shm code +dnl doesn't need to run any configure tests, it really just needs to emit the +dnl AM_CONDITIONAL for the moment +dnl +dnl AM_COND_IF([BUILD_MPID_COMMON_SHM],[ +dnl AC_MSG_NOTICE([RUNNING CONFIGURE FOR src/mpid/common/shm]) +dnl +dnl ])dnl end AM_COND_IF(BUILD_MPID_COMMON_SHM,...) +])dnl end _BODY +dnl +[#] end of __file__ diff --git a/src/mpid/pamid/subconfigure.m4 b/src/mpid/pamid/subconfigure.m4 index a353c4e..817b240 100644 --- a/src/mpid/pamid/subconfigure.m4 +++ b/src/mpid/pamid/subconfigure.m4 @@ -20,6 +20,7 @@ dnl -*- mode: makefile-gmake; -*- dnl MPICH_SUBCFG_BEFORE=src/mpid/common/sched dnl MPICH_SUBCFG_BEFORE=src/mpid/common/datatype dnl MPICH_SUBCFG_BEFORE=src/mpid/common/thread +dnl MPICH_SUBCFG_BEFORE=src/mpid/common/shm dnl _PREREQ handles the former role of mpichprereq, setup_device, etc [#] expansion is: PAC_SUBCFG_PREREQ_[]PAC_SUBCFG_AUTO_SUFFIX @@ -64,10 +65,11 @@ fi MPID_MAX_THREAD_LEVEL=MPI_THREAD_MULTIPLE -dnl the PAMID device depends on the common NBC scheduler code +dnl the PAMID device depends on the common NBC scheduler SHM code build_mpid_common_sched=yes build_mpid_common_datatype=yes build_mpid_common_thread=yes +build_mpid_common_shm=yes ])dnl end AM_COND_IF(BUILD_PAMID,...) http://git.mpich.org/mpich.git/commitdiff/9c05f2563d632f9a855c9a78f86aa9132… commit 9c05f2563d632f9a855c9a78f86aa913292f1bd2 Author: Yanfei Guo <yguo(a)anl.gov> Date: Thu Jan 28 18:25:20 2016 -0600 mpid/common/shm: fix namespace MPIDU_Seg_* should be MPIDU_shm_seg_*. diff --git a/src/mpid/ch3/channels/nemesis/src/mpid_nem_finalize.c b/src/mpid/ch3/channels/nemesis/src/mpid_nem_finalize.c index 24b6e5a..0de5b2b 100644 --- a/src/mpid/ch3/channels/nemesis/src/mpid_nem_finalize.c +++ b/src/mpid/ch3/channels/nemesis/src/mpid_nem_finalize.c @@ -50,7 +50,7 @@ int MPID_nem_finalize(void) if (mpi_errno) MPIR_ERR_POP (mpi_errno); /* free the shared memory segment */ - mpi_errno = MPIDU_Seg_destroy(&MPID_nem_mem_region.memory, MPID_nem_mem_region.num_local); + mpi_errno = MPIDU_shm_seg_destroy(&MPID_nem_mem_region.memory, MPID_nem_mem_region.num_local); if (mpi_errno) MPIR_ERR_POP (mpi_errno); #ifdef PAPI_MONITOR diff --git a/src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c b/src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c index c115974..c335461 100644 --- a/src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c +++ b/src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c @@ -238,29 +238,29 @@ MPID_nem_init(int pg_rank, MPIDI_PG_t *pg_p, int has_parent ATTRIBUTE((unused))) #endif /*FORCE_ASYM */ /* Request fastboxes region */ - mpi_errno = MPIDU_Seg_alloc(MPL_MAX((num_local*((num_local-1)*sizeof(MPID_nem_fastbox_t))), MPID_NEM_ASYMM_NULL_VAL), + mpi_errno = MPIDU_shm_seg_alloc(MPL_MAX((num_local*((num_local-1)*sizeof(MPID_nem_fastbox_t))), MPID_NEM_ASYMM_NULL_VAL), (void **)&fastboxes_p); if (mpi_errno) MPIR_ERR_POP(mpi_errno); /* Request data cells region */ - mpi_errno = MPIDU_Seg_alloc(num_local * MPID_NEM_NUM_CELLS * sizeof(MPID_nem_cell_t), (void **)&cells_p); + mpi_errno = MPIDU_shm_seg_alloc(num_local * MPID_NEM_NUM_CELLS * sizeof(MPID_nem_cell_t), (void **)&cells_p); if (mpi_errno) MPIR_ERR_POP(mpi_errno); /* Request free q region */ - mpi_errno = MPIDU_Seg_alloc(num_local * sizeof(MPID_nem_queue_t), (void **)&free_queues_p); + mpi_errno = MPIDU_shm_seg_alloc(num_local * sizeof(MPID_nem_queue_t), (void **)&free_queues_p); if (mpi_errno) MPIR_ERR_POP(mpi_errno); /* Request recv q region */ - mpi_errno = MPIDU_Seg_alloc(num_local * sizeof(MPID_nem_queue_t), (void **)&recv_queues_p); + mpi_errno = MPIDU_shm_seg_alloc(num_local * sizeof(MPID_nem_queue_t), (void **)&recv_queues_p); if (mpi_errno) MPIR_ERR_POP(mpi_errno); /* Request shared collectives barrier vars region */ - mpi_errno = MPIDU_Seg_alloc(MPID_NEM_NUM_BARRIER_VARS * sizeof(MPID_nem_barrier_vars_t), + mpi_errno = MPIDU_shm_seg_alloc(MPID_NEM_NUM_BARRIER_VARS * sizeof(MPID_nem_barrier_vars_t), (void **)&MPID_nem_mem_region.barrier_vars); if (mpi_errno) MPIR_ERR_POP(mpi_errno); /* Actually allocate the segment and assign regions to the pointers */ - mpi_errno = MPIDU_Seg_commit(&MPID_nem_mem_region.memory, &MPID_nem_mem_region.barrier, + mpi_errno = MPIDU_shm_seg_commit(&MPID_nem_mem_region.memory, &MPID_nem_mem_region.barrier, num_local, local_rank, MPID_nem_mem_region.local_procs[0], MPID_nem_mem_region.rank); /* check_alloc steps */ diff --git a/src/mpid/common/shm/mpidu_shm.h b/src/mpid/common/shm/mpidu_shm.h index 74d9c8c..8558a5f 100644 --- a/src/mpid/common/shm/mpidu_shm.h +++ b/src/mpid/common/shm/mpidu_shm.h @@ -32,10 +32,10 @@ typedef struct MPIDU_shm_seg_info char *addr; } MPIDU_shm_seg_info_t, *MPIDU_shm_seg_info_ptr_t; -int MPIDU_Seg_alloc(size_t len, void **ptr_p); -int MPIDU_Seg_commit(MPIDU_shm_seg_ptr_t memory, MPIDU_shm_barrier_ptr_t *barrier, +int MPIDU_shm_seg_alloc(size_t len, void **ptr_p); +int MPIDU_shm_seg_commit(MPIDU_shm_seg_ptr_t memory, MPIDU_shm_barrier_ptr_t *barrier, int num_local, int local_rank, int local_procs_0, int rank); -int MPIDU_Seg_destroy(MPIDU_shm_seg_ptr_t memory, int num_local); +int MPIDU_shm_seg_destroy(MPIDU_shm_seg_ptr_t memory, int num_local); int MPIDU_shm_barrier_init(MPIDU_shm_barrier_t *barrier_region, MPIDU_shm_barrier_ptr_t *barrier, int init_values); diff --git a/src/mpid/common/shm/mpidu_shm_alloc.c b/src/mpid/common/shm/mpidu_shm_alloc.c index 45a3d35..d85a40d 100644 --- a/src/mpid/common/shm/mpidu_shm_alloc.c +++ b/src/mpid/common/shm/mpidu_shm_alloc.c @@ -59,7 +59,7 @@ typedef struct asym_check_region static asym_check_region* asym_check_region_p = NULL; -/* MPIDU_Seg_alloc(len, ptr_p) +/* MPIDU_shm_seg_alloc(len, ptr_p) This function is used to allow the caller to reserve a len sized region in the shared memory segment. Once the shared memory @@ -72,10 +72,10 @@ static asym_check_region* asym_check_region_p = NULL; MPIDU_SHM_Seg_commit() is called. */ #undef FUNCNAME -#define FUNCNAME MPIDU_Seg_alloc +#define FUNCNAME MPIDU_shm_seg_alloc #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -int MPIDU_Seg_alloc(size_t len, void **ptr_p) +int MPIDU_shm_seg_alloc(size_t len, void **ptr_p) { int mpi_errno = MPI_SUCCESS; alloc_elem_t *ep; @@ -109,11 +109,11 @@ int MPIDU_Seg_alloc(size_t len, void **ptr_p) goto fn_exit; } -/* MPIDU_Seg_commit(memory, num_local, local_rank) +/* MPIDU_shm_seg_commit(memory, num_local, local_rank) This function allocates a shared memory segment large enough to hold all of the regions previously requested by calls to - MPIDU_Seg_alloc(). For each request, this function sets the + MPIDU_shm_seg_alloc(). For each request, this function sets the associated pointer to point to the reserved region in the allocated shared memory segment. @@ -125,10 +125,10 @@ int MPIDU_Seg_alloc(size_t len, void **ptr_p) calling MPIDU_SHM_Seg_commit(). */ #undef FUNCNAME -#define FUNCNAME MPIDU_Seg_commit +#define FUNCNAME MPIDU_shm_seg_commit #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -int MPIDU_Seg_commit(MPIDU_shm_seg_ptr_t memory, MPIDU_shm_barrier_ptr_t *barrier, +int MPIDU_shm_seg_commit(MPIDU_shm_seg_ptr_t memory, MPIDU_shm_barrier_ptr_t *barrier, int num_local, int local_rank, int local_procs_0, int rank) { int mpi_errno = MPI_SUCCESS; @@ -158,7 +158,7 @@ int MPIDU_Seg_commit(MPIDU_shm_seg_ptr_t memory, MPIDU_shm_barrier_ptr_t *barrie MPIU_Assert(segment_len > 0); /* allocate an area to check if the segment was allocated symmetrically */ - mpi_errno = MPIDU_Seg_alloc(sizeof(asym_check_region), (void **) &asym_check_region_p); + mpi_errno = MPIDU_shm_seg_alloc(sizeof(asym_check_region), (void **) &asym_check_region_p); if (mpi_errno) MPIR_ERR_POP(mpi_errno); mpi_errno = MPIU_SHMW_Hnd_init(&(memory->hnd)); @@ -455,10 +455,10 @@ int MPIDU_Seg_commit(MPIDU_shm_seg_ptr_t memory, MPIDU_shm_barrier_ptr_t *barrie /* MPIDU_SHM_Seg_destroy() free the shared memory segment */ #undef FUNCNAME -#define FUNCNAME MPIDU_Seg_destroy +#define FUNCNAME MPIDU_shm_seg_destroy #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -int MPIDU_Seg_destroy(MPIDU_shm_seg_ptr_t memory, int num_local) +int MPIDU_shm_seg_destroy(MPIDU_shm_seg_ptr_t memory, int num_local) { int mpi_errno = MPI_SUCCESS; MPIDI_STATE_DECL(MPID_STATE_MPIDU_SHM_SEG_DESTROY); http://git.mpich.org/mpich.git/commitdiff/743a5ca7420f918503ff46f2737e92833… commit 743a5ca7420f918503ff46f2737e92833aeeee4a Author: Yanfei Guo <yguo(a)anl.gov> Date: Thu Jan 28 14:33:50 2016 -0600 mpid/common/shm: remove dependency on CH3 Moving MPID_nem_barrier and MPID_nem_seg from CH3 to MPIDU_shm. Revising interface to make it work with CH3. diff --git a/src/mpid/ch3/channels/nemesis/include/mpid_nem_defs.h b/src/mpid/ch3/channels/nemesis/include/mpid_nem_defs.h index aabf033..f3fee81 100644 --- a/src/mpid/ch3/channels/nemesis/include/mpid_nem_defs.h +++ b/src/mpid/ch3/channels/nemesis/include/mpid_nem_defs.h @@ -11,8 +11,7 @@ #include "mpi.h" #include "mpiimpl.h" #include "mpiu_os_wrappers_pre.h" - -#define MPID_NEM_MAX_FNAME_LEN 256 +#include "mpidu_shm.h" /* FIXME: This definition should be gotten from mpidi_ch3_impl.h */ #ifndef MAX_HOSTNAME_LEN @@ -60,32 +59,6 @@ static inline MPID_nem_cell_rel_ptr_t MPID_NEM_ABS_TO_REL (MPID_nem_cell_ptr_t a #define MPID_NEM_ABS_TO_REL(ptr) (ptr) #endif /*MPID_NEM_SYMMETRIC_QUEUES */ -typedef struct MPID_nem_barrier -{ - OPA_int_t val; - OPA_int_t wait; -} -MPID_nem_barrier_t; - -typedef struct MPID_nem_seg -{ - size_t segment_len; - /* Handle to shm seg */ - MPIU_SHMW_Hnd_t hnd; - /* Pointers */ - char *base_addr; - /* Misc */ - char file_name[MPID_NEM_MAX_FNAME_LEN]; - int base_descs; - int symmetrical; -} MPID_nem_seg_t, *MPID_nem_seg_ptr_t; - -typedef struct MPID_nem_seg_info -{ - size_t size; - char *addr; -} MPID_nem_seg_info_t, *MPID_nem_seg_info_ptr_t; - /* NOTE: MPID_NEM_IS_LOCAL should only be used when the process is known to be in your comm_world (such as at init time). This will generally not work for dynamic processes. Check vc_ch->is_local instead. If that is true, then @@ -111,8 +84,8 @@ MPID_nem_barrier_vars_t; typedef struct MPID_nem_mem_region { - MPID_nem_seg_t memory; - MPID_nem_seg_info_t *seg; + MPIDU_shm_seg_t memory; + MPIDU_shm_seg_info_t *seg; int num_seg; int map_lock; int num_local; @@ -126,7 +99,7 @@ typedef struct MPID_nem_mem_region MPID_nem_cell_ptr_t Elements; MPID_nem_queue_ptr_t *FreeQ; MPID_nem_queue_ptr_t *RecvQ; - MPID_nem_barrier_t *barrier; + MPIDU_shm_barrier_t *barrier; MPID_nem_queue_ptr_t my_freeQ; MPID_nem_queue_ptr_t my_recvQ; MPID_nem_barrier_vars_t *barrier_vars; diff --git a/src/mpid/ch3/channels/nemesis/include/mpid_nem_impl.h b/src/mpid/ch3/channels/nemesis/include/mpid_nem_impl.h index 521048e..f16eabb 100644 --- a/src/mpid/ch3/channels/nemesis/include/mpid_nem_impl.h +++ b/src/mpid/ch3/channels/nemesis/include/mpid_nem_impl.h @@ -19,10 +19,6 @@ #define MPID_NEM__BYPASS_Q_MAX_VAL ((MPID_NEM_MPICH_DATA_LEN) - (sizeof(MPIDI_CH3_Pkt_t))) -int MPIDU_Seg_alloc(size_t len, void **ptr_p); -int MPIDU_Seg_commit(MPID_nem_seg_ptr_t memory, int num_local, int local_rank); -int MPIDU_Seg_destroy(void); -int MPID_nem_check_alloc(int); int MPID_nem_mpich_init(void); int MPID_nem_coll_init (void); int MPID_nem_send_iov(MPIDI_VC_t *vc, MPIR_Request **sreq_ptr, MPL_IOV *iov, int n_iov); diff --git a/src/mpid/ch3/channels/nemesis/include/mpid_nem_post.h b/src/mpid/ch3/channels/nemesis/include/mpid_nem_post.h index f8dd2d4..a30d5fa 100644 --- a/src/mpid/ch3/channels/nemesis/include/mpid_nem_post.h +++ b/src/mpid/ch3/channels/nemesis/include/mpid_nem_post.h @@ -7,6 +7,8 @@ #ifndef MPID_NEM_POST_H #define MPID_NEM_POST_H +#include "mpidu_shm.h" + struct MPIDI_PG; union MPIDI_CH3_Pkt; @@ -19,8 +21,6 @@ extern int (*MPID_nem_local_lmt_progress)(void); will happen if this is included in mpidpost.h or mpidpre.h) */ int MPID_nem_init(int rank, struct MPIDI_PG *pg_p, int has_parent); int MPID_nem_finalize(void); -int MPIDU_shm_barrier_init(MPID_nem_barrier_t *barrier_region, int init_values); -int MPIDU_shm_barrier(void); int MPID_nem_vc_init(struct MPIDI_VC *vc); int MPID_nem_vc_destroy(struct MPIDI_VC *vc); int MPID_nem_get_business_card(int myRank, char *value, int length); diff --git a/src/mpid/ch3/channels/nemesis/src/mpid_nem_ckpt.c b/src/mpid/ch3/channels/nemesis/src/mpid_nem_ckpt.c index 1b1fce5..7621cb7 100644 --- a/src/mpid/ch3/channels/nemesis/src/mpid_nem_ckpt.c +++ b/src/mpid/ch3/channels/nemesis/src/mpid_nem_ckpt.c @@ -480,7 +480,7 @@ int MPIDI_nem_ckpt_finish(void) channels between local procs), we don't have to flush those channels, just make sure no one is sending or receiving during the checkpoint */ - mpi_errno = MPIDU_shm_barrier(); + mpi_errno = MPIDU_shm_barrier(MPID_nem_mem_region.barrier, MPID_nem_mem_region.num_local); if (mpi_errno) MPIR_ERR_POP(mpi_errno); do { @@ -493,7 +493,7 @@ int MPIDI_nem_ckpt_finish(void) } while (ret == -1 && errno == EINTR); MPIR_ERR_CHKANDJUMP1(ret, mpi_errno, MPI_ERR_OTHER, "**sem_wait", "**sem_wait %s", MPIR_Strerror(errno)); - mpi_errno = MPIDU_shm_barrier(); + mpi_errno = MPIDU_shm_barrier(MPID_nem_mem_region.barrier, MPID_nem_mem_region.num_local); if (mpi_errno) MPIR_ERR_POP(mpi_errno); if (ckpt_result == CKPT_CONTINUE) { diff --git a/src/mpid/ch3/channels/nemesis/src/mpid_nem_finalize.c b/src/mpid/ch3/channels/nemesis/src/mpid_nem_finalize.c index 2d38d1c..24b6e5a 100644 --- a/src/mpid/ch3/channels/nemesis/src/mpid_nem_finalize.c +++ b/src/mpid/ch3/channels/nemesis/src/mpid_nem_finalize.c @@ -50,7 +50,7 @@ int MPID_nem_finalize(void) if (mpi_errno) MPIR_ERR_POP (mpi_errno); /* free the shared memory segment */ - mpi_errno = MPIDU_Seg_destroy(); + mpi_errno = MPIDU_Seg_destroy(&MPID_nem_mem_region.memory, MPID_nem_mem_region.num_local); if (mpi_errno) MPIR_ERR_POP (mpi_errno); #ifdef PAPI_MONITOR diff --git a/src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c b/src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c index 1c4d45a..c115974 100644 --- a/src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c +++ b/src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c @@ -167,7 +167,7 @@ MPID_nem_init(int pg_rank, MPIDI_PG_t *pg_p, int has_parent ATTRIBUTE((unused))) #endif /* MEM_REGION_IN_HEAP */ MPID_nem_mem_region.num_seg = 7; - MPIR_CHKPMEM_MALLOC (MPID_nem_mem_region.seg, MPID_nem_seg_info_ptr_t, MPID_nem_mem_region.num_seg * sizeof(MPID_nem_seg_info_t), mpi_errno, "mem_region segments"); + MPIR_CHKPMEM_MALLOC (MPID_nem_mem_region.seg, MPIDU_shm_seg_info_ptr_t, MPID_nem_mem_region.num_seg * sizeof(MPIDU_shm_seg_info_t), mpi_errno, "mem_region segments"); MPID_nem_mem_region.rank = pg_rank; MPID_nem_mem_region.num_local = num_local; MPID_nem_mem_region.num_procs = num_procs; @@ -260,7 +260,19 @@ MPID_nem_init(int pg_rank, MPIDI_PG_t *pg_p, int has_parent ATTRIBUTE((unused))) if (mpi_errno) MPIR_ERR_POP(mpi_errno); /* Actually allocate the segment and assign regions to the pointers */ - mpi_errno = MPIDU_Seg_commit(&MPID_nem_mem_region.memory, num_local, local_rank); + mpi_errno = MPIDU_Seg_commit(&MPID_nem_mem_region.memory, &MPID_nem_mem_region.barrier, + num_local, local_rank, MPID_nem_mem_region.local_procs[0], + MPID_nem_mem_region.rank); + /* check_alloc steps */ + if (MPID_nem_mem_region.memory.symmetrical == 1) { + MPID_nem_asymm_base_addr = NULL; + } else { + MPID_nem_asymm_base_addr = MPID_nem_mem_region.memory.base_addr; +#ifdef MPID_NEM_SYMMETRIC_QUEUES + MPIR_ERR_INTERNALANDJUMP(mpi_errno, "queues are not symmetrically allocated as expected"); +#endif + } + if (mpi_errno) MPIR_ERR_POP (mpi_errno); /* init shared collectives barrier region */ @@ -268,7 +280,7 @@ MPID_nem_init(int pg_rank, MPIDI_PG_t *pg_p, int has_parent ATTRIBUTE((unused))) if (mpi_errno) MPIR_ERR_POP (mpi_errno); /* local procs barrier */ - mpi_errno = MPIDU_shm_barrier(); + mpi_errno = MPIDU_shm_barrier(MPID_nem_mem_region.barrier, num_local); if (mpi_errno) MPIR_ERR_POP (mpi_errno); /* find our cell region */ @@ -343,7 +355,7 @@ MPID_nem_init(int pg_rank, MPIDI_PG_t *pg_p, int has_parent ATTRIBUTE((unused))) /* local barrier */ - mpi_errno = MPIDU_shm_barrier(); + mpi_errno = MPIDU_shm_barrier(MPID_nem_mem_region.barrier, num_local); if (mpi_errno) MPIR_ERR_POP(mpi_errno); @@ -394,11 +406,11 @@ MPID_nem_init(int pg_rank, MPIDI_PG_t *pg_p, int has_parent ATTRIBUTE((unused))) MPL_free(publish_bc_orig); - mpi_errno = MPIDU_shm_barrier(); + mpi_errno = MPIDU_shm_barrier(MPID_nem_mem_region.barrier, num_local); if (mpi_errno) MPIR_ERR_POP(mpi_errno); mpi_errno = MPID_nem_mpich_init(); if (mpi_errno) MPIR_ERR_POP(mpi_errno); - mpi_errno = MPIDU_shm_barrier(); + mpi_errno = MPIDU_shm_barrier(MPID_nem_mem_region.barrier, num_local); if (mpi_errno) MPIR_ERR_POP(mpi_errno); #ifdef ENABLE_CHECKPOINTING mpi_errno = MPIDI_nem_ckpt_init(); diff --git a/src/mpid/common/shm/Makefile.mk b/src/mpid/common/shm/Makefile.mk index 4f50dad..d897eb3 100644 --- a/src/mpid/common/shm/Makefile.mk +++ b/src/mpid/common/shm/Makefile.mk @@ -13,6 +13,8 @@ mpi_core_sources += \ AM_CPPFLAGS += -I$(top_srcdir)/src/mpid/common/shm noinst_HEADERS += \ - src/mpid/common/shm/mpidu_generic_queue.h + src/mpid/common/shm/mpidu_generic_queue.h \ + src/mpid/common/shm/mpidu_shm_impl.h \ + src/mpid/common/shm/mpidu_shm.h diff --git a/src/mpid/common/shm/mpidu_generic_queue.h b/src/mpid/common/shm/mpidu_generic_queue.h index c6a01fc..9332214 100644 --- a/src/mpid/common/shm/mpidu_generic_queue.h +++ b/src/mpid/common/shm/mpidu_generic_queue.h @@ -4,8 +4,8 @@ * See COPYRIGHT in top-level directory. */ -#ifndef MPID_NEM_GENERIC_QUEUE_H -#define MPID_NEM_GENERIC_QUEUE_H +#ifndef MPIDU_SHM_GENERIC_QUEUE_H +#define MPIDU_SHM_GENERIC_QUEUE_H /* Generic queue macros -- "next_field" should be set to the name of the next pointer field in the element (e.g., "ch.tcp_sendq_next") */ @@ -336,4 +336,4 @@ *(epp) = (sp)->top; \ (sp)->top = (*(epp))->next_field; \ } while (0) -#endif /* MPID_NEM_GENERIC_QUEUE_H */ +#endif /* MPIDU_SHM_GENERIC_QUEUE_H */ diff --git a/src/mpid/common/shm/mpidu_shm.h b/src/mpid/common/shm/mpidu_shm.h new file mode 100644 index 0000000..74d9c8c --- /dev/null +++ b/src/mpid/common/shm/mpidu_shm.h @@ -0,0 +1,45 @@ + +#ifndef MPIDU_SHM_H +#define MPIDU_SHM_H + +#include "mpiutil.h" + +#define MPIDU_SHM_MAX_FNAME_LEN 256 +#define MPIDU_SHM_CACHE_LINE_LEN 64 + +typedef struct MPIDU_shm_barrier +{ + OPA_int_t val; + OPA_int_t wait; +} MPIDU_shm_barrier_t, *MPIDU_shm_barrier_ptr_t; + +typedef struct MPIDU_shm_seg +{ + MPIU_Size_t segment_len; + /* Handle to shm seg */ + MPIU_SHMW_Hnd_t hnd; + /* Pointers */ + char *base_addr; + /* Misc */ + char file_name[MPIDU_SHM_MAX_FNAME_LEN]; + int base_descs; + int symmetrical; +} MPIDU_shm_seg_t, *MPIDU_shm_seg_ptr_t; + +typedef struct MPIDU_shm_seg_info +{ + size_t size; + char *addr; +} MPIDU_shm_seg_info_t, *MPIDU_shm_seg_info_ptr_t; + +int MPIDU_Seg_alloc(size_t len, void **ptr_p); +int MPIDU_Seg_commit(MPIDU_shm_seg_ptr_t memory, MPIDU_shm_barrier_ptr_t *barrier, + int num_local, int local_rank, int local_procs_0, int rank); +int MPIDU_Seg_destroy(MPIDU_shm_seg_ptr_t memory, int num_local); + +int MPIDU_shm_barrier_init(MPIDU_shm_barrier_t *barrier_region, + MPIDU_shm_barrier_ptr_t *barrier, int init_values); +int MPIDU_shm_barrier(MPIDU_shm_barrier_t *barrier, int num_local); + +#endif /* MPIDU_SHM_H */ +/* vim: ft=c */ diff --git a/src/mpid/common/shm/mpidu_shm_alloc.c b/src/mpid/common/shm/mpidu_shm_alloc.c index b927745..45a3d35 100644 --- a/src/mpid/common/shm/mpidu_shm_alloc.c +++ b/src/mpid/common/shm/mpidu_shm_alloc.c @@ -28,48 +28,48 @@ extern int mkstemp(char *t); #endif +#include "mpidu_shm_impl.h" #include "mpidu_generic_queue.h" - -static int check_alloc(int num_local, int local_rank); - typedef struct alloc_elem { struct alloc_elem *next; void **ptr_p; size_t len; } alloc_elem_t; -static struct { - alloc_elem_t *head, *tail; -} allocq = { 0 }; +static struct { alloc_elem_t *head, *tail; } allocq = {0}; + +static int check_alloc(MPIDU_shm_seg_ptr_t memory, MPIDU_shm_barrier_ptr_t barrier, + int num_local, int local_rank); #define ALLOCQ_HEAD() GENERIC_Q_HEAD(allocq) #define ALLOCQ_EMPTY() GENERIC_Q_EMPTY(allocq) #define ALLOCQ_ENQUEUE(ep) GENERIC_Q_ENQUEUE(&allocq, ep, next) #define ALLOCQ_DEQUEUE(epp) GENERIC_Q_DEQUEUE(&allocq, epp, next) -#define ROUND_UP_8(x) (((x) + (size_t)7) & ~(size_t)7) /* rounds up to multiple of 8 */ +#define ROUND_UP_8(x) (((x) + (size_t)7) & ~(size_t)7) /* rounds up to multiple of 8 */ static size_t segment_len = 0; -typedef struct asym_check_region { +typedef struct asym_check_region +{ void *base_ptr; OPA_int_t is_asym; } asym_check_region; -static asym_check_region *asym_check_region_p = NULL; +static asym_check_region* asym_check_region_p = NULL; /* MPIDU_Seg_alloc(len, ptr_p) This function is used to allow the caller to reserve a len sized region in the shared memory segment. Once the shared memory - segment is actually allocated, when MPIDI_CH3I_Seg_commit() is + segment is actually allocated, when MPIDU_SHM_Seg_commit() is called, the pointer *ptr_p will be set to point to the reserved region in the shared memory segment. Note that no shared memory is actually allocated by this function, and the *ptr_p pointer will be valid only after - MPIDI_CH3I_Seg_commit() is called. + MPIDU_SHM_Seg_commit() is called. */ #undef FUNCNAME #define FUNCNAME MPIDU_Seg_alloc @@ -80,29 +80,29 @@ int MPIDU_Seg_alloc(size_t len, void **ptr_p) int mpi_errno = MPI_SUCCESS; alloc_elem_t *ep; MPIU_CHKPMEM_DECL(1); - MPIDI_STATE_DECL(MPID_STATE_MPIDI_CH3I_SEG_ALLOC); + MPIDI_STATE_DECL(MPID_STATE_MPIDU_SHM_SEG_ALLOC); - MPIDI_FUNC_ENTER(MPID_STATE_MPIDI_CH3I_SEG_ALLOC); + MPIDI_FUNC_ENTER(MPID_STATE_MPIDU_SHM_SEG_ALLOC); /* round up to multiple of 8 to ensure the start of the next - * region is 64-bit aligned. */ + region is 64-bit aligned. */ len = ROUND_UP_8(len); MPIU_Assert(len); MPIU_Assert(ptr_p); MPIU_CHKPMEM_MALLOC(ep, alloc_elem_t *, sizeof(alloc_elem_t), mpi_errno, "el"); - + ep->ptr_p = ptr_p; ep->len = len; ALLOCQ_ENQUEUE(ep); segment_len += len; - - fn_exit: + + fn_exit: MPIU_CHKPMEM_COMMIT(); - MPIDI_FUNC_EXIT(MPID_STATE_MPIDI_CH3I_SEG_ALLOC); + MPIDI_FUNC_EXIT(MPID_STATE_MPIDU_SHM_SEG_ALLOC); return mpi_errno; fn_fail: MPIU_CHKPMEM_REAP(); @@ -121,14 +121,15 @@ int MPIDU_Seg_alloc(size_t len, void **ptr_p) memory region is not allocated. Instead, memory is allocated from the heap. - At least one call to MPIDI_CH3I_Seg_alloc() must be made before - calling MPIDI_CH3I_Seg_commit(). + At least one call to MPIDU_SHM_Seg_alloc() must be made before + calling MPIDU_SHM_Seg_commit(). */ #undef FUNCNAME #define FUNCNAME MPIDU_Seg_commit #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -int MPIDU_Seg_commit(MPID_nem_seg_ptr_t memory, int num_local, int local_rank) +int MPIDU_Seg_commit(MPIDU_shm_seg_ptr_t memory, MPIDU_shm_barrier_ptr_t *barrier, + int num_local, int local_rank, int local_procs_0, int rank) { int mpi_errno = MPI_SUCCESS; int pmi_errno; @@ -148,36 +149,34 @@ int MPIDU_Seg_commit(MPID_nem_seg_ptr_t memory, int num_local, int local_rank) size_t size_left; MPIU_CHKPMEM_DECL(1); MPIU_CHKLMEM_DECL(2); - MPIDI_STATE_DECL(MPID_STATE_MPIDI_CH3I_SEG_COMMIT); + MPIDI_STATE_DECL(MPID_STATE_MPIDU_SHM_SEG_COMMIT); - MPIDI_FUNC_ENTER(MPID_STATE_MPIDI_CH3I_SEG_COMMIT); + MPIDI_FUNC_ENTER(MPID_STATE_MPIDU_SHM_SEG_COMMIT); - /* MPIDI_CH3I_Seg_alloc() needs to have been called before this function */ + /* MPIDU_SHM_Seg_alloc() needs to have been called before this function */ MPIU_Assert(!ALLOCQ_EMPTY()); MPIU_Assert(segment_len > 0); /* allocate an area to check if the segment was allocated symmetrically */ mpi_errno = MPIDU_Seg_alloc(sizeof(asym_check_region), (void **) &asym_check_region_p); - if (mpi_errno) - MPIR_ERR_POP(mpi_errno); + if (mpi_errno) MPIR_ERR_POP(mpi_errno); mpi_errno = MPIU_SHMW_Hnd_init(&(memory->hnd)); - if (mpi_errno != MPI_SUCCESS) - MPIR_ERR_POP(mpi_errno); + if (mpi_errno != MPI_SUCCESS) MPIR_ERR_POP (mpi_errno); /* Shared memory barrier variables are in the front of the shared - * memory region. We do this here explicitly, rather than use the - * Seg_alloc() function because we need to use the barrier inside - * this function, before we've assigned the variables to their - * regions. To do this, we add extra space to the segment_len, - * initialize the variables as soon as the shared memory region is - * allocated/attached, then before we do the assignments of the - * pointers provided in Seg_alloc(), we make sure to skip the - * region containing the barrier vars. */ - + memory region. We do this here explicitly, rather than use the + Seg_alloc() function because we need to use the barrier inside + this function, before we've assigned the variables to their + regions. To do this, we add extra space to the segment_len, + initialize the variables as soon as the shared memory region is + allocated/attached, then before we do the assignments of the + pointers provided in Seg_alloc(), we make sure to skip the + region containing the barrier vars. */ + /* add space for local barrier region. Use a whole cacheline. */ - MPIU_Assert(MPID_NEM_CACHE_LINE_LEN >= sizeof(MPID_nem_barrier_t)); - segment_len += MPID_NEM_CACHE_LINE_LEN; + MPIU_Assert(MPIDU_SHM_CACHE_LINE_LEN >= sizeof(MPIDU_shm_barrier_t)); + segment_len += MPIDU_SHM_CACHE_LINE_LEN; #ifdef OPA_USE_LOCK_BASED_PRIMITIVES /* We have a similar bootstrapping problem when using OpenPA in @@ -187,76 +186,68 @@ int MPIDU_Seg_commit(MPID_nem_seg_ptr_t memory, int num_local, int local_rank) * right after the barrier var space. */ /* offset from memory->base_addr to the start of ipc_lock */ - ipc_lock_offset = MPID_NEM_CACHE_LINE_LEN; + ipc_lock_offset = MPIDU_SHM_CACHE_LINE_LEN; MPIU_Assert(ipc_lock_offset >= sizeof(OPA_emulation_ipl_t)); - segment_len += MPID_NEM_CACHE_LINE_LEN; + segment_len += MPIDU_SHM_CACHE_LINE_LEN; #endif memory->segment_len = segment_len; #ifdef USE_PMI2_API /* if there is only one process on this processor, don't use shared memory */ - if (num_local == 1) { + if (num_local == 1) + { char *addr; - MPIU_CHKPMEM_MALLOC(addr, char *, segment_len + MPID_NEM_CACHE_LINE_LEN, mpi_errno, - "segment"); + MPIU_CHKPMEM_MALLOC(addr, char *, segment_len + MPIDU_SHM_CACHE_LINE_LEN, mpi_errno, "segment"); memory->base_addr = addr; current_addr = - (char *) (((MPIU_Upint) addr + (MPIU_Upint) MPID_NEM_CACHE_LINE_LEN - 1) & - (~((MPIU_Upint) MPID_NEM_CACHE_LINE_LEN - 1))); + (char *) (((MPIU_Upint) addr + (MPIU_Upint) MPIDU_SHM_CACHE_LINE_LEN - 1) & + (~((MPIU_Upint) MPIDU_SHM_CACHE_LINE_LEN - 1))); memory->symmetrical = 0; /* must come before barrier_init since we use OPA in that function */ #ifdef OPA_USE_LOCK_BASED_PRIMITIVES - ipc_lock = (OPA_emulation_ipl_t *) ((char *) memory->base_addr + ipc_lock_offset); - ret = OPA_Interprocess_lock_init(ipc_lock, TRUE /*isLeader */); + ipc_lock = (OPA_emulation_ipl_t *)((char *)memory->base_addr + ipc_lock_offset); + ret = OPA_Interprocess_lock_init(ipc_lock, TRUE/*isLeader*/); MPIR_ERR_CHKANDJUMP1(ret != 0, mpi_errno, MPI_ERR_OTHER, "**fail", "**fail %d", ret); #endif - mpi_errno = MPIDU_shm_barrier_init((MPID_nem_barrier_t *) memory->base_addr, TRUE); - if (mpi_errno) - MPIR_ERR_POP(mpi_errno); + mpi_errno = MPIDU_shm_barrier_init((MPIDU_shm_barrier_t *) memory->base_addr, barrier, TRUE); + if (mpi_errno) MPIR_ERR_POP(mpi_errno); } else { if (local_rank == 0) { - mpi_errno = - MPIU_SHMW_Seg_create_and_attach(memory->hnd, memory->segment_len, - &(memory->base_addr), 0); - if (mpi_errno) - MPIR_ERR_POP(mpi_errno); + mpi_errno = MPIU_SHMW_Seg_create_and_attach(memory->hnd, memory->segment_len, &(memory->base_addr), 0); + if (mpi_errno) MPIR_ERR_POP(mpi_errno); /* post name of shared file */ - MPIU_Assert(MPID_nem_mem_region.local_procs[0] == MPID_nem_mem_region.rank); + MPIU_Assert(local_procs_0 == rank); mpi_errno = MPIU_SHMW_Hnd_get_serialized_by_ref(memory->hnd, &serialized_hnd); - if (mpi_errno) - MPIR_ERR_POP(mpi_errno); + if (mpi_errno) MPIR_ERR_POP(mpi_errno); /* must come before barrier_init since we use OPA in that function */ #ifdef OPA_USE_LOCK_BASED_PRIMITIVES - ipc_lock = (OPA_emulation_ipl_t *) ((char *) memory->base_addr + ipc_lock_offset); + ipc_lock = (OPA_emulation_ipl_t *)((char *)memory->base_addr + ipc_lock_offset); ret = OPA_Interprocess_lock_init(ipc_lock, local_rank == 0); MPIR_ERR_CHKANDJUMP1(ret != 0, mpi_errno, MPI_ERR_OTHER, "**fail", "**fail %d", ret); #endif - mpi_errno = MPIDU_shm_barrier_init((MPID_nem_barrier_t *) memory->base_addr, TRUE); - if (mpi_errno) - MPIR_ERR_POP(mpi_errno); + mpi_errno = MPIDU_shm_barrier_init((MPIDU_shm_barrier_t *) memory->base_addr, barrier, TRUE); + if (mpi_errno) MPIR_ERR_POP(mpi_errno); - /* The opa and nem barrier initializations must come before we (the + /* The opa and barrier initializations must come before we (the * leader) put the sharedFilename attribute. Since this is a * serializing operation with our peers on the local node this * ensures that these initializations have occurred before any peer * attempts to use the resources. */ mpi_errno = PMI2_Info_PutNodeAttr("sharedFilename", serialized_hnd); - if (mpi_errno) - MPIR_ERR_POP(mpi_errno); - } - else { + if (mpi_errno) MPIR_ERR_POP(mpi_errno); + } else { int found = FALSE; /* Allocate space for pmi key and val */ @@ -264,23 +255,18 @@ int MPIDU_Seg_commit(MPID_nem_seg_ptr_t memory, int num_local, int local_rank) /* get name of shared file */ mpi_errno = PMI2_Info_GetNodeAttr("sharedFilename", val, PMI2_MAX_VALLEN, &found, TRUE); - if (mpi_errno) - MPIR_ERR_POP(mpi_errno); + if (mpi_errno) MPIR_ERR_POP(mpi_errno); MPIR_ERR_CHKINTERNAL(!found, mpi_errno, "nodeattr not found"); mpi_errno = MPIU_SHMW_Hnd_deserialize(memory->hnd, val, strlen(val)); - if (mpi_errno) - MPIR_ERR_POP(mpi_errno); + if (mpi_errno) MPIR_ERR_POP(mpi_errno); - mpi_errno = - MPIU_SHMW_Seg_attach(memory->hnd, memory->segment_len, (char **) &memory->base_addr, - 0); - if (mpi_errno) - MPIR_ERR_POP(mpi_errno); + mpi_errno = MPIU_SHMW_Seg_attach(memory->hnd, memory->segment_len, (char **)&memory->base_addr, 0); + if (mpi_errno) MPIR_ERR_POP(mpi_errno); /* must come before barrier_init since we use OPA in that function */ #ifdef OPA_USE_LOCK_BASED_PRIMITIVES - ipc_lock = (OPA_emulation_ipl_t *) ((char *) memory->base_addr + ipc_lock_offset); + ipc_lock = (OPA_emulation_ipl_t *)((char *)memory->base_addr + ipc_lock_offset); ret = OPA_Interprocess_lock_init(ipc_lock, local_rank == 0); MPIR_ERR_CHKANDJUMP1(ret != 0, mpi_errno, MPI_ERR_OTHER, "**fail", "**fail %d", ret); @@ -292,53 +278,48 @@ int MPIDU_Seg_commit(MPID_nem_seg_ptr_t memory, int num_local, int local_rank) * this behavior. */ #endif - mpi_errno = MPIDU_shm_barrier_init((MPID_nem_barrier_t *) memory->base_addr, FALSE); - if (mpi_errno) - MPIR_ERR_POP(mpi_errno); + mpi_errno = MPIDU_shm_barrier_init((MPIDU_shm_barrier_t *) memory->base_addr, barrier, FALSE); + if (mpi_errno) MPIR_ERR_POP(mpi_errno); } - mpi_errno = MPIDU_shm_barrier(); - if (mpi_errno) - MPIR_ERR_POP(mpi_errno); + mpi_errno = MPIDU_shm_barrier(*barrier, num_local); + if (mpi_errno) MPIR_ERR_POP(mpi_errno); if (local_rank == 0) { mpi_errno = MPIU_SHMW_Seg_remove(memory->hnd); - if (mpi_errno) - MPIR_ERR_POP(mpi_errno); + if (mpi_errno) MPIR_ERR_POP(mpi_errno); } current_addr = memory->base_addr; - memory->symmetrical = 0; + memory->symmetrical = 0 ; } #else /* we are using PMIv1 */ /* if there is only one process on this processor, don't use shared memory */ - if (num_local == 1) { + if (num_local == 1) + { char *addr; - MPIU_CHKPMEM_MALLOC(addr, char *, segment_len + MPID_NEM_CACHE_LINE_LEN, mpi_errno, - "segment"); + MPIU_CHKPMEM_MALLOC(addr, char *, segment_len + MPIDU_SHM_CACHE_LINE_LEN, mpi_errno, "segment"); memory->base_addr = addr; current_addr = - (char *) (((MPIU_Upint) addr + (MPIU_Upint) MPID_NEM_CACHE_LINE_LEN - 1) & - (~((MPIU_Upint) MPID_NEM_CACHE_LINE_LEN - 1))); + (char *) (((MPIU_Upint) addr + (MPIU_Upint) MPIDU_SHM_CACHE_LINE_LEN - 1) & + (~((MPIU_Upint) MPIDU_SHM_CACHE_LINE_LEN - 1))); memory->symmetrical = 0; /* we still need to call barrier */ - pmi_errno = PMI_Barrier(); - MPIR_ERR_CHKANDJUMP1(pmi_errno != PMI_SUCCESS, mpi_errno, MPI_ERR_OTHER, "**pmi_barrier", - "**pmi_barrier %d", pmi_errno); + pmi_errno = PMI_Barrier(); + MPIR_ERR_CHKANDJUMP1 (pmi_errno != PMI_SUCCESS, mpi_errno, MPI_ERR_OTHER, "**pmi_barrier", "**pmi_barrier %d", pmi_errno); /* must come before barrier_init since we use OPA in that function */ #ifdef OPA_USE_LOCK_BASED_PRIMITIVES - ipc_lock = (OPA_emulation_ipl_t *) ((char *) memory->base_addr + ipc_lock_offset); - ret = OPA_Interprocess_lock_init(ipc_lock, TRUE /*isLeader */); + ipc_lock = (OPA_emulation_ipl_t *)((char *)memory->base_addr + ipc_lock_offset); + ret = OPA_Interprocess_lock_init(ipc_lock, TRUE/*isLeader*/); MPIR_ERR_CHKANDJUMP1(ret != 0, mpi_errno, MPI_ERR_OTHER, "**fail", "**fail %d", ret); #endif - mpi_errno = MPIDU_shm_barrier_init((MPID_nem_barrier_t *) memory->base_addr, TRUE); - if (mpi_errno) - MPIR_ERR_POP(mpi_errno); + mpi_errno = MPIDU_shm_barrier_init((MPIDU_shm_barrier_t *) memory->base_addr, barrier, TRUE); + if (mpi_errno) MPIR_ERR_POP(mpi_errno); } - else { + else{ /* Allocate space for pmi key and val */ pmi_errno = PMI_KVS_Get_key_length_max(&key_max_sz); MPIR_ERR_CHKANDJUMP1(pmi_errno, mpi_errno, MPI_ERR_OTHER, "**fail", "**fail %d", pmi_errno); @@ -348,94 +329,77 @@ int MPIDU_Seg_commit(MPID_nem_seg_ptr_t memory, int num_local, int local_rank) MPIR_ERR_CHKANDJUMP1(pmi_errno, mpi_errno, MPI_ERR_OTHER, "**fail", "**fail %d", pmi_errno); MPIU_CHKLMEM_MALLOC(val, char *, val_max_sz, mpi_errno, "val"); -// mpi_errno = MPIDI_PG_GetConnKVSname (&kvs_name); - kvs_name = "my_kvs_name"; - if (mpi_errno) - MPIR_ERR_POP(mpi_errno); + mpi_errno = MPIDI_PG_GetConnKVSname (&kvs_name); + if (mpi_errno) MPIR_ERR_POP (mpi_errno); - if (local_rank == 0) { - mpi_errno = - MPIU_SHMW_Seg_create_and_attach(memory->hnd, memory->segment_len, - &(memory->base_addr), 0); - if (mpi_errno != MPI_SUCCESS) - MPIR_ERR_POP(mpi_errno); + if (local_rank == 0){ + mpi_errno = MPIU_SHMW_Seg_create_and_attach(memory->hnd, memory->segment_len, &(memory->base_addr), 0); + if (mpi_errno != MPI_SUCCESS) MPIR_ERR_POP (mpi_errno); /* post name of shared file */ - MPIU_Assert(MPID_nem_mem_region.local_procs[0] == MPID_nem_mem_region.rank); - MPL_snprintf(key, key_max_sz, "sharedFilename[%i]", MPID_nem_mem_region.rank); + MPIU_Assert(local_procs_0 == rank); + MPL_snprintf(key, key_max_sz, "sharedFilename[%i]", rank); mpi_errno = MPIU_SHMW_Hnd_get_serialized_by_ref(memory->hnd, &serialized_hnd); - if (mpi_errno != MPI_SUCCESS) - MPIR_ERR_POP(mpi_errno); + if (mpi_errno != MPI_SUCCESS) MPIR_ERR_POP (mpi_errno); - pmi_errno = PMI_KVS_Put(kvs_name, key, serialized_hnd); - MPIR_ERR_CHKANDJUMP1(pmi_errno != PMI_SUCCESS, mpi_errno, MPI_ERR_OTHER, - "**pmi_kvs_put", "**pmi_kvs_put %d", pmi_errno); + pmi_errno = PMI_KVS_Put (kvs_name, key, serialized_hnd); + MPIR_ERR_CHKANDJUMP1 (pmi_errno != PMI_SUCCESS, mpi_errno, MPI_ERR_OTHER, "**pmi_kvs_put", "**pmi_kvs_put %d", pmi_errno); - pmi_errno = PMI_KVS_Commit(kvs_name); - MPIR_ERR_CHKANDJUMP1(pmi_errno != PMI_SUCCESS, mpi_errno, MPI_ERR_OTHER, - "**pmi_kvs_commit", "**pmi_kvs_commit %d", pmi_errno); + pmi_errno = PMI_KVS_Commit (kvs_name); + MPIR_ERR_CHKANDJUMP1 (pmi_errno != PMI_SUCCESS, mpi_errno, MPI_ERR_OTHER, "**pmi_kvs_commit", "**pmi_kvs_commit %d", pmi_errno); /* must come before barrier_init since we use OPA in that function */ #ifdef OPA_USE_LOCK_BASED_PRIMITIVES - ipc_lock = (OPA_emulation_ipl_t *) ((char *) memory->base_addr + ipc_lock_offset); + ipc_lock = (OPA_emulation_ipl_t *)((char *)memory->base_addr + ipc_lock_offset); ret = OPA_Interprocess_lock_init(ipc_lock, local_rank == 0); MPIR_ERR_CHKANDJUMP1(ret != 0, mpi_errno, MPI_ERR_OTHER, "**fail", "**fail %d", ret); #endif - mpi_errno = MPIDU_shm_barrier_init((MPID_nem_barrier_t *) memory->base_addr, TRUE); - if (mpi_errno) - MPIR_ERR_POP(mpi_errno); + mpi_errno = MPIDU_shm_barrier_init((MPIDU_shm_barrier_t *) memory->base_addr, barrier, TRUE); + if (mpi_errno) MPIR_ERR_POP(mpi_errno); pmi_errno = PMI_Barrier(); - MPIR_ERR_CHKANDJUMP1(pmi_errno != PMI_SUCCESS, mpi_errno, MPI_ERR_OTHER, - "**pmi_barrier", "**pmi_barrier %d", pmi_errno); + MPIR_ERR_CHKANDJUMP1 (pmi_errno != PMI_SUCCESS, mpi_errno, MPI_ERR_OTHER, "**pmi_barrier", "**pmi_barrier %d", pmi_errno); } - else { + else + { pmi_errno = PMI_Barrier(); - MPIR_ERR_CHKANDJUMP1(pmi_errno != PMI_SUCCESS, mpi_errno, MPI_ERR_OTHER, - "**pmi_barrier", "**pmi_barrier %d", pmi_errno); + MPIR_ERR_CHKANDJUMP1 (pmi_errno != PMI_SUCCESS, mpi_errno, MPI_ERR_OTHER, "**pmi_barrier", "**pmi_barrier %d", pmi_errno); /* get name of shared file */ - MPL_snprintf(key, key_max_sz, "sharedFilename[%i]", MPID_nem_mem_region.local_procs[0]); + MPL_snprintf(key, key_max_sz, "sharedFilename[%i]", local_procs_0); pmi_errno = PMI_KVS_Get(kvs_name, key, val, val_max_sz); MPIR_ERR_CHKANDJUMP1(pmi_errno != PMI_SUCCESS, mpi_errno, MPI_ERR_OTHER, "**pmi_kvs_get", "**pmi_kvs_get %d", pmi_errno); mpi_errno = MPIU_SHMW_Hnd_deserialize(memory->hnd, val, strlen(val)); - mpi_errno = MPIU_SHMW_Hnd_deserialize(memory->hnd, val, strlen(val)); - if (mpi_errno != MPI_SUCCESS) - MPIR_ERR_POP(mpi_errno); + if(mpi_errno != MPI_SUCCESS) MPIR_ERR_POP(mpi_errno); - mpi_errno = - MPIU_SHMW_Seg_attach(memory->hnd, memory->segment_len, (char **) &memory->base_addr, - 0); - if (mpi_errno) - MPIR_ERR_POP(mpi_errno); + mpi_errno = MPIU_SHMW_Seg_attach(memory->hnd, memory->segment_len, (char **)&memory->base_addr, 0); + if (mpi_errno) MPIR_ERR_POP (mpi_errno); /* must come before barrier_init since we use OPA in that function */ #ifdef OPA_USE_LOCK_BASED_PRIMITIVES - ipc_lock = (OPA_emulation_ipl_t *) ((char *) memory->base_addr + ipc_lock_offset); + ipc_lock = (OPA_emulation_ipl_t *)((char *)memory->base_addr + ipc_lock_offset); ret = OPA_Interprocess_lock_init(ipc_lock, local_rank == 0); MPIR_ERR_CHKANDJUMP1(ret != 0, mpi_errno, MPI_ERR_OTHER, "**fail", "**fail %d", ret); #endif - mpi_errno = MPIDU_shm_barrier_init((MPID_nem_barrier_t *) memory->base_addr, FALSE); - if (mpi_errno) - MPIR_ERR_POP(mpi_errno); + mpi_errno = MPIDU_shm_barrier_init((MPIDU_shm_barrier_t *) memory->base_addr, barrier, FALSE); + if (mpi_errno) MPIR_ERR_POP(mpi_errno); } - mpi_errno = MPIDU_shm_barrier(); - if (mpi_errno) - MPIR_ERR_POP(mpi_errno); + mpi_errno = MPIDU_shm_barrier(*barrier, num_local); + if (mpi_errno) MPIR_ERR_POP(mpi_errno); - if (local_rank == 0) { + if (local_rank == 0) + { mpi_errno = MPIU_SHMW_Seg_remove(memory->hnd); - if (mpi_errno != MPI_SUCCESS) - MPIR_ERR_POP(mpi_errno); + if (mpi_errno != MPI_SUCCESS) MPIR_ERR_POP (mpi_errno); } current_addr = memory->base_addr; - memory->symmetrical = 0; + memory->symmetrical = 0 ; } #endif /* assign sections of the shared memory segment to their pointers */ @@ -444,18 +408,19 @@ int MPIDU_Seg_commit(MPID_nem_seg_ptr_t memory, int num_local, int local_rank) size_left = segment_len; /* reserve room for shared mem barrier (We used a whole cacheline) */ - current_addr = (char *) current_addr + MPID_NEM_CACHE_LINE_LEN; - MPIU_Assert(size_left >= MPID_NEM_CACHE_LINE_LEN); - size_left -= MPID_NEM_CACHE_LINE_LEN; + current_addr = (char *) current_addr + MPIDU_SHM_CACHE_LINE_LEN; + MPIU_Assert(size_left >= MPIDU_SHM_CACHE_LINE_LEN); + size_left -= MPIDU_SHM_CACHE_LINE_LEN; #ifdef OPA_USE_LOCK_BASED_PRIMITIVES /* reserve room for the opa emulation lock */ - current_addr = (char *) current_addr + MPID_NEM_CACHE_LINE_LEN; - MPIU_Assert(size_left >= MPID_NEM_CACHE_LINE_LEN); - size_left -= MPID_NEM_CACHE_LINE_LEN; + current_addr = (char *) current_addr + MPIDU_SHM_CACHE_LINE_LEN; + MPIU_Assert(size_left >= MPIDU_SHM_CACHE_LINE_LEN); + size_left -= MPIDU_SHM_CACHE_LINE_LEN; #endif - do { + do + { alloc_elem_t *ep; ALLOCQ_DEQUEUE(&ep); @@ -463,24 +428,23 @@ int MPIDU_Seg_commit(MPID_nem_seg_ptr_t memory, int num_local, int local_rank) *(ep->ptr_p) = current_addr; MPIU_Assert(size_left >= ep->len); size_left -= ep->len; - current_addr = (char *) current_addr + ep->len; + current_addr = (char *)current_addr + ep->len; MPL_free(ep); - MPIU_Assert((char *) current_addr <= (char *) start_addr + segment_len); + MPIU_Assert((char *)current_addr <= (char *)start_addr + segment_len); } while (!ALLOCQ_EMPTY()); - mpi_errno = check_alloc(num_local, local_rank); - if (mpi_errno) - MPIR_ERR_POP(mpi_errno); - + mpi_errno = check_alloc(memory, *barrier, num_local, local_rank); + if (mpi_errno) MPIR_ERR_POP(mpi_errno); + MPIU_CHKPMEM_COMMIT(); - fn_exit: + fn_exit: MPIU_CHKLMEM_FREEALL(); - MPIDI_FUNC_EXIT(MPID_STATE_MPIDI_CH3I_SEG_COMMIT); + MPIDI_FUNC_EXIT(MPID_STATE_MPIDU_SHM_SEG_COMMIT); return mpi_errno; - fn_fail: + fn_fail: /* --BEGIN ERROR HANDLING-- */ MPIU_SHMW_Seg_remove(memory->hnd); MPIU_SHMW_Hnd_finalize(&(memory->hnd)); @@ -489,33 +453,32 @@ int MPIDU_Seg_commit(MPID_nem_seg_ptr_t memory, int num_local, int local_rank) /* --END ERROR HANDLING-- */ } -/* MPIDI_CH3I_Seg_destroy() free the shared memory segment */ +/* MPIDU_SHM_Seg_destroy() free the shared memory segment */ #undef FUNCNAME #define FUNCNAME MPIDU_Seg_destroy #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -int MPIDU_Seg_destroy(void) +int MPIDU_Seg_destroy(MPIDU_shm_seg_ptr_t memory, int num_local) { int mpi_errno = MPI_SUCCESS; - MPIDI_STATE_DECL(MPID_STATE_MPIDI_CH3I_SEG_DESTROY); + MPIDI_STATE_DECL(MPID_STATE_MPIDU_SHM_SEG_DESTROY); - MPIDI_FUNC_ENTER(MPID_STATE_MPIDI_CH3I_SEG_DESTROY); + MPIDI_FUNC_ENTER(MPID_STATE_MPIDU_SHM_SEG_DESTROY); - if (MPID_nem_mem_region.num_local == 1) - MPL_free(MPID_nem_mem_region.memory.base_addr); - else { - mpi_errno = MPIU_SHMW_Seg_detach(MPID_nem_mem_region.memory.hnd, - &(MPID_nem_mem_region.memory.base_addr), - MPID_nem_mem_region.memory.segment_len); - if (mpi_errno) - MPIR_ERR_POP(mpi_errno); + if (num_local == 1) + MPL_free(memory->base_addr); + else + { + mpi_errno = MPIU_SHMW_Seg_detach(memory->hnd, + &(memory->base_addr), memory->segment_len); + if (mpi_errno) MPIR_ERR_POP (mpi_errno); } fn_exit: - MPIU_SHMW_Hnd_finalize(&(MPID_nem_mem_region.memory.hnd)); - MPIDI_FUNC_EXIT(MPID_STATE_MPIDI_CH3I_SEG_DESTROY); + MPIU_SHMW_Hnd_finalize(&(memory->hnd)); + MPIDI_FUNC_EXIT(MPID_STATE_MPIDU_SHM_SEG_DESTROY); return mpi_errno; - fn_fail: + fn_fail: goto fn_exit; } @@ -526,46 +489,39 @@ int MPIDU_Seg_destroy(void) #define FUNCNAME check_alloc #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -static int check_alloc(int num_local, int local_rank) +static int check_alloc(MPIDU_shm_seg_ptr_t memory, MPIDU_shm_barrier_ptr_t barrier, + int num_local, int local_rank) { int mpi_errno = MPI_SUCCESS; - MPIDI_STATE_DECL(MPID_STATE_CHECK_ALLOC); + MPIDI_STATE_DECL(MPID_STATE_MPIDU_SHM_CHECK_ALLOC); - MPIDI_FUNC_ENTER(MPID_STATE_CHECK_ALLOC); + MPIDI_FUNC_ENTER(MPID_STATE_MPIDU_SHM_CHECK_ALLOC); if (local_rank == 0) { - asym_check_region_p->base_ptr = MPID_nem_mem_region.memory.base_addr; + asym_check_region_p->base_ptr = memory->base_addr; OPA_store_int(&asym_check_region_p->is_asym, 0); } - mpi_errno = MPIDU_shm_barrier(); - if (mpi_errno) - MPIR_ERR_POP(mpi_errno); + mpi_errno = MPIDU_shm_barrier(barrier, num_local); + if (mpi_errno) MPIR_ERR_POP(mpi_errno); - if (asym_check_region_p->base_ptr != MPID_nem_mem_region.memory.base_addr) + if (asym_check_region_p->base_ptr != memory->base_addr) OPA_store_int(&asym_check_region_p->is_asym, 1); OPA_read_write_barrier(); - mpi_errno = MPIDU_shm_barrier(); - if (mpi_errno) - MPIR_ERR_POP(mpi_errno); + mpi_errno = MPIDU_shm_barrier(barrier, num_local); + if (mpi_errno) MPIR_ERR_POP(mpi_errno); if (OPA_load_int(&asym_check_region_p->is_asym)) { - MPID_nem_mem_region.memory.symmetrical = 0; - MPID_nem_asymm_base_addr = MPID_nem_mem_region.memory.base_addr; -#ifdef MPID_NEM_SYMMETRIC_QUEUES - MPIR_ERR_INTERNALANDJUMP(mpi_errno, "queues are not symmetrically allocated as expected"); -#endif - } - else { - MPID_nem_mem_region.memory.symmetrical = 1; - MPID_nem_asymm_base_addr = NULL; + memory->symmetrical = 0; + } else { + memory->symmetrical = 1; } fn_exit: - MPIDI_FUNC_EXIT(MPID_STATE_CHECK_ALLOC); + MPIDI_FUNC_EXIT(MPID_STATE_MPIDU_SHM_CHECK_ALLOC); return mpi_errno; - fn_fail: + fn_fail: goto fn_exit; } diff --git a/src/mpid/common/shm/mpidu_shm_barrier.c b/src/mpid/common/shm/mpidu_shm_barrier.c index 21b9f57..bfefb26 100644 --- a/src/mpid/common/shm/mpidu_shm_barrier.c +++ b/src/mpid/common/shm/mpidu_shm_barrier.c @@ -4,8 +4,7 @@ * See COPYRIGHT in top-level directory. */ -#include "mpidimpl.h" -#include "mpiu_os_wrappers.h" +#include "mpidu_shm_impl.h" static int sense; static int barrier_init = 0; @@ -14,22 +13,23 @@ static int barrier_init = 0; #define FUNCNAME MPIDU_shm_barrier_init #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -int MPIDU_shm_barrier_init(MPID_nem_barrier_t *barrier_region, int init_values) +int MPIDU_shm_barrier_init(MPIDU_shm_barrier_t *barrier_region, + MPIDU_shm_barrier_ptr_t *barrier, int init_values) { - MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPID_NEM_BARRIER_INIT); + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDU_SHM_BARRIER_INIT); - MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPID_NEM_BARRIER_INIT); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDU_SHM_BARRIER_INIT); - MPID_nem_mem_region.barrier = barrier_region; + *barrier = barrier_region; if (init_values) { - OPA_store_int(&MPID_nem_mem_region.barrier->val, 0); - OPA_store_int(&MPID_nem_mem_region.barrier->wait, 0); + OPA_store_int(&(*barrier)->val, 0); + OPA_store_int(&(*barrier)->wait, 0); OPA_write_barrier(); } sense = 0; barrier_init = 1; - MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPID_NEM_BARRIER_INIT); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDU_SHM_BARRIER_INIT); return MPI_SUCCESS; } @@ -39,34 +39,34 @@ int MPIDU_shm_barrier_init(MPID_nem_barrier_t *barrier_region, int init_values) #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) /* FIXME: this is not a scalable algorithm because everyone is polling on the same cacheline */ -int MPIDU_shm_barrier(void) +int MPIDU_shm_barrier(MPIDU_shm_barrier_t *barrier, int num_local) { int mpi_errno = MPI_SUCCESS; - MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPID_NEM_BARRIER); + MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDU_SHM_BARRIER); - MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPID_NEM_BARRIER); + MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDU_SHM_BARRIER); - if (MPID_nem_mem_region.num_local == 1) + if (num_local == 1) goto fn_exit; MPIR_ERR_CHKINTERNAL(!barrier_init, mpi_errno, "barrier not initialized"); - if (OPA_fetch_and_incr_int(&MPID_nem_mem_region.barrier->val) == MPID_nem_mem_region.num_local - 1) + if (OPA_fetch_and_incr_int(&barrier->val) == num_local - 1) { - OPA_store_int(&MPID_nem_mem_region.barrier->val, 0); - OPA_store_int(&MPID_nem_mem_region.barrier->wait, 1 - sense); + OPA_store_int(&barrier->val, 0); + OPA_store_int(&barrier->wait, 1 - sense); OPA_write_barrier(); } else { /* wait */ - while (OPA_load_int(&MPID_nem_mem_region.barrier->wait) == sense) - MPL_sched_yield(); /* skip */ + while (OPA_load_int(&barrier->wait) == sense) + MPIU_PW_Sched_yield(); /* skip */ } sense = 1 - sense; fn_fail: fn_exit: - MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPID_NEM_BARRIER); + MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDU_SHM_BARRIER); return mpi_errno; } diff --git a/src/mpid/common/shm/mpidu_shm_impl.h b/src/mpid/common/shm/mpidu_shm_impl.h new file mode 100644 index 0000000..b822101 --- /dev/null +++ b/src/mpid/common/shm/mpidu_shm_impl.h @@ -0,0 +1,15 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2016 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + */ +#ifndef MPIDU_SHM_IMPL_H +#define MPIDU_SHM_IMPL_H + +#include "mpidimpl.h" +#include "mpiu_os_wrappers.h" + +#include "mpidu_shm.h" + +#endif /* ifndef MPIDU_SHM_IMPL_H */ +/* vim: ft=c */ diff --git a/src/mpid/common/shm/mpidu_shm_states.h b/src/mpid/common/shm/mpidu_shm_states.h new file mode 100644 index 0000000..57dacb1 --- /dev/null +++ b/src/mpid/common/shm/mpidu_shm_states.h @@ -0,0 +1,19 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2001 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + */ + +#ifndef MPIDU_SHM_STATES_H +#define MPIDU_SHM_STATES_H + +#define MPID_STATE_LIST_MPIDU_SHM \ +MPID_STATE_MPIDU_SHM_BARRIER, \ +MPID_STATE_MPIDU_SHM_BARRIER_INIT, \ +MPID_STATE_MPIDU_SHM_SEG_ALLOC, \ +MPID_STATE_MPIDU_SHM_SEG_DESTROY, \ +MPID_STATE_MPIDU_SHM_SEG_COMMIT, \ +MPID_STATE_MPIDU_SHM_CHECK_ALLOC, + +#endif +/* vim: ft=c */ http://git.mpich.org/mpich.git/commitdiff/d0ce2eae76bc1eb3ceeb865886632ccaf… commit d0ce2eae76bc1eb3ceeb865886632ccaf6937d4a Author: Yanfei Guo <yguo(a)anl.gov> Date: Wed Jan 20 11:46:41 2016 -0600 mpid/common/shm: cleanup format diff --git a/src/mpid/common/shm/Makefile.mk b/src/mpid/common/shm/Makefile.mk index 8d7010e..4f50dad 100644 --- a/src/mpid/common/shm/Makefile.mk +++ b/src/mpid/common/shm/Makefile.mk @@ -1,7 +1,7 @@ ## -*- Mode: Makefile; -*- ## vim: set ft=automake : ## -## (C) 2011 by Argonne National Laboratory. +## (C) 2016 by Argonne National Laboratory. ## See COPYRIGHT in top-level directory. ## http://git.mpich.org/mpich.git/commitdiff/44388f15b7f13f291d40d7d4e71dfb2c4… commit 44388f15b7f13f291d40d7d4e71dfb2c419de6bd Author: Gengbin Zheng <gengbin.zheng(a)intel.com> Date: Wed Jan 6 19:03:32 2016 -0600 CH4 SHM: Move shm barrier common code to mpid/common. also renamed to MPIDU_shm_ Signed-off-by: Yanfei Guo <yguo(a)anl.gov> diff --git a/src/mpid/ch3/channels/nemesis/include/mpid_nem_post.h b/src/mpid/ch3/channels/nemesis/include/mpid_nem_post.h index cf77ca8..f8dd2d4 100644 --- a/src/mpid/ch3/channels/nemesis/include/mpid_nem_post.h +++ b/src/mpid/ch3/channels/nemesis/include/mpid_nem_post.h @@ -19,8 +19,8 @@ extern int (*MPID_nem_local_lmt_progress)(void); will happen if this is included in mpidpost.h or mpidpre.h) */ int MPID_nem_init(int rank, struct MPIDI_PG *pg_p, int has_parent); int MPID_nem_finalize(void); -int MPID_nem_barrier_init(MPID_nem_barrier_t *barrier_region, int init_values); -int MPID_nem_barrier(void); +int MPIDU_shm_barrier_init(MPID_nem_barrier_t *barrier_region, int init_values); +int MPIDU_shm_barrier(void); int MPID_nem_vc_init(struct MPIDI_VC *vc); int MPID_nem_vc_destroy(struct MPIDI_VC *vc); int MPID_nem_get_business_card(int myRank, char *value, int length); diff --git a/src/mpid/ch3/channels/nemesis/src/Makefile.mk b/src/mpid/ch3/channels/nemesis/src/Makefile.mk index 439ec23..58ade27 100644 --- a/src/mpid/ch3/channels/nemesis/src/Makefile.mk +++ b/src/mpid/ch3/channels/nemesis/src/Makefile.mk @@ -18,7 +18,6 @@ mpi_core_sources += \ src/mpid/ch3/channels/nemesis/src/ch3i_comm.c \ src/mpid/ch3/channels/nemesis/src/ch3i_eagernoncontig.c \ src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c \ - src/mpid/ch3/channels/nemesis/src/mpid_nem_barrier.c \ src/mpid/ch3/channels/nemesis/src/mpid_nem_mpich.c \ src/mpid/ch3/channels/nemesis/src/mpid_nem_ckpt.c \ src/mpid/ch3/channels/nemesis/src/mpid_nem_finalize.c \ diff --git a/src/mpid/ch3/channels/nemesis/src/mpid_nem_ckpt.c b/src/mpid/ch3/channels/nemesis/src/mpid_nem_ckpt.c index 2c4a19c..1b1fce5 100644 --- a/src/mpid/ch3/channels/nemesis/src/mpid_nem_ckpt.c +++ b/src/mpid/ch3/channels/nemesis/src/mpid_nem_ckpt.c @@ -480,7 +480,7 @@ int MPIDI_nem_ckpt_finish(void) channels between local procs), we don't have to flush those channels, just make sure no one is sending or receiving during the checkpoint */ - mpi_errno = MPID_nem_barrier(); + mpi_errno = MPIDU_shm_barrier(); if (mpi_errno) MPIR_ERR_POP(mpi_errno); do { @@ -493,7 +493,7 @@ int MPIDI_nem_ckpt_finish(void) } while (ret == -1 && errno == EINTR); MPIR_ERR_CHKANDJUMP1(ret, mpi_errno, MPI_ERR_OTHER, "**sem_wait", "**sem_wait %s", MPIR_Strerror(errno)); - mpi_errno = MPID_nem_barrier(); + mpi_errno = MPIDU_shm_barrier(); if (mpi_errno) MPIR_ERR_POP(mpi_errno); if (ckpt_result == CKPT_CONTINUE) { diff --git a/src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c b/src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c index c68e095..1c4d45a 100644 --- a/src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c +++ b/src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c @@ -268,7 +268,7 @@ MPID_nem_init(int pg_rank, MPIDI_PG_t *pg_p, int has_parent ATTRIBUTE((unused))) if (mpi_errno) MPIR_ERR_POP (mpi_errno); /* local procs barrier */ - mpi_errno = MPID_nem_barrier(); + mpi_errno = MPIDU_shm_barrier(); if (mpi_errno) MPIR_ERR_POP (mpi_errno); /* find our cell region */ @@ -343,7 +343,7 @@ MPID_nem_init(int pg_rank, MPIDI_PG_t *pg_p, int has_parent ATTRIBUTE((unused))) /* local barrier */ - mpi_errno = MPID_nem_barrier(); + mpi_errno = MPIDU_shm_barrier(); if (mpi_errno) MPIR_ERR_POP(mpi_errno); @@ -394,11 +394,11 @@ MPID_nem_init(int pg_rank, MPIDI_PG_t *pg_p, int has_parent ATTRIBUTE((unused))) MPL_free(publish_bc_orig); - mpi_errno = MPID_nem_barrier(); + mpi_errno = MPIDU_shm_barrier(); if (mpi_errno) MPIR_ERR_POP(mpi_errno); mpi_errno = MPID_nem_mpich_init(); if (mpi_errno) MPIR_ERR_POP(mpi_errno); - mpi_errno = MPID_nem_barrier(); + mpi_errno = MPIDU_shm_barrier(); if (mpi_errno) MPIR_ERR_POP(mpi_errno); #ifdef ENABLE_CHECKPOINTING mpi_errno = MPIDI_nem_ckpt_init(); diff --git a/src/mpid/common/shm/Makefile.mk b/src/mpid/common/shm/Makefile.mk index 90459f9..8d7010e 100644 --- a/src/mpid/common/shm/Makefile.mk +++ b/src/mpid/common/shm/Makefile.mk @@ -6,7 +6,8 @@ ## mpi_core_sources += \ - src/mpid/common/shm/mpidu_shm_alloc.c + src/mpid/common/shm/mpidu_shm_alloc.c \ + src/mpid/common/shm/mpidu_shm_barrier.c # there are no AC_OUTPUT_FILES headers, so builddir is unnecessary AM_CPPFLAGS += -I$(top_srcdir)/src/mpid/common/shm diff --git a/src/mpid/common/shm/mpidu_shm_alloc.c b/src/mpid/common/shm/mpidu_shm_alloc.c index 5e1fad3..b927745 100644 --- a/src/mpid/common/shm/mpidu_shm_alloc.c +++ b/src/mpid/common/shm/mpidu_shm_alloc.c @@ -216,7 +216,7 @@ int MPIDU_Seg_commit(MPID_nem_seg_ptr_t memory, int num_local, int local_rank) MPIR_ERR_CHKANDJUMP1(ret != 0, mpi_errno, MPI_ERR_OTHER, "**fail", "**fail %d", ret); #endif - mpi_errno = MPID_nem_barrier_init((MPID_nem_barrier_t *) memory->base_addr, TRUE); + mpi_errno = MPIDU_shm_barrier_init((MPID_nem_barrier_t *) memory->base_addr, TRUE); if (mpi_errno) MPIR_ERR_POP(mpi_errno); } @@ -243,7 +243,7 @@ int MPIDU_Seg_commit(MPID_nem_seg_ptr_t memory, int num_local, int local_rank) MPIR_ERR_CHKANDJUMP1(ret != 0, mpi_errno, MPI_ERR_OTHER, "**fail", "**fail %d", ret); #endif - mpi_errno = MPID_nem_barrier_init((MPID_nem_barrier_t *) memory->base_addr, TRUE); + mpi_errno = MPIDU_shm_barrier_init((MPID_nem_barrier_t *) memory->base_addr, TRUE); if (mpi_errno) MPIR_ERR_POP(mpi_errno); @@ -292,12 +292,12 @@ int MPIDU_Seg_commit(MPID_nem_seg_ptr_t memory, int num_local, int local_rank) * this behavior. */ #endif - mpi_errno = MPID_nem_barrier_init((MPID_nem_barrier_t *) memory->base_addr, FALSE); + mpi_errno = MPIDU_shm_barrier_init((MPID_nem_barrier_t *) memory->base_addr, FALSE); if (mpi_errno) MPIR_ERR_POP(mpi_errno); } - mpi_errno = MPID_nem_barrier(); + mpi_errno = MPIDU_shm_barrier(); if (mpi_errno) MPIR_ERR_POP(mpi_errno); @@ -334,7 +334,7 @@ int MPIDU_Seg_commit(MPID_nem_seg_ptr_t memory, int num_local, int local_rank) ret = OPA_Interprocess_lock_init(ipc_lock, TRUE /*isLeader */); MPIR_ERR_CHKANDJUMP1(ret != 0, mpi_errno, MPI_ERR_OTHER, "**fail", "**fail %d", ret); #endif - mpi_errno = MPID_nem_barrier_init((MPID_nem_barrier_t *) memory->base_addr, TRUE); + mpi_errno = MPIDU_shm_barrier_init((MPID_nem_barrier_t *) memory->base_addr, TRUE); if (mpi_errno) MPIR_ERR_POP(mpi_errno); } @@ -383,7 +383,7 @@ int MPIDU_Seg_commit(MPID_nem_seg_ptr_t memory, int num_local, int local_rank) MPIR_ERR_CHKANDJUMP1(ret != 0, mpi_errno, MPI_ERR_OTHER, "**fail", "**fail %d", ret); #endif - mpi_errno = MPID_nem_barrier_init((MPID_nem_barrier_t *) memory->base_addr, TRUE); + mpi_errno = MPIDU_shm_barrier_init((MPID_nem_barrier_t *) memory->base_addr, TRUE); if (mpi_errno) MPIR_ERR_POP(mpi_errno); @@ -420,12 +420,12 @@ int MPIDU_Seg_commit(MPID_nem_seg_ptr_t memory, int num_local, int local_rank) MPIR_ERR_CHKANDJUMP1(ret != 0, mpi_errno, MPI_ERR_OTHER, "**fail", "**fail %d", ret); #endif - mpi_errno = MPID_nem_barrier_init((MPID_nem_barrier_t *) memory->base_addr, FALSE); + mpi_errno = MPIDU_shm_barrier_init((MPID_nem_barrier_t *) memory->base_addr, FALSE); if (mpi_errno) MPIR_ERR_POP(mpi_errno); } - mpi_errno = MPID_nem_barrier(); + mpi_errno = MPIDU_shm_barrier(); if (mpi_errno) MPIR_ERR_POP(mpi_errno); @@ -538,7 +538,7 @@ static int check_alloc(int num_local, int local_rank) OPA_store_int(&asym_check_region_p->is_asym, 0); } - mpi_errno = MPID_nem_barrier(); + mpi_errno = MPIDU_shm_barrier(); if (mpi_errno) MPIR_ERR_POP(mpi_errno); @@ -547,7 +547,7 @@ static int check_alloc(int num_local, int local_rank) OPA_read_write_barrier(); - mpi_errno = MPID_nem_barrier(); + mpi_errno = MPIDU_shm_barrier(); if (mpi_errno) MPIR_ERR_POP(mpi_errno); diff --git a/src/mpid/ch3/channels/nemesis/src/mpid_nem_barrier.c b/src/mpid/common/shm/mpidu_shm_barrier.c similarity index 89% rename from src/mpid/ch3/channels/nemesis/src/mpid_nem_barrier.c rename to src/mpid/common/shm/mpidu_shm_barrier.c index 7de3eb8..21b9f57 100644 --- a/src/mpid/ch3/channels/nemesis/src/mpid_nem_barrier.c +++ b/src/mpid/common/shm/mpidu_shm_barrier.c @@ -4,17 +4,17 @@ * See COPYRIGHT in top-level directory. */ -#include "mpid_nem_impl.h" +#include "mpidimpl.h" #include "mpiu_os_wrappers.h" static int sense; static int barrier_init = 0; #undef FUNCNAME -#define FUNCNAME MPID_nem_barrier_init +#define FUNCNAME MPIDU_shm_barrier_init #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -int MPID_nem_barrier_init(MPID_nem_barrier_t *barrier_region, int init_values) +int MPIDU_shm_barrier_init(MPID_nem_barrier_t *barrier_region, int init_values) { MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPID_NEM_BARRIER_INIT); @@ -35,11 +35,11 @@ int MPID_nem_barrier_init(MPID_nem_barrier_t *barrier_region, int init_values) } #undef FUNCNAME -#define FUNCNAME MPID_nem_barrier +#define FUNCNAME MPIDU_shm_barrier #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) /* FIXME: this is not a scalable algorithm because everyone is polling on the same cacheline */ -int MPID_nem_barrier(void) +int MPIDU_shm_barrier(void) { int mpi_errno = MPI_SUCCESS; MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPID_NEM_BARRIER); http://git.mpich.org/mpich.git/commitdiff/5a53eb25a721a0f46da5559d9e10e3cd1… commit 5a53eb25a721a0f46da5559d9e10e3cd16ce2911 Author: Gengbin Zheng <gengbin.zheng(a)intel.com> Date: Wed Jan 6 13:15:47 2016 -0600 Refactor shm common code between ch3 and ch4 to mpid/common Created mpid/common/shm directory for common code shared between ch3 and ch4. Moved some common code including seg_alloc and generic queue header file to the common diretory. Renamed MPID_CH3I MPIDU_. Signed-off-by: Yanfei Guo <yguo(a)anl.gov> diff --git a/src/mpid/ch3/channels/nemesis/Makefile.mk b/src/mpid/ch3/channels/nemesis/Makefile.mk index 8078ba5..597d86e 100644 --- a/src/mpid/ch3/channels/nemesis/Makefile.mk +++ b/src/mpid/ch3/channels/nemesis/Makefile.mk @@ -20,7 +20,6 @@ noinst_HEADERS += \ src/mpid/ch3/channels/nemesis/include/mpid_nem_debug.h \ src/mpid/ch3/channels/nemesis/include/mpid_nem_defs.h \ src/mpid/ch3/channels/nemesis/include/mpid_nem_fbox.h \ - src/mpid/ch3/channels/nemesis/include/mpid_nem_generic_queue.h \ src/mpid/ch3/channels/nemesis/include/mpid_nem_impl.h \ src/mpid/ch3/channels/nemesis/include/mpid_nem_inline.h \ src/mpid/ch3/channels/nemesis/include/mpid_nem_nets.h \ diff --git a/src/mpid/ch3/channels/nemesis/include/mpid_nem_impl.h b/src/mpid/ch3/channels/nemesis/include/mpid_nem_impl.h index addac24..521048e 100644 --- a/src/mpid/ch3/channels/nemesis/include/mpid_nem_impl.h +++ b/src/mpid/ch3/channels/nemesis/include/mpid_nem_impl.h @@ -19,9 +19,9 @@ #define MPID_NEM__BYPASS_Q_MAX_VAL ((MPID_NEM_MPICH_DATA_LEN) - (sizeof(MPIDI_CH3_Pkt_t))) -int MPIDI_CH3I_Seg_alloc(size_t len, void **ptr_p); -int MPIDI_CH3I_Seg_commit(MPID_nem_seg_ptr_t memory, int num_local, int local_rank); -int MPIDI_CH3I_Seg_destroy(void); +int MPIDU_Seg_alloc(size_t len, void **ptr_p); +int MPIDU_Seg_commit(MPID_nem_seg_ptr_t memory, int num_local, int local_rank); +int MPIDU_Seg_destroy(void); int MPID_nem_check_alloc(int); int MPID_nem_mpich_init(void); int MPID_nem_coll_init (void); diff --git a/src/mpid/ch3/channels/nemesis/include/mpidi_ch3_impl.h b/src/mpid/ch3/channels/nemesis/include/mpidi_ch3_impl.h index 4037253..92edc88 100644 --- a/src/mpid/ch3/channels/nemesis/include/mpidi_ch3_impl.h +++ b/src/mpid/ch3/channels/nemesis/include/mpidi_ch3_impl.h @@ -9,7 +9,7 @@ #include "mpidimpl.h" #include "mpiu_os_wrappers.h" -#include "mpid_nem_generic_queue.h" +#include "mpidu_generic_queue.h" #include "mpl_utlist.h" #if defined(HAVE_ASSERT_H) diff --git a/src/mpid/ch3/channels/nemesis/include/mpidi_ch3_pre.h b/src/mpid/ch3/channels/nemesis/include/mpidi_ch3_pre.h index 3e80f7e..44aedc8 100644 --- a/src/mpid/ch3/channels/nemesis/include/mpidi_ch3_pre.h +++ b/src/mpid/ch3/channels/nemesis/include/mpidi_ch3_pre.h @@ -7,7 +7,7 @@ #if !defined(MPICH_MPIDI_CH3_PRE_H_INCLUDED) #define MPICH_MPIDI_CH3_PRE_H_INCLUDED #include "mpid_nem_pre.h" -#include "mpid_nem_generic_queue.h" +#include "mpidu_generic_queue.h" #if defined(HAVE_NETINET_IN_H) #include <netinet/in.h> diff --git a/src/mpid/ch3/channels/nemesis/src/Makefile.mk b/src/mpid/ch3/channels/nemesis/src/Makefile.mk index d55697c..439ec23 100644 --- a/src/mpid/ch3/channels/nemesis/src/Makefile.mk +++ b/src/mpid/ch3/channels/nemesis/src/Makefile.mk @@ -17,7 +17,6 @@ mpi_core_sources += \ src/mpid/ch3/channels/nemesis/src/ch3_win_fns.c \ src/mpid/ch3/channels/nemesis/src/ch3i_comm.c \ src/mpid/ch3/channels/nemesis/src/ch3i_eagernoncontig.c \ - src/mpid/ch3/channels/nemesis/src/mpid_nem_alloc.c \ src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c \ src/mpid/ch3/channels/nemesis/src/mpid_nem_barrier.c \ src/mpid/ch3/channels/nemesis/src/mpid_nem_mpich.c \ diff --git a/src/mpid/ch3/channels/nemesis/src/mpid_nem_finalize.c b/src/mpid/ch3/channels/nemesis/src/mpid_nem_finalize.c index dd50ac3..2d38d1c 100644 --- a/src/mpid/ch3/channels/nemesis/src/mpid_nem_finalize.c +++ b/src/mpid/ch3/channels/nemesis/src/mpid_nem_finalize.c @@ -50,7 +50,7 @@ int MPID_nem_finalize(void) if (mpi_errno) MPIR_ERR_POP (mpi_errno); /* free the shared memory segment */ - mpi_errno = MPIDI_CH3I_Seg_destroy(); + mpi_errno = MPIDU_Seg_destroy(); if (mpi_errno) MPIR_ERR_POP (mpi_errno); #ifdef PAPI_MONITOR diff --git a/src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c b/src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c index cf08b32..c68e095 100644 --- a/src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c +++ b/src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c @@ -238,29 +238,29 @@ MPID_nem_init(int pg_rank, MPIDI_PG_t *pg_p, int has_parent ATTRIBUTE((unused))) #endif /*FORCE_ASYM */ /* Request fastboxes region */ - mpi_errno = MPIDI_CH3I_Seg_alloc(MPL_MAX((num_local*((num_local-1)*sizeof(MPID_nem_fastbox_t))), MPID_NEM_ASYMM_NULL_VAL), + mpi_errno = MPIDU_Seg_alloc(MPL_MAX((num_local*((num_local-1)*sizeof(MPID_nem_fastbox_t))), MPID_NEM_ASYMM_NULL_VAL), (void **)&fastboxes_p); if (mpi_errno) MPIR_ERR_POP(mpi_errno); /* Request data cells region */ - mpi_errno = MPIDI_CH3I_Seg_alloc(num_local * MPID_NEM_NUM_CELLS * sizeof(MPID_nem_cell_t), (void **)&cells_p); + mpi_errno = MPIDU_Seg_alloc(num_local * MPID_NEM_NUM_CELLS * sizeof(MPID_nem_cell_t), (void **)&cells_p); if (mpi_errno) MPIR_ERR_POP(mpi_errno); /* Request free q region */ - mpi_errno = MPIDI_CH3I_Seg_alloc(num_local * sizeof(MPID_nem_queue_t), (void **)&free_queues_p); + mpi_errno = MPIDU_Seg_alloc(num_local * sizeof(MPID_nem_queue_t), (void **)&free_queues_p); if (mpi_errno) MPIR_ERR_POP(mpi_errno); /* Request recv q region */ - mpi_errno = MPIDI_CH3I_Seg_alloc(num_local * sizeof(MPID_nem_queue_t), (void **)&recv_queues_p); + mpi_errno = MPIDU_Seg_alloc(num_local * sizeof(MPID_nem_queue_t), (void **)&recv_queues_p); if (mpi_errno) MPIR_ERR_POP(mpi_errno); /* Request shared collectives barrier vars region */ - mpi_errno = MPIDI_CH3I_Seg_alloc(MPID_NEM_NUM_BARRIER_VARS * sizeof(MPID_nem_barrier_vars_t), + mpi_errno = MPIDU_Seg_alloc(MPID_NEM_NUM_BARRIER_VARS * sizeof(MPID_nem_barrier_vars_t), (void **)&MPID_nem_mem_region.barrier_vars); if (mpi_errno) MPIR_ERR_POP(mpi_errno); /* Actually allocate the segment and assign regions to the pointers */ - mpi_errno = MPIDI_CH3I_Seg_commit(&MPID_nem_mem_region.memory, num_local, local_rank); + mpi_errno = MPIDU_Seg_commit(&MPID_nem_mem_region.memory, num_local, local_rank); if (mpi_errno) MPIR_ERR_POP (mpi_errno); /* init shared collectives barrier region */ diff --git a/src/mpid/common/Makefile.mk b/src/mpid/common/Makefile.mk index d041162..c607619 100644 --- a/src/mpid/common/Makefile.mk +++ b/src/mpid/common/Makefile.mk @@ -12,3 +12,4 @@ include $(top_srcdir)/src/mpid/common/sock/Makefile.mk include $(top_srcdir)/src/mpid/common/thread/Makefile.mk include $(top_srcdir)/src/mpid/common/hcoll/Makefile.mk include $(top_srcdir)/src/mpid/common/timers/Makefile.mk +include $(top_srcdir)/src/mpid/common/shm/Makefile.mk diff --git a/src/mpid/common/shm/Makefile.mk b/src/mpid/common/shm/Makefile.mk new file mode 100644 index 0000000..90459f9 --- /dev/null +++ b/src/mpid/common/shm/Makefile.mk @@ -0,0 +1,17 @@ +## -*- Mode: Makefile; -*- +## vim: set ft=automake : +## +## (C) 2011 by Argonne National Laboratory. +## See COPYRIGHT in top-level directory. +## + +mpi_core_sources += \ + src/mpid/common/shm/mpidu_shm_alloc.c + +# there are no AC_OUTPUT_FILES headers, so builddir is unnecessary +AM_CPPFLAGS += -I$(top_srcdir)/src/mpid/common/shm + +noinst_HEADERS += \ + src/mpid/common/shm/mpidu_generic_queue.h + + diff --git a/src/mpid/ch3/channels/nemesis/include/mpid_nem_generic_queue.h b/src/mpid/common/shm/mpidu_generic_queue.h similarity index 100% rename from src/mpid/ch3/channels/nemesis/include/mpid_nem_generic_queue.h rename to src/mpid/common/shm/mpidu_generic_queue.h diff --git a/src/mpid/ch3/channels/nemesis/src/mpid_nem_alloc.c b/src/mpid/common/shm/mpidu_shm_alloc.c similarity index 52% rename from src/mpid/ch3/channels/nemesis/src/mpid_nem_alloc.c rename to src/mpid/common/shm/mpidu_shm_alloc.c index f093850..5e1fad3 100644 --- a/src/mpid/ch3/channels/nemesis/src/mpid_nem_alloc.c +++ b/src/mpid/common/shm/mpidu_shm_alloc.c @@ -4,7 +4,9 @@ * See COPYRIGHT in top-level directory. */ -#include "mpid_nem_impl.h" +#include <mpidimpl.h> +#include "mpiu_shm_wrappers.h" + #ifdef USE_PMI2_API #include "pmi2.h" #else @@ -26,35 +28,38 @@ extern int mkstemp(char *t); #endif +#include "mpidu_generic_queue.h" + + static int check_alloc(int num_local, int local_rank); -typedef struct alloc_elem -{ +typedef struct alloc_elem { struct alloc_elem *next; void **ptr_p; size_t len; } alloc_elem_t; -static struct { alloc_elem_t *head, *tail; } allocq = {0}; +static struct { + alloc_elem_t *head, *tail; +} allocq = { 0 }; #define ALLOCQ_HEAD() GENERIC_Q_HEAD(allocq) #define ALLOCQ_EMPTY() GENERIC_Q_EMPTY(allocq) #define ALLOCQ_ENQUEUE(ep) GENERIC_Q_ENQUEUE(&allocq, ep, next) #define ALLOCQ_DEQUEUE(epp) GENERIC_Q_DEQUEUE(&allocq, epp, next) -#define ROUND_UP_8(x) (((x) + (size_t)7) & ~(size_t)7) /* rounds up to multiple of 8 */ +#define ROUND_UP_8(x) (((x) + (size_t)7) & ~(size_t)7) /* rounds up to multiple of 8 */ static size_t segment_len = 0; -typedef struct asym_check_region -{ +typedef struct asym_check_region { void *base_ptr; OPA_int_t is_asym; } asym_check_region; -static asym_check_region* asym_check_region_p = NULL; +static asym_check_region *asym_check_region_p = NULL; -/* MPIDI_CH3I_Seg_alloc(len, ptr_p) +/* MPIDU_Seg_alloc(len, ptr_p) This function is used to allow the caller to reserve a len sized region in the shared memory segment. Once the shared memory @@ -67,48 +72,48 @@ static asym_check_region* asym_check_region_p = NULL; MPIDI_CH3I_Seg_commit() is called. */ #undef FUNCNAME -#define FUNCNAME MPIDI_CH3I_Seg_alloc +#define FUNCNAME MPIDU_Seg_alloc #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -int MPIDI_CH3I_Seg_alloc(size_t len, void **ptr_p) +int MPIDU_Seg_alloc(size_t len, void **ptr_p) { int mpi_errno = MPI_SUCCESS; alloc_elem_t *ep; - MPIR_CHKPMEM_DECL(1); - MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDI_CH3I_SEG_ALLOC); + MPIU_CHKPMEM_DECL(1); + MPIDI_STATE_DECL(MPID_STATE_MPIDI_CH3I_SEG_ALLOC); - MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDI_CH3I_SEG_ALLOC); + MPIDI_FUNC_ENTER(MPID_STATE_MPIDI_CH3I_SEG_ALLOC); /* round up to multiple of 8 to ensure the start of the next - region is 64-bit aligned. */ + * region is 64-bit aligned. */ len = ROUND_UP_8(len); - MPIR_Assert(len); - MPIR_Assert(ptr_p); + MPIU_Assert(len); + MPIU_Assert(ptr_p); + + MPIU_CHKPMEM_MALLOC(ep, alloc_elem_t *, sizeof(alloc_elem_t), mpi_errno, "el"); - MPIR_CHKPMEM_MALLOC(ep, alloc_elem_t *, sizeof(alloc_elem_t), mpi_errno, "el"); - ep->ptr_p = ptr_p; ep->len = len; ALLOCQ_ENQUEUE(ep); segment_len += len; - - fn_exit: - MPIR_CHKPMEM_COMMIT(); - MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDI_CH3I_SEG_ALLOC); + + fn_exit: + MPIU_CHKPMEM_COMMIT(); + MPIDI_FUNC_EXIT(MPID_STATE_MPIDI_CH3I_SEG_ALLOC); return mpi_errno; - fn_fail: - MPIR_CHKPMEM_REAP(); + fn_fail: + MPIU_CHKPMEM_REAP(); goto fn_exit; } -/* MPIDI_CH3I_Seg_commit(memory, num_local, local_rank) +/* MPIDU_Seg_commit(memory, num_local, local_rank) This function allocates a shared memory segment large enough to hold all of the regions previously requested by calls to - MPIDI_CH3I_Seg_alloc(). For each request, this function sets the + MPIDU_Seg_alloc(). For each request, this function sets the associated pointer to point to the reserved region in the allocated shared memory segment. @@ -120,10 +125,10 @@ int MPIDI_CH3I_Seg_alloc(size_t len, void **ptr_p) calling MPIDI_CH3I_Seg_commit(). */ #undef FUNCNAME -#define FUNCNAME MPIDI_CH3I_Seg_commit +#define FUNCNAME MPIDU_Seg_commit #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -int MPIDI_CH3I_Seg_commit(MPID_nem_seg_ptr_t memory, int num_local, int local_rank) +int MPIDU_Seg_commit(MPID_nem_seg_ptr_t memory, int num_local, int local_rank) { int mpi_errno = MPI_SUCCESS; int pmi_errno; @@ -141,35 +146,37 @@ int MPIDI_CH3I_Seg_commit(MPID_nem_seg_ptr_t memory, int num_local, int local_ra void *current_addr; void *start_addr ATTRIBUTE((unused)); size_t size_left; - MPIR_CHKPMEM_DECL (1); - MPIR_CHKLMEM_DECL (2); - MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDI_CH3I_SEG_COMMIT); + MPIU_CHKPMEM_DECL(1); + MPIU_CHKLMEM_DECL(2); + MPIDI_STATE_DECL(MPID_STATE_MPIDI_CH3I_SEG_COMMIT); - MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDI_CH3I_SEG_COMMIT); + MPIDI_FUNC_ENTER(MPID_STATE_MPIDI_CH3I_SEG_COMMIT); /* MPIDI_CH3I_Seg_alloc() needs to have been called before this function */ - MPIR_Assert(!ALLOCQ_EMPTY()); - MPIR_Assert(segment_len > 0); + MPIU_Assert(!ALLOCQ_EMPTY()); + MPIU_Assert(segment_len > 0); /* allocate an area to check if the segment was allocated symmetrically */ - mpi_errno = MPIDI_CH3I_Seg_alloc(sizeof(asym_check_region), (void **)&asym_check_region_p); - if (mpi_errno) MPIR_ERR_POP(mpi_errno); + mpi_errno = MPIDU_Seg_alloc(sizeof(asym_check_region), (void **) &asym_check_region_p); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); mpi_errno = MPIU_SHMW_Hnd_init(&(memory->hnd)); - if (mpi_errno != MPI_SUCCESS) MPIR_ERR_POP (mpi_errno); + if (mpi_errno != MPI_SUCCESS) + MPIR_ERR_POP(mpi_errno); /* Shared memory barrier variables are in the front of the shared - memory region. We do this here explicitly, rather than use the - Seg_alloc() function because we need to use the barrier inside - this function, before we've assigned the variables to their - regions. To do this, we add extra space to the segment_len, - initialize the variables as soon as the shared memory region is - allocated/attached, then before we do the assignments of the - pointers provided in Seg_alloc(), we make sure to skip the - region containing the barrier vars. */ - + * memory region. We do this here explicitly, rather than use the + * Seg_alloc() function because we need to use the barrier inside + * this function, before we've assigned the variables to their + * regions. To do this, we add extra space to the segment_len, + * initialize the variables as soon as the shared memory region is + * allocated/attached, then before we do the assignments of the + * pointers provided in Seg_alloc(), we make sure to skip the + * region containing the barrier vars. */ + /* add space for local barrier region. Use a whole cacheline. */ - MPIR_Assert(MPID_NEM_CACHE_LINE_LEN >= sizeof(MPID_nem_barrier_t)); + MPIU_Assert(MPID_NEM_CACHE_LINE_LEN >= sizeof(MPID_nem_barrier_t)); segment_len += MPID_NEM_CACHE_LINE_LEN; #ifdef OPA_USE_LOCK_BASED_PRIMITIVES @@ -182,7 +189,7 @@ int MPIDI_CH3I_Seg_commit(MPID_nem_seg_ptr_t memory, int num_local, int local_ra /* offset from memory->base_addr to the start of ipc_lock */ ipc_lock_offset = MPID_NEM_CACHE_LINE_LEN; - MPIR_Assert(ipc_lock_offset >= sizeof(OPA_emulation_ipl_t)); + MPIU_Assert(ipc_lock_offset >= sizeof(OPA_emulation_ipl_t)); segment_len += MPID_NEM_CACHE_LINE_LEN; #endif @@ -190,47 +197,55 @@ int MPIDI_CH3I_Seg_commit(MPID_nem_seg_ptr_t memory, int num_local, int local_ra #ifdef USE_PMI2_API /* if there is only one process on this processor, don't use shared memory */ - if (num_local == 1) - { + if (num_local == 1) { char *addr; - MPIR_CHKPMEM_MALLOC (addr, char *, segment_len + MPID_NEM_CACHE_LINE_LEN, mpi_errno, "segment"); + MPIU_CHKPMEM_MALLOC(addr, char *, segment_len + MPID_NEM_CACHE_LINE_LEN, mpi_errno, + "segment"); memory->base_addr = addr; - current_addr = (char *)(((uintptr_t)addr + (uintptr_t)MPID_NEM_CACHE_LINE_LEN-1) & (~((uintptr_t)MPID_NEM_CACHE_LINE_LEN-1))); + current_addr = + (char *) (((MPIU_Upint) addr + (MPIU_Upint) MPID_NEM_CACHE_LINE_LEN - 1) & + (~((MPIU_Upint) MPID_NEM_CACHE_LINE_LEN - 1))); memory->symmetrical = 0; /* must come before barrier_init since we use OPA in that function */ #ifdef OPA_USE_LOCK_BASED_PRIMITIVES - ipc_lock = (OPA_emulation_ipl_t *)((char *)memory->base_addr + ipc_lock_offset); - ret = OPA_Interprocess_lock_init(ipc_lock, TRUE/*isLeader*/); + ipc_lock = (OPA_emulation_ipl_t *) ((char *) memory->base_addr + ipc_lock_offset); + ret = OPA_Interprocess_lock_init(ipc_lock, TRUE /*isLeader */); MPIR_ERR_CHKANDJUMP1(ret != 0, mpi_errno, MPI_ERR_OTHER, "**fail", "**fail %d", ret); #endif - mpi_errno = MPID_nem_barrier_init((MPID_nem_barrier_t *)memory->base_addr, TRUE); - if (mpi_errno) MPIR_ERR_POP(mpi_errno); + mpi_errno = MPID_nem_barrier_init((MPID_nem_barrier_t *) memory->base_addr, TRUE); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); } else { if (local_rank == 0) { - mpi_errno = MPIU_SHMW_Seg_create_and_attach(memory->hnd, memory->segment_len, &(memory->base_addr), 0); - if (mpi_errno) MPIR_ERR_POP(mpi_errno); + mpi_errno = + MPIU_SHMW_Seg_create_and_attach(memory->hnd, memory->segment_len, + &(memory->base_addr), 0); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); /* post name of shared file */ - MPIR_Assert (MPID_nem_mem_region.local_procs[0] == MPID_nem_mem_region.rank); + MPIU_Assert(MPID_nem_mem_region.local_procs[0] == MPID_nem_mem_region.rank); mpi_errno = MPIU_SHMW_Hnd_get_serialized_by_ref(memory->hnd, &serialized_hnd); - if (mpi_errno) MPIR_ERR_POP(mpi_errno); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); /* must come before barrier_init since we use OPA in that function */ #ifdef OPA_USE_LOCK_BASED_PRIMITIVES - ipc_lock = (OPA_emulation_ipl_t *)((char *)memory->base_addr + ipc_lock_offset); + ipc_lock = (OPA_emulation_ipl_t *) ((char *) memory->base_addr + ipc_lock_offset); ret = OPA_Interprocess_lock_init(ipc_lock, local_rank == 0); MPIR_ERR_CHKANDJUMP1(ret != 0, mpi_errno, MPI_ERR_OTHER, "**fail", "**fail %d", ret); #endif - mpi_errno = MPID_nem_barrier_init((MPID_nem_barrier_t *)memory->base_addr, TRUE); - if (mpi_errno) MPIR_ERR_POP(mpi_errno); + mpi_errno = MPID_nem_barrier_init((MPID_nem_barrier_t *) memory->base_addr, TRUE); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); /* The opa and nem barrier initializations must come before we (the * leader) put the sharedFilename attribute. Since this is a @@ -238,27 +253,34 @@ int MPIDI_CH3I_Seg_commit(MPID_nem_seg_ptr_t memory, int num_local, int local_ra * ensures that these initializations have occurred before any peer * attempts to use the resources. */ mpi_errno = PMI2_Info_PutNodeAttr("sharedFilename", serialized_hnd); - if (mpi_errno) MPIR_ERR_POP(mpi_errno); - } else { + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + } + else { int found = FALSE; /* Allocate space for pmi key and val */ - MPIR_CHKLMEM_MALLOC(val, char *, PMI2_MAX_VALLEN, mpi_errno, "val"); + MPIU_CHKLMEM_MALLOC(val, char *, PMI2_MAX_VALLEN, mpi_errno, "val"); /* get name of shared file */ mpi_errno = PMI2_Info_GetNodeAttr("sharedFilename", val, PMI2_MAX_VALLEN, &found, TRUE); - if (mpi_errno) MPIR_ERR_POP(mpi_errno); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); MPIR_ERR_CHKINTERNAL(!found, mpi_errno, "nodeattr not found"); mpi_errno = MPIU_SHMW_Hnd_deserialize(memory->hnd, val, strlen(val)); - if (mpi_errno) MPIR_ERR_POP(mpi_errno); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); - mpi_errno = MPIU_SHMW_Seg_attach(memory->hnd, memory->segment_len, (char **)&memory->base_addr, 0); - if (mpi_errno) MPIR_ERR_POP(mpi_errno); + mpi_errno = + MPIU_SHMW_Seg_attach(memory->hnd, memory->segment_len, (char **) &memory->base_addr, + 0); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); /* must come before barrier_init since we use OPA in that function */ #ifdef OPA_USE_LOCK_BASED_PRIMITIVES - ipc_lock = (OPA_emulation_ipl_t *)((char *)memory->base_addr + ipc_lock_offset); + ipc_lock = (OPA_emulation_ipl_t *) ((char *) memory->base_addr + ipc_lock_offset); ret = OPA_Interprocess_lock_init(ipc_lock, local_rank == 0); MPIR_ERR_CHKANDJUMP1(ret != 0, mpi_errno, MPI_ERR_OTHER, "**fail", "**fail %d", ret); @@ -270,125 +292,150 @@ int MPIDI_CH3I_Seg_commit(MPID_nem_seg_ptr_t memory, int num_local, int local_ra * this behavior. */ #endif - mpi_errno = MPID_nem_barrier_init((MPID_nem_barrier_t *)memory->base_addr, FALSE); - if (mpi_errno) MPIR_ERR_POP(mpi_errno); + mpi_errno = MPID_nem_barrier_init((MPID_nem_barrier_t *) memory->base_addr, FALSE); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); } mpi_errno = MPID_nem_barrier(); - if (mpi_errno) MPIR_ERR_POP(mpi_errno); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); if (local_rank == 0) { mpi_errno = MPIU_SHMW_Seg_remove(memory->hnd); - if (mpi_errno) MPIR_ERR_POP(mpi_errno); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); } current_addr = memory->base_addr; - memory->symmetrical = 0 ; + memory->symmetrical = 0; } #else /* we are using PMIv1 */ /* if there is only one process on this processor, don't use shared memory */ - if (num_local == 1) - { + if (num_local == 1) { char *addr; - MPIR_CHKPMEM_MALLOC (addr, char *, segment_len + MPID_NEM_CACHE_LINE_LEN, mpi_errno, "segment"); + MPIU_CHKPMEM_MALLOC(addr, char *, segment_len + MPID_NEM_CACHE_LINE_LEN, mpi_errno, + "segment"); memory->base_addr = addr; - current_addr = (char *)(((uintptr_t)addr + (uintptr_t)MPID_NEM_CACHE_LINE_LEN-1) & (~((uintptr_t)MPID_NEM_CACHE_LINE_LEN-1))); - memory->symmetrical = 0 ; + current_addr = + (char *) (((MPIU_Upint) addr + (MPIU_Upint) MPID_NEM_CACHE_LINE_LEN - 1) & + (~((MPIU_Upint) MPID_NEM_CACHE_LINE_LEN - 1))); + memory->symmetrical = 0; /* we still need to call barrier */ - pmi_errno = PMI_Barrier(); - MPIR_ERR_CHKANDJUMP1 (pmi_errno != PMI_SUCCESS, mpi_errno, MPI_ERR_OTHER, "**pmi_barrier", "**pmi_barrier %d", pmi_errno); + pmi_errno = PMI_Barrier(); + MPIR_ERR_CHKANDJUMP1(pmi_errno != PMI_SUCCESS, mpi_errno, MPI_ERR_OTHER, "**pmi_barrier", + "**pmi_barrier %d", pmi_errno); /* must come before barrier_init since we use OPA in that function */ #ifdef OPA_USE_LOCK_BASED_PRIMITIVES - ipc_lock = (OPA_emulation_ipl_t *)((char *)memory->base_addr + ipc_lock_offset); - ret = OPA_Interprocess_lock_init(ipc_lock, TRUE/*isLeader*/); + ipc_lock = (OPA_emulation_ipl_t *) ((char *) memory->base_addr + ipc_lock_offset); + ret = OPA_Interprocess_lock_init(ipc_lock, TRUE /*isLeader */); MPIR_ERR_CHKANDJUMP1(ret != 0, mpi_errno, MPI_ERR_OTHER, "**fail", "**fail %d", ret); #endif - mpi_errno = MPID_nem_barrier_init((MPID_nem_barrier_t *)memory->base_addr, TRUE); - if (mpi_errno) MPIR_ERR_POP(mpi_errno); + mpi_errno = MPID_nem_barrier_init((MPID_nem_barrier_t *) memory->base_addr, TRUE); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); } - else{ + else { /* Allocate space for pmi key and val */ pmi_errno = PMI_KVS_Get_key_length_max(&key_max_sz); MPIR_ERR_CHKANDJUMP1(pmi_errno, mpi_errno, MPI_ERR_OTHER, "**fail", "**fail %d", pmi_errno); - MPIR_CHKLMEM_MALLOC(key, char *, key_max_sz, mpi_errno, "key"); + MPIU_CHKLMEM_MALLOC(key, char *, key_max_sz, mpi_errno, "key"); pmi_errno = PMI_KVS_Get_value_length_max(&val_max_sz); MPIR_ERR_CHKANDJUMP1(pmi_errno, mpi_errno, MPI_ERR_OTHER, "**fail", "**fail %d", pmi_errno); - MPIR_CHKLMEM_MALLOC(val, char *, val_max_sz, mpi_errno, "val"); + MPIU_CHKLMEM_MALLOC(val, char *, val_max_sz, mpi_errno, "val"); - mpi_errno = MPIDI_PG_GetConnKVSname (&kvs_name); - if (mpi_errno) MPIR_ERR_POP (mpi_errno); +// mpi_errno = MPIDI_PG_GetConnKVSname (&kvs_name); + kvs_name = "my_kvs_name"; + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); - if (local_rank == 0){ - mpi_errno = MPIU_SHMW_Seg_create_and_attach(memory->hnd, memory->segment_len, &(memory->base_addr), 0); - if (mpi_errno != MPI_SUCCESS) MPIR_ERR_POP (mpi_errno); + if (local_rank == 0) { + mpi_errno = + MPIU_SHMW_Seg_create_and_attach(memory->hnd, memory->segment_len, + &(memory->base_addr), 0); + if (mpi_errno != MPI_SUCCESS) + MPIR_ERR_POP(mpi_errno); /* post name of shared file */ - MPIR_Assert (MPID_nem_mem_region.local_procs[0] == MPID_nem_mem_region.rank); - MPL_snprintf (key, key_max_sz, "sharedFilename[%i]", MPID_nem_mem_region.rank); + MPIU_Assert(MPID_nem_mem_region.local_procs[0] == MPID_nem_mem_region.rank); + MPL_snprintf(key, key_max_sz, "sharedFilename[%i]", MPID_nem_mem_region.rank); mpi_errno = MPIU_SHMW_Hnd_get_serialized_by_ref(memory->hnd, &serialized_hnd); - if (mpi_errno != MPI_SUCCESS) MPIR_ERR_POP (mpi_errno); + if (mpi_errno != MPI_SUCCESS) + MPIR_ERR_POP(mpi_errno); - pmi_errno = PMI_KVS_Put (kvs_name, key, serialized_hnd); - MPIR_ERR_CHKANDJUMP1 (pmi_errno != PMI_SUCCESS, mpi_errno, MPI_ERR_OTHER, "**pmi_kvs_put", "**pmi_kvs_put %d", pmi_errno); + pmi_errno = PMI_KVS_Put(kvs_name, key, serialized_hnd); + MPIR_ERR_CHKANDJUMP1(pmi_errno != PMI_SUCCESS, mpi_errno, MPI_ERR_OTHER, + "**pmi_kvs_put", "**pmi_kvs_put %d", pmi_errno); - pmi_errno = PMI_KVS_Commit (kvs_name); - MPIR_ERR_CHKANDJUMP1 (pmi_errno != PMI_SUCCESS, mpi_errno, MPI_ERR_OTHER, "**pmi_kvs_commit", "**pmi_kvs_commit %d", pmi_errno); + pmi_errno = PMI_KVS_Commit(kvs_name); + MPIR_ERR_CHKANDJUMP1(pmi_errno != PMI_SUCCESS, mpi_errno, MPI_ERR_OTHER, + "**pmi_kvs_commit", "**pmi_kvs_commit %d", pmi_errno); /* must come before barrier_init since we use OPA in that function */ #ifdef OPA_USE_LOCK_BASED_PRIMITIVES - ipc_lock = (OPA_emulation_ipl_t *)((char *)memory->base_addr + ipc_lock_offset); + ipc_lock = (OPA_emulation_ipl_t *) ((char *) memory->base_addr + ipc_lock_offset); ret = OPA_Interprocess_lock_init(ipc_lock, local_rank == 0); MPIR_ERR_CHKANDJUMP1(ret != 0, mpi_errno, MPI_ERR_OTHER, "**fail", "**fail %d", ret); #endif - mpi_errno = MPID_nem_barrier_init((MPID_nem_barrier_t *)memory->base_addr, TRUE); - if (mpi_errno) MPIR_ERR_POP(mpi_errno); + mpi_errno = MPID_nem_barrier_init((MPID_nem_barrier_t *) memory->base_addr, TRUE); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); pmi_errno = PMI_Barrier(); - MPIR_ERR_CHKANDJUMP1 (pmi_errno != PMI_SUCCESS, mpi_errno, MPI_ERR_OTHER, "**pmi_barrier", "**pmi_barrier %d", pmi_errno); + MPIR_ERR_CHKANDJUMP1(pmi_errno != PMI_SUCCESS, mpi_errno, MPI_ERR_OTHER, + "**pmi_barrier", "**pmi_barrier %d", pmi_errno); } - else - { + else { pmi_errno = PMI_Barrier(); - MPIR_ERR_CHKANDJUMP1 (pmi_errno != PMI_SUCCESS, mpi_errno, MPI_ERR_OTHER, "**pmi_barrier", "**pmi_barrier %d", pmi_errno); + MPIR_ERR_CHKANDJUMP1(pmi_errno != PMI_SUCCESS, mpi_errno, MPI_ERR_OTHER, + "**pmi_barrier", "**pmi_barrier %d", pmi_errno); /* get name of shared file */ - MPL_snprintf (key, key_max_sz, "sharedFilename[%i]", MPID_nem_mem_region.local_procs[0]); - pmi_errno = PMI_KVS_Get (kvs_name, key, val, val_max_sz); - MPIR_ERR_CHKANDJUMP1 (pmi_errno != PMI_SUCCESS, mpi_errno, MPI_ERR_OTHER, "**pmi_kvs_get", "**pmi_kvs_get %d", pmi_errno); + MPL_snprintf(key, key_max_sz, "sharedFilename[%i]", MPID_nem_mem_region.local_procs[0]); + pmi_errno = PMI_KVS_Get(kvs_name, key, val, val_max_sz); + MPIR_ERR_CHKANDJUMP1(pmi_errno != PMI_SUCCESS, mpi_errno, MPI_ERR_OTHER, + "**pmi_kvs_get", "**pmi_kvs_get %d", pmi_errno); mpi_errno = MPIU_SHMW_Hnd_deserialize(memory->hnd, val, strlen(val)); - if(mpi_errno != MPI_SUCCESS) MPIR_ERR_POP(mpi_errno); + mpi_errno = MPIU_SHMW_Hnd_deserialize(memory->hnd, val, strlen(val)); + if (mpi_errno != MPI_SUCCESS) + MPIR_ERR_POP(mpi_errno); - mpi_errno = MPIU_SHMW_Seg_attach(memory->hnd, memory->segment_len, (char **)&memory->base_addr, 0); - if (mpi_errno) MPIR_ERR_POP (mpi_errno); + mpi_errno = + MPIU_SHMW_Seg_attach(memory->hnd, memory->segment_len, (char **) &memory->base_addr, + 0); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); /* must come before barrier_init since we use OPA in that function */ #ifdef OPA_USE_LOCK_BASED_PRIMITIVES - ipc_lock = (OPA_emulation_ipl_t *)((char *)memory->base_addr + ipc_lock_offset); + ipc_lock = (OPA_emulation_ipl_t *) ((char *) memory->base_addr + ipc_lock_offset); ret = OPA_Interprocess_lock_init(ipc_lock, local_rank == 0); MPIR_ERR_CHKANDJUMP1(ret != 0, mpi_errno, MPI_ERR_OTHER, "**fail", "**fail %d", ret); #endif - mpi_errno = MPID_nem_barrier_init((MPID_nem_barrier_t *)memory->base_addr, FALSE); - if (mpi_errno) MPIR_ERR_POP(mpi_errno); + mpi_errno = MPID_nem_barrier_init((MPID_nem_barrier_t *) memory->base_addr, FALSE); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); } mpi_errno = MPID_nem_barrier(); - if (mpi_errno) MPIR_ERR_POP(mpi_errno); - - if (local_rank == 0) - { + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + if (local_rank == 0) { mpi_errno = MPIU_SHMW_Seg_remove(memory->hnd); - if (mpi_errno != MPI_SUCCESS) MPIR_ERR_POP (mpi_errno); + if (mpi_errno != MPI_SUCCESS) + MPIR_ERR_POP(mpi_errno); } current_addr = memory->base_addr; - memory->symmetrical = 0 ; + memory->symmetrical = 0; } #endif /* assign sections of the shared memory segment to their pointers */ @@ -397,77 +444,78 @@ int MPIDI_CH3I_Seg_commit(MPID_nem_seg_ptr_t memory, int num_local, int local_ra size_left = segment_len; /* reserve room for shared mem barrier (We used a whole cacheline) */ - current_addr = (char *)current_addr + MPID_NEM_CACHE_LINE_LEN; - MPIR_Assert(size_left >= MPID_NEM_CACHE_LINE_LEN); + current_addr = (char *) current_addr + MPID_NEM_CACHE_LINE_LEN; + MPIU_Assert(size_left >= MPID_NEM_CACHE_LINE_LEN); size_left -= MPID_NEM_CACHE_LINE_LEN; #ifdef OPA_USE_LOCK_BASED_PRIMITIVES /* reserve room for the opa emulation lock */ - current_addr = (char *)current_addr + MPID_NEM_CACHE_LINE_LEN; - MPIR_Assert(size_left >= MPID_NEM_CACHE_LINE_LEN); + current_addr = (char *) current_addr + MPID_NEM_CACHE_LINE_LEN; + MPIU_Assert(size_left >= MPID_NEM_CACHE_LINE_LEN); size_left -= MPID_NEM_CACHE_LINE_LEN; #endif - do - { + do { alloc_elem_t *ep; ALLOCQ_DEQUEUE(&ep); *(ep->ptr_p) = current_addr; - MPIR_Assert(size_left >= ep->len); + MPIU_Assert(size_left >= ep->len); size_left -= ep->len; - current_addr = (char *)current_addr + ep->len; + current_addr = (char *) current_addr + ep->len; MPL_free(ep); - MPIR_Assert((char *)current_addr <= (char *)start_addr + segment_len); + MPIU_Assert((char *) current_addr <= (char *) start_addr + segment_len); } while (!ALLOCQ_EMPTY()); mpi_errno = check_alloc(num_local, local_rank); - if (mpi_errno) MPIR_ERR_POP(mpi_errno); - - MPIR_CHKPMEM_COMMIT(); - fn_exit: - MPIR_CHKLMEM_FREEALL(); - MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDI_CH3I_SEG_COMMIT); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); + + MPIU_CHKPMEM_COMMIT(); + fn_exit: + MPIU_CHKLMEM_FREEALL(); + MPIDI_FUNC_EXIT(MPID_STATE_MPIDI_CH3I_SEG_COMMIT); return mpi_errno; - fn_fail: + fn_fail: /* --BEGIN ERROR HANDLING-- */ MPIU_SHMW_Seg_remove(memory->hnd); MPIU_SHMW_Hnd_finalize(&(memory->hnd)); - MPIR_CHKPMEM_REAP(); + MPIU_CHKPMEM_REAP(); goto fn_exit; /* --END ERROR HANDLING-- */ } /* MPIDI_CH3I_Seg_destroy() free the shared memory segment */ #undef FUNCNAME -#define FUNCNAME MPIDI_CH3I_Seg_destroy +#define FUNCNAME MPIDU_Seg_destroy #undef FCNAME #define FCNAME MPL_QUOTE(FUNCNAME) -int MPIDI_CH3I_Seg_destroy(void) +int MPIDU_Seg_destroy(void) { int mpi_errno = MPI_SUCCESS; - MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_MPIDI_CH3I_SEG_DESTROY); + MPIDI_STATE_DECL(MPID_STATE_MPIDI_CH3I_SEG_DESTROY); - MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_MPIDI_CH3I_SEG_DESTROY); + MPIDI_FUNC_ENTER(MPID_STATE_MPIDI_CH3I_SEG_DESTROY); if (MPID_nem_mem_region.num_local == 1) MPL_free(MPID_nem_mem_region.memory.base_addr); - else - { - mpi_errno = MPIU_SHMW_Seg_detach(MPID_nem_mem_region.memory.hnd, - &(MPID_nem_mem_region.memory.base_addr), MPID_nem_mem_region.memory.segment_len); - if (mpi_errno) MPIR_ERR_POP (mpi_errno); + else { + mpi_errno = MPIU_SHMW_Seg_detach(MPID_nem_mem_region.memory.hnd, + &(MPID_nem_mem_region.memory.base_addr), + MPID_nem_mem_region.memory.segment_len); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); } - fn_exit: + fn_exit: MPIU_SHMW_Hnd_finalize(&(MPID_nem_mem_region.memory.hnd)); - MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_MPIDI_CH3I_SEG_DESTROY); + MPIDI_FUNC_EXIT(MPID_STATE_MPIDI_CH3I_SEG_DESTROY); return mpi_errno; - fn_fail: + fn_fail: goto fn_exit; } @@ -481,9 +529,9 @@ int MPIDI_CH3I_Seg_destroy(void) static int check_alloc(int num_local, int local_rank) { int mpi_errno = MPI_SUCCESS; - MPIR_FUNC_VERBOSE_STATE_DECL(MPID_STATE_CHECK_ALLOC); + MPIDI_STATE_DECL(MPID_STATE_CHECK_ALLOC); - MPIR_FUNC_VERBOSE_ENTER(MPID_STATE_CHECK_ALLOC); + MPIDI_FUNC_ENTER(MPID_STATE_CHECK_ALLOC); if (local_rank == 0) { asym_check_region_p->base_ptr = MPID_nem_mem_region.memory.base_addr; @@ -491,7 +539,8 @@ static int check_alloc(int num_local, int local_rank) } mpi_errno = MPID_nem_barrier(); - if (mpi_errno) MPIR_ERR_POP(mpi_errno); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); if (asym_check_region_p->base_ptr != MPID_nem_mem_region.memory.base_addr) OPA_store_int(&asym_check_region_p->is_asym, 1); @@ -499,26 +548,24 @@ static int check_alloc(int num_local, int local_rank) OPA_read_write_barrier(); mpi_errno = MPID_nem_barrier(); - if (mpi_errno) MPIR_ERR_POP(mpi_errno); + if (mpi_errno) + MPIR_ERR_POP(mpi_errno); - if (OPA_load_int(&asym_check_region_p->is_asym)) - { - MPID_nem_mem_region.memory.symmetrical = 0; - MPID_nem_asymm_base_addr = MPID_nem_mem_region.memory.base_addr; + if (OPA_load_int(&asym_check_region_p->is_asym)) { + MPID_nem_mem_region.memory.symmetrical = 0; + MPID_nem_asymm_base_addr = MPID_nem_mem_region.memory.base_addr; #ifdef MPID_NEM_SYMMETRIC_QUEUES MPIR_ERR_INTERNALANDJUMP(mpi_errno, "queues are not symmetrically allocated as expected"); #endif } - else - { - MPID_nem_mem_region.memory.symmetrical = 1; - MPID_nem_asymm_base_addr = NULL; + else { + MPID_nem_mem_region.memory.symmetrical = 1; + MPID_nem_asymm_base_addr = NULL; } - - fn_exit: - MPIR_FUNC_VERBOSE_EXIT(MPID_STATE_CHECK_ALLOC); + + fn_exit: + MPIDI_FUNC_EXIT(MPID_STATE_CHECK_ALLOC); return mpi_errno; - fn_fail: + fn_fail: goto fn_exit; } - ----------------------------------------------------------------------- Summary of changes: src/mpid/ch3/channels/nemesis/Makefile.mk | 1 - .../ch3/channels/nemesis/include/mpid_nem_defs.h | 36 +- .../nemesis/include/mpid_nem_generic_queue.h | 339 ------- .../ch3/channels/nemesis/include/mpid_nem_impl.h | 4 - .../ch3/channels/nemesis/include/mpid_nem_post.h | 4 +- .../ch3/channels/nemesis/include/mpid_nem_pre.h | 6 +- .../ch3/channels/nemesis/include/mpidi_ch3_impl.h | 3 +- .../ch3/channels/nemesis/include/mpidi_ch3_pre.h | 6 +- src/mpid/ch3/channels/nemesis/src/Makefile.mk | 2 - src/mpid/ch3/channels/nemesis/src/ch3_progress.c | 4 +- src/mpid/ch3/channels/nemesis/src/ch3_rma_shm.c | 12 +- src/mpid/ch3/channels/nemesis/src/ch3_win_fns.c | 54 +- src/mpid/ch3/channels/nemesis/src/mpid_nem_alloc.c | 524 ----------- .../ch3/channels/nemesis/src/mpid_nem_barrier.c | 72 -- src/mpid/ch3/channels/nemesis/src/mpid_nem_ckpt.c | 4 +- .../ch3/channels/nemesis/src/mpid_nem_finalize.c | 2 +- src/mpid/ch3/channels/nemesis/src/mpid_nem_init.c | 54 +- .../ch3/channels/nemesis/src/mpid_nem_lmt_shm.c | 39 +- src/mpid/ch3/channels/nemesis/subconfigure.m4 | 11 +- src/mpid/common/Makefile.mk | 1 + src/mpid/common/shm/Makefile.mk | 22 + src/mpid/common/shm/mpidu_generic_queue.h | 339 +++++++ src/mpid/common/shm/mpidu_shm.h | 45 + src/mpid/common/shm/mpidu_shm_alloc.c | 526 +++++++++++ src/mpid/common/shm/mpidu_shm_barrier.c | 72 ++ src/mpid/common/shm/mpidu_shm_impl.h | 14 + src/mpid/common/shm/mpidu_shm_states.h | 19 + src/mpid/common/shm/subconfigure.m4 | 21 + src/mpid/pamid/subconfigure.m4 | 4 +- src/mpl/Makefile.am | 3 +- src/mpl/configure.ac | 25 + src/mpl/include/mpl.h | 1 + src/mpl/include/mpl_shm.h | 160 ++++ src/mpl/include/mpl_shm_mmap.h | 36 + src/mpl/include/mpl_shm_sysv.h | 38 + src/mpl/include/mpl_shm_win.h | 70 ++ src/mpl/src/Makefile.mk | 1 + src/mpl/src/shm/Makefile.mk | 12 + src/mpl/src/shm/mpl_shm.c | 112 +++ src/mpl/src/shm/mpl_shm_mmap.c | 188 ++++ src/mpl/src/shm/mpl_shm_sysv.c | 152 ++++ src/mpl/src/shm/mpl_shm_win.c | 137 +++ src/util/wrappers/Makefile.mk | 3 - src/util/wrappers/mpiu_os_wrappers.h | 13 - src/util/wrappers/mpiu_os_wrappers_pre.h | 61 -- src/util/wrappers/mpiu_shm_wrappers.h | 938 -------------------- src/util/wrappers/mpiu_util_wrappers.h | 13 - 47 files changed, 2106 insertions(+), 2097 deletions(-) delete mode 100644 src/mpid/ch3/channels/nemesis/include/mpid_nem_generic_queue.h delete mode 100644 src/mpid/ch3/channels/nemesis/src/mpid_nem_alloc.c delete mode 100644 src/mpid/ch3/channels/nemesis/src/mpid_nem_barrier.c create mode 100644 src/mpid/common/shm/Makefile.mk create mode 100644 src/mpid/common/shm/mpidu_generic_queue.h create mode 100644 src/mpid/common/shm/mpidu_shm.h create mode 100644 src/mpid/common/shm/mpidu_shm_alloc.c create mode 100644 src/mpid/common/shm/mpidu_shm_barrier.c create mode 100644 src/mpid/common/shm/mpidu_shm_impl.h create mode 100644 src/mpid/common/shm/mpidu_shm_states.h create mode 100644 src/mpid/common/shm/subconfigure.m4 create mode 100644 src/mpl/include/mpl_shm.h create mode 100644 src/mpl/include/mpl_shm_mmap.h create mode 100644 src/mpl/include/mpl_shm_sysv.h create mode 100644 src/mpl/include/mpl_shm_win.h create mode 100644 src/mpl/src/shm/Makefile.mk create mode 100644 src/mpl/src/shm/mpl_shm.c create mode 100644 src/mpl/src/shm/mpl_shm_mmap.c create mode 100644 src/mpl/src/shm/mpl_shm_sysv.c create mode 100644 src/mpl/src/shm/mpl_shm_win.c delete mode 100644 src/util/wrappers/mpiu_os_wrappers.h delete mode 100644 src/util/wrappers/mpiu_os_wrappers_pre.h delete mode 100644 src/util/wrappers/mpiu_shm_wrappers.h hooks/post-receive -- MPICH primary repository

1 0