August 2014 - commits - mailman3.cels.anl.gov

[mpich] MPICH primary repository branch, master, updated. v3.1.2-97-g29323e5
by noreply＠mpich.org 19 Aug '14

19 Aug '14

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "MPICH primary repository". The branch, master has been updated via 29323e596753dbb455d8c242dd47a5bd8daefde9 (commit) from c4ee1fbe686735d42a05cfe06e867da1e2d93536 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- http://git.mpich.org/mpich.git/commitdiff/29323e596753dbb455d8c242dd47a5bd8… commit 29323e596753dbb455d8c242dd47a5bd8daefde9 Author: Xin Zhao <xinzhao3(a)illinois.edu> Date: Wed Aug 13 11:09:28 2014 -0500 Free SHM comm for platforms without SHM support In test/mpi/rma/win_shared_zerobyte.c, free SHM comm handle even for the platforms without SHM support. In such case, SHM comm created is comm_self. Signed-off-by: Min Si <msi(a)il.is.s.u-tokyo.ac.jp> Signed-off-by: Antonio J. Pena <apenya(a)mcs.anl.gov> diff --git a/test/mpi/rma/win_shared_zerobyte.c b/test/mpi/rma/win_shared_zerobyte.c index 4bec8e5..2b9eac1 100644 --- a/test/mpi/rma/win_shared_zerobyte.c +++ b/test/mpi/rma/win_shared_zerobyte.c @@ -96,7 +96,6 @@ int main(int argc, char **argv) MPI_Win_unlock_all(shm_win); MPI_Win_free(&shm_win); - MPI_Comm_free(&shm_comm); MPI_Reduce(&errors, &all_errors, 1, MPI_INT, MPI_SUM, 0, MPI_COMM_WORLD); @@ -105,6 +104,8 @@ int main(int argc, char **argv) if (rank == 0 && all_errors == 0) printf(" No Errors\n"); + MPI_Comm_free(&shm_comm); + MPI_Finalize(); if (bases) ----------------------------------------------------------------------- Summary of changes: test/mpi/rma/win_shared_zerobyte.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-) hooks/post-receive -- MPICH primary repository

1 0

[mpich] MPICH primary repository branch, master, updated. v3.1.2-96-gc4ee1fb
by noreply＠mpich.org 18 Aug '14

18 Aug '14

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "MPICH primary repository". The branch, master has been updated via c4ee1fbe686735d42a05cfe06e867da1e2d93536 (commit) via 14958c758e435ad246e8282ccbb72e12eec94acc (commit) via b6564c6374ce3ae9d553f212d1f707aabffb84c2 (commit) from e8585d7c7f5666a29bf9d74ef6daa56db84f4f86 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- http://git.mpich.org/mpich.git/commitdiff/c4ee1fbe686735d42a05cfe06e867da1e… commit c4ee1fbe686735d42a05cfe06e867da1e2d93536 Author: Norio Yamaguchi <norio.yamaguchi(a)riken.jp> Date: Fri Aug 8 14:11:49 2014 +0900 Fix memory leak in netmod-IB Some memories allocated in an initialization process are not released when using shared memory for communication. Signed-off-by: Pavan Balaji <balaji(a)anl.gov> diff --git a/src/mpid/ch3/channels/nemesis/netmod/ib/ib_init.c b/src/mpid/ch3/channels/nemesis/netmod/ib/ib_init.c index 35b2ab2..fba362b 100644 --- a/src/mpid/ch3/channels/nemesis/netmod/ib/ib_init.c +++ b/src/mpid/ch3/channels/nemesis/netmod/ib/ib_init.c @@ -604,7 +604,16 @@ int MPID_nem_ib_init(MPIDI_PG_t * pg_p, int pg_rank, char **bc_val_p, int *val_m #endif #else /* define(MPID_NEM_IB_ONDEMAND) */ /* We need to communicate with all other ranks in close sequence. */ - MPID_nem_ib_conns_ref_count = MPID_nem_ib_nranks - 1; + MPID_nem_ib_conns_ref_count = MPID_nem_ib_nranks - MPID_nem_mem_region.num_local; + + for (i = 0; i < MPID_nem_mem_region.num_local; i++) { + if (MPID_nem_mem_region.local_procs[i] != MPID_nem_ib_myrank) { + ibcom_errno = + MPID_nem_ib_com_close(MPID_nem_ib_scratch_pad_fds + [MPID_nem_mem_region.local_procs[i]]); + MPID_nem_ib_scratch_pad_fds_ref_count--; + } + } #endif MPIU_Free(remote_rank_str); http://git.mpich.org/mpich.git/commitdiff/14958c758e435ad246e8282ccbb72e12e… commit 14958c758e435ad246e8282ccbb72e12eec94acc Author: Norio Yamaguchi <norio.yamaguchi(a)riken.jp> Date: Mon Aug 4 16:15:07 2014 +0900 Fix memory release operation in race-condition There is a case to operate a memory released in race-condition of CM-operation. In order to avoid it, set ref_count to 3 and release the memory after completing the operation. Signed-off-by: Pavan Balaji <balaji(a)anl.gov> diff --git a/src/mpid/ch3/channels/nemesis/netmod/ib/ib_poll.c b/src/mpid/ch3/channels/nemesis/netmod/ib/ib_poll.c index 710c1de..7523127 100644 --- a/src/mpid/ch3/channels/nemesis/netmod/ib/ib_poll.c +++ b/src/mpid/ch3/channels/nemesis/netmod/ib/ib_poll.c @@ -2561,6 +2561,7 @@ int MPID_nem_ib_cm_drain_scq() dprintf("cm_drain_scq,tx=%d\n", shadow_cm->req->ibcom->outstanding_connection_tx); dprintf("cm_drain_scq,syn,buf_from=%p,sz=%d\n", shadow_cm->buf_from, shadow_cm->buf_from_sz); + MPID_nem_ib_cm_request_release(shadow_cm->req); MPID_nem_ib_rdmawr_from_free(shadow_cm->buf_from, shadow_cm->buf_from_sz); MPIU_Free(shadow_cm); break; @@ -3228,9 +3229,11 @@ int MPID_nem_ib_cm_poll() MPID_nem_ib_send_progress(MPID_nem_ib_conns[req->responder_rank].vc); /* Let the following connection request go */ VC_FIELD(MPID_nem_ib_conns[req->responder_rank].vc, connection_guard) = 0; - /* free memory : req->ref_count is 2, so call MPIU_Free() directly */ - //MPID_nem_ib_cm_request_release(req); - MPIU_Free(req); + /* Call cm_request_release twice. + * If ref_count == 2, the memory of request is released here. + * If ref_count == 3, the memory of request will be released on draining SCQ of SYN. */ + MPID_nem_ib_cm_request_release(req); + MPID_nem_ib_cm_request_release(req); } //goto common_tail; break; diff --git a/src/mpid/ch3/channels/nemesis/netmod/ib/ib_send.c b/src/mpid/ch3/channels/nemesis/netmod/ib/ib_send.c index 66ef497..901dbd8 100644 --- a/src/mpid/ch3/channels/nemesis/netmod/ib/ib_send.c +++ b/src/mpid/ch3/channels/nemesis/netmod/ib/ib_send.c @@ -1677,7 +1677,7 @@ int MPID_nem_ib_cm_cas(MPIDI_VC_t * vc, uint32_t ask_on_connect) MPIU_ERR_CHKANDJUMP(!req, mpi_errno, MPI_ERR_OTHER, "**malloc"); dprintf("req=%p\n", req); req->state = MPID_NEM_IB_CM_CAS; - req->ref_count = 2; /* Released on receiving ACK2 and draining SCQ of ACK1 */ + req->ref_count = 3; /* Released on receiving ACK2 and draining SCQ of SYN and ACK1 */ req->retry_backoff = 0; req->initiator_rank = MPID_nem_ib_myrank; req->responder_rank = vc->pg_rank; http://git.mpich.org/mpich.git/commitdiff/b6564c6374ce3ae9d553f212d1f707aab… commit b6564c6374ce3ae9d553f212d1f707aabffb84c2 Author: Masamichi Takagi <masamichi.takagi(a)riken.jp> Date: Fri Aug 1 19:08:35 2014 +0900 Remove always-true comparison Remove (0 <= <variable>) comparison using a variable with the type of uint16_t to eliminate a compile-time warning. Signed-off-by: Pavan Balaji <balaji(a)anl.gov> diff --git a/src/mpid/ch3/channels/nemesis/netmod/ib/ib_poll.c b/src/mpid/ch3/channels/nemesis/netmod/ib/ib_poll.c index 4195152..710c1de 100644 --- a/src/mpid/ch3/channels/nemesis/netmod/ib/ib_poll.c +++ b/src/mpid/ch3/channels/nemesis/netmod/ib/ib_poll.c @@ -1477,7 +1477,7 @@ int MPID_nem_ib_recv_buf_released(struct MPIDI_VC *vc, void *user_data) (unsigned long) ((uint8_t *) user_data - (uint8_t *) vc_ib->ibcom->remote_ringbuf->start) / MPID_NEM_IB_COM_RDMABUF_SZSEG; - MPIU_Assert(0 <= index_slot && index_slot < (uint16_t) (vc_ib->ibcom->remote_ringbuf->nslot)); + MPIU_Assert(index_slot < (uint16_t) (vc_ib->ibcom->remote_ringbuf->nslot)); dprintf("released,user_data=%p,mem=%p,sub=%08lx,index_slot=%d\n", user_data, vc_ib->ibcom->remote_ringbuf->start, (unsigned long) user_data - ----------------------------------------------------------------------- Summary of changes: src/mpid/ch3/channels/nemesis/netmod/ib/ib_init.c | 11 ++++++++++- src/mpid/ch3/channels/nemesis/netmod/ib/ib_poll.c | 11 +++++++---- src/mpid/ch3/channels/nemesis/netmod/ib/ib_send.c | 2 +- 3 files changed, 18 insertions(+), 6 deletions(-) hooks/post-receive -- MPICH primary repository

1 0

[mpich] MPICH primary repository branch, master, updated. v3.1.2-93-ge8585d7
by noreply＠mpich.org 13 Aug '14

13 Aug '14

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "MPICH primary repository". The branch, master has been updated via e8585d7c7f5666a29bf9d74ef6daa56db84f4f86 (commit) from f8bd87eb68621ecca31f07042f7e3548766d75a8 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- http://git.mpich.org/mpich.git/commitdiff/e8585d7c7f5666a29bf9d74ef6daa56db… commit e8585d7c7f5666a29bf9d74ef6daa56db84f4f86 Author: Wesley Bland <wbland(a)anl.gov> Date: Wed Aug 13 10:17:08 2014 -0500 Mark multi_isendalive as xfail for now. diff --git a/test/mpi/ft/testlist b/test/mpi/ft/testlist index eda8f96..b60859e 100644 --- a/test/mpi/ft/testlist +++ b/test/mpi/ft/testlist @@ -2,7 +2,7 @@ die 4 mpiexecarg=-disable-auto-cleanup timeLimit=10 strict=false resultTest=Test abort 2 mpiexecarg=-disable-auto-cleanup timeLimit=10 strict=false xfail=ticket1537 sendalive 4 mpiexecarg=-disable-auto-cleanup timeLimit=10 strict=false resultTest=TestStatusNoErrors xfail=ticket1945 isendalive 3 mpiexecarg=-disable-auto-cleanup resultTest=TestStatusNoErrors strict=false xfail=ticket1945 -multi_isendalive 4 mpiexecarg=-disable-auto-cleanup resultTest=TestStatusNoErrors strict=false +multi_isendalive 4 mpiexecarg=-disable-auto-cleanup resultTest=TestStatusNoErrors strict=false xfail=ticket1945 senddead 2 mpiexecarg=-disable-auto-cleanup resultTest=TestStatusNoErrors strict=false timeLimit=10 xfail=ticket1945 recvdead 2 mpiexecarg=-disable-auto-cleanup resultTest=TestStatusNoErrors strict=false timeLimit=10 xfail=ticket1945 isenddead 2 mpiexecarg=-disable-auto-cleanup resultTest=TestStatusNoErrors strict=false timeLimit=10 xfail=ticket1945 ----------------------------------------------------------------------- Summary of changes: test/mpi/ft/testlist | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) hooks/post-receive -- MPICH primary repository

1 0

[mpich] MPICH primary repository branch, master, updated. v3.1.2-92-gf8bd87e
by noreply＠mpich.org 12 Aug '14

12 Aug '14

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "MPICH primary repository". The branch, master has been updated via f8bd87eb68621ecca31f07042f7e3548766d75a8 (commit) from 04a1907aaed46486857f995e4b8332cfdcd865e7 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- http://git.mpich.org/mpich.git/commitdiff/f8bd87eb68621ecca31f07042f7e35487… commit f8bd87eb68621ecca31f07042f7e3548766d75a8 Author: Min Si <msi(a)il.is.s.u-tokyo.ac.jp> Date: Sat Aug 9 18:58:58 2014 -0500 Add process number check for SHM communicator. Some platforms do not support shared memory, they will create a SHM communicator which essentially equals to comm_self. This test should not be executed on such platforms. Thus we check the number of processes in SHM communicator and exit if only one process exists. Signed-off-by: Xin Zhao <xinzhao3(a)illinois.edu> Signed-off-by: Antonio J. Pena <apenya(a)mcs.anl.gov> diff --git a/test/mpi/rma/win_shared_zerobyte.c b/test/mpi/rma/win_shared_zerobyte.c index 1f602a7..4bec8e5 100644 --- a/test/mpi/rma/win_shared_zerobyte.c +++ b/test/mpi/rma/win_shared_zerobyte.c @@ -35,6 +35,11 @@ int main(int argc, char **argv) MPI_Comm_rank(shm_comm, &shm_rank); MPI_Comm_size(shm_comm, &shm_nproc); + /* Platform does not support shared memory, just return. */ + if (shm_nproc < 2) { + goto exit; + } + bases = calloc(shm_nproc, sizeof(int *)); if (shm_rank == 0 || shm_rank == shm_nproc - 1) { @@ -95,12 +100,15 @@ int main(int argc, char **argv) MPI_Reduce(&errors, &all_errors, 1, MPI_INT, MPI_SUM, 0, MPI_COMM_WORLD); + exit: + if (rank == 0 && all_errors == 0) printf(" No Errors\n"); MPI_Finalize(); - free(bases); + if (bases) + free(bases); return 0; } ----------------------------------------------------------------------- Summary of changes: test/mpi/rma/win_shared_zerobyte.c | 10 +++++++++- 1 files changed, 9 insertions(+), 1 deletions(-) hooks/post-receive -- MPICH primary repository

1 0

[mpich] MPICH primary repository branch, master, updated. v3.1.2-91-g04a1907
by noreply＠mpich.org 11 Aug '14

11 Aug '14

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "MPICH primary repository". The branch, master has been updated via 04a1907aaed46486857f995e4b8332cfdcd865e7 (commit) via c03c3a79f19e772c1e2bd5fe32b6bc8117e59ca4 (commit) via 26fb5462d44b835e04ca192341bf9ff490def266 (commit) via c68494ad0be062336b6108ad8f1be4f70132a7de (commit) via 4b9c244961103d1c192143b7a819c062c0145382 (commit) from 5dac839337d807f45f46ffd8c1d3905badb2f910 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- http://git.mpich.org/mpich.git/commitdiff/04a1907aaed46486857f995e4b8332cfd… commit 04a1907aaed46486857f995e4b8332cfdcd865e7 Author: Rob Latham <robl(a)mcs.anl.gov> Date: Tue Aug 5 12:51:58 2014 -0500 Promote 'len' to something bigger than int It's now possible to have a very large file domain (one exceeding 2 GiB). We need to now relax some assertions and promote this type. Signed-off-by: Ken Raffenetti <raffenet(a)mcs.anl.gov> diff --git a/src/mpi/romio/adio/ad_gpfs/ad_gpfs_aggrs.c b/src/mpi/romio/adio/ad_gpfs/ad_gpfs_aggrs.c index c83d91f..517db04 100644 --- a/src/mpi/romio/adio/ad_gpfs/ad_gpfs_aggrs.c +++ b/src/mpi/romio/adio/ad_gpfs/ad_gpfs_aggrs.c @@ -536,8 +536,8 @@ void ADIOI_GPFS_Calc_my_req(ADIO_File fd, ADIO_Offset *offset_list, ADIO_Offset if (count_my_req_per_proc[i]) { my_req[i].offsets = (ADIO_Offset *) ADIOI_Malloc(count_my_req_per_proc[i] * sizeof(ADIO_Offset)); - my_req[i].lens = (int *) - ADIOI_Malloc(count_my_req_per_proc[i] * sizeof(int)); + my_req[i].lens = + ADIOI_Malloc(count_my_req_per_proc[i] * sizeof(ADIO_Offset)); count_my_req_procs++; } my_req[i].count = 0; /* will be incremented where needed @@ -574,8 +574,7 @@ void ADIOI_GPFS_Calc_my_req(ADIO_File fd, ADIO_Offset *offset_list, ADIO_Offset * and the associated count. */ my_req[proc].offsets[l] = off; - ADIOI_Assert(fd_len == (int) fd_len); - my_req[proc].lens[l] = (int) fd_len; + my_req[proc].lens[l] = fd_len; my_req[proc].count++; while (rem_len > 0) { @@ -595,8 +594,7 @@ void ADIOI_GPFS_Calc_my_req(ADIO_File fd, ADIO_Offset *offset_list, ADIO_Offset rem_len -= fd_len; my_req[proc].offsets[l] = off; - ADIOI_Assert(fd_len == (int) fd_len); - my_req[proc].lens[l] = (int) fd_len; + my_req[proc].lens[l] = fd_len; my_req[proc].count++; } } @@ -718,8 +716,8 @@ void ADIOI_GPFS_Calc_others_req(ADIO_File fd, int count_my_req_procs, others_req[i].offsets = (ADIO_Offset *) ADIOI_Malloc(count_others_req_per_proc[i]*sizeof(ADIO_Offset)); - others_req[i].lens = (int *) - ADIOI_Malloc(count_others_req_per_proc[i]*sizeof(int)); + others_req[i].lens = + ADIOI_Malloc(count_others_req_per_proc[i]*sizeof(ADIO_Offset)); if ( (MPIR_Upint)others_req[i].offsets < (MPIR_Upint)recvBufForOffsets ) recvBufForOffsets = others_req[i].offsets; @@ -813,7 +811,7 @@ void ADIOI_GPFS_Calc_others_req(ADIO_File fd, int count_my_req_procs, sdispls[i] = (int) ( ( (MPIR_Upint)my_req[i].lens - (MPIR_Upint)sendBufForLens ) / - (MPIR_Upint) sizeof(int) ); + (MPIR_Upint) sizeof(ADIO_Offset) ); /* Receive these offsets from process i. */ rcounts[i] = count_others_req_per_proc[i]; @@ -823,14 +821,14 @@ void ADIOI_GPFS_Calc_others_req(ADIO_File fd, int count_my_req_procs, rdispls[i] = (int) ( ( (MPIR_Upint)others_req[i].lens - (MPIR_Upint)recvBufForLens ) / - (MPIR_Upint) sizeof(int) ); + (MPIR_Upint) sizeof(ADIO_Offset) ); } /* Exchange the lengths */ MPI_Alltoallv(sendBufForLens, - scounts, sdispls, MPI_INT, + scounts, sdispls, ADIO_OFFSET, recvBufForLens, - rcounts, rdispls, MPI_INT, + rcounts, rdispls, ADIO_OFFSET, fd->comm); /* Clean up */ diff --git a/src/mpi/romio/adio/ad_gpfs/ad_gpfs_rdcoll.c b/src/mpi/romio/adio/ad_gpfs/ad_gpfs_rdcoll.c index 2172702..92b6336 100644 --- a/src/mpi/romio/adio/ad_gpfs/ad_gpfs_rdcoll.c +++ b/src/mpi/romio/adio/ad_gpfs/ad_gpfs_rdcoll.c @@ -829,8 +829,8 @@ static void ADIOI_R_Exchange_data(ADIO_File fd, void *buf, ADIOI_Flatlist_node tmp = others_req[i].lens[k]; others_req[i].lens[k] = partial_send[i]; } - MPI_Type_hindexed(count[i], - &(others_req[i].lens[start_pos[i]]), + ADIOI_Type_create_hindexed_x(count[i], + &(others_req[i].lens[start_pos[i]]), &(others_req[i].mem_ptrs[start_pos[i]]), MPI_BYTE, &send_type); /* absolute displacement; use MPI_BOTTOM in send */ diff --git a/src/mpi/romio/adio/ad_gpfs/ad_gpfs_wrcoll.c b/src/mpi/romio/adio/ad_gpfs/ad_gpfs_wrcoll.c index ba0ded5..d049340 100644 --- a/src/mpi/romio/adio/ad_gpfs/ad_gpfs_wrcoll.c +++ b/src/mpi/romio/adio/ad_gpfs/ad_gpfs_wrcoll.c @@ -931,8 +931,8 @@ static void ADIOI_W_Exchange_data(ADIO_File fd, const void *buf, char *write_buf tmp_len[i] = others_req[i].lens[k]; others_req[i].lens[k] = partial_recv[i]; } - MPI_Type_hindexed(count[i], - &(others_req[i].lens[start_pos[i]]), + ADIOI_Type_create_hindexed_x(count[i], + &(others_req[i].lens[start_pos[i]]), &(others_req[i].mem_ptrs[start_pos[i]]), MPI_BYTE, recv_types+j); /* absolute displacements; use MPI_BOTTOM in recv */ @@ -1286,7 +1286,7 @@ static void ADIOI_Heap_merge(ADIOI_Access *others_req, int *count, { typedef struct { ADIO_Offset *off_list; - int *len_list; + ADIO_Offset *len_list; int nelem; } heap_struct; diff --git a/src/mpi/romio/adio/ad_lustre/ad_lustre_aggregate.c b/src/mpi/romio/adio/ad_lustre/ad_lustre_aggregate.c index a43f565..cd55282 100644 --- a/src/mpi/romio/adio/ad_lustre/ad_lustre_aggregate.c +++ b/src/mpi/romio/adio/ad_lustre/ad_lustre_aggregate.c @@ -215,8 +215,8 @@ void ADIOI_LUSTRE_Calc_my_req(ADIO_File fd, ADIO_Offset *offset_list, my_req[i].offsets = (ADIO_Offset *) ADIOI_Malloc(count_my_req_per_proc[i] * sizeof(ADIO_Offset)); - my_req[i].lens = (int *) ADIOI_Malloc(count_my_req_per_proc[i] * - sizeof(int)); + my_req[i].lens = ADIOI_Malloc(count_my_req_per_proc[i] * + sizeof(ADIO_Offset)); count_my_req_procs++; } my_req[i].count = 0; /* will be incremented where needed later */ diff --git a/src/mpi/romio/adio/ad_lustre/ad_lustre_wrcoll.c b/src/mpi/romio/adio/ad_lustre/ad_lustre_wrcoll.c index 58d75c9..3299ad5 100644 --- a/src/mpi/romio/adio/ad_lustre/ad_lustre_wrcoll.c +++ b/src/mpi/romio/adio/ad_lustre/ad_lustre_wrcoll.c @@ -656,7 +656,7 @@ static void ADIOI_LUSTRE_W_Exchange_data(ADIO_File fd, const void *buf, j = 0; for (i = 0; i < nprocs; i++) { if (recv_size[i]) { - MPI_Type_hindexed(count[i], + ADIOI_Type_create_hindexed_x(count[i], &(others_req[i].lens[start_pos[i]]), &(others_req[i].mem_ptrs[start_pos[i]]), MPI_BYTE, recv_types + j); diff --git a/src/mpi/romio/adio/common/ad_aggregate.c b/src/mpi/romio/adio/common/ad_aggregate.c index 4749256..55b5544 100644 --- a/src/mpi/romio/adio/common/ad_aggregate.c +++ b/src/mpi/romio/adio/common/ad_aggregate.c @@ -335,8 +335,8 @@ void ADIOI_Calc_my_req(ADIO_File fd, ADIO_Offset *offset_list, ADIO_Offset *len_ if (count_my_req_per_proc[i]) { my_req[i].offsets = (ADIO_Offset *) ADIOI_Malloc(count_my_req_per_proc[i] * sizeof(ADIO_Offset)); - my_req[i].lens = (int *) - ADIOI_Malloc(count_my_req_per_proc[i] * sizeof(int)); + my_req[i].lens = + ADIOI_Malloc(count_my_req_per_proc[i] * sizeof(ADIO_Offset)); count_my_req_procs++; } my_req[i].count = 0; /* will be incremented where needed @@ -373,8 +373,7 @@ void ADIOI_Calc_my_req(ADIO_File fd, ADIO_Offset *offset_list, ADIO_Offset *len_ * and the associated count. */ my_req[proc].offsets[l] = off; - ADIOI_Assert(fd_len == (int) fd_len); - my_req[proc].lens[l] = (int) fd_len; + my_req[proc].lens[l] = fd_len; my_req[proc].count++; while (rem_len != 0) { @@ -394,8 +393,7 @@ void ADIOI_Calc_my_req(ADIO_File fd, ADIO_Offset *offset_list, ADIO_Offset *len_ rem_len -= fd_len; my_req[proc].offsets[l] = off; - ADIOI_Assert(fd_len == (int) fd_len); - my_req[proc].lens[l] = (int) fd_len; + my_req[proc].lens[l] = fd_len; my_req[proc].count++; } } @@ -463,8 +461,8 @@ void ADIOI_Calc_others_req(ADIO_File fd, int count_my_req_procs, others_req[i].count = count_others_req_per_proc[i]; others_req[i].offsets = (ADIO_Offset *) ADIOI_Malloc(count_others_req_per_proc[i]*sizeof(ADIO_Offset)); - others_req[i].lens = (int *) - ADIOI_Malloc(count_others_req_per_proc[i]*sizeof(int)); + others_req[i].lens = + ADIOI_Malloc(count_others_req_per_proc[i]*sizeof(ADIO_Offset)); others_req[i].mem_ptrs = (MPI_Aint *) ADIOI_Malloc(count_others_req_per_proc[i]*sizeof(MPI_Aint)); count_others_req_procs++; @@ -485,7 +483,7 @@ void ADIOI_Calc_others_req(ADIO_File fd, int count_my_req_procs, ADIO_OFFSET, i, i+myrank, fd->comm, &requests[j]); j++; MPI_Irecv(others_req[i].lens, others_req[i].count, - MPI_INT, i, i+myrank+1, fd->comm, &requests[j]); + ADIO_OFFSET, i, i+myrank+1, fd->comm, &requests[j]); j++; } } @@ -496,7 +494,7 @@ void ADIOI_Calc_others_req(ADIO_File fd, int count_my_req_procs, ADIO_OFFSET, i, i+myrank, fd->comm, &requests[j]); j++; MPI_Isend(my_req[i].lens, my_req[i].count, - MPI_INT, i, i+myrank+1, fd->comm, &requests[j]); + ADIO_OFFSET, i, i+myrank+1, fd->comm, &requests[j]); j++; } } diff --git a/src/mpi/romio/adio/common/ad_read_coll.c b/src/mpi/romio/adio/common/ad_read_coll.c index 35090a4..c5202a3 100644 --- a/src/mpi/romio/adio/common/ad_read_coll.c +++ b/src/mpi/romio/adio/common/ad_read_coll.c @@ -853,8 +853,8 @@ static void ADIOI_R_Exchange_data(ADIO_File fd, void *buf, ADIOI_Flatlist_node tmp = others_req[i].lens[k]; others_req[i].lens[k] = partial_send[i]; } - MPI_Type_hindexed(count[i], - &(others_req[i].lens[start_pos[i]]), + ADIOI_Type_create_hindexed_x(count[i], + &(others_req[i].lens[start_pos[i]]), &(others_req[i].mem_ptrs[start_pos[i]]), MPI_BYTE, &send_type); /* absolute displacement; use MPI_BOTTOM in send */ diff --git a/src/mpi/romio/adio/common/ad_write_coll.c b/src/mpi/romio/adio/common/ad_write_coll.c index bf92a75..d585f9e 100644 --- a/src/mpi/romio/adio/common/ad_write_coll.c +++ b/src/mpi/romio/adio/common/ad_write_coll.c @@ -611,8 +611,8 @@ static void ADIOI_W_Exchange_data(ADIO_File fd, void *buf, char *write_buf, tmp_len[i] = others_req[i].lens[k]; others_req[i].lens[k] = partial_recv[i]; } - MPI_Type_hindexed(count[i], - &(others_req[i].lens[start_pos[i]]), + ADIOI_Type_create_hindexed_x(count[i], + &(others_req[i].lens[start_pos[i]]), &(others_req[i].mem_ptrs[start_pos[i]]), MPI_BYTE, recv_types+j); /* absolute displacements; use MPI_BOTTOM in recv */ @@ -974,7 +974,7 @@ void ADIOI_Heap_merge(ADIOI_Access *others_req, int *count, { typedef struct { ADIO_Offset *off_list; - int *len_list; + ADIO_Offset *len_list; int nelem; } heap_struct; diff --git a/src/mpi/romio/adio/include/adioi.h b/src/mpi/romio/adio/include/adioi.h index 26aad93..65e2bcd 100644 --- a/src/mpi/romio/adio/include/adioi.h +++ b/src/mpi/romio/adio/include/adioi.h @@ -307,9 +307,7 @@ struct ADIOI_Fns_struct { as array of structures indexed by process number. */ typedef struct { ADIO_Offset *offsets; /* array of offsets */ - int *lens; /* array of lengths */ - /* consider aints or offsets for lens? Seems to be used as in-memory - buffer lengths, so it should be < 2G and ok as an int */ + ADIO_Offset *lens; /* array of lengths */ MPI_Aint *mem_ptrs; /* array of pointers. used in the read/write phase to indicate where the data is stored in memory */ @@ -320,7 +318,7 @@ typedef struct { file realms among other things */ typedef struct { ADIO_Offset *offsets; /* array of offsets */ - int *lens; /* array of lengths */ + ADIO_Offset *lens; /* array of lengths */ int count; /* size of above arrays */ } ADIOI_Offlen; http://git.mpich.org/mpich.git/commitdiff/c03c3a79f19e772c1e2bd5fe32b6bc811… commit c03c3a79f19e772c1e2bd5fe32b6bc8117e59ca4 Author: Rob Latham <robl(a)mcs.anl.gov> Date: Thu Aug 7 21:45:43 2014 -0500 create a big-count HINDEXED type standard-HINDEXED takes an integer array for its "array of lengths" parameter. ROMIO can generate array_of_lengths that are too big for an integer. Jump through hoops. Signed-off-by: Ken Raffenetti <raffenet(a)mcs.anl.gov> diff --git a/src/mpi/romio/adio/common/Makefile.mk b/src/mpi/romio/adio/common/Makefile.mk index 043b198..6cfd118 100644 --- a/src/mpi/romio/adio/common/Makefile.mk +++ b/src/mpi/romio/adio/common/Makefile.mk @@ -68,5 +68,6 @@ romio_other_sources += \ adio/common/system_hints.c \ adio/common/hint_fns.c \ adio/common/ad_threaded_io.c \ - adio/common/p2p_aggregation.c + adio/common/p2p_aggregation.c \ + adio/common/utils.c diff --git a/src/mpi/romio/adio/common/utils.c b/src/mpi/romio/adio/common/utils.c new file mode 100644 index 0000000..9814220 --- /dev/null +++ b/src/mpi/romio/adio/common/utils.c @@ -0,0 +1,83 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * + * Copyright (C) 2014 UChicgo/Argonne, LLC. + * See COPYRIGHT notice in top-level directory. + */ + +#include <adio.h> +#include <limits.h> + + +/* utility function for creating large contiguous types: algorithim from BigMPI + * https://github.com/jeffhammond/BigMPI */ + +static int type_create_contiguous_x(MPI_Count count, + MPI_Datatype oldtype, MPI_Datatype *newtype) +{ + /* to make 'count' fit MPI-3 type processing routines (which take integer + * counts), we construct a type consisting of N INT_MAX chunks followed by + * a remainder. e.g for a count of 4000000000 bytes you would end up with + * one 2147483647-byte chunk followed immediately by a 1852516353-byte + * chunk */ + MPI_Datatype chunks, remainder; + MPI_Aint lb, extent, disps[2]; + int blocklens[2]; + MPI_Datatype types[2]; + + MPI_Count c = count/INT_MAX; + MPI_Count r = count%INT_MAX; + + MPI_Type_vector(c, INT_MAX, INT_MAX, oldtype, &chunks); + MPI_Type_contiguous(r, oldtype, &remainder); + + MPI_Type_get_extent(oldtype, &lb, &extent); + + blocklens[0] = 1; blocklens[1] = 1; + disps[0] = 0; disps[1] = c*extent*INT_MAX; + types[0] = chunks; types[1] = remainder; + + MPI_Type_create_struct(2, blocklens, disps, types, newtype); + + MPI_Type_free(&chunks); + MPI_Type_free(&remainder); + + return MPI_SUCCESS; +} +/* like MPI_Type_create_hindexed, except array_of_lengths can be a larger datatype. + * + * Hindexed provides 'count' pairs of (displacement, length), but what if + * length is longer than an integer? We will create 'count' types, using + * contig if length is small enough, or something more complex if not */ + +int ADIOI_Type_create_hindexed_x(int count, + const MPI_Count array_of_blocklengths[], + const MPI_Aint array_of_displacements[], + MPI_Datatype oldtype, + MPI_Datatype *newtype) +{ + int i, ret; + MPI_Datatype *types; + int *blocklens; + + types = ADIOI_Malloc(count*sizeof(MPI_Datatype)); + blocklens = ADIOI_Malloc(count*sizeof(int)); + + for(i=0; i<count; i++) { + blocklens[i] = 1; + type_create_contiguous_x(array_of_blocklengths[i], oldtype, &(types[i])); + } + + ret = MPI_Type_create_struct(count, blocklens, array_of_displacements, + types, newtype); + for (i=0; i< count; i++) + MPI_Type_free(&(types[i])); + ADIOI_Free(types); + ADIOI_Free(blocklens); + + return ret; +} + +/* + * vim: ts=8 sts=4 sw=4 noexpandtab + */ diff --git a/src/mpi/romio/adio/include/adioi.h b/src/mpi/romio/adio/include/adioi.h index 944e099..26aad93 100644 --- a/src/mpi/romio/adio/include/adioi.h +++ b/src/mpi/romio/adio/include/adioi.h @@ -590,6 +590,11 @@ int MPIR_Status_set_bytes(MPI_Status *status, MPI_Datatype datatype, MPI_Count n int ADIOI_Uses_generic_read(ADIO_File fd); int ADIOI_Uses_generic_write(ADIO_File fd); int ADIOI_Err_create_code(const char *myname, const char *filename, int my_errno); +int ADIOI_Type_create_hindexed_x(int count, + const MPI_Count array_of_blocklengths[], + const MPI_Aint array_of_displacements[], + MPI_Datatype oldtype, + MPI_Datatype *newtype); int ADIOI_FAKE_IODone(ADIO_Request *request, ADIO_Status *status, http://git.mpich.org/mpich.git/commitdiff/26fb5462d44b835e04ca192341bf9ff49… commit 26fb5462d44b835e04ca192341bf9ff490def266 Author: Rob Latham <robl(a)mcs.anl.gov> Date: Wed Aug 6 11:23:04 2014 -0500 portability: only BGQ uses the MPIX routines mpix.h only exists for the pami device. BGPLATFORM is our closest proxy for "need this header for some PAMI-implemented routines" Signed-off-by: Ken Raffenetti <raffenet(a)mcs.anl.gov> diff --git a/src/mpi/romio/adio/ad_gpfs/ad_gpfs_wrcoll.c b/src/mpi/romio/adio/ad_gpfs/ad_gpfs_wrcoll.c index 17299f2..ba0ded5 100644 --- a/src/mpi/romio/adio/ad_gpfs/ad_gpfs_wrcoll.c +++ b/src/mpi/romio/adio/ad_gpfs/ad_gpfs_wrcoll.c @@ -17,7 +17,9 @@ #include "ad_gpfs.h" #include "ad_gpfs_aggrs.h" +#ifdef BGQPLATFORM #include <mpix.h> +#endif #ifdef AGGREGATION_PROFILE #include "mpe.h" http://git.mpich.org/mpich.git/commitdiff/c68494ad0be062336b6108ad8f1be4f70… commit c68494ad0be062336b6108ad8f1be4f70132a7de Author: Rob Latham <robl(a)mcs.anl.gov> Date: Wed Aug 6 14:58:01 2014 -0500 clean up bigtype test a bit - free memory in test to appease valgrind - more helpful failure message Signed-off-by: Ken Raffenetti <raffenet(a)mcs.anl.gov> diff --git a/test/mpi/io/bigtype.c b/test/mpi/io/bigtype.c index 571a0a3..d500714 100644 --- a/test/mpi/io/bigtype.c +++ b/test/mpi/io/bigtype.c @@ -119,11 +119,14 @@ int main(int argc, char** argv) for (k=0 ; k<NUM_X*NUM_Y ; k++) { if(buf_read[k] != buf_write[k]) { - fprintf(stderr, "Verfiy Failed at index %zu\n", k); + fprintf(stderr, "Verfiy Failed index %zu: expected %d found %d\n", + k, buf_write[k], buf_read[k]); assert(0); } } + free(buf_write); + free(buf_read); MPI_File_close(&fh); MPI_Type_free (&mem_type); http://git.mpich.org/mpich.git/commitdiff/4b9c244961103d1c192143b7a819c062c… commit 4b9c244961103d1c192143b7a819c062c0145382 Author: Rob Latham <robl(a)mcs.anl.gov> Date: Fri Aug 8 11:13:07 2014 -0500 shameful but necessary wholesale reformatting DOS-newlines snuck into this file when I wasn't paying attention. Signed-off-by: Ken Raffenetti <raffenet(a)mcs.anl.gov> diff --git a/src/mpi/romio/adio/ad_gpfs/ad_gpfs_aggrs.c b/src/mpi/romio/adio/ad_gpfs/ad_gpfs_aggrs.c index 350a336..c83d91f 100644 --- a/src/mpi/romio/adio/ad_gpfs/ad_gpfs_aggrs.c +++ b/src/mpi/romio/adio/ad_gpfs/ad_gpfs_aggrs.c @@ -1,848 +1,848 @@ -/* ---------------------------------------------------------------- */ -/* (C)Copyright IBM Corp. 2007, 2008 */ -/* ---------------------------------------------------------------- */ -/** - * \file ad_gpfs_aggrs.c - * \brief The externally used function from this file is is declared in ad_gpfs_aggrs.h - */ - -/* -*- Mode: C; c-basic-offset:4 ; -*- */ -/* - * Copyright (C) 1997-2001 University of Chicago. - * See COPYRIGHT notice in top-level directory. - */ - - -#include "adio.h" -#include "adio_cb_config_list.h" -#include "ad_gpfs.h" -#include "ad_gpfs_aggrs.h" - -#ifdef AGGREGATION_PROFILE -#include "mpe.h" -#endif - - -#ifdef USE_DBG_LOGGING - #define AGG_DEBUG 1 -#endif - -#ifndef TRACE_ERR -# define TRACE_ERR(format...) -#endif - -/* Comments copied from common: - * This file contains four functions: - * - * ADIOI_Calc_aggregator() - * ADIOI_Calc_file_domains() - * ADIOI_Calc_my_req() - * ADIOI_Calc_others_req() - * - * The last three of these were originally in ad_read_coll.c, but they are - * also shared with ad_write_coll.c. I felt that they were better kept with - * the rest of the shared aggregation code. - */ - -/* Discussion of values available from above: - * - * ADIO_Offset st_offsets[0..nprocs-1] - * ADIO_Offset end_offsets[0..nprocs-1] - * These contain a list of start and end offsets for each process in - * the communicator. For example, an access at loc 10, size 10 would - * have a start offset of 10 and end offset of 19. - * int nprocs - * number of processors in the collective I/O communicator - * ADIO_Offset min_st_offset - * ADIO_Offset fd_start[0..nprocs_for_coll-1] - * starting location of "file domain"; region that a given process will - * perform aggregation for (i.e. actually do I/O) - * ADIO_Offset fd_end[0..nprocs_for_coll-1] - * start + size - 1 roughly, but it can be less, or 0, in the case of - * uneven distributions - */ - -/* Description from common/ad_aggregate.c. (Does it completely apply to bg?) - * ADIOI_Calc_aggregator() - * - * The intention here is to implement a function which provides basically - * the same functionality as in Rajeev's original version of - * ADIOI_Calc_my_req(). He used a ceiling division approach to assign the - * file domains, and we use the same approach here when calculating the - * location of an offset/len in a specific file domain. Further we assume - * this same distribution when calculating the rank_index, which is later - * used to map to a specific process rank in charge of the file domain. - * - * A better (i.e. more general) approach would be to use the list of file - * domains only. This would be slower in the case where the - * original ceiling division was used, but it would allow for arbitrary - * distributions of regions to aggregators. We'd need to know the - * nprocs_for_coll in that case though, which we don't have now. - * - * Note a significant difference between this function and Rajeev's old code: - * this code doesn't necessarily return a rank in the range - * 0..nprocs_for_coll; instead you get something in 0..nprocs. This is a - * result of the rank mapping; any set of ranks in the communicator could be - * used now. - * - * Returns an integer representing a rank in the collective I/O communicator. - * - * The "len" parameter is also modified to indicate the amount of data - * actually available in this file domain. - */ -/* - * This is more general aggregator search function which does not base on the assumption - * that each aggregator hosts the file domain with the same size - */ -int ADIOI_GPFS_Calc_aggregator(ADIO_File fd, - ADIO_Offset off, - ADIO_Offset min_off, - ADIO_Offset *len, - ADIO_Offset fd_size, - ADIO_Offset *fd_start, - ADIO_Offset *fd_end) -{ - int rank_index, rank; - ADIO_Offset avail_bytes; - TRACE_ERR("Entering ADIOI_GPFS_Calc_aggregator\n"); - - ADIOI_Assert ( (off <= fd_end[fd->hints->cb_nodes-1] && off >= min_off && fd_start[0] >= min_off ) ); - - /* binary search --> rank_index is returned */ - int ub = fd->hints->cb_nodes; - int lb = 0; - /* get an index into our array of aggregators */ - /* Common code for striping - bg doesn't use it but it's - here to make diff'ing easier. - rank_index = (int) ((off - min_off + fd_size)/ fd_size - 1); - - if (fd->hints->striping_unit > 0) { - * wkliao: implementation for file domain alignment - fd_start[] and fd_end[] have been aligned with file lock - boundaries when returned from ADIOI_Calc_file_domains() so cannot - just use simple arithmatic as above * - rank_index = 0; - while (off > fd_end[rank_index]) rank_index++; - } - bg does it's own striping below - */ - rank_index = fd->hints->cb_nodes / 2; - while ( off < fd_start[rank_index] || off > fd_end[rank_index] ) { - if ( off > fd_end [rank_index] ) { - lb = rank_index; - rank_index = (rank_index + ub) / 2; - } - else - if ( off < fd_start[rank_index] ) { - ub = rank_index; - rank_index = (rank_index + lb) / 2; - } - } - /* we index into fd_end with rank_index, and fd_end was allocated to be no - * bigger than fd->hins->cb_nodes. If we ever violate that, we're - * overrunning arrays. Obviously, we should never ever hit this abort */ - if (rank_index >= fd->hints->cb_nodes || rank_index < 0) { - FPRINTF(stderr, "Error in ADIOI_Calc_aggregator(): rank_index(%d) >= fd->hints->cb_nodes (%d) fd_size=%lld off=%lld\n", - rank_index,fd->hints->cb_nodes,fd_size,off); - MPI_Abort(MPI_COMM_WORLD, 1); - } - /* DBG_FPRINTF ("ADIOI_GPFS_Calc_aggregator: rank_index = %d\n", - rank_index ); */ - - /* - * remember here that even in Rajeev's original code it was the case that - * different aggregators could end up with different amounts of data to - * aggregate. here we use fd_end[] to make sure that we know how much - * data this aggregator is working with. - * - * the +1 is to take into account the end vs. length issue. - */ - avail_bytes = fd_end[rank_index] + 1 - off; - if (avail_bytes < *len && avail_bytes > 0) { - /* this file domain only has part of the requested contig. region */ - - *len = avail_bytes; - } - - /* map our index to a rank */ - /* NOTE: FOR NOW WE DON'T HAVE A MAPPING...JUST DO 0..NPROCS_FOR_COLL */ - rank = fd->hints->ranklist[rank_index]; - TRACE_ERR("Leaving ADIOI_GPFS_Calc_aggregator\n"); - - return rank; -} - -/* - * Compute a dynamic access range based file domain partition among I/O aggregators, - * which align to the GPFS block size - * Divide the I/O workload among "nprocs_for_coll" processes. This is - * done by (logically) dividing the file into file domains (FDs); each - * process may directly access only its own file domain. - * Additional effort is to make sure that each I/O aggregator get - * a file domain that aligns to the GPFS block size. So, there will - * not be any false sharing of GPFS file blocks among multiple I/O nodes. - * - * The common version of this now accepts a min_fd_size and striping_unit. - * It doesn't seem necessary here (using GPFS block sizes) but keep it in mind - * (e.g. we could pass striping unit instead of using fs_ptr->blksize). - */ -void ADIOI_GPFS_Calc_file_domains(ADIO_File fd, - ADIO_Offset *st_offsets, - ADIO_Offset *end_offsets, - int nprocs, - int nprocs_for_coll, - ADIO_Offset *min_st_offset_ptr, - ADIO_Offset **fd_start_ptr, - ADIO_Offset **fd_end_ptr, - ADIO_Offset *fd_size_ptr, - void *fs_ptr) -{ - ADIO_Offset min_st_offset, max_end_offset, *fd_start, *fd_end, *fd_size; - int i, aggr; - TRACE_ERR("Entering ADIOI_GPFS_Calc_file_domains\n"); - blksize_t blksize; - -#ifdef AGGREGATION_PROFILE - MPE_Log_event (5004, 0, NULL); -#endif - -# if AGG_DEBUG - static char myname[] = "ADIOI_GPFS_Calc_file_domains"; - DBG_FPRINTF(stderr, "%s(%d): %d aggregator(s)\n", - myname,__LINE__,nprocs_for_coll); -# endif - if (fd->blksize <= 0) - /* default to 1M if blksize unset */ - fd->blksize = 1048576; - blksize = fd->blksize; - -# if AGG_DEBUG - DBG_FPRINTF(stderr,"%s(%d): Blocksize=%ld\n",myname,__LINE__,blksize); -# endif -/* find min of start offsets and max of end offsets of all processes */ - min_st_offset = st_offsets [0]; - max_end_offset = end_offsets[0]; - for (i=1; i<nprocs; i++) { - min_st_offset = ADIOI_MIN(min_st_offset, st_offsets[i]); - max_end_offset = ADIOI_MAX(max_end_offset, end_offsets[i]); - } - - /* DBG_FPRINTF(stderr, "_calc_file_domains, min_st_offset, max_ - = %qd, %qd\n", min_st_offset, max_end_offset );*/ - - /* determine the "file domain (FD)" of each process, i.e., the portion of - the file that will be "owned" by each process */ - - ADIO_Offset gpfs_ub = (max_end_offset +blksize-1) / blksize * blksize - 1; - ADIO_Offset gpfs_lb = min_st_offset / blksize * blksize; - ADIO_Offset gpfs_ub_rdoff = (max_end_offset +blksize-1) / blksize * blksize - 1 - max_end_offset; - ADIO_Offset gpfs_lb_rdoff = min_st_offset - min_st_offset / blksize * blksize; - ADIO_Offset fd_gpfs_range = gpfs_ub - gpfs_lb + 1; - - int naggs = nprocs_for_coll; - - /* Tweak the file domains so that no fd is smaller than a threshold. We - * have to strike a balance between efficency and parallelism: somewhere - * between 10k processes sending 32-byte requests and one process sending a - * 320k request is a (system-dependent) sweet spot - - This is from the common code - the new min_fd_size parm that we didn't implement. - (And common code uses a different declaration of fd_size so beware) - - if (fd_size < min_fd_size) - fd_size = min_fd_size; - */ - fd_size = (ADIO_Offset *) ADIOI_Malloc(nprocs_for_coll * sizeof(ADIO_Offset)); - *fd_start_ptr = (ADIO_Offset *) ADIOI_Malloc(nprocs_for_coll * sizeof(ADIO_Offset)); - *fd_end_ptr = (ADIO_Offset *) ADIOI_Malloc(nprocs_for_coll * sizeof(ADIO_Offset)); - fd_start = *fd_start_ptr; - fd_end = *fd_end_ptr; - - /* each process will have a file domain of some number of gpfs blocks, but - * the division of blocks is not likely to be even. Some file domains will - * be "large" and others "small" - * - * Example: consider 17 blocks distributed over 3 aggregators. - * nb_cn_small = 17/3 = 5 - * naggs_large = 17 - 3*(17/3) = 17 - 15 = 2 - * naggs_small = 3 - 2 = 1 - * - * and you end up with file domains of {5-blocks, 6-blocks, 6-blocks} - * - * what about (relatively) small files? say, a file of 1000 blocks - * distributed over 2064 aggregators: - * nb_cn_small = 1000/2064 = 0 - * naggs_large = 1000 - 2064*(1000/2064) = 1000 - * naggs_small = 2064 - 1000 = 1064 - * and you end up with domains of {0, 0, 0, ... 1, 1, 1 ...} - * - * it might be a good idea instead of having all the zeros up front, to - * "mix" those zeros into the fd_size array. that way, no pset/bridge-set - * is left with zero work. In fact, even if the small file domains aren't - * zero, it's probably still a good idea to mix the "small" file domains - * across the fd_size array to keep the io nodes in balance */ - - - ADIO_Offset n_gpfs_blk = fd_gpfs_range / blksize; - ADIO_Offset nb_cn_small = n_gpfs_blk/naggs; - ADIO_Offset naggs_large = n_gpfs_blk - naggs * (n_gpfs_blk/naggs); - ADIO_Offset naggs_small = naggs - naggs_large; - -#ifdef BGQPLATFORM - if (gpfsmpio_balancecontig == 1) { - /* File domains blocks are assigned to aggregators in a breadth-first - * fashion relative to the ions - additionally, file domains on the - * aggregators sharing the same bridgeset and ion have contiguous - * offsets. */ - - // initialize everything to small - for (i=0; i<naggs; i++) - fd_size[i] = nb_cn_small * blksize; - - // go thru and distribute the large across the bridges - - /* bridelistoffset: agg rank list offsets using the bridgelist - each - * entry is created by adding up the indexes for the aggs from all - * previous bridges */ - int *bridgelistoffset = - (int *) ADIOI_Malloc(fd->hints->fs_hints.bg.numbridges*sizeof(int)); - /* tmpbridgelistnum: copy of the bridgelistnum whose entries can be - * decremented to keep track of bridge assignments during the actual - * large block assignments to the agg rank list*/ - int *tmpbridgelistnum = - (int *) ADIOI_Malloc(fd->hints->fs_hints.bg.numbridges*sizeof(int)); - - int j; - for (j=0;j<fd->hints->fs_hints.bg.numbridges;j++) { - int k, bridgerankoffset = 0; - for (k=0;k<j;k++) { - bridgerankoffset += fd->hints->fs_hints.bg.bridgelistnum[k]; - } - bridgelistoffset[j] = bridgerankoffset; - } - - for (j=0;j<fd->hints->fs_hints.bg.numbridges;j++) - tmpbridgelistnum[j] = fd->hints->fs_hints.bg.bridgelistnum[j]; - int bridgeiter = 0; - - /* distribute the large blocks across the aggs going breadth-first - * across the bridgelist - this distributes the fd sizes across the - * ions, so later in the file domain assignment when it iterates thru - * the ranklist the offsets will be contiguous within the bridge and - * ion as well */ - for (j=0;j<naggs_large;j++) { - int foundbridge = 0; - int numbridgelistpasses = 0; - while (!foundbridge) { - if (tmpbridgelistnum[bridgeiter] > 0) { - foundbridge = 1; - /* - printf("bridgeiter is %d tmpbridgelistnum[bridgeiter] is %d bridgelistoffset[bridgeiter] is %d\n",bridgeiter,tmpbridgelistnum[bridgeiter],bridgelistoffset[bridgeiter]); - printf("naggs is %d bridgeiter is %d bridgelistoffset[bridgeiter] is %d tmpbridgelistnum[bridgeiter] is %d\n",naggs, bridgeiter,bridgelistoffset[bridgeiter],tmpbridgelistnum[bridgeiter]); - printf("naggs is %d bridgeiter is %d setting fd_size[%d]\n",naggs, bridgeiter,bridgelistoffset[bridgeiter]+(fd->hints->bridgelistnum[bridgeiter]-tmpbridgelistnum[bridgeiter])); - */ - int currentbridgelistnum = - (fd->hints->fs_hints.bg.bridgelistnum[bridgeiter]- - tmpbridgelistnum[bridgeiter]); - int currentfdsizeindex = bridgelistoffset[bridgeiter] + - currentbridgelistnum; - fd_size[currentfdsizeindex] = (nb_cn_small+1) * blksize; - tmpbridgelistnum[bridgeiter]--; - } - if (bridgeiter == (fd->hints->fs_hints.bg.numbridges-1)) { - /* guard against infinite loop - should only ever make 1 pass - * thru bridgelist */ - ADIOI_Assert(numbridgelistpasses == 0); - numbridgelistpasses++; - bridgeiter = 0; - } - else - bridgeiter++; - } - } - ADIOI_Free(tmpbridgelistnum); - ADIOI_Free(bridgelistoffset); - - } else { - /* BG/L- and BG/P-style distribution of file domains: simple allocation of - * file domins to each aggregator */ - for (i=0; i<naggs; i++) { - if (i < naggs_small) { - fd_size[i] = nb_cn_small * blksize; - } else { - fd_size[i] = (nb_cn_small+1) * blksize; - } - } - } -#ifdef balancecontigtrace - int myrank; - MPI_Comm_rank(fd->comm,&myrank); - if (myrank == 0) { - fprintf(stderr,"naggs_small is %d nb_cn_small is %d\n",naggs_small,nb_cn_small); - for (i=0; i<naggs; i++) { - fprintf(stderr,"fd_size[%d] set to %d agg rank is %d\n",i,fd_size[i],fd->hints->ranklist[i]); - } - } -#endif - -#else // not BGQ platform - for (i=0; i<naggs; i++) { - if (i < naggs_small) { - fd_size[i] = nb_cn_small * blksize; - } else { - fd_size[i] = (nb_cn_small+1) * blksize; - } - } - -#endif - - -# if AGG_DEBUG - DBG_FPRINTF(stderr,"%s(%d): " - "gpfs_ub %llu, " - "gpfs_lb %llu, " - "gpfs_ub_rdoff %llu, " - "gpfs_lb_rdoff %llu, " - "fd_gpfs_range %llu, " - "n_gpfs_blk %llu, " - "nb_cn_small %llu, " - "naggs_large %llu, " - "naggs_small %llu, " - "\n", - myname,__LINE__, - gpfs_ub , - gpfs_lb , - gpfs_ub_rdoff, - gpfs_lb_rdoff, - fd_gpfs_range, - n_gpfs_blk , - nb_cn_small , - naggs_large , - naggs_small - ); -# endif - - fd_size[0] -= gpfs_lb_rdoff; - fd_size[naggs-1] -= gpfs_ub_rdoff; - - /* compute the file domain for each aggr */ - ADIO_Offset offset = min_st_offset; - for (aggr=0; aggr<naggs; aggr++) { - fd_start[aggr] = offset; - fd_end [aggr] = offset + fd_size[aggr] - 1; - offset += fd_size[aggr]; - } - - *fd_size_ptr = fd_size[0]; - *min_st_offset_ptr = min_st_offset; - -#ifdef AGGREGATION_PROFILE - MPE_Log_event (5005, 0, NULL); -#endif - ADIOI_Free (fd_size); - TRACE_ERR("Leaving ADIOI_GPFS_Calc_file_domains\n"); -} - -/* - * ADIOI_GPFS_Calc_my_req() overrides ADIOI_Calc_my_req for the default implementation - * is specific for static file domain partitioning. - * - * ADIOI_Calc_my_req() - calculate what portions of the access requests - * of this process are located in the file domains of various processes - * (including this one) - */ -void ADIOI_GPFS_Calc_my_req(ADIO_File fd, ADIO_Offset *offset_list, ADIO_Offset *len_list, - int contig_access_count, ADIO_Offset - min_st_offset, ADIO_Offset *fd_start, - ADIO_Offset *fd_end, ADIO_Offset fd_size, - int nprocs, - int *count_my_req_procs_ptr, - int **count_my_req_per_proc_ptr, - ADIOI_Access **my_req_ptr, - int **buf_idx_ptr) -/* Possibly reconsider if buf_idx's are ok as int's, or should they be aints/offsets? - They are used as memory buffer indices so it seems like the 2G limit is in effect */ -{ - int *count_my_req_per_proc, count_my_req_procs, *buf_idx; - int i, l, proc; - ADIO_Offset fd_len, rem_len, curr_idx, off; - ADIOI_Access *my_req; - TRACE_ERR("Entering ADIOI_GPFS_Calc_my_req\n"); - -#ifdef AGGREGATION_PROFILE - MPE_Log_event (5024, 0, NULL); -#endif - *count_my_req_per_proc_ptr = (int *) ADIOI_Calloc(nprocs,sizeof(int)); - count_my_req_per_proc = *count_my_req_per_proc_ptr; -/* count_my_req_per_proc[i] gives the no. of contig. requests of this - process in process i's file domain. calloc initializes to zero. - I'm allocating memory of size nprocs, so that I can do an - MPI_Alltoall later on.*/ - - buf_idx = (int *) ADIOI_Malloc(nprocs*sizeof(int)); -/* buf_idx is relevant only if buftype_is_contig. - buf_idx[i] gives the index into user_buf where data received - from proc. i should be placed. This allows receives to be done - without extra buffer. This can't be done if buftype is not contig. */ - - /* initialize buf_idx to -1 */ - for (i=0; i < nprocs; i++) buf_idx[i] = -1; - - /* one pass just to calculate how much space to allocate for my_req; - * contig_access_count was calculated way back in ADIOI_Calc_my_off_len() - */ - for (i=0; i < contig_access_count; i++) { - /* short circuit offset/len processing if len == 0 - * (zero-byte read/write */ - if (len_list[i] == 0) - continue; - off = offset_list[i]; - fd_len = len_list[i]; - /* note: we set fd_len to be the total size of the access. then - * ADIOI_Calc_aggregator() will modify the value to return the - * amount that was available from the file domain that holds the - * first part of the access. - */ - /* BES */ - proc = ADIOI_GPFS_Calc_aggregator(fd, off, min_st_offset, &fd_len, fd_size, - fd_start, fd_end); - count_my_req_per_proc[proc]++; - - /* figure out how much data is remaining in the access (i.e. wasn't - * part of the file domain that had the starting byte); we'll take - * care of this data (if there is any) in the while loop below. - */ - rem_len = len_list[i] - fd_len; - - while (rem_len > 0) { - off += fd_len; /* point to first remaining byte */ - fd_len = rem_len; /* save remaining size, pass to calc */ - proc = ADIOI_GPFS_Calc_aggregator(fd, off, min_st_offset, &fd_len, - fd_size, fd_start, fd_end); - - count_my_req_per_proc[proc]++; - rem_len -= fd_len; /* reduce remaining length by amount from fd */ - } - } - -/* now allocate space for my_req, offset, and len */ - - *my_req_ptr = (ADIOI_Access *) - ADIOI_Malloc(nprocs*sizeof(ADIOI_Access)); - my_req = *my_req_ptr; - - count_my_req_procs = 0; - for (i=0; i < nprocs; i++) { - if (count_my_req_per_proc[i]) { - my_req[i].offsets = (ADIO_Offset *) - ADIOI_Malloc(count_my_req_per_proc[i] * sizeof(ADIO_Offset)); - my_req[i].lens = (int *) - ADIOI_Malloc(count_my_req_per_proc[i] * sizeof(int)); - count_my_req_procs++; - } - my_req[i].count = 0; /* will be incremented where needed - later */ - } - -/* now fill in my_req */ - curr_idx = 0; - for (i=0; i<contig_access_count; i++) { - /* short circuit offset/len processing if len == 0 - * (zero-byte read/write */ - if (len_list[i] == 0) - continue; - off = offset_list[i]; - fd_len = len_list[i]; - proc = ADIOI_GPFS_Calc_aggregator(fd, off, min_st_offset, &fd_len, fd_size, - fd_start, fd_end); - - /* for each separate contiguous access from this process */ - if (buf_idx[proc] == -1) - { - ADIOI_Assert(curr_idx == (int) curr_idx); - buf_idx[proc] = (int) curr_idx; - } - - l = my_req[proc].count; - curr_idx += fd_len; - - rem_len = len_list[i] - fd_len; - - /* store the proc, offset, and len information in an array - * of structures, my_req. Each structure contains the - * offsets and lengths located in that process's FD, - * and the associated count. - */ - my_req[proc].offsets[l] = off; - ADIOI_Assert(fd_len == (int) fd_len); - my_req[proc].lens[l] = (int) fd_len; - my_req[proc].count++; - - while (rem_len > 0) { - off += fd_len; - fd_len = rem_len; - proc = ADIOI_GPFS_Calc_aggregator(fd, off, min_st_offset, &fd_len, - fd_size, fd_start, fd_end); - - if (buf_idx[proc] == -1) - { - ADIOI_Assert(curr_idx == (int) curr_idx); - buf_idx[proc] = (int) curr_idx; - } - - l = my_req[proc].count; - curr_idx += fd_len; - rem_len -= fd_len; - - my_req[proc].offsets[l] = off; - ADIOI_Assert(fd_len == (int) fd_len); - my_req[proc].lens[l] = (int) fd_len; - my_req[proc].count++; - } - } - - - -#ifdef AGG_DEBUG - for (i=0; i<nprocs; i++) { - if (count_my_req_per_proc[i] > 0) { - DBG_FPRINTF(stderr, "data needed from %d (count = %d):\n", i, - my_req[i].count); - for (l=0; l < my_req[i].count; l++) { - DBG_FPRINTF(stderr, " off[%d] = %lld, len[%d] = %d\n", l, - my_req[i].offsets[l], l, my_req[i].lens[l]); - } - } - DBG_FPRINTF(stderr, "buf_idx[%d] = 0x%x\n", i, buf_idx[i]); - } -#endif - - *count_my_req_procs_ptr = count_my_req_procs; - *buf_idx_ptr = buf_idx; -#ifdef AGGREGATION_PROFILE - MPE_Log_event (5025, 0, NULL); -#endif - TRACE_ERR("Leaving ADIOI_GPFS_Calc_my_req\n"); -} - -/* - * ADIOI_Calc_others_req (copied to bg and switched to all to all for performance) - * - * param[in] count_my_req_procs Number of processes whose file domain my - * request touches. - * param[in] count_my_req_per_proc count_my_req_per_proc[i] gives the no. of - * contig. requests of this process in - * process i's file domain. - * param[in] my_req A structure defining my request - * param[in] nprocs Number of nodes in the block - * param[in] myrank Rank of this node - * param[out] count_others_req_proc_ptr Number of processes whose requests lie in - * my process's file domain (including my - * process itself) - * param[out] others_req_ptr Array of other process' requests that lie - * in my process's file domain - */ -void ADIOI_GPFS_Calc_others_req(ADIO_File fd, int count_my_req_procs, - int *count_my_req_per_proc, - ADIOI_Access *my_req, - int nprocs, int myrank, - int *count_others_req_procs_ptr, - ADIOI_Access **others_req_ptr) -{ - TRACE_ERR("Entering ADIOI_GPFS_Calc_others_req\n"); -/* determine what requests of other processes lie in this process's - file domain */ - -/* count_others_req_procs = number of processes whose requests lie in - this process's file domain (including this process itself) - count_others_req_per_proc[i] indicates how many separate contiguous - requests of proc. i lie in this process's file domain. */ - - int *count_others_req_per_proc, count_others_req_procs; - int i; - ADIOI_Access *others_req; - - /* Parameters for MPI_Alltoallv */ - int *scounts, *sdispls, *rcounts, *rdispls; - - /* Parameters for MPI_Alltoallv. These are the buffers, which - * are later computed to be the lowest address of all buffers - * to be sent/received for offsets and lengths. Initialize to - * the highest possible address which is the current minimum. - */ - void *sendBufForOffsets=(void*)0xFFFFFFFFFFFFFFFF, - *sendBufForLens =(void*)0xFFFFFFFFFFFFFFFF, - *recvBufForOffsets=(void*)0xFFFFFFFFFFFFFFFF, - *recvBufForLens =(void*)0xFFFFFFFFFFFFFFFF; - -/* first find out how much to send/recv and from/to whom */ -#ifdef AGGREGATION_PROFILE - MPE_Log_event (5026, 0, NULL); -#endif - /* Send 1 int to each process. count_my_req_per_proc[i] is the number of - * requests that my process will do to the file domain owned by process[i]. - * Receive 1 int from each process. count_others_req_per_proc[i] is the number of - * requests that process[i] will do to the file domain owned by my process. - */ - count_others_req_per_proc = (int *) ADIOI_Malloc(nprocs*sizeof(int)); -/* cora2a1=timebase(); */ -/*for(i=0;i<nprocs;i++) ?*/ - MPI_Alltoall(count_my_req_per_proc, 1, MPI_INT, - count_others_req_per_proc, 1, MPI_INT, fd->comm); - -/* total_cora2a+=timebase()-cora2a1; */ - - /* Allocate storage for an array of other nodes' accesses of our - * node's file domain. Also allocate storage for the alltoallv - * parameters. - */ - *others_req_ptr = (ADIOI_Access *) - ADIOI_Malloc(nprocs*sizeof(ADIOI_Access)); - others_req = *others_req_ptr; - - scounts = ADIOI_Malloc(nprocs*sizeof(int)); - sdispls = ADIOI_Malloc(nprocs*sizeof(int)); - rcounts = ADIOI_Malloc(nprocs*sizeof(int)); - rdispls = ADIOI_Malloc(nprocs*sizeof(int)); - - /* If process[i] has any requests in my file domain, - * initialize an ADIOI_Access structure that will describe each request - * from process[i]. The offsets, lengths, and buffer pointers still need - * to be obtained to complete the setting of this structure. - */ - count_others_req_procs = 0; - for (i=0; i<nprocs; i++) { - if (count_others_req_per_proc[i]) - { - others_req[i].count = count_others_req_per_proc[i]; - - others_req[i].offsets = (ADIO_Offset *) - ADIOI_Malloc(count_others_req_per_proc[i]*sizeof(ADIO_Offset)); - others_req[i].lens = (int *) - ADIOI_Malloc(count_others_req_per_proc[i]*sizeof(int)); - - if ( (MPIR_Upint)others_req[i].offsets < (MPIR_Upint)recvBufForOffsets ) - recvBufForOffsets = others_req[i].offsets; - if ( (MPIR_Upint)others_req[i].lens < (MPIR_Upint)recvBufForLens ) - recvBufForLens = others_req[i].lens; - - others_req[i].mem_ptrs = (MPI_Aint *) - ADIOI_Malloc(count_others_req_per_proc[i]*sizeof(MPI_Aint)); - - count_others_req_procs++; - } - else - { - others_req[i].count = 0; - others_req[i].offsets = NULL; - others_req[i].lens = NULL; - } - } - /* If no recv buffer was allocated in the loop above, make it NULL */ - if ( recvBufForOffsets == (void*)0xFFFFFFFFFFFFFFFF) recvBufForOffsets = NULL; - if ( recvBufForLens == (void*)0xFFFFFFFFFFFFFFFF) recvBufForLens = NULL; - - /* Now send the calculated offsets and lengths to respective processes */ - - /************************/ - /* Exchange the offsets */ - /************************/ - - /* Determine the lowest sendBufForOffsets/Lens */ - for (i=0; i<nprocs; i++) - { - if ( (my_req[i].count) && - ((MPIR_Upint)my_req[i].offsets <= (MPIR_Upint)sendBufForOffsets) ) - { - sendBufForOffsets = my_req[i].offsets; - } - - if ( (my_req[i].count) && - ((MPIR_Upint)my_req[i].lens <= (MPIR_Upint)sendBufForLens) ) - { - sendBufForLens = my_req[i].lens; - } - } - - /* If no send buffer was found in the loop above, make it NULL */ - if ( sendBufForOffsets == (void*)0xFFFFFFFFFFFFFFFF) sendBufForOffsets = NULL; - if ( sendBufForLens == (void*)0xFFFFFFFFFFFFFFFF) sendBufForLens = NULL; - - /* Calculate the displacements from the sendBufForOffsets/Lens */ - for (i=0; i<nprocs; i++) - { - /* Send these offsets to process i.*/ - scounts[i] = count_my_req_per_proc[i]; - if ( scounts[i] == 0 ) - sdispls[i] = 0; - else - sdispls[i] = (int) - ( ( (MPIR_Upint)my_req[i].offsets - - (MPIR_Upint)sendBufForOffsets ) / - (MPIR_Upint)sizeof(ADIO_Offset) ); - - /* Receive these offsets from process i.*/ - rcounts[i] = count_others_req_per_proc[i]; - if ( rcounts[i] == 0 ) - rdispls[i] = 0; - else - rdispls[i] = (int) - ( ( (MPIR_Upint)others_req[i].offsets - - (MPIR_Upint)recvBufForOffsets ) / - (MPIR_Upint)sizeof(ADIO_Offset) ); - } - - /* Exchange the offsets */ - MPI_Alltoallv(sendBufForOffsets, - scounts, sdispls, ADIO_OFFSET, - recvBufForOffsets, - rcounts, rdispls, ADIO_OFFSET, - fd->comm); - - /************************/ - /* Exchange the lengths */ - /************************/ - - for (i=0; i<nprocs; i++) - { - /* Send these lengths to process i.*/ - scounts[i] = count_my_req_per_proc[i]; - if ( scounts[i] == 0 ) - sdispls[i] = 0; - else - sdispls[i] = (int) - ( ( (MPIR_Upint)my_req[i].lens - - (MPIR_Upint)sendBufForLens ) / - (MPIR_Upint) sizeof(int) ); - - /* Receive these offsets from process i. */ - rcounts[i] = count_others_req_per_proc[i]; - if ( rcounts[i] == 0 ) - rdispls[i] = 0; - else - rdispls[i] = (int) - ( ( (MPIR_Upint)others_req[i].lens - - (MPIR_Upint)recvBufForLens ) / - (MPIR_Upint) sizeof(int) ); - } - - /* Exchange the lengths */ - MPI_Alltoallv(sendBufForLens, - scounts, sdispls, MPI_INT, - recvBufForLens, - rcounts, rdispls, MPI_INT, - fd->comm); - - /* Clean up */ - ADIOI_Free(count_others_req_per_proc); - ADIOI_Free (scounts); - ADIOI_Free (sdispls); - ADIOI_Free (rcounts); - ADIOI_Free (rdispls); - - *count_others_req_procs_ptr = count_others_req_procs; -#ifdef AGGREGATION_PROFILE - MPE_Log_event (5027, 0, NULL); -#endif - TRACE_ERR("Leaving ADIOI_GPFS_Calc_others_req\n"); -} +/* ---------------------------------------------------------------- */ +/* (C)Copyright IBM Corp. 2007, 2008 */ +/* ---------------------------------------------------------------- */ +/** + * \file ad_gpfs_aggrs.c + * \brief The externally used function from this file is is declared in ad_gpfs_aggrs.h + */ + +/* -*- Mode: C; c-basic-offset:4 ; -*- */ +/* + * Copyright (C) 1997-2001 University of Chicago. + * See COPYRIGHT notice in top-level directory. + */ + + +#include "adio.h" +#include "adio_cb_config_list.h" +#include "ad_gpfs.h" +#include "ad_gpfs_aggrs.h" + +#ifdef AGGREGATION_PROFILE +#include "mpe.h" +#endif + + +#ifdef USE_DBG_LOGGING + #define AGG_DEBUG 1 +#endif + +#ifndef TRACE_ERR +# define TRACE_ERR(format...) +#endif + +/* Comments copied from common: + * This file contains four functions: + * + * ADIOI_Calc_aggregator() + * ADIOI_Calc_file_domains() + * ADIOI_Calc_my_req() + * ADIOI_Calc_others_req() + * + * The last three of these were originally in ad_read_coll.c, but they are + * also shared with ad_write_coll.c. I felt that they were better kept with + * the rest of the shared aggregation code. + */ + +/* Discussion of values available from above: + * + * ADIO_Offset st_offsets[0..nprocs-1] + * ADIO_Offset end_offsets[0..nprocs-1] + * These contain a list of start and end offsets for each process in + * the communicator. For example, an access at loc 10, size 10 would + * have a start offset of 10 and end offset of 19. + * int nprocs + * number of processors in the collective I/O communicator + * ADIO_Offset min_st_offset + * ADIO_Offset fd_start[0..nprocs_for_coll-1] + * starting location of "file domain"; region that a given process will + * perform aggregation for (i.e. actually do I/O) + * ADIO_Offset fd_end[0..nprocs_for_coll-1] + * start + size - 1 roughly, but it can be less, or 0, in the case of + * uneven distributions + */ + +/* Description from common/ad_aggregate.c. (Does it completely apply to bg?) + * ADIOI_Calc_aggregator() + * + * The intention here is to implement a function which provides basically + * the same functionality as in Rajeev's original version of + * ADIOI_Calc_my_req(). He used a ceiling division approach to assign the + * file domains, and we use the same approach here when calculating the + * location of an offset/len in a specific file domain. Further we assume + * this same distribution when calculating the rank_index, which is later + * used to map to a specific process rank in charge of the file domain. + * + * A better (i.e. more general) approach would be to use the list of file + * domains only. This would be slower in the case where the + * original ceiling division was used, but it would allow for arbitrary + * distributions of regions to aggregators. We'd need to know the + * nprocs_for_coll in that case though, which we don't have now. + * + * Note a significant difference between this function and Rajeev's old code: + * this code doesn't necessarily return a rank in the range + * 0..nprocs_for_coll; instead you get something in 0..nprocs. This is a + * result of the rank mapping; any set of ranks in the communicator could be + * used now. + * + * Returns an integer representing a rank in the collective I/O communicator. + * + * The "len" parameter is also modified to indicate the amount of data + * actually available in this file domain. + */ +/* + * This is more general aggregator search function which does not base on the assumption + * that each aggregator hosts the file domain with the same size + */ +int ADIOI_GPFS_Calc_aggregator(ADIO_File fd, + ADIO_Offset off, + ADIO_Offset min_off, + ADIO_Offset *len, + ADIO_Offset fd_size, + ADIO_Offset *fd_start, + ADIO_Offset *fd_end) +{ + int rank_index, rank; + ADIO_Offset avail_bytes; + TRACE_ERR("Entering ADIOI_GPFS_Calc_aggregator\n"); + + ADIOI_Assert ( (off <= fd_end[fd->hints->cb_nodes-1] && off >= min_off && fd_start[0] >= min_off ) ); + + /* binary search --> rank_index is returned */ + int ub = fd->hints->cb_nodes; + int lb = 0; + /* get an index into our array of aggregators */ + /* Common code for striping - bg doesn't use it but it's + here to make diff'ing easier. + rank_index = (int) ((off - min_off + fd_size)/ fd_size - 1); + + if (fd->hints->striping_unit > 0) { + * wkliao: implementation for file domain alignment + fd_start[] and fd_end[] have been aligned with file lock + boundaries when returned from ADIOI_Calc_file_domains() so cannot + just use simple arithmatic as above * + rank_index = 0; + while (off > fd_end[rank_index]) rank_index++; + } + bg does it's own striping below + */ + rank_index = fd->hints->cb_nodes / 2; + while ( off < fd_start[rank_index] || off > fd_end[rank_index] ) { + if ( off > fd_end [rank_index] ) { + lb = rank_index; + rank_index = (rank_index + ub) / 2; + } + else + if ( off < fd_start[rank_index] ) { + ub = rank_index; + rank_index = (rank_index + lb) / 2; + } + } + /* we index into fd_end with rank_index, and fd_end was allocated to be no + * bigger than fd->hins->cb_nodes. If we ever violate that, we're + * overrunning arrays. Obviously, we should never ever hit this abort */ + if (rank_index >= fd->hints->cb_nodes || rank_index < 0) { + FPRINTF(stderr, "Error in ADIOI_Calc_aggregator(): rank_index(%d) >= fd->hints->cb_nodes (%d) fd_size=%lld off=%lld\n", + rank_index,fd->hints->cb_nodes,fd_size,off); + MPI_Abort(MPI_COMM_WORLD, 1); + } + /* DBG_FPRINTF ("ADIOI_GPFS_Calc_aggregator: rank_index = %d\n", + rank_index ); */ + + /* + * remember here that even in Rajeev's original code it was the case that + * different aggregators could end up with different amounts of data to + * aggregate. here we use fd_end[] to make sure that we know how much + * data this aggregator is working with. + * + * the +1 is to take into account the end vs. length issue. + */ + avail_bytes = fd_end[rank_index] + 1 - off; + if (avail_bytes < *len && avail_bytes > 0) { + /* this file domain only has part of the requested contig. region */ + + *len = avail_bytes; + } + + /* map our index to a rank */ + /* NOTE: FOR NOW WE DON'T HAVE A MAPPING...JUST DO 0..NPROCS_FOR_COLL */ + rank = fd->hints->ranklist[rank_index]; + TRACE_ERR("Leaving ADIOI_GPFS_Calc_aggregator\n"); + + return rank; +} + +/* + * Compute a dynamic access range based file domain partition among I/O aggregators, + * which align to the GPFS block size + * Divide the I/O workload among "nprocs_for_coll" processes. This is + * done by (logically) dividing the file into file domains (FDs); each + * process may directly access only its own file domain. + * Additional effort is to make sure that each I/O aggregator get + * a file domain that aligns to the GPFS block size. So, there will + * not be any false sharing of GPFS file blocks among multiple I/O nodes. + * + * The common version of this now accepts a min_fd_size and striping_unit. + * It doesn't seem necessary here (using GPFS block sizes) but keep it in mind + * (e.g. we could pass striping unit instead of using fs_ptr->blksize). + */ +void ADIOI_GPFS_Calc_file_domains(ADIO_File fd, + ADIO_Offset *st_offsets, + ADIO_Offset *end_offsets, + int nprocs, + int nprocs_for_coll, + ADIO_Offset *min_st_offset_ptr, + ADIO_Offset **fd_start_ptr, + ADIO_Offset **fd_end_ptr, + ADIO_Offset *fd_size_ptr, + void *fs_ptr) +{ + ADIO_Offset min_st_offset, max_end_offset, *fd_start, *fd_end, *fd_size; + int i, aggr; + TRACE_ERR("Entering ADIOI_GPFS_Calc_file_domains\n"); + blksize_t blksize; + +#ifdef AGGREGATION_PROFILE + MPE_Log_event (5004, 0, NULL); +#endif + +# if AGG_DEBUG + static char myname[] = "ADIOI_GPFS_Calc_file_domains"; + DBG_FPRINTF(stderr, "%s(%d): %d aggregator(s)\n", + myname,__LINE__,nprocs_for_coll); +# endif + if (fd->blksize <= 0) + /* default to 1M if blksize unset */ + fd->blksize = 1048576; + blksize = fd->blksize; + +# if AGG_DEBUG + DBG_FPRINTF(stderr,"%s(%d): Blocksize=%ld\n",myname,__LINE__,blksize); +# endif +/* find min of start offsets and max of end offsets of all processes */ + min_st_offset = st_offsets [0]; + max_end_offset = end_offsets[0]; + for (i=1; i<nprocs; i++) { + min_st_offset = ADIOI_MIN(min_st_offset, st_offsets[i]); + max_end_offset = ADIOI_MAX(max_end_offset, end_offsets[i]); + } + + /* DBG_FPRINTF(stderr, "_calc_file_domains, min_st_offset, max_ + = %qd, %qd\n", min_st_offset, max_end_offset );*/ + + /* determine the "file domain (FD)" of each process, i.e., the portion of + the file that will be "owned" by each process */ + + ADIO_Offset gpfs_ub = (max_end_offset +blksize-1) / blksize * blksize - 1; + ADIO_Offset gpfs_lb = min_st_offset / blksize * blksize; + ADIO_Offset gpfs_ub_rdoff = (max_end_offset +blksize-1) / blksize * blksize - 1 - max_end_offset; + ADIO_Offset gpfs_lb_rdoff = min_st_offset - min_st_offset / blksize * blksize; + ADIO_Offset fd_gpfs_range = gpfs_ub - gpfs_lb + 1; + + int naggs = nprocs_for_coll; + + /* Tweak the file domains so that no fd is smaller than a threshold. We + * have to strike a balance between efficency and parallelism: somewhere + * between 10k processes sending 32-byte requests and one process sending a + * 320k request is a (system-dependent) sweet spot + + This is from the common code - the new min_fd_size parm that we didn't implement. + (And common code uses a different declaration of fd_size so beware) + + if (fd_size < min_fd_size) + fd_size = min_fd_size; + */ + fd_size = (ADIO_Offset *) ADIOI_Malloc(nprocs_for_coll * sizeof(ADIO_Offset)); + *fd_start_ptr = (ADIO_Offset *) ADIOI_Malloc(nprocs_for_coll * sizeof(ADIO_Offset)); + *fd_end_ptr = (ADIO_Offset *) ADIOI_Malloc(nprocs_for_coll * sizeof(ADIO_Offset)); + fd_start = *fd_start_ptr; + fd_end = *fd_end_ptr; + + /* each process will have a file domain of some number of gpfs blocks, but + * the division of blocks is not likely to be even. Some file domains will + * be "large" and others "small" + * + * Example: consider 17 blocks distributed over 3 aggregators. + * nb_cn_small = 17/3 = 5 + * naggs_large = 17 - 3*(17/3) = 17 - 15 = 2 + * naggs_small = 3 - 2 = 1 + * + * and you end up with file domains of {5-blocks, 6-blocks, 6-blocks} + * + * what about (relatively) small files? say, a file of 1000 blocks + * distributed over 2064 aggregators: + * nb_cn_small = 1000/2064 = 0 + * naggs_large = 1000 - 2064*(1000/2064) = 1000 + * naggs_small = 2064 - 1000 = 1064 + * and you end up with domains of {0, 0, 0, ... 1, 1, 1 ...} + * + * it might be a good idea instead of having all the zeros up front, to + * "mix" those zeros into the fd_size array. that way, no pset/bridge-set + * is left with zero work. In fact, even if the small file domains aren't + * zero, it's probably still a good idea to mix the "small" file domains + * across the fd_size array to keep the io nodes in balance */ + + + ADIO_Offset n_gpfs_blk = fd_gpfs_range / blksize; + ADIO_Offset nb_cn_small = n_gpfs_blk/naggs; + ADIO_Offset naggs_large = n_gpfs_blk - naggs * (n_gpfs_blk/naggs); + ADIO_Offset naggs_small = naggs - naggs_large; + +#ifdef BGQPLATFORM + if (gpfsmpio_balancecontig == 1) { + /* File domains blocks are assigned to aggregators in a breadth-first + * fashion relative to the ions - additionally, file domains on the + * aggregators sharing the same bridgeset and ion have contiguous + * offsets. */ + + // initialize everything to small + for (i=0; i<naggs; i++) + fd_size[i] = nb_cn_small * blksize; + + // go thru and distribute the large across the bridges + + /* bridelistoffset: agg rank list offsets using the bridgelist - each + * entry is created by adding up the indexes for the aggs from all + * previous bridges */ + int *bridgelistoffset = + (int *) ADIOI_Malloc(fd->hints->fs_hints.bg.numbridges*sizeof(int)); + /* tmpbridgelistnum: copy of the bridgelistnum whose entries can be + * decremented to keep track of bridge assignments during the actual + * large block assignments to the agg rank list*/ + int *tmpbridgelistnum = + (int *) ADIOI_Malloc(fd->hints->fs_hints.bg.numbridges*sizeof(int)); + + int j; + for (j=0;j<fd->hints->fs_hints.bg.numbridges;j++) { + int k, bridgerankoffset = 0; + for (k=0;k<j;k++) { + bridgerankoffset += fd->hints->fs_hints.bg.bridgelistnum[k]; + } + bridgelistoffset[j] = bridgerankoffset; + } + + for (j=0;j<fd->hints->fs_hints.bg.numbridges;j++) + tmpbridgelistnum[j] = fd->hints->fs_hints.bg.bridgelistnum[j]; + int bridgeiter = 0; + + /* distribute the large blocks across the aggs going breadth-first + * across the bridgelist - this distributes the fd sizes across the + * ions, so later in the file domain assignment when it iterates thru + * the ranklist the offsets will be contiguous within the bridge and + * ion as well */ + for (j=0;j<naggs_large;j++) { + int foundbridge = 0; + int numbridgelistpasses = 0; + while (!foundbridge) { + if (tmpbridgelistnum[bridgeiter] > 0) { + foundbridge = 1; + /* + printf("bridgeiter is %d tmpbridgelistnum[bridgeiter] is %d bridgelistoffset[bridgeiter] is %d\n",bridgeiter,tmpbridgelistnum[bridgeiter],bridgelistoffset[bridgeiter]); + printf("naggs is %d bridgeiter is %d bridgelistoffset[bridgeiter] is %d tmpbridgelistnum[bridgeiter] is %d\n",naggs, bridgeiter,bridgelistoffset[bridgeiter],tmpbridgelistnum[bridgeiter]); + printf("naggs is %d bridgeiter is %d setting fd_size[%d]\n",naggs, bridgeiter,bridgelistoffset[bridgeiter]+(fd->hints->bridgelistnum[bridgeiter]-tmpbridgelistnum[bridgeiter])); + */ + int currentbridgelistnum = + (fd->hints->fs_hints.bg.bridgelistnum[bridgeiter]- + tmpbridgelistnum[bridgeiter]); + int currentfdsizeindex = bridgelistoffset[bridgeiter] + + currentbridgelistnum; + fd_size[currentfdsizeindex] = (nb_cn_small+1) * blksize; + tmpbridgelistnum[bridgeiter]--; + } + if (bridgeiter == (fd->hints->fs_hints.bg.numbridges-1)) { + /* guard against infinite loop - should only ever make 1 pass + * thru bridgelist */ + ADIOI_Assert(numbridgelistpasses == 0); + numbridgelistpasses++; + bridgeiter = 0; + } + else + bridgeiter++; + } + } + ADIOI_Free(tmpbridgelistnum); + ADIOI_Free(bridgelistoffset); + + } else { + /* BG/L- and BG/P-style distribution of file domains: simple allocation of + * file domins to each aggregator */ + for (i=0; i<naggs; i++) { + if (i < naggs_small) { + fd_size[i] = nb_cn_small * blksize; + } else { + fd_size[i] = (nb_cn_small+1) * blksize; + } + } + } +#ifdef balancecontigtrace + int myrank; + MPI_Comm_rank(fd->comm,&myrank); + if (myrank == 0) { + fprintf(stderr,"naggs_small is %d nb_cn_small is %d\n",naggs_small,nb_cn_small); + for (i=0; i<naggs; i++) { + fprintf(stderr,"fd_size[%d] set to %d agg rank is %d\n",i,fd_size[i],fd->hints->ranklist[i]); + } + } +#endif + +#else // not BGQ platform + for (i=0; i<naggs; i++) { + if (i < naggs_small) { + fd_size[i] = nb_cn_small * blksize; + } else { + fd_size[i] = (nb_cn_small+1) * blksize; + } + } + +#endif + + +# if AGG_DEBUG + DBG_FPRINTF(stderr,"%s(%d): " + "gpfs_ub %llu, " + "gpfs_lb %llu, " + "gpfs_ub_rdoff %llu, " + "gpfs_lb_rdoff %llu, " + "fd_gpfs_range %llu, " + "n_gpfs_blk %llu, " + "nb_cn_small %llu, " + "naggs_large %llu, " + "naggs_small %llu, " + "\n", + myname,__LINE__, + gpfs_ub , + gpfs_lb , + gpfs_ub_rdoff, + gpfs_lb_rdoff, + fd_gpfs_range, + n_gpfs_blk , + nb_cn_small , + naggs_large , + naggs_small + ); +# endif + + fd_size[0] -= gpfs_lb_rdoff; + fd_size[naggs-1] -= gpfs_ub_rdoff; + + /* compute the file domain for each aggr */ + ADIO_Offset offset = min_st_offset; + for (aggr=0; aggr<naggs; aggr++) { + fd_start[aggr] = offset; + fd_end [aggr] = offset + fd_size[aggr] - 1; + offset += fd_size[aggr]; + } + + *fd_size_ptr = fd_size[0]; + *min_st_offset_ptr = min_st_offset; + +#ifdef AGGREGATION_PROFILE + MPE_Log_event (5005, 0, NULL); +#endif + ADIOI_Free (fd_size); + TRACE_ERR("Leaving ADIOI_GPFS_Calc_file_domains\n"); +} + +/* + * ADIOI_GPFS_Calc_my_req() overrides ADIOI_Calc_my_req for the default implementation + * is specific for static file domain partitioning. + * + * ADIOI_Calc_my_req() - calculate what portions of the access requests + * of this process are located in the file domains of various processes + * (including this one) + */ +void ADIOI_GPFS_Calc_my_req(ADIO_File fd, ADIO_Offset *offset_list, ADIO_Offset *len_list, + int contig_access_count, ADIO_Offset + min_st_offset, ADIO_Offset *fd_start, + ADIO_Offset *fd_end, ADIO_Offset fd_size, + int nprocs, + int *count_my_req_procs_ptr, + int **count_my_req_per_proc_ptr, + ADIOI_Access **my_req_ptr, + int **buf_idx_ptr) +/* Possibly reconsider if buf_idx's are ok as int's, or should they be aints/offsets? + They are used as memory buffer indices so it seems like the 2G limit is in effect */ +{ + int *count_my_req_per_proc, count_my_req_procs, *buf_idx; + int i, l, proc; + ADIO_Offset fd_len, rem_len, curr_idx, off; + ADIOI_Access *my_req; + TRACE_ERR("Entering ADIOI_GPFS_Calc_my_req\n"); + +#ifdef AGGREGATION_PROFILE + MPE_Log_event (5024, 0, NULL); +#endif + *count_my_req_per_proc_ptr = (int *) ADIOI_Calloc(nprocs,sizeof(int)); + count_my_req_per_proc = *count_my_req_per_proc_ptr; +/* count_my_req_per_proc[i] gives the no. of contig. requests of this + process in process i's file domain. calloc initializes to zero. + I'm allocating memory of size nprocs, so that I can do an + MPI_Alltoall later on.*/ + + buf_idx = (int *) ADIOI_Malloc(nprocs*sizeof(int)); +/* buf_idx is relevant only if buftype_is_contig. + buf_idx[i] gives the index into user_buf where data received + from proc. i should be placed. This allows receives to be done + without extra buffer. This can't be done if buftype is not contig. */ + + /* initialize buf_idx to -1 */ + for (i=0; i < nprocs; i++) buf_idx[i] = -1; + + /* one pass just to calculate how much space to allocate for my_req; + * contig_access_count was calculated way back in ADIOI_Calc_my_off_len() + */ + for (i=0; i < contig_access_count; i++) { + /* short circuit offset/len processing if len == 0 + * (zero-byte read/write */ + if (len_list[i] == 0) + continue; + off = offset_list[i]; + fd_len = len_list[i]; + /* note: we set fd_len to be the total size of the access. then + * ADIOI_Calc_aggregator() will modify the value to return the + * amount that was available from the file domain that holds the + * first part of the access. + */ + /* BES */ + proc = ADIOI_GPFS_Calc_aggregator(fd, off, min_st_offset, &fd_len, fd_size, + fd_start, fd_end); + count_my_req_per_proc[proc]++; + + /* figure out how much data is remaining in the access (i.e. wasn't + * part of the file domain that had the starting byte); we'll take + * care of this data (if there is any) in the while loop below. + */ + rem_len = len_list[i] - fd_len; + + while (rem_len > 0) { + off += fd_len; /* point to first remaining byte */ + fd_len = rem_len; /* save remaining size, pass to calc */ + proc = ADIOI_GPFS_Calc_aggregator(fd, off, min_st_offset, &fd_len, + fd_size, fd_start, fd_end); + + count_my_req_per_proc[proc]++; + rem_len -= fd_len; /* reduce remaining length by amount from fd */ + } + } + +/* now allocate space for my_req, offset, and len */ + + *my_req_ptr = (ADIOI_Access *) + ADIOI_Malloc(nprocs*sizeof(ADIOI_Access)); + my_req = *my_req_ptr; + + count_my_req_procs = 0; + for (i=0; i < nprocs; i++) { + if (count_my_req_per_proc[i]) { + my_req[i].offsets = (ADIO_Offset *) + ADIOI_Malloc(count_my_req_per_proc[i] * sizeof(ADIO_Offset)); + my_req[i].lens = (int *) + ADIOI_Malloc(count_my_req_per_proc[i] * sizeof(int)); + count_my_req_procs++; + } + my_req[i].count = 0; /* will be incremented where needed + later */ + } + +/* now fill in my_req */ + curr_idx = 0; + for (i=0; i<contig_access_count; i++) { + /* short circuit offset/len processing if len == 0 + * (zero-byte read/write */ + if (len_list[i] == 0) + continue; + off = offset_list[i]; + fd_len = len_list[i]; + proc = ADIOI_GPFS_Calc_aggregator(fd, off, min_st_offset, &fd_len, fd_size, + fd_start, fd_end); + + /* for each separate contiguous access from this process */ + if (buf_idx[proc] == -1) + { + ADIOI_Assert(curr_idx == (int) curr_idx); + buf_idx[proc] = (int) curr_idx; + } + + l = my_req[proc].count; + curr_idx += fd_len; + + rem_len = len_list[i] - fd_len; + + /* store the proc, offset, and len information in an array + * of structures, my_req. Each structure contains the + * offsets and lengths located in that process's FD, + * and the associated count. + */ + my_req[proc].offsets[l] = off; + ADIOI_Assert(fd_len == (int) fd_len); + my_req[proc].lens[l] = (int) fd_len; + my_req[proc].count++; + + while (rem_len > 0) { + off += fd_len; + fd_len = rem_len; + proc = ADIOI_GPFS_Calc_aggregator(fd, off, min_st_offset, &fd_len, + fd_size, fd_start, fd_end); + + if (buf_idx[proc] == -1) + { + ADIOI_Assert(curr_idx == (int) curr_idx); + buf_idx[proc] = (int) curr_idx; + } + + l = my_req[proc].count; + curr_idx += fd_len; + rem_len -= fd_len; + + my_req[proc].offsets[l] = off; + ADIOI_Assert(fd_len == (int) fd_len); + my_req[proc].lens[l] = (int) fd_len; + my_req[proc].count++; + } + } + + + +#ifdef AGG_DEBUG + for (i=0; i<nprocs; i++) { + if (count_my_req_per_proc[i] > 0) { + DBG_FPRINTF(stderr, "data needed from %d (count = %d):\n", i, + my_req[i].count); + for (l=0; l < my_req[i].count; l++) { + DBG_FPRINTF(stderr, " off[%d] = %lld, len[%d] = %d\n", l, + my_req[i].offsets[l], l, my_req[i].lens[l]); + } + } + DBG_FPRINTF(stderr, "buf_idx[%d] = 0x%x\n", i, buf_idx[i]); + } +#endif + + *count_my_req_procs_ptr = count_my_req_procs; + *buf_idx_ptr = buf_idx; +#ifdef AGGREGATION_PROFILE + MPE_Log_event (5025, 0, NULL); +#endif + TRACE_ERR("Leaving ADIOI_GPFS_Calc_my_req\n"); +} + +/* + * ADIOI_Calc_others_req (copied to bg and switched to all to all for performance) + * + * param[in] count_my_req_procs Number of processes whose file domain my + * request touches. + * param[in] count_my_req_per_proc count_my_req_per_proc[i] gives the no. of + * contig. requests of this process in + * process i's file domain. + * param[in] my_req A structure defining my request + * param[in] nprocs Number of nodes in the block + * param[in] myrank Rank of this node + * param[out] count_others_req_proc_ptr Number of processes whose requests lie in + * my process's file domain (including my + * process itself) + * param[out] others_req_ptr Array of other process' requests that lie + * in my process's file domain + */ +void ADIOI_GPFS_Calc_others_req(ADIO_File fd, int count_my_req_procs, + int *count_my_req_per_proc, + ADIOI_Access *my_req, + int nprocs, int myrank, + int *count_others_req_procs_ptr, + ADIOI_Access **others_req_ptr) +{ + TRACE_ERR("Entering ADIOI_GPFS_Calc_others_req\n"); +/* determine what requests of other processes lie in this process's + file domain */ + +/* count_others_req_procs = number of processes whose requests lie in + this process's file domain (including this process itself) + count_others_req_per_proc[i] indicates how many separate contiguous + requests of proc. i lie in this process's file domain. */ + + int *count_others_req_per_proc, count_others_req_procs; + int i; + ADIOI_Access *others_req; + + /* Parameters for MPI_Alltoallv */ + int *scounts, *sdispls, *rcounts, *rdispls; + + /* Parameters for MPI_Alltoallv. These are the buffers, which + * are later computed to be the lowest address of all buffers + * to be sent/received for offsets and lengths. Initialize to + * the highest possible address which is the current minimum. + */ + void *sendBufForOffsets=(void*)0xFFFFFFFFFFFFFFFF, + *sendBufForLens =(void*)0xFFFFFFFFFFFFFFFF, + *recvBufForOffsets=(void*)0xFFFFFFFFFFFFFFFF, + *recvBufForLens =(void*)0xFFFFFFFFFFFFFFFF; + +/* first find out how much to send/recv and from/to whom */ +#ifdef AGGREGATION_PROFILE + MPE_Log_event (5026, 0, NULL); +#endif + /* Send 1 int to each process. count_my_req_per_proc[i] is the number of + * requests that my process will do to the file domain owned by process[i]. + * Receive 1 int from each process. count_others_req_per_proc[i] is the number of + * requests that process[i] will do to the file domain owned by my process. + */ + count_others_req_per_proc = (int *) ADIOI_Malloc(nprocs*sizeof(int)); +/* cora2a1=timebase(); */ +/*for(i=0;i<nprocs;i++) ?*/ + MPI_Alltoall(count_my_req_per_proc, 1, MPI_INT, + count_others_req_per_proc, 1, MPI_INT, fd->comm); + +/* total_cora2a+=timebase()-cora2a1; */ + + /* Allocate storage for an array of other nodes' accesses of our + * node's file domain. Also allocate storage for the alltoallv + * parameters. + */ + *others_req_ptr = (ADIOI_Access *) + ADIOI_Malloc(nprocs*sizeof(ADIOI_Access)); + others_req = *others_req_ptr; + + scounts = ADIOI_Malloc(nprocs*sizeof(int)); + sdispls = ADIOI_Malloc(nprocs*sizeof(int)); + rcounts = ADIOI_Malloc(nprocs*sizeof(int)); + rdispls = ADIOI_Malloc(nprocs*sizeof(int)); + + /* If process[i] has any requests in my file domain, + * initialize an ADIOI_Access structure that will describe each request + * from process[i]. The offsets, lengths, and buffer pointers still need + * to be obtained to complete the setting of this structure. + */ + count_others_req_procs = 0; + for (i=0; i<nprocs; i++) { + if (count_others_req_per_proc[i]) + { + others_req[i].count = count_others_req_per_proc[i]; + + others_req[i].offsets = (ADIO_Offset *) + ADIOI_Malloc(count_others_req_per_proc[i]*sizeof(ADIO_Offset)); + others_req[i].lens = (int *) + ADIOI_Malloc(count_others_req_per_proc[i]*sizeof(int)); + + if ( (MPIR_Upint)others_req[i].offsets < (MPIR_Upint)recvBufForOffsets ) + recvBufForOffsets = others_req[i].offsets; + if ( (MPIR_Upint)others_req[i].lens < (MPIR_Upint)recvBufForLens ) + recvBufForLens = others_req[i].lens; + + others_req[i].mem_ptrs = (MPI_Aint *) + ADIOI_Malloc(count_others_req_per_proc[i]*sizeof(MPI_Aint)); + + count_others_req_procs++; + } + else + { + others_req[i].count = 0; + others_req[i].offsets = NULL; + others_req[i].lens = NULL; + } + } + /* If no recv buffer was allocated in the loop above, make it NULL */ + if ( recvBufForOffsets == (void*)0xFFFFFFFFFFFFFFFF) recvBufForOffsets = NULL; + if ( recvBufForLens == (void*)0xFFFFFFFFFFFFFFFF) recvBufForLens = NULL; + + /* Now send the calculated offsets and lengths to respective processes */ + + /************************/ + /* Exchange the offsets */ + /************************/ + + /* Determine the lowest sendBufForOffsets/Lens */ + for (i=0; i<nprocs; i++) + { + if ( (my_req[i].count) && + ((MPIR_Upint)my_req[i].offsets <= (MPIR_Upint)sendBufForOffsets) ) + { + sendBufForOffsets = my_req[i].offsets; + } + + if ( (my_req[i].count) && + ((MPIR_Upint)my_req[i].lens <= (MPIR_Upint)sendBufForLens) ) + { + sendBufForLens = my_req[i].lens; + } + } + + /* If no send buffer was found in the loop above, make it NULL */ + if ( sendBufForOffsets == (void*)0xFFFFFFFFFFFFFFFF) sendBufForOffsets = NULL; + if ( sendBufForLens == (void*)0xFFFFFFFFFFFFFFFF) sendBufForLens = NULL; + + /* Calculate the displacements from the sendBufForOffsets/Lens */ + for (i=0; i<nprocs; i++) + { + /* Send these offsets to process i.*/ + scounts[i] = count_my_req_per_proc[i]; + if ( scounts[i] == 0 ) + sdispls[i] = 0; + else + sdispls[i] = (int) + ( ( (MPIR_Upint)my_req[i].offsets - + (MPIR_Upint)sendBufForOffsets ) / + (MPIR_Upint)sizeof(ADIO_Offset) ); + + /* Receive these offsets from process i.*/ + rcounts[i] = count_others_req_per_proc[i]; + if ( rcounts[i] == 0 ) + rdispls[i] = 0; + else + rdispls[i] = (int) + ( ( (MPIR_Upint)others_req[i].offsets - + (MPIR_Upint)recvBufForOffsets ) / + (MPIR_Upint)sizeof(ADIO_Offset) ); + } + + /* Exchange the offsets */ + MPI_Alltoallv(sendBufForOffsets, + scounts, sdispls, ADIO_OFFSET, + recvBufForOffsets, + rcounts, rdispls, ADIO_OFFSET, + fd->comm); + + /************************/ + /* Exchange the lengths */ + /************************/ + + for (i=0; i<nprocs; i++) + { + /* Send these lengths to process i.*/ + scounts[i] = count_my_req_per_proc[i]; + if ( scounts[i] == 0 ) + sdispls[i] = 0; + else + sdispls[i] = (int) + ( ( (MPIR_Upint)my_req[i].lens - + (MPIR_Upint)sendBufForLens ) / + (MPIR_Upint) sizeof(int) ); + + /* Receive these offsets from process i. */ + rcounts[i] = count_others_req_per_proc[i]; + if ( rcounts[i] == 0 ) + rdispls[i] = 0; + else + rdispls[i] = (int) + ( ( (MPIR_Upint)others_req[i].lens - + (MPIR_Upint)recvBufForLens ) / + (MPIR_Upint) sizeof(int) ); + } + + /* Exchange the lengths */ + MPI_Alltoallv(sendBufForLens, + scounts, sdispls, MPI_INT, + recvBufForLens, + rcounts, rdispls, MPI_INT, + fd->comm); + + /* Clean up */ + ADIOI_Free(count_others_req_per_proc); + ADIOI_Free (scounts); + ADIOI_Free (sdispls); + ADIOI_Free (rcounts); + ADIOI_Free (rdispls); + + *count_others_req_procs_ptr = count_others_req_procs; +#ifdef AGGREGATION_PROFILE + MPE_Log_event (5027, 0, NULL); +#endif + TRACE_ERR("Leaving ADIOI_GPFS_Calc_others_req\n"); +} ----------------------------------------------------------------------- Summary of changes: src/mpi/romio/adio/ad_gpfs/ad_gpfs_aggrs.c | 1694 ++++++++++---------- src/mpi/romio/adio/ad_gpfs/ad_gpfs_rdcoll.c | 4 +- src/mpi/romio/adio/ad_gpfs/ad_gpfs_wrcoll.c | 8 +- src/mpi/romio/adio/ad_lustre/ad_lustre_aggregate.c | 4 +- src/mpi/romio/adio/ad_lustre/ad_lustre_wrcoll.c | 2 +- src/mpi/romio/adio/common/Makefile.mk | 3 +- src/mpi/romio/adio/common/ad_aggregate.c | 18 +- src/mpi/romio/adio/common/ad_read_coll.c | 4 +- src/mpi/romio/adio/common/ad_write_coll.c | 6 +- src/mpi/romio/adio/common/utils.c | 83 + src/mpi/romio/adio/include/adioi.h | 11 +- test/mpi/io/bigtype.c | 5 +- 12 files changed, 965 insertions(+), 877 deletions(-) create mode 100644 src/mpi/romio/adio/common/utils.c hooks/post-receive -- MPICH primary repository

1 0

[mpich] MPICH primary repository branch, master, updated. v3.1.2-86-g5dac839
by noreply＠mpich.org 11 Aug '14

11 Aug '14

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "MPICH primary repository". The branch, master has been updated via 5dac839337d807f45f46ffd8c1d3905badb2f910 (commit) from 95cfe8a33eca95eb722a6216f997c6d7306ded98 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- http://git.mpich.org/mpich.git/commitdiff/5dac839337d807f45f46ffd8c1d3905ba… commit 5dac839337d807f45f46ffd8c1d3905badb2f910 Author: Wesley Bland <wbland(a)anl.gov> Date: Mon Aug 11 09:46:39 2014 -0500 Change the func used to poke the progress engine Inside mpid_comm_get_all_failed_procs.c, we were using `MPIDI_CH3I_Progress` to poke the progress engine. This isn't the right function however since the signature is different in sock and nemesis. Now it uses `MPID_Progress_poke` which is the correct way to do this. Signed-off-by: Rob Latham <robl(a)mcs.anl.gov> diff --git a/src/mpid/ch3/src/mpid_comm_get_all_failed_procs.c b/src/mpid/ch3/src/mpid_comm_get_all_failed_procs.c index 882f30a..c8eb7f6 100644 --- a/src/mpid/ch3/src/mpid_comm_get_all_failed_procs.c +++ b/src/mpid/ch3/src/mpid_comm_get_all_failed_procs.c @@ -94,7 +94,7 @@ int MPID_Comm_get_all_failed_procs(MPID_Comm *comm_ptr, MPID_Group **failed_grou /* Kick the progress engine in case it's been a while so we get all the * latest updates about failures */ - MPIDI_CH3I_Progress(NULL, false); + MPID_Progress_poke(); /* Generate the list of failed processes */ MPIDI_CH3U_Check_for_failed_procs(); ----------------------------------------------------------------------- Summary of changes: src/mpid/ch3/src/mpid_comm_get_all_failed_procs.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) hooks/post-receive -- MPICH primary repository

1 0

[mpich] MPICH primary repository branch, master, updated. v3.1.2-85-g95cfe8a
by noreply＠mpich.org 08 Aug '14

08 Aug '14

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "MPICH primary repository". The branch, master has been updated via 95cfe8a33eca95eb722a6216f997c6d7306ded98 (commit) via cf8ec4e69b6a675e2602e8f84d56ae283c823ddd (commit) from 8b73bd97d382c6ac8bd8d5158a475bfe6b78a063 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- http://git.mpich.org/mpich.git/commitdiff/95cfe8a33eca95eb722a6216f997c6d73… commit 95cfe8a33eca95eb722a6216f997c6d7306ded98 Author: Igor Ivanov <Igor.Ivanov(a)itseez.com> Date: Fri Aug 8 16:19:32 2014 +0300 netmod/mxm: Update MXM List of changes: - Adapted cancel recv to meet current code; - Changed active messaging callback flag; - Added possibility to configurate MXM netmode using MPICH_NETMOD_MXM_BULK_CONNECT and MPICH_NETMOD_MXM_BULK_DISCONNECT; - Improved debug output; Signed-off-by: Igor Ivanov <Igor.Ivanov(a)itseez.com> Signed-off-by: Pavan Balaji <balaji(a)anl.gov> diff --git a/src/mpid/ch3/channels/nemesis/netmod/mxm/mxm_cancel.c b/src/mpid/ch3/channels/nemesis/netmod/mxm/mxm_cancel.c index 6144ace..df49239 100644 --- a/src/mpid/ch3/channels/nemesis/netmod/mxm/mxm_cancel.c +++ b/src/mpid/ch3/channels/nemesis/netmod/mxm/mxm_cancel.c @@ -66,7 +66,10 @@ int MPID_nem_mxm_cancel_recv(MPIDI_VC_t * vc, MPID_Request * req) _dbg_mxm_out_req(req); fn_exit: - return mpi_errno; + /* This function returns sero in case request is canceled + * and nonzero otherwise + */ + return (!MPIR_STATUS_GET_CANCEL_BIT(req->status)); fn_fail:ATTRIBUTE((unused)) goto fn_exit; } diff --git a/src/mpid/ch3/channels/nemesis/netmod/mxm/mxm_init.c b/src/mpid/ch3/channels/nemesis/netmod/mxm/mxm_init.c index bc6d6d4..da5409a 100644 --- a/src/mpid/ch3/channels/nemesis/netmod/mxm/mxm_init.c +++ b/src/mpid/ch3/channels/nemesis/netmod/mxm/mxm_init.c @@ -67,6 +67,7 @@ static int _mxm_connect(MPID_nem_mxm_ep_t * ep, const char *business_card, static int _mxm_disconnect(MPID_nem_mxm_ep_t * ep); static int _mxm_add_comm(MPID_Comm * comm, void *param); static int _mxm_del_comm(MPID_Comm * comm, void *param); +static int _mxm_conf(void); #undef FUNCNAME @@ -78,6 +79,7 @@ int MPID_nem_mxm_post_init(void) int mpi_errno = MPI_SUCCESS; #if MXM_API >= MXM_VERSION(3,1) + /* Current logic guarantees that all VCs have been initialized before post init call */ if (_mxm_obj.conf.bulk_connect) { mxm_ep_wireup(_mxm_obj.mxm_ep); } @@ -308,8 +310,16 @@ int MPID_nem_mxm_vc_terminate(MPIDI_VC_t * vc) MPIDI_STATE_DECL(MPID_STATE_MXM_VC_TERMINATE); MPIDI_FUNC_ENTER(MPID_STATE_MXM_VC_TERMINATE); - while ((VC_FIELD(vc, pending_sends)) > 0) - MPID_nem_mxm_poll(FALSE); + if (vc->state != MPIDI_VC_STATE_CLOSED) { + /* VC is terminated as a result of a fault. Complete + * outstanding sends with an error and terminate connection + * immediately. */ + MPIU_ERR_SET1(mpi_errno, MPI_ERR_OTHER, "**comm_fail", "**comm_fail %d", vc->pg_rank); + } + else { + while ((VC_FIELD(vc, pending_sends)) > 0) + MPID_nem_mxm_poll(FALSE); + } mpi_errno = MPIDI_CH3U_Handle_connection(vc, MPIDI_VC_EVENT_TERMINATED); if (mpi_errno) @@ -322,11 +332,12 @@ int MPID_nem_mxm_vc_terminate(MPIDI_VC_t * vc) goto fn_exit; } -static int _mxm_init(int rank, int size) +static int _mxm_conf(void) { int mpi_errno = MPI_SUCCESS; mxm_error_t ret = MXM_OK; unsigned long cur_ver; + char *env_val = NULL; cur_ver = mxm_get_version(); if (cur_ver != MXM_API) { @@ -347,14 +358,34 @@ static int _mxm_init(int rank, int size) "%ld.%ld", (cur_ver >> MXM_MAJOR_BIT) & 0xff, (cur_ver >> MXM_MINOR_BIT) & 0xff); #endif - if (cur_ver < MXM_VERSION(3, 2)) { + env_val = getenv("MPICH_NETMOD_MXM_BULK_CONNECT"); + _mxm_obj.conf.bulk_connect = (env_val ? atoi(env_val) : (cur_ver < MXM_VERSION(3, 2) ? 0 : 1)); + env_val = getenv("MPICH_NETMOD_MXM_BULK_DISCONNECT"); + _mxm_obj.conf.bulk_disconnect = (env_val ? + atoi(env_val) : (cur_ver < MXM_VERSION(3, 2) ? 0 : 1)); + if (cur_ver < MXM_VERSION(3, 2) && + (_mxm_obj.conf.bulk_connect || _mxm_obj.conf.bulk_disconnect)) { _mxm_obj.conf.bulk_connect = 0; _mxm_obj.conf.bulk_disconnect = 0; + MPIU_DBG_MSG_FMT(CH3_CHANNEL, VERBOSE, + (MPIU_DBG_FDEST, + "WARNING: MPICH runs with %s version of MXM that is less than 3.2, " + "so bulk connect/disconnect cannot work properly and will be turn off.", + _mxm_obj.runtime_version)); } - else { - _mxm_obj.conf.bulk_connect = 1; - _mxm_obj.conf.bulk_disconnect = 1; - } + + fn_exit: + return mpi_errno; + fn_fail: + goto fn_exit; +} + +static int _mxm_init(int rank, int size) +{ + int mpi_errno = MPI_SUCCESS; + mxm_error_t ret = MXM_OK; + + ret = _mxm_conf(); ret = mxm_config_read_opts(&_mxm_obj.mxm_ctx_opts, &_mxm_obj.mxm_ep_opts, "MPICH2", NULL, 0); MPIU_ERR_CHKANDJUMP1(ret != MXM_OK, @@ -369,7 +400,7 @@ static int _mxm_init(int rank, int size) ret = mxm_set_am_handler(_mxm_obj.mxm_context, MXM_MPICH_HID_ADI_MSG, MPID_nem_mxm_get_adi_msg, - MXM_AM_FLAG_THREAD_SAFE); + 0); MPIU_ERR_CHKANDJUMP1(ret != MXM_OK, mpi_errno, MPI_ERR_OTHER, "**mxm_set_am_handler", "**mxm_set_am_handler %s", mxm_error_string(ret)); diff --git a/src/mpid/ch3/channels/nemesis/netmod/mxm/mxm_poll.c b/src/mpid/ch3/channels/nemesis/netmod/mxm/mxm_poll.c index 50487fa..df6fa59 100644 --- a/src/mpid/ch3/channels/nemesis/netmod/mxm/mxm_poll.c +++ b/src/mpid/ch3/channels/nemesis/netmod/mxm/mxm_poll.c @@ -69,10 +69,10 @@ void MPID_nem_mxm_get_adi_msg(mxm_conn_h conn, mxm_imm_t imm, void *data, MPIU_DBG_MSG(CH3_CHANNEL, VERBOSE, "MPID_nem_mxm_get_adi_msg"); - _dbg_mxm_output(5, "========> Getting ADI msg (data_size %d) \n", length); - vc = mxm_conn_ctx_get(conn); + _dbg_mxm_output(5, "========> Getting ADI msg (from=%d data_size %d) \n", vc->pg_rank, length); + MPID_nem_handle_pkt(vc, data, (MPIDI_msg_sz_t) (length)); } @@ -379,7 +379,6 @@ static int _mxm_irecv(MPID_nem_mxm_ep_t * ep, MPID_nem_mxm_req_area * req, int i mpi_errno = MPI_ERR_OTHER; goto fn_fail; } -// list_enqueue(&ep->out_queue, &req->mxm_req->queue); fn_exit: return mpi_errno; diff --git a/src/mpid/ch3/channels/nemesis/netmod/mxm/mxm_send.c b/src/mpid/ch3/channels/nemesis/netmod/mxm/mxm_send.c index 7a8f12a..3e60d56 100644 --- a/src/mpid/ch3/channels/nemesis/netmod/mxm/mxm_send.c +++ b/src/mpid/ch3/channels/nemesis/netmod/mxm/mxm_send.c @@ -54,8 +54,10 @@ int MPID_nem_mxm_iSendContig(MPIDI_VC_t * vc, MPID_Request * sreq, void *hdr, MP REQ_FIELD(sreq, iov_buf)[1].length = data_sz; } - _dbg_mxm_output(5, "iSendContig ========> Sending ADI msg for req %p (data_size %d, %d) \n", - sreq, sizeof(MPIDI_CH3_Pkt_t), data_sz); + _dbg_mxm_output(5, + "iSendContig ========> Sending ADI msg (to=%d type=%d) for req %p (data_size %d, %d) \n", + vc->pg_rank, sreq->dev.pending_pkt.type, sreq, sizeof(MPIDI_CH3_Pkt_t), + data_sz); (VC_FIELD(vc, pending_sends)) += 1; sreq->ch.vc = vc; @@ -100,8 +102,10 @@ int MPID_nem_mxm_iStartContigMsg(MPIDI_VC_t * vc, void *hdr, MPIDI_msg_sz_t hdr_ sreq->dev.OnDataAvail = NULL; sreq->dev.tmpbuf = NULL; - _dbg_mxm_output(5, "iStartContigMsg ========> Sending ADI msg for req %p (data_size %d, %d) \n", - sreq, sizeof(MPIDI_CH3_Pkt_t), data_sz); + _dbg_mxm_output(5, + "iStartContigMsg ========> Sending ADI msg (to=%d type=%d) for req %p (data_size %d, %d) \n", + vc->pg_rank, sreq->dev.pending_pkt.type, sreq, sizeof(MPIDI_CH3_Pkt_t), + data_sz); REQ_FIELD(sreq, ctx) = sreq; REQ_FIELD(sreq, iov_buf) = REQ_FIELD(sreq, tmp_buf); @@ -151,8 +155,10 @@ int MPID_nem_mxm_SendNoncontig(MPIDI_VC_t * vc, MPID_Request * sreq, void *hdr, MPIU_Memcpy(&(sreq->dev.pending_pkt), (char *) hdr, sizeof(MPIDI_CH3_Pkt_t)); sreq->dev.tmpbuf = NULL; - _dbg_mxm_output(5, "SendNoncontig ========> Sending ADI msg for req %p (data_size %d, %d) \n", - sreq, sizeof(MPIDI_CH3_Pkt_t), sreq->dev.segment_size); + _dbg_mxm_output(5, + "SendNoncontig ========> Sending ADI msg (to=%d type=%d) for req %p (data_size %d, %d) \n", + vc->pg_rank, sreq->dev.pending_pkt.type, sreq, sizeof(MPIDI_CH3_Pkt_t), + sreq->dev.segment_size); REQ_FIELD(sreq, ctx) = sreq; REQ_FIELD(sreq, iov_buf) = REQ_FIELD(sreq, tmp_buf); @@ -269,8 +275,7 @@ int MPID_nem_mxm_send(MPIDI_VC_t * vc, const void *buf, int count, MPI_Datatype mpi_errno = _mxm_isend(VC_FIELD(vc, mxm_ep), REQ_BASE(sreq), MXM_MPICH_ISEND, (mxm_mq_h) comm->ch.netmod_comm, comm->rank, tag, _mxm_tag_mpi2mxm(tag, - comm-> - context_id + comm->context_id + context_offset), 1); @@ -367,8 +372,7 @@ int MPID_nem_mxm_ssend(MPIDI_VC_t * vc, const void *buf, int count, MPI_Datatype mpi_errno = _mxm_isend(VC_FIELD(vc, mxm_ep), REQ_BASE(sreq), MXM_MPICH_ISEND_SYNC, (mxm_mq_h) comm->ch.netmod_comm, comm->rank, tag, _mxm_tag_mpi2mxm(tag, - comm-> - context_id + comm->context_id + context_offset), 1); @@ -465,8 +469,7 @@ int MPID_nem_mxm_isend(MPIDI_VC_t * vc, const void *buf, int count, MPI_Datatype mpi_errno = _mxm_isend(VC_FIELD(vc, mxm_ep), REQ_BASE(sreq), MXM_MPICH_ISEND, (mxm_mq_h) comm->ch.netmod_comm, comm->rank, tag, _mxm_tag_mpi2mxm(tag, - comm-> - context_id + comm->context_id + context_offset), 0); @@ -564,8 +567,7 @@ int MPID_nem_mxm_issend(MPIDI_VC_t * vc, const void *buf, int count, MPI_Datatyp mpi_errno = _mxm_isend(VC_FIELD(vc, mxm_ep), REQ_BASE(sreq), MXM_MPICH_ISEND_SYNC, (mxm_mq_h) comm->ch.netmod_comm, comm->rank, tag, _mxm_tag_mpi2mxm(tag, - comm-> - context_id + comm->context_id + context_offset), 0); @@ -712,7 +714,6 @@ static int _mxm_isend(MPID_nem_mxm_ep_t * ep, MPID_nem_mxm_req_area * req, if (block) _mxm_req_wait(&mxm_sreq->base); -// list_enqueue(&ep->out_queue, &req->mxm_req->queue); fn_exit: return mpi_errno; http://git.mpich.org/mpich.git/commitdiff/cf8ec4e69b6a675e2602e8f84d56ae283… commit cf8ec4e69b6a675e2602e8f84d56ae283c823ddd Author: Igor Ivanov <Igor.Ivanov(a)itseez.com> Date: Tue Jul 29 18:58:31 2014 +0300 nemesis: Fix memory leak related MPID_nem_register_initcomp_cb Every call of MPID_nem_register_initcomp_cb allocates memory but it is not freed after completion of all post netmod init callbacks. Signed-off-by: Igor Ivanov <Igor.Ivanov(a)itseez.com> Change-Id: Iacbbf4d95cf4ed11c2fc9f753d42608b19ef0d35 Signed-off-by: Pavan Balaji <balaji(a)anl.gov> diff --git a/src/mpid/ch3/channels/nemesis/src/ch3_init.c b/src/mpid/ch3/channels/nemesis/src/ch3_init.c index 78a44a9..0cb3a36 100644 --- a/src/mpid/ch3/channels/nemesis/src/ch3_init.c +++ b/src/mpid/ch3/channels/nemesis/src/ch3_init.c @@ -346,6 +346,7 @@ int MPIDI_CH3_InitCompleted(void) { int mpi_errno = MPI_SUCCESS; initcomp_cb_t *ep; + initcomp_cb_t *ep_tmp; MPIDI_STATE_DECL(MPID_STATE_MPIDI_CH3_INITCOMPLETED); MPIDI_FUNC_ENTER(MPID_STATE_MPIDI_CH3_INITCOMPLETED); @@ -354,7 +355,9 @@ int MPIDI_CH3_InitCompleted(void) { mpi_errno = ep->callback(); if (mpi_errno) MPIU_ERR_POP(mpi_errno); + ep_tmp = ep; ep = ep->next; + MPIU_Free(ep_tmp); } fn_exit: ----------------------------------------------------------------------- Summary of changes: .../ch3/channels/nemesis/netmod/mxm/mxm_cancel.c | 5 ++- .../ch3/channels/nemesis/netmod/mxm/mxm_init.c | 49 ++++++++++++++++---- .../ch3/channels/nemesis/netmod/mxm/mxm_poll.c | 5 +- .../ch3/channels/nemesis/netmod/mxm/mxm_send.c | 31 ++++++------ src/mpid/ch3/channels/nemesis/src/ch3_init.c | 3 + 5 files changed, 65 insertions(+), 28 deletions(-) hooks/post-receive -- MPICH primary repository

1 0

[mpich] MPICH primary repository branch, master, updated. v3.1.2-83-g8b73bd9
by noreply＠mpich.org 07 Aug '14

07 Aug '14

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "MPICH primary repository". The branch, master has been updated via 8b73bd97d382c6ac8bd8d5158a475bfe6b78a063 (commit) from a415703dcef90dc2282856da481955f11ebb17a0 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- http://git.mpich.org/mpich.git/commitdiff/8b73bd97d382c6ac8bd8d5158a475bfe6… commit 8b73bd97d382c6ac8bd8d5158a475bfe6b78a063 Author: Sangmin Seo <sseo(a)anl.gov> Date: Tue Aug 5 23:12:42 2014 -0500 Add dependency check for external libs Since automake cannot determine the value of 'external_libs = @mpllib@ @opalib@' in Makefile.am, $(external_libs) is missed in lib_lib@PMPILIBNAME@_la_DEPENDENCIES and lib_lib@MPILIBNAME@_la_DEPENDENCIES. This patch adds $(external_libs) to EXTRA_lib_lib@PMPILIBNAME@_la_DEPENDENCIES and EXTRA_lib_lib@MPILIBNAME@_la_DEPENDENCIES so that dependency for $(external_libs) is correctly checked. This automake email thread describes the issue: http://lists.gnu.org/archive/html/automake/2008-06/msg00077.html Fixes #2131 Signed-off-by: Ken Raffenetti <raffenet(a)mcs.anl.gov> diff --git a/Makefile.am b/Makefile.am index 654bd82..6af4b62 100644 --- a/Makefile.am +++ b/Makefile.am @@ -151,6 +151,7 @@ lib_LTLIBRARIES += lib/lib@[email protected] lib_lib@PMPILIBNAME@_la_SOURCES = $(mpi_sources) $(mpi_f77_sources) $(mpi_core_sources) lib_lib@PMPILIBNAME@_la_LDFLAGS = $(external_ldflags) $(ABIVERSIONFLAGS) lib_lib@PMPILIBNAME@_la_LIBADD = $(external_libs) $(pmpi_convenience_libs) +EXTRA_lib_lib@PMPILIBNAME@_la_DEPENDENCIES = $(external_libs) # lib@[email protected] might depend on lib@[email protected]. We add them # in that order to lib_LTLIBRARIES so libtool doesn't get @@ -168,6 +169,7 @@ lib_lib@MPILIBNAME@_la_SOURCES = $(mpi_sources) $(mpi_core_sources) lib_lib@MPILIBNAME@_la_LDFLAGS = $(external_ldflags) $(ABIVERSIONFLAGS) lib_lib@MPILIBNAME@_la_CPPFLAGS = $(AM_CPPFLAGS) lib_lib@MPILIBNAME@_la_LIBADD = $(external_libs) $(pmpi_convenience_libs) $(mpi_convenience_libs) +EXTRA_lib_lib@MPILIBNAME@_la_DEPENDENCIES = $(external_libs) endif !BUILD_PROFILING_LIB if BUILD_F77_BINDING ----------------------------------------------------------------------- Summary of changes: Makefile.am | 2 ++ 1 files changed, 2 insertions(+), 0 deletions(-) hooks/post-receive -- MPICH primary repository

1 0

[mpich] MPICH primary repository branch, master, updated. v3.1.2-82-ga415703
by noreply＠mpich.org 07 Aug '14

07 Aug '14

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "MPICH primary repository". The branch, master has been updated via a415703dcef90dc2282856da481955f11ebb17a0 (commit) via 0e675b029b3aea43bdb16e732fe87eaac9ad06c8 (commit) via cef2f1ecfe5be54825776982567c2398bed71af3 (commit) from 206af6d56ce9e1f965a26c5f82b8c484f8bb7596 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- http://git.mpich.org/mpich.git/commitdiff/a415703dcef90dc2282856da481955f11… commit a415703dcef90dc2282856da481955f11ebb17a0 Author: Rob Latham <robl(a)mcs.anl.gov> Date: Mon Aug 4 14:57:30 2014 -0500 test case for hindexed_block type Mohamad Chaarawi <chaarawi(a)hdfgroup.org> discovered that ROMIO did not correctly implement hindexed_block datatypes. Signed-off-by: Ken Raffenetti <raffenet(a)mcs.anl.gov> diff --git a/test/mpi/io/Makefile.am b/test/mpi/io/Makefile.am index 12dd69d..4d5ba77 100644 --- a/test/mpi/io/Makefile.am +++ b/test/mpi/io/Makefile.am @@ -24,7 +24,8 @@ noinst_PROGRAMS = \ userioerr \ resized \ resized2 \ - bigtype + bigtype \ + hindexed_io clean-local: -rm -f testfile testfile.* diff --git a/test/mpi/io/hindexed_io.c b/test/mpi/io/hindexed_io.c new file mode 100644 index 0000000..3261278 --- /dev/null +++ b/test/mpi/io/hindexed_io.c @@ -0,0 +1,106 @@ +#include <mpi.h> +#include <stdio.h> +#include <string.h> +#include <stdlib.h> +#include <unistd.h> + +#define DATA_SIZE 324*4 +#define PAD 256 +#define HEADER 144 +#define BLK_COUNT 3 + +static void handle_error(int errcode, char *str) +{ + char msg[MPI_MAX_ERROR_STRING]; + int resultlen; + MPI_Error_string(errcode, msg, &resultlen); + fprintf(stderr, "%s: %s\n", str, msg); + MPI_Abort(MPI_COMM_WORLD, 1); +} +#define CHECK(fn) { int errcode; errcode = (fn); if (errcode != MPI_SUCCESS) handle_error(errcode, #fn ); } + +int main(int argc, char** argv) +{ + + MPI_File fh; + MPI_Datatype file_type, mem_type; + int *data = NULL; + int *verify = NULL; + int data_size = DATA_SIZE; + int i, j,k, nr_errors=0; + MPI_Aint disp[BLK_COUNT]; + int block_lens[BLK_COUNT]; + char* filename = "unnamed.dat"; + + MPI_Init (&argc, &argv); + disp[0] = (MPI_Aint)(PAD); + disp[1] = (MPI_Aint)(data_size*1 + PAD); + disp[2] = (MPI_Aint)(data_size*2 + PAD); + + block_lens[0] = data_size; + block_lens[1] = data_size; + block_lens[2] = data_size; + + data = malloc(data_size); + verify = malloc(data_size*BLK_COUNT + HEADER + PAD); + for(i=0 ; i<data_size/sizeof(int) ; i++) + data[i] = i; + + MPI_Type_create_hindexed_block(BLK_COUNT, data_size, disp, MPI_BYTE, &file_type); + MPI_Type_commit(&file_type); + + MPI_Type_create_hvector(BLK_COUNT, data_size, 0, MPI_BYTE, &mem_type); + MPI_Type_commit(&mem_type); + + if( 1 < argc ) filename = argv[1]; + + CHECK(MPI_File_open (MPI_COMM_WORLD, filename, + MPI_MODE_RDWR | MPI_MODE_CREATE | MPI_MODE_DELETE_ON_CLOSE, + MPI_INFO_NULL, &fh) != 0); + + CHECK(MPI_File_set_view(fh, HEADER, MPI_BYTE, + file_type, "native", MPI_INFO_NULL)); + + /* write everything */ + CHECK(MPI_File_write_at_all (fh, 0, + data, 1, mem_type, + MPI_STATUS_IGNORE)); + /* verify */ + CHECK(MPI_File_set_view(fh, 0, MPI_BYTE, MPI_BYTE, + "native", MPI_INFO_NULL)); + CHECK(MPI_File_read_at_all(fh, 0, + verify, (HEADER+PAD+BLK_COUNT*DATA_SIZE)/sizeof(int), MPI_INT, + MPI_STATUS_IGNORE)); + + /* header and block padding should have no data */ + for (i=0; i<(HEADER+PAD)/sizeof(int); i++) { + if (verify[i] != 0) { + nr_errors++; + fprintf(stderr, "expected 0, read %d\n", verify[i]); + } + } + /* blocks are replicated */ + for (j=0; j<BLK_COUNT; j++ ) { + for (k=0; k<(DATA_SIZE/sizeof(int)); k++) { + if (verify[(HEADER+PAD)/sizeof(int) + k + j*(DATA_SIZE/sizeof(int))] != + data[k]) { + nr_errors++; + fprintf(stderr, "expcted %d, read %d\n", data[k], + verify[(HEADER+PAD)/sizeof(int) + k + j*(DATA_SIZE/sizeof(int))]); + } + i++; + } + } + + MPI_File_close(&fh); + + MPI_Type_free (&mem_type); + MPI_Type_free(&file_type); + + if (nr_errors == 0) printf(" No Errors\n"); + + MPI_Finalize (); + + free(data); + return 0; +} diff --git a/test/mpi/io/testlist b/test/mpi/io/testlist index 6dfaafb..2549091 100644 --- a/test/mpi/io/testlist +++ b/test/mpi/io/testlist @@ -10,3 +10,4 @@ userioerr 1 resized 1 resized2 1 xfail=ticket2088 bigtype 1 +hindexed_io 1 http://git.mpich.org/mpich.git/commitdiff/0e675b029b3aea43bdb16e732fe87eaac… commit 0e675b029b3aea43bdb16e732fe87eaac9ad06c8 Author: Rob Latham <robl(a)mcs.anl.gov> Date: Fri Aug 1 13:11:36 2014 -0500 HINDEXED_BLOCK is not quite an INDEXED_BLOCK Someone (Mohamad Chaarawi <chaarawi(a)hdfgroup.org>) finally used HINDEXED_BLOCK and discovered that ROMIO's HINDEXED_BLOCK implementation was.... incomplete. or at least untested. - ADIOI_Count_contiguous_blocks simply aborted when fed a type it did not know about. hindexed_block blocks are counted same as indexed_block blocks. - But, the stride between hindexed_block blocks is given by the explicit addresses. indexed_block, on the other hand, computes a stride based on type. Signed-off-by: Ken Raffenetti <raffenet(a)mcs.anl.gov> diff --git a/src/mpi/romio/adio/common/flatten.c b/src/mpi/romio/adio/common/flatten.c index bc5f1e2..86c2875 100644 --- a/src/mpi/romio/adio/common/flatten.c +++ b/src/mpi/romio/adio/common/flatten.c @@ -107,7 +107,7 @@ void ADIOI_Flatten_datatype(MPI_Datatype datatype) void ADIOI_Flatten(MPI_Datatype datatype, ADIOI_Flatlist_node *flat, ADIO_Offset st_offset, MPI_Count *curr_index) { - int i, k, m, n, basic_num, nonzeroth; + int i, k, m, n, basic_num, nonzeroth, is_hindexed_block=0; int combiner, old_combiner, old_is_contig; int nints, nadds, ntypes, old_nints, old_nadds, old_ntypes; /* By using ADIO_Offset we preserve +/- sign and @@ -484,11 +484,10 @@ void ADIOI_Flatten(MPI_Datatype datatype, ADIOI_Flatlist_node *flat, } break; - /* FIXME: using the same code as indexed_block for - * hindexed_block doesn't look correct. Needs to be carefully - * looked into. */ #if defined HAVE_DECL_MPI_COMBINER_HINDEXED_BLOCK && HAVE_DECL_MPI_COMBINER_HINDEXED_BLOCK case MPI_COMBINER_HINDEXED_BLOCK: + is_hindexed_block=1; + /* deliberate fall-through */ #endif case MPI_COMBINER_INDEXED_BLOCK: #ifdef FLATTEN_DEBUG @@ -506,8 +505,13 @@ void ADIOI_Flatten(MPI_Datatype datatype, ADIOI_Flatlist_node *flat, /* By using ADIO_Offset we preserve +/- sign and avoid >2G integer arithmetic problems */ ADIO_Offset stride = ints[1+1]; - ADIOI_Flatten(types[0], flat, - st_offset+stride* ADIOI_AINT_CAST_TO_OFFSET old_extent, curr_index); + if (is_hindexed_block) { + ADIOI_Flatten(types[0], flat, + st_offset+adds[0], curr_index); + } else { + ADIOI_Flatten(types[0], flat, + st_offset+stride* ADIOI_AINT_CAST_TO_OFFSET old_extent, curr_index); + } } if (prev_index == *curr_index) { @@ -516,8 +520,14 @@ void ADIOI_Flatten(MPI_Datatype datatype, ADIOI_Flatlist_node *flat, for (i=j; i<j+top_count; i++) { /* By using ADIO_Offset we preserve +/- sign and avoid >2G integer arithmetic problems */ - ADIO_Offset blocklength = ints[1], stride = ints[1+1+i-j]; - flat->indices[i] = st_offset + stride* ADIOI_AINT_CAST_TO_OFFSET old_extent; + ADIO_Offset blocklength = ints[1]; + if (is_hindexed_block) { + flat->indices[i] = st_offset + adds[i-j]; + } else { + ADIO_Offset stride = ints[1+1+i-j]; + flat->indices[i] = st_offset + + stride* ADIOI_AINT_CAST_TO_OFFSET old_extent; + } flat->blocklens[i] = blocklength* ADIOI_AINT_CAST_TO_OFFSET old_extent; } *curr_index = i; @@ -532,7 +542,13 @@ void ADIOI_Flatten(MPI_Datatype datatype, ADIOI_Flatlist_node *flat, and then strided. Replicate the first one. */ for (m=1; m<ints[1]; m++) { for (i=0; i<num; i++) { - flat->indices[j] = flat->indices[j-num] + ADIOI_AINT_CAST_TO_OFFSET old_extent; + if (is_hindexed_block) { + /* this is the one place the hindexed case uses the + * extent of a type */ + MPI_Type_extent(types[0], &old_extent); + } + flat->indices[j] = flat->indices[j-num] + + ADIOI_AINT_CAST_TO_OFFSET old_extent; flat->blocklens[j] = flat->blocklens[j-num]; j++; } @@ -543,10 +559,16 @@ void ADIOI_Flatten(MPI_Datatype datatype, ADIOI_Flatlist_node *flat, num = *curr_index - prev_index; for (i=1; i<top_count; i++) { for (m=0; m<num; m++) { - /* By using ADIO_Offset we preserve +/- sign and - avoid >2G integer arithmetic problems */ - ADIO_Offset stride = ints[2+i]-ints[1+i]; - flat->indices[j] = flat->indices[j-num] + stride* ADIOI_AINT_CAST_TO_OFFSET old_extent; + if (is_hindexed_block) { + flat->indices[j] = flat->indices[j-num] + + adds[i] - adds[i-1]; + } else { + /* By using ADIO_Offset we preserve +/- sign and + avoid >2G integer arithmetic problems */ + ADIO_Offset stride = ints[2+i]-ints[1+i]; + flat->indices[j] = flat->indices[j-num] + + stride* ADIOI_AINT_CAST_TO_OFFSET old_extent; + } flat->blocklens[j] = flat->blocklens[j-num]; j++; } @@ -963,6 +985,9 @@ MPI_Count ADIOI_Count_contiguous_blocks(MPI_Datatype datatype, MPI_Count *curr_i } break; +#if defined HAVE_DECL_MPI_COMBINER_HINDEXED_BLOCK && HAVE_DECL_MPI_COMBINER_HINDEXED_BLOCK + case MPI_COMBINER_HINDEXED_BLOCK: +#endif case MPI_COMBINER_INDEXED_BLOCK: top_count = ints[0]; MPI_Type_get_envelope(types[0], &old_nints, &old_nadds, diff --git a/src/mpi/romio/configure.ac b/src/mpi/romio/configure.ac index 1fe952d..68801ad 100644 --- a/src/mpi/romio/configure.ac +++ b/src/mpi/romio/configure.ac @@ -1558,6 +1558,7 @@ if test $FROM_OMPI = yes ; then # Open MPI: see comments in mpi-io/mpioprof.h AC_DEFINE(MPIO_BUILD_PROFILING, 1, [hack to make ROMIO build without profiling]) DEFINE_HAVE_MPI_GREQUEST="#define HAVE_MPI_GREQUEST" + AC_DEFINE(HAVE_DECL_MPI_COMBINER_HINDEXED_BLOCK, 1, [Define if MPI library provides HINDEXED_BLOCK datatype]) elif test $FROM_LAM = yes ; then # LAM does have the status set bytes functionality AC_DEFINE(HAVE_STATUS_SET_BYTES,1,[Define if have MPIR_Status_set_bytes]) @@ -1609,6 +1610,7 @@ elif test $FROM_MPICH = yes ; then AC_DEFINE(HAVE_MPIIO_CONST, const, Set if MPI-IO prototypes use const qualifier) AC_DEFINE(HAVE_MPI_TYPE_SIZE_X, 1, [Define if MPI library provides MPI_TYPE_SIZE_X]) AC_DEFINE(HAVE_MPI_STATUS_SET_ELEMENTS_X, 1, [Define if MPI library provides MPI_STATUS_SET_ELEMENTS_X]) + AC_DEFINE(HAVE_DECL_MPI_COMBINER_HINDEXED_BLOCK, 1, [Define if MPI library provides HINDEXED_BLOCK datatype]) fi # # http://git.mpich.org/mpich.git/commitdiff/cef2f1ecfe5be54825776982567c2398b… commit cef2f1ecfe5be54825776982567c2398bed71af3 Author: Rob Latham <robl(a)mcs.anl.gov> Date: Wed Jul 30 09:59:06 2014 -0500 rma-using tests belong in rma even though get-struct does exercise datatype processing code, it's use of RMA operations means it belongs in rma directory. Closes #2141 Signed-off-by: Ken Raffenetti <raffenet(a)mcs.anl.gov> diff --git a/test/mpi/datatype/Makefile.am b/test/mpi/datatype/Makefile.am index b19321a..03c13b5 100644 --- a/test/mpi/datatype/Makefile.am +++ b/test/mpi/datatype/Makefile.am @@ -79,8 +79,7 @@ noinst_PROGRAMS = \ unusual-noncontigs \ vecblklen \ zeroblks \ - zeroparms \ - get-struct + zeroparms # Some of the tests use a more comprehensive set of datatype tests. # These must specify a different LDADD that includes the object file diff --git a/test/mpi/datatype/testlist.in b/test/mpi/datatype/testlist.in index b2875b2..549c3eb 100644 --- a/test/mpi/datatype/testlist.in +++ b/test/mpi/datatype/testlist.in @@ -59,4 +59,3 @@ cxx-types 1 mpiversion=3.0 @largetest@large_type 1 mpiversion=3.0 @largetest@large_type_sendrec 2 arg=31 mpiversion=3.0 @largetest@large_type_sendrec 2 arg=32 mpiversion=3.0 timeLimit=360 -get-struct 2 diff --git a/test/mpi/rma/Makefile.am b/test/mpi/rma/Makefile.am index 40f3717..b2bf942 100644 --- a/test/mpi/rma/Makefile.am +++ b/test/mpi/rma/Makefile.am @@ -124,7 +124,8 @@ noinst_PROGRAMS = \ badrma \ nb_test \ acc-loc \ - fence_shm + fence_shm \ + get-struct strided_acc_indexed_LDADD = $(LDADD) -lm strided_acc_onelock_LDADD = $(LDADD) -lm diff --git a/test/mpi/datatype/get-struct.c b/test/mpi/rma/get-struct.c similarity index 99% rename from test/mpi/datatype/get-struct.c rename to test/mpi/rma/get-struct.c index 8970008..8a91e5f 100644 --- a/test/mpi/datatype/get-struct.c +++ b/test/mpi/rma/get-struct.c @@ -14,7 +14,7 @@ * * The observed failure was a SEGV in the MPI_Get * - * + * */ #define MAX_KEY_SIZE 64 #define MAX_VALUE_SIZE 256 diff --git a/test/mpi/rma/testlist.in b/test/mpi/rma/testlist.in index 72606d2..c1acd24 100644 --- a/test/mpi/rma/testlist.in +++ b/test/mpi/rma/testlist.in @@ -109,6 +109,7 @@ badrma 2 mpiversion=3.0 acc-loc 4 fence_shm 2 mpiversion=3.0 win_shared_zerobyte 4 mpiversion=3.0 +get-struct 2 ## This test is not strictly correct. This was meant to test out the ## case when MPI_Test is not nonblocking. However, we ended up ----------------------------------------------------------------------- Summary of changes: src/mpi/romio/adio/common/flatten.c | 51 +++++++++++---- src/mpi/romio/configure.ac | 2 + test/mpi/datatype/Makefile.am | 3 +- test/mpi/datatype/testlist.in | 1 - test/mpi/io/Makefile.am | 3 +- test/mpi/io/hindexed_io.c | 106 +++++++++++++++++++++++++++++++ test/mpi/io/testlist | 1 + test/mpi/rma/Makefile.am | 3 +- test/mpi/{datatype => rma}/get-struct.c | 2 +- test/mpi/rma/testlist.in | 1 + 10 files changed, 154 insertions(+), 19 deletions(-) create mode 100644 test/mpi/io/hindexed_io.c rename test/mpi/{datatype => rma}/get-struct.c (99%) hooks/post-receive -- MPICH primary repository

1 0

[mpich] MPICH primary repository branch, master, updated. v3.1.2-79-g206af6d
by noreply＠mpich.org 07 Aug '14

07 Aug '14

This is an automated email from the git hooks/post-receive script. It was generated because a ref change was pushed to the repository containing the project "MPICH primary repository". The branch, master has been updated via 206af6d56ce9e1f965a26c5f82b8c484f8bb7596 (commit) via d1df5c9c792397dde7b4ddcea859c89aaa9e3955 (commit) via 5c328ccb3d1752224a30740e5e1b5c3ed4e129f8 (commit) via dfce63a02289fbaebd776c76d2c90a7a21e96450 (commit) via 88f05f80a9e5b4c41a5a670484503fc413d7868f (commit) from 43bbd02bab428e1f6dab359ba77d49f5be9efd15 (commit) Those revisions listed above that are new to this repository have not appeared on any other notification email; so we list those revisions in full, below. - Log ----------------------------------------------------------------- http://git.mpich.org/mpich.git/commitdiff/206af6d56ce9e1f965a26c5f82b8c484f… commit 206af6d56ce9e1f965a26c5f82b8c484f8bb7596 Author: Antonio Pena Monferrer <apenya(a)mcs.anl.gov> Date: Mon Aug 4 18:54:52 2014 -0500 Fixed MPI_Probe/MPI/Mprobe/MPI_Mrecv (+I variants) First working version. See #2152. Signed-off-by: Ken Raffenetti <raffenet(a)mcs.anl.gov> diff --git a/src/mpid/ch3/channels/nemesis/netmod/portals4/Makefile.mk b/src/mpid/ch3/channels/nemesis/netmod/portals4/Makefile.mk index 40b1539..3901503 100644 --- a/src/mpid/ch3/channels/nemesis/netmod/portals4/Makefile.mk +++ b/src/mpid/ch3/channels/nemesis/netmod/portals4/Makefile.mk @@ -14,7 +14,8 @@ mpi_core_sources += \ src/mpid/ch3/channels/nemesis/netmod/portals4/ptl_probe.c \ src/mpid/ch3/channels/nemesis/netmod/portals4/ptl_recv.c \ src/mpid/ch3/channels/nemesis/netmod/portals4/ptl_nm.c \ - src/mpid/ch3/channels/nemesis/netmod/portals4/ptl_send.c + src/mpid/ch3/channels/nemesis/netmod/portals4/ptl_send.c \ + src/mpid/ch3/channels/nemesis/netmod/portals4/ptl_lmt.c noinst_HEADERS += \ src/mpid/ch3/channels/nemesis/netmod/portals4/ptl_impl.h diff --git a/src/mpid/ch3/channels/nemesis/netmod/portals4/ptl_impl.h b/src/mpid/ch3/channels/nemesis/netmod/portals4/ptl_impl.h index 9a2d3cf..9681bae 100644 --- a/src/mpid/ch3/channels/nemesis/netmod/portals4/ptl_impl.h +++ b/src/mpid/ch3/channels/nemesis/netmod/portals4/ptl_impl.h @@ -181,6 +181,13 @@ void MPID_nem_ptl_anysource_posted(MPID_Request *rreq); int MPID_nem_ptl_anysource_matched(MPID_Request *rreq); int MPID_nem_ptl_init_id(MPIDI_VC_t *vc); +int MPID_nem_ptl_lmt_initiate_lmt(MPIDI_VC_t *vc, MPIDI_CH3_Pkt_t *rts_pkt, MPID_Request *req); +int MPID_nem_ptl_lmt_start_recv(MPIDI_VC_t *vc, MPID_Request *rreq, MPID_IOV s_cookie); +int MPID_nem_ptl_lmt_start_send(MPIDI_VC_t *vc, MPID_Request *sreq, MPID_IOV r_cookie); +int MPID_nem_ptl_lmt_handle_cookie(MPIDI_VC_t *vc, MPID_Request *req, MPID_IOV s_cookie); +int MPID_nem_ptl_lmt_done_send(MPIDI_VC_t *vc, MPID_Request *req); +int MPID_nem_ptl_lmt_done_recv(MPIDI_VC_t *vc, MPID_Request *req); + /* debugging */ const char *MPID_nem_ptl_strerror(int ret); diff --git a/src/mpid/ch3/channels/nemesis/netmod/portals4/ptl_init.c b/src/mpid/ch3/channels/nemesis/netmod/portals4/ptl_init.c index 9496048..f5161db 100644 --- a/src/mpid/ch3/channels/nemesis/netmod/portals4/ptl_init.c +++ b/src/mpid/ch3/channels/nemesis/netmod/portals4/ptl_init.c @@ -97,7 +97,9 @@ static int ptl_init(MPIDI_PG_t *pg_p, int pg_rank, char **bc_val_p, int *val_max mpi_errno = MPIDI_CH3I_Register_anysource_notification(MPID_nem_ptl_anysource_posted, MPID_nem_ptl_anysource_matched); if (mpi_errno) MPIU_ERR_POP(mpi_errno); - + + MPIDI_Anysource_improbe_fn = MPID_nem_ptl_anysource_improbe; + /* init portals */ ret = PtlInit(); MPIU_ERR_CHKANDJUMP1(ret, mpi_errno, MPI_ERR_OTHER, "**ptlinit", "**ptlinit %s", MPID_nem_ptl_strerror(ret)); @@ -281,6 +283,13 @@ static int vc_init(MPIDI_VC_t *vc) vc_ch->iStartContigMsg = MPID_nem_ptl_iStartContigMsg; vc_ch->iSendContig = MPID_nem_ptl_iSendContig; + vc_ch->lmt_initiate_lmt = MPID_nem_ptl_lmt_initiate_lmt; + vc_ch->lmt_start_recv = MPID_nem_ptl_lmt_start_recv; + vc_ch->lmt_start_send = MPID_nem_ptl_lmt_start_send; + vc_ch->lmt_handle_cookie = MPID_nem_ptl_lmt_handle_cookie; + vc_ch->lmt_done_send = MPID_nem_ptl_lmt_done_send; + vc_ch->lmt_done_recv = MPID_nem_ptl_lmt_done_recv; + vc->comm_ops = &comm_ops; vc_ch->next = NULL; diff --git a/src/mpid/ch3/channels/nemesis/netmod/portals4/ptl_lmt.c b/src/mpid/ch3/channels/nemesis/netmod/portals4/ptl_lmt.c new file mode 100644 index 0000000..7979db8 --- /dev/null +++ b/src/mpid/ch3/channels/nemesis/netmod/portals4/ptl_lmt.c @@ -0,0 +1,70 @@ +/* -*- Mode: C; c-basic-offset:4 ; indent-tabs-mode:nil ; -*- */ +/* + * (C) 2014 by Argonne National Laboratory. + * See COPYRIGHT in top-level directory. + */ + +#include "ptl_impl.h" + + + +#undef FUNCNAME +#define FUNCNAME MPID_nem_ptl_lmt_initiate_lmt +#undef FCNAME +#define FCNAME MPIU_QUOTE(FUNCNAME) +int MPID_nem_ptl_lmt_initiate_lmt(MPIDI_VC_t *vc, MPIDI_CH3_Pkt_t *rts_pkt, MPID_Request *req) +{ + /* Nothing to do here, but has to be defined for CH3 to follow the right path to + MPID_nem_ptl_lmt_start_recv */ + return MPI_SUCCESS; +} + + + +/* The following function is implemented in ptl_recv.c to make use of the handlers defined there */ +/* int MPID_nem_ptl_lmt_start_recv(MPIDI_VC_t *vc, MPID_Request *rreq, MPID_IOV s_cookie) */ + + + +#undef FUNCNAME +#define FUNCNAME MPID_nem_ptl_lmt_start_send +#undef FCNAME +#define FCNAME MPIU_QUOTE(FUNCNAME) +int MPID_nem_ptl_lmt_start_send(MPIDI_VC_t *vc, MPID_Request *sreq, MPID_IOV r_cookie) +{ + MPIU_Assertp(0 && "This function shouldn't be called."); + return MPI_ERR_INTERN; +} + + +#undef FUNCNAME +#define FUNCNAME MPID_nem_ptl_lmt_handle_cookie +#undef FCNAME +#define FCNAME MPIU_QUOTE(FUNCNAME) +int MPID_nem_ptl_lmt_handle_cookie(MPIDI_VC_t *vc, MPID_Request *req, MPID_IOV s_cookie) +{ + MPIU_Assertp(0 && "This function shouldn't be called."); + return MPI_ERR_INTERN; +} + + +#undef FUNCNAME +#define FUNCNAME MPID_nem_ptl_lmt_done_send +#undef FCNAME +#define FCNAME MPIU_QUOTE(FUNCNAME) +int MPID_nem_ptl_lmt_done_send(MPIDI_VC_t *vc, MPID_Request *req) +{ + MPIU_Assertp(0 && "This function shouldn't be called."); + return MPI_ERR_INTERN; +} + + +#undef FUNCNAME +#define FUNCNAME MPID_nem_ptl_lmt_done_recv +#undef FCNAME +#define FCNAME MPIU_QUOTE(FUNCNAME) +int MPID_nem_ptl_lmt_done_recv(MPIDI_VC_t *vc, MPID_Request *req) +{ + MPIU_Assertp(0 && "This function shouldn't be called."); + return MPI_ERR_INTERN; +} diff --git a/src/mpid/ch3/channels/nemesis/netmod/portals4/ptl_probe.c b/src/mpid/ch3/channels/nemesis/netmod/portals4/ptl_probe.c index 0dbfc7e..b7c7dcc 100644 --- a/src/mpid/ch3/channels/nemesis/netmod/portals4/ptl_probe.c +++ b/src/mpid/ch3/channels/nemesis/netmod/portals4/ptl_probe.c @@ -26,17 +26,66 @@ static int handle_probe(const ptl_event_t *e) REQ_PTL(req)->found = TRUE; req->status.MPI_SOURCE = NPTL_MATCH_GET_RANK(e->match_bits); req->status.MPI_TAG = NPTL_MATCH_GET_TAG(e->match_bits); - MPIR_STATUS_SET_COUNT(req->status, NPTL_HEADER_GET_LENGTH(e->match_bits)); + MPIR_STATUS_SET_COUNT(req->status, NPTL_HEADER_GET_LENGTH(e->hdr_data)); + fn_exit: MPIDI_CH3U_Request_complete(req); - - fn_exit: MPIDI_FUNC_EXIT(MPID_STATE_HANDLE_PROBE); return mpi_errno; fn_fail: goto fn_exit; } +static int handle_mprobe(const ptl_event_t *e) +{ + int mpi_errno = MPI_SUCCESS; + MPID_Request *const req = e->user_ptr; + MPIU_CHKPMEM_DECL(1); + MPIDI_STATE_DECL(MPID_STATE_HANDLE_PROBE); + + MPIDI_FUNC_ENTER(MPID_STATE_HANDLE_PROBE); + + if (e->ni_fail_type == PTL_NI_NO_MATCH) { + REQ_PTL(req)->found = FALSE; + goto fn_exit; + } + + REQ_PTL(req)->found = TRUE; + req->status.MPI_SOURCE = NPTL_MATCH_GET_RANK(e->match_bits); + req->status.MPI_TAG = NPTL_MATCH_GET_TAG(e->match_bits); + MPIR_STATUS_SET_COUNT(req->status, NPTL_HEADER_GET_LENGTH(e->hdr_data)); + MPIDI_Request_set_sync_send_flag(req, e->hdr_data & NPTL_SSEND); + + MPIU_CHKPMEM_MALLOC(req->dev.tmpbuf, void *, e->mlength, mpi_errno, "tmpbuf"); + MPIU_Memcpy((char *)req->dev.tmpbuf, e->start, e->mlength); + req->dev.recv_data_sz = e->mlength; + + if (!(e->hdr_data & NPTL_LARGE)) { + MPIDI_Request_set_msg_type(req, MPIDI_REQUEST_EAGER_MSG); + } + else { + MPIU_Assert (e->mlength == PTL_LARGE_THRESHOLD); + req->dev.match.parts.tag = req->status.MPI_TAG; + req->dev.match.parts.context_id = NPTL_MATCH_GET_CTX(e->match_bits); + req->dev.match.parts.rank = req->status.MPI_SOURCE; + MPIDI_Request_set_msg_type(req, MPIDI_REQUEST_RNDV_MSG); + } + + /* At this point we know the ME is unlinked. Invalidate the handle to + prevent further accesses, e.g. an attempted cancel. */ + REQ_PTL(req)->me = PTL_INVALID_HANDLE; + req->dev.recv_pending_count = 1; + + fn_exit: + MPIU_CHKPMEM_COMMIT(); + MPIDI_CH3U_Request_complete(req); + MPIDI_FUNC_EXIT(MPID_STATE_HANDLE_PROBE); + return mpi_errno; + fn_fail: + MPIU_CHKPMEM_REAP(); + goto fn_exit; +} + #undef FUNCNAME #define FUNCNAME MPID_nem_ptl_probe @@ -84,8 +133,13 @@ int MPID_nem_ptl_iprobe(MPIDI_VC_t *vc, int source, int tag, MPID_Comm *comm, in if (source == MPI_ANY_SOURCE) me.match_id = id_any; - else + else { + if (!vc_ptl->id_initialized) { + mpi_errno = MPID_nem_ptl_init_id(vc); + if (mpi_errno) MPIU_ERR_POP(mpi_errno); + } me.match_id = vc_ptl->id; + } if (tag == MPI_ANY_TAG) me.ignore_bits = NPTL_MATCH_IGNORE_ANY_TAG; @@ -95,8 +149,8 @@ int MPID_nem_ptl_iprobe(MPIDI_VC_t *vc, int source, int tag, MPID_Comm *comm, in /* submit a search request */ ret = PtlMESearch(MPIDI_nem_ptl_ni, MPIDI_nem_ptl_pt, &me, PTL_SEARCH_ONLY, req); MPIU_ERR_CHKANDJUMP1(ret, mpi_errno, MPI_ERR_OTHER, "**ptlmesearch", "**ptlmesearch %s", MPID_nem_ptl_strerror(ret)); - DBG_MSG_MESearch("REG", vc->pg_rank, me, req); - + DBG_MSG_MESearch("REG", vc ? vc->pg_rank : MPI_ANY_SOURCE, me, req); + /* wait for search request to complete */ do { mpi_errno = MPID_nem_ptl_poll(FALSE); @@ -123,11 +177,72 @@ int MPID_nem_ptl_improbe(MPIDI_VC_t *vc, int source, int tag, MPID_Comm *comm, i MPID_Request **message, MPI_Status *status) { int mpi_errno = MPI_SUCCESS; + MPID_nem_ptl_vc_area *const vc_ptl = VC_PTL(vc); + int ret; + ptl_process_t id_any; + ptl_me_t me; + MPID_Request *req; + MPIDI_STATE_DECL(MPID_STATE_MPID_NEM_PTL_IMPROBE); MPIDI_FUNC_ENTER(MPID_STATE_MPID_NEM_PTL_IMPROBE); + id_any.phys.nid = PTL_NID_ANY; + id_any.phys.pid = PTL_PID_ANY; + + /* create a request */ + req = MPID_Request_create(); + MPID_nem_ptl_init_req(req); + MPIU_ERR_CHKANDJUMP1(!req, mpi_errno, MPI_ERR_OTHER, "**nomem", "**nomem %s", "MPID_Request_create"); + MPIU_Object_set_ref(req, 2); /* 1 ref for progress engine and 1 ref for us */ + REQ_PTL(req)->event_handler = handle_mprobe; + req->kind = MPID_REQUEST_MPROBE; + + /* create a dummy ME to use for searching the list */ + me.start = NULL; + me.length = 0; + me.ct_handle = PTL_CT_NONE; + me.uid = PTL_UID_ANY; + me.options = ( PTL_ME_OP_PUT | PTL_ME_USE_ONCE ); + me.min_free = 0; + me.match_bits = NPTL_MATCH(tag, comm->context_id + context_offset, source); + + if (source == MPI_ANY_SOURCE) + me.match_id = id_any; + else { + if (!vc_ptl->id_initialized) { + mpi_errno = MPID_nem_ptl_init_id(vc); + if (mpi_errno) MPIU_ERR_POP(mpi_errno); + } + me.match_id = vc_ptl->id; + } + + if (tag == MPI_ANY_TAG) + me.ignore_bits = NPTL_MATCH_IGNORE_ANY_TAG; + else + me.ignore_bits = NPTL_MATCH_IGNORE; + /* submit a search request */ + ret = PtlMESearch(MPIDI_nem_ptl_ni, MPIDI_nem_ptl_pt, &me, PTL_SEARCH_DELETE, req); + MPIU_ERR_CHKANDJUMP1(ret, mpi_errno, MPI_ERR_OTHER, "**ptlmesearch", "**ptlmesearch %s", MPID_nem_ptl_strerror(ret)); + DBG_MSG_MESearch("REG", vc ? vc->pg_rank : 0, me, req); + /* wait for search request to complete */ + do { + mpi_errno = MPID_nem_ptl_poll(FALSE); + if (mpi_errno) MPIU_ERR_POP(mpi_errno); + } while (!MPID_Request_is_complete(req)); + + *flag = REQ_PTL(req)->found; + if (*flag) { + req->comm = comm; + MPIR_Comm_add_ref(comm); + MPIR_Request_extract_status(req, status); + *message = req; + } + else { + MPID_Request_release(req); + *message = MPI_MESSAGE_NULL; + } fn_exit: MPIDI_FUNC_EXIT(MPID_STATE_MPID_NEM_PTL_IMPROBE); @@ -147,6 +262,7 @@ int MPID_nem_ptl_anysource_iprobe(int tag, MPID_Comm * comm, int context_offset, MPIDI_FUNC_ENTER(MPID_STATE_MPID_NEM_PTL_ANYSOURCE_IPROBE); + return MPID_nem_ptl_iprobe(NULL, MPI_ANY_SOURCE, tag, comm, context_offset, flag, status); fn_exit: MPIDI_FUNC_EXIT(MPID_STATE_MPID_NEM_PTL_ANYSOURCE_IPROBE); @@ -167,6 +283,7 @@ int MPID_nem_ptl_anysource_improbe(int tag, MPID_Comm * comm, int context_offset MPIDI_FUNC_ENTER(MPID_STATE_MPID_NEM_PTL_ANYSOURCE_IMPROBE); + return MPID_nem_ptl_improbe(NULL, MPI_ANY_SOURCE, tag, comm, context_offset, flag, message, status); fn_exit: MPIDI_FUNC_EXIT(MPID_STATE_MPID_NEM_PTL_ANYSOURCE_IMPROBE); diff --git a/src/mpid/ch3/channels/nemesis/netmod/portals4/ptl_recv.c b/src/mpid/ch3/channels/nemesis/netmod/portals4/ptl_recv.c index 5e323f6..4e97a8d 100644 --- a/src/mpid/ch3/channels/nemesis/netmod/portals4/ptl_recv.c +++ b/src/mpid/ch3/channels/nemesis/netmod/portals4/ptl_recv.c @@ -598,3 +598,117 @@ int MPID_nem_ptl_cancel_recv(MPIDI_VC_t *vc, MPID_Request *rreq) fn_fail: goto fn_exit; } + + + +#undef FUNCNAME +#define FUNCNAME MPID_nem_ptl_lmt_start_recv +#undef FCNAME +#define FCNAME MPIU_QUOTE(FUNCNAME) +int MPID_nem_ptl_lmt_start_recv(MPIDI_VC_t *vc, MPID_Request *rreq, MPID_IOV s_cookie) +{ + /* This function should only be called as a result of an Mrecv because of the CH3 protocol for + Rendezvous Mrecvs. The regular CH3 protocol is not optimal for portals, since we don't need + to exchange CTS/RTS. We need this code here because at the time of the Mprobe we don't know + the target buffer, but we dequeue (and lose) the portals entry. This doesn't happen on + regular large transfers because we handle them directly on the netmod. */ + int mpi_errno = MPI_SUCCESS; + int dt_contig; + MPIDI_msg_sz_t data_sz; + MPID_Datatype *dt_ptr; + MPI_Aint dt_true_lb; + ptl_match_bits_t match_bits; + int was_incomplete; + int ret; + MPID_nem_ptl_vc_area *vc_ptl = VC_PTL(vc); + MPIU_CHKPMEM_DECL(1); + + MPIDI_STATE_DECL(MPID_STATE_MPID_NEM_PTL_LMT_START_RECV); + + MPIDI_FUNC_ENTER(MPID_STATE_MPID_NEM_PTL_LMT_START_RECV); + + /* This Rendezvous protocol does not do RTS-CTS. Since we have all the data, we limit to get it */ + /* The following code is inspired on handler_recv_dqueue_large */ + + match_bits = NPTL_MATCH(rreq->dev.match.parts.tag, rreq->dev.match.parts.context_id, + rreq->dev.match.parts.rank); + MPIDI_CH3U_Request_increment_cc(rreq, &was_incomplete); + MPIU_Assert(was_incomplete == 0); + MPIU_Object_add_ref(rreq); + + MPIDI_Datatype_get_info(rreq->dev.user_count, rreq->dev.datatype, dt_contig, data_sz, dt_ptr, + dt_true_lb); + if (dt_contig) { + void * real_user_buf = (char *)rreq->dev.user_buf + dt_true_lb; + + REQ_PTL(rreq)->event_handler = handler_recv_complete; + ret = PtlGet(MPIDI_nem_ptl_global_md, (ptl_size_t)((char *)real_user_buf + PTL_LARGE_THRESHOLD), + data_sz - PTL_LARGE_THRESHOLD, vc_ptl->id, vc_ptl->ptg, match_bits, 0, rreq); + DBG_MSG_GET("global", data_sz - PTL_LARGE_THRESHOLD, vc->pg_rank, match_bits); + MPIU_DBG_MSG_P(CH3_CHANNEL, VERBOSE, " buf=%p", (char *)real_user_buf + PTL_LARGE_THRESHOLD); + MPIU_ERR_CHKANDJUMP1(ret, mpi_errno, MPI_ERR_OTHER, "**ptlget", "**ptlget %s", + MPID_nem_ptl_strerror(ret)); + /* The memcpy is done after the get purposely for overlapping */ + MPIU_Memcpy(real_user_buf, rreq->dev.tmpbuf, PTL_LARGE_THRESHOLD); + } + else { + MPI_Aint last; + + rreq->dev.segment_ptr = MPID_Segment_alloc(); + MPIU_ERR_CHKANDJUMP1(rreq->dev.segment_ptr == NULL, mpi_errno, MPI_ERR_OTHER, "**nomem", + "**nomem %s", "MPID_Segment_alloc"); + MPID_Segment_init(rreq->dev.user_buf, rreq->dev.user_count, rreq->dev.datatype, + rreq->dev.segment_ptr, 0); + rreq->dev.segment_first = 0; + rreq->dev.segment_size = data_sz - PTL_LARGE_THRESHOLD; + last = PTL_LARGE_THRESHOLD; + MPID_Segment_unpack(rreq->dev.segment_ptr, rreq->dev.segment_first, &last, rreq->dev.tmpbuf); + MPIU_Assert(last == PTL_LARGE_THRESHOLD); + rreq->dev.segment_first = PTL_LARGE_THRESHOLD; + last = data_sz - PTL_LARGE_THRESHOLD; + rreq->dev.iov_count = MPID_IOV_LIMIT; + MPID_Segment_pack_vector(rreq->dev.segment_ptr, rreq->dev.segment_first, &last, rreq->dev.iov, + &rreq->dev.iov_count); + if (last == rreq->dev.segment_size) { + /* Rest of message fits in one IOV */ + ptl_md_t md; + + md.start = rreq->dev.iov; + md.length = rreq->dev.iov_count; + md.options = PTL_IOVEC; + md.eq_handle = MPIDI_nem_ptl_eq; + md.ct_handle = PTL_CT_NONE; + ret = PtlMDBind(MPIDI_nem_ptl_ni, &md, &REQ_PTL(rreq)->md); + MPIU_ERR_CHKANDJUMP1(ret, mpi_errno, MPI_ERR_OTHER, "**ptlmdbind", "**ptlmdbind %s", + MPID_nem_ptl_strerror(ret)); + + REQ_PTL(rreq)->event_handler = handler_recv_complete; + ret = PtlGet(REQ_PTL(rreq)->md, 0, rreq->dev.segment_size, vc_ptl->id, vc_ptl->ptg, + match_bits, PTL_LARGE_THRESHOLD, rreq); + MPIU_ERR_CHKANDJUMP1(ret, mpi_errno, MPI_ERR_OTHER, "**ptlget", "**ptlget %s", + MPID_nem_ptl_strerror(ret)); + } + else { + /* message won't fit in a single IOV, allocate buffer and unpack when received */ + /* FIXME: For now, allocate a single large buffer to hold entire message */ + MPIU_CHKPMEM_MALLOC(REQ_PTL(rreq)->chunk_buffer[0], void *, rreq->dev.segment_size, + mpi_errno, "chunk_buffer"); + REQ_PTL(rreq)->event_handler = handler_recv_unpack_complete; + ret = PtlGet(MPIDI_nem_ptl_global_md, (ptl_size_t)REQ_PTL(rreq)->chunk_buffer[0], + rreq->dev.segment_size, vc_ptl->id, vc_ptl->ptg, match_bits, + PTL_LARGE_THRESHOLD, rreq); + MPIU_ERR_CHKANDJUMP1(ret, mpi_errno, MPI_ERR_OTHER, "**ptlget", "**ptlget %s", + MPID_nem_ptl_strerror(ret)); + } + } + MPIU_Free(rreq->dev.tmpbuf); + rreq->ch.lmt_tmp_cookie.MPID_IOV_LEN = 0; /* Required for do_cts in mpid_nem_lmt.c */ + + fn_exit: + MPIU_CHKPMEM_COMMIT(); + MPIDI_FUNC_EXIT(MPID_STATE_MPID_NEM_PTL_LMT_START_RECV); + return mpi_errno; + fn_fail: + MPIU_CHKPMEM_REAP(); + goto fn_exit; +} http://git.mpich.org/mpich.git/commitdiff/d1df5c9c792397dde7b4ddcea859c89aa… commit d1df5c9c792397dde7b4ddcea859c89aaa9e3955 Author: Antonio Pena Monferrer <apenya(a)mcs.anl.gov> Date: Mon Aug 4 18:35:25 2014 -0500 Added 4 more tests to mprobe The following tests are added: - MPI_Ssend - Large transfers (Rendezvous) - Small noncontiguous datatypes - Large noncontiguous datatypes Signed-off-by: Ken Raffenetti <raffenet(a)mcs.anl.gov> diff --git a/test/mpi/pt2pt/mprobe.c b/test/mpi/pt2pt/mprobe.c index b45834e..07edde1 100644 --- a/test/mpi/pt2pt/mprobe.c +++ b/test/mpi/pt2pt/mprobe.c @@ -28,18 +28,22 @@ } \ } while (0) +#define LARGE_DIM 512 +#define LARGE_SZ (LARGE_DIM * LARGE_DIM) + int main(int argc, char **argv) { int errs = 0; int found, completed; int rank, size; - int sendbuf[8], recvbuf[8]; - int count; + int sendbuf[LARGE_SZ], recvbuf[LARGE_SZ]; + int count, i; #ifdef TEST_MPROBE_ROUTINES MPI_Message msg; #endif MPI_Request rreq; MPI_Status s1, s2; + MPI_Datatype vectype; MPI_Init(&argc, &argv); @@ -355,6 +359,162 @@ int main(int argc, char **argv) check(count == 0); } + /* test 8: simple ssend & mprobe+mrecv */ + if (rank == 0) { + sendbuf[0] = 0xdeadbeef; + sendbuf[1] = 0xfeedface; + MPI_Ssend(sendbuf, 2, MPI_INT, 1, 5, MPI_COMM_WORLD); + } + else { + memset(&s1, 0xab, sizeof(MPI_Status)); + memset(&s2, 0xab, sizeof(MPI_Status)); + /* the error field should remain unmodified */ + s1.MPI_ERROR = MPI_ERR_DIMS; + s2.MPI_ERROR = MPI_ERR_TOPOLOGY; + + msg = MPI_MESSAGE_NULL; + MPI_Mprobe(0, 5, MPI_COMM_WORLD, &msg, &s1); + check(s1.MPI_SOURCE == 0); + check(s1.MPI_TAG == 5); + check(s1.MPI_ERROR == MPI_ERR_DIMS); + check(msg != MPI_MESSAGE_NULL); + + count = -1; + MPI_Get_count(&s1, MPI_INT, &count); + check(count == 2); + + recvbuf[0] = 0x01234567; + recvbuf[1] = 0x89abcdef; + MPI_Mrecv(recvbuf, count, MPI_INT, &msg, &s2); + check(recvbuf[0] == 0xdeadbeef); + check(recvbuf[1] == 0xfeedface); + check(s2.MPI_SOURCE == 0); + check(s2.MPI_TAG == 5); + check(s2.MPI_ERROR == MPI_ERR_TOPOLOGY); + check(msg == MPI_MESSAGE_NULL); + } + + /* test 9: mprobe+mrecv LARGE */ + if (rank == 0) { + for (i = 0; i < LARGE_SZ; i++) + sendbuf[i] = i; + MPI_Send(sendbuf, LARGE_SZ, MPI_INT, 1, 5, MPI_COMM_WORLD); + } + else { + memset(&s1, 0xab, sizeof(MPI_Status)); + memset(&s2, 0xab, sizeof(MPI_Status)); + /* the error field should remain unmodified */ + s1.MPI_ERROR = MPI_ERR_DIMS; + s2.MPI_ERROR = MPI_ERR_TOPOLOGY; + + msg = MPI_MESSAGE_NULL; + MPI_Mprobe(0, 5, MPI_COMM_WORLD, &msg, &s1); + check(s1.MPI_SOURCE == 0); + check(s1.MPI_TAG == 5); + check(s1.MPI_ERROR == MPI_ERR_DIMS); + check(msg != MPI_MESSAGE_NULL); + + count = -1; + MPI_Get_count(&s1, MPI_INT, &count); + check(count == LARGE_SZ); + + memset(recvbuf, 0xFF, LARGE_SZ * sizeof(int)); + MPI_Mrecv(recvbuf, count, MPI_INT, &msg, &s2); + for (i = 0; i < LARGE_SZ; i++) + check(recvbuf[i] == i); + check(s2.MPI_SOURCE == 0); + check(s2.MPI_TAG == 5); + check(s2.MPI_ERROR == MPI_ERR_TOPOLOGY); + check(msg == MPI_MESSAGE_NULL); + } + + /* test 10: mprobe+mrecv noncontiguous datatype */ + MPI_Type_vector(2, 1, 4, MPI_INT, &vectype); + MPI_Type_commit(&vectype); + if (rank == 0) { + memset(sendbuf, 0, 8 * sizeof(int)); + sendbuf[0] = 0xdeadbeef; + sendbuf[4] = 0xfeedface; + MPI_Send(sendbuf, 1, vectype, 1, 5, MPI_COMM_WORLD); + } + else { + memset(&s1, 0xab, sizeof(MPI_Status)); + memset(&s2, 0xab, sizeof(MPI_Status)); + /* the error field should remain unmodified */ + s1.MPI_ERROR = MPI_ERR_DIMS; + s2.MPI_ERROR = MPI_ERR_TOPOLOGY; + + msg = MPI_MESSAGE_NULL; + MPI_Mprobe(0, 5, MPI_COMM_WORLD, &msg, &s1); + check(s1.MPI_SOURCE == 0); + check(s1.MPI_TAG == 5); + check(s1.MPI_ERROR == MPI_ERR_DIMS); + check(msg != MPI_MESSAGE_NULL); + + count = -1; + MPI_Get_count(&s1, vectype, &count); + check(count == 1); + + memset(recvbuf, 0, 8 * sizeof(int)); + MPI_Mrecv(recvbuf, 1, vectype, &msg, &s2); + check(recvbuf[0] == 0xdeadbeef); + for (i = 1; i < 4; i++) + check(recvbuf[i] == 0); + check(recvbuf[4] = 0xfeedface); + for (i = 5; i < 8; i++) + check(recvbuf[i] == 0); + check(s2.MPI_SOURCE == 0); + check(s2.MPI_TAG == 5); + check(s2.MPI_ERROR == MPI_ERR_TOPOLOGY); + check(msg == MPI_MESSAGE_NULL); + } + MPI_Type_free(&vectype); + + /* test 11: mprobe+mrecv noncontiguous datatype LARGE */ + MPI_Type_vector(LARGE_DIM, LARGE_DIM - 1, LARGE_DIM, MPI_INT, &vectype); + MPI_Type_commit(&vectype); + if (rank == 0) { + for (i = 0; i < LARGE_SZ; i++) + sendbuf[i] = i; + MPI_Send(sendbuf, 1, vectype, 1, 5, MPI_COMM_WORLD); + } + else { + int idx = 0; + + memset(&s1, 0xab, sizeof(MPI_Status)); + memset(&s2, 0xab, sizeof(MPI_Status)); + /* the error field should remain unmodified */ + s1.MPI_ERROR = MPI_ERR_DIMS; + s2.MPI_ERROR = MPI_ERR_TOPOLOGY; + + msg = MPI_MESSAGE_NULL; + MPI_Mprobe(0, 5, MPI_COMM_WORLD, &msg, &s1); + check(s1.MPI_SOURCE == 0); + check(s1.MPI_TAG == 5); + check(s1.MPI_ERROR == MPI_ERR_DIMS); + check(msg != MPI_MESSAGE_NULL); + + count = -1; + MPI_Get_count(&s1, vectype, &count); + check(count == 1); + + memset(recvbuf, 0, LARGE_SZ * sizeof(int)); + MPI_Mrecv(recvbuf, 1, vectype, &msg, &s2); + for (i = 0; i < LARGE_DIM; i++) { + int j; + for (j = 0; j < LARGE_DIM - 1; j++) { + check(recvbuf[idx] == idx); + ++idx; + } + check(recvbuf[idx++] == 0); + } + check(s2.MPI_SOURCE == 0); + check(s2.MPI_TAG == 5); + check(s2.MPI_ERROR == MPI_ERR_TOPOLOGY); + check(msg == MPI_MESSAGE_NULL); + } + MPI_Type_free(&vectype); + /* TODO MPI_ANY_SOURCE and MPI_ANY_TAG should be tested as well */ /* TODO a full range of message sizes should be tested too */ /* TODO threaded tests are also needed, but they should go in a separate http://git.mpich.org/mpich.git/commitdiff/5c328ccb3d1752224a30740e5e1b5c3ed… commit 5c328ccb3d1752224a30740e5e1b5c3ed4e129f8 Author: Antonio Pena Monferrer <apenya(a)mcs.anl.gov> Date: Mon Aug 4 09:42:57 2014 -0500 portals4: Avoid setting error on cancelling a request Sometimes the ch3 layer asks to cancel a request which is not in the netmod queues any more. If portals reports anything but PTL_OK or PTL_IN_USE, just return and don't set an error. We can consider the request no longer active. Signed-off-by: Ken Raffenetti <raffenet(a)mcs.anl.gov> diff --git a/src/mpid/ch3/channels/nemesis/netmod/portals4/ptl_recv.c b/src/mpid/ch3/channels/nemesis/netmod/portals4/ptl_recv.c index 2db6691..5e323f6 100644 --- a/src/mpid/ch3/channels/nemesis/netmod/portals4/ptl_recv.c +++ b/src/mpid/ch3/channels/nemesis/netmod/portals4/ptl_recv.c @@ -533,8 +533,10 @@ static int cancel_recv(MPID_Request *rreq, int *cancelled) ptl_err = PtlMEUnlink(REQ_PTL(rreq)->me); if (ptl_err == PTL_OK) *cancelled = TRUE; - else if (ptl_err != PTL_IN_USE) - mpi_errno = MPI_ERR_INTERN; + /* FIXME: if we properly invalidate matching list entry handles, we should be + able to ensure an unlink operation results in either PTL_OK or PTL_IN_USE. + Anything else would be an error. For now, though, we assume anything but PTL_OK + is uncancelable and return. */ } fn_exit: @@ -560,12 +562,12 @@ int MPID_nem_ptl_anysource_matched(MPID_Request *rreq) mpi_errno = cancel_recv(rreq, &cancelled); /* FIXME: This function is does not return an error because the queue functions (where the posted_recv hooks are called) return no error - code. */ + code. See also comment on cancel_recv. */ MPIU_Assertp(mpi_errno == MPI_SUCCESS); fn_exit: MPIDI_FUNC_EXIT(MPID_STATE_MPID_NEM_PTL_ANYSOURCE_MATCHED); - return !cancelled; + return MPI_SUCCESS; fn_fail: goto fn_exit; } http://git.mpich.org/mpich.git/commitdiff/dfce63a02289fbaebd776c76d2c90a7a2… commit dfce63a02289fbaebd776c76d2c90a7a21e96450 Author: Antonio Pena Monferrer <apenya(a)mcs.anl.gov> Date: Mon Aug 4 09:29:18 2014 -0500 Fixed accessing to NULL struct in portals receive This happened in anysource with debug logging enabled. Signed-off-by: Ken Raffenetti <raffenet(a)mcs.anl.gov> diff --git a/src/mpid/ch3/channels/nemesis/netmod/portals4/ptl_recv.c b/src/mpid/ch3/channels/nemesis/netmod/portals4/ptl_recv.c index 57e5dc3..2db6691 100644 --- a/src/mpid/ch3/channels/nemesis/netmod/portals4/ptl_recv.c +++ b/src/mpid/ch3/channels/nemesis/netmod/portals4/ptl_recv.c @@ -477,7 +477,7 @@ int MPID_nem_ptl_recv_posted(MPIDI_VC_t *vc, MPID_Request *rreq) ret = PtlMEAppend(MPIDI_nem_ptl_ni, MPIDI_nem_ptl_pt, &me, PTL_PRIORITY_LIST, rreq, &REQ_PTL(rreq)->me); MPIU_ERR_CHKANDJUMP1(ret, mpi_errno, MPI_ERR_OTHER, "**ptlmeappend", "**ptlmeappend %s", MPID_nem_ptl_strerror(ret)); - DBG_MSG_MEAPPEND("REG", vc->pg_rank, me, rreq); + DBG_MSG_MEAPPEND("REG", vc ? vc->pg_rank : MPI_ANY_SOURCE, me, rreq); MPIU_DBG_MSG_P(CH3_CHANNEL, VERBOSE, " buf=%p", me.start); MPIU_DBG_MSG_D(CH3_CHANNEL, VERBOSE, "MPIDI_nem_ptl_pt = %d", MPIDI_nem_ptl_pt); http://git.mpich.org/mpich.git/commitdiff/88f05f80a9e5b4c41a5a670484503fc41… commit 88f05f80a9e5b4c41a5a670484503fc413d7868f Author: Antonio Pena Monferrer <apenya(a)mcs.anl.gov> Date: Mon Aug 4 09:21:39 2014 -0500 Fixed Portals matching masking The out-of-interest bits must be zeroed-out to avoid them colliding with their neighbor bits. This is relevant In cases of special values, i.e., negative values such as MPI_*_ANY. Signed-off-by: Ken Raffenetti <raffenet(a)mcs.anl.gov> diff --git a/src/mpid/ch3/channels/nemesis/netmod/portals4/ptl_impl.h b/src/mpid/ch3/channels/nemesis/netmod/portals4/ptl_impl.h index 43c1c1d..9a2d3cf 100644 --- a/src/mpid/ch3/channels/nemesis/netmod/portals4/ptl_impl.h +++ b/src/mpid/ch3/channels/nemesis/netmod/portals4/ptl_impl.h @@ -129,9 +129,9 @@ typedef struct { #define NPTL_MATCH_RANK_MASK (((ptl_match_bits_t)(1) << 16) - 1) #define NPTL_MATCH_CTX_MASK ((((ptl_match_bits_t)(1) << 16) - 1) << NPTL_MATCH_CTX_OFFSET) #define NPTL_MATCH_TAG_MASK ((((ptl_match_bits_t)(1) << 32) - 1) << NPTL_MATCH_TAG_OFFSET) -#define NPTL_MATCH(tag_, ctx_, rank_) (((ptl_match_bits_t)(tag_) << NPTL_MATCH_TAG_OFFSET) | \ - ((ptl_match_bits_t)(ctx_) << NPTL_MATCH_CTX_OFFSET) | \ - ((ptl_match_bits_t)(rank_))) +#define NPTL_MATCH(tag_, ctx_, rank_) ((((ptl_match_bits_t)(tag_) << NPTL_MATCH_TAG_OFFSET) & NPTL_MATCH_TAG_MASK) | \ + (((ptl_match_bits_t)(ctx_) << NPTL_MATCH_CTX_OFFSET) & NPTL_MATCH_CTX_MASK) | \ + ((ptl_match_bits_t)(rank_) & NPTL_MATCH_RANK_MASK)) #define NPTL_MATCH_IGNORE NPTL_MATCH_RANK_MASK #define NPTL_MATCH_IGNORE_ANY_TAG (NPTL_MATCH_IGNORE | NPTL_MATCH_TAG_MASK) ----------------------------------------------------------------------- Summary of changes: .../channels/nemesis/netmod/portals4/Makefile.mk | 3 +- .../channels/nemesis/netmod/portals4/ptl_impl.h | 13 ++- .../channels/nemesis/netmod/portals4/ptl_init.c | 11 ++- .../ch3/channels/nemesis/netmod/portals4/ptl_lmt.c | 70 +++++++++ .../channels/nemesis/netmod/portals4/ptl_probe.c | 129 +++++++++++++++- .../channels/nemesis/netmod/portals4/ptl_recv.c | 126 ++++++++++++++- test/mpi/pt2pt/mprobe.c | 164 +++++++++++++++++++- 7 files changed, 498 insertions(+), 18 deletions(-) create mode 100644 src/mpid/ch3/channels/nemesis/netmod/portals4/ptl_lmt.c hooks/post-receive -- MPICH primary repository

1 0