October 2015 - devel - mailman3.cels.anl.gov

Re: [mpich-devel] MPI_Dist_graph_create self-edge
by Daniel Ibanez 24 Oct '15

24 Oct '15

Rajeev, Thank you ! This was indeed a bug in my code that called Dist_graph_create, good thing I copied the resulting values exactly into the demo code. I should actually be specifying n=1 at all times for my application. Things are working as I expect now, sorry if I wasted someone's time. On Sat, Oct 24, 2015 at 9:32 PM, Thakur, Rajeev <thakur(a)mcs.anl.gov> wrote: > On rank 0, if you are passing n=2, shouldn’t the sources and degrees > arrays be of size 2? > > Rajeev > > > On Oct 24, 2015, at 6:37 PM, Daniel Ibanez <dan.a.ibanez(a)gmail.com> > wrote: > > > > Pavan, > > > > Here is a short program that reproduces the issue: > > > > > > > > #include <mpi.h> > > #include <assert.h> > > > > int main(int argc, char** argv) > > { > > MPI_Init(&argc, &argv); > > int size; > > MPI_Comm_size(MPI_COMM_WORLD, &size); > > assert(size == 2); > > int rank; > > MPI_Comm_rank(MPI_COMM_WORLD, &rank); > > MPI_Comm newcomm; > > if (rank == 0) { > > int sources[1] = {0}; > > int degrees[1] = {2}; > > int destinations[2] = {1,0}; > > int weights[2] = {2,1}; > > MPI_Dist_graph_create(MPI_COMM_WORLD, 2, sources, degrees, > destinations, > > weights, MPI_INFO_NULL, 0, &newcomm); > > } else { > > int sources[1] = {1}; > > int degrees[1] = {1}; > > int destinations[1] = {0}; > > int weights[1] = {2}; > > MPI_Dist_graph_create(MPI_COMM_WORLD, 1, sources, degrees, > destinations, > > weights, MPI_INFO_NULL, 0, &newcomm); > > } > > MPI_Comm_free(&newcomm); > > MPI_Finalize(); > > } > > > > > > > > > > Thanks, > > > > Dan > > > > On Sat, Oct 24, 2015 at 7:21 PM, Balaji, Pavan <balaji(a)anl.gov> wrote: > > Daniel, > > > > Do you have a test program that shows these errors? > > > > The algorithms are a first cut, for now, but we are hoping to optimize > them in 2016 (together with one of our vendor partners, who will be > contributing code for it). > > > > Thanks, > > > > -- Pavan > > > > From: Daniel Ibanez <dan.a.ibanez(a)gmail.com> > > Reply-To: "devel(a)mpich.org" <devel(a)mpich.org> > > Date: Saturday, October 24, 2015 at 5:05 PM > > To: "devel(a)mpich.org" <devel(a)mpich.org> > > Subject: [mpich-devel] MPI_Dist_graph_create self-edge > > > > Hello, > > > > Using MPICH 3.1.4, I'm getting internal assertions of this kind > > of when using MPI_Dist_graph_create : > > > > Assertion failed in file src/mpi/topo/dist_gr_create.c at line 223: > s_rank >= 0 > > Assertion failed in file src/mpi/topo/dist_gr_create.c at line 195: > sources[i] < comm_size > > > > I've checked that ranks passed in are in the proper range, > > rather I think the issue is caused by requesting an edge > > from rank 0 to itself. > > (Another hint is that its non-deterministic which assertion > > I get; it depends on the order in which the ranks get scheduled > > by the OS, so there is a bit of a bug in the implementation, > > or at least no check for self-edges). > > > > Does the MPI standard allow for self-edges in these graphs ? > > > > Thank you, > > > > P.S. - I'll throw in a bigger question while I'm at it: are > > MPI_Dist_graph_create and MPI_Neighbor_alltoallv > > implemented with optimally scalable algorithms ? > > I'm betting my scalability on them being roughly > > O(log(P)) where P is communicator size, assuming > > neighborhood sizes and message sizes are constant. > > > > _______________________________________________ > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/devel > > > > _______________________________________________ > > To manage subscription options or unsubscribe: > > https://lists.mpich.org/mailman/listinfo/devel > > _______________________________________________ > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/devel >

1 0

Re: [mpich-devel] MPI_Dist_graph_create self-edge
by Thakur, Rajeev 24 Oct '15

24 Oct '15

On rank 0, if you are passing n=2, shouldn’t the sources and degrees arrays be of size 2? Rajeev > On Oct 24, 2015, at 6:37 PM, Daniel Ibanez <dan.a.ibanez(a)gmail.com> wrote: > > Pavan, > > Here is a short program that reproduces the issue: > > > > #include <mpi.h> > #include <assert.h> > > int main(int argc, char** argv) > { > MPI_Init(&argc, &argv); > int size; > MPI_Comm_size(MPI_COMM_WORLD, &size); > assert(size == 2); > int rank; > MPI_Comm_rank(MPI_COMM_WORLD, &rank); > MPI_Comm newcomm; > if (rank == 0) { > int sources[1] = {0}; > int degrees[1] = {2}; > int destinations[2] = {1,0}; > int weights[2] = {2,1}; > MPI_Dist_graph_create(MPI_COMM_WORLD, 2, sources, degrees, destinations, > weights, MPI_INFO_NULL, 0, &newcomm); > } else { > int sources[1] = {1}; > int degrees[1] = {1}; > int destinations[1] = {0}; > int weights[1] = {2}; > MPI_Dist_graph_create(MPI_COMM_WORLD, 1, sources, degrees, destinations, > weights, MPI_INFO_NULL, 0, &newcomm); > } > MPI_Comm_free(&newcomm); > MPI_Finalize(); > } > > > > > Thanks, > > Dan > > On Sat, Oct 24, 2015 at 7:21 PM, Balaji, Pavan <balaji(a)anl.gov> wrote: > Daniel, > > Do you have a test program that shows these errors? > > The algorithms are a first cut, for now, but we are hoping to optimize them in 2016 (together with one of our vendor partners, who will be contributing code for it). > > Thanks, > > -- Pavan > > From: Daniel Ibanez <dan.a.ibanez(a)gmail.com> > Reply-To: "devel(a)mpich.org" <devel(a)mpich.org> > Date: Saturday, October 24, 2015 at 5:05 PM > To: "devel(a)mpich.org" <devel(a)mpich.org> > Subject: [mpich-devel] MPI_Dist_graph_create self-edge > > Hello, > > Using MPICH 3.1.4, I'm getting internal assertions of this kind > of when using MPI_Dist_graph_create : > > Assertion failed in file src/mpi/topo/dist_gr_create.c at line 223: s_rank >= 0 > Assertion failed in file src/mpi/topo/dist_gr_create.c at line 195: sources[i] < comm_size > > I've checked that ranks passed in are in the proper range, > rather I think the issue is caused by requesting an edge > from rank 0 to itself. > (Another hint is that its non-deterministic which assertion > I get; it depends on the order in which the ranks get scheduled > by the OS, so there is a bit of a bug in the implementation, > or at least no check for self-edges). > > Does the MPI standard allow for self-edges in these graphs ? > > Thank you, > > P.S. - I'll throw in a bigger question while I'm at it: are > MPI_Dist_graph_create and MPI_Neighbor_alltoallv > implemented with optimally scalable algorithms ? > I'm betting my scalability on them being roughly > O(log(P)) where P is communicator size, assuming > neighborhood sizes and message sizes are constant. > > _______________________________________________ > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/devel > > _______________________________________________ > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/devel

1 0

Re: [mpich-devel] MPI_Dist_graph_create self-edge
by Daniel Ibanez 24 Oct '15

24 Oct '15

Pavan, Here is a short program that reproduces the issue: #include <mpi.h> #include <assert.h> int main(int argc, char** argv) { MPI_Init(&argc, &argv); int size; MPI_Comm_size(MPI_COMM_WORLD, &size); assert(size == 2); int rank; MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm newcomm; if (rank == 0) { int sources[1] = {0}; int degrees[1] = {2}; int destinations[2] = {1,0}; int weights[2] = {2,1}; MPI_Dist_graph_create(MPI_COMM_WORLD, 2, sources, degrees, destinations, weights, MPI_INFO_NULL, 0, &newcomm); } else { int sources[1] = {1}; int degrees[1] = {1}; int destinations[1] = {0}; int weights[1] = {2}; MPI_Dist_graph_create(MPI_COMM_WORLD, 1, sources, degrees, destinations, weights, MPI_INFO_NULL, 0, &newcomm); } MPI_Comm_free(&newcomm); MPI_Finalize(); } Thanks, Dan On Sat, Oct 24, 2015 at 7:21 PM, Balaji, Pavan <balaji(a)anl.gov> wrote: > Daniel, > > Do you have a test program that shows these errors? > > The algorithms are a first cut, for now, but we are hoping to optimize > them in 2016 (together with one of our vendor partners, who will be > contributing code for it). > > Thanks, > > -- Pavan > > From: Daniel Ibanez <dan.a.ibanez(a)gmail.com> > Reply-To: "devel(a)mpich.org" <devel(a)mpich.org> > Date: Saturday, October 24, 2015 at 5:05 PM > To: "devel(a)mpich.org" <devel(a)mpich.org> > Subject: [mpich-devel] MPI_Dist_graph_create self-edge > > Hello, > > Using MPICH 3.1.4, I'm getting internal assertions of this kind > of when using MPI_Dist_graph_create : > > Assertion failed in file src/mpi/topo/dist_gr_create.c at line 223: s_rank > >= 0 > Assertion failed in file src/mpi/topo/dist_gr_create.c at line 195: > sources[i] < comm_size > > I've checked that ranks passed in are in the proper range, > rather I think the issue is caused by requesting an edge > from rank 0 to itself. > (Another hint is that its non-deterministic which assertion > I get; it depends on the order in which the ranks get scheduled > by the OS, so there is a bit of a bug in the implementation, > or at least no check for self-edges). > > Does the MPI standard allow for self-edges in these graphs ? > > Thank you, > > P.S. - I'll throw in a bigger question while I'm at it: are > MPI_Dist_graph_create and MPI_Neighbor_alltoallv > implemented with optimally scalable algorithms ? > I'm betting my scalability on them being roughly > O(log(P)) where P is communicator size, assuming > neighborhood sizes and message sizes are constant. > > _______________________________________________ > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/devel >

1 0

Re: [mpich-devel] MPI_Dist_graph_create self-edge
by Balaji, Pavan 24 Oct '15

24 Oct '15

Daniel, Do you have a test program that shows these errors? The algorithms are a first cut, for now, but we are hoping to optimize them in 2016 (together with one of our vendor partners, who will be contributing code for it). Thanks, -- Pavan From: Daniel Ibanez <dan.a.ibanez(a)gmail.com<mailto:[email protected]>> Reply-To: "devel(a)mpich.org<mailto:[email protected]>" <devel(a)mpich.org<mailto:[email protected]>> Date: Saturday, October 24, 2015 at 5:05 PM To: "devel(a)mpich.org<mailto:[email protected]>" <devel(a)mpich.org<mailto:[email protected]>> Subject: [mpich-devel] MPI_Dist_graph_create self-edge Hello, Using MPICH 3.1.4, I'm getting internal assertions of this kind of when using MPI_Dist_graph_create : Assertion failed in file src/mpi/topo/dist_gr_create.c at line 223: s_rank >= 0 Assertion failed in file src/mpi/topo/dist_gr_create.c at line 195: sources[i] < comm_size I've checked that ranks passed in are in the proper range, rather I think the issue is caused by requesting an edge from rank 0 to itself. (Another hint is that its non-deterministic which assertion I get; it depends on the order in which the ranks get scheduled by the OS, so there is a bit of a bug in the implementation, or at least no check for self-edges). Does the MPI standard allow for self-edges in these graphs ? Thank you, P.S. - I'll throw in a bigger question while I'm at it: are MPI_Dist_graph_create and MPI_Neighbor_alltoallv implemented with optimally scalable algorithms ? I'm betting my scalability on them being roughly O(log(P)) where P is communicator size, assuming neighborhood sizes and message sizes are constant.

1 0

MPI_Dist_graph_create self-edge
by Daniel Ibanez 24 Oct '15

24 Oct '15

Hello, Using MPICH 3.1.4, I'm getting internal assertions of this kind of when using MPI_Dist_graph_create : Assertion failed in file src/mpi/topo/dist_gr_create.c at line 223: s_rank >= 0 Assertion failed in file src/mpi/topo/dist_gr_create.c at line 195: sources[i] < comm_size I've checked that ranks passed in are in the proper range, rather I think the issue is caused by requesting an edge from rank 0 to itself. (Another hint is that its non-deterministic which assertion I get; it depends on the order in which the ranks get scheduled by the OS, so there is a bit of a bug in the implementation, or at least no check for self-edges). Does the MPI standard allow for self-edges in these graphs ? Thank you, P.S. - I'll throw in a bigger question while I'm at it: are MPI_Dist_graph_create and MPI_Neighbor_alltoallv implemented with optimally scalable algorithms ? I'm betting my scalability on them being roughly O(log(P)) where P is communicator size, assuming neighborhood sizes and message sizes are constant.

1 0

Meet us at SC '15
by Seo, Sangmin 22 Oct '15

22 Oct '15

The MPICH team will have a series of events at SC ’15 (http://sc15.supercomputing.org) including tutorials, talks, posters, BoFs, and demos. Come and meet us at the following events: Tutorials - Nov. 15 (Sun) / 08:30am – 05:00pm / 12A / Parallel I/O In Practice - Nov. 16 (Mon) / 08:30am – 05:00pm / 18B / Advanced MPI Programming Keynote speech - Nov. 16 (Mon) / Pavan Balaji @ ExaMPI Papers - Nov. 17 (Tue) / 04:00pm – 04:30pm / 18CD / Improving Concurrency and Asynchrony in Multithreaded MPI Applications using Software Offloading - Nov. 18 (Wed) / 11:30am – 12:00pm / 19AB / Fault Tolerant MapReduce-MPI for HPC Clusters - Nov. 19 (Thu) / 02:00pm – 02:30pm / 19AB / VOCL-FT: Introducing Techniques for Efficient Soft Error Coprocessor Recovery Posters - Nov. 17 (Tue) / 05:15pm – 07:00pm / Level 4 – Concourse / Argo: An Exascale Operating System and Runtime BoFs - Nov. 17 (Tue) / 05:30pm – 07:00pm / 17AB / MPICH: A High-Performance Open-Source MPI Implementation - Nov. 18 (Wed) / 03:30pm – 05:00pm / 15 / The Message Passing Interface: MPI 3.1 Released, Next Stop MPI 4.0 - Nov. 18 (Wed) / 05:30pm – 07:00pm / 19AB / Towards Standardized, Portable and Lightweight User-Level Threads and Tasks Demos/Presentations - DOE Booth: Argo demo/presentation (TBD) For links and updated information for all events, please visit http://www.mpich.org/2015/10/15/meet-us-at-sc-15/ We are looking forward to seeing you at SC '15. Regards, The MPICH Team

1 0

Re: [mpich-devel] MPI_Alltoall fails when called with newly created communicators (Giuseppe Congiu)
by Paul Coffman 21 Oct '15

21 Oct '15

Giuseppe - So if you're creating the separate communicators to address issues with collective IO aggregation memory and performance, actually in MPICH 3.2rc1 there already exists a one-sided aggregation method that does away with the alltoall calls with significant performance and memory enhancements, although currently this is only implemented for GPFS. See src/mpi/romio/adio/ad_gpfs/ad_gpfs_tuning.c and search for 'one-sided' essentially it is enabled with the env vars GPFSMPIO_WRITE_AGGMETHOD and GPFSMPIO_READ_AGGMETHOD with some others for additional tweaking. And actually if you are using NFS there are a couple of tricks to get it to work with that also let me know and I can help you. -----Original Message----- From: devel-request(a)mpich.org [mailto:[email protected]] Sent: Wednesday, October 21, 2015 2:07 AM To: devel(a)mpich.org Subject: devel Digest, Vol 32, Issue 2 Send devel mailing list submissions to devel(a)mpich.org To subscribe or unsubscribe via the World Wide Web, visit https://lists.mpich.org/mailman/listinfo/devel or, via email, send a message with subject or body 'help' to devel-request(a)mpich.org You can reach the person managing the list at devel-owner(a)mpich.org When replying, please edit your Subject line so it is more specific than "Re: Contents of devel digest..." Today's Topics: 1. Re: Argonne Spam Quarantine Notification (Bland, Wesley) 2. Re: Argonne Spam Quarantine Notification (Jeff Hammond) 3. Re: Argonne Spam Quarantine Notification (Kenneth Raffenetti) 4. [PATCH] MPI_Aint_* tests: use MPI_INTEGER datatype in Fortran (Jeff Squyres) 5. Re: [PATCH] MPI_Aint_* tests: use MPI_INTEGER datatype in Fortran (Kenneth Raffenetti) 6. MPI_Alltoall fails when called with newly created communicators (Giuseppe Congiu) ---------------------------------------------------------------------- Message: 1 Date: Thu, 1 Oct 2015 11:41:39 +0000 From: "Bland, Wesley" <wesley.bland(a)intel.com> To: "devel(a)mpich.org" <devel(a)mpich.org> Subject: Re: [mpich-devel] Argonne Spam Quarantine Notification Message-ID: <AF5B0AAE-9AEE-4F8A-BABB-9FA86EFE6D30(a)intel.com> Content-Type: text/plain; charset="us-ascii" Looks like it didn't work. On Oct 1, 2015, at 6:00 AM, Argonne Anti Spam Quarantine <spam-quarantine(a)anl.gov<mailto:[email protected]>> wrote: IronPort Spam Quarantine Notification If a spam message was missed, or if a message was incorrectly flagged as spam, you can add the sender to a personal safelist or blocklist via the quarantine. Senders on the safelist will not be flagged as spam, and senders on the blocklist will be automatically quarantined. For more information, visit: https://wiki.inside.anl.gov/inside/Ironport_Anti-Spam_Appliances/Documentati on_for_Users The message(s) below have been quarantined by the Argonne National Laboratory spam filtering system. There are 2 new messages in your Email Quarantine since you received your last Spam Quarantine Notification. If the messages below are spam, you do not need to take any action. Messages will be automatically deleted from the quarantine after 28 day(s). If any of the messages below are not spam, click the 'Release' link to have them sent to your Inbox. Messages that have been correctly flagged as spam do not need to be reported. Quarantined Email From Subject Date Release<http://mailgateway.anl.gov:8080/Message?action=Release&mid=95404728& h=99ea5934cf8758aecc0fe59fc5e4af52&email=devel%40mpich.org> "Lawyer Central" <homo@joka-interna... *****SPAM*****We Provide you the best Lawyers in your area<http://mailgateway.anl.gov:8080/Message?action=Detail&mid=95404728&h=99 ea5934cf8758aecc0fe59fc5e4af52&email=devel%40mpich.org> 30 Sep 2015 Release<http://mailgateway.anl.gov:8080/Message?action=Release&mid=95404711& h=004b11152d9d7c8f42a99e091808c1af&email=devel%40mpich.org> "Lawyer Connection" <disobligingnes... *****SPAM*****We have the right lawyer to represent you<http://mailgateway.anl.gov:8080/Message?action=Detail&mid=95404711&h=004 b11152d9d7c8f42a99e091808c1af&email=devel%40mpich.org> 30 Sep 2015 View All Quarantined Messages(4)<http://mailgateway.anl.gov:8080/Search?h=2ffb2bd0571f729efb89bf1 1a9fc382c&email=devel%40mpich.org> Note: This message has been sent by a notification only system. Please do not reply If the above links do not work, please copy and paste the following URL into a Web browser: http://mailgateway.anl.gov:8080/Search?h=2ffb2bd0571f729efb89bf11a9fc382c&em ail=devel%40mpich.org If you have any questions regarding the Laboratory spam filtering product, please contact the CIS helpdesk at help(a)anl.gov<mailto:[email protected]> or x-9999 option 2. _______________________________________________ To manage subscription options or unsubscribe: https://lists.mpich.org/mailman/listinfo/devel ------------------------------ Message: 2 Date: Thu, 1 Oct 2015 04:53:20 -0700 From: Jeff Hammond <jeff.science(a)gmail.com> To: "devel(a)mpich.org" <devel(a)mpich.org> Subject: Re: [mpich-devel] Argonne Spam Quarantine Notification Message-ID: <CAGKz=u++GoNvu4DD+eGW_6XYpyaUEy0JkL+sqV0uPV=n+EuhzA(a)mail.gmail.com> Content-Type: text/plain; charset="utf-8" If all else fails, individuals can apply the time-tested solution of email filters... Jeff On Thursday, October 1, 2015, Bland, Wesley <wesley.bland(a)intel.com> wrote: > Looks like it didn't work. > > On Oct 1, 2015, at 6:00 AM, Argonne Anti Spam Quarantine < > spam-quarantine(a)anl.gov <javascript:;><mailto:[email protected] > <javascript:;>>> wrote: > > IronPort Spam Quarantine Notification > > > If a spam message was missed, or if a message was incorrectly flagged > as spam, you can add the sender to a personal safelist or blocklist > via the quarantine. Senders on the safelist will not be flagged as > spam, and senders on the blocklist will be automatically quarantined. > For more information, visit: > > > https://wiki.inside.anl.gov/inside/Ironport_Anti-Spam_Appliances/Docum > entation_for_Users > > The message(s) below have been quarantined by the Argonne National > Laboratory spam filtering system. > > There are 2 new messages in your Email Quarantine since you received > your last Spam Quarantine Notification. If the messages below are > spam, you do not need to take any action. Messages will be > automatically deleted from the quarantine after 28 day(s). > > If any of the messages below are not spam, click the 'Release' link to > have them sent to your Inbox. > > Messages that have been correctly flagged as spam do not need to be > reported. > > Quarantined Email > From Subject Date > Release< > http://mailgateway.anl.gov:8080/Message?action=Release&mid=95404728&h= > 99ea5934cf8758aecc0fe59fc5e4af52&email=devel%40mpich.org> > "Lawyer Central" <homo@joka-interna... *****SPAM*****We Provide you > the best Lawyers in your area< > http://mailgateway.anl.gov:8080/Message?action=Detail&mid=95404728&h=99ea593 4cf8758aecc0fe59fc5e4af52&email=devel%40mpich.org> > 30 Sep 2015 > Release< > http://mailgateway.anl.gov:8080/Message?action=Release&mid=95404711&h= > 004b11152d9d7c8f42a99e091808c1af&email=devel%40mpich.org> > "Lawyer Connection" <disobligingnes... *****SPAM*****We have the > right lawyer to represent you< > http://mailgateway.anl.gov:8080/Message?action=Detail&mid=95404711&h=0 > 04b11152d9d7c8f42a99e091808c1af&email=devel%40mpich.org> > 30 Sep 2015 > View All Quarantined Messages(4)< > http://mailgateway.anl.gov:8080/Search?h=2ffb2bd0571f729efb89bf11a9fc3 > 82c&email=devel%40mpich.org > > > > Note: This message has been sent by a notification only system. Please > do not reply > > If the above links do not work, please copy and paste the following > URL into a Web browser: > > http://mailgateway.anl.gov:8080/Search?h=2ffb2bd0571f729efb89bf11a9fc3 > 82c&email=devel%40mpich.org > > If you have any questions regarding the Laboratory spam filtering > product, please contact the CIS helpdesk at help(a)anl.gov <javascript:;><mailto: > help(a)anl.gov <javascript:;>> or x-9999 option 2. > _______________________________________________ > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/devel > _______________________________________________ > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/devel > -- Jeff Hammond jeff.science(a)gmail.com http://jeffhammond.github.io/

1 0

Re: [mpich-devel] MPI_Alltoall fails when called with newly created communicators
by Rob Latham 21 Oct '15

21 Oct '15

You've provided a lot of information. thanks. Nothing looks obviously wrong. You've set up 4 communicators, each to handle a separate portion of the collective I/O request, but then when it comes time to actually do collective i/o you get a crash. It sounds like maybe a comm_world snuck in somewhere? Or there is a collective routine mis-matched with another process somewhere? Here's how I'd start debugging this: - put in a few MPI_Barriers (over fd->comm) in selected places, just for debugging. These barriers will help (a bit) to keep things temporally in step, and if there is a mis-matched collecitve you'll find a hanging process. - When it comes time to start two-phase I/O, set up a token so that only one communicator at a time is operating. Again, not something you want long-term but you'll find out which communicator is causing headaches. - then I'd stare at my screen for a while until I came up with a next thing to try... ==rob On 10/21/2015 02:06 AM, Giuseppe Congiu wrote: > Hello everyone, > > I am currently working on a partitioned collective I/O implementation > for a EU project. The idea is to partition the accessed file range into > disjoint access regions (i.e., regions that do not overlap with each > other) and assign processes from these regions to independent > communicators created by splitting the original communicator with > MPI_Comm_split. Very much like it is done in the ParColl paper or in the > Memory Conscious Collective I/O paper. > > The code is pretty simple. It just uses the access pattern information > to create the new communicators, assign aggregators to each of them and > then compute file domains and data dependencies for every process using > default functions from ROMIO (i.e. ADIOI_Calc_file_domains, > ADIOI_Calc_my_req, ADIOI_Calc_others_req). > > I am currently testing my implementation using coll_perf with 512 > processes. Every process writes 64MB of data for a total of 32GBs (no > reads are performed). The ranks belonging to non-overlapping regions (of > file areas) are printed out by the process with rank 0 in the global > communicator: > > [romio/adio/common/ad_aggregate.c:0] file_area_count = 8 > [romio/adio/common/ad_aggregate.c:0] file_area_ranklist[0] = 0 1 2 > 3 ... 63 > [romio/adio/common/ad_aggregate.c:0] file_area_ranklist[1] = 64 65 66 > 67 ... 127 > [romio/adio/common/ad_aggregate.c:0] file_area_ranklist[2] = 128 129 130 > 131 ... 191 > [romio/adio/common/ad_aggregate.c:0] file_area_ranklist[3] = 192 193 194 > 195 ... 255 > [romio/adio/common/ad_aggregate.c:0] file_area_ranklist[4] = 256 257 258 > 259 ... 319 > [romio/adio/common/ad_aggregate.c:0] file_area_ranklist[5] = 320 321 322 > 323 ... 383 > [romio/adio/common/ad_aggregate.c:0] file_area_ranklist[6] = 384 385 386 > 387 ... 447 > [romio/adio/common/ad_aggregate.c:0] file_area_ranklist[7] = 448 449 450 > 451 ... 511 > > > Afterwards, every aggregator in each communication group prints its rank > and aggregator number in ranklist: > > [romio/adio/common/ad_aggregate.c:0] my_cb_nodes_index = 0 > [romio/adio/common/ad_aggregate.c:0] fd->hints->ranklist[0] = 0 > [romio/adio/common/ad_aggregate.c:0] my_cb_nodes_index = 0 > [romio/adio/common/ad_aggregate.c:0] fd->hints->ranklist[0] = 0 > [romio/adio/common/ad_aggregate.c:0] my_cb_nodes_index = 0 > [romio/adio/common/ad_aggregate.c:0] fd->hints->ranklist[0] = 0 > [romio/adio/common/ad_aggregate.c:0] my_cb_nodes_index = 0 > [romio/adio/common/ad_aggregate.c:0] fd->hints->ranklist[0] = 0 > > > In this particular case I have set fd->hints->cb_nodes to 4. This means > that there are 4 aggregators but 8 non-overlapping regions (file areas). > Thus, I create only 4 communicators and assign to each of them 2 file > areas. In the previous printed messages the only aggregator for each > communicator is printing its information. > > Then every process prints its group name, the number of process in the > group, its rank in the old and new communicator: > > group-0:128:0:0 > group-0:128:1:1 > group-0:128:2:2 > group-0:128:3:3 > --- > group-0:128:127:127 > group-1:128:128:0 > group-1:128:129:1 > group-1:128:130:2 > group-1:128:131:3 > --- > group-1:128:255:127 > group-2:128:256:0 > group-2:128:257:1 > group-2:128:258:2 > group-2:128:259:3 > --- > group-2:128:383:127 > group-3:128:384:0 > group-3:128:385:1 > group-3:128:386:2 > group-3:128:387:3 > --- > group-3:128:511:127 > > > So far it looks like the communicators are created properly. After > calling the ADIOI_Calc_* functions every aggregator prints MIN and MAX > offsets for every independent access range: > > [romio/adio/common/ad_write_coll.c:0] st_loc = 17179869184, end_loc = > 25769803775 > [romio/adio/common/ad_write_coll.c:0] st_loc = 8589934592, end_loc = > 17179869183 > [romio/adio/common/ad_write_coll.c:0] st_loc = 25769803776, end_loc = > 34359738367 > [romio/adio/common/ad_write_coll.c:0] st_loc = 0, end_loc = 8589934591 > > > Even though the new communicators seems to be created correctly, > something bad happens when two phase I/O starts and the first > MPI_Alltoall() is called to exchange access information among processes. > MPI_Alltoall is called using the new communicators separately and every > buffer passed to it has the new nprocs size for the corresponding > communicator. Nevertheless, for some reason I am getting the following > error message: > > Fatal error in PMPI_Alltoall: Other MPI error, error stack: > PMPI_Alltoall(888)......: MPI_Alltoall(sbuf=0x2551e98, scount=1, > MPI_INT, rbuf=0x2551be8, rcount=1, MPI_INT, comm=0x84000001) failed > MPIR_Alltoall_impl(760).: > MPIR_Alltoall(725)......: > MPIR_Alltoall_intra(283): > > repeated many more times. > > This is actually strange because if I look at the file after coll_perf > crashes there is some data in it (~200MB). Like if the problem was > caused by only one communicator among the four. > > Does anybody have any idea of what is happening and why? > > Thanks, > > -- > Giuseppe Congiu *ï¿½* Research Engineer II > Seagate Technology, LLC > office: +44 (0)23 9249 6082 *ï¿½* mobile: > www.seagate.com <http://www.seagate.com> > > > _______________________________________________ > To manage subscription options or unsubscribe: > https://lists.mpich.org/mailman/listinfo/devel > -- Rob Latham Mathematics and Computer Science Division Argonne National Lab, IL USA

1 0

MPI_Alltoall fails when called with newly created communicators
by Giuseppe Congiu 21 Oct '15

21 Oct '15

Hello everyone, I am currently working on a partitioned collective I/O implementation for a EU project. The idea is to partition the accessed file range into disjoint access regions (i.e., regions that do not overlap with each other) and assign processes from these regions to independent communicators created by splitting the original communicator with MPI_Comm_split. Very much like it is done in the ParColl paper or in the Memory Conscious Collective I/O paper. The code is pretty simple. It just uses the access pattern information to create the new communicators, assign aggregators to each of them and then compute file domains and data dependencies for every process using default functions from ROMIO (i.e. ADIOI_Calc_file_domains, ADIOI_Calc_my_req, ADIOI_Calc_others_req). I am currently testing my implementation using coll_perf with 512 processes. Every process writes 64MB of data for a total of 32GBs (no reads are performed). The ranks belonging to non-overlapping regions (of file areas) are printed out by the process with rank 0 in the global communicator: [romio/adio/common/ad_aggregate.c:0] file_area_count = 8 [romio/adio/common/ad_aggregate.c:0] file_area_ranklist[0] = 0 1 2 3 ... 63 [romio/adio/common/ad_aggregate.c:0] file_area_ranklist[1] = 64 65 66 67 ... 127 [romio/adio/common/ad_aggregate.c:0] file_area_ranklist[2] = 128 129 130 131 ... 191 [romio/adio/common/ad_aggregate.c:0] file_area_ranklist[3] = 192 193 194 195 ... 255 [romio/adio/common/ad_aggregate.c:0] file_area_ranklist[4] = 256 257 258 259 ... 319 [romio/adio/common/ad_aggregate.c:0] file_area_ranklist[5] = 320 321 322 323 ... 383 [romio/adio/common/ad_aggregate.c:0] file_area_ranklist[6] = 384 385 386 387 ... 447 [romio/adio/common/ad_aggregate.c:0] file_area_ranklist[7] = 448 449 450 451 ... 511 Afterwards, every aggregator in each communication group prints its rank and aggregator number in ranklist: [romio/adio/common/ad_aggregate.c:0] my_cb_nodes_index = 0 [romio/adio/common/ad_aggregate.c:0] fd->hints->ranklist[0] = 0 [romio/adio/common/ad_aggregate.c:0] my_cb_nodes_index = 0 [romio/adio/common/ad_aggregate.c:0] fd->hints->ranklist[0] = 0 [romio/adio/common/ad_aggregate.c:0] my_cb_nodes_index = 0 [romio/adio/common/ad_aggregate.c:0] fd->hints->ranklist[0] = 0 [romio/adio/common/ad_aggregate.c:0] my_cb_nodes_index = 0 [romio/adio/common/ad_aggregate.c:0] fd->hints->ranklist[0] = 0 In this particular case I have set fd->hints->cb_nodes to 4. This means that there are 4 aggregators but 8 non-overlapping regions (file areas). Thus, I create only 4 communicators and assign to each of them 2 file areas. In the previous printed messages the only aggregator for each communicator is printing its information. Then every process prints its group name, the number of process in the group, its rank in the old and new communicator: group-0:128:0:0 group-0:128:1:1 group-0:128:2:2 group-0:128:3:3 --- group-0:128:127:127 group-1:128:128:0 group-1:128:129:1 group-1:128:130:2 group-1:128:131:3 --- group-1:128:255:127 group-2:128:256:0 group-2:128:257:1 group-2:128:258:2 group-2:128:259:3 --- group-2:128:383:127 group-3:128:384:0 group-3:128:385:1 group-3:128:386:2 group-3:128:387:3 --- group-3:128:511:127 So far it looks like the communicators are created properly. After calling the ADIOI_Calc_* functions every aggregator prints MIN and MAX offsets for every independent access range: [romio/adio/common/ad_write_coll.c:0] st_loc = 17179869184, end_loc = 25769803775 [romio/adio/common/ad_write_coll.c:0] st_loc = 8589934592, end_loc = 17179869183 [romio/adio/common/ad_write_coll.c:0] st_loc = 25769803776, end_loc = 34359738367 [romio/adio/common/ad_write_coll.c:0] st_loc = 0, end_loc = 8589934591 Even though the new communicators seems to be created correctly, something bad happens when two phase I/O starts and the first MPI_Alltoall() is called to exchange access information among processes. MPI_Alltoall is called using the new communicators separately and every buffer passed to it has the new nprocs size for the corresponding communicator. Nevertheless, for some reason I am getting the following error message: Fatal error in PMPI_Alltoall: Other MPI error, error stack: PMPI_Alltoall(888)......: MPI_Alltoall(sbuf=0x2551e98, scount=1, MPI_INT, rbuf=0x2551be8, rcount=1, MPI_INT, comm=0x84000001) failed MPIR_Alltoall_impl(760).: MPIR_Alltoall(725)......: MPIR_Alltoall_intra(283): repeated many more times. This is actually strange because if I look at the file after coll_perf crashes there is some data in it (~200MB). Like if the problem was caused by only one communicator among the four. Does anybody have any idea of what is happening and why? Thanks, -- Giuseppe Congiu *·* Research Engineer II Seagate Technology, LLC office: +44 (0)23 9249 6082 *·* mobile: www.seagate.com

1 0

Re: [mpich-devel] [PATCH] MPI_Aint_* tests: use MPI_INTEGER datatype in Fortran
by Kenneth Raffenetti 15 Oct '15

15 Oct '15

Thanks Jeff. The patch looks good. We are in a freeze now leading up to release, so this probably won't make 3.2. Either way, it will land in master soon. Ken On 10/14/2015 08:58 PM, Jeff Squyres wrote: > MPI_INT corresponds to type(C_INT); MPI_INTEGER corresponds to INTEGER. > --- > test/mpi/f08/rma/aintf08.f90 | 4 ++-- > test/mpi/f77/rma/aintf.f | 6 +++--- > 2 files changed, 5 insertions(+), 5 deletions(-) > > diff --git a/test/mpi/f08/rma/aintf08.f90 b/test/mpi/f08/rma/aintf08.f90 > index cef75eb..b4dce12 100644 > --- a/test/mpi/f08/rma/aintf08.f90 > +++ b/test/mpi/f08/rma/aintf08.f90 > @@ -42,7 +42,7 @@ program main > endif > > ! Exchange bases > - call MPI_Type_size(MPI_INT, intsize, ierr); > + call MPI_Type_size(MPI_INTEGER, intsize, ierr); > call MPI_Allgather(MPI_IN_PLACE, 0, MPI_DATATYPE_NULL, bases, 1, MPI_AINT, MPI_COMM_WORLD, ierr) > call MPI_Win_create_dynamic(MPI_INFO_NULL, MPI_COMM_WORLD, win, ierr) > > @@ -60,7 +60,7 @@ program main > > ! Get value and verify it > call MPI_Win_fence(MPI_MODE_NOPRECEDE, win, ierr) > - call MPI_Get(val, 1, MPI_INT, target_rank, offset, 1, MPI_INT, win, ierr) > + call MPI_Get(val, 1, MPI_INTEGER, target_rank, offset, 1, MPI_INTEGER, win, ierr) > call MPI_Win_fence(MPI_MODE_NOSUCCEED, win, ierr) > > if (val /= 1234) then > diff --git a/test/mpi/f77/rma/aintf.f b/test/mpi/f77/rma/aintf.f > index 87464ba..9df429b 100644 > --- a/test/mpi/f77/rma/aintf.f > +++ b/test/mpi/f77/rma/aintf.f > @@ -46,7 +46,7 @@ > endif > > ! Exchange bases > - call MPI_Type_size(MPI_INT, intsize, ierr); > + call MPI_Type_size(MPI_INTEGER, intsize, ierr); > > call MPI_Allgather(MPI_IN_PLACE, 0, MPI_DATATYPE_NULL, bases, > & 1, MPI_AINT, MPI_COMM_WORLD, ierr) > @@ -68,8 +68,8 @@ > > ! Get value and verify it > call MPI_Win_fence(MPI_MODE_NOPRECEDE, win, ierr) > - call MPI_Get(val, 1, MPI_INT, target_rank, > - & offset, 1, MPI_INT, win, ierr) > + call MPI_Get(val, 1, MPI_INTEGER, target_rank, > + & offset, 1, MPI_INTEGER, win, ierr) > call MPI_Win_fence(MPI_MODE_NOSUCCEED, win, ierr) > > if (val /= 1234) then >

1 0