April 2013 - devel - mailman3.cels.anl.gov

segfault calling neighbor collectives in communicator with no topology
by Lisandro Dalcin 30 Apr '13

30 Apr '13

I'm adding support for the MPI-3 neighborhood collectives to mpi4py. By mistake, I called a neighbor collective on COMM_SELF, and got a segfault. After running under valgrind, I get the trace below. It seems that MPICH (running 3.0.4) is not checking the communicators for a topology being attached. This should be fixed in MPIR_Topo_canon_nhb_count() at src/mpi/topo/topoutil.c, adding a check after the following line: topo_ptr = MPIR_Topology_get(comm_ptr); BTW, the same kind of check should also be added to MPIR_Topo_canon_nhb(). ==14696== Invalid read of size 4 ==14696== at 0xDE0ED39: MPIR_Topo_canon_nhb_count (topoutil.c:283) ==14696== by 0xDFA870A: MPIR_Ineighbor_allgather_default (inhb_allgather.c:50) ==14696== by 0xDFA8B8B: MPIR_Ineighbor_allgather_impl (inhb_allgather.c:98) ==14696== by 0xDFAE27C: MPIR_Neighbor_allgather_default (nhb_allgather.c:37) ==14696== by 0xDFAE350: MPIR_Neighbor_allgather_impl (nhb_allgather.c:58) ==14696== by 0xDFAE918: PMPI_Neighbor_allgather (nhb_allgather.c:155) ==14696== by 0xDAD7B77: __pyx_pw_6mpi4py_3MPI_9Intracomm_25Neighbor_allgather (mpi4py.MPI.c:87767) ==14696== by 0x31784DD280: PyEval_EvalFrameEx (in /usr/lib64/libpython2.7.so.1.0) ==14696== by 0x31784DCEF0: PyEval_EvalFrameEx (in /usr/lib64/libpython2.7.so.1.0) ==14696== by 0x31784DDCBE: PyEval_EvalCodeEx (in /usr/lib64/libpython2.7.so.1.0) ==14696== by 0x317846DA36: ??? (in /usr/lib64/libpython2.7.so.1.0) ==14696== by 0x3178449C0D: PyObject_Call (in /usr/lib64/libpython2.7.so.1.0) ==14696== Address 0x0 is not stack'd, malloc'd or (recently) free'd -- Lisandro Dalcin --------------- CIMEC (INTEC/CONICET-UNL) Predio CONICET-Santa Fe Colectora RN 168 Km 472, Paraje El Pozo 3000 Santa Fe, Argentina Tel: +54-342-4511594 (ext 1011) Tel/Fax: +54-342-4511169

1 0

Re: [mpich-devel] ROMIO collective i/o memory use
by Jeff Hammond 29 Apr '13

29 Apr '13

I believe that I am on record as complaining vigorously about O(nproc) allocations in BG-MPICH. However, my first rule of parallel IO is to not do IO, so you will not hear me complain about this specific set of issues unless they are due to O(nproc) allocations inside of MPI collectives. Jeff On Mon, Apr 29, 2013 at 10:28 AM, Bob Cernohous <bobc(a)us.ibm.com> wrote: > A customer (Argonne ;) is complaining about O(p) allocations in collective > i/o. A collective read is failing at larger scale. > > Any thoughts or comments or advice? There appears to be lots of O(p) in > ROMIO collective I/O. Plus a lot of (possibly large) aggregated data > buffers. A quick search shows > > The common ROMIO read collective code: > > Find all "ADIOI_Malloc", Match case, Regular expression (UNIX) > File Z:\bgq\comm\lib\dev\mpich2\src\mpi\romio\adio\common\ad_read_coll.c > 124 38: st_offsets = (ADIO_Offset *) > ADIOI_Malloc(nprocs*sizeof(ADIO_Offset)); > 125 39: end_offsets = (ADIO_Offset *) > ADIOI_Malloc(nprocs*sizeof(ADIO_Offset)); > 317 44: *offset_list_ptr = (ADIO_Offset *) > ADIOI_Malloc(2*sizeof(ADIO_Offset)); > 318 41: *len_list_ptr = (ADIO_Offset *) > ADIOI_Malloc(2*sizeof(ADIO_Offset)); > 334 44: *offset_list_ptr = (ADIO_Offset *) > ADIOI_Malloc(2*sizeof(ADIO_Offset)); > 335 41: *len_list_ptr = (ADIO_Offset *) > ADIOI_Malloc(2*sizeof(ADIO_Offset)); > 436 18: > ADIOI_Malloc((contig_access_count+1)*sizeof(ADIO_Offset)); > 437 41: *len_list_ptr = (ADIO_Offset *) > ADIOI_Malloc((contig_access_count+1)*sizeof(ADIO_Offset)); > 573 37: if (ntimes) read_buf = (char *) ADIOI_Malloc(coll_bufsize); > 578 21: count = (int *) ADIOI_Malloc(nprocs * sizeof(int)); > 587 25: send_size = (int *) ADIOI_Malloc(nprocs * sizeof(int)); > 590 25: recv_size = (int *) ADIOI_Malloc(nprocs * sizeof(int)); > 598 25: start_pos = (int *) ADIOI_Malloc(nprocs*sizeof(int)); > 739 32: tmp_buf = (char *) ADIOI_Malloc(for_next_iter); > 744 33: read_buf = (char *) > ADIOI_Malloc(for_next_iter+coll_bufsize); > 805 9: > ADIOI_Malloc((nprocs_send+nprocs_recv+1)*sizeof(MPI_Request)); > 827 30: recv_buf = (char **) ADIOI_Malloc(nprocs * sizeof(char*)); > 830 44: (char *) > ADIOI_Malloc(recv_size[i]); > 870 31: statuses = (MPI_Status *) > ADIOI_Malloc((nprocs_send+nprocs_recv+1) * \ > 988 35: curr_from_proc = (unsigned *) ADIOI_Malloc(nprocs * > sizeof(unsigned)); > 989 35: done_from_proc = (unsigned *) ADIOI_Malloc(nprocs * > sizeof(unsigned)); > 990 35: recv_buf_idx = (unsigned *) ADIOI_Malloc(nprocs * > sizeof(unsigned)); > Total found: 22 > > Our BG version of read collective: > > File Z:\bgq\comm\lib\dev\mpich2\src\mpi\romio\adio\ad_bg\ad_bg_rdcoll.c > 179 40: st_offsets = (ADIO_Offset *) > ADIOI_Malloc(nprocs*sizeof(ADIO_Offset)); > 180 40: end_offsets = (ADIO_Offset *) > ADIOI_Malloc(nprocs*sizeof(ADIO_Offset)); > 183 43: bg_offsets0 = (ADIO_Offset *) > ADIOI_Malloc(2*nprocs*sizeof(ADIO_Offset)); > 184 43: bg_offsets = (ADIO_Offset *) > ADIOI_Malloc(2*nprocs*sizeof(ADIO_Offset)); > 475 37: if (ntimes) read_buf = (char *) ADIOI_Malloc(coll_bufsize); > 480 21: count = (int *) ADIOI_Malloc(nprocs * sizeof(int)); > 489 25: send_size = (int *) ADIOI_Malloc(nprocs * sizeof(int)); > 492 25: recv_size = (int *) ADIOI_Malloc(nprocs * sizeof(int)); > 500 25: start_pos = (int *) ADIOI_Malloc(nprocs*sizeof(int)); > 676 32: tmp_buf = (char *) ADIOI_Malloc(for_next_iter); > 681 33: read_buf = (char *) > ADIOI_Malloc(for_next_iter+coll_bufsize); > 761 9: > ADIOI_Malloc((nprocs_send+nprocs_recv+1)*sizeof(MPI_Request)); > 783 30: recv_buf = (char **) ADIOI_Malloc(nprocs * sizeof(char*)); > 786 44: (char *) > ADIOI_Malloc(recv_size[i]); > 826 31: statuses = (MPI_Status *) > ADIOI_Malloc((nprocs_send+nprocs_recv+1) * \ > 944 35: curr_from_proc = (unsigned *) ADIOI_Malloc(nprocs * > sizeof(unsigned)); > 945 35: done_from_proc = (unsigned *) ADIOI_Malloc(nprocs * > sizeof(unsigned)); > 946 35: recv_buf_idx = (unsigned *) ADIOI_Malloc(nprocs * > sizeof(unsigned)); > 1058 23: rdispls = (int *) ADIOI_Malloc( nprocs * sizeof(int) ); > 1063 29: all_recv_buf = (char *) ADIOI_Malloc( rtail ); > 1064 26: recv_buf = (char **) ADIOI_Malloc(nprocs * sizeof(char *)); > 1068 23: sdispls = (int *) ADIOI_Malloc( nprocs * sizeof(int) ); > 1073 29: all_send_buf = (char *) ADIOI_Malloc( stail ); > Total found: 23 > > > Bob Cernohous: (T/L 553) 507-253-6093 > > BobC(a)us.ibm.com > IBM Rochester, Building 030-2(C335), Department 61L > 3605 Hwy 52 North, Rochester, MN 55901-7829 > >> Chaos reigns within. >> Reflect, repent, and reboot. >> Order shall return. -- Jeff Hammond Argonne Leadership Computing Facility University of Chicago Computation Institute jhammond(a)alcf.anl.gov / (630) 252-5381 http://www.linkedin.com/in/jeffhammond https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond ALCF docs: http://www.alcf.anl.gov/user-guides

1 0

Re: [mpich-devel] ROMIO collective i/o memory use
by Kevin Harms 29 Apr '13

29 Apr '13

Just as a heads up, both Rob L. and Dave G. are out some or all of this week. kevin On Apr 29, 2013, at 10:28 AM, Bob Cernohous <bobc(a)us.ibm.com> wrote: > A customer (Argonne ;) is complaining about O(p) allocations in collective i/o. A collective read is failing at larger scale. > > Any thoughts or comments or advice? There appears to be lots of O(p) in ROMIO collective I/O. Plus a lot of (possibly large) aggregated data buffers. A quick search shows > > The common ROMIO read collective code: > > Find all "ADIOI_Malloc", Match case, Regular expression (UNIX) > File Z:\bgq\comm\lib\dev\mpich2\src\mpi\romio\adio\common\ad_read_coll.c > 124 38: st_offsets = (ADIO_Offset *) ADIOI_Malloc(nprocs*sizeof(ADIO_Offset)); > 125 39: end_offsets = (ADIO_Offset *) ADIOI_Malloc(nprocs*sizeof(ADIO_Offset)); > 317 44: *offset_list_ptr = (ADIO_Offset *) ADIOI_Malloc(2*sizeof(ADIO_Offset)); > 318 41: *len_list_ptr = (ADIO_Offset *) ADIOI_Malloc(2*sizeof(ADIO_Offset)); > 334 44: *offset_list_ptr = (ADIO_Offset *) ADIOI_Malloc(2*sizeof(ADIO_Offset)); > 335 41: *len_list_ptr = (ADIO_Offset *) ADIOI_Malloc(2*sizeof(ADIO_Offset)); > 436 18: ADIOI_Malloc((contig_access_count+1)*sizeof(ADIO_Offset)); > 437 41: *len_list_ptr = (ADIO_Offset *) ADIOI_Malloc((contig_access_count+1)*sizeof(ADIO_Offset)); > 573 37: if (ntimes) read_buf = (char *) ADIOI_Malloc(coll_bufsize); > 578 21: count = (int *) ADIOI_Malloc(nprocs * sizeof(int)); > 587 25: send_size = (int *) ADIOI_Malloc(nprocs * sizeof(int)); > 590 25: recv_size = (int *) ADIOI_Malloc(nprocs * sizeof(int)); > 598 25: start_pos = (int *) ADIOI_Malloc(nprocs*sizeof(int)); > 739 32: tmp_buf = (char *) ADIOI_Malloc(for_next_iter); > 744 33: read_buf = (char *) ADIOI_Malloc(for_next_iter+coll_bufsize); > 805 9: ADIOI_Malloc((nprocs_send+nprocs_recv+1)*sizeof(MPI_Request)); > 827 30: recv_buf = (char **) ADIOI_Malloc(nprocs * sizeof(char*)); > 830 44: (char *) ADIOI_Malloc(recv_size[i]); > 870 31: statuses = (MPI_Status *) ADIOI_Malloc((nprocs_send+nprocs_recv+1) * \ > 988 35: curr_from_proc = (unsigned *) ADIOI_Malloc(nprocs * sizeof(unsigned)); > 989 35: done_from_proc = (unsigned *) ADIOI_Malloc(nprocs * sizeof(unsigned)); > 990 35: recv_buf_idx = (unsigned *) ADIOI_Malloc(nprocs * sizeof(unsigned)); > Total found: 22 > > Our BG version of read collective: > > File Z:\bgq\comm\lib\dev\mpich2\src\mpi\romio\adio\ad_bg\ad_bg_rdcoll.c > 179 40: st_offsets = (ADIO_Offset *) ADIOI_Malloc(nprocs*sizeof(ADIO_Offset)); > 180 40: end_offsets = (ADIO_Offset *) ADIOI_Malloc(nprocs*sizeof(ADIO_Offset)); > 183 43: bg_offsets0 = (ADIO_Offset *) ADIOI_Malloc(2*nprocs*sizeof(ADIO_Offset)); > 184 43: bg_offsets = (ADIO_Offset *) ADIOI_Malloc(2*nprocs*sizeof(ADIO_Offset)); > 475 37: if (ntimes) read_buf = (char *) ADIOI_Malloc(coll_bufsize); > 480 21: count = (int *) ADIOI_Malloc(nprocs * sizeof(int)); > 489 25: send_size = (int *) ADIOI_Malloc(nprocs * sizeof(int)); > 492 25: recv_size = (int *) ADIOI_Malloc(nprocs * sizeof(int)); > 500 25: start_pos = (int *) ADIOI_Malloc(nprocs*sizeof(int)); > 676 32: tmp_buf = (char *) ADIOI_Malloc(for_next_iter); > 681 33: read_buf = (char *) ADIOI_Malloc(for_next_iter+coll_bufsize); > 761 9: ADIOI_Malloc((nprocs_send+nprocs_recv+1)*sizeof(MPI_Request)); > 783 30: recv_buf = (char **) ADIOI_Malloc(nprocs * sizeof(char*)); > 786 44: (char *) ADIOI_Malloc(recv_size[i]); > 826 31: statuses = (MPI_Status *) ADIOI_Malloc((nprocs_send+nprocs_recv+1) * \ > 944 35: curr_from_proc = (unsigned *) ADIOI_Malloc(nprocs * sizeof(unsigned)); > 945 35: done_from_proc = (unsigned *) ADIOI_Malloc(nprocs * sizeof(unsigned)); > 946 35: recv_buf_idx = (unsigned *) ADIOI_Malloc(nprocs * sizeof(unsigned)); > 1058 23: rdispls = (int *) ADIOI_Malloc( nprocs * sizeof(int) ); > 1063 29: all_recv_buf = (char *) ADIOI_Malloc( rtail ); > 1064 26: recv_buf = (char **) ADIOI_Malloc(nprocs * sizeof(char *)); > 1068 23: sdispls = (int *) ADIOI_Malloc( nprocs * sizeof(int) ); > 1073 29: all_send_buf = (char *) ADIOI_Malloc( stail ); > Total found: 23 > > > Bob Cernohous: (T/L 553) 507-253-6093 > > BobC(a)us.ibm.com > IBM Rochester, Building 030-2(C335), Department 61L > 3605 Hwy 52 North, Rochester, MN 55901-7829 > > > Chaos reigns within. > > Reflect, repent, and reboot. > > Order shall return.

1 0

ROMIO collective i/o memory use
by Bob Cernohous 29 Apr '13

29 Apr '13

A customer (Argonne ;) is complaining about O(p) allocations in collective i/o. A collective read is failing at larger scale. Any thoughts or comments or advice? There appears to be lots of O(p) in ROMIO collective I/O. Plus a lot of (possibly large) aggregated data buffers. A quick search shows The common ROMIO read collective code: Find all "ADIOI_Malloc", Match case, Regular expression (UNIX) File Z:\bgq\comm\lib\dev\mpich2\src\mpi\romio\adio\common\ad_read_coll.c 124 38: st_offsets = (ADIO_Offset *) ADIOI_Malloc(nprocs*sizeof(ADIO_Offset)); 125 39: end_offsets = (ADIO_Offset *) ADIOI_Malloc(nprocs*sizeof(ADIO_Offset)); 317 44: *offset_list_ptr = (ADIO_Offset *) ADIOI_Malloc(2*sizeof(ADIO_Offset)); 318 41: *len_list_ptr = (ADIO_Offset *) ADIOI_Malloc(2*sizeof(ADIO_Offset)); 334 44: *offset_list_ptr = (ADIO_Offset *) ADIOI_Malloc(2*sizeof(ADIO_Offset)); 335 41: *len_list_ptr = (ADIO_Offset *) ADIOI_Malloc(2*sizeof(ADIO_Offset)); 436 18: ADIOI_Malloc((contig_access_count+1)*sizeof(ADIO_Offset)); 437 41: *len_list_ptr = (ADIO_Offset *) ADIOI_Malloc((contig_access_count+1)*sizeof(ADIO_Offset)); 573 37: if (ntimes) read_buf = (char *) ADIOI_Malloc(coll_bufsize); 578 21: count = (int *) ADIOI_Malloc(nprocs * sizeof(int)); 587 25: send_size = (int *) ADIOI_Malloc(nprocs * sizeof(int)); 590 25: recv_size = (int *) ADIOI_Malloc(nprocs * sizeof(int)); 598 25: start_pos = (int *) ADIOI_Malloc(nprocs*sizeof(int)); 739 32: tmp_buf = (char *) ADIOI_Malloc(for_next_iter); 744 33: read_buf = (char *) ADIOI_Malloc(for_next_iter+coll_bufsize); 805 9: ADIOI_Malloc((nprocs_send+nprocs_recv+1)*sizeof(MPI_Request)); 827 30: recv_buf = (char **) ADIOI_Malloc(nprocs * sizeof(char*)); 830 44: (char *) ADIOI_Malloc(recv_size[i]); 870 31: statuses = (MPI_Status *) ADIOI_Malloc((nprocs_send+nprocs_recv+1) * \ 988 35: curr_from_proc = (unsigned *) ADIOI_Malloc(nprocs * sizeof(unsigned)); 989 35: done_from_proc = (unsigned *) ADIOI_Malloc(nprocs * sizeof(unsigned)); 990 35: recv_buf_idx = (unsigned *) ADIOI_Malloc(nprocs * sizeof(unsigned)); Total found: 22 Our BG version of read collective: File Z:\bgq\comm\lib\dev\mpich2\src\mpi\romio\adio\ad_bg\ad_bg_rdcoll.c 179 40: st_offsets = (ADIO_Offset *) ADIOI_Malloc(nprocs*sizeof(ADIO_Offset)); 180 40: end_offsets = (ADIO_Offset *) ADIOI_Malloc(nprocs*sizeof(ADIO_Offset)); 183 43: bg_offsets0 = (ADIO_Offset *) ADIOI_Malloc(2*nprocs*sizeof(ADIO_Offset)); 184 43: bg_offsets = (ADIO_Offset *) ADIOI_Malloc(2*nprocs*sizeof(ADIO_Offset)); 475 37: if (ntimes) read_buf = (char *) ADIOI_Malloc(coll_bufsize); 480 21: count = (int *) ADIOI_Malloc(nprocs * sizeof(int)); 489 25: send_size = (int *) ADIOI_Malloc(nprocs * sizeof(int)); 492 25: recv_size = (int *) ADIOI_Malloc(nprocs * sizeof(int)); 500 25: start_pos = (int *) ADIOI_Malloc(nprocs*sizeof(int)); 676 32: tmp_buf = (char *) ADIOI_Malloc(for_next_iter); 681 33: read_buf = (char *) ADIOI_Malloc(for_next_iter+coll_bufsize); 761 9: ADIOI_Malloc((nprocs_send+nprocs_recv+1)*sizeof(MPI_Request)); 783 30: recv_buf = (char **) ADIOI_Malloc(nprocs * sizeof(char*)); 786 44: (char *) ADIOI_Malloc(recv_size[i]); 826 31: statuses = (MPI_Status *) ADIOI_Malloc((nprocs_send+nprocs_recv+1) * \ 944 35: curr_from_proc = (unsigned *) ADIOI_Malloc(nprocs * sizeof(unsigned)); 945 35: done_from_proc = (unsigned *) ADIOI_Malloc(nprocs * sizeof(unsigned)); 946 35: recv_buf_idx = (unsigned *) ADIOI_Malloc(nprocs * sizeof(unsigned)); 1058 23: rdispls = (int *) ADIOI_Malloc( nprocs * sizeof(int) ); 1063 29: all_recv_buf = (char *) ADIOI_Malloc( rtail ); 1064 26: recv_buf = (char **) ADIOI_Malloc(nprocs * sizeof(char *)); 1068 23: sdispls = (int *) ADIOI_Malloc( nprocs * sizeof(int) ); 1073 29: all_send_buf = (char *) ADIOI_Malloc( stail ); Total found: 23 Bob Cernohous: (T/L 553) 507-253-6093 BobC(a)us.ibm.com IBM Rochester, Building 030-2(C335), Department 61L 3605 Hwy 52 North, Rochester, MN 55901-7829 > Chaos reigns within. > Reflect, repent, and reboot. > Order shall return.

1 0

Re: [mpich-devel] MPI_Bsend fails multithreaded
by Bob Cernohous 26 Apr '13

26 Apr '13

If I put a new, big CS_ENTER(BSENDDATA) lock around the bsend processing (that doesn't yield), then MPIU_THREAD_GRANULARITY_PER_OBJECT works ok. Bob Cernohous: (T/L 553) 507-253-6093 BobC(a)us.ibm.com IBM Rochester, Building 030-2(C335), Department 61L 3605 Hwy 52 North, Rochester, MN 55901-7829 > Chaos reigns within. > Reflect, repent, and reboot. > Order shall return. devel-bounces(a)mpich.org wrote on 04/25/2013 11:20:17 PM: > From: Bob Cernohous/Rochester/IBM@IBMUS > To: mpich2-dev(a)mcs.anl.gov, > Cc: Nysal K Jan <jnysal(a)in.ibm.com>, Haizhu Liu/Poughkeepsie/ > IBM@IBMUS, Sameh S Sharkawi/Austin/IBM@IBMUS, Su Huang/Poughkeepsie/IBM@IBMUS > Date: 04/25/2013 11:20 PM > Subject: Re: [mpich-devel] MPI_Bsend fails multithreaded > Sent by: devel-bounces(a)mpich.org > > I think my problem with MPIU_THREAD_GRANULARITY_GLOBAL is a CS_YIELD. > > MPIU_THREAD_CS_ENTER(ALLFUNC,); > .... > > MPIR_Bsend_data_t *active = BsendBuffer.active, *next_active; > while (active) { > fprintf(stderr,"%2.2u:%u:active %p (0x%08x kind=%d) refcount %d\n", > Kernel_ProcessorID(),__LINE__, > (active->request), > (active->request)->handle, > active->request->kind, > MPIU_Object_get_ref((active->request))); > ... > > There's one or more yields somewhere... in test and/or progress. I > haven't tracked it down and I'm out tomorrow. I end up with 3 > threads (26, 54, 48) working on the same active request. 26 frees > it and moves on to the next active. 48 chokes on the freed request. > > stderr[0]: threaded exit > stderr[0]: 26:441:active 0x15d1c78 (0xac000003 kind=1) refcount 2 > stderr[0]: 26:decr 0x15d1aa8 (0xac000001 kind=REQUEST) refcount to 1 > stderr[0]: 26:decr 0x15d1c78 (0xac000003 kind=REQUEST) refcount to 1 > stderr[0]: yield > > stderr[0]: 54:441:active 0x15d1c78 (0xac000003 kind=1) refcount 1 > stderr[0]: 54:set 0x15d1d60 (0xac000004 kind=REQUEST) refcount to 1 > stderr[0]: 54:set 0x15d1d60 (0xac000004 kind=REQUEST) refcount to 2 > stderr[0]: 54:decr 0x15d1d60 (0xac000004 kind=REQUEST) refcount to 1 > stderr[0]: yield > > stderr[0]: 48:441:active 0x15d1c78 (0xac000003 kind=1) refcount 1 > stderr[0]: 48:set 0x15d1e48 (0xac000005 kind=REQUEST) refcount to 1 > stderr[0]: 48:set 0x15d1e48 (0xac000005 kind=REQUEST) refcount to 2 > stderr[0]: 48:decr 0x15d1e48 (0xac000005 kind=REQUEST) refcount to 1 > stderr[0]: yield > > stderr[0]: 26:decr 0x15d1c78 (0xac000003 kind=REQUEST) refcount to 0 > stderr[0]: 26:decr 0x1560f78 (0x44000000 kind=COMM) refcount to 3 > stderr[0]: 26:free 0x15d1c78 (0xac000003 kind=0) refcount 0 > stderr[0]: 26:356:prev 0x15d1c78, active 0x15d1aa8 (0xac000001 > kind=1) refcount 1 > stderr[0]: 26:441:active 0x15d1aa8 (0xac000001 kind=1) refcount 1 > > stderr[0]: yield > stderr[0]: 32:441:active 0x15d1aa8 (0xac000001 kind=1) refcount 1 > > stderr[0]: yield > stderr[0]: 48:badcase 0x15d1c78 (0xac000003 kind=0) refcount 0 > stderr[0]: Abort(1) on node 0 (rank 0 in comm 1140850688): Fatal > error in MPI_Bsend: Internal MPI error!, error stack: > stderr[0]: MPI_Bsend(181)..............: MPI_Bsend(buf=0x19c8a06d70, > count=1024, MPI_CHAR, dest=1, tag=0, MPI_COMM_WORLD) failed > stderr[0]: MPIR_Bsend_isend(226).......: > stderr[0]: MPIR_Bsend_check_active(474): > stderr[0]: MPIR_Test_impl(65)..........: > stderr[0]: MPIR_Request_complete(239)..: INTERNAL ERROR: unexpected > value in case statement (value=0) > > > I'm guessing the problem with MPIU_THREAD_GRANULARITY_PER_OBJECT is > there's no lock and the threads are all over each other... no yield > needed? It's just not thread safe with the static BsendBuffer. > > > Bob Cernohous: (T/L 553) 507-253-6093 > > BobC(a)us.ibm.com > IBM Rochester, Building 030-2(C335), Department 61L > 3605 Hwy 52 North, Rochester, MN 55901-7829 > > > Chaos reigns within. > > Reflect, repent, and reboot. > > Order shall return.

1 0

Re: [mpich-devel] MPI_Bsend fails multithreaded
by Bob Cernohous 25 Apr '13

25 Apr '13

I think my problem with MPIU_THREAD_GRANULARITY_GLOBAL is a CS_YIELD. MPIU_THREAD_CS_ENTER(ALLFUNC,); .... MPIR_Bsend_data_t *active = BsendBuffer.active, *next_active; while (active) { fprintf(stderr,"%2.2u:%u:active %p (0x%08x kind=%d) refcount %d\n", Kernel_ProcessorID(),__LINE__, (active->request), (active->request)->handle, active->request->kind, MPIU_Object_get_ref((active->request))); ... There's one or more yields somewhere... in test and/or progress. I haven't tracked it down and I'm out tomorrow. I end up with 3 threads (26, 54, 48) working on the same active request. 26 frees it and moves on to the next active. 48 chokes on the freed request. stderr[0]: threaded exit stderr[0]: 26:441:active 0x15d1c78 (0xac000003 kind=1) refcount 2 stderr[0]: 26:decr 0x15d1aa8 (0xac000001 kind=REQUEST) refcount to 1 stderr[0]: 26:decr 0x15d1c78 (0xac000003 kind=REQUEST) refcount to 1 stderr[0]: yield stderr[0]: 54:441:active 0x15d1c78 (0xac000003 kind=1) refcount 1 stderr[0]: 54:set 0x15d1d60 (0xac000004 kind=REQUEST) refcount to 1 stderr[0]: 54:set 0x15d1d60 (0xac000004 kind=REQUEST) refcount to 2 stderr[0]: 54:decr 0x15d1d60 (0xac000004 kind=REQUEST) refcount to 1 stderr[0]: yield stderr[0]: 48:441:active 0x15d1c78 (0xac000003 kind=1) refcount 1 stderr[0]: 48:set 0x15d1e48 (0xac000005 kind=REQUEST) refcount to 1 stderr[0]: 48:set 0x15d1e48 (0xac000005 kind=REQUEST) refcount to 2 stderr[0]: 48:decr 0x15d1e48 (0xac000005 kind=REQUEST) refcount to 1 stderr[0]: yield stderr[0]: 26:decr 0x15d1c78 (0xac000003 kind=REQUEST) refcount to 0 stderr[0]: 26:decr 0x1560f78 (0x44000000 kind=COMM) refcount to 3 stderr[0]: 26:free 0x15d1c78 (0xac000003 kind=0) refcount 0 stderr[0]: 26:356:prev 0x15d1c78, active 0x15d1aa8 (0xac000001 kind=1) refcount 1 stderr[0]: 26:441:active 0x15d1aa8 (0xac000001 kind=1) refcount 1 stderr[0]: yield stderr[0]: 32:441:active 0x15d1aa8 (0xac000001 kind=1) refcount 1 stderr[0]: yield stderr[0]: 48:badcase 0x15d1c78 (0xac000003 kind=0) refcount 0 stderr[0]: Abort(1) on node 0 (rank 0 in comm 1140850688): Fatal error in MPI_Bsend: Internal MPI error!, error stack: stderr[0]: MPI_Bsend(181)..............: MPI_Bsend(buf=0x19c8a06d70, count=1024, MPI_CHAR, dest=1, tag=0, MPI_COMM_WORLD) failed stderr[0]: MPIR_Bsend_isend(226).......: stderr[0]: MPIR_Bsend_check_active(474): stderr[0]: MPIR_Test_impl(65)..........: stderr[0]: MPIR_Request_complete(239)..: INTERNAL ERROR: unexpected value in case statement (value=0) I'm guessing the problem with MPIU_THREAD_GRANULARITY_PER_OBJECT is there's no lock and the threads are all over each other... no yield needed? It's just not thread safe with the static BsendBuffer. Bob Cernohous: (T/L 553) 507-253-6093 BobC(a)us.ibm.com IBM Rochester, Building 030-2(C335), Department 61L 3605 Hwy 52 North, Rochester, MN 55901-7829 > Chaos reigns within. > Reflect, repent, and reboot. > Order shall return.

1 0

Re: [mpich-devel] MPI_Bsend under MPIU_THREAD_GRANULARITY_PER_OBJECT
by Dave Goodell 25 Apr '13

25 Apr '13

Thanks for the test case. On my laptop with a debugging build of ch3:nemesis:tcp, I cannot reproduce the problem. I tried with and without valgrind, with and without the "MPI_Comm_dup", and for a range of "-np" values. Obviously this is inconclusive because this is a threading issue, but maybe there's a pamid-specific bug here? -Dave On Apr 25, 2013, at 2:02 PM CDT, Bob Cernohous <bobc(a)us.ibm.com> wrote: > Testcase is pretty simple. I'll keep poking at it myself and look at that ticket too. > > > > Bob Cernohous: (T/L 553) 507-253-6093 > > BobC(a)us.ibm.com > IBM Rochester, Building 030-2(C335), Department 61L > 3605 Hwy 52 North, Rochester, MN 55901-7829 > > > Chaos reigns within. > > Reflect, repent, and reboot. > > Order shall return. > > > devel-bounces(a)mpich.org wrote on 04/25/2013 01:57:35 PM: > > > From: Dave Goodell <goodell(a)mcs.anl.gov> > > To: devel(a)mpich.org, > > Date: 04/25/2013 01:58 PM > > Subject: Re: [mpich-devel] MPI_Bsend under MPIU_THREAD_GRANULARITY_PER_OBJECT > > Sent by: devel-bounces(a)mpich.org > > > > In that case, test programs to help us reproduce this are also welcome. > > > > There's also always a risk that one of the currently missing ALLFUNC > > critical sections is affecting your user: > > > > https://trac.mpich.org/projects/mpich/ticket/1797 > > > > -Dave > > > > On Apr 25, 2013, at 11:38 AM CDT, Bob Cernohous <bobc(a)us.ibm.com> wrote: > > > > Patches could be tricky because I'm even seeing intermittent (and > > different) failures on our 'legacy' libraries which are not per-obj > > but use the big lock. So there's probably more than one problem here. > > > > Abort(1) on node 3 (rank 3 in comm 1140850688): Fatal error in > > MPI_Bsend: Internal MPI error!, error stack: > > MPI_Bsend(181)..............: MPI_Bsend(buf=0x19c8a06da0, > > count=1024, MPI_CHAR, dest=0, tag=0, MPI_COMM_WORLD) failed > > MPIR_Bsend_isend(226).......: > > MPIR_Bsend_check_active(456): > > MPIR_Test_impl(65)..........: > > MPIR_Request_complete(234)..: INTERNAL ERROR: unexpected value in > > case statement (value=0) > > > > > > Bob Cernohous: (T/L 553) 507-253-6093 > > > > BobC(a)us.ibm.com > > IBM Rochester, Building 030-2(C335), Department 61L > > 3605 Hwy 52 North, Rochester, MN 55901-7829 > > > > > Chaos reigns within. > > > Reflect, repent, and reboot. > > > Order shall return. > > > > > > > > > > From: Dave Goodell <goodell(a)mcs.anl.gov> > > To: devel(a)mpich.org, > > Date: 04/25/2013 11:14 AM > > Subject: Re: [mpich-devel] MPI_Bsend under > > MPIU_THREAD_GRANULARITY_PER_OBJECT > > Sent by: devel-bounces(a)mpich.org > > > > > > > > The Bsend paths almost certainly have not been protected correctly. > > Patches to fix the issue are most welcome. > > > > -Dave > > > > On Apr 25, 2013, at 10:49 AM CDT, Bob Cernohous <bobc(a)us.ibm.com> wrote: > > > > > Start by saying that I have not been involved in the nitty gritty > > of the per-object locking design. > > > > > > What protects the attached buffer/data structures/request when > > doing multithreaded MPI_Bsend()'s? All I see in the code path is a > > (no-op) MPIU_THREAD_CS_ENTER(ALLFUNC,). > > > > > > I have a customer test in which the threads seem to be walking all > > over the request around: > > > > > > /bgusr/bobc/bgq/comm/lib/dev/mpich2/src/mpid/pamid/ > > include/../src/mpid_request.h:259 > > > 0000000001088c0c MPIR_Request_complete > > > /bgusr/bobc/bgq/comm/lib/dev/mpich2/src/mpi/pt2pt/mpir_request.c:87 > > > 000000000106e874 MPIR_Test_impl > > > /bgusr/bobc/bgq/comm/lib/dev/mpich2/src/mpi/pt2pt/test.c:62 > > > 00000000010188f0 MPIR_Bsend_check_active > > > /bgusr/bobc/bgq/comm/lib/dev/mpich2/src/mpi/pt2pt/bsendutil.c:455 > > > 0000000001018dc0 MPIR_Bsend_isend > > > /bgusr/bobc/bgq/comm/lib/dev/mpich2/src/mpi/pt2pt/bsendutil.c:226 > > > 0000000001008734 PMPI_Bsend > > > /bgusr/bobc/bgq/comm/lib/dev/mpich2/src/mpi/pt2pt/bsend.c:163 > > > 00000000010009c0 00000012.long_branch_r2off.__libc_start_main+0 > > > :0 > > > 000000000130cbc0 start_thread > > > > > > eg. (fprinting from MPIU_HANDLE_LOG_REFCOUNT_CHANGE) > > > > > > stderr[8]: set 0x15f8048 (0xac0000ff kind=REQUEST) refcount to 2 > > > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to 1 > > > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to 0 > > > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -1 > > > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -2 > > > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -3 > > > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -4 > > > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -5 > > > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -6 > > > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -7 > > > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -8 > > > > > > > > > Bob Cernohous: (T/L 553) 507-253-6093 > > > > > > BobC(a)us.ibm.com > > > IBM Rochester, Building 030-2(C335), Department 61L > > > 3605 Hwy 52 North, Rochester, MN 55901-7829 > > > > > > > Chaos reigns within. > > > > Reflect, repent, and reboot. > > > > Order shall return. > > > <main.c>

1 0

Re: [mpich-devel] MPI_Bsend under MPIU_THREAD_GRANULARITY_PER_OBJECT
by Dave Goodell 25 Apr '13

25 Apr '13

Hard to know what's really going on without a proper test case. Is the test case using cancel? The reference counting is known to be sloppy (at least in CH3) for some of the cancellation paths. -Dave On Apr 25, 2013, at 2:00 PM CDT, Bob Cernohous <bobc(a)us.ibm.com> wrote: > Here's the big lock failure... looks like we use a request after it's complete/refcount is 0 > > ... > stderr[0]: decr 0x15604b0 (0x44000000 kind=COMM) refcount to 3 > stderr[0]: set 0x15d0fe8 (0xac000001 kind=REQUEST) refcount to 2 > stderr[0]: incr 0x15604b0 (0x44000000 kind=COMM) refcount to 4 > stderr[0]: decr 0x15d0fe8 (0xac000001 kind=REQUEST) refcount to 1 > stderr[0]: decr 0x15d0fe8 (0xac000001 kind=REQUEST) refcount to 0 > ^^^^^^^^^ > stderr[0]: decr 0x15604b0 (0x44000000 kind=COMM) refcount to 3 > stderr[0]: decr 0x15d12a0 (0xac000004 kind=REQUEST) refcount to 0 > stderr[0]: decr 0x15604b0 (0x44000000 kind=COMM) refcount to 2 > stderr[0]: badcase 0x15d0fe8 (0xac000001 kind=0) refcount 0 > ^^^^^^^^^ > stderr[0]: Abort(1) on node 0 (rank 0 in comm 1140850688): Fatal error in MPI_Bsend: Internal MPI error!, error stack: > stderr[0]: MPI_Bsend(181)..............: MPI_Bsend(buf=0x19c8606d70, count=1024, MPI_CHAR, dest=1, tag=0, MPI_COMM_WORLD) failed > stderr[0]: MPIR_Bsend_isend(226).......: > stderr[0]: MPIR_Bsend_check_active(456): > stderr[0]: MPIR_Test_impl(65)..........: > stderr[0]: MPIR_Request_complete(239)..: INTERNAL ERROR: unexpected value in case statement (value=0) > > > Bob Cernohous: (T/L 553) 507-253-6093 > > BobC(a)us.ibm.com > IBM Rochester, Building 030-2(C335), Department 61L > 3605 Hwy 52 North, Rochester, MN 55901-7829 > > > Chaos reigns within. > > Reflect, repent, and reboot. > > Order shall return. > > > devel-bounces(a)mpich.org wrote on 04/25/2013 11:38:24 AM: > > > From: Bob Cernohous/Rochester/IBM@IBMUS > > To: devel(a)mpich.org, > > Date: 04/25/2013 11:43 AM > > Subject: Re: [mpich-devel] MPI_Bsend under MPIU_THREAD_GRANULARITY_PER_OBJECT > > Sent by: devel-bounces(a)mpich.org > > > > Patches could be tricky because I'm even seeing intermittent (and > > different) failures on our 'legacy' libraries which are not per-obj > > but use the big lock. So there's probably more than one problem here. > > > > Abort(1) on node 3 (rank 3 in comm 1140850688): Fatal error in > > MPI_Bsend: Internal MPI error!, error stack: > > MPI_Bsend(181)..............: MPI_Bsend(buf=0x19c8a06da0, > > count=1024, MPI_CHAR, dest=0, tag=0, MPI_COMM_WORLD) failed > > MPIR_Bsend_isend(226).......: > > MPIR_Bsend_check_active(456): > > MPIR_Test_impl(65)..........: > > MPIR_Request_complete(234)..: INTERNAL ERROR: unexpected value in > > case statement (value=0) > > > > > > Bob Cernohous: (T/L 553) 507-253-6093 > > > > BobC(a)us.ibm.com > > IBM Rochester, Building 030-2(C335), Department 61L > > 3605 Hwy 52 North, Rochester, MN 55901-7829 > > > > > Chaos reigns within. > > > Reflect, repent, and reboot. > > > Order shall return. > > > > > > > > > > From: Dave Goodell <goodell(a)mcs.anl.gov> > > To: devel(a)mpich.org, > > Date: 04/25/2013 11:14 AM > > Subject: Re: [mpich-devel] MPI_Bsend under > > MPIU_THREAD_GRANULARITY_PER_OBJECT > > Sent by: devel-bounces(a)mpich.org > > > > > > > > The Bsend paths almost certainly have not been protected correctly. > > Patches to fix the issue are most welcome. > > > > -Dave > > > > On Apr 25, 2013, at 10:49 AM CDT, Bob Cernohous <bobc(a)us.ibm.com> wrote: > > > > > Start by saying that I have not been involved in the nitty gritty > > of the per-object locking design. > > > > > > What protects the attached buffer/data structures/request when > > doing multithreaded MPI_Bsend()'s? All I see in the code path is a > > (no-op) MPIU_THREAD_CS_ENTER(ALLFUNC,). > > > > > > I have a customer test in which the threads seem to be walking all > > over the request around: > > > > > > /bgusr/bobc/bgq/comm/lib/dev/mpich2/src/mpid/pamid/ > > include/../src/mpid_request.h:259 > > > 0000000001088c0c MPIR_Request_complete > > > /bgusr/bobc/bgq/comm/lib/dev/mpich2/src/mpi/pt2pt/mpir_request.c:87 > > > 000000000106e874 MPIR_Test_impl > > > /bgusr/bobc/bgq/comm/lib/dev/mpich2/src/mpi/pt2pt/test.c:62 > > > 00000000010188f0 MPIR_Bsend_check_active > > > /bgusr/bobc/bgq/comm/lib/dev/mpich2/src/mpi/pt2pt/bsendutil.c:455 > > > 0000000001018dc0 MPIR_Bsend_isend > > > /bgusr/bobc/bgq/comm/lib/dev/mpich2/src/mpi/pt2pt/bsendutil.c:226 > > > 0000000001008734 PMPI_Bsend > > > /bgusr/bobc/bgq/comm/lib/dev/mpich2/src/mpi/pt2pt/bsend.c:163 > > > 00000000010009c0 00000012.long_branch_r2off.__libc_start_main+0 > > > :0 > > > 000000000130cbc0 start_thread > > > > > > eg. (fprinting from MPIU_HANDLE_LOG_REFCOUNT_CHANGE) > > > > > > stderr[8]: set 0x15f8048 (0xac0000ff kind=REQUEST) refcount to 2 > > > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to 1 > > > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to 0 > > > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -1 > > > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -2 > > > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -3 > > > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -4 > > > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -5 > > > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -6 > > > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -7 > > > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -8 > > > > > > > > > Bob Cernohous: (T/L 553) 507-253-6093 > > > > > > BobC(a)us.ibm.com > > > IBM Rochester, Building 030-2(C335), Department 61L > > > 3605 Hwy 52 North, Rochester, MN 55901-7829 > > > > > > > Chaos reigns within. > > > > Reflect, repent, and reboot. > > > > Order shall return. > >

1 0

Re: [mpich-devel] MPI_Bsend under MPIU_THREAD_GRANULARITY_PER_OBJECT
by Bob Cernohous 25 Apr '13

25 Apr '13

Testcase is pretty simple. I'll keep poking at it myself and look at that ticket too. Bob Cernohous: (T/L 553) 507-253-6093 BobC(a)us.ibm.com IBM Rochester, Building 030-2(C335), Department 61L 3605 Hwy 52 North, Rochester, MN 55901-7829 > Chaos reigns within. > Reflect, repent, and reboot. > Order shall return. devel-bounces(a)mpich.org wrote on 04/25/2013 01:57:35 PM: > From: Dave Goodell <goodell(a)mcs.anl.gov> > To: devel(a)mpich.org, > Date: 04/25/2013 01:58 PM > Subject: Re: [mpich-devel] MPI_Bsend under MPIU_THREAD_GRANULARITY_PER_OBJECT > Sent by: devel-bounces(a)mpich.org > > In that case, test programs to help us reproduce this are also welcome. > > There's also always a risk that one of the currently missing ALLFUNC > critical sections is affecting your user: > > https://trac.mpich.org/projects/mpich/ticket/1797 > > -Dave > > On Apr 25, 2013, at 11:38 AM CDT, Bob Cernohous <bobc(a)us.ibm.com> wrote: > > Patches could be tricky because I'm even seeing intermittent (and > different) failures on our 'legacy' libraries which are not per-obj > but use the big lock. So there's probably more than one problem here. > > Abort(1) on node 3 (rank 3 in comm 1140850688): Fatal error in > MPI_Bsend: Internal MPI error!, error stack: > MPI_Bsend(181)..............: MPI_Bsend(buf=0x19c8a06da0, > count=1024, MPI_CHAR, dest=0, tag=0, MPI_COMM_WORLD) failed > MPIR_Bsend_isend(226).......: > MPIR_Bsend_check_active(456): > MPIR_Test_impl(65)..........: > MPIR_Request_complete(234)..: INTERNAL ERROR: unexpected value in > case statement (value=0) > > > Bob Cernohous: (T/L 553) 507-253-6093 > > BobC(a)us.ibm.com > IBM Rochester, Building 030-2(C335), Department 61L > 3605 Hwy 52 North, Rochester, MN 55901-7829 > > > Chaos reigns within. > > Reflect, repent, and reboot. > > Order shall return. > > > > > From: Dave Goodell <goodell(a)mcs.anl.gov> > To: devel(a)mpich.org, > Date: 04/25/2013 11:14 AM > Subject: Re: [mpich-devel] MPI_Bsend under > MPIU_THREAD_GRANULARITY_PER_OBJECT > Sent by: devel-bounces(a)mpich.org > > > > The Bsend paths almost certainly have not been protected correctly. > Patches to fix the issue are most welcome. > > -Dave > > On Apr 25, 2013, at 10:49 AM CDT, Bob Cernohous <bobc(a)us.ibm.com> wrote: > > > Start by saying that I have not been involved in the nitty gritty > of the per-object locking design. > > > > What protects the attached buffer/data structures/request when > doing multithreaded MPI_Bsend()'s? All I see in the code path is a > (no-op) MPIU_THREAD_CS_ENTER(ALLFUNC,). > > > > I have a customer test in which the threads seem to be walking all > over the request around: > > > > /bgusr/bobc/bgq/comm/lib/dev/mpich2/src/mpid/pamid/ > include/../src/mpid_request.h:259 > > 0000000001088c0c MPIR_Request_complete > > /bgusr/bobc/bgq/comm/lib/dev/mpich2/src/mpi/pt2pt/mpir_request.c:87 > > 000000000106e874 MPIR_Test_impl > > /bgusr/bobc/bgq/comm/lib/dev/mpich2/src/mpi/pt2pt/test.c:62 > > 00000000010188f0 MPIR_Bsend_check_active > > /bgusr/bobc/bgq/comm/lib/dev/mpich2/src/mpi/pt2pt/bsendutil.c:455 > > 0000000001018dc0 MPIR_Bsend_isend > > /bgusr/bobc/bgq/comm/lib/dev/mpich2/src/mpi/pt2pt/bsendutil.c:226 > > 0000000001008734 PMPI_Bsend > > /bgusr/bobc/bgq/comm/lib/dev/mpich2/src/mpi/pt2pt/bsend.c:163 > > 00000000010009c0 00000012.long_branch_r2off.__libc_start_main+0 > > :0 > > 000000000130cbc0 start_thread > > > > eg. (fprinting from MPIU_HANDLE_LOG_REFCOUNT_CHANGE) > > > > stderr[8]: set 0x15f8048 (0xac0000ff kind=REQUEST) refcount to 2 > > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to 1 > > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to 0 > > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -1 > > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -2 > > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -3 > > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -4 > > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -5 > > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -6 > > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -7 > > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -8 > > > > > > Bob Cernohous: (T/L 553) 507-253-6093 > > > > BobC(a)us.ibm.com > > IBM Rochester, Building 030-2(C335), Department 61L > > 3605 Hwy 52 North, Rochester, MN 55901-7829 > > > > > Chaos reigns within. > > > Reflect, repent, and reboot. > > > Order shall return. >

1 0

Re: [mpich-devel] MPI_Bsend under MPIU_THREAD_GRANULARITY_PER_OBJECT
by Bob Cernohous 25 Apr '13

25 Apr '13

Here's the big lock failure... looks like we use a request after it's complete/refcount is 0 ... stderr[0]: decr 0x15604b0 (0x44000000 kind=COMM) refcount to 3 stderr[0]: set 0x15d0fe8 (0xac000001 kind=REQUEST) refcount to 2 stderr[0]: incr 0x15604b0 (0x44000000 kind=COMM) refcount to 4 stderr[0]: decr 0x15d0fe8 (0xac000001 kind=REQUEST) refcount to 1 stderr[0]: decr 0x15d0fe8 (0xac000001 kind=REQUEST) refcount to 0 ^^^^^^^^^ stderr[0]: decr 0x15604b0 (0x44000000 kind=COMM) refcount to 3 stderr[0]: decr 0x15d12a0 (0xac000004 kind=REQUEST) refcount to 0 stderr[0]: decr 0x15604b0 (0x44000000 kind=COMM) refcount to 2 stderr[0]: badcase 0x15d0fe8 (0xac000001 kind=0) refcount 0 ^^^^^^^^^ stderr[0]: Abort(1) on node 0 (rank 0 in comm 1140850688): Fatal error in MPI_Bsend: Internal MPI error!, error stack: stderr[0]: MPI_Bsend(181)..............: MPI_Bsend(buf=0x19c8606d70, count=1024, MPI_CHAR, dest=1, tag=0, MPI_COMM_WORLD) failed stderr[0]: MPIR_Bsend_isend(226).......: stderr[0]: MPIR_Bsend_check_active(456): stderr[0]: MPIR_Test_impl(65)..........: stderr[0]: MPIR_Request_complete(239)..: INTERNAL ERROR: unexpected value in case statement (value=0) Bob Cernohous: (T/L 553) 507-253-6093 BobC(a)us.ibm.com IBM Rochester, Building 030-2(C335), Department 61L 3605 Hwy 52 North, Rochester, MN 55901-7829 > Chaos reigns within. > Reflect, repent, and reboot. > Order shall return. devel-bounces(a)mpich.org wrote on 04/25/2013 11:38:24 AM: > From: Bob Cernohous/Rochester/IBM@IBMUS > To: devel(a)mpich.org, > Date: 04/25/2013 11:43 AM > Subject: Re: [mpich-devel] MPI_Bsend under MPIU_THREAD_GRANULARITY_PER_OBJECT > Sent by: devel-bounces(a)mpich.org > > Patches could be tricky because I'm even seeing intermittent (and > different) failures on our 'legacy' libraries which are not per-obj > but use the big lock. So there's probably more than one problem here. > > Abort(1) on node 3 (rank 3 in comm 1140850688): Fatal error in > MPI_Bsend: Internal MPI error!, error stack: > MPI_Bsend(181)..............: MPI_Bsend(buf=0x19c8a06da0, > count=1024, MPI_CHAR, dest=0, tag=0, MPI_COMM_WORLD) failed > MPIR_Bsend_isend(226).......: > MPIR_Bsend_check_active(456): > MPIR_Test_impl(65)..........: > MPIR_Request_complete(234)..: INTERNAL ERROR: unexpected value in > case statement (value=0) > > > Bob Cernohous: (T/L 553) 507-253-6093 > > BobC(a)us.ibm.com > IBM Rochester, Building 030-2(C335), Department 61L > 3605 Hwy 52 North, Rochester, MN 55901-7829 > > > Chaos reigns within. > > Reflect, repent, and reboot. > > Order shall return. > > > > > From: Dave Goodell <goodell(a)mcs.anl.gov> > To: devel(a)mpich.org, > Date: 04/25/2013 11:14 AM > Subject: Re: [mpich-devel] MPI_Bsend under > MPIU_THREAD_GRANULARITY_PER_OBJECT > Sent by: devel-bounces(a)mpich.org > > > > The Bsend paths almost certainly have not been protected correctly. > Patches to fix the issue are most welcome. > > -Dave > > On Apr 25, 2013, at 10:49 AM CDT, Bob Cernohous <bobc(a)us.ibm.com> wrote: > > > Start by saying that I have not been involved in the nitty gritty > of the per-object locking design. > > > > What protects the attached buffer/data structures/request when > doing multithreaded MPI_Bsend()'s? All I see in the code path is a > (no-op) MPIU_THREAD_CS_ENTER(ALLFUNC,). > > > > I have a customer test in which the threads seem to be walking all > over the request around: > > > > /bgusr/bobc/bgq/comm/lib/dev/mpich2/src/mpid/pamid/ > include/../src/mpid_request.h:259 > > 0000000001088c0c MPIR_Request_complete > > /bgusr/bobc/bgq/comm/lib/dev/mpich2/src/mpi/pt2pt/mpir_request.c:87 > > 000000000106e874 MPIR_Test_impl > > /bgusr/bobc/bgq/comm/lib/dev/mpich2/src/mpi/pt2pt/test.c:62 > > 00000000010188f0 MPIR_Bsend_check_active > > /bgusr/bobc/bgq/comm/lib/dev/mpich2/src/mpi/pt2pt/bsendutil.c:455 > > 0000000001018dc0 MPIR_Bsend_isend > > /bgusr/bobc/bgq/comm/lib/dev/mpich2/src/mpi/pt2pt/bsendutil.c:226 > > 0000000001008734 PMPI_Bsend > > /bgusr/bobc/bgq/comm/lib/dev/mpich2/src/mpi/pt2pt/bsend.c:163 > > 00000000010009c0 00000012.long_branch_r2off.__libc_start_main+0 > > :0 > > 000000000130cbc0 start_thread > > > > eg. (fprinting from MPIU_HANDLE_LOG_REFCOUNT_CHANGE) > > > > stderr[8]: set 0x15f8048 (0xac0000ff kind=REQUEST) refcount to 2 > > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to 1 > > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to 0 > > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -1 > > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -2 > > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -3 > > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -4 > > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -5 > > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -6 > > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -7 > > stderr[8]: decr 0x15f8048 (0xac0000ff kind=REQUEST) refcount to -8 > > > > > > Bob Cernohous: (T/L 553) 507-253-6093 > > > > BobC(a)us.ibm.com > > IBM Rochester, Building 030-2(C335), Department 61L > > 3605 Hwy 52 North, Rochester, MN 55901-7829 > > > > > Chaos reigns within. > > > Reflect, repent, and reboot. > > > Order shall return. >

1 0