Hi Ben,

Are you able to give a sudo access on Haswell nodes to Xiamin, Junchao (Pavan's group),
or create a separate hugetlbfs mount for them (see the command below)?

They may need an isolated environment for performance benchmark.
At this point, it seems very few people uses haswell, probably we don't need any action
for them.

- kaz

-------- Forwarded Message --------
Subject: Re: The progress of stencil-
Date: Fri, 9 Jan 2015 14:17:00 -0600
From: woodslong <woodslong@gmail.com>
To: Kazutomo Yoshii <kazutomo@mcs.anl.gov>
CC: Pavan <balaji@anl.gov>, Junchao Zhang <junchao.zhang@gmail.com>, Shigang Li <shigangli.cs@gmail.com>, Yunquan Zhang <yunquan.cas@gmail.com>


Hi, Kaz

The necessary operations to use huge page are shown below(most copy from your email):

$ sysctl vm.hugetlb_shm_group=X # X is the group id for sharing, be sure that my account is the this group

$ sysctl vm.nr_hugepages=Y

$ echo always > /sys/kernel/mm/transparent_hugepage/enabled  (maybe not needed)

$ mkdir -p /mnt/hugetlbfs

$ mount -t hugetlbfs none /mnt/hugetlbfs -o mode=1770,gid=X


Thanks!
                                                               Xiaomin Zhu
 Original Message 
Sender: woodslong<woodslong@gmail.com>
Recipient: Pavan<balaji@anl.gov>
Cc: Kazutomo Yoshii<kazutomo@mcs.anl.gov>; Junchao Zhang<junchao.zhang@gmail.com>; Shigang Li<shigangli.cs@gmail.com>; Yunquan Zhang<yunquan.cas@gmail.com>
Date: Friday, Jan 9, 2015 10:03
Subject: Re: The progress of stencil

Hi, all

Just now, I discussed with Junchao.

We think Cache miss and page fault are the two factors. I will try to verify that. 

The less floating computation for MPI3 is not accurate. Please just ignore it.


Thanks,


                                                               Xiaomin Zhu
 Original Message 
Sender: woodslong<woodslong@gmail.com>
Recipient: Pavan<balaji@anl.gov>
Cc: Kazutomo Yoshii<kazutomo@mcs.anl.gov>; Junchao Zhang<junchao.zhang@gmail.com>; Shigang Li<shigangli.cs@gmail.com>; Yunquan Zhang<yunquan.cas@gmail.com>
Date: Friday, Jan 9, 2015 09:35
Subject: Re: The progress of stencil

OK!  Pavan

I suddenly remembered another issue which affects the performance, that is the cache miss rate.


As the MPI3 version removes the ghost area then changes the data structure of the matrix, as well as the data layout. According with the data access pattern of a grid update, this cause more cache miss if the matrix size happens to be power of 2. In the former profiling, I used 4096*4096 matrix, so the MPI3 version has more cache miss.


Just now, I did another profiling with matrix 5034*5034, and MPI3 shows a better performance (only between MPI1 without huge page and MPI3 without huge page, noncontig), as MPI3 version has less floating point computations and the cache miss should be nearly the same with this random matrix size.

I am sorry I forgot that. I will add more profiling which contains all the 6 versions. If all the mpi3 version showed a better performance, it will be good. 


                                                               Xiaomin Zhu
 Original Message 
Sender: Pavan" <balaji@anl.gov>
Recipient: woodslong<woodslong@gmail.com>
Cc: Kazutomo" <kazutomo@mcs.anl.gov>; Junchao Zhang<junchao.zhang@gmail.com>; Shigang Li<shigangli.cs@gmail.com>; Yunquan Zhang<yunquan.cas@gmail.com>
Date: Friday, Jan 9, 2015 08:09
Subject: Re: The progress of stencil


We should write the paper as-is.  We can say that we identified some issues, but not all.  That's OK.  But we can't say that it's impossible to fix, unless we know what the reason is and can demonstrate that it's impossible to fix.

Sent from my iPad

On Jan 9, 2015, at 6:56 PM, woodslong <woodslong@gmail.com> wrote:

OK, Pavan

Then how do you think about the future work?  

I think if the reason is shared memory mechanism itself, it will be difficult to fix.  Do you have any suggestions on the future work?


                                                               Xiaomin Zhu
 Original Message 
Sender: Pavan" <balaji@anl.gov>
Recipient: Xiaomin Zhu<woodslong@gmail.com>
Cc: Pavan" <balaji@anl.gov>; Kazutomo" <kazutomo@mcs.anl.gov>; Junchao Zhang<junchao.zhang@gmail.com>; Shigang Li<shigangli.cs@gmail.com>; Yunquan Zhang<yunquan.cas@gmail.com>
Date: Friday, Jan 9, 2015 00:55
Subject: Re: The progress of stencil

> On Jan 9, 2015, at 7:45 AM, woodslong <woodslong@gmail.com> wrote:
> 
> 	(3-4) Our aim is to make the shared memory have the same performance with malloc, but it seems impossible as the related overhead is inevitable. 

I'm not sure how you came to this conclusion.

So far, we identified two problems: (1) compiler aliasing, and (2) huge pages.  You are saying that there is at least one more reason that's causing performance degradation that we haven't identified yet.  But that doesn't mean it's *impossible* to fix.

  -- Pavan

--
Pavan Balaji  ✉️
http://www.mcs.anl.gov/~balaji