Hi Ben, Thanks! (also thank you for helping IPMI!) We're currently interested in an explicit mapping, instead of transparent. Any setting is fine on transparent_hugepage/enabled as long as no other memory-intensive apps are running. - kaz On 01/19/2015 10:41 AM, Allen, Benjamin S. wrote:
Hi Kaz,
I've pushed these changes to thing00-02,04-07 in JLSE:
Added to /etc/fstab:
none /hugetlbfs hugetlbfs mode=1770,gid=501 0 0
Created /etc/sysctl.d/02-hugetlbfs.conf:
vm.hugetlb_shm_group = 501 vm.nr_hugepages = 4096
I did not implement: $ echo always > /sys/kernel/mm/transparent_hugepage/enabled, let me know if this is indeed needed.
Ben
On Jan 12, 2015, at 4:35 PM, Kazutomo Yoshii <[email protected]> wrote:
Hi Ben,
Thank you!
4096 would be enough for now. If you want to set a larger value, of course, no problem :-)
- kaz
On 01/12/2015 04:30 PM, Allen, Benjamin S. wrote:
Hi Kaz,
Started working on this. I'm going to set the two sysctl values by default on the JLSE RHEL7 image, and add a noauto,user entry in fstab. This will let users mount and unmount it as will.
sysctl vm.nr_hugepages=Y This takes a numerical value not "Y":
/proc/sys/vm/nr_hugepages indicates the current number of "persistent" huge pages in the kernel's huge page pool. "Persistent" huge pages will be returned to the huge page pool when freed by a task. A user with root privileges can dynamically allocate more or free some persistent huge pages by increasing or decreasing the value of 'nr_hugepages'.
Please advise on what it should be set to.
Thanks,
Ben
On Jan 9, 2015, at 3:38 PM, Kazutomo Yoshii <[email protected]> wrote:
Hi Ben,
Are you able to give a sudo access on Haswell nodes to Xiamin, Junchao (Pavan's group), or create a separate hugetlbfs mount for them (see the command below)?
They may need an isolated environment for performance benchmark. At this point, it seems very few people uses haswell, probably we don't need any action for them.
- kaz
-------- Forwarded Message -------- Subject: Re: The progress of stencil- Date: Fri, 9 Jan 2015 14:17:00 -0600 From: woodslong <[email protected]> To: Kazutomo Yoshii <[email protected]> CC: Pavan <[email protected]>, Junchao Zhang <[email protected]>, Shigang Li <[email protected]>, Yunquan Zhang <[email protected]>
Hi, Kaz The necessary operations to use huge page are shown below(most copy from your email):
$ sysctl vm.hugetlb_shm_group=X # X is the group id for sharing, be sure that my account is the this group
$ sysctl vm.nr_hugepages=Y $ echo always > /sys/kernel/mm/transparent_hugepage/enabled (maybe not needed) $ mkdir -p /mnt/hugetlbfs
$ mount -t hugetlbfs none /mnt/hugetlbfs -o mode=1770,gid=X
Thanks! Xiaomin Zhu
Original Message
Sender: woodslong<[email protected]> Recipient: Pavan<[email protected]> Cc: Kazutomo Yoshii<[email protected]>; Junchao Zhang<[email protected]>; Shigang Li<[email protected]>; Yunquan Zhang<[email protected]> Date: Friday, Jan 9, 2015 10:03 Subject: Re: The progress of stencil
Hi, all Just now, I discussed with Junchao. We think Cache miss and page fault are the two factors. I will try to verify that. The less floating computation for MPI3 is not accurate. Please just ignore it.
Thanks,
Xiaomin Zhu
Original Message
Sender: woodslong<[email protected]> Recipient: Pavan<[email protected]> Cc: Kazutomo Yoshii<[email protected]>; Junchao Zhang<[email protected]>; Shigang Li<[email protected]>; Yunquan Zhang<[email protected]> Date: Friday, Jan 9, 2015 09:35 Subject: Re: The progress of stencil
OK! Pavan I suddenly remembered another issue which affects the performance, that is the cache miss rate.
As the MPI3 version removes the ghost area then changes the data structure of the matrix, as well as the data layout. According with the data access pattern of a grid update, this cause more cache miss if the matrix size happens to be power of 2. In the former profiling, I used 4096*4096 matrix, so the MPI3 version has more cache miss.
Just now, I did another profiling with matrix 5034*5034, and MPI3 shows a better performance (only between MPI1 without huge page and MPI3 without huge page, noncontig), as MPI3 version has less floating point computations and the cache miss should be nearly the same with this random matrix size.
I am sorry I forgot that. I will add more profiling which contains all the 6 versions. If all the mpi3 version showed a better performance, it will be good.
Xiaomin Zhu
Original Message
Sender: Pavan" <[email protected]> Recipient: woodslong<[email protected]> Cc: Kazutomo" <[email protected]>; Junchao Zhang<[email protected]>; Shigang Li<[email protected]>; Yunquan Zhang<[email protected]> Date: Friday, Jan 9, 2015 08:09 Subject: Re: The progress of stencil
We should write the paper as-is. We can say that we identified some issues, but not all. That's OK. But we can't say that it's impossible to fix, unless we know what the reason is and can demonstrate that it's impossible to fix.
Sent from my iPad
On Jan 9, 2015, at 6:56 PM, woodslong <[email protected]> wrote:
OK, Pavan
Then how do you think about the future work?
I think if the reason is shared memory mechanism itself, it will be difficult to fix. Do you have any suggestions on the future work?
Xiaomin Zhu
Original Message
Sender: Pavan" <[email protected]> Recipient: Xiaomin Zhu<[email protected]> Cc: Pavan" <[email protected]>; Kazutomo" <[email protected]>; Junchao Zhang<[email protected]>; Shigang Li<[email protected]>; Yunquan Zhang<[email protected]> Date: Friday, Jan 9, 2015 00:55 Subject: Re: The progress of stencil
On Jan 9, 2015, at 7:45 AM, woodslong <[email protected] wrote:
(3-4) Our aim is to make the shared memory have the same performance with malloc, but it seems impossible as the related overhead is inevitable. I'm not sure how you came to this conclusion.
So far, we identified two problems: (1) compiler aliasing, and (2) huge pages. You are saying that there is at least one more reason that's causing performance degradation that we haven't identified yet. But that doesn't mean it's *impossible* to fix.
-- Pavan
-- Pavan Balaji ✉️
_______________________________________________ Admins mailing list [email protected] https://lists.jlse.anl.gov/mailman/listinfo/admins