On Sun, Sep 22, 2013 at 08:41:21PM -0500, Jaln wrote:
> Interesting, Thanks Rob,
> So I can assume the Hopper( a cray XE6 with MPT 3.2) contains the
> lustre-specific optimizations?
Hopper does. Just to be clear, don't ask for 3.2: run the newest
possible MPT.
You'll have to experiment a bit with the settings
described in the intro_mpi man page. In my experience, things that
are documented as the default are not actually the default. It's
quite frustrating, but the hopper staff is quite good at answering
site-specific MPI-IO questions.
> Does it work both for read and write?
I don't know for sure (it's a Cray, and I have not see the source
code), but most of the focus was on writes since in the write path
there are lustre-level locks to deal with. In the read path,
lustre-level locks don't really come into play.
==rob
> Jailin
>
>
> On Sun, Sep 22, 2013 at 2:00 PM, Rob Latham <robl(a)mcs.anl.gov> wrote:
>
> > On Sat, Sep 21, 2013 at 11:21:19PM -0500, Jaln wrote:
> > > Hi everyone,
> > >
> > > I'm not sure, whether the lustre or the MPI forum is the right place for
> > my
> > > question.
> >
> > both, i guess :>
> >
> > > The question is about the ROMIO optimization on Lustre,
> > > In one SC'08 paper,
> > > http://users.eecs.northwestern.edu/~wkliao/PAPERS/fd_sc08_revised.pdf<
> > https://mail.ttu.edu/owa/redir.aspx?C=yUmbVUH4hUWLFEWFA2GcoiKOEhnhitAIatZfG…
> > >
> > > , it's said that the way ROMIO assigns the file domains to I/O
> > aggregators
> > > will not make two aggregators access the same OST.
> > >
> > > In my understanding, this means, the data locality on Lustre layer has
> > been
> > > taken care of in the ROMIO, such that the aggregators will not
> > > have competition on the same OST.
> > >
> > > My question is "is this optimization used in all current lustre system,
> > > e.g., Hopper at NERSC?"
> >
> > Wei-keng never contributed the specific ROMIO optimizations he discussed in
> > the SC 08 paper, but his work did spur a lot of community discussion
> > and contributions.
> >
> > Emoly Lu contributed a bunch of Lustre ADIO driver work, which Pascal
> > Deveze and Martin Pokorny improved upon. MPICH-1.3 and newer contain
> > these improvements.
> >
> > David Knaak from Cray implemented his own improvements. Cray's MPI-IO
> > is based on ROMIO but the cray modifications are proprietary. MPT-3.2
> > and newer contain lustre-specific optimizations.
> >
> > The community has been quiet with respect to Lustre MPI-IO work since
> > then. I hope that's because everything "just works".
> >
> > ==rob
> >
> > --
> > Rob Latham
> > Mathematics and Computer Science Division
> > Argonne National Lab, IL USA
> >
>
>
>
--
Rob Latham
Mathematics and Computer Science Division
Argonne National Lab, IL USA
Interesting, Thanks Rob,
So I can assume the Hopper( a cray XE6 with MPT 3.2) contains the
lustre-specific optimizations?
Does it work both for read and write?
Jailin
On Sun, Sep 22, 2013 at 2:00 PM, Rob Latham <robl(a)mcs.anl.gov> wrote:
> On Sat, Sep 21, 2013 at 11:21:19PM -0500, Jaln wrote:
> > Hi everyone,
> >
> > I'm not sure, whether the lustre or the MPI forum is the right place for
> my
> > question.
>
> both, i guess :>
>
> > The question is about the ROMIO optimization on Lustre,
> > In one SC'08 paper,
> > http://users.eecs.northwestern.edu/~wkliao/PAPERS/fd_sc08_revised.pdf<
> https://mail.ttu.edu/owa/redir.aspx?C=yUmbVUH4hUWLFEWFA2GcoiKOEhnhitAIatZfG…
> >
> > , it's said that the way ROMIO assigns the file domains to I/O
> aggregators
> > will not make two aggregators access the same OST.
> >
> > In my understanding, this means, the data locality on Lustre layer has
> been
> > taken care of in the ROMIO, such that the aggregators will not
> > have competition on the same OST.
> >
> > My question is "is this optimization used in all current lustre system,
> > e.g., Hopper at NERSC?"
>
> Wei-keng never contributed the specific ROMIO optimizations he discussed in
> the SC 08 paper, but his work did spur a lot of community discussion
> and contributions.
>
> Emoly Lu contributed a bunch of Lustre ADIO driver work, which Pascal
> Deveze and Martin Pokorny improved upon. MPICH-1.3 and newer contain
> these improvements.
>
> David Knaak from Cray implemented his own improvements. Cray's MPI-IO
> is based on ROMIO but the cray modifications are proprietary. MPT-3.2
> and newer contain lustre-specific optimizations.
>
> The community has been quiet with respect to Lustre MPI-IO work since
> then. I hope that's because everything "just works".
>
> ==rob
>
> --
> Rob Latham
> Mathematics and Computer Science Division
> Argonne National Lab, IL USA
>
--
Genius only means hard-working all one's life
On Sat, Sep 21, 2013 at 11:21:19PM -0500, Jaln wrote:
> Hi everyone,
>
> I'm not sure, whether the lustre or the MPI forum is the right place for my
> question.
both, i guess :>
> The question is about the ROMIO optimization on Lustre,
> In one SC'08 paper,
> http://users.eecs.northwestern.edu/~wkliao/PAPERS/fd_sc08_revised.pdf<https://mail.ttu.edu/owa/redir.aspx?C=yUmbVUH4hUWLFEWFA2GcoiKOEhnhitAIatZfG…>
> , it's said that the way ROMIO assigns the file domains to I/O aggregators
> will not make two aggregators access the same OST.
>
> In my understanding, this means, the data locality on Lustre layer has been
> taken care of in the ROMIO, such that the aggregators will not
> have competition on the same OST.
>
> My question is "is this optimization used in all current lustre system,
> e.g., Hopper at NERSC?"
Wei-keng never contributed the specific ROMIO optimizations he discussed in
the SC 08 paper, but his work did spur a lot of community discussion
and contributions.
Emoly Lu contributed a bunch of Lustre ADIO driver work, which Pascal
Deveze and Martin Pokorny improved upon. MPICH-1.3 and newer contain
these improvements.
David Knaak from Cray implemented his own improvements. Cray's MPI-IO
is based on ROMIO but the cray modifications are proprietary. MPT-3.2
and newer contain lustre-specific optimizations.
The community has been quiet with respect to Lustre MPI-IO work since
then. I hope that's because everything "just works".
==rob
--
Rob Latham
Mathematics and Computer Science Division
Argonne National Lab, IL USA
Hi everyone,
I'm not sure, whether the lustre or the MPI forum is the right place for my
question.
The question is about the ROMIO optimization on Lustre,
In one SC'08 paper,
http://users.eecs.northwestern.edu/~wkliao/PAPERS/fd_sc08_revised.pdf<https://mail.ttu.edu/owa/redir.aspx?C=yUmbVUH4hUWLFEWFA2GcoiKOEhnhitAIatZfG…>
, it's said that the way ROMIO assigns the file domains to I/O aggregators
will not make two aggregators access the same OST.
In my understanding, this means, the data locality on Lustre layer has been
taken care of in the ROMIO, such that the aggregators will not
have competition on the same OST.
My question is "is this optimization used in all current lustre system,
e.g., Hopper at NERSC?"
Thanks,
Jailin Liu
Ph. D student
Texas Tech University
You can create shared memory in regular MPI as well. I'm not sure you need HMPI extensions for this.
-- Pavan
On Aug 29, 2013, at 1:59 PM, Huiwei Lu wrote:
> Thank you Shigang. Your links are very helpful.
>
> Sorry Muhammad I didn't answer your question at the first time. I have just read about HMPI briefly several days ago.
>
>
> Huiwei Lu
> http://www.mcs.anl.gov/~huiweilu/
>
>
> On Thu, Aug 29, 2013 at 1:53 PM, Shigang Li <shigangli.cs(a)gmail.com> wrote:
> Hi Muhammad, Huiwei,
>
> There is a Hybrid MPI(HMPI) library which aims at implementing MPI efficiently on both inter-node network and shared memory architecture. Here are some papers talking about this, which may give you some info.
>
> Ownership Passing: Efficient Distributed Memory Programming on Multi-core Systems
> NUMA-Aware Shared Memory Collective Communication for MPI
> Hybrid MPI: Efficient Message Passing for Multi-core Systems
>
> Best Regards,
> Shigang
>
>
> On Thu, Aug 29, 2013 at 12:31 PM, Muhammad Ansar Javed <muhammad.ansar(a)seecs.edu.pk> wrote:
> Hi Huiwei,
> Thank you so much for suggesting reading material. These are really useful papers but I am looking for models that provide seamless integration of two devices (Network device and Shared Memory device) into one hybrid device. Message passing mechanism should be exactly same for end user at MPI layer.
> Idea is to use to use Network Device for establishing network communication between nodes and Shared memory device should be responsible for Intra-node communication on each multi-core machine. Shared Memory device is implemented using Multi-threading approach. Both Network and Shared Memory device are in working state independently.
>
> I hope it explains what I am really looking for. Please suggest papers that can provide guidelines for achieving this goal.
>
>
>
> On Wed, Aug 28, 2013 at 7:40 PM, Huiwei Lu <huiweilu(a)mcs.anl.gov> wrote:
> Hi, Muhammad,
>
> As I am also starting to exploit this area, I would like to share with you some of my readings.
>
> Following are a few papers to help you start:
> 1. Early Experiments with the OpenMP/MPI Hybrid Programming Model, http://www.mcs.anl.gov/~lusk/hybridpaper.pdf
> 2. Test Suite for Evaluating Performance of MPI Implementations That Support MPI THREAD MULTIPLE, http://www.mcs.anl.gov/~thakur/papers/thread-tests.pdf
>
> In the second paper, you can also find some code examples for using hybrid programming model.
>
> Best,
>
> Huiwei Lu
> http://www.mcs.anl.gov/~huiweilu/
>
>
> On Wed, Aug 28, 2013 at 12:54 AM, Muhammad Ansar Javed <muhammad.ansar(a)seecs.edu.pk> wrote:
> Hi,
> I am looking for message passing models for Hybrid Parallelism that are being used in MPICH implementations for the Cluster of Multi-core machines. Can you please suggest me some papers or study material that can explain communication models that incorporate network communication as well as shared memory communication.
>
> Thanks,
> Ansar
>
>
>
>
> --
>
> Regards:
>
> Muhammad Ansar Javed
> NUST SEECS
> Contact: +92 334 438 9394
> Email: muhammad.ansar(a)seecs.edu.pk
>
>
--
Pavan Balaji
http://www.mcs.anl.gov/~balaji
