I understand the issue, but the problem is what you want to do if the symmetric allocation isn't feasible - do you permit the allocation or do you fail? One option would be an info that required symmetric allocation or failure; this wouldn't prevent the use of shared memory when symmetric allocation wasn't feasible (modulo the discussion about whether info is always a hint or can be a requirement). Bill On Nov 6, 2010, at 11:18 AM, James Dinan wrote:

Hi Bill,

The problem with favoring one, but allowing both is that the representation for pointers changes from symmetric to non-symmetric allocations. In the non-symmetric case, direct pointers are not valid on every process so you have to use displacements instead. In a linked list, for example, you'd store a displacement in next instead of the direct next pointer:

elem_t *next = current->next + ((uint8_t*) segment_base);

versus symmetric:

elem_t *next = current->next;

This means that next has a different type in symmetric vs non- symmetric and will be dereferenced differently - it can get messy supporting both pointers and displacements in the same code. Portable programs would most likely go for the least common denominator and assume non- symmetric so any effort we invest in supporting symmetric allocation would be wasted.

This isn't a huge deal to me, you can easily put together a few macros to make non-symmetric allocations easier to live with. I'm willing to pass on this if we don't feel we can guarantee that it will always be symmetric. This just popped out as a potential opportunity for us to improve on one of the hassles of conventional shared segments.

~Jim.

PS- CC'd back to the Hybrid PM list, I think Ron and others might be interested in the discussion.

On 11/6/10 10:45 AM, William Gropp wrote:

...

Agreed, though having a hint for the implementation would be helpful. There are cases where it may be very hard to deliver the guarantee, as it depends on the allocation history of each process. For CAF-like applications with no allocated data, providing symmetric allocation is (usually) easy. When applications have been running a dynamic mesh refinement application for a few hours, it could get hard :) . Thus, even having a requirement might not work unless your address space is so big (including the capabilities of your paging and swap) that you can always find unallocated addresses in common. More precisely, we can't require that symmetric allocations work, so the application must either be prepared to accept non-symmetric or failure from the allocator. Adding a hint requesting symmetric allocation and an advice to implementors stating that high-quality implementations are expected to pay attention to the hint might be sufficient.

Bill

On Nov 6, 2010, at 10:37 AM, Pavan Balaji wrote:

...

Bill,

On 11/06/2010 08:45 AM, William Gropp wrote:

...

On the specific question, one could use an info hint to request symmetric allocation.

I believe Jim is looking for a "guarantee" on symmetric allocation, which the info argument cannot give as the MPI implementation is allowed to ignore it.

It's possible for the application to check if all the base pointers returned are symmetric or not, and decide what to do after that. But it cannot rely on the fact that it'll always get a symmetric allocation (unless we change the function prototype and add a parameter for it).

-- Pavan

William Gropp Deputy Director for Research Institute for Advanced Computing Applications and Technologies Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign