Hello, A new project on the LCRC cluster has been requested. Please forward the information on to the LCRC Allocation sub-committee. Applicant's name: Xianghui Xiao Applicant's institution: ANL Applicant's division: APS Project Name: MBIR_2014_08 Project title: High Performance Supercomputing for Model Based Iterative Reconstruction Associated funding: Multidisciplinary University Research Initiative (MURI) under award AFOSR FA9550-12-1-0458 DOE NNSA Stewardship Science Graduate Fellowship under grant no. DE-FC52-08NA28752 Other Systems: NA Science: Our understanding of the materials science paradigm, that links structure, process, and the properties of a material, has enabled the development of transformative technologies central to modern civilization. However, in order to assess the extraordinary morphological and topological complexity of technologically relevant materials, we require a fully four-dimensional (i.e., time and space resolved) analysis. The interfacial evolution of such structures is at the core of many materials processing techniques, and our ability to characterize these interfaces in 4D provides unprecedented insights into microstructural evolution. Four-dimensional X-ray Tomography (4D-XRT) is nondestructive, thereby enabling the study of the interfacial evolution of morphologically complex materials. In 4D-XRT, a parallel monochromatic beam of X-rays is transmitted through a rotating cylindrical sample; then, the corresponding X-ray projections on the detector plane are used to reconstruct the sample in 3D. Following reconstruction, the data are binarized and meshed, e.g., in Matlab or IDL. Recent advances in the time resolution (0.25s) and spatial resolution (1μm) in 4D-XRT experiments at APS sector 2-BM allow us to investigate the in situ interfacial dynamics in binary alloys, e.g., the Al-Cu system and Al-Si system. Each data set is on the order of 500 GB to 1 TB. Project description: The predominant challenge with 4D-XRT is the growing size of data collected during experiments, rendering manual processing of such data sets impractical. In particular, 4D reconstruction from a collection of X-ray projections involves solving a large-scale inverse problem; tomographic inversion is further challenging due to the nature of detectors and the varying imaging conditions. Therefore, we use a model based iterative reconstruction (MBIR) algorithm that can handle anomalies in the data, e.g., high-energy photons or “zinger” artifacts, without any manual intervention. In MBIR, the reconstruction is typically formulated as the maximum a posteriori (MAP) estimate of the unknowns given the measurable data. The “forward model” p(y|x,ϕ) incorporates the physics of X-ray transmission as well as measurement variations and the presence of zingers; the “prior model” p(x) accounts for nearest-neighbor interactions between voxels. For solving tomographic inverse problems using MBIR the likelihood of forward model and prior model should be maximized. In practice this is done by using Non-Homogeneous Iterative Coordinate Descent, which works by more frequently updating those voxels with a greater need for updates. Furthermore, we use a multi-resolution initialization to speed convergence, which performs reconstructions at coarser resolutions first, in order to initialize the reconstruction at finer resolution. Voxel updates in MBIR are parallelized using both MPI and OpenMP for the case of 3D reconstruction. A spatial “slice” here refers to the 2D plane whose normal is parallel to the axis of rotation. The total number of slices is decomposed across the available nodes. Each node handles a total of eight slices. Boundary information is transmitted from node to node via non-blocking MPI point-to-point communications. Due to the boundary conditions, each node must be allocated no less than four slices. Within each node, using shared memory parallelization, each process handles alternating even and odd slices in the case of 3D reconstruction, or alternating time-steps for 4D reconstruction. Performance of 3D MBIR has been evaluated on the Edison cluster at the National Energy Research Scientific Computing Center; compute nodes contain Intel “Ivy Bridge” processors rated at 2.4 GHz. The approximate walltime scales linearly with the number of nodes and the number of slices. The results suggest that strong and weak efficiencies are approximately 100% up to 256 slices. Scaling tests are well underway for larger datasets involving 1024 slices, as of July 2014. It should also be noted that the walltime also scales linearly with the number of projections. MBIR is a promising algorithm that involves X-ray attenuation modeling and incorporates non-idealities in the measurement system, enabling high spatial resolution reconstruction. Nevertheless, in general, reconstruction may take days if a few thousand slices and several hundred time-steps are required, using only one or two nodes. As such, reconstruction is currently the rate-limiting step in our data-driven discovery. We would like to have our data reconstructed on-site via MBIR so that we can evaluate the experiment progress, and make changes as necessary during beam-time. In this way, reconstruction can keep pace with the rapid rate of data collection, during our weeklong experiments starting on August 16, 2014. In terms of data storage, since each dataset is on the order of 500 GB to one TB, three TB of storage is requested on the supercomputer such that we can reconstruct multiple data sets concurrently. We have following estimation of core hours. (assume 16 cores/node for Intel Sandy Bridge in LCRC) * (8 nodes to reconstruct several slices in z) * (10 hours wall time for medium resolution reconstruction) * (100 different runs) + (5000 hours development) = 133000 core-hours Project URL: Requested allocation: 133000 Q1: 0 Q2: 0 Q3: 0 Q4: 133000 Justification: The requester has used undetermined amount hours of their initial startup project. In addition to approving an initial amount, please specify a Category and Subcategory for this project. For a list of the current selection of approved categories, please see: https://wiki.lcrc.anl.gov/wiki/Processes/Categories Once the Allocation committee has approved the project, please go to the Project Management page to create it: https://accounts.lcrc.anl.gov/projects.php Thank You, The LCRC Accounts System