This project has been created with the correct storage amount and pushed to Blues. -- John Roberts Argonne National Laboratory CELS Systems [email protected] On 6/7/17, 2:22 PM, "allocations-admins on behalf of Bair, Raymond A." <[email protected] on behalf of [email protected]> wrote: P.S. 12TB is OK for storage, since we discussed it with them. However, if they want more later, we need to talk about disk purchase. ----------------------------------------- Ray Bair Argonne National Laboratory On 6/7/17, 2:17 PM, "allocations-admins on behalf of Bair, Raymond A." <[email protected] on behalf of [email protected]> wrote: This is a well written project plan. Let’s approve the 50K for Q3 now and the Committee can consider Q4 when it meets. Ray ----------------------------------------- Ray Bair Argonne National Laboratory On 6/6/17, 4:31 PM, "allocations-admins on behalf of [email protected]" <[email protected] on behalf of [email protected]> wrote: Hello, A new project on the LCRC cluster has been requested. Please forward the information on to the LCRC Allocation sub-committee. Applicant's name: Calem Hoffman Applicant's institution: ANL Applicant's division: Physics Project Name: HELIOS Project title: Real-Time Processing of Digitized Data from the ATLAS User Facility Associated funding: DOE Office of Science - Nuclear Physics Other Systems: None Science: The mission of the ATLAS User Facility is to provide the nuclear physics community with the most intense stable heavy-ion beams, and the associated experimental equipment, to allow for fundamental studies of the structure of nuclei. In particular, our community is focused on answering fundamental questions pertaining to the origin of the elements and how protons and neutrons behave under extreme conditions. The experiments performed at ATLAS range across many fundamental questions, while also touching on practical applications such as isotope production / harvesting and the robustness of materials to radiation. The proposed project aims to increase our data analysis and processing speed by more than ten-times our current capability with our in-division computing power. Utilizing the LCRC resource will allow for a large improvement in the physics output from one of the major user facilities here at ANL, specifically in the field of nuclear structure p hysics. As an initial input into the LCRC computing, we will focus on processing data from the HELIOS Spectrometer and its new data acquisition DigiOS to better understand the single-particle structure of nuclei through transfer reactions. This device has been chosen as it is the precursor to a device currently under design or order $2-4 Million for future use at the future Facility for Rare Isotope Beams under construction on the campus of Michigan State University. Project description: Summary: A major upgrade of the data taking capabilities has recently occurred for three major pieces of experimental equipment affiliated with the Argonne Tandem Linac Accelerator System (ATLAS) Facility located on-site here at Argonne. Specifically, we are now capable of digitizing all signals generated in our detectors systems, hence, not restricting our decision making about types of data collection to the hardware. While creating an opportunity for flexible and increased throughput, and ultimately deeper reaches into the structure of nuclei, it also has the side effect of generating larger amounts of data than our community is accustomed to handling. Therefore, we hope to leverage the availability of the LCRC for processing, both in real-time (eventually), as well as post-processing, data taken here at ANL from ATLAS. Goals: The initial goal of this project for the remainder of FY2017 will be to develop the infrastructure to send, process, and return data taken from actual experiments at ATLAS using the HELIOS Spectrometer and its digital data acquisition (DigiOS). Plan: Utilizing the individual 8000 hours provided by the LCRC, we have benchmarked our current analysis pipeline, and without any modifications to the software have identified improvement factors between 10 - 22 over our current analysis model. Hence, we will first solidify this pipeline within the project and develop some ease of use for real nuclear physics experiments. The current analysis pipeline is as follows: a) Data is collected at HELIOS a rate of ~5 MB / s over a 5 day period approximately once per month. Data is in form of 20 individual files with timestamps. - [~ 2 TB of data storage / month] b) Data must be passed to an LCRC machine. Not in real time at first. A gigabit transfer rate was already demonstrated from our PHY network using GLOBUS. We are pushing hard for a larger (20 gbit pipe in the near future). c) Data files must be inspected and sorted every hour [~20 GB / experiment run hr to process -> 1 core hr / experiment hr for each data sort] Uses a large amount of memory, current software is not developed for parallel processing. First will get this working then develop more parallel software. There are straight forward ways to make parallel however using SWIFT for example as the data can be broken up into ~1 hour “runs”. We have demonstrated that this software can be run on blues. d) Time sorted data must be processed to generate events defined by a time window [~20 GB / experiment run hr to process -> 1 core hr / experiment hr for each data sort on blues]. Again, not presently setup for parallel computing but straightforward to run in batches, eventually can be modified for better use of multiple cores. We have demonstrated that this software can be run on blues. e) Final step is to generate physics results using the ROOT software [~20 GB / experiment run hr to process -> 1 core hr / experiment hr for each data sort]. This software has simple built-in features for using multiple cores on a single node. We have demonstrated that the software and the use of a full complement of cores on a single node is possible with our data. f) Output file from ROOT is small (<500 MB) and will be transferred back to local machines for display and additional analysis. At the beginning we estimate <25% of the jobs will be submitted for single-core use until we develop the proper batch submission software. However, these jobs will not be long in wall clock time (<1 hr each), and they will utilize the large memory capabilities of blues. The development of this software will be the highest priority once the complete analysis pipeline has been established. The information provided in a) above is what we used to determine our requested allocation for the completion of this fiscal year. Namely, 2 experiments spaced out over ~2 months, processing 20 GB / hr for 7 days with 2 TB of storage for the raw data and 6 TB storage for the processed data (time sorting essentially duplicates the data). At present, we need no additional software than has already been installed on blues. We foresee some development of the system through July and we have planned experiments that will utilize the resource starting in Mid-August and through September. We see no roadblocks with scalability or future software needs. The project will start with a limited number of users <10, however, it will scale up in the future when larger experiments make use of the resource. Initial project members will likely include: Calem Hoffman (PHY), Chris Peters (PHY), Torben Lauritsen (PHY), Michael Carpenter (PHY), and Sean Kuvin (UConn / PHY) Industry partnership: None Project URL: http://www.phy.anl.gov/atlas/helios https://wiki.anl.gov/heliosdaq/Main_Page Requested allocation: 300000 Q1: 0 Q2: 0 Q3: 50000 Q4: 250000 Justification: None Storage requirements: We request 12 TB of storage to accommodate one development experiment and two actual experiments along with the duplication of the raw data that takes place following the time sorting. Hence, ~4 TB / experiment through FY2017. The requester has used undetermined amount hours of their initial startup project. In addition to approving an initial amount, please specify a Category and Subcategory for this project. For a list of the current selection of approved categories, please see: https://wiki.lcrc.anl.gov/wiki/Processes/Categories Once the Allocation committee has approved the project, please go to the Project Management page to create it: https://accounts.lcrc.anl.gov/projects.php Thank You, The LCRC Accounts System _______________________________________________ allocations-admins mailing list [email protected] https://lists.lcrc.anl.gov/mailman/listinfo/allocations-admins _______________________________________________ allocations-admins mailing list [email protected] https://lists.lcrc.anl.gov/mailman/listinfo/allocations-admins