Hello, A new project on the LCRC cluster has been requested. Please forward the information on to the LCRC Allocation sub-committee. Applicant's name: Eric Schwenker Applicant's institution: ANL Applicant's division: NST Project Name: DeepMic Project title: Deep Learning Atomistic Structure from Microscopy Images Associated funding: LDRD Other Systems: JLSE, Carbon (100,000 core-hours/year), NERSC (250,000 core-hours/year) Science: In the present work, we are developing a framework for experimentally-driven atomic structure optimization by combining atomistic modeling with electron microscopy image simulation and deep learning. Existing studies that integrate deep learning techniques into microscopy treat content analysis as a general classification problem, attempting to assign classification labels such as austenitic, ferritic, or martensitic, for example, to a micrograph of steel. While this provides a useful framework for organizing large datasets of specific microscopy images, it fails to use the content in a generative way (i.e. to inform model creation or update an iterative optimization scheme). In order to use atomic resolution images for such tasks, it is critical that image feature detection be carried out in a robust manner. Currently the image features are extracted using a hand-crafted peak detection methods, which limits the types of microscopy images that thi s representation can be applied to. A wealth of open source deep learning approaches to automatic feature extraction exist, and we believe they could be leveraged to automate microscopy interpretation in the design of physical models and microscopy image analysis and retrieval across large datasets. Project description: Deep neural networks (DNN’s) have been shown to be able to capture high-level features describing image data, even when trained on datasets of unrelated objects (i.e. faces or natural scenes). This is attributed to the notion that the features learned are fundamental to visual recognition, and are thus transferable across all images in the general sense. In this project we propose to leverage pre-trained DNN architectures such as YOLO [J. Redmon, CVPR (2016)], DeepMask and SharpMask [P. Pinheiro et al. NIPS (2015), ECCV (2016)], and inception-v3 [C. Szegedy, arxiv (2015)] (pre-trained for other recognition tasks) to test their accuracy in identifying the atomic columns present over a dataset of several thousand simulated atomic resolution microscopy images that are subject to varying levels of synthetic imaging noise. Additionally, we will use these pre-trained DNN architectures, as the basis for “fine-tuning” new feature detection models on si mulated microscopy images. Our (Eric Schwenker, Maria Chan) previous efforts in this area have led to the development of a framework for matching atomic resolution microscopy images. There is a noticeable decline in matching performance as the images become increasingly corrupted with noise and blur. The belief is that the matching performance is not significantly hindered by the metric used for matching, but rather the inability to consistently identify the atomic columns across all the degraded images. Our goal is to use the automatic feature extraction tools from deep learning to improve the image matching framework, and then show that this can be applied to both experimentally-driven atomic structure optimization, and across different imaging modalities to quantify similarities between images. The breakdown of the required calculations is as follows: (1) Image simulation and labelling for a complete set of ICSD prototypes: Currently, the Inorganic Crystal Structure Database (ICSD) contains ~9,000 unique crystal structure prototypes. We will construct an initial set size of ~36,000 pristine simulated microscopy images (~ 4 images per structure prototype) where the image peaks (atomic columns) are labeled automatically based on the projection of the structure on the imaging plane. In addition, we will create 2 complete datasets of corresponding image matches that are corrupted with varying levels of simulated imaging noise. With this, the entire dataset will total (9000 x 4) x 3 = 108,000 images. To simulate these images, a new electron scattering simulation algorithm (PRISM) [C. Ophus, Adv. Struct. Chem. Imaging (2017)], will be used. With the settings we have determined for the algorithm, a single image requires 2 hours of simulation time on a single core. With ~108,000 images, these calculations estimated to take 216,000 core hours. An additional 28,000 core hours will be used to evaluate the accuracy of 3 separate pre-trained DNN detection models. TOTAL CALCULATION TIME: 244,000 core hours (2) Fine-tuning pre-trained DNN’s on ICSD dataset: Training DNN’s from scratch is a laborious process that can take several weeks on modern computing platforms. To avoid this, we will “fine-tune” the existing pre-trained DNN architectures using their respective open-source computing frameworks: darknet (http://pjreddie.com/darknet) for YOLO, Torch (http://torch.ch/) for Deep/SharpMask, and TesorFlow (https://www.tensorflow.org/) for inception-v3 . This involves adjusting the weights of the later layers that are more specific to the details of the microscopy images themselves. For this, we will allot ~ 60,000 core hours for each of the three existing DNN architectures explored. TOTAL CALCULATION TIME: 180,000 core hours (3) Image matching for atomic resolution datasets: The current image matching framework is setup to test matching performance on a dataset of ~25,000 simulated microscopy images A full performance run of a single DNN detector over the entire set will require ~20,000 core hours. The 3 pre-trained DNN detectors, along with the 3 fine-tuned DNN’s will require 6 x 12,000 = 72,000 core hours to evaluate. TOTAL CALCULATION TIME: 120,000 core hours Industry partnership: Project URL: Requested allocation: 496000 Q1: 124000 Q2: 124000 Q3: 124000 Q4: 124000 Justification: Storage requirements: The requester has used undetermined amount hours of their initial startup project. In addition to approving an initial amount, please specify a Category and Subcategory for this project. For a list of the current selection of approved categories, please see: https://wiki.lcrc.anl.gov/wiki/Processes/Categories Once the Allocation committee has approved the project, please go to the Project Management page to create it: https://accounts.lcrc.anl.gov/projects.php Thank You, The LCRC Accounts System