|
|
Oasis Of
Data In The SAHARA
by Richard Fry and Luca Delle Monache, Ph.D. Defense Threat Reduction Agency's Chemical and Biological Technologies Department September 29, 2018 Ninety percent of all data was created in the last two years — and that percentage continues to grow as the world becomes more interconnected and researchers discover the power of big data. Much like the Sahara Desert, Big data is massive and unforgiving. Navigating and deciphering through the more than 2.5 quintillion data bytes generated each day can be difficult. Researchers from the National Center for Atmospheric Research (NCAR), managed by the Defense Threat Reduction Agency’s Chemical and Biological Technologies Department, have addressed this issue with the development of the Self-Organizing Map (SOM) – Assisted Hazard Area Risk Analysis (SAHARA) tool, which reduces large climate datasets into manageable sizes while accurately modeling chemical, biological, radiological and nuclear (CBRN) events.  ----------------------------------------------------------- Evaluating the effects of CBRN releases requires lengthy records of meteorological conditions that represent the climatic variability of a given location. As such, the computational and data storage requirements are ample. Sifting through this much information is time-consuming; however, crisis management and disaster planning require rapid response to ensure recovery and continued mission. Arming warfighters and first responders with accurate and timely information, SAHARA integrates meteorological data, a weather forecast model, an atmospheric transport and dispersion model, the SOM algorithm and plotting routines into a single user interface. Researchers demonstrated the scalability of SAHARA on systems ranging from high-end, high-performance computing centers such as multi-core central processing units and multi-node computing clusters, down to resource-constrained laptops with a single processor and limited data storage. One component of the software, the SOM algorithm, is an approach based on a neural network to identify patterns in a large-dimensional dataset. Similar to “cluster” methods used in climatological studies, and machine-learning tasks such as handwriting and facial recognition, the algorithm has two primary phases: training and mapping. In the training phase, meteorological data generates a matrix of related nodes, corresponding to the dominant patterns of the data. For CBRN applications, meteorological data includes near-surface winds, vertical profiles of temperature and moisture in the planetary boundary layer. During the mapping phase, researchers assign variables at each step to their closest representative vector, ranking them by how well they match. Each match is treated as a distribution and days are sampled to generate a snapshot of prevailing conditions. In validation and verification studies in two CBRN scenarios, SAHARA mapped one month of climate data per year for the past 30 years and reduced it to 150 typical days. To estimate the skill of this approach, the measure of effectiveness metric was used to compare the reduced data set and the full 30-year climatology. Researchers found that the climate subsets fell within 85-90 percent overlap with the full set while using only 15 percent of the input data, representing significant computational and data storage savings. Bridging massive data storage and resource-intensive atmospheric modeling capabilities with operational constraints, SAHARA enables warfighters and first responders to react rapidly and improve decision-making capability during CBRN events. This innovative solution improves focus on combat support and enhances readiness in efforts to build a more lethal force. |
|
