High performance computing

Map of the eastern Indian Ocean and surrounding regions. Location of the drilling expedition and the Sunda subduction zone also shown. The Indo-Australian plate subducts beneath the Eurasian plate at the subduction zone and it was the source of the 2004 earthquake and tsunami offshore Sumatra to Andaman Islands (rupture area shaded in yellow). Ocean drilling boreholes are red dots (U1480, U1481). The Bengal and Nicobar submarine fans are fed by river sediments eroded from the Himalaya and Tibetan Plateau, creating very large thicknesses of sediment. (Credit: Lisa McNeill, University of Southampton.)
Map of the eastern Indian Ocean and surrounding regions. The Indo-Australian that plate was the source of the 2004 Sumatra earthquake and tsunami subducts beneath the Eurasian plate at the subduction zone (rupture area shaded in yellow). Ocean drilling boreholes are red dots (U1480, U1481).  (Credit: Lisa McNeill, University of Southampton.)

An international team of scientists has found evidence suggesting the dehydration of minerals deep below the ocean floor influenced the severity of the Sumatra earthquake, which took place on December 26, 2004, off the west coast of Indonesia.

The magnitude 9.2 earthquake and subsequent tsunami devastated coastal communities of the Indian Ocean, killing over 250,000 people.

Research into the earthquake was conducted during a scientific ocean drilling expedition to the region August through October 2016 as part of the International Ocean Discovery Program (IODP). Expedition 362 was led by researchers from Colorado School of Mines and the University of Southampton in collaboration with IODP scientist Katerina Petronotis.

On board the research vessel JOIDES Resolution, the researchers sampled, for the first time, sediments and rocks from the oceanic tectonic plate that feeds the Sumatra subduction zone. A subduction zone is an area where two of the Earth’s tectonic plates converge, one sliding beneath the other, generating the largest earthquakes on Earth, many with destructive tsunamis.

Findings of a study on sediment samples found far below the seabed are now detailed in a new paper authored by Dr. Andre Hüpers of the MARUM-Center for Marine Environmental Sciences at University of Bremen and published in the journal Science. Colorado School of Mines Associate Professor of Geophysics Brandon Dugan was one of the study’s coauthors and coleader of Expedition 362.

“It raised a lot of questions, because that wasn't a place in the world where we thought a magnitude 9 earthquake would occur,” said Dugan.

Expedition coleader Professor Lisa McNeill of the University of Southampton said “the 2004 Indian Ocean tsunami was triggered by an unusually strong earthquake with an extensive rupture area.” By unearthing the cause of such a large earthquake and tsunami, the scientists hope to be able to assess potential hazards in other regions with similar geological properties.

The scientists concentrated their research on a process of dehydration of sedimentary minerals deep below the ground, which usually occurs within the subduction zone. It is believed this dehydration process, which is influenced by the temperature and composition of the sediments, normally controls the location and extent of slip between the plates, and therefore the severity of an earthquake.

Expedition leaders from left: Lisa McNeill, Brandon Dugan, Katerina Petronotis.
Expedition leaders from left: Lisa McNeill, Brandon Dugan, Katerina Petronotis. (Photo credit: Tim Fulton, IODP JRSO.)

The Sumatra research team used the latest advances in ocean drilling to extract samples from 1.5 km below the seabed, taking measurements of sediment composition including chemical, thermal and physical properties.

At a certain depth, the researchers identified a layer where the water had lower salinity than the overlying and underlying sediment. This evidence of freshwater suggests that the water must have been released from within minerals in the sediment, as ocean water would have been high in salinity.

The researchers found that the sediments on the ocean floor, eroded from the Himalayan mountain range and Tibetan Plateau and transported thousands of kilometers by rivers on land and in the ocean, were subjected to geologic processes over millions of years. These sediments formed a sort of thick shell over minerals far below the seabed, causing chemical transformations within the subsurface.

 A 'free-fall funnel', part of the drilling process.(Photo Credit: Tim Fulton, IODP JRSO)

 A 'free-fall funnel', part of the drilling process.
Credit:Tim Fulton, IODP JRSO.)

These transformations caused the mineral bed to heat, pushing freshwater out of the mineral crystals up through the sediment layers.

At first, this water would have softened the sediment, actually decreasing the risk of a big earthquake by allowing it to absorb more force, Dugan explained. However, as the sediment moved closer to the fault over millions of years, the water flowed away, leaving the sediment dehydrated and brittle—the perfect setup for a megaquake.

The scientists ran simulations to calculate how the Sumatra sediments (currently not yet to the fault) would behave once they had traveled 250 km to the east toward the subduction zone and been buried significantly deeper. The simulations showed the sediment reaching higher temperatures, thus supporting their findings.

Hüpers said that the findings suggest that other subduction zones with thick and hotter sediment and rock could also experience this phenomenon.

“The 2004 Sumatra and 2011 Tohoku earthquakes made us reexamine our understanding of large earthquakes,” said Dugan. “This new analysis extends our knowledge of the conditions that can contribute to large earthquakes that generate tsunamis. We now can assess the potential for megaquakes in subduction margins with limited or no historical earthquake record.”

Subduction zone earthquakes typically have a return time of a few hundred to a thousand years, so applying this research to similar geological regions will allow scientists to better predict these hazards.

Similar subduction zones exist in the Caribbean (Lesser Antilles), off Iran and Pakistan (Makran), and off the western United States and Canada (Cascadia). The team will continue research on the samples and data obtained from the Sumatra drilling expedition over the next few years, including laboratory experiments and further numerical simulations, and will use their results to assess the potential future hazards both in Sumatra and at these comparable subduction zones.

Agata Bogucka, Communications Manager, College of Earth Resource Sciences & Engineering | 303-384-2657 | abogucka@mines.edu
Mark Ramirez, Managing Editor, Communications and Marketing | 303-273-3088 | ramirez@mines.edu


Mines Mechanical Engineering faculty are part of a project that has been awarded $1.6 million to improve the design of soft magnetic materials, which would increase the efficiency of electrical systems. Mechanical Engineering professors Aaron Stebner and Cristian Ciobanu are collaborating with faculty at Case Western Reserve University and the University of Minnesota on the project, titled “Accelerated Soft Magnetic Alloy Design and Synthesis Guided by Theory and Simulations.”

Soft magnetic materials, such as iron and nickel-iron alloys, are labeled soft due to their low loss and high permeability characteristics. They are essential to all aspects of daily life, especially in the generation, conversion and conditioning of electric power. Even small improvements in energy efficiency from the magnetic components can result in large energy, financial and CO2 emissions savings. Stebner and Ciobanu plan to advance the alloy design cycle through a multidisciplinary approach involving materials science, applied physics and electrical engineering.  

Graph showing the iterative process in creating a database of alloy properties

“The research in this grant is aimed at creating an ability to computationally explore new chemistries and microstructures for soft magnetic materials,” said Stebner. “By developing computations that can assist with evaluating new chemistries in the computer instead of in the lab, that process could be reduced to five years or less, and we could also discover well-performing magnetic materials that metallurgists have not thought of before.”

As technological advances lead to soft magnetics operating at higher frequencies and higher temperatures, the challenge becomes retaining their inherent strengths. Stebner explained, “If we think about our power grids, they operate at 60-120 Hz and near room temperature. Looking to applications like the new hybrid-electric airplane program announced by NASA, the goal is for the energy conversion systems to operate at the same efficiencies as our power grids, but at 50-100 kHz and temperatures up to 400 C.”

Ciobanu’s research will focus on the density functional theory, while Stebner’s team will investigate fine element modeling in micromagnetics. The $1.6 million grant award is funded by the National Science Foundation’s Designing Materials to Revolutionize and Engineer Our Future program. Figure 2, below, captures the experimental and computational collaboration between Mines faculty, Matthew Willard from Case Western’s Department of Materials Science and Engineering and Richard James from the University of Minnesota’s Department of Aerospace Engineering and Mechanics. 

Workflow chart showing collaborative process of alloy processing to thermodynamic modeling and micromagnetic modeling on the computational end as well as experimental models looking at structural and magnetic characterizations


Deirdre Keating, Communications Manager, College of Engineering & Computational Sciences | 303-384-2358 | dkeating@mines.edu
Mark Ramirez, Communications Manager, College of Applied Science & Engineering | 303-384-2622 | ramirez@mines.edu

This is a story of humans and hardware. What happened when professors, tops in their different fields of energy research, gained campus access to a world-class supercomputer?

The story began four years ago with the institutional vision to bring a supercomputer named Ra to Mines. Dag Nummedal, director of the Colorado Energy Research Institute, and Physics Professor Mark Lusk had been working to acquire a supercomputer, and when Vice President of Research and Technology Transfer John Poate got involved, “the idea resonated campus-wide,” said Lusk.

With the horsepower of Mines leadership behind this well-timed initiative, it became a commitment to much more than hardware. Five years, one supercomputer, 10 new faculty hires, 15 classes, 60 PhD students and 120 journal publications since that original vision, Mines has become a global leader in computationally guided energy science research.

The initiative has made a huge impact on research volume, led to many important discoveries, and catalyzed interdisciplinary collaboration across campus. More than 90 percent of Mines’ academic departments are pursuing projects supported by the Golden Energy Computing Organization. As scientists from different areas come together, ideas begin to cross-fertilize and surprising synergies emerge. As Lusk notes, “Once all these people start working together under one virtual roof, good things happen.”

One of the new supercomputer hires was Amadeu Sum, a professor in the Chemical and Biological Engineering Department and co-director of Mines’ Center for Hydrate Research. He used Ra to explain the nucleation and growth of hydrates, and his work landed on the cover of Science.

On another side of campus, REMRSEC, the Renewable Energy Materials Research Science & Engineering Center, was chosen for funding by the National Science Foundation in part due to the computing power Ra could bring to the table. 

Lusk’s solar cell work with REMRSEC on multiple-exciton generation (MEG) was successful. MEG theorizes it is possible for an electron that has absorbed light energy to transfer some of that energy to other electrons, resulting in more electricity from the same amount of absorbed light.

In a cross-fertilizing leap, REMRSEC decided to look at hydrates as a way to store hydrogen. The center provided seed money to Carolyn Koh, a professor in the Chemical and Biological Engineering Department, to lead a combined team of REMRSEC solar energy scientists and Center for Hydrate Research experts. Together they developed a computer analysis to assess the potential of hydrates for hydrogen storage.

Then they began thinking of other materials that can be assembled into the cage-like clathrates. Could they build a silicon clathrate structure to store hydrogen? Using experimental facilities at the National Renewable Energy Laboratory (NREL), they determined the answer was yes.

The synergy continued. What are the photovoltaic properties of these new silicon clathrates? Can they be used to build a better solar cell? The answer, once again, appears to be yes.

In summary, hydrate engineers and solar energy physicists have founded two new facets of energy research because a world-class  supercomputing facility came to Mines. “The successes that have come from our original vision have been snapping together a Lego™ at a time,” said Lusk. “We have a cool system going now and there’s no end in sight.”


A New Season

Now plans are underway to purchase a new machine to become the campus flagship for high-performance computing, with Ra maintained as a set of smaller clones for less demanding projects and student training. Requests for bids have gone out to industry, and by autumn 2012 the new machine should be on campus, humming alongside its predecessor.

The next supercomputer will be a radical step forward, with at least five times Ra’s computing power and roughly 16,000 processor cores to Ra’s 2,144. Even so, it will consume just a fraction of Ra’s physical space and electrical power, thanks to technological developments over the past few years.

Ra’s successor will give a boost to many of Mines’ most ambitious efforts, and Lusk predicts the new machine will be the basis for frontier energy research for years to come. “It’s exciting to be part of this vision,” he said. “The campus now fields several big teams that do high performance computing in close collaboration with experimentalists. The original leadership has evolved into some amazing self-assemblies, and I can’t wait to see what advances come out next.”


Clathrate Hydrates

Illustration of hydrate nucleationResearchers have achieved the first real insight into the birth and growth of the cage-like structures known as clathrate hydrates. These materials can form naturally —for example, out of natural gas in pi/gas pipelines, where they form an “icy slush” that can accumulate in the pipelines and eventually clog the flow. Using Ra, Mines researchers have been able to simulate for the first time the molecular processes that cause such hydrates to nucleate and grow, adding – atom by atom – to each rigid molecular cage.

It’s not an easy task. Hydrates form out of disordered systems, with atoms starting out adrift and then coming together in precise ways to form a complex network of water molecules enclosing gas molecules. Simulating how that transition happens takes a lot of computing power, said Amadeu Sum. “That’s why we need to use large resources like we have on campus to do these large and long simulations,” he said.

Knowing how hydrates nucleate will help researchers better understand how to prevent/control them from forming and harness them for useful purposes as well. One impactful area for hydrates is the recovery of methane gas from natural hydrate deposits in the permafrost and ocean seafloor, and the utilization of hydrates as an energy storage medium for natural gas and hydrogen.


Oil and Gas

With the need for traditional fossil fuels still great, Associate Professor Paul Sava is using Ra to discover new sources of oil and gas. Active in the Center for Wave Phenomena in the Geophysics Department, Sava specializes in developing new methods for probing the earth’s interior with seismic waves. Doing so requires running simulation after simulation of how quickly waves travel through the earth, then comparing those to real-world observations to see how closely the two match.

So far, Sava’s team has been able to refine a flagship exploration technique used in industry. The Mines scientists simulate how rock from inside the planet can be squeezed (like a fluid, as industry models it) or twisted around (like an elastic) — a difference that affects the speed of passing seismic waves. Updating this knowledge allows oil and gas companies to better predict where a promising prospect might turn into a lucrative discovery. “The information relevant to them requires a big computer like this,” Sava said.



Mines’ dedication to high-performance computing has helped draw high-profile faculty to the university. “When I showed up, Ra was being unboxed,” said Reed Maxwell, a hydrologist who moved from the Lawrence Livermore National Laboratory in California.

In the Department of Geology and Geological Engineering, Maxwell uses Ra to simulate how water flows from deep within the ground to shallower levels, and also from there into the atmosphere. His computer code, dubbed ParFlow, is one of the few such models to integrate this entire hydrologic cycle. Maxwell has used ParFlow to explore all sorts of important questions, such as how agriculture draws down groundwater and how changes in hydrology affect local atmospheric patterns —for example, the wind energy potential over a particular plot of land.

Because his simulations require so much computing power, Maxwell uses not just Ra but also several other supercomputers, including ones at the Oak Ridge National Laboratory in Tennessee and at a facility in Jülich, Germany. “I always envision running on a range of supercomputers,” he said.

Most recently, Maxwell has built a high-resolution hydrological model of the entire continental United States, which covers 6.3 million square kilometers at a resolution of 1 kilometer. This simulation, which he said is one of hydrology’s “grand challenges,” is tied into leading climate models so that Maxwell can, for example, probe how water flow may affect regional climate change in the decades to come.



From the scale of continents down to the scale of nanoparticles, Ra’s simulations are doing it all. For Cristian Ciobanu, a materials scientist in the Department of Mechanical Engineering, Mines’ computing resources involve the very small. He works to understand the basic chemistry and physics of materials crucial for energy applications, from lithium-ion batteries to biomass to solar cells. “In all these cases, the campus facility is important,” Ciobanu says. “It’s basically a lot of computing power on site, and you can do things faster, closer to real time.”

For instance, he and his collaborators at NREL have shown that using a material as common as quartz can lead to an increase in the capacity of lithium-ion batteries over the first couple hundred cycles of charging and discharging, thus hinting at new ways to prolong battery life. Other simulations have shown that adding nanoparticles of gold or other precious metals to a particular chemical process speeds up the reaction, while also making it yield a desired reaction product, hastening the conversion of biomass into energy. Ciobanu is now running calculation after calculation on Ra to find the best possible shape and composition for nanoparticles to catalyze the biomass conversion reactions. With such information, an experimentalist can make nanoparticles that work efficiently the first time around, without having to run though the trial-and-error of testing particle after particle in real life.

For solar cells, Ciobanu has been testing how to make the perfect nanoparticles out of germanium and tin with the best electronic properties for absorbing light. Such alternative materials might be used in future photovoltaic cells, especially if they can be designed through supercomputer simulations and then tailor-made to fit those designations.



Lusk is also using the power of Ra to figure out how to make better photovoltaics. In 2011, he and colleagues discovered one way to beef up the efficiency with which a solar cell transforms sunlight into energy. Supercomputer simulations done at Mines suggest that in a specially designed material, a particle of light (photon) can knock loose not just one electron (its flow creates the electricity that powers solar cells) but two or more, in a process known as multiple exciton generation. The set of excited electrons would turn more of the original solar energy into useful electricity because not as much is lost to generate heat.

The race is now on to make better photovoltaic materials by exploiting multiple exciton generation and other quirky quantum mechanical properties that Lusk’s team has discovered. They use Ra to model how to best design what amounts to a new form of matter composed of quantum dots. These tiny particles, just a nanometer or two across, both help to capture solar energy and to move it through the solar cell to create useful current.

Lusk is also looking to build on nature’s own solar cell — the leaf — by co-opting its photosynthetic tricks. Nature has evolved some very clever ways of harvesting solar energy, but its solar panels don’t last very long. “We’re using the computer to unravel some of the quantum mechanical secrets that are going on all around us. And then we want to use that information to build inorganic solar cells that do the same thing better and without wearing out as easily,” Lusk said.


This article appears in the 2012-13 issue of Energy and the Earth magazine.


Subscribe to RSS - High performance computing