High performance computing

Mines Mechanical Engineering faculty are part of a project that has been awarded $1.6 million to improve the design of soft magnetic materials, which would increase the efficiency of electrical systems. Mechanical Engineering professors Aaron Stebner and Cristian Ciobanu are collaborating with faculty at Case Western Reserve University and the University of Minnesota on the project, titled “Accelerated Soft Magnetic Alloy Design and Synthesis Guided by Theory and Simulations.”

Soft magnetic materials, such as iron and nickel-iron alloys, are labeled soft due to their low loss and high permeability characteristics. They are essential to all aspects of daily life, especially in the generation, conversion and conditioning of electric power. Even small improvements in energy efficiency from the magnetic components can result in large energy, financial and CO2 emissions savings. Stebner and Ciobanu plan to advance the alloy design cycle through a multidisciplinary approach involving materials science, applied physics and electrical engineering.  

Graph showing the iterative process in creating a database of alloy properties

“The research in this grant is aimed at creating an ability to computationally explore new chemistries and microstructures for soft magnetic materials,” said Stebner. “By developing computations that can assist with evaluating new chemistries in the computer instead of in the lab, that process could be reduced to five years or less, and we could also discover well-performing magnetic materials that metallurgists have not thought of before.”

As technological advances lead to soft magnetics operating at higher frequencies and higher temperatures, the challenge becomes retaining their inherent strengths. Stebner explained, “If we think about our power grids, they operate at 60-120 Hz and near room temperature. Looking to applications like the new hybrid-electric airplane program announced by NASA, the goal is for the energy conversion systems to operate at the same efficiencies as our power grids, but at 50-100 kHz and temperatures up to 400 C.”

Ciobanu’s research will focus on the density functional theory, while Stebner’s team will investigate fine element modeling in micromagnetics. The $1.6 million grant award is funded by the National Science Foundation’s Designing Materials to Revolutionize and Engineer Our Future program. Figure 2, below, captures the experimental and computational collaboration between Mines faculty, Matthew Willard from Case Western’s Department of Materials Science and Engineering and Richard James from the University of Minnesota’s Department of Aerospace Engineering and Mechanics. 

Workflow chart showing collaborative process of alloy processing to thermodynamic modeling and micromagnetic modeling on the computational end as well as experimental models looking at structural and magnetic characterizations


Deirdre Keating, Communications Manager, College of Engineering & Computational Sciences | 303-384-2358 | dkeating@mines.edu
Mark Ramirez, Communications Manager, College of Applied Science & Engineering | 303-384-2622 | ramirez@mines.edu

This is a story of humans and hardware. What happened when professors, tops in their different fields of energy research, gained campus access to a world-class supercomputer?

The story began four years ago with the institutional vision to bring a supercomputer named Ra to Mines. Dag Nummedal, director of the Colorado Energy Research Institute, and Physics Professor Mark Lusk had been working to acquire a supercomputer, and when Vice President of Research and Technology Transfer John Poate got involved, “the idea resonated campus-wide,” said Lusk.

With the horsepower of Mines leadership behind this well-timed initiative, it became a commitment to much more than hardware. Five years, one supercomputer, 10 new faculty hires, 15 classes, 60 PhD students and 120 journal publications since that original vision, Mines has become a global leader in computationally guided energy science research.

The initiative has made a huge impact on research volume, led to many important discoveries, and catalyzed interdisciplinary collaboration across campus. More than 90 percent of Mines’ academic departments are pursuing projects supported by the Golden Energy Computing Organization. As scientists from different areas come together, ideas begin to cross-fertilize and surprising synergies emerge. As Lusk notes, “Once all these people start working together under one virtual roof, good things happen.”

One of the new supercomputer hires was Amadeu Sum, a professor in the Chemical and Biological Engineering Department and co-director of Mines’ Center for Hydrate Research. He used Ra to explain the nucleation and growth of hydrates, and his work landed on the cover of Science.

On another side of campus, REMRSEC, the Renewable Energy Materials Research Science & Engineering Center, was chosen for funding by the National Science Foundation in part due to the computing power Ra could bring to the table. 

Lusk’s solar cell work with REMRSEC on multiple-exciton generation (MEG) was successful. MEG theorizes it is possible for an electron that has absorbed light energy to transfer some of that energy to other electrons, resulting in more electricity from the same amount of absorbed light.

In a cross-fertilizing leap, REMRSEC decided to look at hydrates as a way to store hydrogen. The center provided seed money to Carolyn Koh, a professor in the Chemical and Biological Engineering Department, to lead a combined team of REMRSEC solar energy scientists and Center for Hydrate Research experts. Together they developed a computer analysis to assess the potential of hydrates for hydrogen storage.

Then they began thinking of other materials that can be assembled into the cage-like clathrates. Could they build a silicon clathrate structure to store hydrogen? Using experimental facilities at the National Renewable Energy Laboratory (NREL), they determined the answer was yes.

The synergy continued. What are the photovoltaic properties of these new silicon clathrates? Can they be used to build a better solar cell? The answer, once again, appears to be yes.

In summary, hydrate engineers and solar energy physicists have founded two new facets of energy research because a world-class  supercomputing facility came to Mines. “The successes that have come from our original vision have been snapping together a Lego™ at a time,” said Lusk. “We have a cool system going now and there’s no end in sight.”


A New Season

Now plans are underway to purchase a new machine to become the campus flagship for high-performance computing, with Ra maintained as a set of smaller clones for less demanding projects and student training. Requests for bids have gone out to industry, and by autumn 2012 the new machine should be on campus, humming alongside its predecessor.

The next supercomputer will be a radical step forward, with at least five times Ra’s computing power and roughly 16,000 processor cores to Ra’s 2,144. Even so, it will consume just a fraction of Ra’s physical space and electrical power, thanks to technological developments over the past few years.

Ra’s successor will give a boost to many of Mines’ most ambitious efforts, and Lusk predicts the new machine will be the basis for frontier energy research for years to come. “It’s exciting to be part of this vision,” he said. “The campus now fields several big teams that do high performance computing in close collaboration with experimentalists. The original leadership has evolved into some amazing self-assemblies, and I can’t wait to see what advances come out next.”


Clathrate Hydrates

Illustration of hydrate nucleationResearchers have achieved the first real insight into the birth and growth of the cage-like structures known as clathrate hydrates. These materials can form naturally —for example, out of natural gas in pi/gas pipelines, where they form an “icy slush” that can accumulate in the pipelines and eventually clog the flow. Using Ra, Mines researchers have been able to simulate for the first time the molecular processes that cause such hydrates to nucleate and grow, adding – atom by atom – to each rigid molecular cage.

It’s not an easy task. Hydrates form out of disordered systems, with atoms starting out adrift and then coming together in precise ways to form a complex network of water molecules enclosing gas molecules. Simulating how that transition happens takes a lot of computing power, said Amadeu Sum. “That’s why we need to use large resources like we have on campus to do these large and long simulations,” he said.

Knowing how hydrates nucleate will help researchers better understand how to prevent/control them from forming and harness them for useful purposes as well. One impactful area for hydrates is the recovery of methane gas from natural hydrate deposits in the permafrost and ocean seafloor, and the utilization of hydrates as an energy storage medium for natural gas and hydrogen.


Oil and Gas

With the need for traditional fossil fuels still great, Associate Professor Paul Sava is using Ra to discover new sources of oil and gas. Active in the Center for Wave Phenomena in the Geophysics Department, Sava specializes in developing new methods for probing the earth’s interior with seismic waves. Doing so requires running simulation after simulation of how quickly waves travel through the earth, then comparing those to real-world observations to see how closely the two match.

So far, Sava’s team has been able to refine a flagship exploration technique used in industry. The Mines scientists simulate how rock from inside the planet can be squeezed (like a fluid, as industry models it) or twisted around (like an elastic) — a difference that affects the speed of passing seismic waves. Updating this knowledge allows oil and gas companies to better predict where a promising prospect might turn into a lucrative discovery. “The information relevant to them requires a big computer like this,” Sava said.



Mines’ dedication to high-performance computing has helped draw high-profile faculty to the university. “When I showed up, Ra was being unboxed,” said Reed Maxwell, a hydrologist who moved from the Lawrence Livermore National Laboratory in California.

In the Department of Geology and Geological Engineering, Maxwell uses Ra to simulate how water flows from deep within the ground to shallower levels, and also from there into the atmosphere. His computer code, dubbed ParFlow, is one of the few such models to integrate this entire hydrologic cycle. Maxwell has used ParFlow to explore all sorts of important questions, such as how agriculture draws down groundwater and how changes in hydrology affect local atmospheric patterns —for example, the wind energy potential over a particular plot of land.

Because his simulations require so much computing power, Maxwell uses not just Ra but also several other supercomputers, including ones at the Oak Ridge National Laboratory in Tennessee and at a facility in Jülich, Germany. “I always envision running on a range of supercomputers,” he said.

Most recently, Maxwell has built a high-resolution hydrological model of the entire continental United States, which covers 6.3 million square kilometers at a resolution of 1 kilometer. This simulation, which he said is one of hydrology’s “grand challenges,” is tied into leading climate models so that Maxwell can, for example, probe how water flow may affect regional climate change in the decades to come.



From the scale of continents down to the scale of nanoparticles, Ra’s simulations are doing it all. For Cristian Ciobanu, a materials scientist in the Department of Mechanical Engineering, Mines’ computing resources involve the very small. He works to understand the basic chemistry and physics of materials crucial for energy applications, from lithium-ion batteries to biomass to solar cells. “In all these cases, the campus facility is important,” Ciobanu says. “It’s basically a lot of computing power on site, and you can do things faster, closer to real time.”

For instance, he and his collaborators at NREL have shown that using a material as common as quartz can lead to an increase in the capacity of lithium-ion batteries over the first couple hundred cycles of charging and discharging, thus hinting at new ways to prolong battery life. Other simulations have shown that adding nanoparticles of gold or other precious metals to a particular chemical process speeds up the reaction, while also making it yield a desired reaction product, hastening the conversion of biomass into energy. Ciobanu is now running calculation after calculation on Ra to find the best possible shape and composition for nanoparticles to catalyze the biomass conversion reactions. With such information, an experimentalist can make nanoparticles that work efficiently the first time around, without having to run though the trial-and-error of testing particle after particle in real life.

For solar cells, Ciobanu has been testing how to make the perfect nanoparticles out of germanium and tin with the best electronic properties for absorbing light. Such alternative materials might be used in future photovoltaic cells, especially if they can be designed through supercomputer simulations and then tailor-made to fit those designations.



Lusk is also using the power of Ra to figure out how to make better photovoltaics. In 2011, he and colleagues discovered one way to beef up the efficiency with which a solar cell transforms sunlight into energy. Supercomputer simulations done at Mines suggest that in a specially designed material, a particle of light (photon) can knock loose not just one electron (its flow creates the electricity that powers solar cells) but two or more, in a process known as multiple exciton generation. The set of excited electrons would turn more of the original solar energy into useful electricity because not as much is lost to generate heat.

The race is now on to make better photovoltaic materials by exploiting multiple exciton generation and other quirky quantum mechanical properties that Lusk’s team has discovered. They use Ra to model how to best design what amounts to a new form of matter composed of quantum dots. These tiny particles, just a nanometer or two across, both help to capture solar energy and to move it through the solar cell to create useful current.

Lusk is also looking to build on nature’s own solar cell — the leaf — by co-opting its photosynthetic tricks. Nature has evolved some very clever ways of harvesting solar energy, but its solar panels don’t last very long. “We’re using the computer to unravel some of the quantum mechanical secrets that are going on all around us. And then we want to use that information to build inorganic solar cells that do the same thing better and without wearing out as easily,” Lusk said.


This article appears in the 2012-13 issue of Energy and the Earth magazine.


Subscribe to RSS - High performance computing