New Hikari supercomputer starts solar HVDC

The roar can be deafening. Cooling fans and power supplies whoosh and whine from rows and rows of supercomputers at the main data center of the Texas Advanced Computing Center in Austin. The power bill at TACC can reach over a million dollars a year to keep the machines humming. But there's a stranger in town that might change how data centers power their systems.

A new kind of advanced computing system called Hikari (Japanese for the word "light") came online at TACC late August, 2016. What's new is that Hikari runs on solar power and high voltage direct current, or HVDC.

Hikari is a microgrid that supports a supercomputer, a first for the U.S. By day solar panels that shade a TACC parking lot provide nearly all of Hikari's power, up to 208 kilowatts. At night it switches back to conventional AC power from the utility grid.

"The Hikari project is a Japan and Texas collaboration project, and it aims to demonstrate the potential of the HVDC system," said Toshihiro Hayashi, assistant manager in the engineering divisions of NTT FACILITIES, INC., Japan.

Engineers of the Hikari HVDC power feeding system predict it will save 15 percent compared to conventional systems. "The 380 volt design reduces the number of power conversions when compared to AC voltage systems," said James Stark, director of Engineering and Construction at the Electronic Environments Corporation (EEC), a Division of NTT FACILITIES, INC..

"What's interesting about that," Stark added, "is the computers themselves - the supercomputer, the blade servers, cooling units, and lighting - are really all designed to run on DC voltage. By supplying 380 volts DC to Hikari instead of having an AC supply with conversion steps, it just makes a lot more sense. That's really the largest technical innovation."

Data centers in the U.S. consumed an estimated 70 billion kilowatt hours in 2014, which represents about 1.8 percent of total U.S. electricity consumption. That's according to a June 2016 Department of Energy report by the Lawrence Berkeley National Laboratory.

High Voltage Direct Current also allows for ease in connection to renewable energy, solar for Hikari but potentially other sources like wind and hydrogen fuel cells. "That's really one of our main focuses, trying to make data centers more sustainable so that we're reducing overall power consumption within an industry that has traditionally consumed a lot of power," James Stark said.

With sustainability comes reliability. This held true for another NTT FACILITIES, INC. project called the Sendai Microgrid on the fateful day of the Tohoku earthquake, March 11, 2011. The Sendai Microgrid was equipped with gas engines, a fuel cell, and a photovoltaic array. It continued to supply power and heat to facilities at Tohoku Fukushi University for hospital patients and elderly nursing home residents despite days of blackouts from the catastrophic damage to the district's energy supply system caused by the earthquake.

"This microgrid power supply system activated very well after the earthquake," Hayashi said. "It showed that it's a very highly reliable energy system."

Another NTT FACILITIES project leading to Hikari is the Tsukuba Research and Development Center, a HVDC microgrid that generates 70 kilowatts for a modular data center. "We have various experiences in Japan of having this HVDC system," Hayashi said.

These projects, including Hikari, are supported by NEDO, the New Energy and Industrial Technology Development Organization, a public agency of Japan. The Hikari project partners are NEDO, NTT FACILITIES, INC., and the University of Texas at Austin through the Texas Advanced Computing Center.

The collaboration started with a visit to TACC by engineers at NTT FACILITIES, INC. in early 2014, fresh off the heels of basic studies they'd done for installing HVDC at U.S. sites. TACC also shares a strong interest in developing new technologies, including energy savings.

"We're very interested in UT Austin's motto," said Hayashi, "which is 'what starts here changes the world.' We very much agree with this motto."

Hayashi's team worked with TACC to develop feasibility studies of the Hikari HVDC project from December 2014 to May 2015. This effort led to a Memorandum of Understanding between the State of Texas and NEDO in August of 2015.

NTT FACILITIES, INC. worked with EEC to build out Hikari, which completed the system installation late August 2016. "If there wasn't such partnership, we wouldn't have launched this project. I would like to express my gratitude to NEDO for establishing the partnership," Hayashi said.

The Hikari supercomputer cluster consists of 432 Hewlett Packard Enterprise (HPE) Apollo 8000 XL730f servers coupled with HPE DL380 and DL360 nodes that are interconnected with a first-of-its-kind Mellanox End-to-End EDR InfinBand at 100 gigabytes per second. Over 10,000 cores from "Haswell" Xeon processors will deliver more than 400 teraflops.

The Hikari project also aims to demonstrate energy efficiency through more than just HVDC. The HPE Apollo 8000 systems use a warm water-based liquid cooling system that eliminates the need for fans within the nodes and reduces the energy that would normally be required for water refrigeration and excess heat removal. The solar energy that would have been used for powering fans and chillers can be used for computational work.

When it reaches production later in 2017, Hikari will be used by the University of Texas medical researchers to make progress on diseases like cancer and disorders like autism.

"We really hope this project will demonstrate the efficiency advantages of using 380 volt DC, not only in data centers, but in any commercial building," James Stark said. "The hope is that the research that comes out of this demonstration project will help to open the door to more widespread use of 380 volt systems throughout data centers and commercial buildings worldwide."