ETH research among the best supercomputer applications in 2015
Every year at the SC conference, the Gordon Bell Prize is presented to recognise outstanding work in the field of high-performance computing. This year, two research groups from ETH Zurich are competing in the finals – a remarkable achievement considering only five teams have been preselected world-wide.
November 9, 2015 – Simone Ulmer
Simulations on supercomputers support experimental research in many scientific fields. The calculations are normally hugely complex and time-consuming, even for the most powerful supercomputers, and include, for example, the simulation of nanoscale electronic components such as transistors, or the simulation of blood flow through microscopic channels, separating out metastasis-building tumour cells from red blood cells.
In order to simulate physical processes in a reasonable time, it is essential to have not only the most powerful supercomputers but, above all, highly efficient software. Both of the Gordon Bell finalists from ETH used the CSCS supercomputer Piz Daint to refine and optimise their simulation software. The simulations that were subsequently carried out on one of the fastest supercomputers in the world, Titan, at the Oak Ridge Leadership Computing Facility at the Oak Ridge National Laboratory in Tennessee, enabled the two ETH researchers to achieve their nominations.
Fifty times bigger and faster
ETH professors Mathieu Luisier from the Institute for Integrated Systems and Joost VandeVondele, Head of the Nanoscale Simulations Group, and their teams, combined two software programs and also developed a new algorithm to maximise the throughput rate for hybrid computer systems consisting of conventional CPUs and graphic processors (GPUs). This enabled them not only to reduce the time taken to simulate nanoscale devices by fifty times, but also to increase the size of the nanocomponents to over 50,000 atoms – a number corresponding to the actual size of a nanodevice. “Up to now, even the most CPU-intensive models usually only enabled systems to be simulated that consisted of a maximum of 1000 atoms,” says Luisier.
When simulating the behaviour of nanocomponents, the quantum effects that occur in these tiny components have to be taken into account. To this end, Luisier has been working for over ten years on software called OMEN (see “Nanotransistors on the virtual test bench”). Now the researcher has joined forces with VandeVondele, who helped to write and develop the CP2K code. The latter is specially designed to analyse the electron density “ab initio”, – that means from first principles, on the basis of physical laws without any empirical data. For example the atoms that build up a nanotransistor, by computing wave functions in a closed or periodic system. However, when a voltage is applied to the transistor, electrons start to flow through it. To simulate the transport of electrons, OMEN calculates the probability that injected electrons pass through the transistor, considering, if necessary, possible interactions with the crystal that forms the transistor. Both the ab initio calculation of the properties of the crystal using CP2K, and of the transport of electrons using OMEN, are enormously complex.
Now, both software codes have been optimised for hybrid computer systems on Piz Daint, and doctoral students Sascha Brück and Mauro Calderara have developed a new algorithm called SplitSolve to greatly reduce the computing time and allow for the analysis of structures with over 50,000 atoms. SplitSolve enables parallel and simultaneous computing on CPUs and GPUs. Luisier is convinced that “with the help of Titan, we have redefined the boundaries for quantum transport simulations. The simulations are now realistic enough that they can be of direct benefit to experimental research into nanodevices.”
In silico lab-on-a-chip
The second research team from ETH that has been nominated for the Gordon Bell Prize is led by ETH Professor Petros Koumoutsakos of the Computational Science & Engineering Laboratory (CSElab). In collaboration with researchers from the Università della Svizzera italiana, the United States (Brown University, NVIDIA, udevicex.github.io/uDeviceX/) and Italy (CNR and University of Rome), the team used the Titan supercomputer to run state-of-the-art simulations of the flow of micron-sized red blood cells and tumour cells through microfluidic channels. According to the researchers, the geometric details and the number of cells have been improved by two orders of magnitude and thus the study redefines the boundaries for simulating flows through micron-scale devices.
This enabled the computational scientists to emulate the laboratory experiments (lab-on-a-chip) of other researchers looking at how tumour cells could be filtered out of the blood thus allowing for the early diagnosis of metastatic cancer. Just like in an experiment, the simulation shows the behaviour of blood that flows through a system made up of columns of a certain size and shape, arranged in rows. Because of the geometry of the device and the structure of the cells, the larger tumour cells, which measure in diameter about 25 microns – three times larger than red blood cells – drift to the side of the device so they can be separated out of the bloodstream and identified.
Looking for a tumour cell in blood is like looking for a needle in a flowing haystack, says Koumoutsakos. There may be one tumour cell in a billion red blood cells. However, tumour cells in blood can help diagnose for metastasis, responsible for 90% of the eight million deaths a year caused by cancer. Humans have between four and six billion red blood cells in every millilitre of blood. This kind of device could therefore be used to diagnose and one day perhaps even eliminate tumour cells from the blood.
The “in silico lab-on-a-chip”, as Koumoutsakos calls the simulation, reproduces and assists in further optimising laboratory experiments and designing novel lab-on-a-chip-devices that are essential to researchers in the pharmaceutical industry. Unlike Moore’s Law, which says that computing power doubles every 18 months, the pharmaceutical industry is faced with “Eroom’s Law”. This relates to the approval of new drugs in exactly the opposite way to Moore’s Law: the number of newly approved drugs is being halved every two years.
Focus on cancer
Koumoutsakos and his team have been working for over a decade on researching the processes involved in cancer. Having first simulated angiogenesis, i.e. how tumour cells stimulate the growth of blood vessels and so enable the transport of tumour cells in the bloodstream, his next goal was to simulate the flow of blood and tumour cells in the capillaries of the circulatory system. He wanted to demonstrate the interaction of cells and drug carrying nanoparticles in the capillaries. “The problem is that the tumour induced blood vessels are so entangled that there is no easy way to describe them. That means we are not able to reconstruct them in simulations,” says Koumoutsakos.
Having watched real experiments to separate tumour cells, he had the idea of using this approach to investigate the behaviour of the cells in blood. George Karniadakis of Brown University had the necessary expertise in what is called dissipative particle dynamics, where the flow of blood is simulated as a collection of individual particles. Diego Rossinelli from CSElab and other members of the team put these techniques together into a model and optimised the codes before making the simulation on Titan that put them in the running for the Gordon Bell Prize.
| Gordon Bell Prize |
| The “Gordon Bell Prize” has been endowed by the Association for Computing Machinery since 1987 and is named after the famous computing engineer and pioneer in the field of high-performance computing. It has been presented at the SC conference in the United States every year since 1987. The prize is worth USD 10,000 and recognises outstanding achievements in supercomputing. It is intended to promote and drive forward the development of large-scale parallel computing, in which multiple computing operations run simultaneously in parallel. |