Accelerating New Zealand’s Climate research

Global collaboration and eResearch infrastructure.
Infrastructure and data transfer capability are crucial to supporting climate modelling research.

 

  • Climate modelling research is made up of massive amounts of data and relies heavily on international collaboration.

  • NIWA scientists need to transfer and process this climate modelling data. This requires regular and reliable access to key eResearch infrastructure like the REANNZ network.

  • The speed of data transfer across the network is enhanced by using New Zealand's National Data Transfer Platform, managed by New Zealand eScience Infrastructure (NeSI) and powered by Globus to provide fast, secure, and reliable transfers.

  • Together with HPC processing, these tools make new workflows possible for Dr Jonny Williams and his colleagues.

For NIWA climate scientist Dr Jonny Williams, global climate models are key to uncovering the future state of New Zealand’s climate. The mountains of numerical data that make up these simulations enable scientists to forecast climate scenarios by tracking and predicting future sea surface temperatures, weather and greenhouse gas levels. Jonny’s research contributes to global projects that compare the modelling data from New Zealand to these other global models.

Fast data transfers for global co-operation

Jonny uses REANNZ, NeSI's National Data Transfer Platform, and Globus' data management tools to shift data worldwide with the click of a button.

“It allows us to transfer data much faster than traditional methods. Because the data volumes that we need to transfer are so large, traditional methods are just simply not fast enough,”

“One simulation that we run with our collaborators can easily produce 50-to-100 terabytes of data. That’s not seen as a large amount of data in the climate research world. On a daily basis, I can deal with individual files that are 100 gigabytes in size.”

What has that speed done for your workflow?

The way I see it personally, is that it hasn't just sped up a workflow it's actually enabled a new workflow; that is, to  routinely shift multiple tens or hundreds of terabytes around the world at the click of a button.

Having access to these resources means that we can take part in projects that we would not be able to otherwise, like working on the Aerosol and Chemistry Model Intercomparison Project (AerChemMIP) as a part of the recent IPCC report. This involved us running some simulations on behalf of the UK Met Office and collaborating with them in that data analysis. However, the data analysis tools were installed on machines in the UK on a dedicated data processing platform and so we shifted the data over to that platform and performed the data analysis there.

People used to joke that  you shouldn’t underestimate the bandwidth of a horse and cart filled with USB sticks if you just wanted to shift one large amount of data once - but we needed to do this routinely and regularly. If we were still relying on the older computer transfer technologies - such as SCP and rsync - then that workflow would just simply not be possible. It's not just a case of speeding up something that we were able to do before and making it more efficient, this has actually enabled us to perform completely new workflows and to take part in these very large international collaborations.

 Dr Jonny Williams, NIWA

Zooming in on New Zealand

Regional modelling teams have been enhancing the resolution of New Zealand’s climate modelling, contributing their techniques and the data collected to build more accurate models. 

“We work very closely with regional modelling teams. Even though they're modelling a much smaller area, they’re modelling it at a higher resolution. The amount of data that they produce can be even larger,” says Jonny.

The scale of information collected, stored and shared in global climate research makes it extremely data intensive. While comparing a New Zealand model to a UK model, Jonny and his colleagues may shift tens, sometimes hundreds, of terabytes of data across the world. That’s where access to national eResearch infrastructure with international reach becomes crucial.

“Without international collaboration, we wouldn't be able to do the work we do. The computer models that we run are some of the most complex pieces of software in the world. Our international partners provide us with a code base to work with. The code that we use has been developed by hundreds of scientists over decades, it has a very long history. That is just simply not something that you can do on your own.”

The scale of data being produced in fields like climate research continues to grow. In New Zealand, two High Performance Computing (HPC) systems are used to power the Deep South Challenge's large-scale modelling and analysis of massive datasets.

NIWA’s contribution to the latest Intergovernmental Panel on Climate Change Report requires Jonny and his colleagues to share data reliably overseas. Projects like this need to access remote equipment and analysis platforms.

“The HPC at NIWA produces the data that needs to be analysed, but we also rely heavily on communication with other parts of the world. The ability to share our data comes from the interconnection between the HPC and the network. The international network that REANNZ provides here is essential.”

What do you think that we can do as REANNZ to help support your work further?

I think one thing that we as a community can do is actually just learn more about workflows. I've worked with colleagues who have tried shifting large amounts of data from one place to another. When the technologies that you are using work it’s not always clear to see that there are improved ways of doing the workflow. Ways that make it quicker, easier and more efficient. Raising awareness of new ways of doing your work more efficiently - and without a steep learning curve - can help everyone involved.

 Dr Jonny Williams, NIWA

 

Computing NZ’s challenges

The research that Jonny and the team at NIWA are collaborating on is a part of the Deep South Challenge: Changing with our Climate. The National Science Challenges tackle the biggest science-based issues facing New Zealand. They bring together top researchers to work across disciplines, institutions and borders to solve these challenges. 

“Other parts of the world have many thousands of climate researchers and are very heavily studied. Compared to that, the Southern Ocean and Antarctica are very poorly studied. They're very big, they're very remote and they're difficult to get to.”

Global climate processes are seeing significant impacts to the future climate of the New Zealand region.

“By strengthening our ability to understand and anticipate our future climate, we’re giving New Zealanders the best possible chance to adapt and manage risk in the years to come.” Olaf Morgernstern, Earth System Modelling Programme Lead, Deep South Challenge.

The physical isolation of the Southern Ocean and Antarctica is not limiting researchers' ability to investigate and discover the impacts of global climate as a part of the Deep South Challenge. New Zealand’s isolation should not limit its ability to participate in collaboration and contribute to world class research. In fact, it drives the need for continued, reliable connectivity so that the work of New Zealand’s researchers is accessible to the world and continues to make a valuable contribution to the field of climate research. 

Where do you see the trend for research going in terms of data production, transfer, analysis and storage?

There has been very large growth in other data intensive fields, particularly bioinformatics for example. I think there's a lot of data sets out there that people would like to get their hands on but can't because of the data transfer speeds. There’s always an appetite for increased power from HPC. It's all very well to produce this data but you need to do something with it, you need to transfer it somewhere, you need to archive it. With the increasing speeds the next bottleneck becomes how fast you can process it.

Dr Jonny Williams, NIWA 

NIWA is a Crown Research Institute that works on climate, freshwater and ocean science. One aspect of NIWA’s work is in global climate modelling. They collaborate with international partners on simulations that simulate the flow of air, water and sea ice around the world. They also study how these physical processes react and interact with chemical and biological cycles in the air, on the land and in the sea.

REANNZ is New Zealand’s research and education network (NREN). As an NREN, the network enables research, productivity and collaboration through data movement at a scale rarely found outside of the research and education sector.

New Zealand eScience Infrastructure (NeSI) designs, builds, and operates a specialised platform of shared high performance computing (HPC) infrastructure and a range of eResearch services. As the New Zealand research sector looks to answer national science imperatives across institutional boundaries, NeSI seeks to build national capability in running and optimising use of HPC and eResearch infrastructure. Collaborations with partners like NIWA, REANNZ, and Globus have been vital to supporting the needs of researchers like Jonny who require powerful computing resources and the ability to transfer research data at speed and scale.

Case Studies
Find more examples of how REANNZ members use our network and services.

Products and Services

About REANNZ

Find anything about our products, services, and more. Enter a query in the search input above.