Last Mile Transfer: enabling local data transfers on the global WLCG infrastructure

Abstract

The computing challenge at CERN is of a global nature. To make real world-wide distributed computing possible, more than 150 computer centers must be seamlessly integrated. This means integrating CPU, storage and network.

The File Transfer Service (FTS) is a tool that emerges to solve the data movement problem. It is used to schedule data transfers between different storage resources. Its optimizer takes care of increasing the parallelism to improve throughput, without exhausting the storage resources. It also has a web interface (WebFTS) which makes it quite easy for users to invoke reliable, managed data transfers on distributed infrastructure.

However, FTS only solves part of the problem, as an increasing number of grid users run simulations on their personal laptops, generating files that can amount to several gigabytes. Normally, users would want to move these files from their personal computers to a remote Grid storage for long-term archiving, sharing, or running further processing on them. The issue here is that these users might be sitting behind a firewall, which means that their computer will not be able to listen to inbound connections.

Last Mile Transfer is a solution that was developed to enable local file uploads on the Worldwide LHC Computing Grid (WLCG) infrastructure.