Skip to main content

How to transfer files between INCD and a local machine

NOTE Please note that INCD Lustre storage is not designed for small files. Attempting to store or retrieve files less than a few megabytes will result in extremely poor performance for all users. If you wish to store lots of small files to massdata, please use a utility such as tar to combine them into a single, larger file.

There are at least two choices to share and transfer files:

  • rsync
  • sshfs

rsync

To transfer files from your local machine to INCD, we recommend to use rsync. With appropriate options, rsync is resumable allowing the transfer to continue in case of drop mid-transfer. The recommended command line to use on your local machine is:

rsync -avPS src dst

where src his the path of the files on your local machine and dst will be the path to your destination on the appropriate host.

For example:

//1. Transfer files from local machine to ui7-hpc: 
 
rsync -avPS /home/myfiles user@ui7-hpc.ncg.ingrid.pt:/data/unixgrp/my_folder/my_files
//where user is your INCD username, and unixgrp is the unix group identifying your project. 
 
 
//2. Transfer files from INCD to your local machine:
rsync -avPS user@ui7-hpc.ncg.ingrid.pt:/data/unixgrp/my_folder/my_files /home/mylocalfolder/ 
//where user is your INCD username, unixgrp is the unix group identifying your projec and /home/mylocalfolder/ it's a local machine folder.

sshfs

The use of sshfs is a secure convenient way to share external volumes as long the user is able to open a ssh session.

Suppose we want to share a directory /remote/dir from some remote server named server.remote.pt on the local user interface cirrus at subdirectory mydata. If the remote username is remuser, execute on CIRRUS user interface:

$ mkdir mydata
$ sshfs remuser@server.remote.pt:/remote/dir mydata
$ df
Filesystem                   1K-blocks     Used Available Use% Mounted on
server.remote.pt:/remote/dir 841572128 83471036 715328408  11% /users2/<group>/<user>/mydata
$ ls -l mydata
$ cp mydata/some_file local/workdir

when finish the user can unmount the remote volume with the command

$ fusermount -u mydata

The user will have on local mount directory the same permission to access the volume as on the remote server.

The mount point will be available only on the local node. For example, if the user mount the volume on the user interface this directory content will not be available on the workernodes. This is a convenient way to access and copy files between sites but it is not suited to be used within a batch job, particularly on a MPI batch job with multiple nodes.

Back to top