How to transfer files between INCD and a local machine
NOTE Please note that INCD Lustre storage is not designed for small files. Attempting to store or retrieve files less than a few megabytes will result in extremely poor performance for all users. If you wish to store lots of small files to massdata, please use a utility such as tar to combine them into a single, larger file.
To transfer files from your local machine to INCD, we recommend to use rsync. With appropriate options, rsync is resumable allowing the transfer to continue in case of drop mid-transfer. The recommended command line to use on your local machine is:
rsync -avPS src dst
where src his the path of the files on your local machine and dst will be the path to your destination on the appropriate host.
//1. Transfer files from local machine to cirrus.ncg.ingrid.pt: rsync -avPS /home/myfiles email@example.com:/data/unixgrp/my_folder/my_files //where user is your INCD username, and unixgrp is the unix group identifying your project. //2. Transfer files from INCD to your local machine: rsync -avPS firstname.lastname@example.org:/data/unixgrp/my_folder/my_files /home/mylocalfolder/ //where user is your INCD username, unixgrp is the unix group identifying your projec and /home/mylocalfolder/ it's a local machine folder.
The use of sshfs is a secure convenient way to share external volumes as long the user is able to open a ssh session.
Suppose we want to share a directory /remote/dir from some remote server named server.remote.pt on the local user interface cirrus at subdirectory mydata. If the remote username is remuser, execute on CIRRUS user interface:
$ mkdir mydata $ sshfs email@example.com:/remote/dir mydata $ df Filesystem 1K-blocks Used Available Use% Mounted on server.remote.pt:/remote/dir 841572128 83471036 715328408 11% /users2/<group>/<user>/mydata $ ls -l mydata $ cp mydata/some_file local/workdir
when finish the user can unmount the remote volume with the command
$ fusermount -u mydata
The user will have on local mount directory the same permission to access the volume as on the remote server.
The mount point will be available only on the local node. For example, if the user mount the volume on the user interface this directory content will not be available on the workernodes. This is a convenient way to access and copy files between sites but it is not suited to be used within a batch job, particularly on a MPI batch job with multiple nodes.