How to: File/Data Transfer - Data Science/Machine Learning Platform (DSMLP)


File Transfer


Standard utilities such as 'git', 'scp', 'sftp', and 'curl' are included in the standard container image and may be used to retrieve code or data from on- or off-campus servers.    

Files also may be copied into the cluster from the outside using the following procedures.

Note that file transfer is only offered through 'dsmlp-login.ucsd.edu', even if you normally launch jobs from 'ieng6'

 

Copying Data Into the Cluster: SCP/SFTP from your computer

Data may be copied to/from the cluster using the "SCP" or "SFTP" file transfer protocol from a Mac or Linux terminal window, or on Windows using a freely downloadable utility.  We recommend this option for most users.

 

Example using the Mac/Linux 'sftp' command line program:

slithy:Downloads agt$ sftp <username>@dsmlp-login.ucsd.edu
pod agt-4049 up and running; starting sftp
Connected to ieng6.ucsd.edu
sftp> put 2017-11-29-raspbian-stretch-lite.img
Uploading 2017-11-29-raspbian-stretch-lite.img to /datasets/home/08/108/agt/2017-11-29-raspbian-stretch-lite.img
2017-11-29-raspbian-stretch-lite.img             100% 1772MB  76.6MB/s   00:23
sftp> quit
sftp complete; deleting pod agt-4049
slithy:Downloads agt$

 

On Windows, we recommend the WinSCP utility.

 

Copying Data Into the Cluster: rsync

'rsync' also may be used from a Mac or Linux terminal window to synchronize data sets:

slithy:ME198 agt$ rsync -avr tub_1_17-11-18 <username>@dsmlp-login.ucsd.edu
pod agt-9924 up and running; starting rsync
building file list ... done
rsync complete; deleting pod agt-9924
sent 557671 bytes  received 20 bytes  53113.43 bytes/sec
size is 41144035  speedup is 73.78
slithy:ME198 agt$

 

Find the original version of this guide.

For more information, check the FAQ.

Your instructor or TA will be your best resource for course-specific questions.

If you still have questions or need additional assistance, please submit a ticket or visit support.ucsd.edu.