Skip to content

CephFS via our Data Transfer Nodes

To request a new group on the DTN, please contact hsrn-support@nyu.edu. Once a group has been created, users can be added from NYU's Grouper system.

Accessing the DTN

The DTNs are accessible via SSH at dtn.hsrn.nyu.edu. Please use your NetID and password to log in.

For example:

ssh rr2369@dtn.hsrn.nyu.edu

You can add the following SSH key entries to your ~/.ssh/known_hosts file to avoid manually verifying our hosts:

dtn.hsrn.nyu.edu ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOK7rpr2qzO7Jf/kIJwbS49/VGVFKScyjELdErBGYLodT8buSAQ1h/VBVQnOPteaqIHSpZ8UX81LImaqRFNMHZM=
dtn.hsrn.nyu.edu ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIB5cmdJKe/KFi9tx9DFLcjRL6NECPv3IEgEkCeTq/gVY
dtn.hsrn.nyu.edu ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFdyDx5J8Vy4srgeyZ2qA2blcRdLQIi+sfEhXNFsbTpi2e3wmQPxCjzLwds0XZoQOXTRsNSaJRZJBK92PvHlOM6ossDSjMHRiZ1OhlUAp1y0vksGRLpcQ7b8diCEng31ffuuaE92e58zLM+Qgn0mSaiYrUhlGpAkGbNAeXuXVowwoSnTlrynjcjjiHSJks36r9Z8zCx+YgJ2J2y9l9pF4yio3FBlzIX2C0iyO2NWsTqUgdsbL3V0ZauMJnoZLUWTvd/tShlKRueKmlvn4e2BizefF/i4EUty8LkAsOe/rVfcKRKXcew8nZtL57PQJq88vSKl/s4WkupexhZ+oCKDfoT4vQWyPKZqhjVib2P8K+PluMf+axJd/CBe2NiUCrjg6cavuUl9Ntcv3mykHzXKiHMMXAZdGYxz2BuxUky0DxGtEgySn5zdtEivd+9wuojQZrRMQtlvmh9NBfDqORlNIjtLPzLkrTe5Iqv6M50zBcGNn8CySGhkdydg00XcLPLzM=

You will find the Ceph filesystem (CephFS) at /ceph/

Using rsync

You can use the rsync tool to efficiently copy files from or to the filesystem. For example:

rsync -av --progress my/local/data rr2369@dtn.hsrn.nyu.edu:/ceph/rr2369/data

Using SFTP

SFTP is available and should work with any compatible file browser application. Use the URL sftp://dtn.hsrn.nyu.edu:/ceph.

On MacOS, use Finder > Go > Connect to Server, and enter the URL.

Using Globus

Globus is a fast, managed solution for large data transfers. You can use it to transfer data between our Ceph filesystem and:

  • cloud storage, such as Google Drive, Dropbox, Amazon S3
  • other Globus endpoints, such as NYU's Torch supercomputer or other research institutions
  • a personal endpoint running on your machine
  • your computer, directly through a web browser (for small transfers only)

See also the Globus documentation on transferring files and NYU HPC's guide.

Network restrictions

Access to the DTNs is only possible from NYU Networks, including the High Speed Research Network (HSRN), campus workstations, and wifi.

If you are not on an NYU network, you will need to connect to NYU's Virtual Private Network (VPN).

This restriction does not apply to Globus.

Troubleshooting

Connection Timeouts

Cause: Network connectivity issues or misconfigured firewall settings.

Solution:

  • Check if the DTN is reachable by running ping dtn.hsrn.nyu.edu in your Terminal.
  • Make sure you are in either on the HSRN or NYU network, otherwise use a VPN.

Permission Errors

If you're experiencing permission errors while accessing files or directories, it's usually due to improper file access rights or user configurations.

Check file permissions:

ls -l /path/to/file

This will show the file's permissions, notably read (r) and write (w) permissions.

Slow Transfers

Use the Right Tool for Large Transfers

Transferring many files with scp can be slow and inefficient.

Better Options:

  • rsync (for Mac/Linux): Efficiently syncs directories and resumes interrupted transfers. It only sends the parts of files that have changed.
  • robocopy (for Windows): A built-in command-line tool optimized for large, reliable transfers.