Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

Version 1 Next »

Overview

There are many different ways to access or transfer your data from your ASU Research Computing (ASURC) account from or to elsewhere. One very convenient way is to use Globus.

Before Globus may be used, certain objects affiliated with your account and linked to paths on the filesystem, called collections, must be created. These collections are generated from ASURC “Managed Mapped Collections.”

Currently, five classes of collections may be created. These collections then allow for transferring data to or from your Agave Home directory, Scratch directory, project-based storage directory, legacy mounted storage directory, or your /wiki/spaces/RC/pages/201490433

This page outlines the steps necessary to create the collections from the five managed endpoints.

Globus may make subtle changes to the setup process that date this configuration guide. We host workshops for setting up Globus every two months (Educational Opportunities and Workshops) and are more than happy to schedule a one-on-one meeting to assist in the setup. Our weekly office hours are also an excellent resource for any technical assistance.

Configuring Globus

  1. Sign into globus.org using ASU CILogin

  2. Click Collections from the left navigation bar

  3. Search all endpoints for ASURC Agave. This involves filling out a search bar in the upper right of the web view.

  4. The five relevant managed mapped collection results will come up like in figure 1 below, and clicking on a collection will produce a new webview detailing the specific collection. Note that the Note that the figure and this text are hyperlinks to the search results.

     

  5. After selecting a managed mapped collection, view, select the Collections tab, and then select the Add a Guest Collection button from the interface. The following subitem names are shortcut links directly to the new collection creation pages:

    1. ASURC Agave Scratch: for access to /scratch mounts

    2. ASURC Agave Home: for access to /home mounts

    3. ASURC Agave Google Drive: for access to ASU provided Google drives

    4. ASURC Agave Data Directories: for access to /data mounts (project-based storage)

    5. ASURC Agave Mnt Directories: for access to legacy /mnt mounts

  6. For each collection, only two fields have to be supplied:

    1. the base directory: see the specific instructions section below (tl;dr: /<username for home and scratch, / for Google drive, and be specific for the data and legacy mount collections).

    2. a display name: <username>-agave-<collection_type> is a recommend style, where <username> should be replaced by your username on the Agave system (likely your asurite) and <collection_type> should be replaced by either scratch, home, drive, data, or legacymnt.

    3. additionally, you’ll need to follow the on-screen instructions to provide Globus with formal permission to access your files and Google drive.

Specific instructions for “ASURC Agave Scratch” and “ASURC Agave Home”

For the base directory, specify /<username> , where <username> should be replaced by your username on the Agave system (likely your asurite).

Specific instructions for “ASURC Agave Google Drive”

For the base directory, specify / , which will allow you to access your entire Google drive filesystem from within Globus, including your “My Drive” and “Team Drives” (shared drives).

Specific instructions for “ASURC Agave Data Directories” and the legacy mount

For the base directory, specify /<mount-name> . For instance, if your lab has 50 TB of space in /data/mylab, you may connect it with /mylab specified as the base directory.

Post Configuration

Once created, a user may find their own collections through a variety of means, including through the File Manager view and Bookmarks view.

Finding Collections in the Bookmarks view

  1. Navigate to the Bookmarks view by selecting the Bookmarks icon from the left navigation bar.

  2. Select the Collections tab.

Finding Collections in the File Manager view

  1. Navigate to the File Manager view by selecting the File Manager icon from the left navigation bar.

  2. Click within the uppermost search bar, as indicated by “Collection” and a magnifying glass icon followed by the word, “Search”.

  3. Find your collection within the Collections tab.

Time for Transfers

  1. Select two collections within the File Manager view.

  2. Find the vertical menu item Transfer or Sync to...

    1. If just a single collection is active (single pane view), the option will be on the right of the screen

    2. If two collections are active in a dual pane view, the option will be in between the collections

  3. Select the files/directories from you want to transfer (single click to select, double click to navigate)

  4. Select the Transfer & Sync Options (figure 2)

    Figure 2, a screenshot of the Transfer and Sync Options button

  5. SELECT OR VERIFY that sync - only transfer new or changed files is enabled (figure 3)

    Figure 3, a screenshot of the expanded Transfer and Sync Options, detailing that the sync sub option should be selected

  6. Click Start -> to transfer from the “left” pane collection to the “right” or <- Start for the opposite.

  7. Jump for Joy (optional)

Failing to ensure that the transfer is a sync operation may result in files that were already transferred being re-transferred to the destination. Enabling sync pragmatically improves transfer efficiency.

While transferring (syncing) data, do not delete or move files or directories out of the host filesystem. Such actions may slow an active transfer significantly. At the start of a transfer, Globus cultivates a list of files and directories to transfer from host to target. When paths within that list no longer exist, Globus throws a warning and seemingly loops in an attempt to find that path. If filesystem changes must be made, wait for the transfer to finish (or stop the transfer), make the filesystem changes, and then resubmit the transfer as a sync.

Advanced Technical Notes

Globus provides a command-line tool that is powered by python. Instructions for the Globus command-line tool are provided here (https://docs.globus.org/cli/), but please use pip install globus-cli from within an Anaconda environment /wiki/spaces/RC/pages/125829137.

 

Additional Help

If you require further assistance on this topic, please don't hesitate to contact the Research Computing Team. To create a support ticket, kindly send an email to rtshelp@asu.edu. For quick inquiries, you're welcome to reach out via our #rc-support Slack Channel or attend our office hours for live assistance

We also offer a series of workshops. More information here: Educational Opportunities and Workshops

  • No labels

Arizona State University