Google Drive with Globus

Owned by Johnathan Lee, created
Last updated: May 15, 2024 by rahull1
3 min read

Overview

The University Technology Office provides all staff, faculty, and students storage on the cloud via Google Drive.

Using Globus, the process of archiving or moving files and data from the ASU supercomputer to Google Drive’s cloud becomes a browser-enabled task.

Before documenting how to transfer files, please note some of the stipulations of Google Drive:

  1. Shared Drives should be used for each research initiative. Place a README in the root of each created team Drive for documentation purposes.

    1. Shared drives may not exceed 400,000 files (and folders), which can be a very unforgiving limit for certain workflows.

    2. Google recommends creating team Drives for organizing on the “project and team” level rather than for general filesystem use.

    3. Up to 1,000 shared Drives will be navigable from within the Google Drive web interface. Beyond that, a direct link must be used to access unlisted shared drives.

  2. Uploads are limited to 750 GB per user per day. Transferred files cannot exceed 5 TB (individually).

    1. Users' Drives will not sync nor upload once this daily limit is reached.

  3. Google Drive rate throttles on the file level. These rules will restrict rclone to transferring about 2 new files per second only. Uploading hundreds of small files will take longer than a few large files.

Note the implication of point (2) – Google Drive will allow a file that has begun transferring to finish transferring so long as it does not exceed 5 TB. This further implies that a user may upload significant amounts of per day by starting the transfer of multiple large files. For data that are being archived, this suggests that tarballing may ideal, especially when considering the implications of point (3). However by point (2a), this will prevent a user from uploading or syncing their Google Drive until the day usage quota resets.

Setting up a Shared Drive

Out of the box, no configuration has to be done on the Google Drive side to begin uploading to a user’s Drive. However, it is highly recommended that Shared Drives are created for each research computing initiative. Following Google's documentation on Shared Drive creation:

  1. Open Google Drive and sign in with ASU-SSO (e.g. signing in with asurite@asu.edu)

  2. On the left, click Shared Drives

  3. At the top, click the stylized “+ New”

  4. Enter a name and click “CREATE”

  5. Add members to the Shared Drive, such as collaborators and P.I.

Configuring & Using Globus

This configuration assumes that your research directories have collections associated with them (the help docs for that may be found here).

Once the Shared Drive is initialized, it may be accessed from Globus. Note that the default user Drive, “My Drive” is always accessible from Globus without any additional configuration. This is also true for Drives that are shared with users.

Please reference the “Configuring Globus Collections” documentation for guidance on configuring and using Globus with Google Drive (as well as the four other collection types).

Additional Help

Arizona State University