How is Aloe Different from Sol and Agave/Phoenix?

If you are familiar with supercomputing on Sol or Agave/Phoenix, Aloe will feel quite similar. However, due to the security requirements and technological differences, there are several changes between general-use systems like Sol and HIPAA complement systems like Aloe.

Internet Access is Restricted

While Sol and Agave/Phoenix can access the internet the same way any ASU-owned device would, Aloe is part of a restricted network with a tightly controlled list of allowed websites, and most of the internet is blocked.

Partitions and QoS

While Sol has a variety of QoSs and portions for different types of jobs, Aloe uses a simplified Partition and QoS setup. Currently, only the general partition and normal QoS exist. These are automatically set as defaults and do not need to be explicitly stated in job submission scripts.

See Requesting Resources on Aloe for more info

There is no /scratch, only /tmp

While Sol and Agave/Phoenix have very large scratch filesystems for high IO computing, Aloe does not. Computing directly from your Home directory or project space is perfectly acceptable as these are designed to be used for computing in this environment. If you have a job that benefits from fast disk access, then you can use the per-job ephemeral storage located at /tmp for faster results.

See Using /tmp in Aloe for more information.

Private Slurm Data

Many aspects of the Slurm scheduler have been marked as private and are not viewable to users. For example, you may see the message: slurm_load_partitions: Access/permission denied when viewing slurm information, such as the squeue or myjobs commands. This is not an error; this is expected behavior. This message can be ignored as it does not limit you from viewing information about your jobs and only restricts viewing overall information about partitions, which has been intentionally restricted.

You are not able to see information about other user's jobs.

While these restrictions are in place, it is still important to keep job names free from any PHI, PII, or anything that could be related to private information

No MPI Support

Aloe is not configured with a high-speed fabric for multinode computations. Parallelization can only be achieved with multicore on a single node.

Non-Accessible Compute Nodes

You are not able to SSH directly to a compute node, even if you have a job running on it. You will receive the error: Access denied by pam_slurm_adopt: you have no active jobs on this node. This is expected behavior.

Job Notifications

Aloe cannot send email notifications about jobs, such as when they start, end, or fail. Instead, Aloe will send Slack messages directly to you via a bot named “aloe-slurm-bot”. You will need to use the official ASU Slack to receive these notifications. To enable notifications, add the following to your job script:

#SBATCH --mail-type=ALL

For general help with Slack, see the official ET Slack page: https://tech.asu.edu/slack

For help with the aloe-slurm-bot, submit a request at https://links.asu.edu/kesc-support.

Web Portal

The Aloe web portal is more restricted than Sol or Agave/Phoenix. For example:

• Downloading files is restricted to using Globus

• Uploading files is discouraged. See using Globus

• The job composer has been disabled

• The Active Jobs will only display your jobs regardless of the filter

• There is no way to submit support tickets via the web portal. A request can be submitted at https://links.asu.edu/kesc-support

Data Ingress and Egress

Sol and Agave/Phoenix have less restricted ways of transferring data, and allow for ease of sharing data between users in the same group.

In a HIPPA environment, data movement and sharing are governed by strict policies to ensure no breach of confidential information. Per section 6 of the Acceptable Use Policy:

Sensitive data (e.g., ePHI, PII, HIPAA, other restricted forms of data) must remain inside the KESC environment unless explicit permission to transfer data has been granted by the PI and a member of KESC Management.

If data is being sent to a subcontractor or other agent, additional contractual documents such as
a Data Use Agreement or Business Associate Agreement.

Data may only be transferred into or out of KE Secure Cloud using approved methods such as Globus, SFTP, and similar authenticated, authorized, and encrypted methods.

Because of this, downloading data from the Web Portal has been restricted.

Please see https://asurc.atlassian.net/wiki/spaces/KESC/pages/1913290767/Data+Storage+and+Transfer for acceptable data transfer methods.

Additional information on transferring data can be found in the Data Management and Storage document.

If you have questions about how to move your data in or out of KESC, please submit a request at https://links.asu.edu/kesc-support.

File Permissions

Aloe uses NFS 4 with FACLs (File Access Control Lists) for all networked storage. This includes /home, /globus, /projects, and /rc/packages. These directory permissions may appear with as the owner or group as ke_ldap_app or domain users. The NFS FACL may not appear to translate to POSIX style permissions (ie. drwxrwx--- or mode 770) as one might expect. Regardless of what POSIX permissions say, the enforced permissions are of the NFS FACL. For an accurate view of the FACL permissions, use the command nfs4_getfacl

For example:

[jeburks2@ac001 ~]$ nfs4_getfacl ~/testjob.sh 
# file: /home/jeburks2/testjob.sh
A::OWNER@:rwadxtTnNcy

For more information about NFS FACLs, see these recommended readings:

https://www.osc.edu/book/export/html/4523

https://linux.die.net/man/5/nfs4_acl