How is Aloe Different from Sol and Agave/Phoenix?
If you are familiar with supercomputing on Sol or Agave/Phoenix, Aloe will feel quite similar. However, due to the security requirements and technological differences, there are several changes between general-use systems like Sol and HIPAA complement systems like Aloe.
Internet Access is Restricted
While Sol and Agave/Phoenix can access the internet the same way any ASU-owned device would, Aloe is part of a restricted network with a tightly controlled list of allowed websites, and most of the internet is blocked.
Partitions and QoS
While Sol has a variety of QoSs and portions for different types of jobs, Aloe uses a simplified Partition and QoS setup. Currently, only the general
partition and normal
QoS exist. These are automatically set as defaults and do not need to be explicitly stated in job submission scripts.
See Requesting Resources on Aloe for more info
There is no /scratch, only /tmp
While Sol and Agave/Phoenix have very large scratch filesystems for high IO computing, Aloe does not. Computing directly from your Home directory or project space is perfectly acceptable as these are designed to be used for computing in this environment. If you have a job that benefits from fast disk access, then you can use the per-job ephemeral storage located at /tmp for faster results.
See Using /tmp in Aloe for more information.
Private Slurm Data
Many aspects of the Slurm scheduler have been marked as private and are not viewable to users. For example, you may see the message: slurm_load_partitions: Access/permission denied
when viewing slurm information, such as the squeue
or myjobs
commands. This is not an error; this is expected behavior. This message can be ignored as it does not limit you from viewing information about your jobs and only restricts viewing overall information about partitions, which has been intentionally restricted.
You are not able to see information about other user's jobs.
While these restrictions are in place, it is still important to keep job names free from any PHI, PII, or anything that could be related to private information
No MPI Support
Aloe is not configured with a high-speed fabric for multinode computations. Parallelization can only be achieved with multicore on a single node.
Non-Accessible Compute Nodes
You are not able to SSH directly to a compute node, even if you have a job running on it. You will receive the error: Access denied by pam_slurm_adopt: you have no active jobs on this node
. This is expected behavior.
Job Notifications
Aloe cannot send email notifications about jobs, such as when they start, end, or fail. Instead, Aloe will send Slack messages directly to you via a bot named “aloe-slurm-bot”. You will need to use the official ASU Slack to receive these notifications. To enable notifications, add the following to your job script:
#SBATCH --mail-type=ALL
Note: MailUser
will always be set to your ASURITE - you cannot change this value.
For general help with Slack, see the official ET Slack page: https://tech.asu.edu/slack
For help with the aloe-slurm-bot, submit a request at https://links.asu.edu/kesc-support.
Web Portal
The Aloe web portal is more restricted than Sol or Agave/Phoenix. For example:
Downloading files is restricted to using Globus
Uploading files is discouraged. See using Globus
The job composer has been disabled
The Active Jobs will only display your jobs regardless of the filter
There is no way to submit support tickets via the web portal. A request can be submitted at https://links.asu.edu/kesc-support
Data Ingress and Egress
Sol and Agave/Phoenix have less restricted ways of transferring data, and allow for ease of sharing data between users in the same group.
In a HIPAA environment, data movement and sharing are governed by strict policies to ensure no breach of confidential information. Per section 6 of the Acceptable Use Policy:
Sensitive data (e.g., ePHI, PII, HIPAA, other restricted forms of data) must remain inside the KESC environment unless explicit permission to transfer data has been granted by the PI and a member of KESC Management.
If data is being sent to a subcontractor or other agent, additional contractual documents such as
a Data Use Agreement or Business Associate Agreement.Data may only be transferred into or out of KE Secure Cloud using approved methods such as Globus, SFTP, and similar authenticated, authorized, and encrypted methods.
Because of this, downloading data from the Web Portal has been restricted.
Please see https://asurc.atlassian.net/wiki/spaces/KESC/pages/1913290767/Data+Storage+and+Transfer for acceptable data transfer methods.
Additional information on transferring data can be found in the Data Management and Storage document.
If you have questions about how to move your data in or out of KESC, please submit a request at https://links.asu.edu/kesc-support.
If you suspect a data leak of PII leak has occurred, report it immediately to KESC staff!
File Permissions
Aloe uses NFS 4 with FACLs (File Access Control Lists) for all networked storage. This includes /home
, /globus
, /projects
, and /rc/packages
. These directory permissions may appear with as the owner or group as ke_ldap_app
or domain users
. The NFS FACL may not appear to translate to POSIX style permissions (ie. drwxrwx---
or mode 770) as one might expect. Regardless of what POSIX permissions say, the enforced permissions are of the NFS FACL. For an accurate view of the FACL permissions, use the command nfs4_getfacl
For example:
[jeburks2@ac001 ~]$ nfs4_getfacl ~/testjob.sh
# file: /home/jeburks2/testjob.sh
A::OWNER@:rwadxtTnNcy
For more information about NFS FACLs, see these recommended readings:
https://www.osc.edu/book/export/html/4523
https://linux.die.net/man/5/nfs4_acl
This is a secure environment and it should go without saying but do NOT set files or directories to be readable, writeable, executable, traversable, or in any other way accessed by another user. Home directory permissions are inherited by files, there should be no need to use chmod
except for enabling a file to be executable or to further restrict permissions.
If you suspect a data leak of PII leak has occurred, report it immediately to KESC staff!