Home Directory Cleanup Guide

Owned by Josh Burks, created
Last updated: Nov 26, 2024
3 min read

Overview

Home directories on ASU supercomputers have a 100 GiB limit, and unfortunately, there’s no option to expand them. For additional storage, we recommend using scratch or project storage.

If your home directory becomes full, you’ll encounter errors such as:

No space left on devices

When this happens, you may also be unable to create new sessions from the web portal. However, SSH access is not affected by home directory capacity, and all the commands in this guide assume you are working via SSH. For help connecting with SSH, refer to our Connecting to the Supercomputers with SSH guide.

When this happens, you’ll need to either delete unnecessary files or move them to a different storage system. Here’s how to check for space usage and address common issues.

Identifying Large Files and Directories

To see which files or directories are taking up space, use the command:

gdu ~

gdu (Go Disk Usage) is a fast and user-friendly tool for analyzing disk usage. Once in gdu:

  • Use the arrow keys to navigate up and down the list or into directories.

  • Press d to delete a file or directory directly from the interface.

gdu --help Pretty fast disk usage analyzer written in Go. Usage: gdu [directory_to_scan] [flags] Flags: --config-file string Read config from file (default is $HOME/.gdu.yaml) -g, --const-gc Enable memory garbage collection during analysis with constant level set by GOGC --enable-profiling Enable collection of profiling data and provide it on http://localhost:6060/debug/pprof/ -L, --follow-symlinks Follow symlinks for files, i.e. show the size of the file to which symlink points to (symlinks to directories are not followed) -h, --help help for gdu -i, --ignore-dirs strings Absolute paths to ignore (separated by comma) (default [/proc,/dev,/sys,/run]) -I, --ignore-dirs-pattern strings Absolute path patterns to ignore (separated by comma) -X, --ignore-from string Read absolute path patterns to ignore from file -f, --input-file string Import analysis from JSON file -l, --log-file string Path to a logfile (default "/dev/null") -m, --max-cores int Set max cores that GDU will use. 8 cores available (default 8) -c, --no-color Do not use colorized output -x, --no-cross Do not cross filesystem boundaries --no-delete Do not allow deletions -H, --no-hidden Ignore hidden directories (beginning with dot) --no-mouse Do not use mouse --no-prefix Show sizes as raw numbers without any prefixes (SI or binary) in non-interactive mode -p, --no-progress Do not show progress in non-interactive mode -u, --no-unicode Do not use Unicode symbols (for size bar) -n, --non-interactive Do not run in interactive mode -o, --output-file string Export all info into file as JSON -r, --read-from-storage Read analysis data from persistent key-value storage --sequential Use sequential scanning (intended for rotating HDDs) -a, --show-apparent-size Show apparent size -d, --show-disks Show all mounted disks -C, --show-item-count Show number of items in directory -M, --show-mtime Show latest mtime of items in directory -B, --show-relative-size Show relative size --si Show sizes with decimal SI prefixes (kB, MB, GB) instead of binary prefixes (KiB, MiB, GiB) --storage-path string Path to persistent key-value storage directory (default is /tmp/badger) (default "/tmp/badger") -s, --summarize Show only a total in non-interactive mode --use-storage Use persistent key-value storage for analysis data (experimental) -v, --version Print version --write-config Write current configuration to file (default is $HOME/.gdu.yaml)

For more details on gdu, visit its GitHub page.

Alternatively, you can use:

ncdu ~

ncdu (ncurses Disk Usage) offers similar functionality, though it may run a bit slower.

Common Space-Consuming Applications and Solutions

Mamba

mamba, the Python package manager, stores temporary files, cached tarballs, and environments in your home directory. To free up space:

  1. Remove any unused environments:

  2. Clean up unnecessary packages, tarballs, and cache:

pip

We recommend using mamba over pip for Python package management whenever possible. For more information, refer to our Python Package Installation Comparison.

However, if you must use pip, it often creates cache files in ~/.cache/pip. These files can be safely removed with:

Hugging Face

Hugging Face libraries frequently download large models to your home directory, storing them in ~/.cache/huggingface by default. To free up space and relocate these files to scratch storage:

  1. Move the cache directory:

  2. Update your shell to reflect this change:

  3. Reload your shell:

     

 

Additional Help

Arizona State University