General remarks

Always compile on the compute nodes! The system gcc 7.4 leads to spurious segfaults. Additional software is available from /glob. MPI and other ways of using multiple nodes has been disabled to prevent abuse from cryptominers.

There is no screen but tmux is available.

module use /glob/module-files/development-tools/tmux

Connection

Connection to the login nodes via ssh requires an access key and adding an entry to ~/.ssh/config:

Host devcloud
User u?????
IdentityFile ~/.ssh/devcloud-access-key-?????.txt
ProxyCommand ssh -T -i ~/.ssh/devcloud-access-key-?????.txt guest@ssh.devcloud.intel.com

Connection can then be established via:

ssh devcloud

To mount (and unmount) a remote directory, sshfs can be used as such: (This is handy especially when looking at profiling data and the like)

[on local machine:]
mkdir remote
sshfs devcloud:/home/u?????/remotedir remote

[unmount:]
fusermount -u remote

GCC

The system gcc is not good. There is a fairly recent one available thorugh modules

module use /glob/module-files/development-tools/gcc
module load 11.2.0

Compute nodes

Submission works via qsub, monitoring via qstat, delete jobs with qdel.

To enable access to the nda nodes, the current environment needs this:

export PBS_DEFAULT=v-qsvr-nda

To get a list of all available nodes and an overview of their capabilities, do:

pbsnodes | grep properties | sort | uniq -c

Here are some example qsub commands:

[Interactive 1hr session on a single node with bog-standard intel gpu]
export PBS_DEFAULT
qsub -I -l nodes=1:gpu:ppn=2 -l walltime=01:00:00 -d .

[Submission of 4hr job to the same kind of node:]
export PBS_DEFAULT
qsub -I -l nodes=1:gpu:ppn=2 -l walltime=04:00:00 -d . submit.sh

# Note the ATS nodes require recompilation due to different software stack
[Interactive 1hr session on a single node with intel ats gpu]
export PBS_DEFAULT=v-qsvr-nda
qsub -I -l nodes=1:ats:ppn=2 -l walltime=01:00:00 -d .

Debugger

gdb-oneapi