Caution: I began taking these notes around Step 6, and had to go back and remember the previous ones. It’s possible I’ve forgotten to include something before that point.
ssh userID@klone.hyak.uw.edu. If you do so from
RStudioServer, you may want to create an alias for this. In the file
.ssh/config, add the text below; then you can just use
ssh klone. You probably already have comparable text for
mox:Host klone klone.hyak.uw.edu
User userID
HostName klone.hyak.uw.edu
ControlPath ~/.ssh/master-%r@%h:%p
ControlMaster auto
ControlPersist yes
Compression yesRe-create all your aliases. These are in
~/.bashrc. You will want to edit them to point to klone
instead of mox, of course.
Learn about the directory structure. A good place for more info is: https://hyak.uw.edu/docs/storage/gscratch/. Basically:
Your home directory is at /usr/lusers/userID on Mox,
but /mmfs1/home/userID on Klone. If you have any aliases
pointing here, you’ll need to update them.
On mox you had gscratch drives for yourself and within your org,
i.e. /gscratch/home/userID and
/gscratch/csde/userID. On klone, there is no
/gscratch/home; it’s just by org, so
/gscratch/csde/userID.
Note that /gscratch is mirrored at /mmfs1/gscratch/. My guess is that this is done to allow some relative paths that worked on mox to keep doing so on klone.
Re-build your project’s directory structure. I did this manually, since mine isn’t that complicated.
Copy up all your project files. Your aliases will likely help you here.
Set up the loadR.sh file. This is
in your home directory. It fill need some different content than the
version on mox, including the syntax for how R is loaded. Here is the
version from Adrien. You should be able to change the version of R to
any that is built. Note that you are not yet running this.
#!/bin/bash
. /gscratch/csde/spack_alg/share/spack/setup-env.sh
spack unload -a
spack load r@4.3Note for reference: an earlier version of this file contained
#!/bin/bash
. /gscratch/csde/spack/spack/share/spack/setup-env.sh
spack unload -a
spack load r@4.1.1
spack load gitMake note of the fact that there are no build nodes on
Klone! This means that you will need to use a compute node to
do the things you used to do on a build node. This means that you’ll
want to replace your build alias with a comparable
compute one:
alias compute='srun -p compute --time=3:00:00 --mem=20G --pty /bin/bash'
Find the compiler and copy the file. Before you
can load R the first time, you need to tell Spack to find the compiler.
You only need to do this once on Klone. Load spack with
. /gscratch/csde/spack/spack/share/spack/setup-env.sh and
then type spack compiler find. This should give you the
path to a .yaml file; copy it to ~/.spack/linux/compilers.yaml. This may
not work (it didn’t for Steve G). If not, you can grab the file manually
from a colleague’s directory
(e.g. mmfs1/home/goodreau/.spack/linux/compilers.yaml) and copy it to
~/.spack/linux/compilers.yaml.
Find the available versions of R with
spack find r. Edit your loadR.sh file to use the version
you prefer.
Open R. If you haven’t loaded spack yet this
session, type
. /gscratch/csde/spack/spack/share/spack/setup-env.sh).
Then spack load r@4.1.1, using whichever version number you
prefer.
Set your Github PAT. There are multiple ways to
do this, but I find the easiest is to add the following line directly
into a file .Renviron in your home directory:
GITHUB_PAT=xxxxxxxxxxxxxxxxxxxxxxxxxxxx.
Set up your R packages, using renv or otherwise. This should all be one the same way you would do it on Mox.
Optional add some additional useful aliases.
Drawn from here,
I use
alias lspack='. /gscratch/csde/spack/spack/share/spack/setup-env.sh'
and
alias lr='lspack; spack load git; spack load r@4.1.1; R --no-save --no-restore'.
Learn klone’s added functionality. The first one
I’ve discovered is the very useful command
hyakstorage.
**Edit your calls to sbatch_master. This is
necessary when ckpt=TRUE. On mox, the text in
master.xxxx.sh begins sbatch -p csde -A csde-ckpt, whereas
on klone it should being sbatch -p compute -A csde. To make
the change, add the following two arguments to a
sbatch_master call after ckpt = TRUE:
partition.ckpt = "compute",
account.ckpt = "csde",Here is a description of each file:
## loadR.sh
This files loads all the component required to run R and it's dependencies. Because on the HPC R is not really installed, we need to tell it where R is. In our case we use Spack to manage different R version.
## setup-env.sh
This is the utility script that actually loads Spack. The HPC does not have spack installed either so we needed to download it and then this script populate the `PATH` so you can run functions like `spack load` etc.
## compilers.yaml
Spack allows the creation of reproducible environment. To do so, it will build everything from source. But because it's very thourough, it will actually also record which compilers have been used for which package.
This files record all the compilers known to `spack`. There are the default ones (provided by the HPC) and also the ones installed by `spack` itself when running `spack install gcc@12.0` for example.
More on that below
## . /gscratch/csde/spack_alg/share/spack/setup-env.sh
So previously we used `. /gscratch/csde/spack/spack/share/spack/setup-env.sh` and now I tend to use `. /gscratch/csde/spack_alg/share/spack/setup-env.sh`.
As the *alg* in the second one suggest, it has been installed by me.
I initially installed it to play around with `spack` on my own but in the end used it for analysis because I encountered a similar error as you with the other. The problem with the other version is that it is *owned* by Sam, so I cannot fix it myself.
Also, on this new version I massively simplified the setup by simply using the `gcc` version provided by the system and not ones installed by spack.
If you compare the 2 *compilers.yaml* files, you can see that mine contains only `gcc@8.5`. Whereas the other one as I think at least 5 compilers, each depending on the previous one. And the first one, `gcc@8.3.0` is not available on the system since the migration from CentOS to Rocky linux.