Overview

NCC has several GPU nodes. NCC is reachable from within the Durham network via

ncc1.clients.dur.ac.uk

The best machine to compile on is cpu6.

Compiling llvm for cuda offload

We need to do a bootsrap procedure, that is, we first build clang et al using a system gcc and in a second build step we use that clang to build also libopenmptarget. We need to build with options such that all nvidia devices on NCC are supported. Also, some flags need to be set for CMake to find some headers in non-standard locations.

First build step

module load cuda/11.4
mkdir -p build1
cd build1
 
export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu:$LD_LIBRARY_PATH

CFLAGS=-I/usr/include/x86_64-linux-gnu CXXFLAGS=-I/usr/include/x86_64-linux-gnu cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$(pwd)/../install -DCLANG_OPENMP_NVPTX_DEFAULT_ARCH=sm_61 -DCLANG_DEFAULT_CXX_STDLIB=libc++ -DLLVM_TARGETS_TO_BUILD="X86;NVPTX" -DLIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES="61;70;75" -DLLVM_ENABLE_PROJECTS="clang;compiler-rt;libcxx;libcxxabi;openmp;libunwind;clang-tools-extra;ldd" -DLIBOMPTARGET_ENABLE_DEBUG=YES -LIBOMPTARGET_NVPTX_DEBUG=YES  -DCMAKE_C_COMPILER=gcc-10 -DCMAKE_CXX_COMPILER=g++-10 ../llvm-13.0.0.src
 
make -j install

Second build

cd ..
mkdir -p build2
cd build2

cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$(pwd)/../install -DCLANG_OPENMP_NVPTX_DEFAULT_ARCH=sm_61 -DLIBOMPTARGET_NVPTX_COMPUTE_CAPABILITIES="61;70;75" -DCLANG_DEFAULT_CXX_STDLIB=libc++ -DLLVM_TARGETS_TO_BUILD="X86;NVPTX" -DLLVM_ENABLE_PROJECTS="clang;compiler-rt;libcxx;libcxxabi;openmp;libunwind;clang-tools-extra;lld" -DLIBOMPTARGET_ENABLE_DEBUG=YES -LIBOMPTARGET_NVPTX_DEBUG=YES -DCMAKE_C_COMPILER=$(pwd)/../install/bin/clang -DCMAKE_CXX_COMPILER=$(pwd)/../install/bin/clang++  ../llvm-13.0.0.src

make install -j

Using the compiler

To set up the environment some variables must be set. PATH must include the new clang and clang++ binaries, LD_LIBRARY_PATH must contain the newly installed libs.

module load cuda/11.4
clang++ test_omp.cxx -Wall -g -std=c++17 -fopenmp=libiomp5 -fopenmp-targets=nvptx64 --cuda-path=/apps/cuda/cuda-11.4 -Xopenmp-target  -march=sm_61

Note that the switches -Xopenmp-target -march=sm_61 set the build image to be compatible with cuda devices of compute capability 6.1, such as the Titan Xp cards on gpu2.

Debug info

Debug info can be obtained with environment variables prepended to the command you are running:

 LIBOMPTARGET_INFO=7 ./a.out
 LIBOMPTARGET_DEBUG=1 ./a.out

Addendum

This is how I download and prepare the source code environment:

wget https://github.com/llvm/llvm-project/releases/download/llvmorg-13.0.0/openmp-13.0.0.src.tar.xz  
wget https://github.com/llvm/llvm-project/releases/download/llvmorg-13.0.0/llvm-13.0.0.src.tar.xz      
wget https://github.com/llvm/llvm-project/releases/download/llvmorg-13.0.0/clang-13.0.0.src.tar.xz 
wget https://github.com/llvm/llvm-project/releases/download/llvmorg-13.0.0/clang-tools-extra-13.0.0.src.tar.xz tar 
wget https://github.com/llvm/llvm-project/releases/download/llvmorg-13.0.0/compiler-rt-13.0.0.src.tar.xz
wget https://github.com/llvm/llvm-project/releases/download/llvmorg-13.0.0/libcxx-13.0.0.src.tar.xz  
wget https://github.com/llvm/llvm-project/releases/download/llvmorg-13.0.0/libcxxabi-13.0.0.src.tar.xz 
wget https://github.com/llvm/llvm-project/releases/download/llvmorg-13.0.0/libunwind-13.0.0.src.tar.xz 
wget https://github.com/llvm/llvm-project/releases/download/llvmorg-13.0.0/lld-13.0.0.src.tar.xz

for i in *xz;do tar xf $i;done

ln -s clang-13.0.0.src clang
ln -s openmp-13.0.0.src openmp
ln -s clang-tools-extra-13.0.0.src clang-tools-extra
ln -s compiler-rt-13.0.0.src compiler-rt
ln -s lld-13.0.0.src lld
ln -s libunwind-13.0.0.src libunwind
ln -s libcxxabi-13.0.0.src libcxxabi
ln -s libcxx-13.0.0.src libcxx