Otter Overview

Otter is using the OMP tools interface.

The otf2 branch is the actively developed one:

https://github.com/adamtuft/otter/tree/dev-trace-otf2

Some general info regarding OMPT:

https://openmpcon.org/wp-content/uploads/openmpcon2017/Day2-Session3-Protze.pdf

Also:

https://durhamuniversity.zoom.us/j/4441588340?pwd=amM3UGdnRDl2aXVlMitTWjdZRlNaUT09

We need an LLVM OpenMP runtime. So intel’s latest compiler or clang will do. GNU is out.

Current capabilities

Otter is capable of generating OTF2 traces.

The work process consists of:

  • run the (OpenMP) application as such:
    • source pathto/otter-env.sh
    • OMP_TOOL_LIBRARIES=pathto/libotter.so ./a.out
    • This generates the OTF2 trace in a location defined in otter-env.sh
  • Post-processing:
    • here we have full control of what we want to do.
    • One example is the generation of task graphs
    • The post-processing is done with python

How it works

The LLVM OpenMP runtime reads out OMP_TOOL_LIBRARIES to see if the user wants to hook up something with the OMPT interface.

The runtime looks for the function ompt_initialise in the shared lib.

Call backs

The OMPT standard defines a range of callbacks the user can implement.

The registration of the callbacks within otter was done by Adam Tuft quite elegantly using macros that map events onto enum types. Meaning there is no need to keep track of or update any numbering schemes if additional callbacks are implemented or existing callbacks are removed.

Writing to trace

The best resource for OTF2 are the examples that they ship. There are a few gotchas. First of all, everything is stored in a custom binary format. A trace consists of several definitions and the actual event trace with timestamps and so on.

The tricky bit is string handling. All strings that one would want in the trace info must be known at run time. For example, info strings on what a certain recorded event does must be encoded in otter. Also, OTF2 only deals with integer/enum types. So each info string is internally mapped (again) via macros in a similar fashion that avoids having to keep track of any numbers.

Here is an example that actually writes to the trace:

Post-processing

E.g. ottergraph libary — produces svg images from otf2 traces. All the logic is in the python libary. When processing the otf2 traces, the libary collects bit that belong to e.g. the same parallel region and so on.

python3 -i -m ottergraph pathto/trace.otf2

Limitations

Otter does not know about MPI. In principle this is not difficult to do, as presumably all that is necessary is to run one trace per process. In how far this works with migration of tasks from one rank to another remains to be seen.

Target tasks are not (yet) in.

Installation

Prerequisites

We need

OTF2 installation

tar xzf otf2-2.3.tar.gz
cd otf2-2.3
./configure --prefix=$PWD/local --enable-shared CXX=clang++ CC=clang FC=flang
make install -j

Building Otter

The Makefile must be informed of:

  • location of otf2.h
  • location of libotf.so
  • location of ompt.h

With these adjustments made we can call:

CXX=clang++ CC=clang make -f Makefile.mycopy

Running Otter

Make sure OTF libs are in env. — ldd on libotter…

cd /home/gcgt96/Exa/otter
export LD_LIBRARY_PATH=$PWD/../otf2-2.3/local/lib:$LD_LIBRARY_PATH
source otter-env.sh

OMP_TOOL_LIBRARIES=$PWD/lib/libotter.so ./omp-demo

Analysis with scalasca, cube

To use cubgui, we need to convert the otf trace to cubex. This is best done with scalasca’s scout:

./scout  /PATH/TO/mytrace.otf2

By default (and only if conversion was successful), scout will write the file

/PATH/TO/scout.cubex

Scalasca installation

Scalasca ships a recent version of otf and cubelib, so this is usually sufficient:


wget http://apps.fz-juelich.de/scalasca/releases/scalasca/2.6/dist/scalasca-2.6.tar.gz
tar xzf scalasca-2.6.tar.gz
cd scalasca-2.6
./configure --prefix=$PWD/scalasca-2.6/local
make install -j

Cube installation

This one is a bit more involved and in particular requires a number of QT5 libs and cubelib

wget http://apps.fz-juelich.de/scalasca/releases/cube/4.6/dist/cubelib-4.6.tar.gz
tar xzf cubelib-4.6.tar.gz
cd cubelib-4.6
./configure --prefix=$PWD/local
make install -j
cd ../

# Fedora qt5 devel
sudo dnf install qt5-qtbase-devel qt5-qtsvg-devel qt5-qtwebengine-devel

wget http://apps.fz-juelich.de/scalasca/releases/cube/4.6/dist/cubegui-4.6.tar.gz
tar xzf cubegui-4.6.tar.gz
cd  cubegui-4.6
./configure --prefix=$PWD/local --with-cubelib=/PATH/TO/cubelib-4.6/local
make install -j

BEDE notes

# Software env
module purge
module load cuda/10.1.243 llvm openmpi

# OTF2
cd /nobackup/projects/bddur02
wget http://perftools.pages.jsc.fz-juelich.de/cicd/otf2/tags/otf2-2.3/otf2-2.3.tar.gz
tar xzf otf2-2.3.tar.gz
cd otf2-2.3
./configure --prefix=$PWD/local --enable-shared CXX=clang++ CC=clang FC=flang
make install -j

# Otter
cd ../
git clone https://github.com/adamtuft/otter.git
cd otter
git checkout dev-trace-otf2

# Changes in Makefile:
5c5
< INCLUDE    = -Iinclude -I/opt/otf2/include -I/ddn/data/$(USER)/local/include
---
> INCLUDE    = -Iinclude -I/nobackup/projects/bddur02/otf2-2.3/local/include -I/opt/software/builder/developers/compilers/llvm/11.0.0/1/default/lib/clang/11.0.0/include
8c8
< LDFLAGS    = -Llib/ -L/opt/otf2/lib -L/ddn/data/$(USER)/local/lib -Wl,-rpath=`pwd -P`/lib/,-rpath=/ddn/data/$(USER)/local/lib
---
> LDFLAGS    = -Llib/ -L/nobackup/projects/bddur02/otf2-2.3/local/lib -Wl,-rpath=`pwd -P`/lib/

# Compile step
CXX=clang++ CC=clang make -f Makefile.mycopy