README.md 2.15 KB
Newer Older
1
Prototype of a gitlab-runner using a custom executor to run CI jobs on hemera cluster with Slurm.
Pape, David (FWCC) - 139658's avatar
Pape, David (FWCC) - 139658 committed
2
3


Pape, David (FWCC) - 139658's avatar
Pape, David (FWCC) - 139658 committed
4
# Instructions
5

Pape, David (FWCC) - 139658's avatar
Pape, David (FWCC) - 139658 committed
6
## Usage in .gitlab-ci.yml
Pape, David (FWCC) - 139658's avatar
Pape, David (FWCC) - 139658 committed
7
8

- CI jobs that are supposed to run on the cluster must be tagged with `hemera`
9
10
11
- Pass options to the Slurm job by adding them to your job as variables and prepending "CI_SLURM_"
  to their name. Options must be all upper case, hyphens become underscores. Currently supported
  options are:
12
13
14
15
16
17
    - CI_SLURM_NODES
    - CI_SLURM_NTASKS
    - CI_SLURM_NTASKS_PER_NODE
    - CI_SLURM_CPUS_PER_TASK
    - CI_SLURM_PARTITION
    - CI_SLURM_GRES
Pape, David (FWCC) - 139658's avatar
Pape, David (FWCC) - 139658 committed
18

Pape, David (FWCC) - 139658's avatar
Pape, David (FWCC) - 139658 committed
19
20
### Example 1: Run a hybrid MPI/OpenMP job

21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
```yml
run_mpi_omp:
    stage: deploy
    dependencies:
        - build_mpi_omp
    tags:
        - hemera
    variables:
        CI_SLURM_NODES: 2
        CI_SLURM_NTASKS_PER_NODE: 4
        CI_SLURM_CPUS_PER_TASK: 8
    script:
        - export OMP_NUM_THREADS=8
        - mpirun ./mpi_omp
```
Pape, David (FWCC) - 139658's avatar
Pape, David (FWCC) - 139658 committed
36

Pape, David (FWCC) - 139658's avatar
Pape, David (FWCC) - 139658 committed
37
38
### Example 2: Run a CUDA job on the GPU partition

39
40
41
42
43
44
45
46
47
48
49
50
51
```yml
run_cuda:
    stage: deploy
    dependencies:
        - build_cuda
    tags:
        - hemera
    variables:
        CI_SLURM_PARTITION: "gpu"
        CI_SLURM_GRES: "gpu:1"
    script:
        - ./cuda
```
52

Pape, David (FWCC) - 139658's avatar
Pape, David (FWCC) - 139658 committed
53
## Runner setup
Pape, David (FWCC) - 139658's avatar
Pape, David (FWCC) - 139658 committed
54

55
- Run `./setup.sh` to symlink the runner configuration file to the apropriate place. You will be
Pape, David (FWCC) - 139658's avatar
Pape, David (FWCC) - 139658 committed
56
57
  warned if the gitlab-runner executable (which you will need when working with artifacts) is not in
  your path.
Pape, David (FWCC) - 139658's avatar
Pape, David (FWCC) - 139658 committed
58
- Start the runner: `gitlab-runner run:working-directory`
59
- Register runners to your project: `gitlab-runner register`
Pape, David (FWCC) - 139658's avatar
Pape, David (FWCC) - 139658 committed
60
61
62
63
64
65
66
  You'll need one runner with the tag "hemera" and one without.
- Edit the runner-config.toml and place the tokens of the newly generated configuration blocks in
  the two old blocks and delete the rest of the new ones.

This process is necessary when executing the runner as a non-privileged user.


Pape, David (FWCC) - 139658's avatar
Pape, David (FWCC) - 139658 committed
67
# Code overview
Pape, David (FWCC) - 139658's avatar
Pape, David (FWCC) - 139658 committed
68

69
70
71
72
73
74
- **setup.sh:** creates a symlink for the configuration file
- **runner-config.toml.example:** example configuration of the gitlab-runner
- **base.sh:** contains variables commonly used by the other scripts
- **config.sh:** is executed in the config stage
- **run.sh:** is executed in the run stage
- **cleanup.sh:** is executed in the cleanup stage