Commit ca937832 authored by Frere, Jonathan (FWCC) - 142176's avatar Frere, Jonathan (FWCC) - 142176
Browse files

Apply 18 suggestion(s) to 1 file(s)

parent 09e0d733
Pipeline #42012 passed with stages
in 7 minutes and 33 seconds
......@@ -31,7 +31,7 @@ how do we get other people to download, and do something useful with, our Docker
Likewise, how do we use our Docker image in a wider set of circumstances, for example on HPC systems, or a shared team VM?
This post is more specific to looking at scientific applications,
so we'll look mainly at using Gitlab and Singularity for these purposes,
so we'll look mainly at using GitLab and Singularity for these purposes,
as these are the most commonly-used tools in scientific computing.
# Sharing Docker Images
......@@ -45,7 +45,7 @@ as well as the host of a number of unofficial, third-party images (`jupyter/scip
If you're working with truly open-source code, and want to share that with the wider Docker ecosystem,
you can create an account and upload images here in the same way that you might host Python packages on PyPI.
However, in practice, a lot of scientific programming is hosted internally via your institution's own git host,
However, in practice, a lot of scientific programming is hosted internally via your institution's own Git host,
and most scientific applications are fairly specific,
and probably not of huge use outside the purpose that they were developed for.
......@@ -56,10 +56,10 @@ For the purposes of this post, I'll assume the use of GitLab, because is one of
When enabled by your administrator, GitLab projects include a private container registry for each project.
You can set it up for your project by going to _Settings > General > Visibility, project features, permissions_.
This will enable a "Packages and Registries > Container Registry" option in the project sidebar, which will take you to an empty page,
because you probably don't have any images stored yet!
because you probably don't have any images stored yet.
How do you store an image here?
Let's start off by doing it manually, and then do it "properly" -- by which I mean get it to happen automatically!
Let's start off by doing it manually, and then do it "properly" -- by which I mean get it to happen automatically.
If you want to follow along, create a new private repository that you can use as a sandbox,
and push the code from the previous post to play around with.
......@@ -96,10 +96,10 @@ However, the project name used by GitLab is different to the one we used then --
why?
Well, in GitLab at least, the name of your image is a combination of the project name, and the group or user who owns that project.
For example, if your username is bauer34, and you create a project called `docker-test` inside your personal area in GitLab,
your image will be called `bauer34/docker-test`.
For example, if your username is "user123", and you create a project called `docker-test` inside your personal area in GitLab,
your image will be called `user123/docker-test`.
In addition, Docker requires that if you use a registry that isn't the main Docker registry, you specify that registry as part of the image name.
So, in this case, you'll actually need to name your image `<registry-url>/bauer34/docker-test`,
So, in this case, you'll actually need to name your image `<registry-url>/user123/docker-test`,
where `<registry-url>` is whatever you used to log in in the previous step.
This isn't a problem at all -- we can just run the build again, and because of Docker's clever caching mechanism,
......@@ -112,9 +112,9 @@ When this is done, we should be able to refresh the container registry page, and
with a little note that says "1 Tag".
Clicking on the image will show that tag -- it should be the `latest` tag that Docker always defaults to if none is specified.
To upload a different tag, just specify it at the end of the image name --
for example: `registry.hzdr.de/bauer34/docker-test:my-tag`.
for example: `registry.hzdr.de/user123/docker-test:my-tag`.
That was the manual version, and there were a few steps, but it wasn't so complicated.
That was the manual version, and there were a few steps, but it needn't be so complicated.
How does the automatic version compare?
## Saving Images -- The Automatic Process
......@@ -152,8 +152,8 @@ release-latest:
```
This creates a pipeline with two jobs, one which builds the docker image, and one which uploads it to the registry.
If you push this to the master branch, you and click on the _CI/CD > Pipelines_ tab in your GitLab project,
you should already be able to see the build taking place.
If you push this to the master branch and click on the _CI/CD > Pipelines_ tab in your GitLab project,
you should already be able to see the pipeline being executed.
The documentation for this template is available [here](https://gitlab.com/hifis/templates/gitlab-ci/-/blob/master/docs/docker.md).
......@@ -170,7 +170,7 @@ GitLab also provides access tokens that can be given to people to allow them the
but not to make other changes.
They don't even need to have a GitLab account!
In a projects settings, under _Settings > Access Tokens_, there is a page where you can create tokens to share with other people.
In a project's settings, under _Settings > Access Tokens_, there is a page where you can create tokens to share with other people.
These tokens are randomly-generated passwords that are linked to a particular project, that specify exactly what a person is able to access.
For the purposes of sharing a Docker image, the `read_registry` permission is enough --
this will allow the bearer of the token to access the registry, but not push new images there, or access other project features.
......@@ -178,7 +178,7 @@ this will allow the bearer of the token to access the registry, but not push new
To create an access token, give the token a name to describe what it's being used for,
select the relevant permissions that you want to grant[^3],
and optionally give an expiry date, if you know that the token will only be needed until a certain time.
In response, GitLab will provide a string of letters, digits, and symbols,
In response, GitLab will provide a string of letters, digits, and other special characters,
which can be copied and sent to the people who need to use it.
To use this token, use the `docker login` command with your normal GitLab username, and the token provided.
......@@ -225,7 +225,7 @@ There are a number of other tools used to do containerisation,
and one tool particularly that is both designed to run on HPC systems,
_and_ can interoperate with Docker, meaning you can usually just run your Docker image just like normal.
This tool is known as [_Singularity_](https://sylabs.io/guides/3.6/user-guide/introduction.html).
This tool is called [_Singularity_](https://sylabs.io/guides/3.6/user-guide/introduction.html).
It is actually a complete containerisation tool in its own right,
with its own format for defining containers,
and its own way of running containers[^4].
......@@ -254,7 +254,7 @@ In addition, it's possible to use SSH keys for authentication in certain circums
Two reasons:
Firstly, Docker is way more popular than Singularity, or indeed any other similar tool.
This means more documentation,
more starter images to base our own changes off,
more starter images to base our own changes on,
more people to find and fix bugs in the software,
and more support in third-party tools like GitLab.
Secondly, Singularity only runs on Linux,
......@@ -263,7 +263,7 @@ In addition, it's possible to use SSH keys for authentication in certain circums
and compiling the whole project ourselves.
Given that Singularity can run Docker images,
we can use Docker in the knowledge that we can also get the advantages of Singularity later.
we can use Docker in the knowledge that we can also get the advantages of Singularity later on.
# Docker in The Wild
......@@ -285,7 +285,7 @@ For example, a CI job that builds a LaTeX document may find it easiest to use a
In fact, in GitLab, it's even possible to use the registries from other projects to access custom images,
and use those custom images in jobs in other projects.
It's even possible to use jobs to create images to use in other jobs, if that's something that you really need!
It's even possible to use jobs to create images to use in other jobs, if that's something that you really need.
# Conclusion
......@@ -312,7 +312,7 @@ However, when different tools, languages, and package management needs come toge
using Docker can often be a good way to make sure that the system really is well-defined,
for example by ensuring that the right system packages are always installed,
as well as the right Python packages,
and the right R or Julia software as well.
or the right R or Julia software.
If you feel like your project is a spiralling mess of complexity,
and you're not sure what packages need to exist, or how to build it on any computer other than your own,
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment