FWCC issueshttps://codebase.helmholtz.cloud/groups/fwcc/-/issues2023-09-29T10:35:27+02:00https://codebase.helmholtz.cloud/fwcc/statistics-collector/-/issues/52refactor generate_statistics.py including interfaces to metrics2023-09-29T10:35:27+02:00Helmert, Stefan (FWCA) - 142172refactor generate_statistics.py including interfaces to metricsgenerate_statistics.py has a confusing architecture with misleading function namesgenerate_statistics.py has a confusing architecture with misleading function namesHelmert, Stefan (FWCA) - 142172Helmert, Stefan (FWCA) - 1421722023-09-29https://codebase.helmholtz.cloud/fwcc/statistics-collector/-/issues/51Exception if csv file is empty2023-09-08T11:02:08+02:00Helmert, Stefan (FWCA) - 142172Exception if csv file is emptyIf the target csv file is empty, there are 0 columns. It is not possible for `pandas` to read this file. An `EmptyDataError` occurs. So no data is appended. - If the file is empty, the exception must be catched and the file must be newly...If the target csv file is empty, there are 0 columns. It is not possible for `pandas` to read this file. An `EmptyDataError` occurs. So no data is appended. - If the file is empty, the exception must be catched and the file must be newly created.Helmert, Stefan (FWCA) - 142172Helmert, Stefan (FWCA) - 142172https://codebase.helmholtz.cloud/fwcc/statistics-collector/-/issues/49add overleaf statistics2023-10-09T13:27:36+02:00Helmert, Stefan (FWCA) - 142172add overleaf statisticsAdd plugins to access newly implemented overleaf statistics API.Add plugins to access newly implemented overleaf statistics API.https://codebase.helmholtz.cloud/fwcc/statistics-collector/-/issues/40Add Tests2022-05-31T16:48:31+02:00Erxleben, Fredof.erxleben@hzdr.deAdd TestsAdd test cases so that further refactoring does not break what you already achieved.
A basic setup using `pytest` has already been established on the `add-tests` branch, which you can build upon.
**Before you do that:**
1. Finish the ...Add test cases so that further refactoring does not break what you already achieved.
A basic setup using `pytest` has already been established on the `add-tests` branch, which you can build upon.
**Before you do that:**
1. Finish the current refactoring cycle
2. Merge `refactoring` → `main`
3. Merge `main` → `add-tests`
**Finish by doing:**
* Add the run of `pytest` as a dedicated CI-JpbSultova, Neda Miroslavova (FWCC) - 138965Sultova, Neda Miroslavova (FWCC) - 138965https://codebase.helmholtz.cloud/fwcc/statistics-collector/-/issues/39Include pyreverse in CI-pipeline2022-05-25T14:40:50+02:00Erxleben, Fredof.erxleben@hzdr.deInclude pyreverse in CI-pipelineIt would be nice to generate an updated class diagram for documentation after each commit.
For that purpose, include the tool `pyreverse` into the CI-pipeline.It would be nice to generate an updated class diagram for documentation after each commit.
For that purpose, include the tool `pyreverse` into the CI-pipeline.https://codebase.helmholtz.cloud/fwcc/statistics-collector/-/issues/38Introduce intermediate classes to abstract GitlabApiMetric and GitlabQueryMetric2023-09-29T13:17:23+02:00Erxleben, Fredof.erxleben@hzdr.deIntroduce intermediate classes to abstract GitlabApiMetric and GitlabQueryMetricThis can also help to reduce code duplication in the current metric classes.This can also help to reduce code duplication in the current metric classes.Sultova, Neda Miroslavova (FWCC) - 138965Sultova, Neda Miroslavova (FWCC) - 138965https://codebase.helmholtz.cloud/fwcc/statistics-collector/-/issues/37Introduce a class to better deal with GraphQL responses2022-05-25T14:32:05+02:00Erxleben, Fredof.erxleben@hzdr.deIntroduce a class to better deal with GraphQL responsesIt is currently very inconvenient to deal with the responses directly. Maybe introduce a wrapper class or look for existing frameworks.It is currently very inconvenient to deal with the responses directly. Maybe introduce a wrapper class or look for existing frameworks.Sultova, Neda Miroslavova (FWCC) - 138965Sultova, Neda Miroslavova (FWCC) - 138965https://codebase.helmholtz.cloud/fwcc/statistics-collector/-/issues/31Prettify formatting of choice menu2022-05-17T13:53:52+02:00Sultova, Neda Miroslavova (FWCC) - 138965Prettify formatting of choice menuThe following discussion from !1 should be addressed:
- [ ] @erxleb87 started a [discussion](https://gitlab.hzdr.de/fwcc/statistics-collector/-/merge_requests/1#note_1386220): (+1 comment)
> The formatting is a bit off, but the id...The following discussion from !1 should be addressed:
- [ ] @erxleb87 started a [discussion](https://gitlab.hzdr.de/fwcc/statistics-collector/-/merge_requests/1#note_1386220): (+1 comment)
> The formatting is a bit off, but the idea is clever.Sultova, Neda Miroslavova (FWCC) - 138965Sultova, Neda Miroslavova (FWCC) - 138965https://codebase.helmholtz.cloud/fwcc/statistics-collector/-/issues/30Add auth into own class?2022-05-31T17:09:26+02:00Sultova, Neda Miroslavova (FWCC) - 138965Add auth into own class?The following discussion from !1 should be addressed:
- [ ] @erxleb87 started a [discussion](https://gitlab.hzdr.de/fwcc/statistics-collector/-/merge_requests/1#note_1386218):
> Isn't that the same as in the _Active Users_-class? ...The following discussion from !1 should be addressed:
- [ ] @erxleb87 started a [discussion](https://gitlab.hzdr.de/fwcc/statistics-collector/-/merge_requests/1#note_1386218):
> Isn't that the same as in the _Active Users_-class? Should they have a common shared super-class in beyond _Metrics_?Sultova, Neda Miroslavova (FWCC) - 138965Sultova, Neda Miroslavova (FWCC) - 138965https://codebase.helmholtz.cloud/fwcc/statistics-collector/-/issues/27Refactor query operations2023-09-05T08:55:35+02:00Sultova, Neda Miroslavova (FWCC) - 138965Refactor query operationsThe following discussion from !1 should be addressed:
- [ ] @erxleb87 started a [discussion](https://gitlab.hzdr.de/fwcc/statistics-collector/-/merge_requests/1#note_1386215):
> These are the same as in the CI-Jobs class… Why woul...The following discussion from !1 should be addressed:
- [ ] @erxleb87 started a [discussion](https://gitlab.hzdr.de/fwcc/statistics-collector/-/merge_requests/1#note_1386215):
> These are the same as in the CI-Jobs class… Why would anyone duplicate this code? Suspicious… :thinking:Sultova, Neda Miroslavova (FWCC) - 138965Sultova, Neda Miroslavova (FWCC) - 138965https://codebase.helmholtz.cloud/fwcc/statistics-collector/-/issues/24Refactor Graphql Queries2022-06-28T20:54:47+02:00Sultova, Neda Miroslavova (FWCC) - 138965Refactor Graphql QueriesThe following discussion from !1 should be addressed:
- [ ] @erxleb87 started a [discussion](https://gitlab.hzdr.de/fwcc/statistics-collector/-/merge_requests/1#note_1386194):
> The way you build queries here needs some thorough a...The following discussion from !1 should be addressed:
- [ ] @erxleb87 started a [discussion](https://gitlab.hzdr.de/fwcc/statistics-collector/-/merge_requests/1#note_1386194):
> The way you build queries here needs some thorough attention.
>
> * Use string interpolation consistently
> * Construct the segments separately and then concatenate them
> * Are there segments that are fixed and can be re-used?
> * Can queries be represented as objects and is there a way to formalize their translation into a string?
> * Could this also be helpful if you would want to build other queries in the future?
> * Revisit the name of the `build_query`-functionSultova, Neda Miroslavova (FWCC) - 138965Sultova, Neda Miroslavova (FWCC) - 138965https://codebase.helmholtz.cloud/fwcc/HZDR-GitLab/-/issues/116Failure to access GitLab Docker registry on shared-hzdr-*2022-03-22T16:25:21+01:00Paul SkopnikFailure to access GitLab Docker registry on shared-hzdr-*The shared-hzdr-* runners (not: hifis-runner-manager-1, gitlab-runner-manager.hemera) have problems when trying to access the Docker Registry integrated with GitLab.
From a job log:
```
$ docker login "$REGISTRY" --username "$REGISTRY_U...The shared-hzdr-* runners (not: hifis-runner-manager-1, gitlab-runner-manager.hemera) have problems when trying to access the Docker Registry integrated with GitLab.
From a job log:
```
$ docker login "$REGISTRY" --username "$REGISTRY_USER" --password "$REGISTRY_PASSWORD"
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
error during connect: Post "http://docker:2375/v1.24/auth": dial tcp: lookup docker on 149.220.4.2:53: no such host
```
The `$REGISTRY` variable is set to `$CI_REGISTRY` (a [predefined variable](https://docs.gitlab.com/ee/ci/variables/predefined_variables.html)) in .gitlab-ci.yml via `variables` stanza.
Its value should be `registry.hzdr.de` but appears to be `docker:2375`.
Examples:
* [Job #556633](https://gitlab.hzdr.de/kit-scc-sdm/onlinestorage/httpd-webdav/docker-basics/-/jobs/556633) fails on shared-hzdr-2
* [Job #556636](https://gitlab.hzdr.de/kit-scc-sdm/onlinestorage/httpd-webdav/docker-basics/-/jobs/556636) succeeds on hifis-runner-manager-1
If there is need, I can build a public project for debugging.https://codebase.helmholtz.cloud/fwcc/HZDR-GitLab/-/issues/114Have "Help" landing page also link to HIFIS Service documentation2021-09-28T09:48:36+02:00Guido JuckelandHave "Help" landing page also link to HIFIS Service documentationThe [help landing page](https://gitlab.hzdr.de/help) currently only has a link to this issue tracker. It should also include a reference to the [service documentation](https://hifis.net/doc/software/gitlab/getting-started/).
Especially ...The [help landing page](https://gitlab.hzdr.de/help) currently only has a link to this issue tracker. It should also include a reference to the [service documentation](https://hifis.net/doc/software/gitlab/getting-started/).
Especially on the login page the "help" link is much more prominent then the documentation link. It was the first link I clicked when checking for information on how to use the AAI and such as I was answering an email from a colleague.Huste, TobiasHuste, Tobiashttps://codebase.helmholtz.cloud/fwcc/HZDR-GitLab/-/issues/111internal Tag not working as expected2021-07-21T12:59:49+02:00Schlegel, Dr. Fabian (FWDC) - 107795f.schlegel@hzdr.deinternal Tag not working as expectedI have a few jobs that update code on hemera2. So far they require the internal tag and this would force gitlab to put it on the `gitlab-runner-manager.hemera` Runners: https://gitlab.hzdr.de/openfoam/fwdc/Developments/-/jobs/330008 Howe...I have a few jobs that update code on hemera2. So far they require the internal tag and this would force gitlab to put it on the `gitlab-runner-manager.hemera` Runners: https://gitlab.hzdr.de/openfoam/fwdc/Developments/-/jobs/330008 However, since a while such jobs with the `internal` tag can be processed by `hifis-runner-manager-1` which results in a connection time out when logging in to hemera (python fabric): https://gitlab.hzdr.de/openfoam/fwdc/Developments/-/jobs/330082 Is the tag still valid?Huste, TobiasHuste, Tobiashttps://codebase.helmholtz.cloud/fwcc/HZDR-GitLab/-/issues/104Publish brutenn2020-12-03T16:32:54+01:00Steinbach, Dr. rer. nat. Peter (FWCC) - 120595Publish brutennI'd like to publish [`brutenn`](https://gitlab.hzdr.de/haicu/internal-projects/brutenn). For this, I'd love to move the repo to `https://github.com/helmholtz-ai` and publish wheels on PyPI.
If support for this is available, that would ...I'd like to publish [`brutenn`](https://gitlab.hzdr.de/haicu/internal-projects/brutenn). For this, I'd love to move the repo to `https://github.com/helmholtz-ai` and publish wheels on PyPI.
If support for this is available, that would be great. Otherwise, please let me know what I have to do to make the publication and ROBIS happy.https://codebase.helmholtz.cloud/fwcc/HZDR-GitLab/-/issues/103Build docker image on self-hosted runner2020-11-20T08:43:13+01:00Ehrig, Simeon (FWKT) - 123249Build docker image on self-hosted runnerIs it possible to build a container on a self-made runner? At the moment we use public runners to build our container for the CI. Unfortunately the wall time of a single job has been reduced to 1h, which is not enough for our jobs (we ne...Is it possible to build a container on a self-made runner? At the moment we use public runners to build our container for the CI. Unfortunately the wall time of a single job has been reduced to 1h, which is not enough for our jobs (we need at least 2h).
So it would be really helpful if we could use our runners to build the container, because they need less time to build and have a higher wall time.
P.S. I have already tried to build a container on our system, without success: https://gitlab.com/hzdr/crp/gitlab-ci-test/-/tree/selfHostedDockerBuilder/self-hosted-Docker-Builderhttps://codebase.helmholtz.cloud/fwcc/HZDR-GitLab/-/issues/99Provide Mattermost push notification service2020-07-13T08:42:22+02:00Huste, TobiasProvide Mattermost push notification serviceProviding a self-hosted push notification service would allow us, to send more useful content as push notifications to mobile devices. This would require us to self-host such a service for iOS and Android (https://developers.mattermost.c...Providing a self-hosted push notification service would allow us, to send more useful content as push notifications to mobile devices. This would require us to self-host such a service for iOS and Android (https://developers.mattermost.com/contribute/mobile/push-notifications/service/).
- [ ] Decision, if that should be done?
- [ ] Does HZDR already have Android and/or iOS developer accounts?
Please also compare to https://www.hzdr.de/db/Support.SupPkg.PData?PMSNR=10832.https://codebase.helmholtz.cloud/fwcc/gitlab-hpc-driver/-/issues/17Breaking changes in GitLab 17.02023-03-02T11:12:56+01:00Huste, TobiasBreaking changes in GitLab 17.0GitLab Runner ~~14.0~~ 17.0 will introduce breaking changes that also require this implementation to be updated. In principle, the step called `build_script` so far will be renamed to `step_script`.
This was already unintentionally intr...GitLab Runner ~~14.0~~ 17.0 will introduce breaking changes that also require this implementation to be updated. In principle, the step called `build_script` so far will be renamed to `step_script`.
This was already unintentionally introduced in runner version `13.1.0` and fixed in `13.1.1` with this Merge Request: https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/2227
Update: Now this is apparently due in version 17.0. See output of gitlab-runner:
```
WARNING: Starting with version 17.0 the 'build_script' stage will be replaced with 'step_script': https://gitlab.com/groups/gitlab-org/-/epics/6112
```
[GitLab epic here.](https://gitlab.com/groups/gitlab-org/-/epics/6112)2021-04-21https://codebase.helmholtz.cloud/fwcc/HZDR-GitLab/-/issues/98Login via Helmholtz AAI not working with GitLab 13.12020-10-15T07:24:15+02:00Huste, TobiasLogin via Helmholtz AAI not working with GitLab 13.1With the upgrade to GitLab 13.1 the login via Helmholt AAI was failing. In order to create a useful issue in the respective projects this issue needs some further investigation.
## Corresponding Sentry Error
https://vlsstack1.fz-rossend...With the upgrade to GitLab 13.1 the login via Helmholt AAI was failing. In order to create a useful issue in the respective projects this issue needs some further investigation.
## Corresponding Sentry Error
https://vlsstack1.fz-rossendorf.de/fwcc/gitlab/issues/2320/ (with login)
https://vlsstack1.fz-rossendorf.de/share/issue/5346cec47ab944818aab786f15204d38/ (HZDR internal)
## Current workaround
Manually downgrading the Ruby Gem `omniauth_openid_connect` from `0.3.5` to `0.3.3` in `/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omniauth_openid_connect-0.3.5/` fixes the issue temporarily.
## Places to dig deeper
- https://github.com/m0n9oose/omniauth_openid_connect, especially https://github.com/m0n9oose/omniauth_openid_connect/blob/ef2942047c866993d8323115c419371d75f05a60/lib/omniauth/strategies/openid_connect.rb#L229
- https://github.com/nov/openid_connect/blob/a76c9c84458a0d8c3ec3783ec6c815c285c05d91/lib/openid_connect/response_object/id_token.rb#L70
- https://github.com/nov/json-jwtHuste, TobiasHuste, Tobiashttps://codebase.helmholtz.cloud/fwcc/HZDR-GitLab/-/issues/94MR suggestions always get applied to branch tip2020-05-14T23:29:27+02:00Schmerler, Dr. Steve (FWCC) - 131252MR suggestions always get applied to branch tipDear GitLab gurus. We observed the following behavior [just
recently](https://gitlab.hzdr.de/haicu/operations/-/merge_requests/18/diffs). I have
tested this separately in a private repo as well. Can anyone confirm this? Is
this a known i...Dear GitLab gurus. We observed the following behavior [just
recently](https://gitlab.hzdr.de/haicu/operations/-/merge_requests/18/diffs). I have
tested this separately in a private repo as well. Can anyone confirm this? Is
this a known issue? Is there a solution? Thanks!
Say you have created a branch `feature-wip` and a MR into `master`.
When you, being on the GitLab diff discussion page of the MR
* edit a file in that branch interactively
* apply a [code
suggestion](https://gitlab.hzdr.de/help/user/discussions/index.md#suggest-changes)
then a commit will be created in `feature-wip`, with a commit message such as
```
Apply suggestion to <filename>
```
The problem: the commit will *always* be made on top of the branch's tip, i.e.
the most recent commit, and not the commit at which the diff discussion
started. This is no problem, unless someone (or you) pushes new commits to
`feature-wip`, moving tip forward. Now all not yet applied code suggestions refer to
an *older* version of the file, but will be applied onto the new tip, which may
lead to merge conflicts.
In the particular case above, we even saw a GitLab message saying something like "this ... referes to
an older version of this file ...", with links to the versions and everything.
To make things more visual, this is what happens:
```
* aab4893 (HEAD -> feature-wip, origin/feature-wip) Apply suggestion to foo (GitLab GUI)
|
* b02d02f Pushed
|
* 3edc93f Apply suggestion to foo (GitLab GUI)
|
* f9c2a59 Apply suggestion to foo (GitLab GUI)
|
* 034b71f WIP: foo
|
...
|
* 7192849 initial commit
```
In this situation, GitLab could instead create new branch by
applying onto the old tip `3edc93f`
```
* b02d02f (origin/feature-wip, feature-wip) Pushed
|
| * bec979c (HEAD -> gitlab-automatic-branch-xzy2342) Apply suggestion to foo (GitLab GUI)
|/
|
* 3edc93f Apply suggestion to foo (GitLab GUI)
|
* f9c2a59 Apply suggestion to foo (GitLab GUI)
|
* 034b71f WIP: foo
|
...
|
* 7192849 initial commit
```
Then we could use normal git to merge those, or GitLab could automatically
merge and only create a branch when it cannot merge without conflict.
So: we have a file-locking-like situation here. Until all the MR's discussion
items are resolved, nobody can (should not) commit to that branch. Bummer.
Huste, TobiasHuste, Tobias