README.md 10.6 KB
Newer Older
mdolling-gfz's avatar
add sqa    
mdolling-gfz committed
1
<!--
2
hifis-surveyval
mdolling-gfz's avatar
add sqa    
mdolling-gfz committed
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
Framework to help developing analysis scripts for the HIFIS Software survey.

SPDX-FileCopyrightText: 2021 HIFIS Software <support@hifis.net>

SPDX-License-Identifier: GPL-3.0-or-later

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.
-->

23
# HIFIS-Surveyval
Huste, Tobias (FWCC) - 111645's avatar
Huste, Tobias (FWCC) - 111645 committed
24
25
26

This project is used to develop analysis scripts for the HIFIS Software survey.

mdolling-gfz's avatar
mdolling-gfz committed
27
28
29
30
31
32
33
## Table of Content

* [Installation](#installation)
* [Getting Started](#getting-started)
* [Start Analysis from Command-Line-Interface](#start-analysis-from-command-line-interface)
* [Contribute with Own Analysis Scripts](#contribute-with-own-analysis-scripts)
* [Resources](#resources)
34
35
* [Author Information](#author-information)
* [Contributors](#contributors)
36
* [License](#license)
mdolling-gfz's avatar
mdolling-gfz committed
37

Huste, Tobias (FWCC) - 111645's avatar
Huste, Tobias (FWCC) - 111645 committed
38
39
## Getting Started

40
41
42
43
44
The project's documentation contains a section to help you as a 
[user of the analysis scripts](#getting-for-users) 
to run the analysis scripts or as a 
[developer of the framework](#getting-started-for-developers)
to set up the development environment.
Huste, Tobias (FWCC) - 111645's avatar
Huste, Tobias (FWCC) - 111645 committed
45

46
### Getting Started for Users
mdolling-gfz's avatar
mdolling-gfz committed
47

48
49
50
51
#### Installation

To install the package locally, you can use 
[Pip](https://pip.pypa.io/en/stable/).
mdolling-gfz's avatar
mdolling-gfz committed
52
53

```shell
54
pip install hifis-surveyval
mdolling-gfz's avatar
mdolling-gfz committed
55
56
```

57
58
After the installation, you can use the tool from the command line with 
`hifis-surveyval --help`.
mdolling-gfz's avatar
mdolling-gfz committed
59

60
### Getting Started for Developers
mdolling-gfz's avatar
mdolling-gfz committed
61

62
#### Installation
mdolling-gfz's avatar
mdolling-gfz committed
63

64
65
66
67
To install the package locally, you can use 
[Poetry](https://python-poetry.org/).

#### Using Poetry
68
69

If you want to actively contribute changes to the project, you are required to
70
also install the development packages alongside the framework.
mdolling-gfz's avatar
mdolling-gfz committed
71
72

```shell
73
74
git clone https://gitlab.hzdr.de/hifis/surveys/hifis-surveyval.git
cd hifis-surveyval
mdolling-gfz's avatar
mdolling-gfz committed
75
76
poetry install
```
77

78
79
After the installation, you can use the tool from the command line with 
`poetry run hifis-surveyval --help`
80

81
Poetry installs some packages that are required for performing quality checks.
82
Usually they are also performed via GitLab CI, but can also be executed locally.
83
84
85
86

It is common practice to run some checks locally before pushing them online.
Therefore, execute below commands:
```console
87
88
$ # Order your imports
$ isort -rc .
89
90
91
$ make lint
```

92
93
94
95
The following documentation references the pip installation.
You can use the same commands with a poetry installation, if you prefix your 
commands with `poetry run COMMAND`.

96
97
98
99
100
101
102
## Start Analysis from Command-Line-Interface

The survey analysis package is a program to be executed on the
Command-Line-Interface (CLI).

### Quick Start Example: Run Analysis

103
Due to sensible defaults of the project's configurations you need to have the 
104
105
106
107
analysis scripts, the preprocessing script as well as metadata and data files
in certain locations in order to run the survey analysis.
This configuration file _hifis-surveyval.yml_ which includes these defaults is
created with the command `hifis-surveyval init`.
108
Please put your analysis scripts into a sub-folder called _scripts_.
109
110
The preprocessing script _preprocess.py_ is expected in the root folder of the
project.
111
Make sure that the file _meta.yml_ is put into sub-folder _metadata_.
112
113
114
Finally, copy the CSV data file of your survey to a central location like a
_data_ sub-folder and tell the program the path to that data file on the
command line when running the survey analysis.
115

116
117
118
Now you can do the following to start the survey analysis from the CLI:

```shell script
119
hifis-surveyval analyze data/<data_file_name>.csv
120
121
```

122
123
124
The output is then put into a sub-folder within the folder _output_ 
which is named after the stamp of the current date-time if not specified 
differently.
125

126
127
128
129
130
131
132
133
**Caution:** 
Depending on the Operating System used an issue with the file 
encoding might occur.
There might be data-CSV-files around which are encoded with `UTF-8-BOM`
which causes errors when read in on Windows OS.
In this case you need to change the encoding to `UTF-8` before running
the survey analysis.

134
135
### Flags

136
The program accepts two flags:
137
138
139
140
141
142
143
144
145
146
147

1. Help flag
2. Verbosity flag

#### Help flag

Calling the program with the _help_-flag is the first thing to do
when being encountered with this program.
It outputs a so-called _Usage_-message to the CLI:

```shell script
148
$ hifis-surveyval --help
149
150
```

151
152
153
154
Please issue this command on the CLI and read the detailed 
_Usage_-message before continuing with reading the documentation
of the _Usage_-message here.

155
156
157
158
159
160
161
#### Verbosity flag

The _verbosity_-flag can be provided in order to specify the verbosity
of the output to the CLI.
This flag is called `--verbose` or `-v` for short:

```shell script
162
hifis-surveyval --verbose <COMMAND>
163
164
```
```shell script
165
hifis-surveyval -v <COMMAND>
166
167
168
169
170
171
```

The verbosity of the output can be increased even more 
by duplicating the flag `--verbose` or `-v` up to two times:

```shell script
172
hifis-surveyval --verbose --verbose --verbose <COMMAND>
173
174
```
```shell script
175
hifis-surveyval -vvv <COMMAND>
176
177
```

178
### Commands
179

180
There are three different commands implemented which come with its own set of
181
flags and parameters:
182

183
184
185
1. Command _version_
2. Command _init_
3. Command _analyze_
186

187
188
189
#### Command _version_
 
The `version` command outputs the version number of this CLI-program like so:
190
191

```shell script
192
hifis-surveyval version
193
194
```

195
#### Command _init_
196

197
198
199
200
Before you start the analysis you may want to change the defaults of the
configuration variables.
In order to do so, you can create a configuration file that is named 
_hifis-surveyval.yml_ by issuing the _init_ command:
201
202

```shell script
203
hifis-surveyval init
204
205
```

206
This file contains the following information:
207
208

```
209
ID_COLUMN_NAME: id
210
211
METADATA: metadata/meta.yml
OUTPUT_FOLDER: output
212
213
OUTPUT_FORMAT: SCREEN
PREPROCESSING_FILENAME: preprocess.py
214
215
SCRIPT_FOLDER: scripts
SCRIPT_NAMES: []
216
217
```

218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
- With `ID_COLUMN_NAME` you may want to specify the name of the _id_ column in
  the CSV data file.
- Each analysis needs metadata about the questions asked in the survey and
  answers that participants may give.
  Setting `METADATA` specifies the location of the Metadata file which is by
  default located in a folder called _metadata_ and named _meta.yml_.
- You may specify the output folder by setting `OUTPUT_FOLDER` which is named
  _output_ by default.
- You may prefer a specific output format like _PDF_, _PNG_, _SVG_ or
  _SCREEN_ which you may select via `OUTPUT_FORMAT`.
  The default value is _SCREEN_.
  Note: Be aware that other output formats like text or markdown files may be
  created, which depends largely upon the implementation of the analysis
  scripts.
- You might want to tell the program where to find the preprocessing file
  _preprocess.py_ that preprocesses and filters your survey data according to 
  specific rules.
  You can do so by setting `PREPROCESSING_FILENAME`.
- You may specify the folder which contains the analysis scripts with setting
  `SCRIPT_FOLDER`, which is the _scripts_ folder by default.
- With `SCRIPT_NAMES` you may select a subset of the analysis scripts available
  as a list that ought to be executed.
  This list is empty by default, which means, all scripts are executed.

Additional to the configuration file there are two more files created:

1. File _preprocess.py_ is created in the root folder of the project.
2. File _example_script.py_ is created in the _scripts_ folder of the project.
246
247
248
249

#### Command _analyze_

The more interesting command is the `analyze` command
250
which comes with a _data_-parameter.
251
252
253
254
255
The _data_-parameter can _not_ be omitted and need to be given explicitly
in order to be able to start the analysis.
This is an example of how to do the analysis:

```shell script
256
hifis-surveyval analyze data/<data_file_name>.csv
257
258
259
260
```

## Contribute with Own Analysis Scripts

261
### Essential Requirements for Developing Own Analysis Scripts
262
263

As you might have read in the previous sections the actual analysis scripts 
264
reside in a specific folder called `scripts`.
265
All scripts in that folder will be automatically discovered by the package 
266
`hifis-surveyval` when running the analysis.
267
In order that the program recognizes the scripts in that folder as
268
analysis scripts they need to fulfill the following two criteria:
269
270
271
272
273
274
275
276

1. The filename need to end on `.py`.
2. The file need to contain a function called `run` without any parameters.

```python
"""
A dummy script for testing the function dispatch

277
.. currentmodule:: hifis_surveyval.scripts.dummy
278
279
280
281
282
283
284
.. moduleauthor:: HIFIS Software <software@hifis.net>
"""

def run():
    print("Example Script")
```

285
If both requirements are satisfied the program will execute the `run`-functions
286
of the analysis scripts in an arbitrary order.
287

288
### File-System Structure of the Core Component
289
290

```shell script
291
292
$ tree hifis_surveyval/
hifis_surveyval/
293
294
├── answer.py
├── cli.py
295
296
297
298
├── core
│   ├── environment.py
│   ├── __init__.py
│   └── settings.py
299
300
301
├── data.py
├── dispatch.py
├── globals.py
302
303
├── __init__.py
├── plot.py
304
├── question.py
305
└── util.py
306
307
```

Huste, Tobias (FWCC) - 111645's avatar
Huste, Tobias (FWCC) - 111645 committed
308
309
## Resources

310
Below are some handy resource links:
Huste, Tobias (FWCC) - 111645's avatar
Huste, Tobias (FWCC) - 111645 committed
311

312
313
* [Project Documentation](TODO)
* [Click](https://click.palletsprojects.com/en/7.x) is a Python package for creating beautiful command line interfaces in a composable way with as little code as necessary.
Huste, Tobias (FWCC) - 111645's avatar
Huste, Tobias (FWCC) - 111645 committed
314
315
316
317
* [Sphinx](http://www.sphinx-doc.org/en/master/) is a tool that makes it easy to create intelligent and beautiful documentation, written by Geog Brandl and licnsed under the BSD license.
* [pytest](https://docs.pytest.org/en/latest/) helps you write better programs.
* [GNU Make](https://www.gnu.org/software/make/) is a tool which controls the generation of executables and other non-source files of a program from the program's source files.

318
319
320
321
322
323
324
325
326
327
328
## Author Information

_HIFIS-Surveyval_ was created by 
[HIFIS Software Services](https://software.hifis.net/).

## Contributors

We would like to thank and give credits to the following contributors of this
project:

* Be the first to be named here!
329
330
331
332
333
334
335
336
337
338

## License

Copyright © 2021 HIFIS Software <support@hifis.net>

This work is licensed under the following license(s):
* Everything else is licensed under [GPL-3.0-or-later](LICENSES/GPL-3.0-or-later.txt)

Please see the individual files for more accurate information.

339
> **Hint:** We provided the copyright and license information in accordance to the [REUSE Specification 3.0](https://reuse.software/spec/).