README.md 8.92 KB
Newer Older
Markus Scheidgen's avatar
Markus Scheidgen committed
1
2
3
[![pipeline status](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/badges/master/pipeline.svg)](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/commits/master)
[![coverage report](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/badges/master/coverage.svg)](https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/commits/master)

4
5
6
7
This project implements the new *nomad@FAIRDI* infrastructure. Contrary to its NOMAD CoE
predecessor, it implements the NOMAD Repository and NOMAD Archive functionality within
a single cohesive application. This project provides all necessary artifacts to develop,
test, deploy, and operate the NOMAD Respository and Archive, e.g. at
8
[https://nomad-lab.eu](https://nomad-lab.eu).
9
10
11
12

In the future, this project's aim is to integrate more NOMAD CoE components, like the NOMAD
Encyclopedia and NOMAD Analytics Toolkit, to fully integrate NOMAD with one GUI and consistent
APIs. Furthermore, this projects aims at establishing NOMAD as a distributed platform for
13
14
15
16
17
18
material science data sharing and management. This includes the on-site deployment of
NOMAD as a standalone service (*oasis*), the federated use of NOMAD through a
serious of full and partial *mirrors*, the integration of 3rd party material science
databases (i.e. [Aflow](http://www.aflow.org/), [OQMD](http://oqmd.org/),
[Materials Project](https://materialsproject.org/)), and support for open APIs and
standards like the [Optimade](http://www.optimade.org/) API.
19

20

21
## Getting started
22

Markus Scheidgen's avatar
Markus Scheidgen committed
23
24
25
26
27
28
### Using NOMAD as a Python package

You can install the `nomad` Python package from source distribution with pip. Please
note, that this will only install part of NOMAD's dependencies that will only allow
your to use NOMAD's client library, e.g. to access the NOMAD Archive.
```
29
pip install nomad-lab
Markus Scheidgen's avatar
Markus Scheidgen committed
30
31
32
33
```

To **use the NOMAD parsers for example**, install the `parsing` extra:
```
34
pip install nomad-lab[parsing]
Markus Scheidgen's avatar
Markus Scheidgen committed
35
nomad parse --show-archive <your-file-to-parse>
Markus Scheidgen's avatar
Markus Scheidgen committed
36
37
38
39
```

### For NOMAD developer

40
Read the [docs](https://nomad-lab.eu/prod/rae/docs/index.html). The documentation is also part
41
42
of the source code. It covers aspects like introduction, architecture, development setup/deployment,
contributing, and API reference.
43

Markus Scheidgen's avatar
Markus Scheidgen committed
44
45

## Change log
46

Markus Scheidgen's avatar
Markus Scheidgen committed
47
Omitted versions are plain bugfix releases with only minor changes and fixes.
Markus Scheidgen's avatar
Markus Scheidgen committed
48

49
50
51
### v1.0.6
- upgraded to Elasticsearch 7.x

Markus Scheidgen's avatar
Markus Scheidgen committed
52
53
54
### v1.0.4
- tabular data schema

Markus Scheidgen's avatar
Markus Scheidgen committed
55
56
57
### v1.0.3
- refactored DCAT to use fast api, added DOIs
- refactored ArchiveQuery client
Markus Scheidgen's avatar
Markus Scheidgen committed
58
- documentation and fixes for Oasis with keycloak
Markus Scheidgen's avatar
Markus Scheidgen committed
59
60
- many minor GUI bugfixes

61
62
63
64
65
66
### v1.0.0
- new search interface
- new v1 API (entries, materials, upload, datasets, sync)
- refactored metainfo and parsers
- new upload UI and incremental uploads

67
68
69
70
### v0.10.9
- new AI Toolkit GUI page
- many minor parser fixes and improvements

71
72
73
### v0.10.7
- adding OpenMX parser

74
75
76
### v0.10.6
- support for NOMAD fields in optimade

77
78
79
80
81
### v0.10.4
- new "basic" parser to cover codes without proper parser
- removed old nomad-coe parser dependencies
- many minor parser fixes and improvements

Markus Scheidgen's avatar
Markus Scheidgen committed
82
83
### v0.10.3
- fixes in the VASP parser
84
- new turbemole parser
Markus Scheidgen's avatar
Markus Scheidgen committed
85
- property placeholders while loading entry page
86
- improved UI navigation with breadcrumbs
Markus Scheidgen's avatar
Markus Scheidgen committed
87

Markus Scheidgen's avatar
Markus Scheidgen committed
88
89
90
91
92
### v0.10.2
- fixes small parser and normalizer issues
- fixes broken embargo lifting
- fixes default keycloak configuration for authenticated access via ArchiveQuery

93
### v0.10.0
Markus Scheidgen's avatar
Markus Scheidgen committed
94
- The entries page shows visualizations for key properties of the underlying data
95
96
97
98
- A new more consistent API (/api/v1) alongside the old API (/api)
- OPTIMADE implementation based on optimade-python-tools
- Re-written parsers for VASP, FHI-aims, exciting, ABINIT, and Crystal

99
100
101
102
### v0.9.9
- A rdf-API that provides dcat datasets and catalog for NOMAD entries.
- Support to directly publish upon upload via API.

103
104
### v0.9.8
- A new library for parsing text-based raw files.
Markus Scheidgen's avatar
Markus Scheidgen committed
105
- A new main menu in the GUI.
106
- Upload OASIS uploads to central NOMAD.
Markus Scheidgen's avatar
Markus Scheidgen committed
107
- Updated documentation.
108

Markus Scheidgen's avatar
Markus Scheidgen committed
109
110
111
### v0.9.3
- Encyclopedia with dedicated materials search index.

112
### v0.9.0
113
114
115
116
117
118
119
120
121
122
- The encyclopedia runs on top of the new infrastructure. The GUIs are integrated via
  bi-lateral navigation between entries.
- The parsers have new documentation and development instructions and can be easily developed
  based on NOMAD's PyPi package nomad-lab
- We introduce workflow metadata (starting with geometry optimizations). This includes search
  for workflow parameter, and 3-tiered archive storage for quick access to results for analytics.
- A new GUI to browse  the Archive and Metainfo
- The Artificial Intelligence Toolkit (at least its tutorial page) is part of the GUI
- The OASIS comprises Repository, Archive, Metainfo, Encyclopedia, and Artificial Intelligence Toolkit.

123
124
125
### v0.8.7
- a new variant of the Metainfo browser

Markus Scheidgen's avatar
Markus Scheidgen committed
126
127
### v0.8.1
- switched to support Python 3.7
128
- client library as pypi package `nomad-lab`
Markus Scheidgen's avatar
Markus Scheidgen committed
129

130
131
### v0.8.0
- new underlying datamodel that allows to maintain multiple domains
Markus Scheidgen's avatar
Markus Scheidgen committed
132
- multiple domains supported the GUI
133
- new metainfo implementation
134
- API endpoint to access the metainfo
135
136
137
138
- new archive based on new metainfo
- client library that serves archive data as objects (with tab completion) not dictionaries
- properties and user tab in the search GUI
- improved performance on most parsers
Markus Scheidgen's avatar
Markus Scheidgen committed
139
- NOMAD source distribution
140

141
142
143
144
### v0.7.9
- Everything to run a simple NOMAD OASIS based on the central user-management
- minor bugfixes

145
146
147
148
### v0.7.7
- Shows dataset contents with embargo data, but hides the entry details (raw-files, archive)
- minor bugfixes

149
### v0.7.5
Markus Scheidgen's avatar
Markus Scheidgen committed
150
- AFLOWLIB prototypes (archive)
151
152
- primitive label search
- improved search performance based on excluded fields
Markus Scheidgen's avatar
Markus Scheidgen committed
153
- improved logs
154
155
- minor bugfixes

156
157
158
159
160
161
162
163
### v0.7.3
- fixed aborted raw-file downloads
- improved representation of data availability (staging, embargo, public) in GUI
- user data uploads ordered by upload time
- user data shows uploads with name
- configurable embargo period length
- minor bugfixes

164
165
166
167
### v0.7.2
- API curl, Python, and results for entries and search queries shown in the GUI
- minor bugfixes

Markus Scheidgen's avatar
Markus Scheidgen committed
168
169
170
171
### v0.7.1
- Download of archive files based on search queries
- minor bugfixes

Markus Scheidgen's avatar
Markus Scheidgen committed
172
### v0.7.0
173
174
- User metadata editing and datasets with DOIs
- Revised GUI lists (entries, grouped entries, datasets, uploads)
175
- Keycloak based user management
176
177
- Rawfile preview
- no dependencies with the old NOMAD CoE Repository
178
179
180
181
182
183
184

### v0.6.2
- GUI performance enhancements
- API /raw/query endpoint takes file pattern to further filter download contents and
  strips potential shared path prefixes for a cleaner download .zip
- Stipped common path prefixes in raw file downloads
- minor bugfixes
185

186
### v0.6.0
Markus Scheidgen's avatar
Markus Scheidgen committed
187
- GUI URL, and API endpoint that resolves NOMAD CoE legacy PIDs
188
189
- Support for datasets in the GUI
- more flexible search python module and repo API
Markus Scheidgen's avatar
Markus Scheidgen committed
190
- support for external_id
191
- support for code-based raw_id
192
- Optimade API 0.10.0
193
- GUI supports Optimade filter query and other quantities
194
195
- minor bugfixes

196
197
198
199
200
### v0.5.2
- allows to download large files over longer time period
- streamlined deployment without API+GUI proxy
- minor bugfixes

Markus Scheidgen's avatar
Markus Scheidgen committed
201
### v0.5.1
Markus Scheidgen's avatar
Markus Scheidgen committed
202
- integrated parsers Dmol3, qbox, molcas, fleur, and onetep
203
- API endpoint for query based raw file download
Markus Scheidgen's avatar
Markus Scheidgen committed
204
- improvements to admin cli: e.g. clean staging files, reprocess uploads based on codes
205
- improved error handling in the GUI
Markus Scheidgen's avatar
Markus Scheidgen committed
206
- lots of parser bugfixes
Markus Scheidgen's avatar
Markus Scheidgen committed
207
208
- lots of minor bugfixes

209
### v0.5.0
210
211
The first production version of nomad@fairdi as the upload API and gui for NOMAD
- Production ready software and deployments (term agreements, better GUI docs)
212
213
214
- Raw file API with support to list directories. This replaces the `files` calculation
  metadata key. It was necessary due to arbitrary large lists of *auxfiles* in some
  calculations.
215
- Search interface that contains all features of the CoE Repository GUI.
216
217
218
- Refactored search API that allows to search for entries (paginated + scroll),
  metrics based on quantity aggregations (+ paginated entries), quantity aggregations
  with all values via `after` key (+ paginated entries).
219
220
221
- reprocessing of published results (e.g. after parser/normalizer improvements)
- mirror functionality
- refactored command line interface (CLI)
222
- potential GUI user tracking capabilities
223
- many minor bugfixes
224

225
226
227
228
### v0.4.7
- more migration scripts
- minor bugfixes

229
230
231
232
233
234
### v0.4.6
- admin commands to directly manipulate upload data
- additional migration scripts
- fixed system normalizer to understand indexed atom labels correctly
- many minor bugfixes

235
236
237
238
239
### v0.4.5
- improved uploads view with published uploads
- support for publishing to the existing nomad CoE repository
- many minor bugfixes

240
### v0.4.4
241
- improved GUI navigation
242
243
- support for multiple domains
- info API endpoint
Markus Scheidgen's avatar
Markus Scheidgen committed
244
- metainfo browser
Markus Scheidgen's avatar
Markus Scheidgen committed
245
- support for latest exciting version
Markus Scheidgen's avatar
Markus Scheidgen committed
246
- bugfixes in system normalization
Markus Scheidgen's avatar
Markus Scheidgen committed
247
- many minor bugfixes
248

249
250
251
252
253
254
255
### v0.4.3
- more flexible celery routing
- config via nomad.yml
- repo_db can be disabled
- publishing of calculations with failed processing
- cli for managing running processing tasks

Markus Scheidgen's avatar
Markus Scheidgen committed
256
257
258
### v0.4.2
- bugfixes regarding the migration
- better migration configurability and reproducibility
Markus Scheidgen's avatar
Markus Scheidgen committed
259
- scales to multi node kubernetes deployment