invenio-uploadbyurl issueshttps://codebase.helmholtz.cloud/rodare/invenio-uploadbyurl/-/issues2017-10-25T13:30:42+02:00https://codebase.helmholtz.cloud/rodare/invenio-uploadbyurl/-/issues/1Redis for progress report2017-10-25T13:30:42+02:00Huste, TobiasRedis for progress reportUse Redis as a key-value store and save `bucket_id` as key and list of `celery_id`. This way, an API endpoint can be provided to request upload progress.Use Redis as a key-value store and save `bucket_id` as key and list of `celery_id`. This way, an API endpoint can be provided to request upload progress.Huste, TobiasHuste, Tobiashttps://codebase.helmholtz.cloud/rodare/invenio-uploadbyurl/-/issues/2Delete redis entries regularly2017-10-25T10:47:54+02:00Huste, TobiasDelete redis entries regularlyThe celery ids stored in redis should be cleaned regularly, as they are not required to be persistent.
Create a celery task to delete the entries stored in the kv-store.The celery ids stored in redis should be cleaned regularly, as they are not required to be persistent.
Create a celery task to delete the entries stored in the kv-store.Huste, TobiasHuste, Tobiashttps://codebase.helmholtz.cloud/rodare/invenio-uploadbyurl/-/issues/3Implement file upload via SFTP2017-10-25T16:03:37+02:00Huste, TobiasImplement file upload via SFTPFES server is accessible via sftp protocol.
Implement a way to provide server-side upload via sftp.
[Paramiko](https://github.com/paramiko/paramiko/) could be suitable for that task.
How could it be achieved: Allow the user to upload ...FES server is accessible via sftp protocol.
Implement a way to provide server-side upload via sftp.
[Paramiko](https://github.com/paramiko/paramiko/) could be suitable for that task.
How could it be achieved: Allow the user to upload his SSH-private key and store it in the database [e.g. like this](http://sqlalchemy-utils.readthedocs.io/en/latest/data_types.html#module-sqlalchemy_utils.types.encrypted). The key can then be used to establish the sftp connection using paramiko.Huste, TobiasHuste, Tobiashttps://codebase.helmholtz.cloud/rodare/invenio-uploadbyurl/-/issues/4Settings entry for key generation2017-12-04T14:16:24+01:00Huste, TobiasSettings entry for key generationThe page could look like shown in the given screenshot. On click the user will be prompted to follow the required steps to create the SSH keys.
![link_accounts](/uploads/571923c9fa3f0de647bd8ca7a08e4937/link_accounts.png)The page could look like shown in the given screenshot. On click the user will be prompted to follow the required steps to create the SSH keys.
![link_accounts](/uploads/571923c9fa3f0de647bd8ca7a08e4937/link_accounts.png)Huste, TobiasHuste, Tobiashttps://codebase.helmholtz.cloud/rodare/invenio-uploadbyurl/-/issues/5Fix SSH key creation2017-10-26T11:45:50+02:00Huste, TobiasFix SSH key creationCreation of SSHKey entry is not successfull: https://gitlab.hzdr.de/fwcc/rodare/invenio-uploadbyurl/blob/a0de6be84396637bc55a9cc8d690cec3185f3c6b/invenio_uploadbyurl/models.py#L146-161
Correct and improve tests.Creation of SSHKey entry is not successfull: https://gitlab.hzdr.de/fwcc/rodare/invenio-uploadbyurl/blob/a0de6be84396637bc55a9cc8d690cec3185f3c6b/invenio_uploadbyurl/models.py#L146-161
Correct and improve tests.Huste, TobiasHuste, Tobiashttps://codebase.helmholtz.cloud/rodare/invenio-uploadbyurl/-/issues/6Verify MD5 checksum2018-02-02T14:45:08+01:00Huste, TobiasVerify MD5 checksumAfter a successful download from a remote server the checksum of the copied file should be verified.After a successful download from a remote server the checksum of the copied file should be verified.https://codebase.helmholtz.cloud/rodare/invenio-uploadbyurl/-/issues/7Improve SFTP download performance2017-12-05T08:58:41+01:00Huste, TobiasImprove SFTP download performanceSFTP performance is currently terrible.
Improve the download task.
Using the `get` method from Paramiko SFTP-client instead of file-like object looks promising.
`FileInstance` and `ObjectVersion` need to be created and handled manually...SFTP performance is currently terrible.
Improve the download task.
Using the `get` method from Paramiko SFTP-client instead of file-like object looks promising.
`FileInstance` and `ObjectVersion` need to be created and handled manually with this approach.https://codebase.helmholtz.cloud/rodare/invenio-uploadbyurl/-/issues/8Rebuild encrypted private keys2018-07-02T15:54:35+02:00Huste, TobiasRebuild encrypted private keysIn case the app's `SECRET_KEY` changes, the encrypted private keys in db need to be rebuilt.
Implement an asynchronous celery task.In case the app's `SECRET_KEY` changes, the encrypted private keys in db need to be rebuilt.
Implement an asynchronous celery task.https://codebase.helmholtz.cloud/rodare/invenio-uploadbyurl/-/issues/9Administration interface2018-02-16T09:31:44+01:00Huste, TobiasAdministration interfaceCreate administration interface to easily configure the module.Create administration interface to easily configure the module.Huste, TobiasHuste, Tobiashttps://codebase.helmholtz.cloud/rodare/invenio-uploadbyurl/-/issues/10Subprocess and sftp2017-12-05T14:38:26+01:00Huste, TobiasSubprocess and sftpTry to use subprocess and sftp application for maximum performance.Try to use subprocess and sftp application for maximum performance.Huste, TobiasHuste, Tobiashttps://codebase.helmholtz.cloud/rodare/invenio-uploadbyurl/-/issues/11Send Email notification2018-01-22T12:57:27+01:00Huste, TobiasSend Email notificationAfter failing or successful asynchronous download send a notification email to the user.After failing or successful asynchronous download send a notification email to the user.Huste, TobiasHuste, Tobiashttps://codebase.helmholtz.cloud/rodare/invenio-uploadbyurl/-/issues/12API endpoint for sftp upload2017-12-07T13:17:41+01:00Huste, TobiasAPI endpoint for sftp uploadImplement the API endpoint for sftp upload similar to the HTTP version.Implement the API endpoint for sftp upload similar to the HTTP version.Huste, TobiasHuste, Tobiashttps://codebase.helmholtz.cloud/rodare/invenio-uploadbyurl/-/issues/13Consider port number2017-12-06T08:20:29+01:00Huste, TobiasConsider port numberWhen connecting via sftp, the port number is currently ignored.
Add the option `-p` when invoking `sftp`.When connecting via sftp, the port number is currently ignored.
Add the option `-p` when invoking `sftp`.Huste, TobiasHuste, Tobiashttps://codebase.helmholtz.cloud/rodare/invenio-uploadbyurl/-/issues/14Catch all possible errors2017-12-08T14:26:15+01:00Huste, TobiasCatch all possible errorsMany possible errors are not probably caught yet.
Catch them and display a suitable error message.Many possible errors are not probably caught yet.
Catch them and display a suitable error message.Huste, TobiasHuste, Tobiashttps://codebase.helmholtz.cloud/rodare/invenio-uploadbyurl/-/issues/15Display public key in settings2017-12-14T15:46:41+01:00Huste, TobiasDisplay public key in settingsThe user should get access to the created public key in user settings.
Like this, he could add the key manually to the server in case anything goes wrong.The user should get access to the created public key in user settings.
Like this, he could add the key manually to the server in case anything goes wrong.Huste, TobiasHuste, Tobiashttps://codebase.helmholtz.cloud/rodare/invenio-uploadbyurl/-/issues/16Fix alembic recipe2017-12-15T13:58:39+01:00Huste, TobiasFix alembic recipeFix the alembic recipe.
Column server_address is wrong.
Create new revision and alter table.Fix the alembic recipe.
Column server_address is wrong.
Create new revision and alter table.Huste, TobiasHuste, Tobiashttps://codebase.helmholtz.cloud/rodare/invenio-uploadbyurl/-/issues/17File browsing2018-01-25T14:11:32+01:00Huste, TobiasFile browsingProvide a user interface for browsing files. Interesting packages for index in e.g. Elasticsearch:
- https://github.com/dadoonet/fscrawlerProvide a user interface for browsing files. Interesting packages for index in e.g. Elasticsearch:
- https://github.com/dadoonet/fscrawlerHuste, TobiasHuste, Tobiashttps://codebase.helmholtz.cloud/rodare/invenio-uploadbyurl/-/issues/18Link for upload via sftp2018-01-08T16:22:56+01:00Huste, TobiasLink for upload via sftpAdd link for upload via sftp in `links.py`.Add link for upload via sftp in `links.py`.Huste, TobiasHuste, Tobiashttps://codebase.helmholtz.cloud/rodare/invenio-uploadbyurl/-/issues/19User feedback on upload status2019-02-12T13:34:21+01:00Huste, TobiasUser feedback on upload statusCurrently, the user is not able to gather information about his background uploads.
It would be nice to have an overview of all background jobs and their status displayed in the user interface.Currently, the user is not able to gather information about his background uploads.
It would be nice to have an overview of all background jobs and their status displayed in the user interface.https://codebase.helmholtz.cloud/rodare/invenio-uploadbyurl/-/issues/20Verify return codes2018-01-22T16:30:06+01:00Huste, TobiasVerify return codesVerify return codes of SFTP get command.
SFTP has built-in integrity checks. So, properly check the return code of the get command in the sftp download task.Verify return codes of SFTP get command.
SFTP has built-in integrity checks. So, properly check the return code of the get command in the sftp download task.Huste, TobiasHuste, Tobiashttps://codebase.helmholtz.cloud/rodare/invenio-uploadbyurl/-/issues/21Improve error messages2018-01-23T11:21:09+01:00Huste, TobiasImprove error messagesWhen starting file upload, the user is currently not notified, if the path was invalid, file sizes were too large, or he does not have read access.
Improve the error messages that are return when using the API.When starting file upload, the user is currently not notified, if the path was invalid, file sizes were too large, or he does not have read access.
Improve the error messages that are return when using the API.Huste, TobiasHuste, Tobiashttps://codebase.helmholtz.cloud/rodare/invenio-uploadbyurl/-/issues/22Settings: second button copies wrong key2018-01-23T13:56:58+01:00Huste, TobiasSettings: second button copies wrong keyIn the index page of the settings view, there is a copy button for the SSH public key. All copy buttons copy the first public key field.
Fix the behavior by copying the correct public key.In the index page of the settings view, there is a copy button for the SSH public key. All copy buttons copy the first public key field.
Fix the behavior by copying the correct public key.Huste, TobiasHuste, Tobiashttps://codebase.helmholtz.cloud/rodare/invenio-uploadbyurl/-/issues/23Browsing: add file size2018-01-25T15:49:42+01:00Huste, TobiasBrowsing: add file sizeHuste, TobiasHuste, Tobiashttps://codebase.helmholtz.cloud/rodare/invenio-uploadbyurl/-/issues/24Document the module2018-02-01T17:57:14+01:00Huste, TobiasDocument the moduleThe module needs proper documentation.The module needs proper documentation.Huste, TobiasHuste, Tobiashttps://codebase.helmholtz.cloud/rodare/invenio-uploadbyurl/-/issues/25Proper error messages2018-01-30T10:02:21+01:00Huste, TobiasProper error messagesIf SSH key deployment failed, no proper error message is shown.If SSH key deployment failed, no proper error message is shown.Huste, TobiasHuste, Tobiashttps://codebase.helmholtz.cloud/rodare/invenio-uploadbyurl/-/issues/26Make key deployment function configurable2018-01-30T13:38:49+01:00Huste, TobiasMake key deployment function configurableSome servers require different handling when transfering the SSH public key.
Make the deployment function configurable to allow different behavior for different servers.Some servers require different handling when transfering the SSH public key.
Make the deployment function configurable to allow different behavior for different servers.Huste, TobiasHuste, Tobiashttps://codebase.helmholtz.cloud/rodare/invenio-uploadbyurl/-/issues/27Caching for file browser2018-01-31T13:49:53+01:00Huste, TobiasCaching for file browserImplement caching for the file browser API.Implement caching for the file browser API.Huste, TobiasHuste, Tobiashttps://codebase.helmholtz.cloud/rodare/invenio-uploadbyurl/-/issues/28Fix tests on master2018-02-01T13:09:57+01:00Huste, TobiasFix tests on masterA new isort version introduced different behavior.
Install `isort>=4.3.0`A new isort version introduced different behavior.
Install `isort>=4.3.0`Huste, TobiasHuste, Tobiashttps://codebase.helmholtz.cloud/rodare/invenio-uploadbyurl/-/issues/29Make module public2018-02-01T18:39:47+01:00Huste, TobiasMake module public- [x] Switch license to GPLv3
- [x] Create entry in ROBIS- [x] Switch license to GPLv3
- [x] Create entry in ROBISHuste, TobiasHuste, Tobiashttps://codebase.helmholtz.cloud/rodare/invenio-uploadbyurl/-/issues/32Add examples app to documentation2018-02-02T11:22:46+01:00Huste, TobiasAdd examples app to documentationHuste, TobiasHuste, Tobiashttps://codebase.helmholtz.cloud/rodare/invenio-uploadbyurl/-/issues/33Deployment to Gitlab pages2018-02-06T10:37:27+01:00Huste, TobiasDeployment to Gitlab pagesAdd deployment to GitLab pages if feature is enabled.Add deployment to GitLab pages if feature is enabled.Huste, TobiasHuste, Tobiashttps://codebase.helmholtz.cloud/rodare/invenio-uploadbyurl/-/issues/34Fix timeout for very large directories2018-02-06T10:01:21+01:00Huste, TobiasFix timeout for very large directoriesThere exist very large directories with lots of files and subdirectories.
Currently, this will result in a timeout.
Possible solutions:
- Limit the number of directories to be listed.There exist very large directories with lots of files and subdirectories.
Currently, this will result in a timeout.
Possible solutions:
- Limit the number of directories to be listed.Huste, TobiasHuste, Tobiashttps://codebase.helmholtz.cloud/rodare/invenio-uploadbyurl/-/issues/35Update sftp-docker location2018-02-08T11:28:10+01:00Huste, TobiasUpdate sftp-docker locationThe project `sftp-docker` was moved to the RODARE group. Update the registry links in `.gitlab-ci.yml`.The project `sftp-docker` was moved to the RODARE group. Update the registry links in `.gitlab-ci.yml`.Huste, TobiasHuste, Tobiashttps://codebase.helmholtz.cloud/rodare/invenio-uploadbyurl/-/issues/36Fix failing tests2018-02-22T10:06:34+01:00Huste, TobiasFix failing testsLast builds failed on master. Fix it.Last builds failed on master. Fix it.Huste, TobiasHuste, Tobiashttps://codebase.helmholtz.cloud/rodare/invenio-uploadbyurl/-/issues/37Tests fail with most recent invenio releases2018-03-23T10:02:37+01:00Huste, TobiasTests fail with most recent invenio releasesFix the failing tests.
Probably the values which are put into redis do not have the correct type:
```python
OSError: Provided data is not of type bytes
```Fix the failing tests.
Probably the values which are put into redis do not have the correct type:
```python
OSError: Provided data is not of type bytes
```Huste, TobiasHuste, Tobiashttps://codebase.helmholtz.cloud/rodare/invenio-uploadbyurl/-/issues/39Update invenio dependencies2018-04-11T09:40:27+02:00Huste, TobiasUpdate invenio dependenciesUpdate the invenio-dependencies to their latest versions.Update the invenio-dependencies to their latest versions.Huste, TobiasHuste, Tobiashttps://codebase.helmholtz.cloud/rodare/invenio-uploadbyurl/-/issues/40Send notification mail for HTTP upload2018-05-07T16:47:35+02:00Huste, TobiasSend notification mail for HTTP uploadCurrently, a notification email is only sent for sftp upload. Send the same mail also for the URL upload.Currently, a notification email is only sent for sftp upload. Send the same mail also for the URL upload.Huste, TobiasHuste, Tobiashttps://codebase.helmholtz.cloud/rodare/invenio-uploadbyurl/-/issues/41Return proper error message, if endpoint cannot be resolved2018-05-14T08:35:17+02:00Huste, TobiasReturn proper error message, if endpoint cannot be resolvedCurrently, everything which is a valid URL is accepted. Check if the URL really exists and display a proper error message in case of failure.Currently, everything which is a valid URL is accepted. Check if the URL really exists and display a proper error message in case of failure.Huste, TobiasHuste, Tobiashttps://codebase.helmholtz.cloud/rodare/invenio-uploadbyurl/-/issues/42Replace external requests2018-10-11T12:59:24+02:00Huste, TobiasReplace external requestsReplace external HTTP requests using `httpretty`Replace external HTTP requests using `httpretty`Huste, TobiasHuste, Tobiashttps://codebase.helmholtz.cloud/rodare/invenio-uploadbyurl/-/issues/43Add Python 3.6 to .gitlab-ci.yml2018-09-14T08:14:52+02:00Huste, TobiasAdd Python 3.6 to .gitlab-ci.ymlHuste, TobiasHuste, Tobiashttps://codebase.helmholtz.cloud/rodare/invenio-uploadbyurl/-/issues/44Reduce complexity of .gitlab-ci.yml2018-09-20T10:57:18+02:00Huste, TobiasReduce complexity of .gitlab-ci.ymlThe complexity of our `.gitlab-ci.yml` file can be massively improved by using special YAML features as given in (https://docs.gitlab.com/ee/ci/yaml/#special-yaml-features).The complexity of our `.gitlab-ci.yml` file can be massively improved by using special YAML features as given in (https://docs.gitlab.com/ee/ci/yaml/#special-yaml-features).Huste, TobiasHuste, Tobiashttps://codebase.helmholtz.cloud/rodare/invenio-uploadbyurl/-/issues/45Fix failing builds2018-09-20T16:35:20+02:00Huste, TobiasFix failing buildsSometimes builds fail due to some caching issue.
Use a separate cache for the different jobs.Sometimes builds fail due to some caching issue.
Use a separate cache for the different jobs.Huste, TobiasHuste, Tobiashttps://codebase.helmholtz.cloud/rodare/invenio-uploadbyurl/-/issues/46Create API to get all active background jobs2018-10-19T08:06:59+02:00Huste, TobiasCreate API to get all active background jobsCurrently, it is not easily possible to get the status of the user's jobs while they are running.
Implement an API endpoint, that returns the status of the user's jobs.Currently, it is not easily possible to get the status of the user's jobs while they are running.
Implement an API endpoint, that returns the status of the user's jobs.https://codebase.helmholtz.cloud/rodare/invenio-uploadbyurl/-/issues/47Set TTL for job IDs stored in Redis2018-10-11T15:52:54+02:00Huste, TobiasSet TTL for job IDs stored in RedisThe job IDs are currently stored in Redis, but no Time-To-Live is set. Set the TTL to 1 week.The job IDs are currently stored in Redis, but no Time-To-Live is set. Set the TTL to 1 week.Huste, TobiasHuste, Tobiashttps://codebase.helmholtz.cloud/rodare/invenio-uploadbyurl/-/issues/48Setup pages accessible without being logged in2018-10-12T11:42:04+02:00Huste, TobiasSetup pages accessible without being logged inPutting the `login_required` decorator before `blueprint.route` makes the initialization page accessible without being logged in.Putting the `login_required` decorator before `blueprint.route` makes the initialization page accessible without being logged in.Huste, TobiasHuste, Tobiashttps://codebase.helmholtz.cloud/rodare/invenio-uploadbyurl/-/issues/51Disable mail notifications for specific user2018-12-14T11:04:48+01:00Huste, TobiasDisable mail notifications for specific userIn order to prepare the integration of ROBIS and RODARE the module must be able to exclude specific users from receiving notifications about the job status.
Should be enought to make it configurable via a new configuration variableIn order to prepare the integration of ROBIS and RODARE the module must be able to exclude specific users from receiving notifications about the job status.
Should be enought to make it configurable via a new configuration variableHuste, TobiasHuste, Tobiashttps://codebase.helmholtz.cloud/rodare/invenio-uploadbyurl/-/issues/54Job Failed #4875222023-02-10T10:30:57+01:00Knodel, Dr. Oliver (FWCC) - 132739Job Failed #487522Job [#487522](https://gitlab.hzdr.de/rodare/invenio-uploadbyurl/-/jobs/487522) failed for 2e0791d7c94ca618e4ef1918a8f9d50c0f5b550f:Job [#487522](https://gitlab.hzdr.de/rodare/invenio-uploadbyurl/-/jobs/487522) failed for 2e0791d7c94ca618e4ef1918a8f9d50c0f5b550f:Knodel, Dr. Oliver (FWCC) - 132739Knodel, Dr. Oliver (FWCC) - 132739https://codebase.helmholtz.cloud/rodare/invenio-uploadbyurl/-/issues/55Remove elasticsearch7 dependency2022-04-06T11:10:09+02:00David Paped.pape@hzdr.deRemove elasticsearch7 dependencyDavid Paped.pape@hzdr.deDavid Paped.pape@hzdr.de