Metadata-Version: 2.1
Name: fts3
Version: 3.13.1
Summary: FTS Python3 CLI and libraries
Home-page: https://gitlab.cern.ch/fts/fts-rest-flask
Author: CERN -- FTS team
Author-email: fts-devel@cern.ch
License: Apache License 2.0
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: POSIX :: Linux
Classifier: Programming Language :: Python :: 3.6
Classifier: Programming Language :: Python :: 3.7
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Requires-Python: >=3.6
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: M2Crypto
Requires-Dist: requests
Requires-Dist: setuptools >=39

FTS3-REST-Flask
===============

FTS3-REST-Flask is the RESTful API for the FTS3 Service.  

This is the front-facing component to communicate with the FTS3 Service,
allowing submission of transfers, querying transfer updates, as well as configuration management.  

The project consists of an Apache module running Python3 + Flask. 
It is also a direct continuation of the former [fts-rest][1], where the migration 
decision and evaluation can be seen [here][2].

## Installation

The project should be installed via RPMs, provided via the FTS repository.

### Installing the client

Make sure to have the FTS repositories enabled:
- [production][3] repository: official RPMs deemed ready for production
- [development][4] repository: RPMs built on the latest `develop` branch commit

#### Installing the client:
```shell
$ yum install -y fts-rest-client
```

#### Installing the server

The project needs `mod_wsgi` built for Python 3.6, which on Centos 7 is provided by the `rh-python36-mod_wsgi` package.
All other requirements are specified in the spec files.
```shell
$ yum-config-manager --enable centos-sclo-rh
$ yum install -y rh-python36-mod_wsgi
$ yum install -y fts-rest-server
```

Configuring the server is done via the following two configuration files:
- `/etc/fts3/fts3restconfig`
- `/etc/htpd/conf.d/fts3rest.conf`

## Development

The project is tracked using JIRA, under the [FTS Project][5] (requires CERN log-in). 
Soon the project will be mirrored on GitHub, where issues and pull requests are also accepted (and encouraged!). 

For development purposes, using a virtual environment is encouraged.

### Git Workflow

This project follows a simplified [GitFlow][6] model. The main branch is the `develop` branch,
with `master` being reserved only for tagged released.

Feature branches are developed separately and merged into the `develop` branch.
When preparing a release, either `develop` is merged directly into `master`
or a release branch is created. Production hotfixes are allowed only in case of necessity.
Both release and hotfix branch changes must be merged back into `develop`.

### Development workflow

When taking on a task, the following workflow model should be followed:
1. Create a JIRA ticket for the identified problem
2. Estimate the magnitude of the task:
    - Small changes can go directly on the `develop` branch
    - Larger changes must go into a separate feature branch
3. Implement and test your changes
4. Merge them into the `develop` branch:
    - Merge large changes via merge requests and request code review
    - Changes affecting the database must also be code reviewed!
5. Close the JIRA ticket and mark the appropriate `fixVersion`

### Continuous Integration

The project uses Gitlab-CI for CI/CD. The [pipeline][7] runs for every push, in every branch:
- black: checks code must be formatted with `black`
- pylint: checks for syntax errors (runs for every supported Python3 version)
  - If you are sure that pylint is mistaken, add `# pylint: skip-file` at the beginning of the relevant file
- functional tests: run for every supported Python3 version
- bandit: detects potential security issues in the code
  - Allowed to fail as it flags many false positives.
  - To ignore a false positive, append `# nosec` to the offending line
- build: RPM for the client and server, plus `sdist` and `wheel` for the client
- deploy: upload client and server RPM to the FTS testing repository

Merge requests will proceed only if the pipeline succeeds.  
In case of emergency the pipeline can be [skipped][8].

The pipeline runs in a container from the image tagged as `ci`. The dockerfile is in the `.gitlab-ci` directory and the 
image is hosted in the container registry of this project. The image contains the pre-installed Python environment 
in order to speed up CI execution. When new environment is desired (such as new or upgraded dependencies), 
a new image must be created via the `.gitlab-ci/docker/docker_push.sh` script.

#### During development

Developers may add the `.gitlab-ci/pre-commit` hook to their local repository, 
enabling the following checks before every commit:
- Run `black` to format the changed files
- Run `pylint` only on the changed files
  - Note: As pylint works better when it is run on the full project, some rules have been disabled
- Run `radon` and `bandit` only on the changed files
  - In case of false positives, the hook can be skipped with the commit option `--no-verify`

A convenience script is provided to install the `pre-commit` hook:
```shell
$ cd .gitlab-ci/
$ ./precommit_install.sh
```

### Python dependencies

The project uses [pip-tools][9] to manage dependencies:
- `requirements.in`: list of dependencies for the production app
- `dev-requirements.in`: extra list of packages used for development (e.g. static code analysis)

### Setting up the development environment

The `pipcompile.sh` script is provided to help set up a development environment:
```shell
$ virtualenv venv
$ source venv/bin/activate
(venv) $ pip install --upgrade pip
(venv) $ pip install pip-tools
(venv) $ ./pipcompile.sh --sync
```

Running `pipcompile.sh` will generate `requirements.txt`. Running the command with `--sync`
will also synchronize the virtual environment with the requirements.

Additionally, you will need to configure the HTTPd config file. 
A template version for development is offered at `src/fts3rest/fts3rest_dev_conf.in`. 
To fill it with the right paths and install it, run the `install-httpd-dev.sh` script:
```shell
$ cd src/fts3rest/
$ ./install-httpd-dev.sh
```

### Functional tests

The functional tests reside under `src/fts3rest/fts3rest/tests/functional`. 

In oder to run the tests, a fully functional MySQL-compatible database is required.
The database must have the appropriate FTS schema and the connection details configured 
in the FTS3 REST config file. 

Example on Centos 7 with locally installed `mariadb`: 
```shell
$ wget https://gitlab.cern.ch/fts/fts3/-/raw/v3.11.0/src/db/schema/mysql/fts-schema-7.0.0.sql
$ mysql -u root --execute "CREATE DATABASE ftsflask;"
$ mysql -u root --execute "CREATE USER 'fts3'@'%';"
$ mysql -u root --execute "GRANT ALL PRIVILEGES ON ftsflask.* TO 'fts3'@'%' IDENTIFIED BY 'ftsflaskpass';"
$ mysql -u fts3 --password=ftsflaskpass ftsflask
MariaDB [ftsflask]> source fts-schema-7.0.0.sql
```

Before running the tests, make sure to include the `fts3rest` project in the `PYTHONPATH`.  
Additionally, the `FTS3TESTCONFIG` environment variable can also be set to use a different config file during testing:
```shell
$ export PYTHONPATH="${PROJECT_DIR}/src:${PROJECT_DIR}/src/fts3rest"
$ export FTS3TESTCONFIG="${PROJECT_DIR}/src/fts3rest/fts3rest/tests/fts3testconfig"
```

For ease-of-use, the `runtests.sh` script is provided:
```shell
$ cd src/fts3rest
$ ./runtests.sh
```

**Note**: OIDC tests are not run in the CI as the container would need a client registered and this has not yet been set up.   
To run OIDC tests in a development environment, the environment variables `xdc_ClientId` and `xdc_ClientSecret` must be set.

### Building packages

In order to build RPM packages, two utility scripts are provided 
for the client and server: `fts-rest-client.sh` and `fts-rest-server.sh`.

The following example shows how to build server packages:
```shell
$ ./packaging/server/fts-rest-server.sh
$ tree "build/server/RPMS"
```

## Useful links

- Documentation: https://fts3-docs.web.cern.ch/fts3-docs/fts-rest/docs/index.html
- FTS Repository: https://fts-repo.web.cern.ch/fts-repo/
- Apache License, version 2.0: https://www.apache.org/licenses/LICENSE-2.0
- Support channels: fts-support@cern.ch
- Development team contact: fts-devel@cern.ch 


[1]: https://gitlab.cern.ch/fts/fts-rest
[2]: https://its.cern.ch/jira/browse/FTS-1496
[3]: https://fts-repo.web.cern.ch/fts-repo/el7/x86_64/
[4]: https://fts-repo.web.cern.ch/fts-repo/testing/el7/x86_64/
[5]: https://its.cern.ch/jira/projects/FTS/issues
[6]: https://nvie.com/posts/a-successful-git-branching-model/
[7]: https://gitlab.cern.ch/fts/fts-rest-flask/-/pipelines
[8]: https://docs.gitlab.com/ee/ci/yaml/#skipping-jobs
[9]: https://github.com/jazzband/pip-tools

[10]: https://fts3-docs.web.cern.ch/fts3-docs/fts-rest/docs/index.html
[11]: https://www.apache.org/licenses/LICENSE-2.0