Skip to content

Development

The development of Fractal Server takes place on the fractal-server Github repository. To ask questions or to inform us of a bug or unexpected behavior, please feel free to open an issue.

To contribute code, please fork the repository and submit a pull request.

Set up the development environment

Install poetry

Fractal uses poetry to manage the development environment and dependencies, and to streamline the build and release operations; at least version 1.3 is recommended.

A simple way to install poetry is 

pipx install poetry==1.8.2`
while other options are described here.

Clone repository

You can clone the fractal-server repository via 

git clone https://github.com/fractal-analytics-platform/fractal-server.git

Install package

Running 

poetry install
will initialise a Python virtual environment and install Fractal Server and all its dependencies, including development dependencies. Note that to run commands from within this environment you should prepend them with poetry run, as in poetry run fractalctl start.

To install Fractal Server with some additional extras, use the -E option, as in 

poetry install -E postgres
poetry install --all-extras

It may sometimes be useful to use a different Python interpreter from the one installed in your system. To this end we suggest using pyenv. In the project folder, invoking 

pyenv local <version>
poetry env use <version>
will install Fractal in a development environment using an interpreter pinned at the version provided instead of the system interpreter.

Update database schema

Whenever the models are modified (either in app/models or in app/schemas), you should update them via a migration. To check whether this is needed, run 

poetry run alembic check

If needed, the simplest procedure is to use alembic --autogenerate to create an incremental migration script, as in 

$ export SQLITE_PATH=some-test.db
$ rm some-test.db
$ poetry run fractalctl set-db --skip-init-data
$ poetry run alembic revision --autogenerate -m "Some migration message"

# UserWarning: SQLite is partially supported but discouraged in production environment.SQLite offers partial support for ForeignKey constraints. As such, consistency of the database cannot be guaranteed.
# INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
# INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
# INFO  [alembic.autogenerate.compare] Detected added column 'task.x'
#   Generating /some/path/fractal_server/migrations/versions/155de544c342_.py ...  done

Release

  1. Checkout to branch main.
  2. Check that the current HEAD of the main branch passes all the tests (note: make sure that you are using the poetry-installed local package).
  3. Update the CHANGELOG.md file (e.g. remove (unreleased) from the upcoming version).
  4. If you have modified the models, then you must also create a new migration script (note: in principle the CI will fail if you forget this step).

  5. Use one of the following 

    poetry run bumpver update --tag-num --tag-commit --commit --dry
poetry run bumpver update --patch --tag-commit --commit --dry
poetry run bumpver update --minor --tag-commit --commit --dry
poetry run bumpver update --set-version X.Y.Z --tag-commit --commit --dry
    to test updating the version bump.

  6. If the previous step looks good, remove --dry and re-run to actually bump the version, commit and push the changes.

  7. Approve (or have approved) the new version at Publish package to PyPI.
  8. After the release: If the release was a stable one (e.g. X.Y.Z, not X.Y.Za1 or X.Y.Zrc2), move fractal_server/data_migrations/X_Y_Z.py to fractal_server/data_migrations/old.

Run tests

Unit and integration testing of Fractal Server uses the pytest testing framework.

To test the SLURM backend, we use a custom version of a Docker local SLURM cluster. The pytest plugin pytest-docker is then used to spin up the Docker containers for the duration of the tests.

Important: this requires docker being installed on the development system, and the current user being in the docker group. A simple check for this requirement is to run a command like docker ps, and verify that it does not raise any permission-related error. Note that also docker-compose must be available, but this package is installed as a dependency of pytest-docker (when it is installed with the extra docker-compose-v1, as in the case of Fractal Server).

If you installed the development dependencies, you may run the test suite by invoking 

poetry run pytest
from the main directory of the fractal-server repository. It is sometimes useful to specify additional arguments, e.g. 
poetry run pytest -s -vvv --log-cli-level info --full-trace

Tests are also run as part of GitHub Actions Continuous Integration for the fractal-server repository.

Documentation

The documentations is built with mkdocs and the Material theme. Whenever possible, docstrings should be formatted as in the Google Python Style Guide.

To build the documentation

  1. Setup a python environment and install the requirements from docs/doc-requirements.txt.
  2. Run 
    poetry run mkdocs serve --config-file mkdocs.yml
    and browse the documentation at http://127.0.0.1:8000.