Note: Numbers like (#1234) point to closed Pull Requests on the fractal-server repository.

Unreleased

• API:
  • Add validation for .whl filename (#2147).
  • Trim whitespaces in DatasetCreateV2.zarr_dir (#2138).
  • Support sending emails upon new OAuth signup (#2150).
• App:
  • Introduce configuration for email settings (#2150).
• Command-line interface:
  • Add fractalctl email-settings (#2150).
• Dependencies:
  • Add direct dependency on cryptography (#2150).
• Testing:
  • Introduce mailpit-based end-to-end test of email sending (#2150).

2.10.2

• App:
  • Add FRACTAL_PIP_CACHE_DIR configuration variable (#2141).
• Tasks life cycle:
  • Prevent deactivation of task groups with "github.com" in pip-freeze information (#2144).
• Runner:
  • Handle early shutdown for sudo SLURM executor (#2132).
  • Fix repeated setting of timestamp_ended in task-group reactivation (#2140).

2.10.1

• API:
  • Add POST /api/v2/project/{p_id}/workflow/{w_id}/wftask/replace-task/ endpoint (#2129).
• Testing:
  • Use system postgresql in GitHub actions, rather than independent container (#2199).

2.10.0

• API:
  • Major update of POST /api/v2/task/collect/pip/, to support wheel-file upload (#2113).
• Testing:
  • Add test of private task collection (#2126).

2.9.2

• API
  • Remove cache_dir and use project_dir/.fractal_cache (#2121).
• Docs
  • Improve docstrings and reduce mkdocs warnings (#2122).

2.9.1

• Task collection:
  • Fix bug in wheel-based SSH task-collection (#2119).
• Testing:
  • Re-include a specific test previously skipped for Python 3.12 (#2114).
  • Add metadata to fractal-tasks-mock package (#2117).
• Docs:
  • Add info about working versions.

2.9.0

WARNING 1: This version drops support for sqlite, and removes the configuration variables DB_ENGINE and SQLITE_PATH.

WARNING 2: This version removes the CollectionStateV2 database table. Make sure you have a database dump before running fractalctl set-db, since this operation cannot be undone.

• API
  • Remove GET /api/v2/task/collect/{state_id}/ endpoint (#2010).
  • Remove active property from PATCH /api/v2/task-group/{task_group_id}/ (#2033).
  • Add GET /api/v2/task-group/activity/ endpoint (#2005, #2027).
  • Add GET /api/v2/task-group/activity/{task_group_activity_id}/ endpoint (#2005).
  • Add GET /admin/v2/task-group/activity/ endpoint (#2005, #2027).
  • Add POST /api/v2/task-group/{task_group_id}/{deactivate|reactivate} endpoints (#2033, #2066, #2078).
  • Add POST /admin/v2/task-group/{task_group_id}/{deactivate|reactivate} endpoints (#2062, #2078).
  • Remove GET /auth/current-user/viewer-paths/ (#2096).
  • Add GET /auth/current-user/allowed-viewer-paths/, with logic for fractal-vizarr-viewer authorization (#2096).
  • Add category, modality and author query parameters to GET /admin/v2/task/ (#2102).
  • Add POST /auth/group/{group_id}/add-user/{user_id}/ (#2101).
  • Add POST /auth/group/{group_id}/remove-user/{user_id}/ (#2101, #2111).
  • Add POST /auth/users/{user_id}/set-groups/ (#2106).
  • Remove new_user_ids property from PATCH /auth/group/{group_id}/ (#2101).
  • Remove new_group_ids property from PATCH /auth/users/{user_id}/ (#2106).
  • Internals:
  • Fix bug in _get_collection_task_group_activity_status_message (#2047).
  • Remove valutc validator for timestamps from API schemas, since it does not match with psycopg3 behavior (#2064).
  • Add query parameters timestamp_last_used_{min|max} to GET /admin/v2/task-group/ (#2061).
  • Remove _convert_to_db_timestamp and add _raise_if_naive_datetime: now API only accepts timezone-aware datetimes as query parameters (#2068).
  • Remove _encode_as_utc: now timestamps are serialized in JSONs with their own timezone (#2081).
• Database
  • Drop support for sqlite, and remove the DB_ENGINE and SQLITE_PATH configuration variables (#2052).
  • Add TaskGroupActivityV2 table (#2005).
  • Drop CollectionStateV2 table (#2010).
  • Add TaskGroupV2.pip_freeze nullable column (#2017).
  • Add venv_size_in_kB and venv_file_number to TaskGroupV2 (#2034).
  • Add TaskGroupV2.timestamp_last_used column, updated on job submission (#2049, #2061, #2086).
• Task-lifecycle internals:
  • Refactor task collection and database-session management in background tasks (#2030).
  • Update TaskGroupActivityV2 objects (#2005).
  • Update filename and path for task-collection scripts (#2008).
  • Copy wheel file into task_group.path and update task_group.wheel_path, for local task collection (#2020).
  • Set TaskGroupActivityV2.timestamp_ended when collections terminate (#2026).
  • Refactor bash templates and add install_from_freeze.sh (#2029).
  • Introduce background operations for local reactivate/deactivate (#2033).
  • Introduce background operations for SSH reactivate/deactivate (#2066).
  • Fix escaping of newlines within f-strings, in logs (#2028).
  • Improve handling of task groups created before 2.9.0 (#2050).
  • Add TaskGroupCreateV2Strict for task collections (#2080).
  • Always create script_dir_remote in SSH lifecycle background tasks (#2089).
  • Postpone setting active=False in task-group deactivation to after all preliminary checks (#2100).
• Runner:
  • Improve error handling in _zip_folder_to_file_and_remove (#2057).
  • Improve error handling in FractalSlurmSSHExecutor handshake method (#2083).
  • Use the "spawn" start method for the multiprocessing context, for the ProcessPoolExecutor-based runner (#2084).
  • Extract common functionalities from SLURM/sudo and SLURM/SSH executors (#2107).
• SSH internals:
  • Add FractalSSH.remote_exists method (#2008).
  • Drop FractalSSH.{_get,_put} wrappers of SFTPClient methods (#2077).
  • Try re-opening the connection in FractalSSH.check_connection when an error occurs (#2035).
  • Move NoValidConnectionError exception handling into FractalSSH.log_and_raise method (#2070).
  • Improve closed-socket testing (#2076).
• App:
• Add FRACTAL_VIEWER_AUTHORIZATION_SCHEME and FRACTAL_VIEWER_BASE_FOLDER configuration variables (#2096).
• Testing:
  • Drop fetch-depth from checkout in GitHub actions (#2039).
• Scripts:
  • Introduce scripts/export_v1_workflows.py (#2043).
• Dependencies:
  • Remove passlib dependency (#2112).
  • Bump fastapi-users to v14, which includes switch to pwdlib (#2112).

2.8.1

• API:
  • Validate all user-provided strings that end up in pip-install commands (#2003).

2.8.0

• Task collection
  • Now both the local and SSH versions of the task collection use the bash templates (#1980).
  • Update task-collections database logs incrementally (#1980).
  • Add TaskGroupV2.pinned_package_versions_string property (#1980).
  • Support pinned-package versions for SSH task collection (#1980).
  • Now pip install uses --no-cache (#1980).
• API
  • Deprecate the verbose query parameter in GET /api/v2/task/collect/{state_id}/ (#1980).
  • Add project_dir attribute to UserSettings (#1990).
  • Set a default for DatasetV2.zarr_dir (#1990).
  • Combine the args_schema_parallel and args_schema_non_parallel query parameters in GET /api/v2/task/ into a single parameter args_schema (#1998).

2.7.1

WARNING: As of this version, all extras for pip install are deprecated and the corresponding dependencies become required.

• Database:
  • Drop TaskV2.owner column (#1977).
  • Make TaskV2.taskgroupv2_id column required (#1977).
• Dependencies:
  • Make psycopg[binary] dependency required, and drop postgres-pyscopg-binary extra (#1970).
  • Make gunicorn dependency required, and drop gunicorn extra (#1970).
• Testing:
  • Switch from SQLite to Postgres in the OAuth Github action (#1981).

2.7.0

WARNING: This release comes with several specific notes:

1. It requires running fractalctl update-db-data (after fractalctl set-db).
2. When running fractalctl update-db-data, the environment variable FRACTAL_V27_DEFAULT_USER_EMAIL must be set, e.g. as in FRACTAL_V27_DEFAULT_USER_EMAIL=admin@fractal.yx fractalctl update-db-data. This user must exist, and they will own all previously-common tasks/task-groups.
3. The pip extra postgres is deprecated, in favor of postgres-psycopg-binary.
4. The configuration variable DB_ENGINE="postgres" is deprecated, in favor of DB_ENGINE="postgres-psycopg".
5. Python3.9 is deprecated.

• API:
  • Users and user groups:
    • Replace UserRead.group_names and UserRead.group_ids with UserRead.group_ids_names ordered list (#1844, #1850).
    • Deprecate GET /auth/group-names/ (#1844).
    • Add DELETE /auth/group/{id}/ endpoint (#1885).
    • Add PATCH auth/group/{group_id}/user-settings/ bulk endpoint (#1936).
  • Task groups:
    • Introduce /api/v2/task-group/ routes (#1817, #1847, #1852, #1856, #1943).
    • Respond with 422 error when any task-creating endpoint would break a non-duplication constraint (#1861).
    • Enforce non-duplication constraints on TaskGroupV2 (#1865).
    • Fix non-duplication check in PATCH /api/v2/task-group/{id}/ (#1911).
    • Add cascade operations to DELETE /api/v2/task-group/{task_group_id}/ and to DELETE /admin/v2/task-group/{task_group_id}/ (#1867).
    • Expand use and validators for TaskGroupCreateV2 schema (#1861).
    • Do not process task sources in task/task-group CRUD operations (#1861).
    • Do not process task owners in task/task-group CRUD operations (#1861).
  • Tasks:
    • Drop TaskCreateV2.source (#1909).
    • Drop TaskUpdateV2.version (#1905).
    • Revamp access-control for /api/v2/task/ endpoints, based on task-group attributes (#1817).
    • Update /api/v2/task/ endpoints and schemas with new task attributes (#1856).
    • Forbid changing TaskV2.name (#1925).
  • Task collection:
    • Improve preliminary checks in task-collection endpoints (#1861).
    • Refactor split between task-collection endpoints and background tasks (#1861).
    • Create TaskGroupV2 object within task-collection endpoints (#1861).
    • Fix response of task-collection endpoint (#1902).
    • Automatically discover PyPI package version if missing or invalid (#1858, #1861, #1902).
    • Use appropriate log-file path in collection-status endpoint (#1902).
    • Add task authors to manifest schema (#1856).
    • Do not use source for custom task collection (#1893).
    • Rename custom-task-collection request-body field from source to label (#1896).
    • Improve error messages from task collection (#1913).
    • Forbid non-unique task names in ManifestV2 (#1925).
  • Workflows and workflow tasks:
    • Introduce additional checks in POST-workflowtask endpoint, concerning non-active or non-accessible tasks (#1817).
    • Introduce additional intormation in GET-workflow endpoint, concerning non-active or non-accessible tasks (#1817).
    • Introduce additional intormation in PATCH-workflow endpoint, concerning non-active or non-accessible tasks (#1868, #1869).
    • Stop logging warnings for non-common tasks in workflow export (#1893).
    • Drop WorkflowTaskCreateV2.order (#1906).
    • Update endpoints for workflow import/export (#1925, #1939, #1960).
  • Datasets:
    • Remove TaskDumpV2.owner attribute (#1909).
  • Jobs:
    • Prevent job submission if includes non-active or non-accessible tasks (#1817).
    • Remove rate limit for POST /project/{project_id}/job/submit/ (#1944).
  • Admin:
    • Remove owner from GET admin/v2/task/ (#1909).
    • Deprecate kind query parameter for /admin/v2/task/ (#1893).
    • Add origin and pkg_name query parameters to GET /admin/v2/task-group/ (#1979).
  • Schemas:
    • Forbid extras in TaskCollectPipV2 (#1891).
    • Forbid extras in all Create/Update/Import schemas (#1895).
    • Deprecate internal TaskCollectPip schema in favor of TaskGroupV2 (#1861).
• Database:
  • Introduce TaskGroupV2 table (#1817, #1856).
  • Add timestamp_created column to LinkUserGroup table (#1850).
  • Add TaskV2 attributes authors, tags, category and modality (#1856).
  • Add update-db-data script (#1820, #1888).
  • Add taskgroupv2_id foreign key to CollectionStateV2 (#1867).
  • Make TaskV2.source nullable and drop its uniqueness constraint (#1861).
  • Add TaskGroupV2 columns wheel_path, pinned_package_versions (#1861).
  • Clean up alembic migration scripts (#1894).
  • Verify task-group non-duplication constraint in 2.7.0 data-migration script (#1927).
  • Normalize pkg_name in 2.7.0 data-migration script (#1930).
  • Deprecate DB_ENGINE="postgres" configuration variable (#1946).
• Runner:
  • Do not create local folders with 755 permissions unless FRACTAL_BACKEND_RUNNER="slurm" (#1923).
  • Fix bug of SSH/SFTP commands not acquiring lock (#1949).
  • Fix bug of unhandled exception in SSH/SLURM executor (#1963).
  • Always remove task-subfolder compressed archive (#1949).
• Task collection:
  • Create base directory (in SSH mode), if missing (#1949).
  • Fix bug of SSH/SFTP commands not acquiring lock (#1949).
• SSH:
  • Improve logging for SSH-connection-locking flow (#1949).
  • Introduce FractalSSH.fetch_file and FractalSSH.read_remote_json_file (#1949).
  • Use paramiko.sftp_client.SFTPClient methods directly rathen than fabric wrappers (#1949).
  • Disable prefetching for SFTPClient.get (#1949).
• Internal:
  • Update _create_first_group so that it only searches for UserGroups with a given name (#1964).
• Dependencies:
  • Bump fastapi to 0.115 (#1942).
  • Remove pip extra postgres, corresponding to psycopg2+asyncpg (#1946).
  • Deprecate python3.9 (#1946).
• Testing:
  • Benchmark GET /api/v2/task-group/ (#1922).
  • Use new ubuntu22-slurm-multipy image, with Python3.12 and with Python-version specific venvs (#1946, #1969).
  • Get DB_ENGINE variable from os.environ rather than from installed packages (#1968).

2.6.4

• Database
  • Fix use of naming convention for database schema-migration scripts (#1819).
• Testing:
  • Test alembic downgrade base (#1819).
  • Add GET /api/v2/task/ to benchmarks (#1825).

2.6.3

• API:
  • Introduce GET /auth/current-user/viewer-paths/ endpoint (#1816).
  • Add viewer_paths attribute to UserGroup endpoints (#1816).
• Database:
  • Add viewer_paths column to UserGroup table (#1816).
• Runner:
  • Anticipate wait_thread.shutdown_callback assignment in FractalSlurmExecutor, to avoid an uncaught exception (#1815).

2.6.2

• Allow setting UserSettings attributes to None in standard/strict PATCH endpoints (#1814).

2.6.1

• App (internal):
  • Remove FRACTAL_SLURM_SSH_HOST, FRACTAL_SLURM_SSH_USER, FRACTAL_SLURM_SSH_PRIVATE_KEY_PATH and FRACTAL_SLURM_SSH_WORKING_BASE_DIR from Settings (#1804).
• Database:
  • Drop slurm_user, slurm_accounts and cache_dir columns from UserOAuth (#1804)

2.6.0

WARNING: This release requires running fractalctl update-db-data (after fractalctl set-db).

• API:
  • Introduce user-settings API, in /auth/users/{user_id}/settings/ and /auth/current-user/settings/ (#1778, #1807).
  • Add the creation of empty settings to UserManager.on_after_register hook (#1778).
  • Remove deprecated user's attributes (slurm_user, cache_dir, slurm_accounts) from API, in favor of new UserSetting ones (#1778).
  • Validate user settings in endpoints that rely on them (#1778).
  • Propagate user settings to background tasks when needed (#1778).
• Database:
  • Introduce new user_settings table, and link it to user_oauth (#1778).
• Internal:
• Remove redundant string validation in FractalSSH.remove_folder and TaskCollectCustomV2 (#1810).
• Make validate_cmd more strict about non-string arguments (#1810).

2.5.2

• App:
  • Replace fractal_ssh attribute with fractal_ssh_list, in app.state (#1790).
  • Move creation of SSH connections from app startup to endpoints (#1790).
• Internal
  • Introduce FractalSSHList, in view of support for multiple SSH/Slurm service users (#1790).
  • Make FractalSSH.close() more aggressively close Transport attribute (#1790).
  • Set look_for_keys=False for paramiko/fabric connection (#1790).
• Testing:
  • Add fixture to always test that threads do not accumulate during tests (#1790).

2.5.1

• API:
  • Make WorkflowTaskDumpV2 attributes task_id and task optional (#1784).
  • Add validation for user-provided strings that execute commands with subprocess or remote-shell (#1767).
• Runner and task collection:
  • Validate commands before running them via subprocess or fabric (#1767).

2.5.0

WARNING: This release has a minor API bug when displaying a V2 dataset with a history that contains legacy tasks. It's recommended to update to 2.5.1.

This release removes support for including V1 tasks in V2 workflows. This comes with changes to the database (data and metadata), to the API, and to the V2 runner.

• Runner:
  • Deprecate running v1 tasks within v2 workflows (#1721).
• Database:
  • Remove Task.is_v2_compatible column (#1721).
  • For table WorkflowTaskV2, drop is_legacy_task and task_legacy_id columns, remove task_legacy ORM attribute, make task_id required, make task required (#1721).
• API:
  • Drop v1-v2-task-compatibility admin endpoint (#1721).
  • Drop /task-legacy/ endpoint (#1721).
  • Remove legacy task code branches from WorkflowTaskV2 CRUD endpoints (#1721).
  • Add OAuth accounts info to UserRead at .oauth_accounts (#1765).
• Testing:
  • Improve OAuth Github Action to test OAuth account flow (#1765).

2.4.2

• App:
  • Improve logging in fractalctl set-db (#1764).
• Runner:
  • Add --set-home to sudo -u impersonation command, to fix Ubuntu18 behavior (#1762).
• Testing:
  • Start tests of migrations from valid v2.4.0 database (#1764).

2.4.1

This is mainly a bugfix release, re-implementing a check that was removed in 2.4.0.

• API:
  • Re-introduce check for existing-user-email in PATCH /auth/users/{id}/ (#1760).

2.4.0

This release introduces support for user groups, but without linking it to any access-control rules (which will be introduced later).

NOTE: This release requires running the fractalctl update-db-data script.

• App:
  • Move creation of first user from application startup into fractalctl set-db command (#1738, #1748).
  • Add creation of default user group into fractalctl set-db command (#1738).
  • Create update-db-script for current version, that adds all users to default group (#1738).
• API:
  • Added /auth/group/ and /auth/group-names/ routers (#1738, #1752).
  • Implement /auth/users/{id}/ POST/PATCH routes in fractal-server (#1738, #1747, #1752).
  • Introduce UserUpdateWithNewGroupIds schema for PATCH /auth/users/{id}/ (#1747, #1752).
  • Add UserManager.on_after_register hook to add new users to default user group (#1738).
• Database:
  • Added new usergroup and linkusergroup tables (#1738).
• Internal
  • Refactored fractal_server.app.auth and fractal_server.app.security (#1738)/
  • Export all relevant modules in app.models, since it matters e.g. for autogenerate-ing migration scripts (#1738).
• Testing
  • Add UserGroup validation to scripts/validate_db_data_with_read_schemas.py (#1746).

2.3.11

• SSH runner:
  • Move remote-folder creation from submit_workflow to more specific _process_workflow (#1728).
• Benchmarks:
  • Add GET /auth/token/login/ to tested endpoints (#1720).
• Testing:
  • Update GitHub actions upload-artifact and download-artifact to v4 (#1725).

2.3.10

• Fix minor bug in zipping-job logging (#1716).

2.3.9

• Add logging for zipping-job-folder operations (#1714).

2.3.8

NOTE: FRACTAL_API_V1_MODE="include_without_submission" is now transformed into FRACTAL_API_V1_MODE="include_read_only".

• API:
  • Support read-only mode for V1 (#1701).
  • Improve handling of zipped job-folder in download-logs endpoints (#1702).
• Runner:
  • Improve database-error handling in V2 job execution (#1702).
  • Zip job folder after job execution (#1702).
• App:
  • UvicornWorker is now imported from uvicorn-worker (#1690).
• Testing:
  • Remove HAS_LOCAL_SBATCH variable and related if-branches (#1699).
• Benchmarks:
  • Add GET /auth/current-user/ to tested endpoints (#1700).
• Dependencies:
  • Update mkdocstrings to ^0.25.2 (#1707).
  • Update fastapi to ^0.112.0 (#1705).

2.3.7

• SSH SLURM executor:
  • Handle early shutdown in SSH executor (#1696).
• Task collection:
  • Introduce a new configuration variable FRACTAL_MAX_PIP_VERSION to pin task-collection pip (#1675).

2.3.6

• API:
  • When creating a WorkflowTask, do not pre-populate its top-level arguments based on JSON Schema default values (#1688).
• Dependencies:
  • Update sqlmodel to ^0.0.21 (#1674).
  • Add uvicorn-worker (#1690).

2.3.5

WARNING: The pre_submission_commands SLURM configuration is included as an experimental feature, since it is still not useful for its main intended goal (calling module load before running sbatch).

• SLURM runners:
  • Expose gpus SLURM parameter (#1678).
  • For SSH executor, add pre_submission_commands (#1678).
  • Removed obsolete arguments from get_slurm_config function (#1678).
• SSH features:
  • Add FractalSSH.write_remote_file method (#1678).

2.3.4

• SSH SLURM runner:
  • Refactor compress_folder and extract_archive modules, and stop using tarfile library (#1641).
• API:
  • Introduce FRACTAL_API_V1_MODE=include_without_submission to include V1 API but forbid job submission (#1664).
• Testing:
  • Do not test V1 API with DB_ENGINE="postgres-psycopg" (#1667).
  • Use new Fractal SLURM containers in CI (#1663).
  • Adapt tests so that they always refer to the current Python version (the one running pytest), when needed; this means that we don't require the presence of any additional Python version in the development environment, apart from the current one (#1633).
  • Include Python3.11 in some tests (#1669).
  • Simplify CI SLURM Dockerfile after base-image updates (#1670).
  • Cache ubuntu22-slurm-multipy Docker image in CI (#1671).
  • Add oauth.yaml GitHub action to test OIDC authentication (#1665).

2.3.3

This release fixes a SSH-task-collection bug introduced in version 2.3.1.

• API:
  • Expose new superuser-restricted endpoint GET /api/settings/ (#1662).
• SLURM runner:
  • Make FRACTAL_SLURM_SBATCH_SLEEP configuration variable float (#1658).
• SSH features:
  • Fix wrong removal of task-package folder upon task-collection failure (#1649).
  • Remove FractalSSH.rename_folder method (#1654).
• Testing:
  • Refactor task-collection fixtures (#1637).

2.3.2

WARNING: The remove-remote-venv-folder in the SSH task collection is broken (see issue 1633). Do not deploy this version in an SSH-based fractal-server instance.

• API:
  • Fix incorrect zipping of structured job-log folders (#1648).

2.3.1

This release includes a bugfix for task names with special characters.

WARNING: The remove-remote-venv-folder in the SSH task collection is broken (see issue 1633). Do not deploy this version in an SSH-based fractal-server instance.

• Runner:
  • Improve sanitization of subfolder names (commits from 3d89d6ba104d1c6f11812bc9de5cbdff25f81aa2 to 426fa3522cf2eef90d8bd2da3b2b8a5b646b9bf4).
• API:
  • Improve error message when task-collection Python is not defined (#1640).
  • Use a single endpoint for standard and SSH task collection (#1640).
• SSH features:
  • Remove remote venv folder upon failed task collection in SSH mode (#1634, #1640).
  • Refactor FractalSSH (#1635).
  • Set fabric.Connection.forward_agent=False (#1639).
• Testing:
  • Improved testing of SSH task-collection API (#1640).
  • Improved testing of FractalSSH methods (#1635).
  • Stop testing SQLite database for V1 in CI (#1630).

2.3.0

This release includes two important updates: 1. An Update update to task-collection configuration variables and logic. 2. The first released version of the experimental SSH features.

Re: task-collection configuration, we now support two main use cases:

1. When running a production instance (including on a SLURM cluster), you should set e.g. FRACTAL_TASKS_PYTHON_DEFAULT_VERSION=3.10, and make sure that FRACTAL_TASKS_PYTHON_3_10=/some/python is an absolute path. Optionally, you can define other variables like FRACTAL_TASKS_PYTHON_3_9, FRACTAL_TASKS_PYTHON_3_11 or FRACTAL_TASKS_PYTHON_3_12.

2. If you leave FRACTAL_TASKS_PYTHON_DEFAULT_VERSION unset, then only the Python interpreter that is currently running fractal-server can be used for task collection.

WARNING: If you don't set FRACTAL_TASKS_PYTHON_DEFAULT_VERSION, then you will only have a single Python interpreter available for tasks (namely the one running fractal-server).

• API:
  • Introduce api/v2/task/collect/custom/ endpoint (#1607, #1613, #1617, #1629).
• Task collection:
  • Introduce task-collection Python-related configuration variables (#1587).
  • Always set Python version for task collection, and only use FRACTAL_TASKS_PYTHON_X_Y variables (#1587).
  • Refactor task-collection functions and schemas (#1587, #1617).
  • Remove TaskCollectStatusV2 and get_collection_data internal schema/function (#1598).
  • Introduce CollectionStatusV2 enum for task-collection status (#1598).
  • Reject task-collection request if it includes a wheel file and a version (#1608). SSH features:
  • Introduce fractal_server/ssh subpackage (#1545, #1599, #1611).
  • Introduce SSH executor and runner (#1545).
  • Introduce SSH task collection (#1545, #1599, #1626).
  • Introduce SSH-related configuration variables (#1545).
  • Modify app lifespan to handle SSH connection (#1545).
  • Split app/runner/executor/slurm into sudo and ssh subfolders (#1545).
  • Introduce FractalSSH object which is a wrapper class around fabric.Connection object. It provides a lock to avoid loss of ssh instructions and a custom timeout (#1618)
• Dependencies:
  • Update sqlmodel to ^0.0.19 (#1584).
  • Update pytest-asyncio to ^0.23 (#1558).
• Testing:
  • Test the way FractalProcessPoolExecutor spawns processes and threads (#1579).
  • Remove event_loop fixture: every test will run on its own event loop (#1558).
  • Test task collection with non-canonical package name (#1602).

2.2.0

This release streamlines options for the Gunicorn startup command, and includes two new experimental features.

NOTE 1: you can now enable custom Gunicorn worker/logger by adding the following options to the gunicorn startup command: - --worker-class fractal_server.gunicorn_fractal.FractalWorker - --logger-class fractal_server.gunicorn_fractal.FractalGunicornLogger

NOTE 2: A new experimental local runner is available, which uses processes instead of threads and support shutdown. You can try it out with the configuration variable FRACTAL_BACKEND_RUNNER=local_experimental

NOTE 3: A new PostgreSQL database adapter is available, fully based on psycopg3 (rather than pyscopg2+asyncpg). You can try it out with the configuration variable DB_ENGINE=postgres-psycopg (note that this requires the pip install extra postgres-psycopg-binary).

• API:
  • Add extensive logs to DELETE /api/v2/project/{project_id} (#1532).
  • Remove catch of IntegrityError in POST /api/v1/project (#1530).
• App and deployment:
  • Move FractalGunicornLogger and FractalWorker into fractal_server/gunicorn_fractal.py (#1535).
  • Add custom gunicorn/uvicorn worker to handle SIGABRT signal (#1526).
  • Store list of submitted jobs in app state (#1538).
  • Add logic for graceful shutdown for job slurm executors (#1547).
• Runner:
  • Change structure of job folders, introducing per-task subfolders (#1523).
  • Rename internal workflow_dir and workflow_dir_user variables to local/remote (#1534).
  • Improve handling of errors in submit_workflow background task (#1556, #1566).
  • Add new local_experimental runner, based on ProcessPoolExecutor (#1544, #1566).
• Database:
  • Add new Postgres adapter psycopg (#1562).
• Dependencies
  • Add fabric to dev dependencies (#1518).
  • Add new postgres-psycopg-binary extra (#1562).
• Testing:
  • Extract pytest-docker fixtures into a dedicated module (#1516).
  • Rename SLURM containers in CI (#1516).
  • Install and run SSH daemon in CI containers (#1518).
  • Add unit test of SSH connection via fabric/paramiko (#1518).
  • Remove obsolete folders from tests/data (#1517).

2.1.0

This release fixes a severe bug where SLURM-executor auxiliary threads are not joined when a Fractal job ends.

• App:
  • Add missing join for wait_thread upon FractalSlurmExecutor exit (#1511).
  • Replace startup/shutdown events with lifespan event (#1501).
• API:
  • Remove Path.resolve from the submit-job endpoints and add validator for Settings.FRACTAL_RUNNER_WORKING_BASE_DIR (#1497).
• Testing:
  • Improve dockerfiles for SLURM (#1495, #1496).
  • Set short timeout for docker compose down (#1500).

2.0.6

NOTE: This version changes log formats. For uvicorn logs, this change requires no action. For gunicorn, logs formats are only changed by adding the following command-line option: gunicorn ... --logger-class fractal_server.logger.gunicorn_logger.FractalGunicornLogger.

• API:
  • Add FRACTAL_API_V1_MODE environment variable to include/exclude V1 API (#1480).
  • Change format of uvicorn loggers (#1491).
  • Introduce FractalGunicornLogger class (#1491).
• Runner:
  • Fix missing .log files in server folder for SLURM jobs (#1479).
• Database:
  • Remove UserOAuth.project_list and UserOAuth.project_list_v2 relationships (#1482).
• Dev dependencies:
  • Bump pytest to 8.1.* (#1486).
  • Bump coverage to 7.5.* (#1486).
  • Bump pytest-docker to 3.1.* (#1486).
  • Bump pytest-subprocess to ^1.5 (#1486).
• Benchmarks:
  • Move populate_db scripts into benchmark folder (#1489).

2.0.5

• API:
  • Add GET /admin/v2/task/ (#1465).
  • Improve error message in DELETE-task endpoint (#1471).
• Set JobV2 folder attributes from within the submit-job endpoint (#1464).
• Tests:
  • Make SLURM CI work on MacOS (#1476).

2.0.4

• Add FRACTAL_SLURM_SBATCH_SLEEP configuration variable (#1467).

2.0.3

WARNING: This update requires running a fix-db script, via fractalctl update-db-data.

• Database:
  • Create fix-db script to remove images and history from dataset dumps in V1/V2 jobs (#1456).
• Tests:
  • Split test_full_workflow_v2.py into local/slurm files (#1454).

2.0.2

WARNING: Running this version on a pre-existing database (where the jobsv2 table has some entries) is broken. Running this version on a freshly-created database works as expected.

• API:
  • Fix bug in status endpoint (#1449).
  • Improve handling of out-of-scope scenario in status endpoint (#1449).
  • Do not include dataset history in JobV2.dataset_dump (#1445).
  • Forbid extra arguments in DumpV2 schemas (#1445).
• API V1:
  • Do not include dataset history in ApplyWorkflow.{input,output}_dataset_dump (#1453).
• Move settings logs to check_settings and use fractal-server set_logger (#1452).
• Benchmarks:
  • Handle some more errors in benchmark flow (#1445).
• Tests:
  • Update testing database to version 2.0.1 (#1445).

2.0.1

• Database/API:
  • Do not include dataset_dump.images in JobV2 table (#1441).
• Internal functions:
  • Introduce more robust reset_logger_handlers function (#1425).
• Benchmarks:
  • Add POST /api/v2/project/project_id/dataset/dataset_id/images/query/ in bechmarks to evaluate the impact of the number of images during the query (#1441).
• Development:
  • Use poetry 1.8.2 in GitHub actions and documentation.

2.0.0

Major update.

1.4.10

WARNING: Starting from this version, the dependencies for the slurm extra are required; commands like pip install fractal-server[slurm,postgres] must be replaced by pip install fractal-server[postgres].

• Dependencies:
  • Make clusterfutures and cloudpickle required dependencies (#1255).
  • Remove slurm extra from package (#1255).
• API:
  • Handle invalid history file in GET /project/{project_id}/dataset/{dataset_id}/status/ (#1259).
• Runner:
  • Add custom _jobs_finished function to check the job status and to avoid squeue errors (#1266)

1.4.9

This release is a follow-up of 1.4.7 and 1.4.8, to mitigate the risk of job folders becoming very large.

• Runner:
  • Exclude history from TaskParameters object for parallel tasks, so that it does not end up in input pickle files (#1247).

1.4.8

This release is a follow-up of 1.4.7, to mitigate the risk of job folders becoming very large.

• Runner:
  • Exclude metadata["image"] from TaskParameters object for parallel tasks, so that it does not end up in input pickle files (#1245).
  • Exclude components list from workflow.log logs (#1245).
• Database:
  • Remove spurious logging of fractal_server.app.db string (#1245).

1.4.7

This release provides a bugfix (PR 1239) and a workaround (PR 1238) for the SLURM runner, which became relevant for the use case of processing a large dataset (300 wells with 25 cycles each).

• Runner:
  • Do not include metadata["image"] in JSON file with task arguments (#1238).
  • Add FRACTAL_RUNNER_TASKS_INCLUDE_IMAGE configuration variable, to define exceptions where tasks still require metadata["image"] (#1238).
  • Fix bug in globbing patterns, when copying files from user-side to server-side job folder in SLURM executor (#1239).
• API:
  • Fix error message for rate limits in apply-workflow endpoint (#1231).
• Benchmarks:
  • Add more scenarios, as per issue #1184 (#1232).

1.4.6

• API:
  • Add GET /admin/job/{job_id} (#1230).
  • Handle FileNotFound in GET /project/{project_id}/job/{job_id}/ (#1230).

1.4.5

• Remove CORS middleware (#1228).
• Testing:
  • Fix migrations.yml GitHub action (#1225).

1.4.4

• API:
  • Add rate limiting to POST /{project_id}/workflow/{workflow_id}/apply/ (#1199).
  • Allow users to read the logs of ongoing jobs with GET /project/{project_id}/job/{job_id}/, using show_tmp_logs query parameter (#1216).
  • Add log query parameter in GET {/api/v1/job/,/api/v1/{project.id}/job/,/admin/job/}, to trim response body (#1218).
  • Add args_schema query parameter in GET /api/v1/task/ to trim response body (#1218).
  • Add history query parameter in GET {/api/v1/dataset/,/api/v1/project/{project.id}/dataset/} to trim response body (#1219).
  • Remove task_list from job.workflow_dump creation in /api/v1/{project_id}/workflow/{workflow_id}/apply/(#1219)
  • Remove task_list from WorkflowDump Pydantic schema (#1219)
• Dependencies:
  • Update fastapi to ^0.109.0 (#1222).
  • Update gunicorn to ^21.2.0 (#1222).
  • Update aiosqlite to ^0.19.0 (#1222).
  • Update uvicorn to ^0.27.0 (#1222).

1.4.3

WARNING:

This update requires running a fix-db script, via fractalctl update-db-data.

• API:
  • Improve validation of UserCreate.slurm_accounts (#1162).
  • Add timestamp_created to WorkflowRead, WorkflowDump, DatasetRead and DatasetDump (#1152).
  • Make all dumps in ApplyWorkflowRead non optional (#1175).
  • Ensure that timestamps in Read schemas are timezone-aware, regardless of DB_ENGINE (#1186).
  • Add timezone-aware timestamp query parameters to all /admin endpoints (#1186).
• API (internal):
  • Change the class method Workflow.insert_task into the auxiliary function _workflow_insert_task (#1149).
• Database:
  • Make WorkflowTask.workflow_id and WorfklowTask.task_id not nullable (#1137).
  • Add Workflow.timestamp_created and Dataset.timestamp_created columns (#1152).
  • Start a new current.py fix-db script (#1152, #1195).
  • Add to migrations.yml a new script (validate_db_data_with_read_schemas.py) that validates test-DB data with Read schemas (#1187).
  • Expose fix-db scripts via command-line option fractalctl update-db-data (#1197).
• App (internal):
  • Check in Settings that psycopg2, asyngpg and cfut, if required, are installed (#1167).
  • Split DB.set_db into sync/async methods (#1165).
  • Rename DB.get_db into DB.get_async_db (#1183).
  • Normalize names of task packages (#1188).
• Testing:
  • Update clean_db_fractal_1.4.1.sql to clean_db_fractal_1.4.2.sql, and change migrations.yml target version (#1152).
  • Reorganise the test directory into subdirectories, named according to the order in which we want the CI to execute them (#1166).
  • Split the CI into two independent jobs, Core and Runner, to save time through parallelisation (#1204).
• Dependencies:
  • Update python-dotenv to version 0.21.0 (#1172).
• Runner:
  • Remove JobStatusType.RUNNING, incorporating it into JobStatusType.SUBMITTED (#1179).
• Benchmarks:
  • Add fractal_client.py and populate_script_v2.py for creating different database status scenarios (#1178).
  • Add a custom benchmark suite in api_bench.py.
  • Remove locust.
• Documentation:
  • Add the minimum set of environment variables required to set the database and start the server (#1198).

1.4.2

WARNINGs:

1. This update requires running a fix-db script, available at https://raw.githubusercontent.com/fractal-analytics-platform/fractal-server/1.4.2/scripts/fix_db/current.py.
2. Starting from this version, non-verified users have limited access to /api/v1/ endpoints. Before the upgrade, all existing users must be manually set to verified.

• API:
  • Prevent access to GET/PATCH task endpoints for non-verified users (#1114).
  • Prevent access to task-collection and workflow-apply endpoints for non-verified users (#1099).
  • Make first-admin-user verified (#1110).
  • Add the automatic setting of ApplyWorkflow.end_timestamp when patching ApplyWorkflow.status via PATCH /admin/job/{job_id} (#1121).
  • Change ProjectDump.timestamp_created type from datetime to str (#1120).
  • Change _DatasetHistoryItem.workflowtask type into WorkflowTaskDump (#1139).
  • Change status code of stop-job endpoints to 202 (#1151).
• API (internal):
  • Implement cascade operations explicitly, in DELETE endpoints for datasets, workflows and projects (#1130).
  • Update GET /project/{project_id}/workflow/{workflow_id}/job/ to avoid using Workflow.job_list (#1130).
  • Remove obsolete sync-database dependency from apply-workflow endpoint (#1144).
• Database:
  • Add ApplyWorkflow.project_dump column (#1070).
  • Provide more meaningful names to fix-db scripts (#1107).
  • Add Project.timestamp_created column, with timezone-aware default (#1102, #1131).
  • Remove Dataset.list_jobs_input and Dataset.list_jobs_output relationships (#1130).
  • Remove Workflow.job_list (#1130).
• Runner:
  • In SLURM backend, use slurm_account (as received from apply-workflow endpoint) with top priority (#1145).
  • Forbid setting of SLURM account from WorkflowTask.meta or as part of worker_init variable (#1145).
  • Include more info in error message upon sbatch failure (#1142).
  • Replace sbatch --chdir option with -D, to support also slurm versions before 17.11 (#1159).
• Testing:
  • Extended systematic testing of database models (#1078).
  • Review MockCurrentUser fixture, to handle different kinds of users (#1099).
  • Remove persist from MockCurrentUser (#1098).
  • Update migrations.yml GitHub Action to use up-to-date database and also test fix-db script (#1101).
  • Add more schema-based validation to fix-db current script (#1107).
  • Update .dict() to .model_dump() for SQLModel objects, to fix some DeprecationWarnings(##1133).
  • Small improvement in schema coverage (#1125).
  • Add unit test for security module (#1036).
• Dependencies:
  • Update sqlmodel to version 0.0.14 (#1124).
• Benchmarks:
  • Add automatic benchmark system for API's performances (#1123)
• App (internal):
  • Move _create_first_user from main to security module, and allow it to create multiple regular users (#1036).

1.4.1

• API:
  • Add GET /admin/job/{job_id}/stop/ and GET /admin/job/{job_id}/download/ endpoints (#1059).
  • Use DatasetDump and WorkflowDump models for "dump" attributes of ApplyWorkflowRead (#1049, #1082).
  • Add slurm_accounts to User schemas and add slurm_account to ApplyWorkflow schemas (#1067).
  • Prevent providing a package_version for task collection from a .whl local package (#1069).
  • Add DatasetRead.project and WorkflowRead.project attributes (#1082).
• Database:
  • Make ApplyWorkflow.workflow_dump column non-nullable (#1049).
  • Add UserOAuth.slurm_accounts and ApplyWorkflow.slurm_account columns (#1067).
  • Add script for adding ApplyWorkflow.user_email (#1058).
  • Add Dataset.project and Workflow.project relationships (#1082).
  • Avoid using Project relationships dataset_list or workflow_list within some GET endpoints (#1082).
  • Fully remove Project relationships dataset_list, workflow_list and job_list (#1091).
• Testing:
  • Only use ubuntu-22.04 in GitHub actions (#1061).
  • Improve unit testing of database models (#1082).
• Dependencies:
  • Pin bcrypt to 4.0.1 to avoid warning in passlib (#1060).
• Runner:
  • Set SLURM-job working directory to job.working_dir_user through --chdir option (#1064).

1.4.0

• API:
  • Major endpoint changes:
    • Add trailing slash to all endpoints' paths (#1003).
    • Add new admin-area endpoints restricted to superusers at /admin (#947, #1009, #1032).
    • Add new GET endpoints api/v1/job/ and api/v1/project/{project_id}/workflow/{workflow_id}/job/ (#969, #1003).
    • Add new GET endpoints api/v1/dataset/ and api/v1/workflow/ (#988, #1003).
    • Add new GET endpoint api/v1/project/{project_id}/dataset/ (#993).
    • Add PATCH /admin/job/{job_id}/ endpoint (#1030, #1053).
    • Move GET /auth/whoami/ to GET /auth/current-user/ (#1013).
    • Move PATCH /auth/users/me/ to PATCH /auth/current-user/ (#1013, #1035).
    • Remove DELETE /auth/users/{id}/ endpoint (#994).
    • Remove GET /auth/users/me/ (#1013).
    • Remove POST /auth/forgot-password/, /auth/reset-password/, /auth/request-verify-token/, /auth/verify/ (#1033).
    • Move GET /auth/userlist/ to GET /auth/users/ (#1033).
  • New behaviors or responses of existing endpoints:
    • Change response of /api/v1/project/{project_id}/job/{job_id}/stop/ endpoint to 204 no-content (#967).
    • Remove dataset_list attribute from ProjectRead, which affects all GET endpoints that return some project (#993).
    • Make it possible to delete a Dataset, Workflow or Project, even when it is in relationship to an ApplyWorkflow - provided that the ApplyWorkflow is not pending or running (#927, #973).
    • Align ApplyWorkflowRead with new ApplyWorkflow, which has optional foreign keys project_id, workflow_id, input_dataset_id, and output_dataset_id (#984).
    • Define types for ApplyWorkflowRead "dump" attributes (#990). WARNING: reverted with #999.
  • Internal changes:
    • Move all routes definitions into fractal_server/app/routes (#976).
    • Fix construction of ApplyWorkflow.workflow_dump, within apply endpoint (#968).
    • Fix construction of ApplyWorkflow attributes input_dataset_dump and output_dataset_dump, within apply endpoint (#990).
    • Remove asyncio.gather, in view of SQLAlchemy2 update (#1004).
• Database:
  • Make foreign-keys of ApplyWorkflow (project_id, workflow_id, input_dataset_id, output_dataset_id) optional (#927).
  • Add columns input_dataset_dump, output_dataset_dump and user_email to ApplyWorkflow (#927).
  • Add relations Dataset.list_jobs_input and Dataset.list_jobs_output (#927).
  • Make ApplyWorkflow.start_timestamp non-nullable (#927).
  • Remove "cascade": "all, delete-orphan" from Project.job_list (#927).
  • Add Workflow.job_list relation (#927).
  • Do not use Enums as column types (e.g. for ApplyWorkflow.status), but only for (de-)serialization (#974).
  • Set pool_pre_ping option to True, for asyncpg driver (#1037).
  • Add script for updating DB from 1.4.0 to 1.4.1 (#1010)
  • Fix missing try/except in sync session (#1020).
• App:
  • Skip creation of first-superuser when one superuser already exists (#1006).
• Dependencies:
  • Update sqlalchemy to version >=2.0.23,<2.1 (#1044).
  • Update sqlmodel to version 0.0.12 (#1044).
  • Upgrade asyncpg to version 0.29.0 (#1036).
• Runner:
  • Refresh DB objects within submit_workflow (#927).
• Testing:
  • Add await db_engine.dispose() in db_create_tables fixture (#1047).
  • Set debug=False in event_loop fixture (#1044).
  • Improve test_full_workflow.py (#971).
  • Update pytest-asyncio to v0.21 (#1008).
  • Fix CI issue related to event loop and asyncpg (#1012).
  • Add GitHub Action testing database migrations (#1010).
  • Use greenlet v3 in poetry.lock (#1044).
• Documentation:
  • Add OAuth2 example endpoints to Web API page (#1034, #1038).
• Development:
  • Use poetry 1.7.1 (#1043).

1.3.14 (do not use!)

WARNING: This version introduces a change that is then reverted in 1.4.0, namely it sets the ApplyWorkflow.status type to Enum, when used with PostgreSQL. It is recommended to not use it, and upgrade to 1.4.0 directly.

• Make Dataset.resource_list an ordering_list, ordered by Resource.id (#951).
• Expose redirect_url for OAuth clients (#953).
• Expose JSON Schema for the ManifestV1 Pydantic model (#942).
• Improve delete-resource endpoint (#943).
• Dependencies:
  • Upgrade sqlmodel to 0.0.11 (#949).
• Testing:
  • Fix bug in local tests with Docker/SLURM (#948).

1.3.13

• Configure sqlite WAL to avoid "database is locked" errors (#860).
• Dependencies:
  • Add sqlalchemy[asyncio] extra, and do not directly require greenlet (#895).
  • Fix cloudpickle-version definition in pyproject.toml (#937).
  • Remove obsolete sqlalchemy_utils dependency (#939).
• Testing:
  • Use ubuntu-22 for GitHub CI (#909).
  • Run GitHub CI both with SQLite and Postgres (#915).
  • Disable postgres service in GitHub action when running tests with SQLite (#931).
  • Make test_commands.py tests stateless, also when running with Postgres (#917).
• Documentation:
  • Add information about minimal supported SQLite version (#916).

1.3.12

• Project creation:
  • Do not automatically create a dataset upon project creation (#897).
  • Remove ProjectCreate.default_dataset_name attribute (#897).
• Dataset history:
  • Create a new (non-nullable) history column in Dataset table (#898, #901).
  • Deprecate history handling in /project/{project_id}/job/{job_id} endpoint (#898).
  • Deprecate HISTORY_LEGACY (#898).
• Testing:
  • Remove obsolete fixture slurm_config (#903).

1.3.11

This is mainly a bugfix release for the PermissionError issue.

• Fix PermissionErrors in parallel-task metadata aggregation for the SLURM backend (#893).
• Documentation:
  • Bump mkdocs-render-swagger-plugin to 0.1.0 (#889).
• Testing:
  • Fix poetry install command and poetry version in GitHub CI (#889).

1.3.10

Warning: updating to this version requires changes to the configuration variable

• Updates to SLURM interface:
  • Remove sudo-requiring ls calls from FractalFileWaitThread.check (#885);
  • Change default of FRACTAL_SLURM_POLL_INTERVAL to 5 seconds (#885);
  • Rename FRACTAL_SLURM_OUTPUT_FILE_GRACE_TIME configuration variables into FRACTAL_SLURM_ERROR_HANDLING_INTERVAL (#885);
  • Remove FRACTAL_SLURM_KILLWAIT_INTERVAL variable and corresponding logic (#885);
  • Remove _multiple_paths_exist_as_user helper function (#885);
  • Review type hints and default values of SLURM-related configuration variables (#885).
• Dependencies:
  • Update fastapi to version ^0.103.0 (#877);
  • Update fastapi-users to version ^12.1.0 (#877).

1.3.9

• Make updated-metadata collection robust for metadiff files consisting of a single null value (#879).
• Automate procedure for publishing package to PyPI (#881).

1.3.8

• Backend runner:
  • Add aggregation logic for parallel-task updated metadata (#852);
  • Make updated-metadata collection robust for missing files (#852, #863).
• Database interface:
• API:
  • Prevent user from bypassing workflow-name constraint via the PATCH endpoint (#867).
  • Handle error upon task collection, when tasks exist in the database but not on-disk (#874).
  • Add _check_project_exists helper function (#872).
• Configuration variables:
  • Remove DEPLOYMENT_TYPE variable and update alive endpoint (#875);
  • Introduce Settings.check_db method, and call it during inline/offline migrations (#855);
  • Introduce Settings.check_runner method (#875);
  • Fail if FRACTAL_BACKEND_RUNNER is "local" and FRACTAL_LOCAL_CONFIG_FILE is set but missing on-disk (#875);
  • Clean up Settings.check method and improve its coverage (#875);
• Package, repository, documentation:
  • Change fractal_server.common from being a git-submodule to being a regular folder (#859).
  • Pin documentation dependencies (#865).
  • Split app/models/project.py into two modules for dataset and project (#871).
  • Revamp documentation on database interface and on the corresponding configuration variables (#855).

1.3.7

• Oauth2-related updates (#822):
  • Update configuration of OAuth2 clients, to support OIDC/GitHub/Google;
  • Merge SQLModelBaseOAuthAccount and OAuthAccount models;
  • Update UserOAuth.oauth_accounts relationship and fix list_users endpoint accordingly;
  • Introduce dummy UserManager.on_after_login method;
  • Rename OAuthClient into OAuthClientConfig;
  • Revamp users-related parts of documentation.

1.3.6

• Update output_dataset.meta also when workflow execution fails (#843).
• Improve error message for unknown errors in job execution (#843).
• Fix log message incorrectly marked as "error" (#846).

1.3.5

• Review structure of dataset history (#803):
  • Re-define structure for history property of Dataset.meta;
  • Introduce "api/v1/project/{project_id}/dataset/{dataset_id}/status/" endpoint;
  • Introduce "api/v1/project/{project_id}/dataset/{dataset_id}/export_history/" endpoint;
  • Move legacy history to Dataset.meta["HISTORY_LEGACY"].
• Make first_task_index and last_task_index properties of ApplyWorkflow required (#803).
• Add docs_info and docs_link to Task model (#814)
• Accept TaskUpdate.version=None in task-patch endpoint (#818).
• Store a copy of the Workflow into the optional column ApplyWorkflow.workflow_dump at the time of submission (#804, #834).
• Prevent execution of multiple jobs with the same output dataset (#801).
• Transform non-absolute FRACTAL_TASKS_DIR into absolute paths, relative to the current working directory (#825).
• Error handling:
  • Raise an appropriate error if a task command is not executable (#800).
  • Improve handling of errors raised in get_slurm_config (#800).
• Documentation:
  • Clarify documentation about SlurmConfig (#798).
  • Update documentation configuration and GitHub actions (#811).
• Tests:
  • Move tests/test_common.py into fractal-common repository (#808).
  • Switch to docker compose v2 and unpin pyyaml version (#816).

1.3.4

• Support execution of a workflow subset (#784).
• Fix internal server error for invalid task_id in create_workflowtask endpoint (#782).
• Improve logging in background task collection (#776).
• Handle failures in submit_workflow without raising errors (#787).
• Simplify internal function for execution of a list of task (#780).
• Exclude common/tests and other git-related files from build (#795).
• Remove development dependencies Pillow and pytest-mock (#795).
• Remove obsolete folders from tests/data folder (#795).

1.3.3

• Pin Pydantic to v1 (#779).

1.3.2

• Add sqlalchemy naming convention for DB constraints, and add render_as_batch=True to do_run_migrations (#757).
• Fix bug in job-stop endpoint, due to missing default for FractalSlurmExecutor.wait_thread.shutdown_file (#768, #769).
• Fix bug upon inserting a task with meta=None into a Workflow (#772).

1.3.1

• Fix return value of stop-job endpoint (#764).
• Expose new GET WorkflowTask endpoint (#762).
• Clean up API modules (#762):
  • Split workflow/workflowtask modules;
  • Split tasks/task-collection modules.

1.3.0

• Refactor user model:
  • Switch from UUID4 to int for IDs (#660, #684).
  • Fix many-to-many relationship between users and project (#660).
  • Rename Project.user_member_list into Project.user_list (#660).
  • Add username column (#704).
• Update endpoints (see also 1.2->1.3 upgrade info in the documentation):
  • Review endpoint URLs (#669).
  • Remove foreign keys from payloads (#669).
• Update Task models, task collection and task-related endpoints:
  • Add version and owner columns to Task model (#704).
  • Set Task.version during task collection (#719).
  • Set Task.owner as part of create-task endpoint (#704).
  • For custom tasks, prepend owner to user-provided source (#725).
  • Remove default_args from Tasks model and from manifest tasks (#707).
  • Add args_schema and args_schema_version to Task model (#707).
  • Expose args_schema and args_schema_version in task POST/PATCH endpoints (#749).
  • Make Task.source task-specific rather than package-specific (#719).
  • Make Task.source unique (#725).
  • Update _TaskCollectPip methods, attributes and properties (#719).
  • Remove private/public options for task collection (#704).
  • Improve error message for missing package manifest (#704).
  • Improve behavior when task-collection folder already exists (#704).
  • Expose pinned_package_version for tasks collection (#744).
  • Restrict Task editing to superusers and task owners (#733).
  • Implement delete_task endpoint (#745).
• Update Workflow and WorkflowTask endpoints:
  • Always merge new WorkflowTask.args with defaults from Task.args_schema, in update_workflowtask endpoint (#759).
  • Remove WorkflowTask.overridden_meta property and on-the-fly overriding of meta (#752).
  • Add warning when exporting workflows which include custom tasks (#728).
  • When importing a workflow, only use tasks' source values, instead of (source,name) pairs (#719).
• Job execution:
  • Add FractalSlurmExecutor.shutdown and corresponding endpoint (#631, #691, #696).
  • In FractalSlurmExecutor, make working_dir* attributes required (#679).
  • Remove ApplyWorkflow.overwrite_input column (#684, #694).
  • Make output_dataset_id a required argument of apply-workflow endpoint (#681).
  • Improve error message related to out-of-space disk (#699).
  • Include timestamp in job working directory, to avoid name clashes (#756).
• Other updates to endpoints and database:
  • Add ApplyWorkflow.end_timestamp column (#687, #684).
  • Prevent deletion of a Workflow/Dataset in relationship with existing ApplyWorkflow (#703).
  • Add project-name uniqueness constraint in project-edit endpoint (#689).
• Other updates to internal logic:
  • Drop WorkflowTask.arguments property and WorkflowTask.assemble_args method (#742).
  • Add test for collection of tasks packages with tasks in a subpackage (#743).
  • Expose FRACTAL_CORS_ALLOW_ORIGIN environment variable (#688).
  • Expose FRACTAL_DEFAULT_ADMIN_USERNAME environment variable (#751).
• Package and repository:
  • Remove fastapi-users-db-sqlmodel dependency (#660).
  • Make coverage measure more accurate (#676) and improve coverage (#678).
  • Require pydantic version to be >=1.10.8 (#711, #713).
  • Include multiple fractal-common updates (#705, #719).
  • Add test equivalent to alembic check (#722).
  • Update poetry.lock to address security alerts (#723).
  • Remove sqlmodel from fractal-common, and declare database models with multiple inheritance (#710).
  • Make email generation more robust in MockCurrentUser (#730).
  • Update poetry.lock to cryptography=41, to address security alert (#739).
  • Add greenlet as a direct dependency (#748).
  • Removed tests for IntegrityError (#754).

1.2.5

• Fix bug in task collection when using sqlite (#664, #673).
• Fix bug in task collection from local package, where package extras were not considered (#671).
• Improve error handling in workflow-apply endpoint (#665).
• Fix a bug upon project removal in the presence of project-related jobs (#666). Note: this removes the ApplyWorkflow.Project attribute.

1.2.4

• Review setup for database URLs, especially to allow using UNIX-socket connections for postgresql (#657).

1.2.3

• Fix bug that was keeping multiple database conection open (#649).

1.2.2

• Fix bug related to user_local_exports in SLURM-backend configuration (#642).

1.2.1

• Fix bug upon creation of first user when using multiple workers (#632).
• Allow both ports 5173 and 4173 as CORS origins (#637).

1.2.0

• Drop project.project_dir and replace it with user.cache_dir (#601).
• Update SLURM backend (#582, #612, #614); this includes (1) combining several tasks in a single SLURM job, and (2) offering more granular sources for SLURM configuration options.
• Expose local user exports in SLURM configuration file (#625).
• Make local backend rely on custom FractalThreadPoolExecutor, where parallel_tasks_per_job can affect parallelism (#626).
• Review logging configuration (#619, #623).
• Update to fastapi 0.95 (#587).
• Minor improvements in dataset-edit endpoint (#593) and tests (#589).
• Include test of non-python task (#594).
• Move dummy tasks from package to tests (#601).
• Remove deprecated parsl backend (#607).
• Improve error handling in workflow-import endpoint (#595).
• Also show logs for successful workflow execution (#635).

1.1.1

• Include reordered_workflowtask_ids in workflow-edit endpoint payload, to reorder the task list of a workflow (#585).

1.1.0

• Align with new tasks interface in fractal-tasks-core>=0.8.0, and remove glob_pattern column from resource database table (#544).
• Drop python 3.8 support (#527).
• Improve validation of API request payloads (#545).
• Improve request validation in project-creation endpoint (#537).
• Update the endpoint to patch a Task (#526).
• Add new project-update endpoint, and relax constraints on project_dir in new-project endpoint (#563).
• Update DatasetUpdate schema (#558 and #565).
• Fix redundant task-error logs in slurm backend (#552).
• Improve handling of task-collection errors (#559).
• If FRACTAL_BACKEND_RUNNER=slurm, include some configuration checks at server startup (#529).
• Fail if FRACTAL_SLURM_WORKER_PYTHON has different versions of fractal-server or cloudpickle (#533).

1.0.8

• Fix handling of parallel-tasks errors in FractalSlurmExecutor (#497).
• Add test for custom tasks (#500).
• Improve formatting of job logs (#503).
• Improve error handling in workflow-execution server endpoint (#515).
• Update _TaskBase schema from fractal-common (#517).

1.0.7

• Update endpoints to import/export a workflow (#495).

1.0.6

• Add new endpoints to import/export a workflow (#490).

1.0.5

• Separate workflow-execution folder into two (server- and user-owned) folders, to avoid permission issues (#475).
• Explicitly pin sqlalchemy to v1 (#480).

1.0.4

• Add new POST endpoint to create new Task (#486).

1.0.3

Missing due to releasing error.

1.0.2

• Add FRACTAL_RUNNER_MAX_TASKS_PER_WORKFLOW configuration variable (#469).

1.0.1

• Fix bug with environment variable names (#468).

1.0.0

• First release listed in CHANGELOG.