Note: Numbers like (#1234) point to closed Pull Requests on the fractal-server repository.
2.9.0 (unreleased)¶
WARNING 1: This version drops support for sqlite, and removes the configuration variables
DB_ENGINE
andSQLITE_PATH
.WARNING 2: This version removes the
CollectionStateV2
database table. Make sure you have a database dump before runningfractalctl set-db
, since this operation cannot be undone.
- API
- Remove
GET /api/v2/task/collect/{state_id}/
endpoint (#2010). - Remove
active
property fromPATCH /api/v2/task-group/{task_group_id}/
(#2033). - Add
GET /api/v2/task-group/activity/
endpoint (#2005, #2027). - Add
GET /api/v2/task-group/activity/{task_group_activity_id}/
endpoint (#2005). - Add
GET /admin/v2/task-group/activity/
endpoint (#2005, #2027). - Add
POST /api/v2/task-group/{task_group_id}/{deactivate|reactivate}
endpoints (#2033, #2066, #2078). - Add
POST /admin/v2/task-group/{task_group_id}/{deactivate|reactivate}
endpoints (#2062, #2078). - Internals:
- Fix bug in
_get_collection_task_group_activity_status_message
(#2047). - Remove
valutc
validator for timestamps from API schemas, since it does not match withpsycopg3
behavior (#2064). - Add query parameters
timestamp_last_used_{min|max}
toGET /admin/v2/task-group/
(#2061). - Remove
_convert_to_db_timestamp
and add_raise_if_naive_datetime
: now API only accepts timezone-aware datetimes as query parameters (#2068). - Remove
_encode_as_utc
: now timestamps are serialized in JSONs with their own timezone (#2081).
- Remove
- Database
- Drop support for sqlite, and remove the
DB_ENGINE
andSQLITE_PATH
configuration variables (#2052). - Add
TaskGroupActivityV2
table (#2005). - Drop
CollectionStateV2
table (#2010). - Add
TaskGroupV2.pip_freeze
nullable column (#2017). - Add
venv_size_in_kB
andvenv_file_number
toTaskGroupV2
(#2034). - Add
TaskGroupV2.timestamp_last_used
column, updated on job submission (#2049, #2061, #2086).
- Drop support for sqlite, and remove the
- Task-lifecycle internals:
- Refactor task collection and database-session management in background tasks (#2030).
- Update
TaskGroupActivityV2
objects (#2005). - Update filename and path for task-collection scripts (#2008).
- Copy wheel file into
task_group.path
and updatetask_group.wheel_path
, for local task collection (#2020). - Set
TaskGroupActivityV2.timestamp_ended
when collections terminate (#2026). - Refactor bash templates and add
install_from_freeze.sh
(#2029). - Introduce background operations for local reactivate/deactivate (#2033).
- Introduce background operations for SSH reactivate/deactivate (#2066).
- Fix escaping of newlines within f-strings, in logs (#2028).
- Improve handling of task groups created before 2.9.0 (#2050).
- Add
TaskGroupCreateV2Strict
for task collections (#2080). - Always create
script_dir_remote
in SSH lifecycle background tasks (#2089).
- Runner:
- Improve error handling in
_zip_folder_to_file_and_remove
(#2057). - Improve error handling in
FractalSlurmSSHExecutor
handshake
method (#2083). - Use the "spawn" start method for the multiprocessing context, for the
ProcessPoolExecutor
-based runner (#2084).
- Improve error handling in
- SSH internals:
- Add
FractalSSH.remote_exists
method (#2008). - Drop
FractalSSH.{_get,_put}
wrappers ofSFTPClient
methods (#2077). - Try re-opening the connection in
FractalSSH.check_connection
when an error occurs (#2035). - Move
NoValidConnectionError
exception handling intoFractalSSH.log_and_raise
method (#2070). - Improve closed-socket testing (#2076).
- Add
- Testing:
- Drop
fetch-depth
fromcheckout
in GitHub actions (#2039).
- Drop
- Scripts:
- Introduce
scripts/export_v1_workflows.py
(#2043).
- Introduce
2.8.1¶
- API:
- Validate all user-provided strings that end up in pip-install commands (#2003).
2.8.0¶
- Task collection
- Now both the local and SSH versions of the task collection use the bash templates (#1980).
- Update task-collections database logs incrementally (#1980).
- Add
TaskGroupV2.pinned_package_versions_string
property (#1980). - Support pinned-package versions for SSH task collection (#1980).
- Now
pip install
uses--no-cache
(#1980).
- API
- Deprecate the
verbose
query parameter inGET /api/v2/task/collect/{state_id}/
(#1980). - Add
project_dir
attribute toUserSettings
(#1990). - Set a default for
DatasetV2.zarr_dir
(#1990). - Combine the
args_schema_parallel
andargs_schema_non_parallel
query parameters inGET /api/v2/task/
into a single parameterargs_schema
(#1998).
- Deprecate the
2.7.1¶
WARNING: As of this version, all extras for
pip install
are deprecated and the corresponding dependencies become required.
- Database:
- Drop
TaskV2.owner
column (#1977). - Make
TaskV2.taskgroupv2_id
column required (#1977).
- Drop
- Dependencies:
- Make
psycopg[binary]
dependency required, and droppostgres-pyscopg-binary
extra (#1970). - Make
gunicorn
dependency required, and dropgunicorn
extra (#1970).
- Make
- Testing:
- Switch from SQLite to Postgres in the OAuth Github action (#1981).
2.7.0¶
WARNING: This release comes with several specific notes:
- It requires running
fractalctl update-db-data
(afterfractalctl set-db
).- When running
fractalctl update-db-data
, the environment variableFRACTAL_V27_DEFAULT_USER_EMAIL
must be set, e.g. as inFRACTAL_V27_DEFAULT_USER_EMAIL=admin@fractal.yx fractalctl update-db-data
. This user must exist, and they will own all previously-common tasks/task-groups.- The pip extra
postgres
is deprecated, in favor ofpostgres-psycopg-binary
.- The configuration variable
DB_ENGINE="postgres"
is deprecated, in favor ofDB_ENGINE="postgres-psycopg"
.- Python3.9 is deprecated.
- API:
- Users and user groups:
- Replace
UserRead.group_names
andUserRead.group_ids
withUserRead.group_ids_names
ordered list (#1844, #1850). - Deprecate
GET /auth/group-names/
(#1844). - Add
DELETE /auth/group/{id}/
endpoint (#1885). - Add
PATCH auth/group/{group_id}/user-settings/
bulk endpoint (#1936).
- Replace
- Task groups:
- Introduce
/api/v2/task-group/
routes (#1817, #1847, #1852, #1856, #1943). - Respond with 422 error when any task-creating endpoint would break a non-duplication constraint (#1861).
- Enforce non-duplication constraints on
TaskGroupV2
(#1865). - Fix non-duplication check in
PATCH /api/v2/task-group/{id}/
(#1911). - Add cascade operations to
DELETE /api/v2/task-group/{task_group_id}/
and toDELETE /admin/v2/task-group/{task_group_id}/
(#1867). - Expand use and validators for
TaskGroupCreateV2
schema (#1861). - Do not process task
source
s in task/task-group CRUD operations (#1861). - Do not process task
owner
s in task/task-group CRUD operations (#1861).
- Introduce
- Tasks:
- Drop
TaskCreateV2.source
(#1909). - Drop
TaskUpdateV2.version
(#1905). - Revamp access-control for
/api/v2/task/
endpoints, based on task-group attributes (#1817). - Update
/api/v2/task/
endpoints and schemas with new task attributes (#1856). - Forbid changing
TaskV2.name
(#1925).
- Drop
- Task collection:
- Improve preliminary checks in task-collection endpoints (#1861).
- Refactor split between task-collection endpoints and background tasks (#1861).
- Create
TaskGroupV2
object within task-collection endpoints (#1861). - Fix response of task-collection endpoint (#1902).
- Automatically discover PyPI package version if missing or invalid (#1858, #1861, #1902).
- Use appropriate log-file path in collection-status endpoint (#1902).
- Add task
authors
to manifest schema (#1856). - Do not use
source
for custom task collection (#1893). - Rename custom-task-collection request-body field from
source
tolabel
(#1896). - Improve error messages from task collection (#1913).
- Forbid non-unique task names in
ManifestV2
(#1925).
- Workflows and workflow tasks:
- Introduce additional checks in POST-workflowtask endpoint, concerning non-active or non-accessible tasks (#1817).
- Introduce additional intormation in GET-workflow endpoint, concerning non-active or non-accessible tasks (#1817).
- Introduce additional intormation in PATCH-workflow endpoint, concerning non-active or non-accessible tasks (#1868, #1869).
- Stop logging warnings for non-common tasks in workflow export (#1893).
- Drop
WorkflowTaskCreateV2.order
(#1906). - Update endpoints for workflow import/export (#1925, #1939, #1960).
- Datasets:
- Remove
TaskDumpV2.owner
attribute (#1909).
- Remove
- Jobs:
- Prevent job submission if includes non-active or non-accessible tasks (#1817).
- Remove rate limit for
POST /project/{project_id}/job/submit/
(#1944).
- Admin:
- Remove
owner
fromGET admin/v2/task/
(#1909). - Deprecate
kind
query parameter for/admin/v2/task/
(#1893). - Add
origin
andpkg_name
query parameters toGET /admin/v2/task-group/
(#1979).
- Remove
- Schemas:
- Forbid extras in
TaskCollectPipV2
(#1891). - Forbid extras in all Create/Update/Import schemas (#1895).
- Deprecate internal
TaskCollectPip
schema in favor ofTaskGroupV2
(#1861).
- Forbid extras in
- Users and user groups:
- Database:
- Introduce
TaskGroupV2
table (#1817, #1856). - Add
timestamp_created
column toLinkUserGroup
table (#1850). - Add
TaskV2
attributesauthors
,tags
,category
andmodality
(#1856). - Add
update-db-data
script (#1820, #1888). - Add
taskgroupv2_id
foreign key toCollectionStateV2
(#1867). - Make
TaskV2.source
nullable and drop its uniqueness constraint (#1861). - Add
TaskGroupV2
columnswheel_path
,pinned_package_versions
(#1861). - Clean up
alembic
migration scripts (#1894). - Verify task-group non-duplication constraint in
2.7.0
data-migration script (#1927). - Normalize
pkg_name
in2.7.0
data-migration script (#1930). - Deprecate
DB_ENGINE="postgres"
configuration variable (#1946).
- Introduce
- Runner:
- Do not create local folders with 755 permissions unless
FRACTAL_BACKEND_RUNNER="slurm"
(#1923). - Fix bug of SSH/SFTP commands not acquiring lock (#1949).
- Fix bug of unhandled exception in SSH/SLURM executor (#1963).
- Always remove task-subfolder compressed archive (#1949).
- Do not create local folders with 755 permissions unless
- Task collection:
- Create base directory (in SSH mode), if missing (#1949).
- Fix bug of SSH/SFTP commands not acquiring lock (#1949).
- SSH:
- Improve logging for SSH-connection-locking flow (#1949).
- Introduce
FractalSSH.fetch_file
andFractalSSH.read_remote_json_file
(#1949). - Use
paramiko.sftp_client.SFTPClient
methods directly rathen thanfabric
wrappers (#1949). - Disable prefetching for
SFTPClient.get
(#1949).
- Internal:
- Update
_create_first_group
so that it only searches forUserGroups
with a given name (#1964).
- Update
- Dependencies:
- Bump fastapi to
0.115
(#1942). - Remove pip extra
postgres
, corresponding topsycopg2+asyncpg
(#1946). - Deprecate python3.9 (#1946).
- Bump fastapi to
- Testing:
- Benchmark
GET /api/v2/task-group/
(#1922). - Use new
ubuntu22-slurm-multipy
image, with Python3.12 and with Python-version specific venvs (#1946, #1969). - Get
DB_ENGINE
variable fromos.environ
rather than from installed packages (#1968).
- Benchmark
2.6.4¶
- Database
- Fix use of naming convention for database schema-migration scripts (#1819).
- Testing:
- Test
alembic downgrade base
(#1819). - Add
GET /api/v2/task/
to benchmarks (#1825).
- Test
2.6.3¶
- API:
- Introduce
GET /auth/current-user/viewer-paths/
endpoint (#1816). - Add
viewer_paths
attribute toUserGroup
endpoints (#1816).
- Introduce
- Database:
- Add
viewer_paths
column toUserGroup
table (#1816).
- Add
- Runner:
- Anticipate
wait_thread.shutdown_callback
assignment inFractalSlurmExecutor
, to avoid an uncaught exception (#1815).
- Anticipate
2.6.2¶
- Allow setting
UserSettings
attributes toNone
in standard/strict PATCH endpoints (#1814).
2.6.1¶
- App (internal):
- Remove
FRACTAL_SLURM_SSH_HOST
,FRACTAL_SLURM_SSH_USER
,FRACTAL_SLURM_SSH_PRIVATE_KEY_PATH
andFRACTAL_SLURM_SSH_WORKING_BASE_DIR
fromSettings
(#1804).
- Remove
- Database:
- Drop
slurm_user
,slurm_accounts
andcache_dir
columns fromUserOAuth
(#1804)
- Drop
2.6.0¶
WARNING: This release requires running
fractalctl update-db-data
(afterfractalctl set-db
).
- API:
- Introduce user-settings API, in
/auth/users/{user_id}/settings/
and/auth/current-user/settings/
(#1778, #1807). - Add the creation of empty settings to
UserManager.on_after_register
hook (#1778). - Remove deprecated user's attributes (
slurm_user
,cache_dir
,slurm_accounts
) from API, in favor of newUserSetting
ones (#1778). - Validate user settings in endpoints that rely on them (#1778).
- Propagate user settings to background tasks when needed (#1778).
- Introduce user-settings API, in
- Database:
- Introduce new
user_settings
table, and link it touser_oauth
(#1778).
- Introduce new
- Internal:
- Remove redundant string validation in
FractalSSH.remove_folder
andTaskCollectCustomV2
(#1810). - Make
validate_cmd
more strict about non-string arguments (#1810).
2.5.2¶
- App:
- Replace
fractal_ssh
attribute withfractal_ssh_list
, inapp.state
(#1790). - Move creation of SSH connections from app startup to endpoints (#1790).
- Replace
- Internal
- Introduce
FractalSSHList
, in view of support for multiple SSH/Slurm service users (#1790). - Make
FractalSSH.close()
more aggressively closeTransport
attribute (#1790). - Set
look_for_keys=False
for paramiko/fabric connection (#1790).
- Introduce
- Testing:
- Add fixture to always test that threads do not accumulate during tests (#1790).
2.5.1¶
- API:
- Make
WorkflowTaskDumpV2
attributestask_id
andtask
optional (#1784). - Add validation for user-provided strings that execute commands with subprocess or remote-shell (#1767).
- Make
- Runner and task collection:
- Validate commands before running them via
subprocess
orfabric
(#1767).
- Validate commands before running them via
2.5.0¶
WARNING: This release has a minor API bug when displaying a V2 dataset with a history that contains legacy tasks. It's recommended to update to 2.5.1.
This release removes support for including V1 tasks in V2 workflows. This comes with changes to the database (data and metadata), to the API, and to the V2 runner.
- Runner:
- Deprecate running v1 tasks within v2 workflows (#1721).
- Database:
- Remove
Task.is_v2_compatible
column (#1721). - For table
WorkflowTaskV2
, dropis_legacy_task
andtask_legacy_id
columns, removetask_legacy
ORM attribute, maketask_id
required, maketask
required (#1721).
- Remove
- API:
- Drop v1-v2-task-compatibility admin endpoint (#1721).
- Drop
/task-legacy/
endpoint (#1721). - Remove legacy task code branches from
WorkflowTaskV2
CRUD endpoints (#1721). - Add OAuth accounts info to
UserRead
at.oauth_accounts
(#1765).
- Testing:
- Improve OAuth Github Action to test OAuth account flow (#1765).
2.4.2¶
- App:
- Improve logging in
fractalctl set-db
(#1764).
- Improve logging in
- Runner:
- Add
--set-home
tosudo -u
impersonation command, to fix Ubuntu18 behavior (#1762).
- Add
- Testing:
- Start tests of migrations from valid v2.4.0 database (#1764).
2.4.1¶
This is mainly a bugfix release, re-implementing a check that was removed in 2.4.0.
- API:
- Re-introduce check for existing-user-email in
PATCH /auth/users/{id}/
(#1760).
- Re-introduce check for existing-user-email in
2.4.0¶
This release introduces support for user groups, but without linking it to any access-control rules (which will be introduced later).
NOTE: This release requires running the
fractalctl update-db-data
script.
- App:
- Move creation of first user from application startup into
fractalctl set-db
command (#1738, #1748). - Add creation of default user group into
fractalctl set-db
command (#1738). - Create
update-db-script
for current version, that adds all users to default group (#1738).
- Move creation of first user from application startup into
- API:
- Added
/auth/group/
and/auth/group-names/
routers (#1738, #1752). - Implement
/auth/users/{id}/
POST/PATCH routes infractal-server
(#1738, #1747, #1752). - Introduce
UserUpdateWithNewGroupIds
schema forPATCH /auth/users/{id}/
(#1747, #1752). - Add
UserManager.on_after_register
hook to add new users to default user group (#1738).
- Added
- Database:
- Added new
usergroup
andlinkusergroup
tables (#1738).
- Added new
- Internal
- Refactored
fractal_server.app.auth
andfractal_server.app.security
(#1738)/ - Export all relevant modules in
app.models
, since it matters e.g. forautogenerate
-ing migration scripts (#1738).
- Refactored
- Testing
- Add
UserGroup
validation toscripts/validate_db_data_with_read_schemas.py
(#1746).
- Add
2.3.11¶
- SSH runner:
- Move remote-folder creation from
submit_workflow
to more specific_process_workflow
(#1728).
- Move remote-folder creation from
- Benchmarks:
- Add
GET /auth/token/login/
to tested endpoints (#1720).
- Add
- Testing:
- Update GitHub actions
upload-artifact
anddownload-artifact
tov4
(#1725).
- Update GitHub actions
2.3.10¶
- Fix minor bug in zipping-job logging (#1716).
2.3.9¶
- Add logging for zipping-job-folder operations (#1714).
2.3.8¶
NOTE:
FRACTAL_API_V1_MODE="include_without_submission"
is now transformed intoFRACTAL_API_V1_MODE="include_read_only"
.
- API:
- Support read-only mode for V1 (#1701).
- Improve handling of zipped job-folder in download-logs endpoints (#1702).
- Runner:
- Improve database-error handling in V2 job execution (#1702).
- Zip job folder after job execution (#1702).
- App:
UvicornWorker
is now imported fromuvicorn-worker
(#1690).
- Testing:
- Remove
HAS_LOCAL_SBATCH
variable and related if-branches (#1699).
- Remove
- Benchmarks:
- Add
GET /auth/current-user/
to tested endpoints (#1700).
- Add
- Dependencies:
- Update
mkdocstrings
to^0.25.2
(#1707). - Update
fastapi
to^0.112.0
(#1705).
- Update
2.3.7¶
- SSH SLURM executor:
- Handle early shutdown in SSH executor (#1696).
- Task collection:
- Introduce a new configuration variable
FRACTAL_MAX_PIP_VERSION
to pin task-collection pip (#1675).
- Introduce a new configuration variable
2.3.6¶
- API:
- When creating a WorkflowTask, do not pre-populate its top-level arguments based on JSON Schema default values (#1688).
- Dependencies:
- Update
sqlmodel
to^0.0.21
(#1674). - Add
uvicorn-worker
(#1690).
- Update
2.3.5¶
WARNING: The
pre_submission_commands
SLURM configuration is included as an experimental feature, since it is still not useful for its main intended goal (callingmodule load
before runningsbatch
).
- SLURM runners:
- Expose
gpus
SLURM parameter (#1678). - For SSH executor, add
pre_submission_commands
(#1678). - Removed obsolete arguments from
get_slurm_config
function (#1678).
- Expose
- SSH features:
- Add
FractalSSH.write_remote_file
method (#1678).
- Add
2.3.4¶
- SSH SLURM runner:
- Refactor
compress_folder
andextract_archive
modules, and stop usingtarfile
library (#1641).
- Refactor
- API:
- Introduce
FRACTAL_API_V1_MODE=include_without_submission
to include V1 API but forbid job submission (#1664).
- Introduce
- Testing:
- Do not test V1 API with
DB_ENGINE="postgres-psycopg"
(#1667). - Use new Fractal SLURM containers in CI (#1663).
- Adapt tests so that they always refer to the current Python version (the one running
pytest
), when needed; this means that we don't require the presence of any additional Python version in the development environment, apart from the current one (#1633). - Include Python3.11 in some tests (#1669).
- Simplify CI SLURM Dockerfile after base-image updates (#1670).
- Cache
ubuntu22-slurm-multipy
Docker image in CI (#1671). - Add
oauth.yaml
GitHub action to test OIDC authentication (#1665).
- Do not test V1 API with
2.3.3¶
This release fixes a SSH-task-collection bug introduced in version 2.3.1.
- API:
- Expose new superuser-restricted endpoint
GET /api/settings/
(#1662).
- Expose new superuser-restricted endpoint
- SLURM runner:
- Make
FRACTAL_SLURM_SBATCH_SLEEP
configuration variablefloat
(#1658).
- Make
- SSH features:
- Fix wrong removal of task-package folder upon task-collection failure (#1649).
- Remove
FractalSSH.rename_folder
method (#1654).
- Testing:
- Refactor task-collection fixtures (#1637).
2.3.2¶
WARNING: The remove-remote-venv-folder in the SSH task collection is broken (see issue 1633). Do not deploy this version in an SSH-based
fractal-server
instance.
- API:
- Fix incorrect zipping of structured job-log folders (#1648).
2.3.1¶
This release includes a bugfix for task names with special characters.
WARNING: The remove-remote-venv-folder in the SSH task collection is broken (see issue 1633). Do not deploy this version in an SSH-based
fractal-server
instance.
- Runner:
- Improve sanitization of subfolder names (commits from 3d89d6ba104d1c6f11812bc9de5cbdff25f81aa2 to 426fa3522cf2eef90d8bd2da3b2b8a5b646b9bf4).
- API:
- Improve error message when task-collection Python is not defined (#1640).
- Use a single endpoint for standard and SSH task collection (#1640).
- SSH features:
- Remove remote venv folder upon failed task collection in SSH mode (#1634, #1640).
- Refactor
FractalSSH
(#1635). - Set
fabric.Connection.forward_agent=False
(#1639).
- Testing:
- Improved testing of SSH task-collection API (#1640).
- Improved testing of
FractalSSH
methods (#1635). - Stop testing SQLite database for V1 in CI (#1630).
2.3.0¶
This release includes two important updates: 1. An Update update to task-collection configuration variables and logic. 2. The first released version of the experimental SSH features.
Re: task-collection configuration, we now support two main use cases:
-
When running a production instance (including on a SLURM cluster), you should set e.g.
FRACTAL_TASKS_PYTHON_DEFAULT_VERSION=3.10
, and make sure thatFRACTAL_TASKS_PYTHON_3_10=/some/python
is an absolute path. Optionally, you can define other variables likeFRACTAL_TASKS_PYTHON_3_9
,FRACTAL_TASKS_PYTHON_3_11
orFRACTAL_TASKS_PYTHON_3_12
. -
If you leave
FRACTAL_TASKS_PYTHON_DEFAULT_VERSION
unset, then only the Python interpreter that is currently runningfractal-server
can be used for task collection.
WARNING: If you don't set
FRACTAL_TASKS_PYTHON_DEFAULT_VERSION
, then you will only have a single Python interpreter available for tasks (namely the one runningfractal-server
).
- API:
- Introduce
api/v2/task/collect/custom/
endpoint (#1607, #1613, #1617, #1629).
- Introduce
- Task collection:
- Introduce task-collection Python-related configuration variables (#1587).
- Always set Python version for task collection, and only use
FRACTAL_TASKS_PYTHON_X_Y
variables (#1587). - Refactor task-collection functions and schemas (#1587, #1617).
- Remove
TaskCollectStatusV2
andget_collection_data
internal schema/function (#1598). - Introduce
CollectionStatusV2
enum for task-collection status (#1598). - Reject task-collection request if it includes a wheel file and a version (#1608). SSH features:
- Introduce
fractal_server/ssh
subpackage (#1545, #1599, #1611). - Introduce SSH executor and runner (#1545).
- Introduce SSH task collection (#1545, #1599, #1626).
- Introduce SSH-related configuration variables (#1545).
- Modify app lifespan to handle SSH connection (#1545).
- Split
app/runner/executor/slurm
intosudo
andssh
subfolders (#1545). - Introduce FractalSSH object which is a wrapper class around fabric.Connection object.
It provides a
lock
to avoid loss of ssh instructions and a custom timeout (#1618)
- Dependencies:
- Update
sqlmodel
to^0.0.19
(#1584). - Update
pytest-asyncio
to^0.23
(#1558).
- Update
- Testing:
- Test the way
FractalProcessPoolExecutor
spawns processes and threads (#1579). - Remove
event_loop
fixture: every test will run on its own event loop (#1558). - Test task collection with non-canonical package name (#1602).
- Test the way
2.2.0¶
This release streamlines options for the Gunicorn startup command, and includes two new experimental features.
NOTE 1: you can now enable custom Gunicorn worker/logger by adding the following options to the
gunicorn
startup command: ---worker-class fractal_server.gunicorn_fractal.FractalWorker
---logger-class fractal_server.gunicorn_fractal.FractalGunicornLogger
NOTE 2: A new experimental local runner is available, which uses processes instead of threads and support shutdown. You can try it out with the configuration variable
FRACTAL_BACKEND_RUNNER=local_experimental
NOTE 3: A new PostgreSQL database adapter is available, fully based on
psycopg3
(rather thanpyscopg2
+asyncpg
). You can try it out with the configuration variableDB_ENGINE=postgres-psycopg
(note that this requires thepip install
extrapostgres-psycopg-binary
).
- API:
- Add extensive logs to
DELETE /api/v2/project/{project_id}
(#1532). - Remove catch of
IntegrityError
inPOST /api/v1/project
(#1530).
- Add extensive logs to
- App and deployment:
- Move
FractalGunicornLogger
andFractalWorker
intofractal_server/gunicorn_fractal.py
(#1535). - Add custom gunicorn/uvicorn worker to handle SIGABRT signal (#1526).
- Store list of submitted jobs in app state (#1538).
- Add logic for graceful shutdown for job slurm executors (#1547).
- Move
- Runner:
- Change structure of job folders, introducing per-task subfolders (#1523).
- Rename internal
workflow_dir
andworkflow_dir_user
variables to local/remote (#1534). - Improve handling of errors in
submit_workflow
background task (#1556, #1566). - Add new
local_experimental
runner, based onProcessPoolExecutor
(#1544, #1566).
- Database:
- Add new Postgres adapter
psycopg
(#1562).
- Add new Postgres adapter
- Dependencies
- Add
fabric
todev
dependencies (#1518). - Add new
postgres-psycopg-binary
extra (#1562).
- Add
- Testing:
- Extract
pytest-docker
fixtures into a dedicated module (#1516). - Rename SLURM containers in CI (#1516).
- Install and run SSH daemon in CI containers (#1518).
- Add unit test of SSH connection via fabric/paramiko (#1518).
- Remove obsolete folders from
tests/data
(#1517).
- Extract
2.1.0¶
This release fixes a severe bug where SLURM-executor auxiliary threads are not joined when a Fractal job ends.
- App:
- Add missing join for
wait_thread
uponFractalSlurmExecutor
exit (#1511). - Replace
startup
/shutdown
events withlifespan
event (#1501).
- Add missing join for
- API:
- Remove
Path.resolve
from the submit-job endpoints and add validator forSettings.FRACTAL_RUNNER_WORKING_BASE_DIR
(#1497).
- Remove
- Testing:
- Improve dockerfiles for SLURM (#1495, #1496).
- Set short timeout for
docker compose down
(#1500).
2.0.6¶
NOTE: This version changes log formats. For
uvicorn
logs, this change requires no action. Forgunicorn
, logs formats are only changed by adding the following command-line option:gunicorn ... --logger-class fractal_server.logger.gunicorn_logger.FractalGunicornLogger
.
- API:
- Add
FRACTAL_API_V1_MODE
environment variable to include/exclude V1 API (#1480). - Change format of uvicorn loggers (#1491).
- Introduce
FractalGunicornLogger
class (#1491).
- Add
- Runner:
- Fix missing
.log
files in server folder for SLURM jobs (#1479).
- Fix missing
- Database:
- Remove
UserOAuth.project_list
andUserOAuth.project_list_v2
relationships (#1482).
- Remove
- Dev dependencies:
- Bump
pytest
to8.1.*
(#1486). - Bump
coverage
to7.5.*
(#1486). - Bump
pytest-docker
to3.1.*
(#1486). - Bump
pytest-subprocess
to^1.5
(#1486).
- Bump
- Benchmarks:
- Move
populate_db
scripts intobenchmark
folder (#1489).
- Move
2.0.5¶
- API:
- Add
GET /admin/v2/task/
(#1465). - Improve error message in DELETE-task endpoint (#1471).
- Add
- Set
JobV2
folder attributes from within the submit-job endpoint (#1464). - Tests:
- Make SLURM CI work on MacOS (#1476).
2.0.4¶
- Add
FRACTAL_SLURM_SBATCH_SLEEP
configuration variable (#1467).
2.0.3¶
WARNING: This update requires running a fix-db script, via
fractalctl update-db-data
.
- Database:
- Create fix-db script to remove
images
andhistory
from dataset dumps in V1/V2 jobs (#1456).
- Create fix-db script to remove
- Tests:
- Split
test_full_workflow_v2.py
into local/slurm files (#1454).
- Split
2.0.2¶
WARNING: Running this version on a pre-existing database (where the
jobsv2
table has some entries) is broken. Running this version on a freshly-created database works as expected.
- API:
- Fix bug in status endpoint (#1449).
- Improve handling of out-of-scope scenario in status endpoint (#1449).
- Do not include dataset
history
inJobV2.dataset_dump
(#1445). - Forbid extra arguments in
DumpV2
schemas (#1445).
- API V1:
- Do not include dataset
history
inApplyWorkflow.{input,output}_dataset_dump
(#1453).
- Do not include dataset
- Move settings logs to
check_settings
and use fractal-serverset_logger
(#1452). - Benchmarks:
- Handle some more errors in benchmark flow (#1445).
- Tests:
- Update testing database to version 2.0.1 (#1445).
2.0.1¶
- Database/API:
- Do not include
dataset_dump.images
inJobV2
table (#1441).
- Do not include
- Internal functions:
- Introduce more robust
reset_logger_handlers
function (#1425).
- Introduce more robust
- Benchmarks:
- Add
POST /api/v2/project/project_id/dataset/dataset_id/images/query/
in bechmarks to evaluate the impact of the number of images during the query (#1441).
- Add
- Development:
- Use
poetry
1.8.2 in GitHub actions and documentation.
- Use
2.0.0¶
Major update.
1.4.10¶
WARNING: Starting from this version, the dependencies for the
slurm
extra are required; commands likepip install fractal-server[slurm,postgres]
must be replaced bypip install fractal-server[postgres]
.
- Dependencies:
- Make
clusterfutures
andcloudpickle
required dependencies (#1255). - Remove
slurm
extra from package (#1255).
- Make
- API:
- Handle invalid history file in
GET /project/{project_id}/dataset/{dataset_id}/status/
(#1259).
- Handle invalid history file in
- Runner:
- Add custom
_jobs_finished
function to check the job status and to avoid squeue errors (#1266)
- Add custom
1.4.9¶
This release is a follow-up of 1.4.7 and 1.4.8, to mitigate the risk of job folders becoming very large.
- Runner:
- Exclude
history
fromTaskParameters
object for parallel tasks, so that it does not end up in input pickle files (#1247).
- Exclude
1.4.8¶
This release is a follow-up of 1.4.7, to mitigate the risk of job folders becoming very large.
- Runner:
- Exclude
metadata["image"]
fromTaskParameters
object for parallel tasks, so that it does not end up in input pickle files (#1245). - Exclude components list from
workflow.log
logs (#1245).
- Exclude
- Database:
- Remove spurious logging of
fractal_server.app.db
string (#1245).
- Remove spurious logging of
1.4.7¶
This release provides a bugfix (PR 1239) and a workaround (PR 1238) for the SLURM runner, which became relevant for the use case of processing a large dataset (300 wells with 25 cycles each).
- Runner:
- Do not include
metadata["image"]
in JSON file with task arguments (#1238). - Add
FRACTAL_RUNNER_TASKS_INCLUDE_IMAGE
configuration variable, to define exceptions where tasks still requiremetadata["image"]
(#1238). - Fix bug in globbing patterns, when copying files from user-side to server-side job folder in SLURM executor (#1239).
- Do not include
- API:
- Fix error message for rate limits in apply-workflow endpoint (#1231).
- Benchmarks:
- Add more scenarios, as per issue #1184 (#1232).
1.4.6¶
- API:
- Add
GET /admin/job/{job_id}
(#1230). - Handle
FileNotFound
inGET /project/{project_id}/job/{job_id}/
(#1230).
- Add
1.4.5¶
- Remove CORS middleware (#1228).
- Testing:
- Fix
migrations.yml
GitHub action (#1225).
- Fix
1.4.4¶
- API:
- Add rate limiting to
POST /{project_id}/workflow/{workflow_id}/apply/
(#1199). - Allow users to read the logs of ongoing jobs with
GET /project/{project_id}/job/{job_id}/
, usingshow_tmp_logs
query parameter (#1216). - Add
log
query parameter inGET {/api/v1/job/,/api/v1/{project.id}/job/,/admin/job/}
, to trim response body (#1218). - Add
args_schema
query parameter inGET /api/v1/task/
to trim response body (#1218). - Add
history
query parameter inGET {/api/v1/dataset/,/api/v1/project/{project.id}/dataset/}
to trim response body (#1219). - Remove
task_list
fromjob.workflow_dump
creation in/api/v1/{project_id}/workflow/{workflow_id}/apply/
(#1219) - Remove
task_list
fromWorkflowDump
Pydantic schema (#1219)
- Add rate limiting to
- Dependencies:
- Update fastapi to
^0.109.0
(#1222). - Update gunicorn to
^21.2.0
(#1222). - Update aiosqlite to
^0.19.0
(#1222). - Update uvicorn to
^0.27.0
(#1222).
- Update fastapi to
1.4.3¶
WARNING:
This update requires running a fix-db script, via
fractalctl update-db-data
.
- API:
- Improve validation of
UserCreate.slurm_accounts
(#1162). - Add
timestamp_created
toWorkflowRead
,WorkflowDump
,DatasetRead
andDatasetDump
(#1152). - Make all dumps in
ApplyWorkflowRead
non optional (#1175). - Ensure that timestamps in
Read
schemas are timezone-aware, regardless ofDB_ENGINE
(#1186). - Add timezone-aware timestamp query parameters to all
/admin
endpoints (#1186).
- Improve validation of
- API (internal):
- Change the class method
Workflow.insert_task
into the auxiliary function_workflow_insert_task
(#1149).
- Change the class method
- Database:
- Make
WorkflowTask.workflow_id
andWorfklowTask.task_id
not nullable (#1137). - Add
Workflow.timestamp_created
andDataset.timestamp_created
columns (#1152). - Start a new
current.py
fix-db script (#1152, #1195). - Add to
migrations.yml
a new script (validate_db_data_with_read_schemas.py
) that validates test-DB data with Read schemas (#1187). - Expose
fix-db
scripts via command-line optionfractalctl update-db-data
(#1197).
- Make
- App (internal):
- Check in
Settings
thatpsycopg2
,asyngpg
andcfut
, if required, are installed (#1167). - Split
DB.set_db
into sync/async methods (#1165). - Rename
DB.get_db
intoDB.get_async_db
(#1183). - Normalize names of task packages (#1188).
- Check in
- Testing:
- Update
clean_db_fractal_1.4.1.sql
toclean_db_fractal_1.4.2.sql
, and changemigrations.yml
target version (#1152). - Reorganise the test directory into subdirectories, named according to the order in which we want the CI to execute them (#1166).
- Split the CI into two independent jobs,
Core
andRunner
, to save time through parallelisation (#1204).
- Update
- Dependencies:
- Update
python-dotenv
to version 0.21.0 (#1172).
- Update
- Runner:
- Remove
JobStatusType.RUNNING
, incorporating it intoJobStatusType.SUBMITTED
(#1179).
- Remove
- Benchmarks:
- Add
fractal_client.py
andpopulate_script_v2.py
for creating different database status scenarios (#1178). - Add a custom benchmark suite in
api_bench.py
. - Remove locust.
- Add
- Documentation:
- Add the minimum set of environment variables required to set the database and start the server (#1198).
1.4.2¶
WARNINGs:
- This update requires running a fix-db script, available at https://raw.githubusercontent.com/fractal-analytics-platform/fractal-server/1.4.2/scripts/fix_db/current.py.
- Starting from this version, non-verified users have limited access to
/api/v1/
endpoints. Before the upgrade, all existing users must be manually set to verified.
- API:
- Prevent access to
GET/PATCH
task endpoints for non-verified users (#1114). - Prevent access to task-collection and workflow-apply endpoints for non-verified users (#1099).
- Make first-admin-user verified (#1110).
- Add the automatic setting of
ApplyWorkflow.end_timestamp
when patchingApplyWorkflow.status
viaPATCH /admin/job/{job_id}
(#1121). - Change
ProjectDump.timestamp_created
type fromdatetime
tostr
(#1120). - Change
_DatasetHistoryItem.workflowtask
type intoWorkflowTaskDump
(#1139). - Change status code of stop-job endpoints to 202 (#1151).
- Prevent access to
- API (internal):
- Implement cascade operations explicitly, in
DELETE
endpoints for datasets, workflows and projects (#1130). - Update
GET /project/{project_id}/workflow/{workflow_id}/job/
to avoid usingWorkflow.job_list
(#1130). - Remove obsolete sync-database dependency from apply-workflow endpoint (#1144).
- Implement cascade operations explicitly, in
- Database:
- Add
ApplyWorkflow.project_dump
column (#1070). - Provide more meaningful names to fix-db scripts (#1107).
- Add
Project.timestamp_created
column, with timezone-aware default (#1102, #1131). - Remove
Dataset.list_jobs_input
andDataset.list_jobs_output
relationships (#1130). - Remove
Workflow.job_list
(#1130).
- Add
- Runner:
- In SLURM backend, use
slurm_account
(as received from apply-workflow endpoint) with top priority (#1145). - Forbid setting of SLURM account from
WorkflowTask.meta
or as part ofworker_init
variable (#1145). - Include more info in error message upon
sbatch
failure (#1142). - Replace
sbatch
--chdir
option with-D
, to support also slurm versions before 17.11 (#1159).
- In SLURM backend, use
- Testing:
- Extended systematic testing of database models (#1078).
- Review
MockCurrentUser
fixture, to handle different kinds of users (#1099). - Remove
persist
fromMockCurrentUser
(#1098). - Update
migrations.yml
GitHub Action to use up-to-date database and also test fix-db script (#1101). - Add more schema-based validation to fix-db current script (#1107).
- Update
.dict()
to.model_dump()
forSQLModel
objects, to fix someDeprecationWarnings
(##1133). - Small improvement in schema coverage (#1125).
- Add unit test for
security
module (#1036).
- Dependencies:
- Update
sqlmodel
to version 0.0.14 (#1124).
- Update
- Benchmarks:
- Add automatic benchmark system for API's performances (#1123)
- App (internal):
- Move
_create_first_user
frommain
tosecurity
module, and allow it to create multiple regular users (#1036).
- Move
1.4.1¶
- API:
- Add
GET /admin/job/{job_id}/stop/
andGET /admin/job/{job_id}/download/
endpoints (#1059). - Use
DatasetDump
andWorkflowDump
models for "dump" attributes ofApplyWorkflowRead
(#1049, #1082). - Add
slurm_accounts
toUser
schemas and addslurm_account
toApplyWorkflow
schemas (#1067). - Prevent providing a
package_version
for task collection from a.whl
local package (#1069). - Add
DatasetRead.project
andWorkflowRead.project
attributes (#1082).
- Add
- Database:
- Make
ApplyWorkflow.workflow_dump
column non-nullable (#1049). - Add
UserOAuth.slurm_accounts
andApplyWorkflow.slurm_account
columns (#1067). - Add script for adding
ApplyWorkflow.user_email
(#1058). - Add
Dataset.project
andWorkflow.project
relationships (#1082). - Avoid using
Project
relationshipsdataset_list
orworkflow_list
within someGET
endpoints (#1082). - Fully remove
Project
relationshipsdataset_list
,workflow_list
andjob_list
(#1091).
- Make
- Testing:
- Only use ubuntu-22.04 in GitHub actions (#1061).
- Improve unit testing of database models (#1082).
- Dependencies:
- Pin
bcrypt
to 4.0.1 to avoid warning in passlib (#1060).
- Pin
- Runner:
- Set SLURM-job working directory to
job.working_dir_user
through--chdir
option (#1064).
- Set SLURM-job working directory to
1.4.0¶
- API:
- Major endpoint changes:
- Add trailing slash to all endpoints' paths (#1003).
- Add new admin-area endpoints restricted to superusers at
/admin
(#947, #1009, #1032). - Add new
GET
endpointsapi/v1/job/
andapi/v1/project/{project_id}/workflow/{workflow_id}/job/
(#969, #1003). - Add new
GET
endpointsapi/v1/dataset/
andapi/v1/workflow/
(#988, #1003). - Add new
GET
endpointapi/v1/project/{project_id}/dataset/
(#993). - Add
PATCH /admin/job/{job_id}/
endpoint (#1030, #1053). - Move
GET /auth/whoami/
toGET /auth/current-user/
(#1013). - Move
PATCH /auth/users/me/
toPATCH /auth/current-user/
(#1013, #1035). - Remove
DELETE /auth/users/{id}/
endpoint (#994). - Remove
GET /auth/users/me/
(#1013). - Remove
POST
/auth/forgot-password/
,/auth/reset-password/
,/auth/request-verify-token/
,/auth/verify/
(#1033). - Move
GET /auth/userlist/
toGET /auth/users/
(#1033).
- New behaviors or responses of existing endpoints:
- Change response of
/api/v1/project/{project_id}/job/{job_id}/stop/
endpoint to 204 no-content (#967). - Remove
dataset_list
attribute fromProjectRead
, which affects allGET
endpoints that return some project (#993). - Make it possible to delete a
Dataset
,Workflow
orProject
, even when it is in relationship to anApplyWorkflow
- provided that theApplyWorkflow
is not pending or running (#927, #973). - Align
ApplyWorkflowRead
with newApplyWorkflow
, which has optional foreign keysproject_id
,workflow_id
,input_dataset_id
, andoutput_dataset_id
(#984). - Define types for
ApplyWorkflowRead
"dump" attributes (#990). WARNING: reverted with #999.
- Change response of
- Internal changes:
- Move all routes definitions into
fractal_server/app/routes
(#976). - Fix construction of
ApplyWorkflow.workflow_dump
, within apply endpoint (#968). - Fix construction of
ApplyWorkflow
attributesinput_dataset_dump
andoutput_dataset_dump
, within apply endpoint (#990). - Remove
asyncio.gather
, in view of SQLAlchemy2 update (#1004).
- Move all routes definitions into
- Major endpoint changes:
- Database:
- Make foreign-keys of
ApplyWorkflow
(project_id
,workflow_id
,input_dataset_id
,output_dataset_id
) optional (#927). - Add columns
input_dataset_dump
,output_dataset_dump
anduser_email
toApplyWorkflow
(#927). - Add relations
Dataset.list_jobs_input
andDataset.list_jobs_output
(#927). - Make
ApplyWorkflow.start_timestamp
non-nullable (#927). - Remove
"cascade": "all, delete-orphan"
fromProject.job_list
(#927). - Add
Workflow.job_list
relation (#927). - Do not use
Enum
s as column types (e.g. forApplyWorkflow.status
), but only for (de-)serialization (#974). - Set
pool_pre_ping
option toTrue
, for asyncpg driver (#1037). - Add script for updating DB from 1.4.0 to 1.4.1 (#1010)
- Fix missing try/except in sync session (#1020).
- Make foreign-keys of
- App:
- Skip creation of first-superuser when one superuser already exists (#1006).
- Dependencies:
- Update sqlalchemy to version
>=2.0.23,<2.1
(#1044). - Update sqlmodel to version 0.0.12 (#1044).
- Upgrade asyncpg to version 0.29.0 (#1036).
- Update sqlalchemy to version
- Runner:
- Refresh DB objects within
submit_workflow
(#927).
- Refresh DB objects within
- Testing:
- Add
await db_engine.dispose()
indb_create_tables
fixture (#1047). - Set
debug=False
inevent_loop
fixture (#1044). - Improve
test_full_workflow.py
(#971). - Update
pytest-asyncio
to v0.21 (#1008). - Fix CI issue related to event loop and asyncpg (#1012).
- Add GitHub Action testing database migrations (#1010).
- Use greenlet v3 in
poetry.lock
(#1044).
- Add
- Documentation:
- Add OAuth2 example endpoints to Web API page (#1034, #1038).
- Development:
- Use poetry 1.7.1 (#1043).
1.3.14 (do not use!)¶
WARNING: This version introduces a change that is then reverted in 1.4.0, namely it sets the
ApplyWorkflow.status
type toEnum
, when used with PostgreSQL. It is recommended to not use it, and upgrade to 1.4.0 directly.
- Make
Dataset.resource_list
anordering_list
, ordered byResource.id
(#951). - Expose
redirect_url
for OAuth clients (#953). - Expose JSON Schema for the
ManifestV1
Pydantic model (#942). - Improve delete-resource endpoint (#943).
- Dependencies:
- Upgrade sqlmodel to 0.0.11 (#949).
- Testing:
- Fix bug in local tests with Docker/SLURM (#948).
1.3.13¶
- Configure sqlite WAL to avoid "database is locked" errors (#860).
- Dependencies:
- Add
sqlalchemy[asyncio]
extra, and do not directly requiregreenlet
(#895). - Fix
cloudpickle
-version definition inpyproject.toml
(#937). - Remove obsolete
sqlalchemy_utils
dependency (#939).
- Add
- Testing:
- Use ubuntu-22 for GitHub CI (#909).
- Run GitHub CI both with SQLite and Postgres (#915).
- Disable
postgres
service in GitHub action when running tests with SQLite (#931). - Make
test_commands.py
tests stateless, also when running with Postgres (#917).
- Documentation:
- Add information about minimal supported SQLite version (#916).
1.3.12¶
- Project creation:
- Do not automatically create a dataset upon project creation (#897).
- Remove
ProjectCreate.default_dataset_name
attribute (#897).
- Dataset history:
- Create a new (non-nullable) history column in
Dataset
table (#898, #901). - Deprecate history handling in
/project/{project_id}/job/{job_id}
endpoint (#898). - Deprecate
HISTORY_LEGACY
(#898).
- Create a new (non-nullable) history column in
- Testing:
- Remove obsolete fixture
slurm_config
(#903).
- Remove obsolete fixture
1.3.11¶
This is mainly a bugfix release for the PermissionError
issue.
- Fix
PermissionError
s in parallel-task metadata aggregation for the SLURM backend (#893). - Documentation:
- Bump
mkdocs-render-swagger-plugin
to 0.1.0 (#889).
- Bump
- Testing:
- Fix
poetry install
command andpoetry
version in GitHub CI (#889).
- Fix
1.3.10¶
Warning: updating to this version requires changes to the configuration variable
- Updates to SLURM interface:
- Remove
sudo
-requiringls
calls fromFractalFileWaitThread.check
(#885); - Change default of
FRACTAL_SLURM_POLL_INTERVAL
to 5 seconds (#885); - Rename
FRACTAL_SLURM_OUTPUT_FILE_GRACE_TIME
configuration variables intoFRACTAL_SLURM_ERROR_HANDLING_INTERVAL
(#885); - Remove
FRACTAL_SLURM_KILLWAIT_INTERVAL
variable and corresponding logic (#885); - Remove
_multiple_paths_exist_as_user
helper function (#885); - Review type hints and default values of SLURM-related configuration variables (#885).
- Remove
- Dependencies:
- Update
fastapi
to version^0.103.0
(#877); - Update
fastapi-users
to version^12.1.0
(#877).
- Update
1.3.9¶
- Make updated-metadata collection robust for metadiff files consisting of a single
null
value (#879). - Automate procedure for publishing package to PyPI (#881).
1.3.8¶
- Backend runner:
- Add aggregation logic for parallel-task updated metadata (#852);
- Make updated-metadata collection robust for missing files (#852, #863).
- Database interface:
- API:
- Prevent user from bypassing workflow-name constraint via the PATCH endpoint (#867).
- Handle error upon task collection, when tasks exist in the database but not on-disk (#874).
- Add
_check_project_exists
helper function (#872).
- Configuration variables:
- Remove
DEPLOYMENT_TYPE
variable and updatealive
endpoint (#875); - Introduce
Settings.check_db
method, and call it during inline/offline migrations (#855); - Introduce
Settings.check_runner
method (#875); - Fail if
FRACTAL_BACKEND_RUNNER
is"local"
andFRACTAL_LOCAL_CONFIG_FILE
is set but missing on-disk (#875); - Clean up
Settings.check
method and improve its coverage (#875);
- Remove
- Package, repository, documentation:
- Change
fractal_server.common
from being a git-submodule to being a regular folder (#859). - Pin documentation dependencies (#865).
- Split
app/models/project.py
into two modules for dataset and project (#871). - Revamp documentation on database interface and on the corresponding configuration variables (#855).
- Change
1.3.7¶
- Oauth2-related updates (#822):
- Update configuration of OAuth2 clients, to support OIDC/GitHub/Google;
- Merge
SQLModelBaseOAuthAccount
andOAuthAccount
models; - Update
UserOAuth.oauth_accounts
relationship and fixlist_users
endpoint accordingly; - Introduce dummy
UserManager.on_after_login
method; - Rename
OAuthClient
intoOAuthClientConfig
; - Revamp users-related parts of documentation.
1.3.6¶
- Update
output_dataset.meta
also when workflow execution fails (#843). - Improve error message for unknown errors in job execution (#843).
- Fix log message incorrectly marked as "error" (#846).
1.3.5¶
- Review structure of dataset history (#803):
- Re-define structure for
history
property ofDataset.meta
; - Introduce
"api/v1/project/{project_id}/dataset/{dataset_id}/status/"
endpoint; - Introduce
"api/v1/project/{project_id}/dataset/{dataset_id}/export_history/"
endpoint; - Move legacy history to
Dataset.meta["HISTORY_LEGACY"]
.
- Re-define structure for
- Make
first_task_index
andlast_task_index
properties ofApplyWorkflow
required (#803). - Add
docs_info
anddocs_link
to Task model (#814) - Accept
TaskUpdate.version=None
in task-patch endpoint (#818). - Store a copy of the
Workflow
into the optional columnApplyWorkflow.workflow_dump
at the time of submission (#804, #834). - Prevent execution of multiple jobs with the same output dataset (#801).
- Transform non-absolute
FRACTAL_TASKS_DIR
into absolute paths, relative to the current working directory (#825). - Error handling:
- Raise an appropriate error if a task command is not executable (#800).
- Improve handling of errors raised in
get_slurm_config
(#800).
- Documentation:
- Clarify documentation about
SlurmConfig
(#798). - Update documentation configuration and GitHub actions (#811).
- Clarify documentation about
- Tests:
- Move
tests/test_common.py
intofractal-common
repository (#808). - Switch to
docker compose
v2 and unpinpyyaml
version (#816).
- Move
1.3.4¶
- Support execution of a workflow subset (#784).
- Fix internal server error for invalid
task_id
increate_workflowtask
endpoint (#782). - Improve logging in background task collection (#776).
- Handle failures in
submit_workflow
without raising errors (#787). - Simplify internal function for execution of a list of task (#780).
- Exclude
common/tests
and other git-related files from build (#795). - Remove development dependencies
Pillow
andpytest-mock
(#795). - Remove obsolete folders from
tests/data
folder (#795).
1.3.3¶
- Pin Pydantic to v1 (#779).
1.3.2¶
- Add sqlalchemy naming convention for DB constraints, and add
render_as_batch=True
todo_run_migrations
(#757). - Fix bug in job-stop endpoint, due to missing default for
FractalSlurmExecutor.wait_thread.shutdown_file
(#768, #769). - Fix bug upon inserting a task with
meta=None
into a Workflow (#772).
1.3.1¶
- Fix return value of stop-job endpoint (#764).
- Expose new GET
WorkflowTask
endpoint (#762). - Clean up API modules (#762):
- Split workflow/workflowtask modules;
- Split tasks/task-collection modules.
1.3.0¶
- Refactor user model:
- Switch from UUID4 to int for IDs (#660, #684).
- Fix many-to-many relationship between users and project (#660).
- Rename
Project.user_member_list
intoProject.user_list
(#660). - Add
username
column (#704).
- Update endpoints (see also 1.2->1.3 upgrade info in the documentation):
- Review endpoint URLs (#669).
- Remove foreign keys from payloads (#669).
- Update
Task
models, task collection and task-related endpoints:- Add
version
andowner
columns toTask
model (#704). - Set
Task.version
during task collection (#719). - Set
Task.owner
as part of create-task endpoint (#704). - For custom tasks, prepend
owner
to user-providedsource
(#725). - Remove
default_args
fromTasks
model and from manifest tasks (#707). - Add
args_schema
andargs_schema_version
toTask
model (#707). - Expose
args_schema
andargs_schema_version
in task POST/PATCH endpoints (#749). - Make
Task.source
task-specific rather than package-specific (#719). - Make
Task.source
unique (#725). - Update
_TaskCollectPip
methods, attributes and properties (#719). - Remove private/public options for task collection (#704).
- Improve error message for missing package manifest (#704).
- Improve behavior when task-collection folder already exists (#704).
- Expose
pinned_package_version
for tasks collection (#744). - Restrict Task editing to superusers and task owners (#733).
- Implement
delete_task
endpoint (#745).
- Add
- Update
Workflow
andWorkflowTask
endpoints:- Always merge new
WorkflowTask.args
with defaults fromTask.args_schema
, inupdate_workflowtask
endpoint (#759). - Remove
WorkflowTask.overridden_meta
property and on-the-fly overriding ofmeta
(#752). - Add warning when exporting workflows which include custom tasks (#728).
- When importing a workflow, only use tasks'
source
values, instead of(source,name)
pairs (#719).
- Always merge new
- Job execution:
- Add
FractalSlurmExecutor.shutdown
and corresponding endpoint (#631, #691, #696). - In
FractalSlurmExecutor
, makeworking_dir*
attributes required (#679). - Remove
ApplyWorkflow.overwrite_input
column (#684, #694). - Make
output_dataset_id
a required argument of apply-workflow endpoint (#681). - Improve error message related to out-of-space disk (#699).
- Include timestamp in job working directory, to avoid name clashes (#756).
- Add
- Other updates to endpoints and database:
- Add
ApplyWorkflow.end_timestamp
column (#687, #684). - Prevent deletion of a
Workflow
/Dataset
in relationship with existingApplyWorkflow
(#703). - Add project-name uniqueness constraint in project-edit endpoint (#689).
- Add
- Other updates to internal logic:
- Drop
WorkflowTask.arguments
property andWorkflowTask.assemble_args
method (#742). - Add test for collection of tasks packages with tasks in a subpackage (#743).
- Expose
FRACTAL_CORS_ALLOW_ORIGIN
environment variable (#688). - Expose
FRACTAL_DEFAULT_ADMIN_USERNAME
environment variable (#751).
- Drop
- Package and repository:
- Remove
fastapi-users-db-sqlmodel
dependency (#660). - Make coverage measure more accurate (#676) and improve coverage (#678).
- Require pydantic version to be
>=1.10.8
(#711, #713). - Include multiple
fractal-common
updates (#705, #719). - Add test equivalent to
alembic check
(#722). - Update
poetry.lock
to address security alerts (#723). - Remove
sqlmodel
fromfractal-common
, and declare database models with multiple inheritance (#710). - Make email generation more robust in
MockCurrentUser
(#730). - Update
poetry.lock
tocryptography=41
, to address security alert (#739). - Add
greenlet
as a direct dependency (#748). - Removed tests for
IntegrityError
(#754).
- Remove
1.2.5¶
- Fix bug in task collection when using sqlite (#664, #673).
- Fix bug in task collection from local package, where package extras were not considered (#671).
- Improve error handling in workflow-apply endpoint (#665).
- Fix a bug upon project removal in the presence of project-related jobs (#666). Note: this removes the
ApplyWorkflow.Project
attribute.
1.2.4¶
- Review setup for database URLs, especially to allow using UNIX-socket connections for postgresql (#657).
1.2.3¶
- Fix bug that was keeping multiple database conection open (#649).
1.2.2¶
- Fix bug related to
user_local_exports
in SLURM-backend configuration (#642).
1.2.1¶
- Fix bug upon creation of first user when using multiple workers (#632).
- Allow both ports 5173 and 4173 as CORS origins (#637).
1.2.0¶
- Drop
project.project_dir
and replace it withuser.cache_dir
(#601). - Update SLURM backend (#582, #612, #614); this includes (1) combining several tasks in a single SLURM job, and (2) offering more granular sources for SLURM configuration options.
- Expose local user exports in SLURM configuration file (#625).
- Make local backend rely on custom
FractalThreadPoolExecutor
, whereparallel_tasks_per_job
can affect parallelism (#626). - Review logging configuration (#619, #623).
- Update to fastapi
0.95
(#587). - Minor improvements in dataset-edit endpoint (#593) and tests (#589).
- Include test of non-python task (#594).
- Move dummy tasks from package to tests (#601).
- Remove deprecated parsl backend (#607).
- Improve error handling in workflow-import endpoint (#595).
- Also show logs for successful workflow execution (#635).
1.1.1¶
- Include
reordered_workflowtask_ids
in workflow-edit endpoint payload, to reorder the task list of a workflow (#585).
1.1.0¶
- Align with new tasks interface in
fractal-tasks-core>=0.8.0
, and removeglob_pattern
column fromresource
database table (#544). - Drop python 3.8 support (#527).
- Improve validation of API request payloads (#545).
- Improve request validation in project-creation endpoint (#537).
- Update the endpoint to patch a
Task
(#526). - Add new project-update endpoint, and relax constraints on
project_dir
in new-project endpoint (#563). - Update
DatasetUpdate
schema (#558 and #565). - Fix redundant task-error logs in slurm backend (#552).
- Improve handling of task-collection errors (#559).
- If
FRACTAL_BACKEND_RUNNER=slurm
, include some configuration checks at server startup (#529). - Fail if
FRACTAL_SLURM_WORKER_PYTHON
has different versions offractal-server
orcloudpickle
(#533).
1.0.8¶
- Fix handling of parallel-tasks errors in
FractalSlurmExecutor
(#497). - Add test for custom tasks (#500).
- Improve formatting of job logs (#503).
- Improve error handling in workflow-execution server endpoint (#515).
- Update
_TaskBase
schema from fractal-common (#517).
1.0.7¶
- Update endpoints to import/export a workflow (#495).
1.0.6¶
- Add new endpoints to import/export a workflow (#490).
1.0.5¶
- Separate workflow-execution folder into two (server- and user-owned) folders, to avoid permission issues (#475).
- Explicitly pin sqlalchemy to v1 (#480).
1.0.4¶
- Add new POST endpoint to create new Task (#486).
1.0.3¶
Missing due to releasing error.
1.0.2¶
- Add
FRACTAL_RUNNER_MAX_TASKS_PER_WORKFLOW
configuration variable (#469).
1.0.1¶
- Fix bug with environment variable names (#468).
1.0.0¶
- First release listed in CHANGELOG.