All notable changes to this project will be documented in this file.

The format is based on "Keep a Changelog", and this project adheres to "Semantic Versioning".

Released versions

1.4.8 (2024-08-07)

Added

easypipe launch command: now copies also all pdb and mtz in bulk in a folder named 'ALL_pdb_mtz'

Changed

easypipe ligand command: now generates the cif file only once for a given ligand name if it appears more than once. Alerts for the case there are 2 different smiles for the same ligand name

Fixed

easyget: some process names where not found when folder was just 'autoprocessing' and not 'autoprocessing_...'
easypipe prep command: deals with the error raised when data files are directly in dataset folder without any intermediate process folder, create a "mtz001_process" folder (no process name catched)
easypipe launch command: stopped at space group search in the reference pdb file when there was an empty row in the pdb file before the row with 'CRYST1'

1.4.7 (2024-02-01)

Changed

easypipe launch command: if launched in allsg and completeness/resolution > cutoff => tagged in fast mode and 'yes' in 'to treat', now 'no' in the 'to treat' column (still 'yes' for fast and full modes)
easypipe launch command: autobuild not run when allsg mode because very long and not necessary except if ligand search, and ligand search generally run in fast or full mode

Fixed

easyget: following a change in the path of autoprocessing folders at ESRF, some "autoprocessing" folder didn't contain any folder or were for characterization, they are no more included
easyget: bug fixed, when found the first "autoprocessing" folder, didn't look for others for the same dataset (may be a problem when there are autoprocessing_line and autoprocessing_mesh folders)
easypipe prep command: when several mtz in a folder and no one matches templates, first mtz in the folder should be considered, but it wasn't copied! Now consider first mtz without 'unmerged' in its name
easypipe prep and summary commands: dataset names were not sorted alphabetically, case-insensitive => fixed
easypipe reindex command: if a dataset was added afterwards, and consequently prep and reindex steps re-run, if this dataset didn't need a re-indexation, it wasn't copied in the csv file 'mtz_to_treat_ALL_reindexed.csv' generated after reindexation, for further treatment. Now, after reindexation step, csv file 'mtz_to_treat_ALL_reindexed.csv' is systematically generated even if no reindexation occurs

1.4.6 (2023-09-21)

Fixed

easyget: following a change in the path of autoprocessing folders at ESRF, now these folders are searched recursively which is longer but more robust (work also for old data organization)

1.4.5 (2023-07-28)

Added

easyget: data from ALBA can now be downloaded too
easyget: error explicitly written when files are not downloadable from synchrotron (rights problem from synchrotron) instead of simply pass
easypipe summary command: new option to generate summary file without unsuccessfull treatments for better clarity, and at the end list of datasets not treated and datasets with no successfull treatment
easypipe prep command: even when already run, prep can be run again with additional new processes, mtz number will be incremented
easypipe prep command: if there are several mtz for a same process and no one fits templates from the list => message and takes the first mtz file

Changed

easypipe prep command: if there are several mtz for a same process, now copy all the mtz fitting the template list and no more only the first one (needed if all the processes are gathered in the same folder)

Fixed

easypipe: when looking for reference space group in reference pdb, space group was not found if a number (>1 character) was at the end of the row tagged CRYST1 in the pdb file

1.4.4 (2023-03-29)

Added

easypipe reindex command: new information at the end of simulation mode to explain what to do
easypipe launch command: run automatically a 'summary' command at the end, but only concerning RESULT file created by this 'launch' command
easypipe launch command: new column in the RESULTS file for unit cell extracted from final pdb (usefull for allsg mode)
easypipe launch command: even when Phenix fails, writes resolution, completeness, space group, and unit cell, in the RESULTS csv file (copy of data in the launch file comming from mtz)
easypipe summary command: create a SUMMARY file that compiles all SUMMARY files in RESULTS folders if several RESULTS folders (case when launched for several space groups, or different templates)

Changed

easypipe summary command: among all sorting criteria, sorting criteria by mode removed, sorting criteria by Rwork added

Fixed

easypipe: in auto mode, simulate option put to False for reindex (else bug)
easypipe reindex command: reindex giving the space group in number didn't work, now it works
easypipe launch command: when re-running the same launch with some modifications in the 'launch' csv file, RESULTS file at the end of the first run was based on the information in the modified 'launch' csv file of the second run and so we could miss some results, now it is based on a copy of the 'launch' csv file in '_mtz_treated' folder

1.4.3 (2022-11-08)

Added

easypipe launch command: new column 'ligand name' in RESULTS csv file, if 'ligand search' = yes write the name of the smiles file of the corresponding ligand if given at 'ligand' step (not the case if cif given)
easypipe reindex command: it is now possible to run it in simulation mode in order to choose the datasets to reindex, if you don't want to reindex all the datasets that do not have the right space group

Changed

easypipe launch command: in result csv, when no ligand is found, wrote '0', now write '0/', like when one or several ligands are found
easypipe launch command: change in the way to increment RESULTS csv files to deal with possible overwriting if some are deleted by user
easypipe and easyget: logos now generated with 'figlet'

Fixed

easypipe launch command: error when trying to delete pipeline_1 folder while another process is working in => catched
easypipe launch command: error when space group not found in list, like for permutation (with Phaser), now write "space group not in official list or not found, see pdb file"

1.4.2 (2022-07-12)

Added

easypipe prep command: mtz from autoPROC 'staraniso' process are now also selected and copied for treatment, but not treated in the launch option '--autoproc'

Changed

easyget: PermissionError catched when persmission denied, with an understandable error message

Fixed

easyget: when there was no data at the date given, but at day+1, data at day+1 were not downloaded => fixed
easypipe launch command: allsg failed in sorting processes before launch, when a process had no information successfully found (resolution, completeness, sg) => fixed

1.4.1 (2022-04-06)

Changed

easyget: adaptation to the new connection mode via sftp at ESRF, now with personal SMIS login
easyget: the password is now given in a hidden way

1.4.0 (2022-03-31)

Changed

easyget: now test next day for the date (for 24h collections) for each date given and not only when just one date
easypipe reindex command: doesn't run again if already done
easypipe reindex command: list at the end all the reindexations that failed
easypipe reindex command: copy xtriage log file of reindexed mtz along with reindexed mtz in data folder
easypipe launch command: addition of space group and template in the name of the log file

Fixed

easypipe reindex command: it happened xtriage failed on mtz after reindexing (on unusually incomplete data for example) and creates no log file where space group can be search. Now check if xtriage log file exists before searching space group.
easypipe reindex command: doesn't fail if error when writing mtz
easypipe reindex command: resolution and unit cell no more copied from csv from prep step, but searched in xtriage log as completeness and space group (for case xtriage failed at prep step but not at reindex step)
easypipe prep command: no more failure when problem with xtriage (error or no log file created)
easypipe launch command: if no resolution or completeness found at 'prep' step (failure with xtriage), no error and mtz marked as not to treat in launch csv file
easypipe launch command: sort by decreasing completeness was done on a string field, now sorted on floats so that 100% is no more at the end but first
easypipe summary command: no more failure when no RESULTS csv file in one of the RESULTS directories

1.3.0 (2021-12-01)

Added

easypipe launch command: new column 'ligand smiles' in RESULTS csv file, if 'ligand search' = yes write the smiles of the corresponding ligand if given at 'ligand' step (not the case if cif given)

Changed

easypipe prep command: copied XDS_ACSII.HKL but not XDS_ANOM.HKL, now copy all .HKL files
easypipe prep command: now also copy mtz files with 'multirecord' in their name (unmerged data for deposition)
easypipe: possible to run 'launch --best' instead of 'launch --best 1'

Fixed

easypipe launch command: copy of csv with mtz to be treated (launch csv) in RESULTS folder was done after running all Phenix. So if you wanted to launch again just modifying the launch csv file, before it ended, the csv with mtz to be treated copied at the end was no more the good one. Now, launch csv file is copied just before running all Phenix.

1.2.0 (2021-09-30)

Added

easypipe launch command: added option to treat the whole mtz processes
easypipe: added option to treat automatically all datasets, without ligand search. Run prep, reindex, launch and summary subcommands successively without any intervention

Changed

easypipe launch command: when no ligand search, no more error when R/Rfree is too high (and so results are copied in the result csv file)
easypipe launch command: in 'Phenix first round after Coot' column of csv result file (for linux), name of resulting pdb after first round has changed with the addition of the mode (full or fast)
easypipe reindex command: don't reindex when already done, just do it on new added datasets

Fixed

easyget: when only one date and given not in the command line, there was an error
easypipe launch command: didn't find anymore the space group in the reference pdb file, now OK
easypipe prep command: name of copied processed data folder deduced from path was wrong when dataset name appeared in a subfolder name
easypipe launch command: column 'failing step' in RESULTS csv file, after 'SUCESS' column. If SUCCESS = "no", get the last title in pipeline.log file, sometimes it wasn't the title, now test if 80 characters ensuring it is a title

1.1.0 (2021-07-28)

Added

easyget: summary at the end of copied datasets, of datasets not copied because folder already exists, and of not copied datasets because no process
easyget: when just one date given, test also the day after (useful for 24h experiments ...)
easypipe / ligand command: at the end, list of datasets for which generation of cif has failed, list of datasets with well generated cif, list of datasets with already a cif in their 'ligand' folder, list of datasets without information for cif generation

Changed

easyget: now when several dates, they are listed without quotes in the command line
easyget: less verbose during datasets copy (not each file name written, only process folder)
easyget: when one date is not correct in a list of dates, doesn't stop and work with the right date(s)
easyget: when program stopped by user without download, this is now written in log
easypipe / reindex command: now csv with reindexed results is in '1b_reindex_`SG`' folder
easypipe / reindex command: when mtz sucessfully reindexed, no more rename previous mtz in .old, avoiding error when running again reindex command
easypipe / reindex command: use of phenix.reflection_file_converter instead of pointless to reindex, avoiding "Environment variable CLIBD not set ... big trouble!" when run with space group in some cases with Pointless
easypipe / reindex command: now reference space group is given directly without the need for a reference mtz
easypipe launch command: change in name of pdb copied from LigandFit to be homogeneous with corresponding pdb and mtz (order of information'CC', 'nb ligands' and 'mode')
easypipe / launch command: when copy of csv file with information to launch in RESULTS folder, _mtz_treated subfolder, the launch csv file is renamed with a counter in case of a new launch with this file but with some parameters changed in the file. Same counter for RESULTS_... csv files for the same reasons in RESULTS folder. Avoid overwriting previous files which leads to a loose of information on previous runs
easypipe / launch command: New column 'failing step' in RESULTS csv file, after 'SUCESS' column. If SUCCESS = "no", get the last title in pipeline.log file so as we can know at which step it failed

Fixed

easypipe: in the command line in 'Phenix first round after Coot' column, the name of files was not good because information 'CC' and 'nb ligands' had been included in the name of files to avoid overwriting results files
easypipe / reindex command: doesn't change anymore the name of reindexed file if resulting space group is not the one expected (so, if reindex is launched again, no file with _reindexed_sg_reindexed_sg.mtz in their name and no other bug like listed files not found)
easypipe / reindex command: problem with Pointless when spacegroup as input solved, but Pointless replaced by phenix.reflection_file_converter after this correction
easypipe / launch command: when completeness < completeness min or resolution > resolution min, process is launched in fast mode, but it should also not search for ligand. Now it is effectively the case

1.0.0 (2021-05-04)

First public release