Skip to content
Snippets Groups Projects

Compare revisions

Changes are shown as if the source revision was being merged into the target revision. Learn more about comparing revisions.

Source

Select target project
No results found

Target

Select target project
  • swain-lab/aliby/aliby-mirror
  • swain-lab/aliby/alibylite
2 results
Show changes
Commits on Source (1332)
1,232 additional commits have been omitted to prevent performance issues.
Expand all Hide whitespace changes
Inline Side-by-side
Showing
with 1399 additions and 343 deletions
.flake8 0 → 100644
View file @ 95958192
[flake8]
ignore = E203, E266, E501, W503, F403, F401
max-line-length = 79
select = B,C,E,F,W,T4,B9
exclude =
# No need to traverse our git directory
.git,
# There's no value in checking cache directories
__pycache__,
# Ignore virtual environment contents
.venv
# The conf file is mostly autogenerated, ignore it
docs/source/conf.py,
# The old directory contains Flake8 2.0
old,
# This contains our built documentation
build,
# This contains builds of flake8 that we don't want to check
dist,
# Any data produced inside the folder during development
data/,
# Temporarily ignore tests
tests/
max-complexity = 18
.gitignore
View file @ 95958192
......@@ -110,7 +110,6 @@ venv.bak/
omero_py/omeroweb/
omero_py/pipeline/
**.ipynb
data/
notebooks/
*.pdf
*.h5
......
.gitlab-ci.yml 0 → 100644
View file @ 95958192
image: python:3.8
cache:
key: "project-${CI_JOB_NAME}"
paths:
- .venv
key:
files:
- poetry.lock
variables:
TRIGGER_PYPI_NAME: ""
stages:
- tests
- checks
# - release
before_script:
- test -e $HOME/.poetry/bin/ || curl -sSL https://install.python-poetry.org | python3 -
- export PATH="$PATH:$HOME/.local/bin/"
- poetry --version
- poetry config virtualenvs.in-project true
- pip install --upgrade pip
# - git remote rm origin && git remote add origin https://${ACCESS_TOKEN_NAME}:${ACCESS_TOKEN}@${CI_SERVER_HOST}/${CI_PROJECT_PATH}.git
# - git config pull.rebase false
# - git pull origin HEAD:master
# - rm -rf ~/.cache/pypoetry
# - if [ ${var+TRIGGER_PYPI_NAME} ]; then echo "Pipeline triggered by ${TRIGGER_PYPI_NAME}"; poetry add ${TRIGGER_PYPI_NAME}@latest; fi
# - export WITHOUT="docs,network";
- export ARGS="--with test,dev";
- if [[ "$CI_STAGE_NAME" == "tests" ]]; then echo "Installing system dependencies for ${CI_STAGE_NAME}"; apt update && apt install -y ffmpeg libsm6 libxext6; fi
- if [[ "$CI_JOB_NAME" == "Static Type" ]]; then echo "Activating development group"; export ARGS="${ARGS},dev"; fi
- if [[ "$CI_JOB_NAME" == "Network Tools Tests" ]]; then echo "Setting flag to compile zeroc-ice"; export ARGS="${ARGS} --all-extras"; fi
- poetry install -vv $ARGS
Local Tests:
stage: tests
script:
# - poetry install -vv
- poetry run coverage run -m --branch pytest ./tests --ignore ./tests/aliby/network --ignore ./tests/aliby/pipeline
- poetry run coverage report -m
- poetry run coverage xml
coverage: '/(?i)total.*? (100(?:\.0+)?\%|[1-9]?\d(?:\.\d+)?\%)$/'
artifacts:
reports:
coverage_report:
coverage_format: cobertura
path: coverage.xml
Network Tools Tests:
stage: tests
script:
- poetry run pytest ./tests/aliby/network
- DIRNAME="test_datasets"
- curl https://zenodo.org/record/7513194/files/test_datasets.tar.gz\?download\=1 -o "test_datasets.tar.gz"
- mkdir -p $DIRNAME
- tar xvf test_datasets.tar.gz -C $DIRNAME
- poetry run pytest -s tests/aliby/pipeline --file $DIRNAME/560_2022_11_30_pypipeline_unit_test_reconstituted_00
Static Type:
stage: checks
allow_failure: true
script:
- poetry run black .
- poetry run isort .
- poetry run mypy . --exclude 'setup\.py$'
# We can remove the flag once this is resolved https://github.com/pypa/setuptools/issues/2345
# TODO add more tests before activating auto-release
# Bump_release:
# stage: release
# script:
# - git config --global user.email ${GITLAB_USER_EMAIL}
# - git config --global user.name ${GITLAB_USER_NAME}
# # - git pull origin HEAD:MASTER && poetry version ${BUMP_RULE} && git add poetry.lock add pyproject.toml poetry.lock && git commit -m "Bump version" && git push -o ci.skip origin HEAD:master && poetry publish --build --username ${PYPI_USER} --password ${PYPI_PASSWORD}
# - echo "TRIGGER_PYPI_NAME=$(cat pyproject.toml | grep '^name =' | head -n 1 | cut -f3 -d' ' | tr -d \")" >> build.env
# - echo "Exporting TRIGGER_PYPI_NAME as ${TRIGGER_PYPI_NAME}"
# only:
# - master
# except:
# changes:
# - tests/
# - .*
# needs:
# job: Unit test
## Custom stages ##
.gitlab/issue_templates/bug.md
View file @ 95958192
## Summary
(Summarize the bug encountered concisely)
{Summarize the bug encountered concisely}
I confirm that I have (if relevant):
- [ ] Read the troubleshooting guide: https://gitlab.com/aliby/aliby/-/wikis/Troubleshooting-(basic)
- [ ] Updated aliby and aliby-baby.
- [ ] Tried the unit test.
- [ ] Tried a scaled-down version of my experiment (distributed=0, filter=0, tps=10)
- [ ] Tried re-postprocessing.
## Steps to reproduce
(How one can reproduce the issue - this is very important)
{How one can reproduce the issue - this is very important}
- aliby version: 0.1.{...}, or if development/unreleased version, commit SHA: {...}
- platform(s):
- [ ] Jura
- [ ] Other Linux, please specify distribution and version: {...}
- [ ] MacOS, please specify version: {...}
- [ ] Windows, please specify version: {...}
- experiment ID: {...}
- Any special things you need to know about this experiment: {...}
## What is the current bug behavior?
......@@ -19,7 +35,12 @@
(Paste any relevant logs - please use code blocks (```) to format console output, logs, and code, as
it's very hard to read otherwise.)
```
{PASTE YOUR ERROR MESSAGE HERE!!}
```
## Possible fixes
(If you can, link to the line of code that might be responsible for the problem)
.pre-commit-config.yaml 0 → 100644
View file @ 95958192
exclude: '^examples/'
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.3.0
hooks:
# - id: check-yaml
- id: end-of-file-fixer
- id: trailing-whitespace
args: [--markdown-linebreak-ext=md]
- id: check-added-large-files
- id: check-docstring-first
- id: name-tests-test
args: [--pytest-test-first]
- repo: https://github.com/psf/black
rev: 22.6.0
hooks:
- id: black
- repo: https://github.com/PyCQA/isort
rev: 5.10.1
hooks:
- id: isort
- repo: https://github.com/PyCQA/flake8
rev: 4.0.1
hooks:
- id: flake8
exclude: ^tests/ # Temporarily exclude tests
.readthedocs.yaml 0 → 100644
View file @ 95958192
# .readthedocs.yaml
# Read the Docs configuration file
# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
# Required
version: 2
# Set the version of Python and other tools you might need
build:
os: ubuntu-20.04
tools:
python: "3.8"
# Build documentation in the docs/ directory with Sphinx
sphinx:
configuration: docs/source/conf.py
# If using Sphinx, optionally build your docs in additional formats such as PDF
# formats:
# - pdf
# Optionally declare the Python requirements required to build your docs
python:
install:
- requirements: docs/requirements.txt
CONTRIBUTING.md 0 → 100644
View file @ 95958192
# Contributing
We focus our work on python 3.8 due to the current neural network being developed on tensorflow 1. In the near future we will migrate the network to pytorch to support more recent versions of all packages.
## Issues
All issues are managed within the gitlab [ repository ](https://gitlab.com/aliby/aliby/-/issues), if you don't have an account on the University of Edinburgh's gitlab instance and would like to submit issues please get in touch with [Prof. Peter Swain](mailto:peter.swain@ed.ac.uk ).
## Data aggregation
ALIBY has been tested by a few research groups, but we welcome new data sources for the models and pipeline to be as general as possible. Please get in touch with [ us ](mailto:peter.swain@ed.ac.uk ) if you are interested in testing it on your data.
INSTALL.md deleted 100644 → 0
View file @ 73b7b230
# Installation
Tested on: Mac OSX Mojave
## Requirements
We strongly recommend installing within a python environment as there are many dependencies that you may not want polluting your regular python environment.
An environment can be created with using the conda package manager:
$ conda create --name <env>
$ conda activate <env>
Which you can deactivate with:
$ conda deactivate
Or using virtualenv:
$ python -m virtualenv /path/to/venv/
$ source /path/to/venv/bin/activate
This will download all of your packages under `/path/to/venv` and then activate it.
Deactivate using
$ deactivate
## Pipeline installation
Once you have created your local environment, run:
$ cd pipeline-core
$ pip install -e ./
You will be asked to put in your GitLab credentials for a couple of the packages installed as dependencies.
The `-e` option will install in 'editable' mode: all of the changes that are made to the code in the repository will immediately be reflected in the installation.
This is very useful to keep up with any changes and branching that we make.
That way, if you want to keep up with the most recent updates, just run:
$ git pull
If you would rather be more in charge of which updates you install and which you don't, remove `-e` from your installation command.
In this case (or if you run into a dependency error) in order to update your installed version you will have to run:
$ cd pipeline-core
$ git pull
$ pip install ./
LICENSE 0 → 100644
View file @ 95958192
This diff is collapsed.
README.md
View file @ 95958192
# Pipeline core
# ALIBY (Analyser of Live-cell Imaging for Budding Yeast)
The core classes and methods for the python microfluidics, microscopy, and
analysis pipeline.
### Installation
See [INSTALL.md](./INSTALL.md) for installation instructions.
[![docs](https://readthedocs.org/projects/aliby/badge/?version=master)](https://aliby.readthedocs.io/en/latest)
[![PyPI version](https://badge.fury.io/py/aliby.svg)](https://badge.fury.io/py/aliby)
[![pipeline](https://gitlab.com/aliby/aliby/badges/master/pipeline.svg?key_text=master)](https://gitlab.com/aliby/aliby/-/pipelines)
[![dev pipeline](https://gitlab.com/aliby/aliby/badges/dev/pipeline.svg?key_text=dev)](https://gitlab.com/aliby/aliby/-/commits/dev)
[![coverage](https://gitlab.com/aliby/aliby/badges/dev/coverage.svg)](https://gitlab.com/aliby/aliby/-/commits/dev)
End-to-end processing of cell microscopy time-lapses. ALIBY automates segmentation, tracking, lineage predictions, post-processing and report production. It leverages the existing Python ecosystem and open-source scientific software available to produce seamless and standardised pipelines.
## Quickstart Documentation
### Setting up a server
For testing and development, the easiest way to set up an OMERO server is by
using Docker images.
[The software carpentry](https://software-carpentry.org/) and the [Open
Microscopy Environment](https://www.openmicroscopy.org), have provided
[instructions](https://ome.github.io/training-docker/) to do this.
The `docker-compose.yml` file can be used to create an OMERO server with an
accompanying PostgreSQL database, and an OMERO web server.
It is described in detail
[here](https://ome.github.io/training-docker/12-dockercompose/).
Our version of the `docker-compose.yml` has been adapted from the above to
use version 5.6 of OMERO.
To start these containers (in background):
```shell script
cd pipeline-core
docker-compose up -d
```
Omit the `-d` to run in foreground.
Installation of [VS Studio](https://visualstudio.microsoft.com/downloads/#microsoft-visual-c-redistributable-for-visual-studio-2022) Native MacOS support for is under work, but you can use containers (e.g., Docker, Podman) in the meantime.
To stop them, in the same directory, run:
```shell script
docker-compose stop
```
To analyse local data
```bash
pip install aliby
```
Add any of the optional flags `omero` and `utils` (e.g., `pip install aliby[omero, utils]`). `omero` provides tools to connect with an OMERO server and `utils` provides visualisation, user interface and additional deep learning tools.
See our [installation instructions]( https://aliby.readthedocs.io/en/latest/INSTALL.html ) for more details.
### Raw data access
Raw data access can be found in `core.experiment` and `core.timelapse`, and
is organised inspired by the Swain Lab MATLAB pipeline.
The `Experiment` classes are basically the only ones that really need to be
accessed by a user. The `ExperimentOMERO` and `ExperimentLocal` classes
implement the different possible sources of the data.
If the experiment is saved locally, we expect the organisation to be as in
[this repository](https://github.com/SynthSys/omero_connect_demo/tree/master/test_data)
`Experiment` cannot be instantiated as it is an abstract class, but calling
`Experiment.from_source()` will instantiate either an `ExperimentOMERO` or an
`ExperimentLocal` class depending on the arguments, and the differences between
the two are invisible to the user from then on.
### CLI
```python
from core.experiment import Experiment
local_expt = Experiment.from_source('path/to/data/directory/')
omero_expt = Experiment.from_source(10421, #Experiment ID on OMERO
'user', #OMERO Username
'password', #OMERO Password
'host.omero.ed.ac.uk', #OMERO host
port=4064 #This is default
)
```
Data is organised in each experiment as `Timelapse` classes. These are also
separated into `TimelapseOMERO` and `TimelapseLocal`.
The main function of these objects is to give a direct interface to the raw
data, whatever form it is saved in.
These objects are sliceable, meaning that data can be accessed like a numpy
array (with some reservations). This can be done directly through the
`Experiment` object.
If installed via poetry, you have access to a Command Line Interface (CLI)
```python
bf_1 = expt[0, 0, :, :, :] # First channel, first timepoint, all x,y,z
```
Aside from the argument parsing, this is implemented through the
`get_hypercube()` function, which can be called directly from the `Experiment`
object.
```python
x, y, width, height, z_positions, channels, timepoints = [None]*7 #Get full pos
expt.get_hypercube(x, y, width, height, z_positions, channels,
timepoints)
```
To change position (`Timelapse`), one simply needs to set `Experiment
.curent_position` to the desired position name.
```bash
aliby-run --expt_id EXPT_PATH --distributed 4 --tps None
```
```python
position = expt.positions[0] #This is the default position when expt initalized
expt.current_position = positions
```
And to run Omero servers, the basic arguments are shown:
```bash
aliby-run --expt_id XXX --host SERVER.ADDRESS --user USER --password PASSWORD
```
### Tiling the raw data
The output is a folder with the original logfiles and a set of hdf5 files, one with the results of each multidimensional inside.
The tiling of raw data is done through a `Tiler` object.
It takes a raw `Experiment` object as an argument.
For more information, including available options, see the page on [running the analysis pipeline](https://aliby.readthedocs.io/en/latest/PIPELINE.html)
```python
from core.segment import Tiler
seg_expt = Tiler(expt)
```
## Using specific components
The initialization should take a few seconds, as it needs to align the images
in time.
### Access raw data
#### Get a timelapse for a given trap
From there, you can obtain a timelapse for a single trap as follows:
```python
channels = [0] #Get only the first channel, this is also the default
z = [0, 1, 2, 3, 4] #Get all z-positions
trap_id = 0
tile_size = 117
# Get a timelapse of the trap
# The default trap size is 96 by 96
# The trap is in the center of the image, except for edge cases
# The output has shape (C, T, X, Y, Z), so in this example: (1, T, 96, 96, 5)
timelapse = seg_expt.get_trap_timelapse(trap_id, tile_size=tile_size,
channels=channels, z=z)
ALIBY's tooling can also be used as an interface to OMERO servers, for example, to fetch a brightfield channel.
```python
from aliby.io.omero import Dataset, Image
server_info= {
"host": "host_address",
"username": "user",
"password": "xxxxxx"}
expt_id = XXXX
tps = [0, 1] # Subset of positions to get.
with Dataset(expt_id, **server_info) as conn:
image_ids = conn.get_images()
#To get the first position
with Image(list(image_ids.values())[0], **server_info) as image:
dimg = image.data
imgs = dimg[tps, image.metadata["channels"].index("Brightfield"), 2, ...].compute()
# tps timepoints, Brightfield channel, z=2, all x,y
```
This can take several seconds at the moment.
For a speed-up: take fewer z-positions if you can.
If you're not sure what indices to use:
```python
seg_expt.channels # Get a list of channels
channel = 'Brightfield'
ch_id = seg_expt.get_channel_index(channel)
### Tiling the raw data
n_traps = seg_expt.n_traps # Get the number of traps
```
#### Get the traps for a given time point
Alternatively, if you want to get all the traps at a given timepoint:
A `Tiler` object performs trap registration. It may be built in different ways but the simplest one is using an image and a the default parameters set.
```python
timepoint = 0
seg_expt.get_traps_timepoints(timepoint, tile_size=96, channels=None,
z=[0,1,2,3,4])
from aliby.tile.tiler import Tiler, TilerParameters
with Image(list(image_ids.values())[0], **server_info) as image:
tiler = Tiler.from_image(image, TilerParameters.default())
tiler.run_tp(0)
```
## Reading MATLAB files
*Disclaimer: this is very much still in development so it may not always
work for you case. If you run into any problems please let me know, or even
better start an Issue on the project describing your problem.*
The initialisation should take a few seconds, as it needs to align the images
in time.
At the moment the best/only way to read matlab files is through a `matObject`:
It fetches the metadata from the Image object, and uses the TilerParameters values (all Processes in aliby depend on an associated Parameters class, which is in essence a dictionary turned into a class.)
#### Get a timelapse for a given tile (remote connection)
```python
from core.io.matlab import matObject
cTimelapse = matObject('/path/to/cTimelapse.mat')
```
fpath = "h5/location"
You can see an overview of what's in the object:
```python
cTimelapse.describe()
```
tile_id = 9
trange = range(0, 10)
ncols = 8
The `matObject` has some dictionary-like features although it is *not* a
dictionary (yet). You can access different parts of the object using keys
, though, and can use the `keys()` function to do so. This will usually
work at the first few levels, but if it doesn't you may have run into an
object that's actually a list or a numpy array.
riv = remoteImageViewer(fpath)
trap_tps = [riv.tiler.get_tiles_timepoint(tile_id, t) for t in trange]
```python
cTimelapse.keys()
# You can also access labelled traps
m_ts = riv.get_labelled_trap(tile_id=0, tps=[0])
# And plot them directly
riv.plot_labelled_trap(trap_id=0, channels=[0, 1, 2, 3], trange=range(10))
```
This should return an iterable of the upper level keys. For example, a
timelapse object will usually have a `timelapseTrapsOmero` key which you
can look deeper into in the same manner. Once you've found what you want
you can usually access it as you would a nested dictionary, for instance:
Depending on the network speed can take several seconds at the moment.
For a speed-up: take fewer z-positions if you can.
```python
cTimelapse['timelapseTrapsOmero']['cTimepoint']['trapLocations']
```
#### Get the tiles for a given time point
Alternatively, if you want to get all the traps at a given timepoint:
For more information about using MATLAB files in python objects, please see
[this page](docs/matlab.md).
## Development guidelines
In order to separate the python2, python3, and "currently working" versions
(\#socialdistancing) of the pipeline, please use the branches:
* python2.7: for any development on the 2 version
* python3.6-dev: for any added features for the python3 version
* master: very sparingly and only for changes that need to be made in both
versions as I will be merging changes from master into the development
branches frequently
* Ideally for adding features into any branch, espeically master, create
a new branch first, then create a pull request (from within Gitlab) before
merging it back so we can check each others' code. This is just to make
sure that we can always use the code that is in the master branch without
any issues.
Branching cheat-sheet:
```git
git branch my_branch # Create a new branch called branch_name from master
git branch my_branch another_branch #Branch from another_branch, not master
git checkout -b my_branch # Create my_branch and switch to it
# Merge changes from master into your branch
git pull #get any remote changes in master
git checkout my_branch
git merge master
# Merge changes from your branch into another branch
git checkout another_branch
git merge my_branch #check the doc for --no-ff option, you might want to use it
```python
timepoint = (4,6)
tiler.get_tiles_timepoint(timepoint, channels=None,
z=[0,1,2,3,4])
```
## TODO
### Tests
* test full pipeline with OMERO experiment (no download.)
### Contributing
See [CONTRIBUTING](https://aliby.readthedocs.io/en/latest/INSTALL.html) on how to help out or get involved.
config.json deleted 100644 → 0
View file @ 73b7b230
{"host": "sce-bio-c04287.bio.ed.ac.uk", "password": "***REMOVED***", "port": 4064,
"user": "upload", "experiment": 10932}
docker-compose.yml deleted 100644 → 0
View file @ 73b7b230
# Modified from https://ome.github.io/training-docker/12-dockercompose/
version: "3"
services:
database:
image: "postgres:11"
environment:
- POSTGRES_USER=omero
- POSTGRES_DB=omero
- POSTGRES_PASSWORD=SeCrEtPaSsWoRd
networks:
- omero-network
volumes:
- "database-volume:/var/lib/postgresql/data"
omeroserver:
image: "openmicroscopy/omero-server:5.6"
environment:
- CONFIG_omero_db_host=database
- CONFIG_omero_db_user=omero
- CONFIG_omero_db_pass=SeCrEtPaSsWoRd
- CONFIG_omero_db_name=omero
- ROOTPASS=omero-root-password
networks:
- omero-network
ports:
- "4063:4063"
- "4064:4064"
volumes:
- "omero-volume:/OMERO"
omeroweb:
image: "openmicroscopy/omero-web-standalone:master"
environment:
- OMEROHOST=omeroserver
networks:
- omero-network
ports:
- "4080:4080"
networks:
omero-network:
volumes:
database-volume:
omero-volume:
docs/matlab.md deleted 100644 → 0
View file @ 73b7b230
# Working with MATLAB files
Working with MATLAB files is possible but very likely to cause issues.
```python
from core.experiment import Experiment
omero_expt = Experiment.from_source(19310, #Experiment ID on OMERO
'upload', #OMERO Username
'***REMOVED***', #OMERO Password
'islay.bio.ed.ac.uk', #OMERO host
port=4064, #This is default
save_dir='../doc/' #Directory to cache files
)
# Download and save the MATLAB files
omero_expt.cache_annotations(matlab=True)
```
## Create a Tiler using the MATLAB information
Use the suffix to your MATLAB files as the input argument for `matlab`. For
example , if your cTimelapse files are named
`experiment_and_position_namecTimelapse_001.mat` then you should use
`'cTimelapse_001.mat'` as an argument, as below.
```python
from core.segment import Tiler
seg_expt = Tiler(omero_expt, matlab='cTimelapse_001.mat')
channels = [0] #Get only the first channel, this is also the default
z = [0]# 1, 2, 3, 4] #Get all z-positions
trap_id = 0
tile_size = 117
# Get trap images at a given timepoint
seg_expt.get_traps_timepoint(0, tile_size=tile_size, channels=channels, z=z)
```
## Create a Cells object using the MATLAB information
At the moment a `Cells` object can take a `cTimelapse` as input and no other
format. In the future we may create an experiment-wide `Cells` object where
the position is set like in the `Experiment` or `Tiler` objects.
The following will work if you have downloaded the matlab objects as
decribed above.
```python
from core.cells import Cells
# Currently hacky way to get the correct cells object. Note you will have to
# re-create a cells object each time you change position
mat_source = omero_expt.root_dir / (seg_expt.current_position + seg_expt.matlab)
cells = Cells.from_source(mat_source)
```
You can combine the images with the segmentations at one timepoint by doing:
```python
timepoint = 0
for image, segmentations in zip(seg_expt.get_traps_timepoint(timepoint
), cells.at_time(timepoint)):
run_analysis(image, segmentations)
```
docs/requirements.txt 0 → 100644
View file @ 95958192
numpydoc>=1.3.1
aliby[network]>=0.1.43
sphinx-autodoc-typehints==1.19.2
sphinx-rtd-theme==1.0.0
sphinxcontrib-applehelp==1.0.2
sphinxcontrib-devhelp==1.0.2
sphinxcontrib-htmlhelp==2.0.0
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.3
sphinxcontrib-serializinghtml==1.1.5
myst-parser
docs/source/INSTALL.md 0 → 100644
View file @ 95958192
# Installation
## Requirements
We strongly recommend installing within a python environment as there are many dependencies that you may not want polluting your regular python environment.
Make sure you are using python 3.
An environment can be created using [Anaconda](https://www.anaconda.com/):
$ conda create --name <env>
$ conda activate <env>
Which you can deactivate with:
$ conda deactivate
Or using virtualenv:
$ python -m virtualenv /path/to/venv/
$ source /path/to/venv/bin/activate
This will download all of your packages under `/path/to/venv` and then activate it.
Deactivate using
$ deactivate
You will also need to make sure you have a recent version of pip.
In your local environment, run:
$ pip install --upgrade pip
Or using [pyenv](https://github.com/pyenv/pyenv) with pyenv-virtualenv:
$ pyenv install 3.8.14
$ pyenv virtualenv 3.8.14 aliby
$ pyenv local aliby
## Pipeline installation
### Pip version
Once you have created and activated your virtual environment, run:
If you are not using an OMERO server setup:
$ pip install aliby
Otherwise, if you are contacting an OMERO server:
$ pip install aliby[network]
NOTE: Support for OMERO servers in GNU/Linux computers requires building ZeroC-Ice, thus it requires build tools. The versions for Windows and MacOS are provided as Python wheels and thus installation is faster.
### FAQ
- Installation fails during zeroc-ice compilation (Windows and MacOS).
For Windows, the simplest way to install it is using conda (or mamba). You can install the (OMERO) network components separately:
$ conda create -n aliby -c conda-forge python=3.8 omero-py
$ conda activate aliby
$ cd c:/Users/Public/Repos/aliby
$ \PATH\TO\POETRY\LOCATION\poetry install
- MacOS
For local access and processing, follow the same instructions as Linux. Remote access to OMERO servers depends on some issues in one of our depedencies being solved (See issue https://github.com/ome/omero-py/issues/317)
### Git version
Install [ poetry ](https://python-poetry.org/docs/#installation) for dependency management.
In case you want to have local version:
$ git clone git@gitlab.com/aliby/aliby.git
$ cd aliby
and then either
$$ poetry install --all-extras
for everything, including tools to access OMERO servers, or
$$ poetry install
for a version with only local access, or
$$ poetry install --with dev
to install with compatible versions of the development tools we use, such as black.
These commands will automatically install the [ BABY ](https://gitlab.com/aliby/baby) segmentation software. Support for additional segmentation and tracking algorithms is under development.
## Omero Server
We use (and recommend) [OMERO](https://www.openmicroscopy.org/omero/) to manage our microscopy database, but ALIBY can process both locally-stored experiments and remote ones hosted on a server.
### Setting up a server
For testing and development, the easiest way to set up an OMERO server is by
using Docker images.
[The software carpentry](https://software-carpentry.org/) and the [Open
Microscopy Environment](https://www.openmicroscopy.org), have provided
[instructions](https://ome.github.io/training-docker/) to do this.
The `docker-compose.yml` file can be used to create an OMERO server with an
accompanying PostgreSQL database, and an OMERO web server.
It is described in detail
[here](https://ome.github.io/training-docker/12-dockercompose/).
Our version of the `docker-compose.yml` has been adapted from the above to
use version 5.6 of OMERO.
To start these containers (in background):
```shell script
cd pipeline-core
docker-compose up -d
```
Omit the `-d` to run in foreground.
To stop them, in the same directory, run:
```shell script
docker-compose stop
```
### Troubleshooting
Segmentation has been tested on: Mac OSX Mojave, Ubuntu 20.04 and Arch Linux.
Data processing has been tested on all the above and Windows 11.
### Detailed Windows installation
#### Create environment
Open anaconda powershell as administrator
```shell script
conda create -n devaliby2 -c conda-forge python=3.8 omero-py
conda activate devaliby2
```
#### Install poetry
You may have to specify the python executable to get this to work :
```shell script
(Invoke-WebRequest -Uri https://install.python-poetry.org -UseBasicParsing).Content | C:\Users\USERNAME\Anaconda3\envs\devaliby2\python.exe -
``` Also specify full path when running poetry (there must be a way to sort this)
- Clone the repository (Assuming you have ssh properly set up)
```shell script
git clone git@gitlab.com:aliby/aliby.git
cd aliby
poetry install --all-extras
```
You may need to run the full poetry path twice - first time gave an error message, worked second time
```shell script
C:\Users\v1iclar2\AppData\Roaming\Python\Scripts\poetry install --all-extras
```
confirm installation of aliby - python...import aliby - get no error message
#### Access the virtual environment from the IDE (e.g., PyCharm)
New project
In location - navigate to the aliby folder (eg c::/Users/Public/Repos/aliby
- Select the correct python interpreter
click the interpreter name at the bottom right
click add local interpreter
on the left click conda environment
click the 3 dots to the right of the interpreter path and navigate to the python executable from the environment created above (eg C:\Users\v1iclar2\Anaconda3\envs\devaliby2\python.exe)
#### Potential Windows issues
- Sometimes the library pywin32 gives trouble, just install it using pip or conda
docs/source/PIPELINE.md 0 → 100644
View file @ 95958192
# Running the analysis pipeline
You can run the analysis pipeline either via the command line interface (CLI) or using a script that incorporates the `aliby.pipeline.Pipeline` object.
## CLI
On a CLI, you can use the `aliby-run` command. This command takes options as follows:
- `--host`: Address of image-hosting server.
- `--username`: Username to access image-hosting server.
- `--password`: Password to access image-hosting server.
- `--expt_id`: Number ID of experiment stored on host server.
- `--distributed`: Number of distributed cores to use for segmentation and signal processing. If 0, there is no parallelisation.
- `--tps`: Optional. Number of time points from the beginning of the experiment to use. If not specified, the pipeline processes all time points.
- `--directory`: Optional. Parent directory to save the data files (HDF5) generated, `./data` by default; the files will be stored in a child directory whose name is the name of the experiment.
- `--filter`: Optional. List of positions to use for analysis. Alternatively, a regex (regular expression) or list of regexes to search for positions. **Note: for the CLI, currently it is not able to take a list of strings as input.**
- `--overwrite`: Optional. Whether to overwrite an existing data directory. True by default.
- `--override_meta`: Optional. Whether to overwrite an existing data directory. True by default.
Example usage:
```bash
aliby-run --expt_id EXPT_PATH --distributed 4 --tps None
```
And to run Omero servers, the basic arguments are shown:
```bash
aliby-run --expt_id XXX --host SERVER.ADDRESS --user USER --password PASSWORD
```
## Script
Use the `aliby.pipeline.Pipeline` object and supply a dictionary, following the example below. The meaning of the parameters are the same as described in the CLI section above.
```python
#!/usr/bin/env python3
from aliby.pipeline import Pipeline, PipelineParameters
# Specify experiment IDs
ids = [101, 102]
for i in ids:
print(i)
try:
params = PipelineParameters.default(
# Create dictionary to define pipeline parameters.
general={
"expt_id": i,
"distributed": 6,
"host": "INSERT ADDRESS HERE",
"username": "INSERT USERNAME HERE",
"password": "INSERT PASSWORD HERE",
# Ensure data will be overwriten
"override_meta": True,
"overwrite": True,
}
)
# Fine-grained control beyond general parameters:
# change specific leaf in the extraction tree.
# This example tells the pipeline to additionally compute the
# nuc_est_conv quantity, which is a measure of the degree of
# localisation of a signal in a cell.
params = params.to_dict()
leaf_to_change = params["extraction"]["tree"]["GFP"]["np_max"]
leaf_to_change.add("nuc_est_conv")
# Regenerate PipelineParameters
p = Pipeline(PipelineParameters.from_dict(params))
# Run pipeline
p.run()
# Error handling
except Exception as e:
print(e)
```
This example code can be the contents of a `run.py` file, and you can run it via
```bash
python run.py
```
in the appropriate virtual environment.
Alternatively, the example code can be the contents of a cell in a jupyter notebook.
docs/source/_templates/custom-class-template.rst 0 → 100644
View file @ 95958192
{{ fullname | escape | underline}}
.. currentmodule:: {{ module }}
.. autoclass:: {{ objname }}
:members:
:show-inheritance:
:inherited-members:
{% block methods %}
.. automethod:: __init__
{% if methods %}
.. rubric:: {{ _('Methods') }}
.. autosummary::
{% for item in methods %}
~{{ name }}.{{ item }}
{%- endfor %}
{% endif %}
{% endblock %}
{% block attributes %}
{% if attributes %}
.. rubric:: {{ _('Attributes') }}
.. autosummary::
{% for item in attributes %}
~{{ name }}.{{ item }}
{%- endfor %}
{% endif %}
{% endblock %}
docs/source/_templates/custom-module-template.rst 0 → 100644
View file @ 95958192
{{ fullname | escape | underline}}
.. automodule:: {{ fullname }}
{% block attributes %}
{% if attributes %}
.. rubric:: Module Attributes
.. autosummary::
:toctree:
{% for item in attributes %}
{{ item }}
{%- endfor %}
{% endif %}
{% endblock %}
{% block functions %}
{% if functions %}
.. rubric:: {{ _('Functions') }}
.. autosummary::
:toctree:
{% for item in functions %}
{{ item }}
{%- endfor %}
{% endif %}
{% endblock %}
{% block classes %}
{% if classes %}
.. rubric:: {{ _('Classes') }}
.. autosummary::
:toctree:
:template: custom-class-template.rst
{% for item in classes %}
{{ item }}
{%- endfor %}
{% endif %}
{% endblock %}
{% block exceptions %}
{% if exceptions %}
.. rubric:: {{ _('Exceptions') }}
.. autosummary::
:toctree:
{% for item in exceptions %}
{{ item }}
{%- endfor %}
{% endif %}
{% endblock %}
{% block modules %}
{% if modules %}
.. rubric:: Modules
.. autosummary::
:toctree:
:template: custom-module-template.rst
:recursive:
{% for item in modules %}
{{ item }}
{%- endfor %}
{% endif %}
{% endblock %}
docs/source/api.rst 0 → 100644
View file @ 95958192
..
DO NOT DELETE THIS FILE! It contains the all-important `.. autosummary::` directive with `:recursive:` option, without
which API documentation wouldn't get extracted from docstrings by the `sphinx.ext.autosummary` engine. It is hidden
(not declared in any toctree) to remove an unnecessary intermediate page; index.rst instead points directly to the
package page. DO NOT REMOVE THIS FILE!.. autosummary::
.. autosummary::
:toctree: _autosummary
:template: custom-module-template.rst
:recursive:
aliby
agora
extraction
postprocessor
logfile_parser
docs/source/conf.py 0 → 100644
View file @ 95958192
# Configuration file for the Sphinx documentation builder.
#
# This file only contains a selection of the most common options. For a full
# list see the documentation:
# https://www.sphinx-doc.org/en/master/usage/configuration.html
# -- Path setup --------------------------------------------------------------
# If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here.
#
import os
import sys
sys.path.insert(0, os.path.abspath("../.."))
sys.setrecursionlimit(1500)
# -- Project information -----------------------------------------------------
project = "aliby"
copyright = "2022, Alan F. Muñoz"
author = "Alan F. Muñoz"
# The full version, including alpha/beta/rc tags
release = "0.1.26"
# -- General configuration ---------------------------------------------------
# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = [
"sphinx.ext.autodoc", # Core library for html generation from docstrings
"sphinx.ext.autosummary", # Create neat summary tables
"sphinx.ext.intersphinx", # Link to other project's documentation (see mapping below)
"sphinx.ext.viewcode", # Add a link to the Python source code for classes, functions etc.
"sphinx_autodoc_typehints", # Automatically document param types (less noise in class signature)
"numpydoc", # Play along with numpydoc format
"myst_parser", # Include markdown in rst files.
]
autosummary_generate = True # Turn on sphinx.ext.autosummary
autoclass_content = "both" # Add __init__ doc (ie. params) to class summaries
html_show_sourcelink = (
False # Remove 'view source code' from top of page (for html, not python)
)
autodoc_inherit_docstrings = True # If no docstring, inherit from base class
set_type_checking_flag = (
True # Enable 'expensive' imports for sphinx_autodoc_typehints
)
nbsphinx_allow_errors = True # Continue through Jupyter errors
# autodoc_typehints = "description" # Sphinx-native method. Not as good as sphinx_autodoc_typehints
add_module_names = False # Remove namespaces from class/method signatures
# Add any paths that contain templates here, relative to this directory.
templates_path = ["_templates"]
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path.
exclude_patterns = []
# -- Options for HTML output -------------------------------------------------
# on_rtd is whether we are on readthedocs.org
import os
on_rtd = os.environ.get("READTHEDOCS", None) == "True"
if not on_rtd: # only import and set the theme if we're building docs locally
import sphinx_rtd_theme
html_theme = "sphinx_rtd_theme"
html_theme_path = [sphinx_rtd_theme.get_html_theme_path()]
# otherwise, readthedocs.org uses their theme by default, so no need to specify it