bioconda http://bioconda-recipes-demo.readthedocs.io/en/docs/ Experiences with workflows for automating data-intensive bioinformatics Question: Workflow management software for pipeline development in NGS ： https://www.biostars.org/p/115745/ https://www.biostars.org/p/91301/ omics_pipe https://pythonhosted.org/omics_pipe/custom_pipelines.html A review of bioinformatic pipeline frameworks https://github.com/pditommaso/awesome-pipeline Awesome Pipeline A curated list of awesome pipeline toolkits inspired by Awesome Sysadmin Pipeline frameworks libraries ActionChain - A workflow system for simple linear success/failure workflows. Airflow - Python-based workflow system created by AirBnb. Anduril - Component-based workflow framework for scientific data analysis. Antha - High-level language for biology. Bds - Scripting language for data pipelines. Bpipe - Tool for running and managing bioinformatics pipelines. Briefly - Python Meta-programming Library for Job Flow Control. Cluster Flow - Command-line tool which uses common cluster managers to run bioinformatics pipelines. Clusterjob - Automated reproducibility, and hassle-free submission of computational jobs to clusters. Compss - Programming model for distributed infrastructures. Conan2 - Light-weight workflow management application. Cosmos - Python library for massively parallel workflows. Cuneiform - Advanced functional workflow language and framework, implemented in Erlang. Doit - Task management automation tool. Dagobah - Simple DAG-based job scheduler in Python. Drake - Robust DSL akin to Make, implemented in Clojure. Flex - Language agnostic framework for building flexible data science pipelines (Python/Shell/Gnuplot). Flowr - Robust and efficient workflows using a simple language agnostic approach (R package). Gwf - Make-like utility for submitting workflows via qsub. Hive - System for creating and running pipelines on a distributed compute resource. Joblib - Set of tools to provide lightweight pipelining in Python. Ketrew - Embedded DSL in the OCAML language alongside a client-server management application. Kronos - Workflow assembler for cancer genome analytics and informatics. Loom - Tool for running bioinformatics workflows locally or in the cloud. Longbow - Job proxying tool for biomolecular simulations. Luigi - Python module that helps you build complex pipelines of batch jobs. Makeflow - Workflow engine for executing large complex workflows on clusters. Mario - Scala library for defining data pipelines. Mistral - Python based workflow engine by the Open Stack project. Moa - Lightweight workflows in bioinformatics. Nextflow - Flow-based computational toolkit for reproducibile and scalable bioinformatics pipelines. NiPype - Workflows and interfaces for neuroimaging packages. OpenGE - Accelerated framework for manipulating and interpreting high-throughput sequencing data. PipEngine Ruby based launcher for complex biological pipelines. Pinball - Python based workflow engine by Pinterest. PyFlow - Lightweight parallel task engine. PypeFlow - Lightweight workflow engine for data analysis scripting. Pwrake - Parallel workflow extension for Rake. Qdo - Lightweight high-throughput queuing system for workflows with many small tasks to perform. Qsubsec - Simple tokenised template system for SGE. Rabix - Python-based workflow toolkit based on the Common Workflow Language and Docker. Remake - Make-like declarative workflows in R. Rmake - Wrapper for the creation of Makefiles, enabling massive parallelization. Rubra - Pipeline system for bioinformatics workflows. Ruffus - Computation Pipeline library for Python. Ruigi - Pipeline tool for R, inspired by Luigi. Sake - Self-documenting build automation tool. SciLuigi - Helper library for writing flexible scientific workflows in Luigi. Scoop - Scalable Concurrent Operations in Python. Snakemake - Tool for running and managing bioinformatics pipelines. Spiff - Based on the Workflow Patterns initiative and implemented in Python. Stpipe - File processing pipelines as a Python library. Suro - Java-based distributed pipeline from Netflix. Swift - Fast easy parallel scripting - on multicores, clusters, clouds and supercomputers. Toil - Distributed pipeline workflow manager (mostly for genomics). Yap - Extensible parallel framework, written in Python using OpenMPI libraries. WorldMake - Easy Collaborative Reproducible Computing. Workflow platforms ActivePapers - Computational science made reproducible and publishable. Apache Iravata - Framework for executing and managing computational workflows on distributed computing resources. Arvados - A container based workflow platform. Biokepler - Bioinformatics Scientific Workflow for Distributed Analysis of Large-Scale Biological Data. Chipster - Open source platform for data analysis. Cromwell - Workflow Management System geared towards scientific workflows from the Broad Institute. Fireworks - Centralized workflow server for dynamic workflows of high-throughput computations. Galaxy - Web-based platform for biomedical research. Kepler - Kepler scientific workflow application from University of California. NextflowWorkbench - Integrated development environment for Nextflow, Docker and Reusable Workflows. OpenMOLE - Workflow Management System for exploration of models and parameter optimization. Ophidia - Data-analytics platform with declarative workflows of distributed operations. Pegasus - Workflow Management System. Sushi - Supporting User for SHell script Integration. Yabi - Online research environment for grid, HPC and cloud computing. Taverna - Domain independent workflow system. VisTrails - Scientific workflow and provenance management system. Wings - Semantic workflow system utilizing Pegasus as execution system. Workflow languages Common Workflow Language Cloudgene Workflow Language OpenMOLE DSL Workflow Definition Language Yet Another Workflow Language Workflow standardization initiatives Workflow 4 Ever Initiative Workflow 4 Ever workflow research object model Workflow Patterns Initiative Workflow Patterns Library ResearchObject.org Literate programming (aka interactive notebooks) Beaker Notebook-style development environment. Binder - Turn a GitHub repo into a collection of interactive notebooks powered by Jupyter and Kubernetes IPython A rich architecture for interactive computing. Jupyter Language-agnostic notebook literate programming environment. Pathomx - Interactive data workflows built on Python. R Notebooks - R Markdown notebook literate programming environment. Wakari - Web-based Python Data Analysis. Zeppelin - Web-based notebook that enables interactive data analytics. Build automation tools Bazel - Build software just as engineers do at Google. DoIt - Highly generalized task-management and automation in Python. Gradle - Unified cross platforms builds. Scons - Python library focused on C/C++ builds. Shake - Define robust build systems akin to GNU Make using Haskell. Make - The GNU Make build system. Other projects HPC Grid Runner noWorkflow - Supporting infrastructure to run scientific experiments without a scientific workflow management system, and still get things like provenance. https://github.com/nextflow-io/awesome-nextflow
Part I: Pipeline,bothadopted in industrial assembly andprocessors, is used for performance enhancement or improvement. In addtion, in processors, superscalar, multi-issues, reorder buffer, cache, etc., are all employed to improve performance. Pipeline control logic: bypass unit --- forwaring, data dependency branch predict unit --- prediction, control dependency Part II: Two places for sharing: SlideShare: as it states, it is the best way to share presentations, documents and professional videos http://www.slideshare.net/ Zoho http://viewer.zoho.com/Upload.jsp After uploading what we want to share, we can get corresponding links for them and copy URLs into Google Translate and let the smart tool finish translating the resources into what language we are interested in.
标签: Pipeline
