systemPipeShiny

Daniela Cassol & Le Zhang

University of California Riverside

2020/08/28 (updated: 2020-08-31)

1 / 43

OutlineIntroductionsystemPipeShiny DemoVisualization FeaturesCustomize and Extend2 / 43

Introduction3 / 43

Motivation

Build an interactive framework for workflow management and visualization by extending all systemPipeR functionalities

4 / 43

Motivation

Build an interactive framework for workflow management and visualization by extending all systemPipeR functionalities

5 / 43

Motivation

Build an interactive framework for workflow management and visualization by extending all systemPipeR functionalities

6 / 43

Motivation

Build an interactive framework for workflow management and visualization by extending all systemPipeR functionalities

7 / 43

Motivation

Build an interactive framework for workflow management and visualization by extending all systemPipeR functionalities

8 / 43

Motivation

Build an interactive framework for workflow management and visualization by extending all systemPipeR functionalities

Provide for non-R users, such as experimentalists, to run many systemPipeR’s workflow designs, control, and visualization functionalities interactively without requiring knowledge of R

Provide a tool that can be used on both local computers as well as centralized server-based deployments that can be accessed remotely as a public web service for using SPR’s functionalities with the community and/or private data

9 / 43

Features

10 / 43

Interactively define experimental designs and provide associated metadata using an easy to use tabular editor and/or file uploader

visualize workflow topologies combined with auto-generation of R Markdown previews for interactively designed workflows

Allows prepare the data for the visualizations tabs

 Structural Features11 / 43

Structural Features

User friendly

12 / 43

Structural Features

User friendly

Progress Tracking

13 / 43

Structural Features

User friendly

Progress Tracking

Canvas

Under this workbench users can take snapshots of different plots, and combine or resize them. This feature is useful for generating complex scientific summary graphics.

drawing

14 / 43

Structural Features

Progress Tracking

Messages, warnings and errors from R functions are automatically captured and logged on both the server and client ends.

drawing

15 / 43

Structural Features

Progress Tracking

App Config

a robust exception handling system has been implemented (similar to Shiny options), that provides error solutions to to users, e.g. invalid parameter settings.

16 / 43

Structural Features

Progress Tracking

App Config

Modular Design

SPS is built on Shiny modules, which provides local scope isolation between each tab. Objects on one tab do not conflict with other tabs. To enable cross-tab communication, SPS also supports global scope interactions

17 / 43

Design

18 / 43

Customize and Extend

Templates are available to generate new visualization tabs

19 / 43

 Live Demo20 / 43

Install Package

Install the systemPipeShiny package from GitHub:

if (!requireNamespace("BiocManager", quietly=TRUE))
    install.packages("BiocManager")
BiocManager::install("systemPipeR/systemPipeShiny", dependencies=TRUE, build_vignettes=TRUE)

Load Package and Documentation

Load packages and accessing help

library("systemPipeShiny")

Access help

library(help="systemPipeShiny")
vignette("systemPipeShiny")

21 / 43

Quick Start

Create the project:

systemPipeShiny::spsInit()
## [SPS-INFO] 2020-08-30 17:03:28 Start to create a new SPS project
## [SPS-INFO] 2020-08-30 17:03:28 Create project under /home/dcassol/danielac@ucr.edu/projects/Presentations/SPS/SPS_20200830
## [SPS-INFO] 2020-08-30 17:03:28 Now copy files
## [SPS-INFO] 2020-08-30 17:03:28 Create SPS database
## [SPS-INFO] 2020-08-30 17:03:28 Created SPS database method container
## [SPS-INFO] 2020-08-30 17:03:28 Creating SPS db...
## [SPS-DANGER] 2020-08-30 17:03:28 Db created at '/home/dcassol/danielac@ucr.edu/projects/Presentations/SPS/SPS_20200830/config/sps.db'. DO NOT share this file with others
## [SPS-INFO] 2020-08-30 17:03:28 Key md5 dc17b12b7cadbb70e2d32a77bb32a17f
## [SPS-INFO] 2020-08-30 17:03:28 SPS project setup done!

22 / 43

SPS Folder Structure

Directory structure:

SPS_YYYYMMDD
├── server.R
├── ui.R
├── global.R ## It will need manual input for new tabs
├── deploy.R ## Deploy helper file
├── config/ ## Folder with app configuration files
│   ├── sps.db             
│   ├── sps_options.yaml   
│   └── tabs.csv  
├── R/ ## All SPS additional tab files and helper R function files
│   └── tabs_xx.R 
├── data/ ## Storage all the input data
│   └── inputData 
├── results/ ## Storage all the results and plot data
│   └── plot_xx.png 
└── www/ ## Folder with all the app resources

23 / 43

Launching the interface

The runApp function from shiny package launches the app in our browser.

shiny::runApp()

Check out our instance of systemPipeShiny: Link

Note: Add iframe here

24 / 43

 Visualization Features25 / 43

Data transformations and visualization

targetspath <- system.file("extdata", "targets.txt", package = "systemPipeR")
targets <- read.delim(targetspath, comment = "#")
cmp <- systemPipeR::readComp(file = targetspath, format = "matrix", delim = "-")
countMatrixPath <- system.file("extdata", "countDFeByg.xls", package = "systemPipeR")
countMatrix <- read.delim(countMatrixPath, row.names = 1)
exploreDDSplot(countMatrix, targets, cmp = cmp[[1]], preFilter = NULL,
    samples = c(3, 4), savePlot = TRUE, filePlot = "transf.png")

26 / 43

Heatmap

Samples

exploredds <- exploreDDS(countMatrix, targets, cmp=cmp[[1]], preFilter=NULL, transformationMethod="rlog")
heatMaplot(exploredds, clust="samples")
heatMaplot(exploredds, clust="samples", plotly = TRUE)

Individuals genes identified in DEG analysis

### DEG analysis with `systemPipeR`
degseqDF <- systemPipeR::run_DESeq2(countDF = countMatrix, targets = targets, cmp = cmp[[1]], independent = FALSE)
DEG_list <- systemPipeR::filterDEGs(degDF = degseqDF, filter = c(Fold = 2, FDR = 10))
heatMaplot(exploredds, clust="ind", DEGlist = unique(as.character(unlist(DEG_list[[1]]))))
heatMaplot(exploredds, clust="ind", DEGlist = unique(as.character(unlist(DEG_list[[1]]))), plotly = TRUE)

27 / 43

Dendrogram

exploredds <- exploreDDS(countMatrix, targets, cmp=cmp[[1]], preFilter=NULL, transformationMethod="rlog")
hclustplot(exploredds, method = "spearman")
hclustplot(exploredds, method = "spearman", savePlot = TRUE, filePlot = "cor.pdf")

28 / 43

PCA plot

library(systemPipeShiny)
targetspath <- system.file("extdata", "targets.txt", package="systemPipeR")
targets <- read.delim(targetspath, comment="#")
cmp <- systemPipeR::readComp(file=targetspath, format="matrix", delim="-")
countMatrixPath <- system.file("extdata", "countDFeByg.xls", package="systemPipeR")
countMatrix <- read.delim(countMatrixPath, row.names=1)
exploredds <- exploreDDS(countMatrix, targets, cmp=cmp[[1]], preFilter=NULL, transformationMethod="rlog")
PCAplot(exploredds, plotly = TRUE)

29 / 43

Multidimensional Scaling (MDS) plot

exploredds <- exploreDDS(countMatrix, targets, cmp=cmp[[1]], preFilter=NULL, transformationMethod="rlog")
MDSplot(exploredds, plotly = FALSE)

30 / 43

Generalized Principal Components Analysis

exploredds <- exploreDDS(countMatrix, targets, cmp=cmp[[1]], preFilter=NULL, transformationMethod="raw")
GLMplot(exploredds, plotly = FALSE)
GLMplot(exploredds, plotly = FALSE, savePlot = TRUE, filePlot = "GML.pdf")

31 / 43

Bland–Altman Plot (MA-Plot)

exploredds <- exploreDDS(countMatrix, targets, cmp=cmp[[1]], preFilter=NULL, transformationMethod="raw")
GLMplot(exploredds, plotly = FALSE)
GLMplot(exploredds, plotly = FALSE, savePlot = TRUE, filePlot = "GML.pdf")

32 / 43

t-SNE Plot

targetspath <- system.file("extdata", "targets.txt", package="systemPipeR")
targets <- read.delim(targetspath, comment="#")
cmp <- systemPipeR::readComp(file=targetspath, format="matrix", delim="-")
countMatrixPath <- system.file("extdata", "countDFeByg.xls", package="systemPipeR")
countMatrix <- read.delim(countMatrixPath, row.names=1)
set.seed(42) ## Set a seed if you want reproducible results
tSNEplot(countMatrix, targets, perplexity = 5)

33 / 43

Volcano Plot

degseqDF <- systemPipeR::run_DESeq2(countDF = countMatrix, targets = targets, cmp = cmp[[1]], independent = FALSE)
DEG_list <- systemPipeR::filterDEGs(degDF = degseqDF, filter = c(Fold = 2, FDR = 10))
volcanoplot(degseqDF, comparison = "M12-A12", filter = c(Fold = 2, FDR = 10))
volcanoplot(degseqDF, comparison = "M12-A12", filter = c(Fold = 1, FDR = 20), genes = "ATCG00280")

34 / 43

 Customize and Extend35 / 43

Development of New Tabs

Create new template tabs with newTabData and newTabPlot.

newTabData(
    tab_id = "data_new", 
    tab_displayname = "my first data tab",
    prepro_methods = list(makePrepro(label = "do nothing",
                                     plot_options = "plot_new"))
)
newTabPlot(
    tab_id = "plot_new",
    tab_displayname = "my first plot tab",
    plot_data = list(makePlotData(dataset_label = "Data from my new tab",
                                  receive_datatab_ids = "data_new"))
)

36 / 43

SPS Folder Structure

Directory structure with the new tabs:

SPS_YYYYMMDD
├── server.R
├── ui.R
├── global.R ## Requires actions!
├── deploy.R 
├── config/ 
│   ├── sps.db             
│   ├── sps_options.yaml   
│   └── tabs.csv ## It will be automatically edit
├── R/
│   ├── tab_vs_data_new.R   
│   └── tabs_vs_plot_new.R
├── data/ 
│   └── inputData 
├── results/ 
│   └── plot_xx.png 
└── www/

37 / 43

Development of New Tabs

Add new tabs on global.R file:

sps_app <- sps(
    vstabs = c("data_new", "plot_new"), # add new tab IDs here
    server_expr = {
        msg("Custom expression runs -- Hello World: checking", "GREETING", "green")
    }
)

Launch the app again:

shiny::runApp()

38 / 43

systemPipeR Team


Daniela Cassol	Le Zhang	Ponmathi Ramasamy	Thomas Girke

39 / 43

Thanks!

Browse source code at

Ask a question about systemPipeShiny at Bioconductor Support Page

systemPipeR at Bioconductor

https://systempipe.org/

40 / 43

 ToDo List41 / 43

 DoingDocumentation: Vignette
Complete the visualizations tabs with the plots controls 
Unit Testing
Add Quality plots
Add Functional Enrichment Analysis plots
Final deployment (shinyapp.io) and Review package
Submission deadline
42 / 43

Functional Enrichment Analysis plots

## Still working

43 / 43

Help

Keyboard shortcuts

↑, ←, Pg Up, k

Go to previous slide

↓, →, Pg Dn, Space, j

Go to next slide

Home

Go to first slide

End

Go to last slide

Number + Return

Go to specific slide

b / m / f

Toggle blackout / mirrored / fullscreen mode

Clone slideshow

Toggle presenter mode

Restart the presentation timer

?, h

Toggle this help

Tile View: Overview of Slides

systemPipeShiny

Daniela Cassol & Le Zhang

University of California Riverside

2020/08/28 (updated: 2020-08-31)

1 / 43

OutlineIntroductionsystemPipeShiny DemoVisualization FeaturesCustomize and Extend2 / 43

Introduction3 / 43

Motivation

Build an interactive framework for workflow management and visualization by extending all systemPipeR functionalities

4 / 43

Motivation

Build an interactive framework for workflow management and visualization by extending all systemPipeR functionalities

5 / 43

Motivation

Build an interactive framework for workflow management and visualization by extending all systemPipeR functionalities

6 / 43

Motivation

Build an interactive framework for workflow management and visualization by extending all systemPipeR functionalities

7 / 43

Motivation

Build an interactive framework for workflow management and visualization by extending all systemPipeR functionalities

8 / 43

Motivation

Build an interactive framework for workflow management and visualization by extending all systemPipeR functionalities

Provide for non-R users, such as experimentalists, to run many systemPipeR’s workflow designs, control, and visualization functionalities interactively without requiring knowledge of R

9 / 43

Features

10 / 43

Interactively define experimental designs and provide associated metadata using an easy to use tabular editor and/or file uploader

visualize workflow topologies combined with auto-generation of R Markdown previews for interactively designed workflows

Allows prepare the data for the visualizations tabs

 Structural Features11 / 43

Structural Features

User friendly

12 / 43

Structural Features

User friendly

Progress Tracking

13 / 43

Structural Features

User friendly

Progress Tracking

Canvas

Under this workbench users can take snapshots of different plots, and combine or resize them. This feature is useful for generating complex scientific summary graphics.

drawing

14 / 43

Structural Features

Progress Tracking

Messages, warnings and errors from R functions are automatically captured and logged on both the server and client ends.

drawing

15 / 43

Structural Features

Progress Tracking

App Config

a robust exception handling system has been implemented (similar to Shiny options), that provides error solutions to to users, e.g. invalid parameter settings.

16 / 43

Structural Features

Progress Tracking

App Config

Modular Design

17 / 43

Design

18 / 43

Customize and Extend

Templates are available to generate new visualization tabs

19 / 43

 Live Demo20 / 43

Install Package

Install the systemPipeShiny package from GitHub:

if (!requireNamespace("BiocManager", quietly=TRUE))
    install.packages("BiocManager")
BiocManager::install("systemPipeR/systemPipeShiny", dependencies=TRUE, build_vignettes=TRUE)

Load Package and Documentation

Load packages and accessing help

library("systemPipeShiny")

Access help

library(help="systemPipeShiny")
vignette("systemPipeShiny")

21 / 43

Quick Start

Create the project:

systemPipeShiny::spsInit()
## [SPS-INFO] 2020-08-30 17:03:28 Start to create a new SPS project
## [SPS-INFO] 2020-08-30 17:03:28 Create project under /home/dcassol/danielac@ucr.edu/projects/Presentations/SPS/SPS_20200830
## [SPS-INFO] 2020-08-30 17:03:28 Now copy files
## [SPS-INFO] 2020-08-30 17:03:28 Create SPS database
## [SPS-INFO] 2020-08-30 17:03:28 Created SPS database method container
## [SPS-INFO] 2020-08-30 17:03:28 Creating SPS db...
## [SPS-DANGER] 2020-08-30 17:03:28 Db created at '/home/dcassol/danielac@ucr.edu/projects/Presentations/SPS/SPS_20200830/config/sps.db'. DO NOT share this file with others
## [SPS-INFO] 2020-08-30 17:03:28 Key md5 dc17b12b7cadbb70e2d32a77bb32a17f
## [SPS-INFO] 2020-08-30 17:03:28 SPS project setup done!

22 / 43

SPS Folder Structure

Directory structure:

SPS_YYYYMMDD
├── server.R
├── ui.R
├── global.R ## It will need manual input for new tabs
├── deploy.R ## Deploy helper file
├── config/ ## Folder with app configuration files
│   ├── sps.db             
│   ├── sps_options.yaml   
│   └── tabs.csv  
├── R/ ## All SPS additional tab files and helper R function files
│   └── tabs_xx.R 
├── data/ ## Storage all the input data
│   └── inputData 
├── results/ ## Storage all the results and plot data
│   └── plot_xx.png 
└── www/ ## Folder with all the app resources

23 / 43

Launching the interface

The runApp function from shiny package launches the app in our browser.

shiny::runApp()

Check out our instance of systemPipeShiny: Link

Note: Add iframe here

24 / 43

 Visualization Features25 / 43

Data transformations and visualization

targetspath <- system.file("extdata", "targets.txt", package = "systemPipeR")
targets <- read.delim(targetspath, comment = "#")
cmp <- systemPipeR::readComp(file = targetspath, format = "matrix", delim = "-")
countMatrixPath <- system.file("extdata", "countDFeByg.xls", package = "systemPipeR")
countMatrix <- read.delim(countMatrixPath, row.names = 1)
exploreDDSplot(countMatrix, targets, cmp = cmp[[1]], preFilter = NULL,
    samples = c(3, 4), savePlot = TRUE, filePlot = "transf.png")

26 / 43

Heatmap

Samples

exploredds <- exploreDDS(countMatrix, targets, cmp=cmp[[1]], preFilter=NULL, transformationMethod="rlog")
heatMaplot(exploredds, clust="samples")
heatMaplot(exploredds, clust="samples", plotly = TRUE)

Individuals genes identified in DEG analysis

### DEG analysis with `systemPipeR`
degseqDF <- systemPipeR::run_DESeq2(countDF = countMatrix, targets = targets, cmp = cmp[[1]], independent = FALSE)
DEG_list <- systemPipeR::filterDEGs(degDF = degseqDF, filter = c(Fold = 2, FDR = 10))
heatMaplot(exploredds, clust="ind", DEGlist = unique(as.character(unlist(DEG_list[[1]]))))
heatMaplot(exploredds, clust="ind", DEGlist = unique(as.character(unlist(DEG_list[[1]]))), plotly = TRUE)

27 / 43

Dendrogram

exploredds <- exploreDDS(countMatrix, targets, cmp=cmp[[1]], preFilter=NULL, transformationMethod="rlog")
hclustplot(exploredds, method = "spearman")
hclustplot(exploredds, method = "spearman", savePlot = TRUE, filePlot = "cor.pdf")

28 / 43

PCA plot

library(systemPipeShiny)
targetspath <- system.file("extdata", "targets.txt", package="systemPipeR")
targets <- read.delim(targetspath, comment="#")
cmp <- systemPipeR::readComp(file=targetspath, format="matrix", delim="-")
countMatrixPath <- system.file("extdata", "countDFeByg.xls", package="systemPipeR")
countMatrix <- read.delim(countMatrixPath, row.names=1)
exploredds <- exploreDDS(countMatrix, targets, cmp=cmp[[1]], preFilter=NULL, transformationMethod="rlog")
PCAplot(exploredds, plotly = TRUE)

29 / 43

Multidimensional Scaling (MDS) plot

exploredds <- exploreDDS(countMatrix, targets, cmp=cmp[[1]], preFilter=NULL, transformationMethod="rlog")
MDSplot(exploredds, plotly = FALSE)

30 / 43

Generalized Principal Components Analysis

exploredds <- exploreDDS(countMatrix, targets, cmp=cmp[[1]], preFilter=NULL, transformationMethod="raw")
GLMplot(exploredds, plotly = FALSE)
GLMplot(exploredds, plotly = FALSE, savePlot = TRUE, filePlot = "GML.pdf")

31 / 43

Bland–Altman Plot (MA-Plot)

exploredds <- exploreDDS(countMatrix, targets, cmp=cmp[[1]], preFilter=NULL, transformationMethod="raw")
GLMplot(exploredds, plotly = FALSE)
GLMplot(exploredds, plotly = FALSE, savePlot = TRUE, filePlot = "GML.pdf")

32 / 43

t-SNE Plot

targetspath <- system.file("extdata", "targets.txt", package="systemPipeR")
targets <- read.delim(targetspath, comment="#")
cmp <- systemPipeR::readComp(file=targetspath, format="matrix", delim="-")
countMatrixPath <- system.file("extdata", "countDFeByg.xls", package="systemPipeR")
countMatrix <- read.delim(countMatrixPath, row.names=1)
set.seed(42) ## Set a seed if you want reproducible results
tSNEplot(countMatrix, targets, perplexity = 5)

33 / 43

Volcano Plot

degseqDF <- systemPipeR::run_DESeq2(countDF = countMatrix, targets = targets, cmp = cmp[[1]], independent = FALSE)
DEG_list <- systemPipeR::filterDEGs(degDF = degseqDF, filter = c(Fold = 2, FDR = 10))
volcanoplot(degseqDF, comparison = "M12-A12", filter = c(Fold = 2, FDR = 10))
volcanoplot(degseqDF, comparison = "M12-A12", filter = c(Fold = 1, FDR = 20), genes = "ATCG00280")

34 / 43

 Customize and Extend35 / 43

Development of New Tabs

Create new template tabs with newTabData and newTabPlot.

newTabData(
    tab_id = "data_new", 
    tab_displayname = "my first data tab",
    prepro_methods = list(makePrepro(label = "do nothing",
                                     plot_options = "plot_new"))
)
newTabPlot(
    tab_id = "plot_new",
    tab_displayname = "my first plot tab",
    plot_data = list(makePlotData(dataset_label = "Data from my new tab",
                                  receive_datatab_ids = "data_new"))
)

36 / 43

SPS Folder Structure

Directory structure with the new tabs:

SPS_YYYYMMDD
├── server.R
├── ui.R
├── global.R ## Requires actions!
├── deploy.R 
├── config/ 
│   ├── sps.db             
│   ├── sps_options.yaml   
│   └── tabs.csv ## It will be automatically edit
├── R/
│   ├── tab_vs_data_new.R   
│   └── tabs_vs_plot_new.R
├── data/ 
│   └── inputData 
├── results/ 
│   └── plot_xx.png 
└── www/

37 / 43

Development of New Tabs

Add new tabs on global.R file:

sps_app <- sps(
    vstabs = c("data_new", "plot_new"), # add new tab IDs here
    server_expr = {
        msg("Custom expression runs -- Hello World: checking", "GREETING", "green")
    }
)

Launch the app again:

shiny::runApp()

38 / 43

systemPipeR Team


Daniela Cassol	Le Zhang	Ponmathi Ramasamy	Thomas Girke

39 / 43

Thanks!

Browse source code at

Ask a question about systemPipeShiny at Bioconductor Support Page

systemPipeR at Bioconductor

https://systempipe.org/

40 / 43

 ToDo List41 / 43

 DoingDocumentation: Vignette
Complete the visualizations tabs with the plots controls 
Unit Testing
Add Quality plots
Add Functional Enrichment Analysis plots
Final deployment (shinyapp.io) and Review package
Submission deadline
42 / 43

Functional Enrichment Analysis plots

## Still working

43 / 43