orca fails when trying to save a heatmap

[ndrubins]

Hi,

I'm creating a heatmap image.
If I try to combine it with a column dendrogram, although it is viewable on the RStudio viewer, using orca to export it as pdf fails .

Here's an example where I build a sparse heatmap and add to it a column dendrogram:

library(ggplot2)
library(dplyr)
library(plotly)

#build the heatmap data.frame
row.text <- strsplit("ALLOGRAFT_REJECTION,APICAL_SURFACE,APOPTOSIS,EPITHELIAL_MESENCHYMAL_TRANSITION,ESTROGEN_RESPONSE_EARLY,HYPOXIA,IL2_STAT5_SIGNALING,IL6_JAK_STAT3_SIGNALING,INFLAMMATORY_RESPONSE,INTERFERON_ALPHA_RESPONSE,INTERFERON_GAMMA_RESPONSE,KRAS_SIGNALING_DN,KRAS_SIGNALING_UP,MYC_TARGETS_V1,P53_PATHWAY,PI3K_AKT_MTOR_SIGNALING,PROTEIN_SECRETION,TNFA_SIGNALING_VIA_NFKB,UV_RESPONSE_UP,ALLOGRAFT_REJECTION,APICAL_SURFACE,APOPTOSIS,EPITHELIAL_MESENCHYMAL_TRANSITION,ESTROGEN_RESPONSE_EARLY,HYPOXIA,IL2_STAT5_SIGNALING,IL6_JAK_STAT3_SIGNALING,INFLAMMATORY_RESPONSE,INTERFERON_ALPHA_RESPONSE,INTERFERON_GAMMA_RESPONSE,KRAS_SIGNALING_DN,KRAS_SIGNALING_UP,MYC_TARGETS_V1,P53_PATHWAY,PI3K_AKT_MTOR_SIGNALING,PROTEIN_SECRETION,TNFA_SIGNALING_VIA_NFKB,UV_RESPONSE_UP,ALLOGRAFT_REJECTION,APICAL_SURFACE,APOPTOSIS,EPITHELIAL_MESENCHYMAL_TRANSITION,ESTROGEN_RESPONSE_EARLY,HYPOXIA,IL2_STAT5_SIGNALING,IL6_JAK_STAT3_SIGNALING,INFLAMMATORY_RESPONSE,INTERFERON_ALPHA_RESPONSE,INTERFERON_GAMMA_RESPONSE,KRAS_SIGNALING_DN,KRAS_SIGNALING_UP,MYC_TARGETS_V1,P53_PATHWAY,PI3K_AKT_MTOR_SIGNALING,PROTEIN_SECRETION,TNFA_SIGNALING_VIA_NFKB,UV_RESPONSE_UP,ALLOGRAFT_REJECTION,APICAL_SURFACE,APOPTOSIS,EPITHELIAL_MESENCHYMAL_TRANSITION,ESTROGEN_RESPONSE_EARLY,HYPOXIA,IL2_STAT5_SIGNALING,IL6_JAK_STAT3_SIGNALING,INFLAMMATORY_RESPONSE,INTERFERON_ALPHA_RESPONSE,INTERFERON_GAMMA_RESPONSE,KRAS_SIGNALING_DN,KRAS_SIGNALING_UP,MYC_TARGETS_V1,P53_PATHWAY,PI3K_AKT_MTOR_SIGNALING,PROTEIN_SECRETION,TNFA_SIGNALING_VIA_NFKB,UV_RESPONSE_UP,ALLOGRAFT_REJECTION,APICAL_SURFACE,APOPTOSIS,EPITHELIAL_MESENCHYMAL_TRANSITION,ESTROGEN_RESPONSE_EARLY,HYPOXIA,IL2_STAT5_SIGNALING,IL6_JAK_STAT3_SIGNALING,INFLAMMATORY_RESPONSE,INTERFERON_ALPHA_RESPONSE,INTERFERON_GAMMA_RESPONSE,KRAS_SIGNALING_DN,KRAS_SIGNALING_UP,MYC_TARGETS_V1,P53_PATHWAY,PI3K_AKT_MTOR_SIGNALING,PROTEIN_SECRETION,TNFA_SIGNALING_VIA_NFKB,UV_RESPONSE_UP",split=",")[[1]]
col.text <- strsplit("APC.nnn.vs.mmmmm,APC.nnn.vs.mmmmm,APC.nnn.vs.mmmmm,APC.nnn.vs.mmmmm,APC.nnn.vs.mmmmm,APC.nnn.vs.mmmmm,APC.nnn.vs.mmmmm,APC.nnn.vs.mmmmm,APC.nnn.vs.mmmmm,APC.nnn.vs.mmmmm,APC.nnn.vs.mmmmm,APC.nnn.vs.mmmmm,APC.nnn.vs.mmmmm,APC.nnn.vs.mmmmm,APC.nnn.vs.mmmmm,APC.nnn.vs.mmmmm,APC.nnn.vs.mmmmm,APC.nnn.vs.mmmmm,APC.nnn.vs.mmmmm,B-cell.nnn.vs.mmmmm,B-cell.nnn.vs.mmmmm,B-cell.nnn.vs.mmmmm,B-cell.nnn.vs.mmmmm,B-cell.nnn.vs.mmmmm,B-cell.nnn.vs.mmmmm,B-cell.nnn.vs.mmmmm,B-cell.nnn.vs.mmmmm,B-cell.nnn.vs.mmmmm,B-cell.nnn.vs.mmmmm,B-cell.nnn.vs.mmmmm,B-cell.nnn.vs.mmmmm,B-cell.nnn.vs.mmmmm,B-cell.nnn.vs.mmmmm,B-cell.nnn.vs.mmmmm,B-cell.nnn.vs.mmmmm,B-cell.nnn.vs.mmmmm,B-cell.nnn.vs.mmmmm,B-cell.nnn.vs.mmmmm,T-cell.nnn.vs.mmmmm,T-cell.nnn.vs.mmmmm,T-cell.nnn.vs.mmmmm,T-cell.nnn.vs.mmmmm,T-cell.nnn.vs.mmmmm,T-cell.nnn.vs.mmmmm,T-cell.nnn.vs.mmmmm,T-cell.nnn.vs.mmmmm,T-cell.nnn.vs.mmmmm,T-cell.nnn.vs.mmmmm,T-cell.nnn.vs.mmmmm,T-cell.nnn.vs.mmmmm,T-cell.nnn.vs.mmmmm,T-cell.nnn.vs.mmmmm,T-cell.nnn.vs.mmmmm,T-cell.nnn.vs.mmmmm,T-cell.nnn.vs.mmmmm,T-cell.nnn.vs.mmmmm,T-cell.nnn.vs.mmmmm,Neutrophil.nnn.vs.mmmmm,Neutrophil.nnn.vs.mmmmm,Neutrophil.nnn.vs.mmmmm,Neutrophil.nnn.vs.mmmmm,Neutrophil.nnn.vs.mmmmm,Neutrophil.nnn.vs.mmmmm,Neutrophil.nnn.vs.mmmmm,Neutrophil.nnn.vs.mmmmm,Neutrophil.nnn.vs.mmmmm,Neutrophil.nnn.vs.mmmmm,Neutrophil.nnn.vs.mmmmm,Neutrophil.nnn.vs.mmmmm,Neutrophil.nnn.vs.mmmmm,Neutrophil.nnn.vs.mmmmm,Neutrophil.nnn.vs.mmmmm,Neutrophil.nnn.vs.mmmmm,Neutrophil.nnn.vs.mmmmm,Neutrophil.nnn.vs.mmmmm,Neutrophil.nnn.vs.mmmmm,Splenic-monocyte.nnn.vs.mmmmm,Splenic-monocyte.nnn.vs.mmmmm,Splenic-monocyte.nnn.vs.mmmmm,Splenic-monocyte.nnn.vs.mmmmm,Splenic-monocyte.nnn.vs.mmmmm,Splenic-monocyte.nnn.vs.mmmmm,Splenic-monocyte.nnn.vs.mmmmm,Splenic-monocyte.nnn.vs.mmmmm,Splenic-monocyte.nnn.vs.mmmmm,Splenic-monocyte.nnn.vs.mmmmm,Splenic-monocyte.nnn.vs.mmmmm,Splenic-monocyte.nnn.vs.mmmmm,Splenic-monocyte.nnn.vs.mmmmm,Splenic-monocyte.nnn.vs.mmmmm,Splenic-monocyte.nnn.vs.mmmmm,Splenic-monocyte.nnn.vs.mmmmm,Splenic-monocyte.nnn.vs.mmmmm,Splenic-monocyte.nnn.vs.mmmmm,Splenic-monocyte.nnn.vs.mmmmm",split=",")[[1]]
vals <- as.numeric(strsplit("1.82390874094432,NA,NA,1.72124639904717,NA,1.69897000433602,1.34678748622466,NA,2.74472749489669,3.76955107862173,3.76955107862173,NA,1.72124639904717,1.3767507096021,1.3767507096021,NA,NA,3.76955107862173,1.72124639904717,NA,NA,NA,NA,NA,NA,NA,NA,NA,3.60205999132796,3.60205999132796,NA,NA,NA,NA,NA,NA,NA,NA,2.20760831050175,NA,2.20760831050175,NA,NA,NA,NA,NA,NA,3.30102999566398,1.69897000433602,1.48148606012211,NA,NA,NA,NA,NA,NA,NA,2.72124639904717,NA,3.60205999132796,2.36653154442041,1.55284196865778,3.13667713987954,2.46852108295775,2.74472749489669,3.13667713987954,2.72124639904717,2.46852108295775,2.29242982390206,NA,NA,NA,2.72124639904717,1.3767507096021,3.60205999132796,NA,NA,1.30980391997149,NA,NA,NA,1.30980391997149,NA,NA,NA,1.79588001734408,NA,NA,NA,NA,NA,NA,NA,NA,1.30980391997149",split=",")[[1]])
plot.df <- data.frame(row = row.text, column = col.text, value = vals)

#build the column dendrogram, via clustering
plot.mat <- reshape2::acast(plot.df,column ~ row)
col.hc <- cluster::agnes(plot.mat,diss=FALSE,method="complete")
sorted.col.hc <- dendsort::dendsort(as.hclust(col.hc))
col.dend <- as.dendrogram(sorted.col.hc)
col.dendex <- dendextend::as.ggdend(col.dend)
leaf.heights <- dplyr::filter(col.dendex$nodes,!is.na(leaf))$height
leaf.xs <- dplyr::filter(col.dendex$nodes,!is.na(leaf))$x
leaf.seqments.idx <- which(col.dendex$segments$yend %in% leaf.heights & col.dendex$segments$x %in% leaf.xs)
col.dendex$segments$yend[leaf.seqments.idx] <- max(col.dendex$segments$yend[leaf.seqments.idx])
col.dendex$segments$col[leaf.seqments.idx] <- "black"
col.dendex$labels$label <- ""
col.dendex$labels$y <- max(col.dendex$segments$yend[leaf.seqments.idx])
col.dendex$labels$x <- col.dendex$segments$x[leaf.seqments.idx]
col.dendex$labels$col <- "black"
col.dendex$segments$lwd <- 0.5
col.ggdend <- ggplot(col.dendex,labels=F)+guides(fill=F)+theme_minimal()+
  theme(axis.title=element_blank(),axis.text=element_blank(),axis.ticks=element_blank(),panel.grid=element_blank(),legend.position="none",legend.text=element_blank(),legend.background=element_blank(),legend.key=element_blank())

legend.title <- "-log10(Q-value)"
color.vec <- c("black","yellow")

#build the heatmap
p <- plot_ly(z=c(plot.df$value),x=plot.df$column,y=plot.df$row,colors=grDevices::colorRamp(color.vec),type="heatmap",colorbar=list(title=legend.title,len=0.4)) %>%
  layout(yaxis=list(title=NULL),xaxis=list(title=NULL))

#add the dendrogram
p <- plotly::subplot(col.ggdend,plotly::plotly_empty(),p %>% plotly::layout(showlegend = F),nrows=2,margin=c(0,0,0,0),heights=c(0.2,0.8),widths=c(0.8,0.2))

As mentioned above, I'm able to view the image of the heatmap + dendrogram on the RStudio viewer. However, when I try to save it to a pdf using:

plotly::orca(p, file="p.pdf")
I get the error:

done with code 1 in 41.38 sec - failed or incomplete task(s)
Error in processx::run("orca", args, echo = TRUE, spinner = TRUE, ...) : 
  System command error

The failure of exporting a heatmap to a pdf using orca is not specific to the example above though.

In the example below I'm creating a big heatmap (1862 x 2511) with long row and column labels and it also fails:

The heatmap is facetted by columns, so first I create a data.frame fr guiding the facets:

set.seed(1)
facet.df <- data.frame(facet = 1:20,
                       n.cols = c(38,32,78,59,303,417,77,119,18,14,262,351,24,11,13,13,175,182,198,127),
                       x.tick.vals = c(296,248.5,616.5,463,2404,3316.5,608,941.5,135.5,103.5,2081.5,2791,191,86,100.5,100.5,1388.5,1447.5,1573,1003.5),stringsAsFactors = F)

row.labels <- paste0("RRRRR",1:1862)
col.labels <- paste0(paste(sample(c("A","C","G","T"),16,replace=T),collapse=""),"-1_aaa.",1:2511)

Now I create a list of heatmap plots, one per each facet:

plot.list <- lapply(1:nrow(facet.df),function(f){
  if(f > 1){
    x.vals <- col.labels[(sum(dplyr::filter(facet.df,facet <= f-1)$n.cols)+1):sum(dplyr::filter(facet.df,facet <= f)$n.cols)]
  } else{
    x.vals <- col.labels[1:facet.df$n.cols[f]]
  }
  facet.heatmap.df <- data.frame(value = rnorm(1862*facet.df$n.cols[f]), x = unlist(lapply(x.vals,function(x) rep(x,length(row.labels)))), y = rep(row.labels,facet.df$n.cols[f]),stringsAsFactors = F)
  plot_ly(z = c(facet.heatmap.df$value), x = facet.heatmap.df$x, y = facet.heatmap.df$y, colors = grDevices::colorRamp(c("cyan","black","yellow")), type = "heatmap",colorbar = list(title="expr",len=0.4)) %>%
    layout(yaxis=list(title=NULL),xaxis=list(tickangle=90,tickvals=facet.df$x.tick.vals[f]))
})

Now I combine the facet plots to a single plot:

heatmap.plot <- plotly::subplot(plot.list,shareX=T,shareY=T,nrows=1,margin=0.001,widths=rep(1/nrow(facet.df),nrow(facet.df))) %>% layout(showlegend=F)
Finally, trying to export the plot using orca:
`
plotly::orca(heatmap.plot %>% plotly::style(hoverinfo = 'none'),file="heatmap.plot.pdf")

`

Fails with:

done with code 1 in 204.44 ms - failed or incomplete task(s)
Error in processx::run("orca", args, echo = TRUE, spinner = TRUE, ...) : 
  System command error

Thanks a lot.

[Haunfelder]

The system command error from processx::run likely means that orca is failing for some reason, which could be due to permissions. Can you try running the below code and let me know the output?

system("orca")
system("where orca")

[ndrubins]

Sure.

For system("orca") I get:

Plotly's image-exporting utilities

  Usage: orca [--version] [--help] <command> [<args>]

  Available commands:
  - graph [or plotly-graph, plotly_graph]
    Generates an image of plotly graph from inputted plotly.js JSON attributes.
    For more info, run `orca graph --help`.
  - serve [or server]
    Boots up a server with one route per available export component
    For more info, run `orca serve --help`.

where orca is not defined:

> system("where orca")
sh: where: command not found
Warning message:
In system("where orca") : error in running command

Perhaps which orca helps:

> system("which orca")
/usr/local/bin/orca

[Haunfelder]

It looks like orca is set up properly from your output, what if you tried the following from the unix command line,

orca graph '{ "data": [{"y": [1,2,1]}] }' -o fig.png

You can also use the system() function to run the same from R if you would like. Let me know if that works with no errors.

[cpsievert]

@ndrubins I was able to replicate your issue.

(@Haunfelder I don't think this has anything to do with configuring the orca CLI).

This is gonna be a difficult issue to track down, but just FYI, you can specify debug = T in orca() to get some more info:

export error 422 for heatmap.plot - json parse error
  undefined

I might not have the bandwidth to investigate further. If you wanted, you could report an issue over at https://github.com/plotly/orca or https://github.com/plotly/plotly.js. If you do, first use plotly_json() to write the relevant JSON to file:

R> plotly_json(heatmap.plot, FALSE) %>% cat(file = "graph.json")

Then provide a link to the JSON and a shell command to replicate the issue:

shell> orca graph graph.json -o fig.pdf --debug

[ndrubins]

Thanks a lot @cpsievert.
Just to confirm, indeed orca graph '{ "data": [{"y": [1,2,1]}] }' -o fig.png from my command land does not throw any error message.

I'll follow your advice and report an issue at (https://github.com/plotly/orca) or (https://github.com/plotly/plotly.js).