Should webGL mode be backed by a framebuffer?

[aferriss]

Should webGL mode be backed by a framebuffer (or p5.Graphics object)?

I was thinking a bit about webGL mode and how we might achieve better parity with the 2d renderer. Currently, webGL mode renders content directly to the canvas. If instead we rendered to a framebuffer, we could use the main scene as a texture, and add a post processing stage, which is what is actually drawn to the screen. In that post processing stage, it would be trivial to write shaders that perform the filter functions we have set up in 2d mode. There might be other benefits we haven't considered as well.

One drawback from this, is that webGL1 framebuffers do not support multisampling, so geometry would look a bit aliased, unless we implement some kind of multisampling or antialiasing ourselves (which could be controlled with noSmooth() / smooth()). webGL2 does support MSAA, but I'm not sure if we're ready to take the plunge into webGL2 yet. If we used an additional p5.Graphics layer, it could be slightly less performant, but wouldn't face the anti-aliasing issues.

I haven't fully thought this through, but there it's likely that there's other benefits and drawbacks to this strategy as well. Thoughts?

[Qianqianye]

@davepagurek recently released a p5.js library called p5.Framebuffer. Maybe @davepagurek can share some thoughts on this?
Also inviting two GSoC contributors to this discussion who will work on webGL related issues this year: @aceslowman mentored by @kjhollen, @ShenpaiSharma mentored by @calebfoss. Any thoughts? Thank you!

[davepagurek]

Hi! I'd love to see first-class support for framebuffers in p5. I could also see a world where p5.Graphics is backed by framebuffers instead of separate canvases for better performance, similar to how openFrameworks functions. The main drawback I can see to that farther-future vision is backwards compatibility with p5.Graphics, since you wouldn't be able to do things like interleave draw calls to multiple graphics (you would need to switch the active framebuffer before each one).

In any case, I think switching the main canvas to a framebuffer would be a great first step towards that. In addition to the benefits you've listed, it would also let us:

• Add in a depth texture, which the p5.Framebuffer library examples use to do depth of field blur shaders. I would love to be able to have built-in postprocessing filters for blurs and even screen-space ambient occlusion shadows
• Optionally use floating point textures. I've talked to some artists wanting to use those for feedback shaders, since the limited precision starts to add artifacts over time otherwise

Although MSAA is not supported in WebGL 1, here are some workarounds I can think of:

• We could multiply the pixel density by 2. This would render four pixels per pixel, and although the grid pattern might cause some artifacts that the MSAA implementation might not have, it's probably better than nothing.
• We could implement a screen-space antialiasing solution as a postprocessing filter, e.g. https://en.wikipedia.org/wiki/Fast_approximate_anti-aliasing
• WebGL2 is pretty backwards compatible to WebGL1. We could feasibly fall back to e.g. doubling the pixel density if we can't grab a WebGL2 context and setup MSAA otherwise

I'm also happy to discuss more/contribute if any of the GSoC people want to work on this 🙂

[Qianqianye]

Thanks @davepagurek. I'm inviting more WebGL stewards to this discussion @stalgiag @jeffawang @AdilRabbani

[davepagurek]

Adding some more thoughts: I feel like the biggest benefit of using framebuffers and shaders where possible instead of separate canvases and loadPixels/updatePixels is performance. Even if it introduces no new features, I think it matters because performance seems like part of accessibility to me. Some cool sketches are currently just not a good experience on older or cheaper hardware (especially things like the Chromebooks used in schools.) If we can make it so p5 code performs as well as possible, we're letting all those people experience p5 closer to the way people on new computers do. I'm not sure if that's part of the definition of access that p5 usually uses, but that's the lens I'm using when thinking about the tradeoffs anyway!

[stalgiag]

I agree that performance is an important part of accessibility.

Do you imagine the interface changing in any way?

Would the performance updates in this proposal only apply to sketcches where p5.Graphics objects are being used to manage shader passes?

Do you see any features other than filter made possible by this proposal that bring it closer to parity with 2D or Processing OpenGL?

[davepagurek]

Do you imagine the interface changing in any way?

For a fast filter() implementation, I think we can keep the same interface.

For the farther future thing where graphics are backed by framebuffers on one canvas, ideally it would also use the same interface, but I'd need to think a bit harder to make sure that's possible. I don't think it's possible to preserve some operations while using framebuffers as graphics (like drawing a webgl graphic to a 2D graphic), so that would take some further discussion.

Would the performance updates in this proposal only apply to sketcches where p5.Graphics objects are being used to manage shader passes?

For the filter() implementation, yes, a framebuffer would just speed up filter shader passes.

If webgl graphics are backed by framebuffers too, then all multi-graphic drawing in webgl mode would get a speedup, shaders or no shaders.

Do you see any features other than filter made possible by this proposal that bring it closer to parity with 2D or Processing OpenGL?

I think that's it for core feature parity.

[stalgiag]

Thanks for that!

My reluctance is that this is a pretty large change if it is just useful in getting filter() working in WebGL. If we think that this large change is worth it because it opens up a larger set of possibilities for GLSL enjoyers using p5 then I am all for it.

I think that p5 WebGL serves a few common purposes currently: accessible aha moment transitioning from 2D to 3D for new coders that aren't ready to commit to three.js, simple 3D interactions in canvas-based web design, and people interested in shader programming. I fully support making changes to the renderer that make it more accessible to that last group.

[davepagurek]

Agreed, it's probably too big of a jump doing everything with framebuffers for now. How would you feel about first reaching parity with Processing for using filter in WebGL mode without using any framebuffers? Trying to improve its performance with just one framebuffer (not touching all graphics for now) could be a follow-up

[davepagurek]

I recently added support in the p5.Framebuffer library for that antialiasing solution I mentioned earlier (WebGL2 antialiasing if available, or rendering at 2x pixel density otherwise.) I'm hoping to learn how reliable this setup is, to see if it could ever end up being an option for the main canvas. I'll keep using this for a while and report back if I run into any issues or if it seems to work OK!

[davepagurek]

After talking to other maintainers about this, we've decided to make Framebuffer objects that one can use as an alternative to p5.Graphics for WebGL only environments: #6018

This means, at least at first, we won't be replacing the backend of existing features with framebuffers. This means we aren't blocking other features, e.g. shader-based filter(), on using this, but once this feature is available, we can evaluate what it would take to make those other features use framebuffers. I'm going to close this issue for now, so discussion of the new Framebuffer class can continue in #6018 and once that's in we can make new issues to talk about updating other features to use framebuffers.