Adds createFilterShader() and custom shader support to the webGL filter() function #6237

wong-justin · 2023-06-23T21:56:38Z

Initial changes:

filter() accepts a shader parameter
Added property for storing current filter shader

Will also change:

Apply the current filter shader and render to main canvas / parent renderer
Add shaders for constants like GRAYSCALE, etc
Clarifying docs to show new functionality

p5.Framebuffer is an alternative graphics layer to pursue if p5.Graphics doesn't work. Either one is nice if it gives a simple way to call shader(), renders to main with something like texture() plane() / copyPixels(), and gives resizing and other coverage for free. Right now I started a framebuffer approach after running into two difficult p5.Graphics errors.

(This PR is a working draft that's part of the GSoC 2023 project 'Supporting shader-based filters in p5.js", mentored by @aferriss and @aceslowman)

npm run lint passes
[Inline documentation] is included / updated
[Unit tests] are included / updated

although this line isn't necessary

@method

1 - inline docs adds new @method signature 2 - build process updates docs/parameterData.json 3 - validateParameters accepts overloaded signature

davepagurek · 2023-06-26T13:05:21Z

src/webgl/p5.RendererGL.js

@@ -576,6 +577,10 @@ p5.RendererGL = class RendererGL extends p5.Renderer {
    // set of framebuffers in use
    this.framebuffers = new Set();

+    // for post processing step
+    this.filterShader = undefined;
+    this.filterFramebuffer = undefined;


btw, in case it's useful, RendererGL has access to a lazily-created framebuffer that matches its size, currently used for its loadPixels implementation:

p5.js/src/webgl/p5.RendererGL.js

Lines 1047 to 1052 in a66195a

* @private

* @returns {p5.Framebuffer} A p5.Framebuffer set to match the size and settings

* of the renderer's canvas. It will be created if it does not yet exist, and

* reused if it does.

*/

_getTempFramebuffer() {

Might work if you use that for filters too? But there could be other reasons why you want a separate one too.

wip

filter() will only receive one parameter, the arguments list since it is an overloaded function

src/webgl/p5.RendererGL.js

src/image/pixels.js

aferriss · 2023-06-30T22:06:52Z

src/webgl/p5.RendererGL.js

+    // Bind the shader to the pg renderer,
+    // since when the user makes it with createShader() it binds to the main
+    userShader =
+      new p5.Shader(pg._renderer, userShader._vertSrc, userShader._fragSrc);


We probably don't wanna create a shader every frame, can we cache it somehow?

aferriss · 2023-06-30T22:28:06Z

src/image/pixels.js

+
+/**
+ * @method filter
+ * @param {p5.Shader}  shaderFilter  A shader that's been loaded


Add in example, alt text, supporting documentation, etc.

aferriss · 2023-06-30T22:28:31Z

src/webgl/p5.RendererGL.js

+    // Bind the shader to the pg renderer,
+    // since when the user makes it with createShader() it binds to the main
+    userShader =
+      new p5.Shader(pg._renderer, userShader._vertSrc, userShader._fragSrc);


Should we provide users with a default vertex shader?

I think so. Here are some related thoughts: #6144
I'm leaning towards the simplest option, just exposing a default vertex shader as a string constant.

Another thought, we could make a function to make a shader that only takes in a fragment shader and automatically uses that default vertex shader? Like createFilterShader()?

A default vertex shader as a string constant honestly makes sense to me, although I'd like to hear other thoughts. Would the idea be that they would use createShader() like normal, but pass in the new constant?

there were other reasons for mismatched canvases sizes like incorrect shader scaling, or different rect modes

src/webgl/p5.Shader.js

aferriss · 2023-07-08T06:40:54Z

src/webgl/p5.RendererGL.js

+    //   this.filterShader = map(args[0], {GRAYSCALE: grayscaleShader, ...})
+    //   filterOperationParameter = undefined or args[1]
+
+    if (typeof args[0] === 'string') {


Is this the right check? I think we need to know if args[0] is a p5.shader, or a filter constant ( BLUR, INVERT, etc), and if not, we print an error. Aren't the filter constants strings?

Yes, that's the check. Although there shouldn't need to be a third case handling errors, since the args get validated in the parent filter() function: pixels.js#L472

Right now the string / filter constant case has an error placeholder. It will be replaced in the next PR that implements the shader versions of filter constants. And the current code handles the p5.Shader case.

src/webgl/material.js

aferriss · 2023-07-14T17:16:31Z

src/webgl/p5.RendererGL.js

+    );
+    if (!isSameUserShader) {
+      this.filterShader =
+        new p5.Shader(pg._renderer, userShader._vertSrc, userShader._fragSrc);


If we're supplying a shader created with createFilterShader() can we skip this?

Agreed to skip this tiny optimization for now

mainly removing unneccesary functions, using brick image

including pre-flipping texcoords.y to make it easier for the user

src/webgl/material.js

to be consistent for the user, since they'll have to flip y eventually

aferriss · 2023-07-28T17:14:29Z

src/webgl/material.js

 *
- * The fragment shader is given a uniform, or variable, called `tex0`.
+ * Note:
+ * - The fragment shader is given a uniform, or variable, called `tex0`.


Can we change the language to clarify that the default input is a texture called tex0?

Something like: "The fragment shader is provided with a single texture input uniform called tex0"

src/webgl/material.js

aferriss · 2023-07-28T17:25:21Z

src/webgl/p5.RendererGL.js

+    // so it's just created here once on the initial filter call.
+    if (!this.filterGraphicsLayer) {
+      this.filterGraphicsLayer =
+        new p5.Graphics(this.width, this.height, constants.WEBGL, this._pInst);


Could it not be working with pgraphics because this._pInst is referring to the main sketch?

including resolving a conflict where a test was added in the same place in test/unit/p5.Renderer.GL

- old flip was using vTexCoord.y, which let some intermediate values remain upside down - but new flip uses scale(1,-1) which makes everything look right-side up

checks for unusual pInst

aferriss · 2023-08-01T17:01:01Z

test/unit/webgl/p5.RendererGL.js

+      assert.notDeepEqual(p1, p2);
+    });
+
+    // test('default constant shader makes changes to main canvas', function() {


@wong-justin can you remove this commented code before we merge?

wong-justin added 5 commits June 23, 2023 15:24

declare framebuffer usage

16def01

although this line isn't necessary

new properties for shader filters

c7abb8c

partial shader filter implementation instead of error

9dbe595

add new filter() signature

a096914

1 - inline docs adds new @method signature 2 - build process updates docs/parameterData.json 3 - validateParameters accepts overloaded signature

branch filter() for either webgl mode or 2d mode, not just 2d

87f347a

davepagurek reviewed Jun 26, 2023

View reviewed changes

wong-justin added 5 commits June 29, 2023 18:06

revert to p5.Graphics instead of p5.Framebuffer, at least for now

e0126bf

apply shader to secondary graphics, then draw onto main

cb1979b

wip

Merge remote-tracking branch 'upstream/main' into shader-filters

644e61a

add manual test for webgl filter()

628fc18

add some automated webgl filter tests

359d16b

wong-justin changed the title ~~Shader Filters~~ WEBGL Shader Filters Jun 30, 2023

wong-justin added 3 commits June 30, 2023 10:59

remove confused arguments

e857d5b

filter() will only receive one parameter, the arguments list since it is an overloaded function

create new shader bound to secondary graphics

a762869

add tests to check sizing

2eb64a0

aferriss reviewed Jun 30, 2023

View reviewed changes

src/webgl/p5.RendererGL.js Outdated Show resolved Hide resolved

aferriss reviewed Jun 30, 2023

View reviewed changes

src/image/pixels.js Show resolved Hide resolved

aferriss reviewed Jun 30, 2023

View reviewed changes

wong-justin added 8 commits July 7, 2023 18:04

docs for filter(shader)

0c5d0ff

add resizing to secondary graphics

10d85ff

only create secondary shader once

f324437

leave out filter(CONSTANT) test until that feature is added

7f39f69

remove resolution check because pixel density wont change

fc50e52

there were other reasons for mismatched canvases sizes like incorrect shader scaling, or different rect modes

remove framebuffer test since framebuffer isnt used for now

f3a41e7

fill in empty tests

7888ad4

wip: copy uniforms to shader copy

b52a467

aferriss reviewed Jul 8, 2023

View reviewed changes

src/webgl/p5.Shader.js Show resolved Hide resolved

aferriss reviewed Jul 8, 2023

View reviewed changes

aferriss reviewed Jul 14, 2023

View reviewed changes

src/webgl/material.js Show resolved Hide resolved

aferriss reviewed Jul 14, 2023

View reviewed changes

wong-justin added 7 commits July 14, 2023 14:07

fix gitignore

bf282f3

document uniforms

25424c7

refine example fragment shaders

d1dd620

mainly removing unneccesary functions, using brick image

default vertex shader

fa178ba

including pre-flipping texcoords.y to make it easier for the user

rest of createFilterShader docs

68ad8ae

adjust test shaders to match default filter shaders

c039991

add test comparing default vertex shader to a supplied one

c476759

wong-justin commented Jul 21, 2023

View reviewed changes

src/webgl/material.js Show resolved Hide resolved

wong-justin added 4 commits July 28, 2023 08:32

revert pre-flipping the y in the vertex shader

4b1e56e

to be consistent for the user, since they'll have to flip y eventually

add test for edge case of filter shader on graphics context

ec76701

move uniform documentation to setUniform

8220071

add disclaimers to createFilterShader docs

44ad94f

aferriss reviewed Jul 28, 2023

View reviewed changes

src/webgl/material.js Show resolved Hide resolved

aferriss reviewed Jul 28, 2023

View reviewed changes

wong-justin added 4 commits July 28, 2023 17:04

Merge remote-tracking branch 'upstream/main' into shader-filters

a532146

including resolving a conflict where a test was added in the same place in test/unit/p5.Renderer.GL

go back to flipping the y

479aca3

- old flip was using vTexCoord.y, which let some intermediate values remain upside down - but new flip uses scale(1,-1) which makes everything look right-side up

fix filter() not working on a secondary p5.Graphics

4250399

checks for unusual pInst

small wording changes in createFilterShader docs

5abd47d

aferriss reviewed Aug 1, 2023

View reviewed changes

aferriss approved these changes Aug 1, 2023

View reviewed changes

aferriss changed the title ~~WEBGL Shader Filters~~ Adds createFilterShader() and custom shader support to the webGL filter() function Aug 1, 2023

aferriss added the Area:WebGL label Aug 1, 2023

remove unused commented code from test

51aa874

wong-justin marked this pull request as ready for review August 2, 2023 07:51

aferriss merged commit e0678fb into processing:main Aug 2, 2023

wong-justin mentioned this pull request Aug 3, 2023

Add shaders for filter() constants, and use them by default in P2D #6324

Merged

3 tasks

	* @private
	* @returns {p5.Framebuffer} A p5.Framebuffer set to match the size and settings
	* of the renderer's canvas. It will be created if it does not yet exist, and
	* reused if it does.
	*/
	_getTempFramebuffer() {

Uh oh!

Adds createFilterShader() and custom shader support to the webGL filter() function #6237

Adds createFilterShader() and custom shader support to the webGL filter() function #6237

Uh oh!

Conversation

wong-justin commented Jun 23, 2023

Initial changes:

Will also change:

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aferriss Aug 1, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

aferriss Aug 1, 2023 •

edited

Loading