Expose an intuitive subset of stencil operations

[QbieShay]

Describe the project you are working on

Godot engine

Describe the problem or limitation you are having in your project

1. No stencil buffer support which makes a number of important effects impossible to achieve without waste of GPU resources
2. Stencil API is extremely convoluted, counterintuitive and brain-breaking to understand
3. People requested (5 years now) stencil support. People need stencil.

After gathering feedback in #3373
After testing with godotengine/godot#78542
After a significant amount of hours spent by clay and myself to wrap our head around this

Considering we don't want to expose the full complexity of stencil outside of lower level API (more on this in the future)
Considering we don't want to 100% copy any other engine in their implementation (expect tutorials from Unity to not apply)

Read along for Godot's stencil proposal ^^
We've tried to think of the use cases brought up in #3373 and they all seem possible with this API.

This will be an iterative process and more features will come little by little. This is huge work. There's a lot of things to consider. Please be patient

Describe the feature / enhancement and how it helps to overcome the problem or limitation

Supersedes #3373

Stencil operations are needed for a wide range of 3D FX. They can also be very useful to optimize a lot of other 3D FX.

Currently, Godot allocates a stencil buffer when allocating the 3D renderbuffers (combined depth + stencil), but does not make use of it.

Stencil operations as exposed by the OpenGL and Vulkan standards are cumbersome, confusing, and exposes a lot of meaningless combinations of similar-sounding settings. We want to implement something that can be used by most users and not just by users with a background in advanced tech art or 3D graphics APIs.

At the same time, there is a huge demand to have some control over stencil operations to do more than just the most common effects (masking and xray).

Masking works like the depth buffer. In other words, at some point in time you render to the stencil buffer and place a value. At another time (or at the same time) you read from the stencil buffer, compare the value and reject based on a set condition (equals, not equals, greater than, etc.). Masking allows users to implement outlines, complex masks, and performance optimizations.

Xray allows users to add overlays that persist through scene geometry. I.e. show an enemy through a wall etc. Xray requires writing a stencil value at one point (i.e. when you first draw the enemy), but only when the depth test passes. In a later pass the "xray" effect is drawn and rejects any pixels that match the xray stencil value.

The important difference between the two is that Masking writes regardless of the depth test.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

Shader API

Similar to render_mode, add the keyword stencil_mode

Mode (enum):

WRITE_DEPTH_FAIL // stencil value is only written if the depth test fails

READ_ONLY // stencil value is never written, but it is tested // must pass stencil and depth // depth testing can be disabled in the depth settings. Stencil testing can be disabled by not using stencil

READ_WRITE // stencil value is written if ref passes stencil test and depth test

WRITE_ONLY // stencil value is written if it passes depth test

DISABLED

Compare mode (enum):

COMPARE_LESS

COMPARE_EQUAL

COMPARE_LESS_OR_EQUAL

COMPARE_GREATER

COMPARE_NOT_EQUAL

COMPARE_GREATER_OR_EQUAL

Ref (an integer 0-255)

These could be implemented as rendering modes or with a new token stencil_mode i.e.:

stencil_mode read_only, compare_less, 1;

Shader

The stencil value should be exposed in the fragment shader by reading from the stencil buffer with a STENCIL_ID keyword.

Pipeline details

There's a lot of work needed to make this fully usable not only in term of exposing the relevant keywords to shaders, but to also make sure that those operations are placed in a sensible manner in the rendering pipeline. For example WRITE_DEPTH_FAIL cannot be run at the depth prepass. Similarly, stencil operations need to run in a consistent manner and to run either in depth prepass or in subsequent passes, but ideally not both, which causes effects to misbehave.

Considerations on how depth sorting buckets are managed for opaque objects must be taken into consideration, possibly supporting appropriate ordering of subsequent draw passes specified via the "next pass API" which currently doesn't seem to sort objects reliably, causing flickering and completely breaking stencil operation.

Tentative step-by-step

1. First implementation will add the above keywords. It will probably be very finnicky on the render ordering and require to be run in some combination of transparent and opaque. How to solve this will be decided in the future once enough feedback on this work is gathered. Solving all the ordering usecases is out of scope for step one. Step one should grant at least the possibility to create
2. masks and impossible geometry (like antichamber's room)
3. outlines
4. xray shaders
   this is the absolute minimal subset of features that should reliably work out of the box. Windwaker style lights have been tested but they don't work reliably due to sorting issues on draw passes mentioned above. they can be however achieved just by putting separate nodes with the same sphere and using sorting offset.
   This needs to be iterative work. There will be back and forth.
5. second pass should address sorting issues and any usecase that's isn't covered in first pass and add presets:
6. StandardMaterial3D

We want users to be able to implement stencil outlines and stencil Xray effects without understanding how stencils work (the same way that users can implement transparent effects without understanding all the depth settings). Accordingly, we propose to expose a high level API in the StandardMaterial3D that makes these much easier.

stencil_mode (enum):

OUTLINE. When selected, exposes:

color

thickness

ID

Internally this would:

set mode to WRITE_ONLY

set ref to ID

Add a next_pass with a basic opaque material set to unshaded and with albedo_color set to the specified color and with the grow property set to thickness and with stencil mode set to READ_ONLY, compare mode set to NOT_EQUAL and ref set to ID

XRAY. When selected, exposes:

color

ID

Internally this would:

Set mode to WRITE_DEPTH_FAIL

set ref to ID

Add a next_pass with a basic transparent material set to unshaded and with albedo_color set to the specified color and with stencil mode set to READ_ONLY, compare mode set to NOT_EQUAL and ref set to ID

CUSTOM. When selected, exposes:

read_write_mode,

compare_mode (only visible when read_write_mode includes reading), and

ref

In parallel stencil explorations for 2D can be made. I have no idea how to do that or even if it's possible. I have exhausted my energy on this so proposals for 2D stencil are welcome. Feel free to experiment with the original PR I opened to then make a proposal!

Alternatives

To the people inevitably unsatisfied with this solution: we know. We hear you. We have decided to not compromise entirely on usability for this work, but it doesn't mean that it will be forever this way. There's the intention to extend the rendering API and to eventually offer low level access to the renderer and to the whole stencil API via the lower level interface. This is out of scope for this initial work. This proposal is a weighted compromise between:

1. maintaining the user friendliness which is at the core of Godot's development
2. keeping the work manageable and the maintenance cost contained: exposing everything now would make it much harder to tweak things in the future. API cannot be unexposed once exposed.
3. getting some form of support for stencil finally in the engine with an amount of work that's feasible

Conclusion

Thank you for reading until here if you have, and thanks to everyone that participated to this discussion and provided their input. Special thanks to @apples whose initial work made it possible at all to finally reach a consensus on how we want this API to look like.

Thank you all <3

If this enhancement will not be used often, can it be worked around with a few lines of script?

No

Is there a reason why this should be core and not an add-on in the asset library?

No

[QbieShay]

It would be super useful and great if someone picks up this work code-wise. Ideally we'd be able to test WIP builds and give feedback in an iterative manner with the community. I do not have the capacity to implement this.

[RedMser]

Thank you for your work on this proposal!

Is there any reason the StandardMaterial3D stencil options only seem to allow specifying a color, and not an entire outline/xray material? Is it for ease of implementation? I don't think this API will find use if color is all you can specify.
Technically an outline/xray material could be realized by godot automatically tweaking the stencil settings of the material sub-resource when it is used for outline/xray. If the inspector allows for it, disable or hide any options which are forced to a fixed value by the parent material.

Note that stencil stuff is all way above my head, so I probably wouldn't be able to benefit from even the mid-level API, let alone anything low-level.

[QbieShay]

Is there any reason the StandardMaterial3D stencil options only seem to allow specifying a color, and not an entire outline/xray material?

The idea is that if you want a custom one you can make a shader yourself, but if you want something quick it's just a matter of a few clicks. Note that it auto-adds a second pass to the standard material.

[fire]

Is it possible to support "half". So instead of an integer 0-255, it's a 16 bit integer? Not sure about the internals of allowing 32bit integers.