designpaca 초기 구현 — 스킬 · 설치 CLI · 배포 파이프라인

웹 디자인 파이프라인 스킬과 이를 5개 에이전트에 설치하는 CLI 를 담은 모노레포.

스킬 (packages/skill)
- SKILL.md 261줄 + 참조 문서 16개 3,349줄. progressive disclosure 로
  본문은 절차와 인덱스만, 지식은 references/ 로 분리
- 0~6단계 파이프라인. 규모에 따라 전체·연장·국소 세 경로로 분기
- 하드 게이트 12개는 grep·카운트로 검증 가능한 것만. 취향 판단은 제외
- 미학 프리셋 5종, AI 슬롭 지문 목록, 한글 조판 규칙,
  SVG 필터·three.js·인터랙티브 모션·HTML-in-Canvas 실전 지침

설치 CLI (packages/cli, packages/core)
- npx designpaca 온보딩 TUI. Claude Code · Codex · Cursor · Windsurf · AGENTS.md
- 매니페스트에 설치 시점 해시를 기록해 사용자가 고친 파일은 update 가 건너뛴다
- 타깃별로 본문의 references/ 경로를 실제 설치 위치로 재작성
- AGENTS.md 는 항상 로드되므로 본문 대신 303자 포인터만 주입
- Windsurf 는 12,000자 상한 초과 시 설치를 차단

배포 (build/ci, .forgejo/workflows)
- 태그 v* → 검사·테스트·빌드 → npmjs 배포 + Forgejo 레지스트리 미러
  → draft 릴리스 → Cloudflare Pages. 재실행 멱등

근거 (research/)
- 약 250개 웹 소스 조사 결과와 도그푸딩 검증 2건. 스킬의 모든 수치는 여기서 나온다

테스트 22개 통과 (core 16 · cli 6)
This commit is contained in:
Yun Chan 2026-08-20 10:48:00 +09:00
commit 8808c672dc
135 changed files with 38838 additions and 0 deletions

View file

@ -0,0 +1,36 @@
# Awesome HTML-in-Canvas
This is a collection of resources to help developers build with HTML-in-Canvas.
Check out the HTML-in-canvas deployed at [chrome.dev](https://chrome.dev/html-in-canvas/) or view the source code [here](https://github.com/GoogleChromeLabs/css-web-ui-demos/tree/main/html-in-canvas).
## HTML-in-Canvas demos by the ecosystem
This is a curated list of links to awesome HTML-in-canvas demos created by the ecosystem. Note that the demos featured here are contributed by third-party developers and are not created or maintained by Google. Read the [contribution guidelines](https://github.com/GoogleChromeLabs/css-web-ui-demos/blob/main/CONTRIBUTING.md#add-a-demo-to-the-awesome-html-in-canvas-list) to suggest another demo.
| Demo | Description | Author | Source code |
| :--- | :--- | :--- | :--- |
| [Duck Hunt TODO](https://x.com/wesbos/status/2041594973674483851) | A form that's also a shooting game | [Wes Bos](https://github.com/wesbos) | [Source](https://github.com/wesbos/hot-tips/blob/main/html-in-canvas/demos/wicg/website-shatter-shooter.html) |
| [Wobble Buttons](https://x.com/wesbos/status/2041974552478052507) | Interactive ripple-effect buttons | [Wes Bos](https://github.com/wesbos) | [Source](https://github.com/wesbos/hot-tips/blob/main/html-in-canvas/demos/wicg/ripple-buttons.html) |
| [Compiz Web](https://compiz-web.vercel.app/) | Shader-driven web page transitions demo | [Max Leiter](https://github.com/MaxLeiter) | [Source](https://github.com/MaxLeiter/compiz-web) |
| [HTML cloth](https://arrival.space/htmlcanvas) | Customize a form on a hanging cloth inside a game | [Thomas Richter-Trummer](https://github.com/fimbox) | [Source](https://github.com/fimbox/html-in-canvas/blob/main/plugins/html-cloth.mjs) |
| [PixiJS HTML Laser](https://pixijs-html-in-canvas.vercel.app) | Interactive landing page that shatters and heals over time | [Zyie](https://github.com/Zyie) | [Source](https://github.com/Zyie/pixijs-html-in-canvas) |
| [Quest Signal](https://vav-labs.com/case-studies/quest-signal/) | A playable Godot scene with accessible, interactive DOM panels rendered as world-space WebGL textures | [Vav Labs](https://vav-labs.com/) | [Source](https://github.com/Vav-Labs/quest-signal) |
| More | demos | coming | soon... |
## Framework Support
This is a list of frameworks that have added support for HTML-in-Canvas along with the documentation
| Framework | Description | Documentation | Sample Code |
| :--- | :--- | :--- | :--- |
| [Three.js](https://threejs.org/) | JavaScript library used to create and display animated 3D computer graphics with WebGL & WebGPU | [HTMLTexture](https://goo.gle/HIC-threejs) | [Sample](https://goo.gle/HIC-threejs-example) |
| [PlayCanvas](https://playcanvas.com/) | Open source engine and tools for building amazing 3D experiences | [html-texture](https://goo.gle/HIC-playcanvas) | [Sample](https://goo.gle/HIC-playcanvas-example) |
| [PixiJS](https://pixijs.com/) | Fast, lightweight 2D rendering library for WebGL & WebGPU | [HTMLSource](https://pixijs.download/release/docs/rendering.HTMLSource.html) | [Sample](https://pixijs-html-in-canvas.vercel.app/) |
| [Babylon.js](https://babylonjs.com/) | Babylon.js: Powerful, Beautiful, Simple, Open 3D engine for the web | [HTML Texture](https://doc.babylonjs.com/features/featuresDeepDive/materials/using/htmlTexture/) | [Sample](https://playground.babylonjs.com/#8RDVXG#1) |
| [CanvasUI](https://canvasui.dev/) | An open source library of tasteful html-in-canvas & WebGL components. | [Introduction](https://canvasui.dev/docs) | [Sample](https://canvasui.dev/docs/components/bend) |
## Disclaimer
**Important note on external content**: The demos linked in the [HTML-in-Canvas demos by the ecosystem](#html-in-canvas-demos-by-the-ecosystem) section are created by third-party developers and are not created, maintained, or supported by Google. Please be aware of the following:
* No endorsement: Inclusion of these links does not constitute an endorsement or recommendation by Google.
* Subject to change: Content, functionality, and availability are at the sole discretion of the third-party owners and may change or be removed without notice.
* No liability: Google assumes no responsibility or liability for the accuracy, legality, or performance of these demos.

View file

@ -0,0 +1,50 @@
<!doctype html>
<meta charset="utf-8" />
<title>Demo of complex text in canvas</title>
<style>
canvas {
border: 1px solid blue;
width: 638px;
height: 318px;
}
</style>
<canvas id="canvas" width="638" height="318" layoutsubtree="true">
<div id="draw_element" style="width: 550px;">
Hello from <a href="https://github.com/WICG/html-in-canvas">html-in-canvas</a>!
<br>I'm multi-line, <b>formatted</b>,
rotated text with emoji (&#128512;), RTL text
<span dir=rtl>من فارسی صحبت میکنم</span>,
vertical text,
<p style="writing-mode: vertical-rl;">
这是垂直文本
</p>
an inline image (<img width="150" src="wolf.jpg">), and
<svg width="50" height="50">
<circle cx="25" cy="25" r="20" fill="green" />
<text x="25" y="30" font-size="15" text-anchor="middle" fill="#fff">
SVG
</text>
</svg>!
</div>
</canvas>
<script>
const canvas = document.getElementById('canvas');
const ctx = canvas.getContext('2d');
canvas.onpaint = (event) => {
ctx.reset();
ctx.rotate((15 * Math.PI) / 180);
ctx.translate(80 * devicePixelRatio, -20 * devicePixelRatio);
let transform = ctx.drawElementImage(draw_element, 0, 0);
draw_element.style.transform = transform.toString();
};
canvas.requestPaint(); // Request an initial paint event.
const observer = new ResizeObserver(([entry]) => {
canvas.width = entry.devicePixelContentBoxSize[0].inlineSize;
canvas.height = entry.devicePixelContentBoxSize[0].blockSize;
});
observer.observe(canvas, {box: 'device-pixel-content-box'});
</script>

View file

@ -0,0 +1,87 @@
<!doctype html>
<meta charset="utf-8" />
<title>Pie chart</title>
<style>
.pie {
width: 250px;
height: 250px;
}
.pie .label {
text-align: center;
max-width: 40%;
font-family: sans-serif;
}
.pie .label .val {
display: block;
font-size: xx-large;
font-weight: bold;
}
</style>
<canvas layoutsubtree class="pie" role="list" aria-label="Pie Chart">
<div class="label" role="listitem" tabindex="0" data-val="0.45" data-color="tomato">
<span class="val">45%</span>Apple
</div>
<div class="label" role="listitem" tabindex="0" data-val="0.35" data-color="cornflowerblue">
<span class="val">35%</span>Blackberry / Bramble
</div>
<div class="label" role="listitem" tabindex="0" data-val="0.20" data-color="gold">
<span class="val">20%</span>Durian
</div>
</canvas>
<script>
const canvas = document.querySelector('canvas');
const ctx = canvas.getContext('2d');
canvas.onpaint = () => {
ctx.reset();
// 1. Center the coordinate system.
const radius = 0.95 * Math.min(canvas.width, canvas.height) / 2;
ctx.translate(canvas.width / 2, canvas.height / 2);
let angle = 0;
let focusedPath = null;
for (const label of canvas.children) {
const slice = Number(label.dataset.val) * Math.PI * 2;
// 2. Draw the wedge.
const grad = ctx.createRadialGradient(0, 0, 0, 0, 0, radius);
grad.addColorStop(0, `color-mix(${label.dataset.color}, white 40%)`);
grad.addColorStop(1, label.dataset.color);
ctx.fillStyle = grad;
const path = new Path2D();
path.moveTo(0, 0);
path.arc(0, 0, radius, angle, angle + slice);
path.closePath();
ctx.fill(path);
if (document.activeElement === label)
focusedPath = path;
// 3. Draw the label element, and update its transform.
const mid = angle + slice / 2;
const label_width = label.offsetWidth * devicePixelRatio;
const label_height = label.offsetHeight * devicePixelRatio;
const x = Math.cos(mid) * radius * 0.60 - label_width / 2;
const y = Math.sin(mid) * radius * 0.60 - label_height / 2;
const transform = ctx.drawElementImage(label, x, y);
label.style.transform = transform;
angle += slice;
}
// 4. Draw the focus ring on top of everything else.
if (focusedPath)
ctx.drawFocusIfNeeded(focusedPath, document.activeElement);
};
canvas.requestPaint(); // Request an initial paint event.
// Setup a resize observer to resize the canvas in response to dpr changes.
new ResizeObserver(([entry]) => {
const box = entry.devicePixelContentBoxSize[0];
canvas.width = box.inlineSize;
canvas.height = box.blockSize;
}).observe(canvas, {box: ['device-pixel-content-box']});
</script>

View file

@ -0,0 +1,68 @@
<!doctype html>
<meta charset="utf-8" />
<title>Demo of interactive content in canvas</title>
<style>
canvas {
border: 1px solid blue;
width: 638px;
height: 318px;
}
form p {
margin: 6px;
}
</style>
<canvas id="canvas" width="638" height="318" layoutsubtree="true">
<div id=draw_element style="width: 578px" >
<form id="demo-form" action="#" method="get">
<fieldset>
<legend>🚀 Spaceship Control Panel</legend>
<p>
<label for="shipName">Ship Name:</label>
<input type="text" id="shipName" value="The 'Canvas' Voyager">
</p>
<p>
<input type="checkbox" id="hyperdrive" checked>
<label for="hyperdrive">Engage Hyperdrive</label>
</p>
<fieldset>
<legend>Target System</legend>
<p>
<input type="radio" id="alpha" name="system" value="alpha" checked>
<label for="alpha">Alpha Centauri</label>
</p>
<p>
<input type="radio" id="beta" name="system" value="beta">
<label for="beta">Betelgeuse</label>
</p>
</fieldset>
<p>
<label for="shieldLevel">Shield Strength:</label>
<input type="range" id="shieldLevel" min="0" max="100" value="75">
</p>
<p style="text-align: right; margin: 0;">
<button type="submit">Launch!</button>
</p>
</fieldset>
</form>
</div>
</canvas>
<script>
const canvas = document.getElementById('canvas');
const ctx = canvas.getContext('2d');
canvas.onpaint = (event) => {
ctx.reset();
let x = canvas.width / 25;
let y = canvas.height / 25;
let transform = ctx.drawElementImage(draw_element, x, y);
draw_element.style.transform = transform.toString();
};
canvas.requestPaint(); // Request an initial paint event.
const observer = new ResizeObserver(([entry]) => {
canvas.width = entry.devicePixelContentBoxSize[0].inlineSize;
canvas.height = entry.devicePixelContentBoxSize[0].blockSize;
});
observer.observe(canvas, {box: 'device-pixel-content-box'});
</script>

View file

@ -0,0 +1,193 @@
<!doctype html>
<meta charset="utf-8" />
<title>Demo of complex text in WebGL</title>
<script
src="https://cdnjs.cloudflare.com/ajax/libs/gl-matrix/2.8.1/gl-matrix-min.js"
integrity="sha512-zhHQR0/H5SEBL3Wn6yYSaTTZej12z0hVZKOv3TwCUXT1z5qeqGcXJLLrbERYRScEDDpYIJhPC1fk31gqR783iQ=="
crossorigin="anonymous"
defer>
</script>
<script src="webGLSetup.js"></script>
<style>
canvas {
border: 1px solid blue;
width: 638px;
height: 318px;
}
#draw_element {
border: 1px solid blue;
width: 400px;
height: 400px;
padding: 10px;
}
</style>
<canvas id="gl-canvas" width="638" height="318" layoutsubtree="true">
<!-- inert to prevent hit testing in this example. -->
<div id="draw_element" inert>
Hello world!<br>I'm multi-line, <b>formatted</b>,
rotated text with emoji (&#128512;), RTL text
<span dir=rtl>من فارسی صحبت میکنم</span>,
vertical text,
<p style="writing-mode: vertical-rl;">
这是垂直文本
</p>
an inline image (<img width="150" src="wolf.jpg">), and
<svg width="50" height="50">
<circle cx="25" cy="25" r="20" fill="green" />
<text x="25" y="30" font-size="15" text-anchor="middle" fill="#fff">
SVG
</text>
</svg>!
</div>
</canvas>
<script>
let cubeRotation = 0.0;
let currentTime = 0;
let deltaTime = 0;
let render_context = null;
//
// Initialize a texture and load an image.
// When the image finished loading copy it into the texture.
//
function loadTexture(gl) {
const texture = gl.createTexture();
gl.bindTexture(gl.TEXTURE_2D, texture);
const internalFormat = gl.RGBA8;
try {
gl.texElementImage2D(gl.TEXTURE_2D, internalFormat, draw_element);
} catch (e) {
// The texElementImage2D API was recently changed (see:
// https://github.com/WICG/html-in-canvas#idl-changes). This snippet
// supports the old syntax temporarily so that the demos do not break.
const level = 0;
const srcFormat = gl.RGBA;
const destType = gl.UNSIGNED_BYTE;
gl.texElementImage2D(gl.TEXTURE_2D, level, internalFormat,
srcFormat, destType, draw_element);
console.log('Note: using old texElementImage2D API');
}
// Linear texture filtering produces better results than mipmap with text.
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.LINEAR);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE);
gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE);
return texture;
}
// Draw the scene repeatedly
function render() {
let new_time = performance.now() * 0.001; // convert to seconds
deltaTime = new_time - currentTime;
currentTime = new_time;
if (render_context === null) {
return;
}
drawScene(render_context.gl,
render_context.program,
render_context.buffers,
render_context.texture,
cubeRotation);
cubeRotation += deltaTime;
requestAnimationFrame(render);
}
function main() {
const canvas = document.querySelector('#gl-canvas');
// Initialize the GL context
const gl = canvas.getContext('webgl2');
// Only continue if WebGL is available and working
if (gl === null) {
alert(
'Unable to initialize WebGL. Your browser or machine may not support it.',
);
return;
}
// Vertex shader program
const vsSource = `
attribute vec4 aVertexPosition;
attribute vec2 aTextureCoord;
uniform mat4 uModelViewMatrix;
uniform mat4 uProjectionMatrix;
varying highp vec2 vTextureCoord;
void main(void) {
gl_Position = uProjectionMatrix * uModelViewMatrix * aVertexPosition;
vTextureCoord = aTextureCoord;
}
`;
// Fragment shader program
const fsSource = `
varying highp vec2 vTextureCoord;
uniform sampler2D uSampler;
void main(void) {
gl_FragColor = texture2D(uSampler, vTextureCoord);
}
`;
// Initialize a shader program; this is where all the lighting
// for the vertices and so forth is established.
const shaderProgram = initShaderProgram(gl, vsSource, fsSource);
// Collect all the info needed to use the shader program.
// Look up which attribute our shader program is using
// for aVertexPosition and look up uniform locations.
const programInfo = {
program: shaderProgram,
attribLocations: {
vertexPosition: gl.getAttribLocation(shaderProgram, 'aVertexPosition'),
textureCoord: gl.getAttribLocation(shaderProgram, 'aTextureCoord'),
},
uniformLocations: {
projectionMatrix: gl.getUniformLocation(shaderProgram, 'uProjectionMatrix'),
modelViewMatrix: gl.getUniformLocation(shaderProgram, 'uModelViewMatrix'),
uSampler: gl.getUniformLocation(shaderProgram, 'uSampler'),
},
};
const buffers = initBuffers(gl);
// Load texture
const texture = loadTexture(gl);
// Flip image pixels into the bottom-to-top order that WebGL expects.
gl.pixelStorei(gl.UNPACK_FLIP_Y_WEBGL, true);
render_context = {
gl: gl,
program: programInfo,
buffers: buffers,
texture:texture,
};
requestAnimationFrame(render);
}
onload = () => {
const canvas = document.querySelector('#gl-canvas');
canvas.onpaint = () => {
main();
}
canvas.requestPaint();
const observer = new ResizeObserver(([entry]) => {
canvas.width = entry.devicePixelContentBoxSize[0].inlineSize;
canvas.height = entry.devicePixelContentBoxSize[0].blockSize;
});
observer.observe(canvas, {box: 'device-pixel-content-box'});
}
</script>

View file

@ -0,0 +1,288 @@
//
// creates a shader of the given type, uploads the source and
// compiles it.
//
function loadShader(gl, type, source) {
const shader = gl.createShader(type);
// Send the source to the shader object
gl.shaderSource(shader, source);
// Compile the shader program
gl.compileShader(shader);
// See if it compiled successfully
if (!gl.getShaderParameter(shader, gl.COMPILE_STATUS)) {
alert(
`An error occurred compiling the shaders: ${gl.getShaderInfoLog(shader)}`,
);
gl.deleteShader(shader);
return null;
}
return shader;
}
//
// Initialize a shader program, so WebGL knows how to draw our data
//
function initShaderProgram(gl, vsSource, fsSource) {
const vertexShader = loadShader(gl, gl.VERTEX_SHADER, vsSource);
const fragmentShader = loadShader(gl, gl.FRAGMENT_SHADER, fsSource);
// Create the shader program
const shaderProgram = gl.createProgram();
gl.attachShader(shaderProgram, vertexShader);
gl.attachShader(shaderProgram, fragmentShader);
gl.linkProgram(shaderProgram);
// If creating the shader program failed, alert
if (!gl.getProgramParameter(shaderProgram, gl.LINK_STATUS)) {
alert(
`Unable to initialize the shader program: ${gl.getProgramInfoLog(
shaderProgram,
)}`,
);
return null;
}
return shaderProgram;
}
function initBuffers(gl) {
const positionBuffer = initPositionBuffer(gl);
const textureCoordBuffer = initTextureBuffer(gl);
const indexBuffer = initIndexBuffer(gl);
return {
position: positionBuffer,
textureCoord: textureCoordBuffer,
indices: indexBuffer,
};
}
function initPositionBuffer(gl) {
// Create a buffer for the square's positions.
const positionBuffer = gl.createBuffer();
// Select the positionBuffer as the one to apply buffer
// operations to from here out.
gl.bindBuffer(gl.ARRAY_BUFFER, positionBuffer);
const positions = [
// Front face
-1.0, -1.0, 1.0, 1.0, -1.0, 1.0, 1.0, 1.0, 1.0, -1.0, 1.0, 1.0,
// Back face
-1.0, -1.0, -1.0, -1.0, 1.0, -1.0, 1.0, 1.0, -1.0, 1.0, -1.0, -1.0,
// Top face
-1.0, 1.0, -1.0, -1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, -1.0,
// Bottom face
-1.0, -1.0, -1.0, 1.0, -1.0, -1.0, 1.0, -1.0, 1.0, -1.0, -1.0, 1.0,
// Right face
1.0, -1.0, -1.0, 1.0, 1.0, -1.0, 1.0, 1.0, 1.0, 1.0, -1.0, 1.0,
// Left face
-1.0, -1.0, -1.0, -1.0, -1.0, 1.0, -1.0, 1.0, 1.0, -1.0, 1.0, -1.0,
];
// Now pass the list of positions into WebGL to build the
// shape. We do this by creating a Float32Array from the
// JavaScript array, then use it to fill the current buffer.
gl.bufferData(gl.ARRAY_BUFFER, new Float32Array(positions), gl.STATIC_DRAW);
return positionBuffer;
}
function initIndexBuffer(gl) {
const indexBuffer = gl.createBuffer();
gl.bindBuffer(gl.ELEMENT_ARRAY_BUFFER, indexBuffer);
// This array defines each face as two triangles, using the
// indices into the vertex array to specify each triangle's
// position.
// prettier-ignore
const indices = [
0, 1, 2, 0, 2, 3, // front
4, 5, 6, 4, 6, 7, // back
8, 9, 10, 8, 10, 11, // top
12, 13, 14, 12, 14, 15, // bottom
16, 17, 18, 16, 18, 19, // right
20, 21, 22, 20, 22, 23, // left
];
// Now send the element array to GL
gl.bufferData(
gl.ELEMENT_ARRAY_BUFFER,
new Uint16Array(indices),
gl.STATIC_DRAW,
);
return indexBuffer;
}
function initTextureBuffer(gl) {
const textureCoordBuffer = gl.createBuffer();
gl.bindBuffer(gl.ARRAY_BUFFER, textureCoordBuffer);
const textureCoordinates = [
// Front
1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0,
// Back
1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0,
// Top
1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0,
// Bottom
1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0,
// Right
1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0,
// Left
1.0, 0.0, 0.0, 0.0, 0.0, 1.0, 1.0, 1.0,
];
gl.bufferData(
gl.ARRAY_BUFFER,
new Float32Array(textureCoordinates),
gl.STATIC_DRAW,
);
return textureCoordBuffer;
}
function drawScene(gl, programInfo, buffers, texture, cubeRotation) {
gl.clearColor(0.0, 0.0, 0.0, 1.0); // Clear to black, fully opaque
gl.clearDepth(1.0); // Clear everything
gl.enable(gl.DEPTH_TEST); // Enable depth testing
gl.depthFunc(gl.LEQUAL); // Near things obscure far things
// Clear the canvas before we start drawing on it.
gl.clear(gl.COLOR_BUFFER_BIT | gl.DEPTH_BUFFER_BIT);
// Create a perspective matrix, a special matrix that is
// used to simulate the distortion of perspective in a camera.
// Our field of view is 35 degrees, with a width/height
// ratio that matches the display size of the canvas
// and we only want to see objects between 0.1 units
// and 100 units away from the camera.
const fieldOfView = (35 * Math.PI) / 180; // in radians
const aspect = gl.canvas.clientWidth / gl.canvas.clientHeight;
const zNear = 0.1;
const zFar = 100.0;
const projectionMatrix = mat4.create();
// note: glMatrix always has the first argument
// as the destination to receive the result.
mat4.perspective(projectionMatrix, fieldOfView, aspect, zNear, zFar);
// Set the drawing position to the "identity" point, which is
// the center of the scene.
const modelViewMatrix = mat4.create();
// Now move the drawing position a bit to where we want to
// start drawing the square.
mat4.translate(
modelViewMatrix, // destination matrix
modelViewMatrix, // matrix to translate
[-0.0, 0.0, -6.0],
); // amount to translate
mat4.rotate(
modelViewMatrix, // destination matrix
modelViewMatrix, // matrix to rotate
cubeRotation, // amount to rotate in radians
[0, 0, 1],
); // axis to rotate around (Z)
mat4.rotate(
modelViewMatrix, // destination matrix
modelViewMatrix, // matrix to rotate
cubeRotation * 0.7, // amount to rotate in radians
[0, 1, 0],
); // axis to rotate around (Y)
mat4.rotate(
modelViewMatrix, // destination matrix
modelViewMatrix, // matrix to rotate
cubeRotation * 0.3, // amount to rotate in radians
[1, 0, 0],
); // axis to rotate around (X)
setPositionAttribute(gl, buffers, programInfo);
setTextureAttribute(gl, buffers, programInfo);
gl.bindBuffer(gl.ELEMENT_ARRAY_BUFFER, buffers.indices);
// Tell WebGL to use our program when drawing
gl.useProgram(programInfo.program);
// Set the shader uniforms
gl.uniformMatrix4fv(
programInfo.uniformLocations.projectionMatrix,
false,
projectionMatrix,
);
gl.uniformMatrix4fv(
programInfo.uniformLocations.modelViewMatrix,
false,
modelViewMatrix,
);
// Tell WebGL we want to affect texture unit 0
gl.activeTexture(gl.TEXTURE0);
// Bind the texture to texture unit 0
gl.bindTexture(gl.TEXTURE_2D, texture);
// Tell the shader we bound the texture to texture unit 0
gl.uniform1i(programInfo.uniformLocations.uSampler, 0);
{
const vertexCount = 36;
const type = gl.UNSIGNED_SHORT;
const offset = 0;
gl.drawElements(gl.TRIANGLES, vertexCount, type, offset);
}
}
// Tell WebGL how to pull out the positions from the position
// buffer into the vertexPosition attribute.
function setPositionAttribute(gl, buffers, programInfo) {
const numComponents = 3; // pull out 2 values per iteration
const type = gl.FLOAT; // the data in the buffer is 32bit floats
const normalize = false; // don't normalize
const stride = 0; // how many bytes to get from one set of values to the next
// 0 = use type and numComponents above
const offset = 0; // how many bytes inside the buffer to start from
gl.bindBuffer(gl.ARRAY_BUFFER, buffers.position);
gl.vertexAttribPointer(
programInfo.attribLocations.vertexPosition,
numComponents,
type,
normalize,
stride,
offset,
);
gl.enableVertexAttribArray(programInfo.attribLocations.vertexPosition);
}
// tell webgl how to pull out the texture coordinates from buffer
function setTextureAttribute(gl, buffers, programInfo) {
const num = 2; // every coordinate composed of 2 values
const type = gl.FLOAT; // the data in the buffer is 32-bit float
const normalize = false; // don't normalize
const stride = 0; // how many bytes to get from one set to the next
const offset = 0; // how many bytes inside the buffer to start from
gl.bindBuffer(gl.ARRAY_BUFFER, buffers.textureCoord);
gl.vertexAttribPointer(
programInfo.attribLocations.textureCoord,
num,
type,
normalize,
stride,
offset,
);
gl.enableVertexAttribArray(programInfo.attribLocations.textureCoord);
}

View file

@ -0,0 +1,379 @@
# HTML-in-Canvas
This is a proposal for using 2D and 3D `<canvas>` to customize the rendering of HTML content.
## Status
This is a living explainer which is continuously updated as we receive feedback.
The APIs described here are implemented behind a flag in Chromium and can be enabled with `chrome://flags/#canvas-draw-element`.
## Motivation
There is no web API to easily render complex layouts of text and other content into a `<canvas>`. As a result, `<canvas>`-based content suffers in accessibility, internationalization, performance, and quality.
### Use cases
* **Styled, Laid Out Content in Canvas.** Theres a strong need for better styled text support in Canvas. Examples include chart components (legend, axes, etc.), rich content boxes in creative tools, and in-game menus.
* **Accessibility Improvements.** There is currently no guarantee that the canvas fallback content used for `<canvas>` accessibility always matches the rendered content, and such fallback content can be hard to generate. With this API, elements drawn into the canvas will match their corresponding canvas fallback.
* **Composing HTML Elements with Effects.** A limited set of CSS effects, such as filters, backdrop-filter, and mix-blend-mode are already available, but there is a desire to use general WebGL shaders with HTML.
* **HTML Rendering in a 3D Context.** 3D aspects of sites and games need to render rich 2D content into surfaces within a 3D scene.
* **Media Export.** There's a need to export HTML content as images or video.
## Proposed solution
The solution introduces three main primitives: an attribute to opt-in canvas elements, methods to draw child elements into the canvas, and an event which fires to handle updates.
### 1. The `layoutsubtree` attribute
The `layoutsubtree` attribute on a `<canvas>` element opts in canvas descendants to layout and participate in hit testing. It causes the direct children of the `<canvas>` to have a stacking context, become a containing block for all descendants, and have paint containment. Canvas element children behave as if they are visible, but their rendering is not visible to the user unless and until they are explicitly drawn into the canvas via a call to `drawElementImage()` (see below).
### 2. `drawElementImage` (and WebGL/WebGPU equivalents)
The `drawElementImage()` method draws a child of the canvas into the canvas, and returns a transform that can be applied to `element.style.transform` to align its DOM location with its drawn location. A snapshot of the rendering of all children of the canvas is recorded just prior to the `paint` event. When called during the `paint` event, `drawElementImage()` will draw the child as it would appear in the current frame. When called outside the `paint` event, the previous frame's snapshot is used. An exception is thrown if `drawElementImage()` is called with a child before an initial snapshot has been recorded.
**Requirements & Constraints:**
* `layoutsubtree` must be specified on the `<canvas>` in the most recent rendering update.
* The `element` must be a direct child of the `<canvas>` in the most recent rendering update.
* The `element` must have generated boxes (i.e., not `display: none`) in the most recent rendering update.
* **Transforms:** The canvas's current transformation matrix is applied when drawing into the canvas. CSS transforms on the source `element` are **ignored** for drawing (but continue to affect hit testing/accessibility, see below).
* **Clipping:** Overflowing content (both layout and ink overflow) is clipped to the element's border box.
* **Sizing:** The optional `width`/`height` arguments specify a destination rect in canvas coordinates. If omitted, the `width`/`height` arguments default to sizing the element so that it has the same on-screen size and proportion in canvas coordinates as it does outside the canvas.
**WebGL/WebGPU Support:**
Similar methods are added for 3D contexts: `WebGLRenderingContext.texElementImage2D` and `copyElementImageToTexture`.
### 3. The `paint` event
A `paint` event is added to `canvas` elements and fires if the rendering of any canvas children has changed. This event fires just after intersection observer steps have run during [update-the-rendering](https://html.spec.whatwg.org/#update-the-rendering). The event contains a list of the canvas children which have changed. Because CSS transforms on canvas children are ignored for rendering, changing the transform does not cause the `paint` event to fire in the next frame. Canvas drawing commands made in the `paint` event will appear in the current frame, but DOM changes made in the `paint` event will not show up until the subsequent frame. If there are multiple `<canvas>` elements, the `paint` event fires in _reverse_ tree order which ensures that descendants fire `paint` before ancestors.
To support application patterns which update every frame, a new `requestPaint()` function is added which will cause the `paint` event to fire once, even if no children have changed (analagous to `requestAnimationFrame()`).
### 4. `captureElementImage`
To support `OffscreenCanvas` in workers, a snapshot of an element can be captured as an `ElementImage` snapshot using `canvas.captureElementImage(element)`. These objects can be transferred to a worker and drawn to an `OffscreenCanvas`.
### Synchronization
Browser features like hit testing, intersection observer, and accessibility rely on an element's DOM location. To ensure these work, the element's `transform` property should be updated so that the DOM location matches the drawn location.
<details>
<summary>Calculating a CSS transform to match a drawn location</summary>
The general formula for the CSS transform is:
<div align="center">$$T_{\text{origin}}^{-1} \cdot S_{\text{css} \to \text{grid}}^{-1} \cdot T_{\text{draw}} \cdot S_{\text{css} \to \text{grid}} \cdot T_{\text{origin}} $$</div>
Where:
* $$T_{\text{draw}}$$: Transform used to draw the element in the canvas grid coordinate system.
For `drawElementImage`, this is $$CTM \cdot T_{(\text{x}, \text{y})} \cdot S_{(\text{destScale})}$$, where $$CTM$$ is the Current Transformation Matrix, $$T_{(\text{x}, \text{y})}$$ is a translation from the x and y arguments, and $$S_{(\text{destScale})}$$ is a scale from the width and height arguments.
* $$T_{\text{origin}}$$: Translation matrix of the element's computed `transform-origin`.
* $$S_{\text{css} \to \text{grid}}$$: Scaling matrix converting CSS pixels to Canvas Grid pixels.
</details>
To assist with synchronization, `drawElementImage()` returns the CSS transform which can be applied to the element to keep its location synchronized. For 3D contexts, the `getElementTransform(element, drawTransform)` helper method is provided which returns the CSS transform, provided a general transformation matrix.
The transform used to draw the element on the worker thread needs to be synced back to the DOM, and can simply be `postMessage()`'d back to the main thread if the position is static. If the position is dynamic, an alternative is to calculate the position on the main thread and update `element.style.transform` at the same time that the `ElementImage` objects is sent to the worker thread.
### Basic Example
<img width="250" height="38" alt="a screenshot showing a form element with a blinking cursor" src="https://github.com/user-attachments/assets/acbdd231-3259-4819-b57e-32e29c460fc9" />
```html
<canvas id="canvas" style="width: 400px; height: 200px;" layoutsubtree>
<form id="form_element">
<label for="name">name:</label>
<input id="name">
</form>
</canvas>
<script>
const ctx = document.getElementById('canvas').getContext('2d');
canvas.onpaint = () => {
ctx.reset();
const transform = ctx.drawElementImage(form_element, 100, 0);
form_element.style.transform = transform.toString();
};
// Size the canvas grid to match the device scale factor to prevent blurriness.
const observer = new ResizeObserver(([entry]) => {
canvas.width = entry.devicePixelContentBoxSize[0].inlineSize;
canvas.height = entry.devicePixelContentBoxSize[0].blockSize;
});
observer.observe(canvas, {box: 'device-pixel-content-box'});
</script>
```
### OffscreenCanvas Example
In this example, `OffscreenCanvas` in a worker is used. The `canvas` child form is captured as an `ElementImage` object in the `paint` event and transferred to the worker for painting.
```html
<!DOCTYPE html>
<canvas id="canvas" style="width: 400px; height: 200px;" layoutsubtree>
<form id="form_element">
<label for="name">name:</label>
<input id="name">
</form>
</canvas>
<script>
const workerCode = `
let ctx;
self.onmessage = (e) => {
if (e.data.canvas) {
ctx = e.data.canvas.getContext('2d');
}
if (e.data.width && e.data.height) {
ctx.canvas.width = e.data.width;
ctx.canvas.height = e.data.height;
}
if (e.data.elementImage) {
ctx.reset();
const transform = ctx.drawElementImage(e.data.elementImage, 100, 0);
self.postMessage({transform: transform});
}
};
`;
const worker = new Worker(URL.createObjectURL(new Blob([workerCode])));
const offscreen = canvas.transferControlToOffscreen();
worker.postMessage({ canvas: offscreen }, [offscreen]);
canvas.onpaint = (event) => {
const elementImage = canvas.captureElementImage(form_element)
worker.postMessage({ elementImage: elementImage }, [elementImage]);
};
// Synchronize the element's CSS transform to match its drawn location.
worker.onmessage = ({data}) => {
form_element.style.transform = data.transform.toString();
};
// Size the canvas grid to match the device scale factor to prevent blurriness.
const observer = new ResizeObserver(([entry]) => {
worker.postMessage({
width: entry.devicePixelContentBoxSize[0].inlineSize,
height: entry.devicePixelContentBoxSize[0].blockSize
});
canvas.requestPaint();
});
observer.observe(canvas, { box: 'device-pixel-content-box' });
</script>
```
### IDL changes
```idl
partial interface HTMLCanvasElement {
[CEReactions, Reflect] attribute boolean layoutSubtree;
attribute EventHandler onpaint;
void requestPaint();
ElementImage captureElementImage(Element element);
DOMMatrix getElementTransform((Element or ElementImage) element, DOMMatrix drawTransform);
};
partial interface OffscreenCanvas {
DOMMatrix getElementTransform((Element or ElementImage) element, DOMMatrix drawTransform);
};
interface mixin CanvasDrawElementImage {
DOMMatrix drawElementImage((Element or ElementImage) element,
unrestricted double dx, unrestricted double dy);
DOMMatrix drawElementImage((Element or ElementImage) element,
unrestricted double dx, unrestricted double dy,
unrestricted double dwidth, unrestricted double dheight);
DOMMatrix drawElementImage((Element or ElementImage) element,
unrestricted double sx, unrestricted double sy,
unrestricted double swidth, unrestricted double sheight,
unrestricted double dx, unrestricted double dy);
DOMMatrix drawElementImage((Element or ElementImage) element,
unrestricted double sx, unrestricted double sy,
unrestricted double swidth, unrestricted double sheight,
unrestricted double dx, unrestricted double dy,
unrestricted double dwidth, unrestricted double dheight);
};
CanvasRenderingContext2D includes CanvasDrawElementImage;
OffscreenCanvasRenderingContext2D includes CanvasDrawElementImage;
dictionary WebGLCopyElementImageConfig {
GLfloat sx;
GLfloat sy;
GLfloat swidth;
GLfloat sheight;
GLsizei width;
GLsizei height;
};
partial interface WebGLRenderingContext {
void texElementImage2D(GLenum target, GLenum internalformat,
(Element or ElementImage) element,
optional WebGLCopyElementImageConfig config = {});
};
dictionary GPUCopyElementImageDestination {
required GPUImageCopyTextureTagged destination;
GPUIntegerCoordinate width;
GPUIntegerCoordinate height;
};
dictionary GPUCopyElementImageSource {
required (Element or ElementImage) source;
float sx;
float sy;
float swidth;
float sheight;
};
partial interface GPUQueue {
void copyElementImageToTexture(GPUCopyElementImageSource source,
GPUCopyElementImageDestination destination);
}
[Exposed=Window]
interface PaintEvent : Event {
constructor(DOMString type, optional PaintEventInit eventInitDict);
readonly attribute FrozenArray<Element> changedElements;
};
dictionary PaintEventInit : EventInit {
sequence<Element> changedElements = [];
};
[Exposed=(Window,Worker), Transferable]
interface ElementImage {
readonly attribute double width;
readonly attribute double height;
undefined close();
};
```
## Demos
#### [Live demo](https://wicg.github.io/html-in-canvas/Examples/complex-text.html) ([source](Examples/complex-text.html)) using the `drawElementImage` API to draw rotated complex text.
<img width="640" height="320" alt="screenshot showing rotated, complex text drawn into canvas" src="https://github.com/user-attachments/assets/3ef73e0f-9119-49de-bf84-dfb3a4f5d77c" />
#### [Live demo](https://wicg.github.io/html-in-canvas/Examples/pie-chart.html) ([source](Examples/pie-chart.html)) using the `drawElementImage` API to draw a pie chart with multi-line labels.
<img width="640" height="320" alt="screenshot showing a pie chart" src="https://github.com/user-attachments/assets/887eefa2-ffc0-49d6-914b-987b05ccb45d" />
#### [Live demo](https://wicg.github.io/html-in-canvas/Examples/webgpu-jelly-slider/) ([source](Examples/webgpu-jelly-slider)) using the WebGPU `copyElementImageToTexture` API to draw a div under a jelly slider.
<img width="640" height="320" alt="screenshot showing a range slider with a jelly effect" src="https://github.com/user-attachments/assets/86ecb8b8-4d3b-49b0-8aa0-5f2df5674045" />
#### [Live demo](https://wicg.github.io/html-in-canvas/Examples/webGL.html) ([source](Examples/webGL.html)) using the WebGL `texElementImage2D` API to draw HTML onto a 3D cube.
<img width="640" height="320" alt="screenshot showing html content on a 3D cube" src="https://github.com/user-attachments/assets/689fefe3-56d9-4ae9-b386-32a01ebb0117" />
A demo of the same thing using an experimental extension of [three.js](https://threejs.org/) is [here](https://raw.githack.com/mrdoob/three.js/htmltexture/examples/webgl_materials_texture_html.html). Further instructions and context are [here](https://github.com/mrdoob/three.js/pull/31233).
#### [Live demo](https://wicg.github.io/html-in-canvas/Examples/text-input.html) ([source](Examples/text-input.html)) of interactive content in canvas.
<img width="640" height="320" alt="screenshot showing a form drawn into canvas" src="https://github.com/user-attachments/assets/be2d098f-17ae-4982-a0f9-a069e3c2d1d5" />
## Read-back-allowed rendering
The `drawElementImage()` method and any other methods that draw element image snapshots, as well as the paint event, must not reveal any security- or privacy-sensitive information that isn't otherwise observable to author code. This concept is called read-back-allowed rendering because it makes it possible to allow pixel read-back, which is always possible with WebGL and WebGPU.
Both painting (via canvas pixel readbacks or timing attacks) and invalidation (via `onpaint`) have the potential to leak sensitive information, and this is prevented by excluding sensitive information when painting and invalidating.
Sensitive information includes:
* Cross-origin data in [embedded content](https://html.spec.whatwg.org/#embedded-content-category) (e.g., `<iframe>`, `<img>`), [`<url>`](https://drafts.csswg.org/css-values-4/#url-value) references (e.g., `background-image`, `clip-path`), `<canvas>` elements tained with cross-origin data, and [SVG](https://svgwg.org/svg2-draft/single-page.html#types-InterfaceSVGURIReference) (e.g., `<use>`, `<pattern>`, `<feImage>`). Note that same-origin iframes would still paint, but cross-origin content in them would not.
* System colors, themes, or preferences.
* Spelling and grammar markers.
* Visited link information.
* Pending form autofill information not otherwise available to JavaScript.
* Subpixel text anti-aliasing.
* User preferences for caption and subtitle selection and appearance.
* IME pop-ups and distinctive IME text formatting.
The following new information is not considered sensitive:
* Search text (find-in-page) and text-fragment (fragment url) markers.
* Scrollbar and form element appearance (these are already detectable in Blink and WebKit through [foreignObject](https://jsfiddle.net/progers/qhawnyeu)).
* Caret blink rate.
* forced-colors (this information is already available to javascript using the `forced-colors` media query and system colors).
## Developer Trial (dev trial) Information
The HTML-in-Canvas features may be enabled with `chrome://flags/#canvas-draw-element` in Chrome Canary.
We are most interested in feedback on the following topics:
* What content works, and what fails? Which failure modes are most important to fix?
* How does the feature interact with accessibility features? How can accessibility support be improved?
Please file bugs or design issues [here](https://github.com/WICG/html-in-canvas/issues/new).
## Alternatives considered: `paint` event timing
A new `paint` event is needed to give developers an opportunity to update their canvas rendering in response to paint changes. This is integrated into [update the rendering](https://html.spec.whatwg.org/#update-the-rendering) so that canvas updates can occur in sync with the DOM.
There are several opportunities in the [update the rendering](https://html.spec.whatwg.org/#update-the-rendering) steps where the `paint` event could fire:
* 14\. Run animation frame callbacks.
* 16.2.1\. Recalculate styles and update layout.
* 16.2.6\. Deliver resize observers, looping back to 16.2.1 if needed.
* _Option A: Fire `paint` at resize observer timing, looping back to 16.2.1 if needed._
* 19\. Run the update intersection observations steps.
* Paint, where the painted output of elements is calculated. This is not an explicitly named step in [update the rendering](https://html.spec.whatwg.org/#update-the-rendering).
* _Option B: Fire `paint` immediately after Paint, looping back to 16.2.1 if needed._
* _Option C: Fire `paint` immediately after Paint._
* Commit / thread handoff, where the painted output is sent to another process. This is not an explicitly named step in [update the rendering](https://html.spec.whatwg.org/#update-the-rendering).
Note that the `paint` event is the new event on canvas introduced in this proposal, and the Paint step is the existing operation that browsers perform to record the painted output of the rendering tree following [paint order](https://drafts.csswg.org/css-position-4/#painting-order).
#### Option A: Fire `paint` at resize observer timing, looping back to 16.2.1 if needed.
Similar to resize observer, a looping approach is needed to handle cases where the paint event performs modifications (including of elements outside the canvas). There is no mechanism for preventing arbitrary javascript from modifying the DOM. Looping will be required for more conditions than those required by ResizeObserver, such as background style changes. A downside of looping is that the user's canvas code may need to run multiple times per frame.
One option is to do a synchronous Paint step to snapshot the painted output of canvas children. A downside of this approach is that the Paint step may be expensive to run, and may need to be run multiple times. This approach has unique implementation challenges in Gecko, and possibly other engines, due to architectural limitations.
A second option is to not run the Paint step synchronously, but instead record a placeholder representing how an element will appear on the next rendering update (see [design](https://docs.google.com/document/d/1YaHCxYqE4uQc4-UTWo4a5pHt2I2MutlwJtsnj5ljEkM/edit?usp=sharing)). This model can be implemented with 2D canvas by buffering the canvas commands until the next Paint step. When the next Paint step occurs, the placeholders would then be replaced with the actual rendering. Canvas operations such as `getImageData` require synchronous flushing of the canvas command buffer and would need to show blank or stale data for the placeholders. Unfortunately, this approach has a fundamental flaw for WebGL because many APIs require flushing (e.g., `getError()`, see callsites of [WaitForCmd](https://source.chromium.org/chromium/chromium/src/+/main:gpu/command_buffer/client/implementation_base.h;drc=b3eab4fd06ddbeee84b37224f4cc9d78094fc2f7;l=102)), and calling any of these APIs would result in a deadlock or inconsistent rendering. Therefore, we must run the `paint` event at a time where we have the complete painted display list of an element already available.
#### Option B: Fire `paint` immediately after Paint, looping back to 16.2.1 if needed.
See above for the reasons and downsides of looping when there are modifications made during the `paint` event.
The upside of option B as compared with option A is that it does not require partial Paint of canvas children. An additional downside is that even more steps of [update the rendering](https://html.spec.whatwg.org/#update-the-rendering) need to run on each iteration of the loop.
#### Option C: Fire `paint` immediately after Paint.
This is the design approach taken for the API.
This approach only runs `paint` once per frame, similar to the browser's own Paint step. To solve the issue of javascript being able to perform arbitrary modifications, it is important to ensure that before `paint` runs we have locked in the contents of the rendering update, except for one intentional carve-out: the drawn content of the canvas. DOM invalidations that may occur in the `paint` event apply to the subsequent frame, not the current frame.
## Alternatives considered: Supporting threaded effects with worker threads
To support threaded effects, we explored a [design](https://docs.google.com/document/d/1TWe6HP7HMn6y-XnNKppIhgf9FtuXJ6LPgenJJxZDjzg/edit?tab=t.0) where canvas children "snapshots" are sent to a worker thread. In response to threaded scrolling and animations, the worker thread could then render the most up-to-date rendering of the snapshots into OffscreenCanvas. This model requires that javascript can be synchronously called on scroll and animation updates, which is difficult for architectures that perform threaded scroll updates in a restricted process.
## Future considerations: Supporting threaded effects with an auto-updating canvas
To support threaded effects such as scrolling and animations, we are considering a future "auto-updating canvas" mode.
In this model, `drawElementImage` records a placeholder representing the latest rendering. Canvas retains a command buffer which can be automatically replayed following every scroll or animation update. This allows the canvas to re-rasterize with updated placeholders that incorporate threaded scrolling and animations, without needing to block on script. This would enable visual effects that stay perfectly in sync with native scrolling or animations within the canvas, independent of the main thread. This design is viable for 2D contexts, and may be viable for WebGPU with some small API additions.
## Other documents
* [Security and Privacy Questionnaire](./security-privacy-questionnaire.md)
## Authors
* [Philip Rogers](mailto:pdr@chromium.org)
* [Stephen Chenney](mailto:schenney@igalia.com)
* [Chris Harrelson](mailto:chrishtr@chromium.org)
* [Philip Jägenstedt](mailto:foolip@chromium.org)
* [Khushal Sagar](mailto:khushalsagar@chromium.org)
* [Vladimir Levin](mailto:vmpstr@chromium.org)
* [Fernando Serboncini](mailto:fserb@chromium.org)

View file

@ -0,0 +1,182 @@
<!DOCTYPE html>
<html lang="en">
<head>
<title>three.js webgl - materials - html texture</title>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, user-scalable=no, minimum-scale=1.0, maximum-scale=1.0">
<meta property="og:title" content="three.js webgl - materials - html texture">
<meta property="og:type" content="website">
<meta property="og:url" content="https://threejs.org/examples/webgl_materials_texture_html.html">
<meta property="og:image" content="https://threejs.org/examples/screenshots/webgl_materials_texture_html.jpg">
<link type="text/css" rel="stylesheet" href="main.css">
<style>
body {
background-color: #ffffff;
}
#draw_element {
width: 600px;
background-color: #aaaaaa;
color: #000000;
font-family: sans-serif;
font-size: 30px;
line-height: 1.5;
text-align: center;
padding: 30px;
/* border: 10px solid #cccccc; */
}
#draw_element img {
animation: swing 1s ease-in-out infinite alternate;
}
#draw_element input[type="text"] {
font-size: 24px;
padding: 8px 12px;
border: 2px solid #888;
border-radius: 6px;
width: 80%;
margin-top: 10px;
}
#draw_element button {
font-size: 24px;
padding: 8px 20px;
margin-top: 10px;
border: none;
border-radius: 6px;
background-color: #4CAF50;
color: white;
cursor: pointer;
}
#draw_element button:hover {
background-color: #2196F3;
}
@keyframes swing {
from { transform: rotate(-15deg); }
to { transform: rotate(15deg); }
}
</style>
</head>
<body>
<div id="info">
<a href="https://threejs.org" target="_blank" rel="noopener">three.js</a> - webgl - HTMLTexture
</div>
<script type="importmap">
{
"imports": {
"three": "../build/three.module.js",
"three/addons/": "./jsm/",
"three-html-render/polyfill": "https://cdn.jsdelivr.net/npm/three-html-render/dist/polyfill.mjs"
}
}
</script>
<script type="module">
import * as THREE from 'three';
import { installHtmlInCanvasPolyfill } from 'three-html-render/polyfill';
import { RoundedBoxGeometry } from 'three/addons/geometries/RoundedBoxGeometry.js';
import { RoomEnvironment } from 'three/addons/environments/RoomEnvironment.js';
import { InteractionManager } from 'three/addons/interaction/InteractionManager.js';
if ( ! ( 'requestPaint' in HTMLCanvasElement.prototype ) ) {
installHtmlInCanvasPolyfill();
info.innerHTML += '<br><a href="https://github.com/WICG/html-in-canvas" target="_blank">HTML-in-Canvas API</a> not available. Using <a href="https://github.com/repalash/three-html-render" target="_blank">polyfill</a>.';
}
let camera, scene, renderer, mesh, interactions;
init();
function init() {
renderer = new THREE.WebGLRenderer( { antialias: true } );
renderer.toneMapping = THREE.NeutralToneMapping;
renderer.setPixelRatio( window.devicePixelRatio );
renderer.setSize( window.innerWidth, window.innerHeight );
renderer.setAnimationLoop( animate );
document.body.appendChild( renderer.domElement );
camera = new THREE.PerspectiveCamera( 50, window.innerWidth / window.innerHeight, 1, 2000 );
camera.position.z = 500;
scene = new THREE.Scene();
scene.background = new THREE.Color( 0xaaaaaa );
scene.environment = new THREE.PMREMGenerator( renderer ).fromScene( new RoomEnvironment(), 0.02 ).texture;
// HTML element
const element = document.createElement( 'div' );
element.id = 'draw_element';
element.innerHTML = `
Hello world!<br>I'm multi-line, <b>formatted</b>,
rotated text with emoji (&#128512;), RTL text
<span dir=rtl>من فارسی صحبت میکنم</span>,
vertical text,
<p style="writing-mode: vertical-rl;">
这是垂直文本
</p>
an inline image (<img width="150" src="textures/758px-Canestra_di_frutta_(Caravaggio).jpg">), and
<svg width="50" height="50">
<circle cx="25" cy="25" r="20" fill="green" />
<text x="25" y="30" font-size="15" text-anchor="middle" fill="#fff">
SVG
</text>
</svg>!
<br>
<input type="text" placeholder="Type here...">
<button>Click me</button>
`;
const geometry = new RoundedBoxGeometry( 200, 200, 200, 10, 10 );
const material = new THREE.MeshStandardMaterial( { roughness: 0, metalness: 0.5 } );
material.map = new THREE.HTMLTexture( element );
mesh = new THREE.Mesh( geometry, material );
scene.add( mesh );
// Interaction
interactions = new InteractionManager();
interactions.connect( renderer, camera );
interactions.add( mesh );
// Button click handler
element.querySelector( 'button' ).addEventListener( 'click', function () {
this.textContent = 'Clicked!';
} );
window.addEventListener( 'resize', onWindowResize );
}
function onWindowResize() {
camera.aspect = window.innerWidth / window.innerHeight;
camera.updateProjectionMatrix();
renderer.setSize( window.innerWidth, window.innerHeight );
}
function animate( time ) {
mesh.rotation.x = Math.sin( time * 0.0005 ) * 0.5;
mesh.rotation.y = Math.cos( time * 0.0008 ) * 0.5;
interactions.update();
renderer.render( scene, camera );
}
</script>
</body>
</html>

View file

@ -0,0 +1,74 @@
import { Texture } from './Texture.js';
/**
* Creates a texture from an HTML element.
*
* This is almost the same as the base texture class, except that it sets {@link Texture#needsUpdate}
* to `true` immediately and listens for the parent canvas's paint events to trigger updates.
*
* @augments Texture
*/
class HTMLTexture extends Texture {
/**
* Constructs a new texture.
*
* @param {HTMLElement} [element] - The HTML element.
* @param {number} [mapping=Texture.DEFAULT_MAPPING] - The texture mapping.
* @param {number} [wrapS=ClampToEdgeWrapping] - The wrapS value.
* @param {number} [wrapT=ClampToEdgeWrapping] - The wrapT value.
* @param {number} [magFilter=LinearFilter] - The mag filter value.
* @param {number} [minFilter=LinearMipmapLinearFilter] - The min filter value.
* @param {number} [format=RGBAFormat] - The texture format.
* @param {number} [type=UnsignedByteType] - The texture type.
* @param {number} [anisotropy=Texture.DEFAULT_ANISOTROPY] - The anisotropy value.
*/
constructor( element, mapping, wrapS, wrapT, magFilter, minFilter, format, type, anisotropy ) {
super( element, mapping, wrapS, wrapT, magFilter, minFilter, format, type, anisotropy );
/**
* This flag can be used for type testing.
*
* @type {boolean}
* @readonly
* @default true
*/
this.isHTMLTexture = true;
this.generateMipmaps = false;
this.needsUpdate = true;
const parent = element ? element.parentNode : null;
if ( parent !== null && 'requestPaint' in parent ) {
parent.onpaint = () => {
this.needsUpdate = true;
};
parent.requestPaint();
}
}
dispose() {
const parent = this.image ? this.image.parentNode : null;
if ( parent !== null && 'onpaint' in parent ) {
parent.onpaint = null;
}
super.dispose();
}
}
export { HTMLTexture };

View file

@ -0,0 +1,938 @@
import * as sdf from '@typegpu/sdf';
import tgpu, { common, d, std } from 'typegpu';
import { randf } from '@typegpu/noise';
import { Slider } from './slider.ts';
import { CameraController } from './camera.ts';
import {
DirectionalLight,
HitInfo,
LineInfo,
ObjectType,
Ray,
rayMarchLayout,
sampleLayout,
SdfBbox,
} from './dataTypes.ts';
import {
beerLambert,
createBackgroundTexture,
createTextures,
fresnelSchlick,
intersectBox,
} from './utils.ts';
import { TAAResolver } from './taa.ts';
import {
AMBIENT_COLOR,
AMBIENT_INTENSITY,
AO_BIAS,
AO_INTENSITY,
AO_RADIUS,
AO_STEPS,
JELLY_IOR,
JELLY_SCATTER_STRENGTH,
LINE_HALF_THICK,
LINE_RADIUS,
MAX_DIST,
MAX_STEPS,
SPECULAR_INTENSITY,
SPECULAR_POWER,
SURF_DIST,
} from './constants.ts';
const root = await tgpu.init({
device: {
optionalFeatures: ['timestamp-query'],
},
});
const presentationFormat = navigator.gpu.getPreferredCanvasFormat();
const canvas = document.querySelector('canvas') as HTMLCanvasElement;
const context = root.configureContext({ canvas, alphaMode: 'premultiplied' });
const NUM_POINTS = 17;
const slider = new Slider(root, d.vec2f(-1, 0), d.vec2f(0.9, 0), NUM_POINTS, -0.03);
const bezierTexture = slider.bezierTexture.createView();
const bezierBbox = slider.bbox;
let qualityScale = 1.0;
let [width, height] = [canvas.width * qualityScale, canvas.height * qualityScale];
let textures = createTextures(root, width, height);
let backgroundTexture = createBackgroundTexture(root, width, height);
const sliderElement = document.getElementById('slider') as HTMLInputElement;
const valueElement = document.getElementById('value') as HTMLDivElement;
const valueRawTexture = root.device.createTexture({
size: [width, height, 1],
format: 'rgba8unorm',
usage: GPUTextureUsage.TEXTURE_BINDING | GPUTextureUsage.COPY_DST | GPUTextureUsage.RENDER_ATTACHMENT
});
const valueTextureView = valueRawTexture.createView();
// Return a number from 0...100 as a string Zero percent...One hundred percent.
function getPercentString(n: number): string {
if (n === 100) return "One-hundred %";
const ones: string[] = [
"Zero", "One", "Two", "Three", "Four", "Five", "Six", "Seven", "Eight", "Nine",
"Ten", "Eleven", "Twelve", "Thirteen", "Fourteen", "Fifteen", "Sixteen", "Seventeen", "Eighteen", "Nineteen"
];
const tens: string[] = [
"", "", "Twenty", "Thirty", "Forty", "Fifty", "Sixty", "Seventy", "Eighty", "Ninety"
];
// Handle 0 through 19
if (n < 20) {
return `${ones[n]} %`;
}
// Handle 20 through 99
const tensWord: string = tens[Math.floor(n / 10)];
const onesWord: string = n % 10 === 0 ? "" : `-${ones[n % 10].toLowerCase()}`;
return `${tensWord}${onesWord} %`;
}
let targetMouseX = 0.9;
let currentMouseX = 0.9;
sliderElement.addEventListener('input', () => {
const t = Number(sliderElement.value) / 100.0;
targetMouseX = t * 1.9 - 1.0;
valueElement.textContent = getPercentString(Number(sliderElement.value));
(canvas as any).requestPaint();
});
valueElement.textContent = getPercentString(Number(sliderElement.value));
const filteringSampler = root['~unstable'].createSampler({
magFilter: 'linear',
minFilter: 'linear',
});
const camera = new CameraController(
root,
d.vec3f(0, 2.7, 1.9),
d.vec3f(0, 0, 0),
d.vec3f(0, 1, 0),
Math.PI / 4,
width,
height,
);
const cameraUniform = camera.cameraUniform;
const lightUniform = root.createUniform(DirectionalLight, {
direction: std.normalize(d.vec3f(0.19, -0.24, 0.75)),
color: d.vec3f(1, 1, 1),
});
const jellyColorUniform = root.createUniform(d.vec4f, d.vec4f(1.0, 0.45, 0.075, 1.0));
const jellyScatterUniform = root.createUniform(d.f32, JELLY_SCATTER_STRENGTH);
const groundColorUniform = root.createUniform(d.vec3f, d.vec3f(1.0));
const groundTextColorUniform = root.createUniform(d.vec3f, d.vec3f(0.5));
const randomUniform = root.createUniform(d.vec2f);
const blurEnabledUniform = root.createUniform(d.u32);
const getRay = (ndc: d.v2f) => {
'use gpu';
const clipPos = d.vec4f(ndc.x, ndc.y, -1.0, 1.0);
const invView = cameraUniform.$.viewInv;
const invProj = cameraUniform.$.projInv;
const viewPos = invProj.mul(clipPos);
const viewPosNormalized = d.vec4f(viewPos.xyz.div(viewPos.w), 1.0);
const worldPos = invView.mul(viewPosNormalized);
const rayOrigin = invView.columns[3].xyz;
const rayDir = std.normalize(worldPos.xyz.sub(rayOrigin));
return Ray({
origin: rayOrigin,
direction: rayDir,
});
};
const getSliderBbox = () => {
'use gpu';
return SdfBbox({
left: d.f32(bezierBbox[3]),
right: d.f32(bezierBbox[1]),
bottom: d.f32(bezierBbox[2]),
top: d.f32(bezierBbox[0]),
});
};
const sdInflatedPolyline2D = (p: d.v2f) => {
'use gpu';
const bbox = getSliderBbox();
const uv = d.vec2f(
(p.x - bbox.left) / (bbox.right - bbox.left),
(bbox.top - p.y) / (bbox.top - bbox.bottom),
);
const clampedUV = std.saturate(uv);
const sampledColor = std.textureSampleLevel(bezierTexture.$, filteringSampler.$, clampedUV, 0);
const segUnsigned = sampledColor.x;
const progress = sampledColor.y;
const normal = sampledColor.zw;
return LineInfo({
t: progress,
distance: segUnsigned,
normal: normal,
});
};
const cap3D = (position: d.v3f) => {
'use gpu';
const endCap = slider.endCapUniform.$;
const secondLastPoint = d.vec2f(endCap.x, endCap.y);
const lastPoint = d.vec2f(endCap.z, endCap.w);
const angle = std.atan2(lastPoint.y - secondLastPoint.y, lastPoint.x - secondLastPoint.x);
const rot = d.mat2x2f(std.cos(angle), -std.sin(angle), std.sin(angle), std.cos(angle));
let pieP = position.sub(d.vec3f(secondLastPoint, 0));
pieP = d.vec3f(rot.mul(pieP.xy), pieP.z);
const hmm = sdf.sdPie(pieP.zx, d.vec2f(1, 0), LINE_HALF_THICK);
const extrudeEnd = sdf.opExtrudeY(pieP, hmm, 0.001) - LINE_RADIUS;
return extrudeEnd;
};
const sliderSdf3D = (position: d.v3f) => {
'use gpu';
const poly2D = sdInflatedPolyline2D(position.xy);
let finalDist = d.f32(0.0);
if (poly2D.t > 0.94) {
finalDist = cap3D(position);
} else {
const body = sdf.opExtrudeZ(position, poly2D.distance, LINE_HALF_THICK) - LINE_RADIUS;
finalDist = body;
}
return LineInfo({
t: poly2D.t,
distance: finalDist,
normal: poly2D.normal,
});
};
const GroundParams = {
groundThickness: 0.03,
groundRoundness: 0.02,
};
const rectangleCutoutDist = (position: d.v2f) => {
'use gpu';
const groundRoundness = GroundParams.groundRoundness;
return sdf.sdRoundedBox2d(
position,
d.vec2f(1 + groundRoundness, 0.2 + groundRoundness),
0.2 + groundRoundness,
);
};
const getMainSceneDist = (position: d.v3f) => {
'use gpu';
const groundThickness = GroundParams.groundThickness;
const groundRoundness = GroundParams.groundRoundness;
return sdf.opUnion(
sdf.sdPlane(position, d.vec3f(0, 1, 0), 0.06),
sdf.opExtrudeY(position, -rectangleCutoutDist(position.xz), groundThickness - groundRoundness) -
groundRoundness,
);
};
const sliderApproxDist = (position: d.v3f) => {
'use gpu';
const bbox = getSliderBbox();
const p = position.xy;
if (p.x < bbox.left || p.x > bbox.right || p.y < bbox.bottom || p.y > bbox.top) {
return 1e9;
}
const poly2D = sdInflatedPolyline2D(p);
const dist3D = sdf.opExtrudeZ(position, poly2D.distance, LINE_HALF_THICK) - LINE_RADIUS;
return dist3D;
};
const getSceneDist = (position: d.v3f) => {
'use gpu';
const mainScene = getMainSceneDist(position);
const poly3D = sliderSdf3D(position);
const hitInfo = HitInfo();
if (poly3D.distance < mainScene) {
hitInfo.distance = poly3D.distance;
hitInfo.objectType = ObjectType.SLIDER;
hitInfo.t = poly3D.t;
} else {
hitInfo.distance = mainScene;
hitInfo.objectType = ObjectType.BACKGROUND;
}
return hitInfo;
};
const getSceneDistForAO = (position: d.v3f) => {
'use gpu';
const mainScene = getMainSceneDist(position);
const sliderApprox = sliderApproxDist(position);
return std.min(mainScene, sliderApprox);
};
const sdfSlot = tgpu.slot<(pos: d.v3f) => number>();
const getNormalFromSdf = tgpu.fn(
[d.vec3f, d.f32],
d.vec3f,
)((position, epsilon) => {
'use gpu';
const k = d.vec3f(1, -1, 0);
const offset1 = k.xyy.mul(epsilon);
const offset2 = k.yyx.mul(epsilon);
const offset3 = k.yxy.mul(epsilon);
const offset4 = k.xxx.mul(epsilon);
const sample1 = offset1.mul(sdfSlot.$(position.add(offset1)));
const sample2 = offset2.mul(sdfSlot.$(position.add(offset2)));
const sample3 = offset3.mul(sdfSlot.$(position.add(offset3)));
const sample4 = offset4.mul(sdfSlot.$(position.add(offset4)));
const gradient = sample1.add(sample2).add(sample3).add(sample4);
return std.normalize(gradient);
});
const getNormalCapSdf = getNormalFromSdf.with(sdfSlot, cap3D);
const getNormalMainSdf = getNormalFromSdf.with(sdfSlot, getMainSceneDist);
const getNormalCap = (pos: d.v3f) => {
'use gpu';
return getNormalCapSdf(pos, 0.01);
};
const getNormalMain = (position: d.v3f) => {
'use gpu';
if (std.abs(position.z) > 0.22 || std.abs(position.x) > 1.02) {
return d.vec3f(0, 1, 0);
}
return getNormalMainSdf(position, 0.0001);
};
const getSliderNormal = (position: d.v3f, hitInfo: d.Infer<typeof HitInfo>) => {
'use gpu';
const poly2D = sdInflatedPolyline2D(position.xy);
const gradient2D = poly2D.normal;
const threshold = LINE_HALF_THICK * 0.85;
const absZ = std.abs(position.z);
const zDistance = std.max(
0,
((absZ - threshold) * LINE_HALF_THICK) / (LINE_HALF_THICK - threshold),
);
const edgeDistance = LINE_RADIUS - poly2D.distance;
const edgeContrib = 0.9;
const zContrib = 1.0 - edgeContrib;
const zDirection = std.sign(position.z);
const zAxisVector = d.vec3f(0, 0, zDirection);
const edgeBlendDistance = edgeContrib * LINE_RADIUS + zContrib * LINE_HALF_THICK;
const blendFactor = std.smoothstep(
edgeBlendDistance,
0.0,
zDistance * zContrib + edgeDistance * edgeContrib,
);
const normal2D = d.vec3f(gradient2D.xy, 0);
const blendedNormal = std.mix(zAxisVector, normal2D, blendFactor * 0.5 + 0.5);
let normal = std.normalize(blendedNormal);
if (hitInfo.t > 0.94) {
const ratio = (hitInfo.t - 0.94) / 0.02;
const fullNormal = getNormalCap(position);
normal = std.normalize(std.mix(normal, fullNormal, ratio));
}
return normal;
};
const getNormal = (position: d.v3f, hitInfo: d.Infer<typeof HitInfo>) => {
'use gpu';
if (hitInfo.objectType === ObjectType.SLIDER && hitInfo.t < 0.96) {
return getSliderNormal(position, hitInfo);
}
return std.select(
getNormalCap(position),
getNormalMain(position),
hitInfo.objectType === ObjectType.BACKGROUND,
);
};
const sqLength = (a: d.v3f) => {
'use gpu';
return std.dot(a, a);
};
const getFakeShadow = (position: d.v3f, lightDir: d.v3f): d.v3f => {
'use gpu';
const jellyColor = jellyColorUniform.$;
const endCapX = slider.endCapUniform.$.x;
if (position.y < -GroundParams.groundThickness) {
// Applying darkening under the ground (the shadow cast by the upper ground layer)
const fadeSharpness = 30;
const inset = 0.02;
const cutout = rectangleCutoutDist(position.xz) + inset;
const edgeDarkening = std.saturate(1 - cutout * fadeSharpness);
// Applying a slight gradient based on the light direction
const lightGradient = std.saturate(-position.z * 4 * lightDir.z + 1);
return d
.vec3f(1)
.mul(edgeDarkening)
.mul(lightGradient * 0.5);
} else {
const finalUV = d.vec2f(
(position.x - position.z * lightDir.x * std.sign(lightDir.z)) * 0.5 + 0.5,
1 - (-position.z / lightDir.z) * 0.5 - 0.2,
);
const data = std.textureSampleLevel(bezierTexture.$, filteringSampler.$, finalUV, 0);
// Normally it would be just data.y, but there transition is too sudden when the jelly is bunched up.
// To mitigate this, we transition into a position-based transition.
const jellySaturation = std.mix(0, data.y, std.saturate(position.x * 1.5 + 1.1));
const shadowColor = std.mix(d.vec3f(0, 0, 0), jellyColor.rgb, jellySaturation);
const contrast = 20 * std.saturate(finalUV.y) * (0.8 + endCapX * 0.2);
const shadowOffset = -0.3;
const featherSharpness = 10;
const uvEdgeFeather =
std.saturate(finalUV.x * featherSharpness) *
std.saturate((1 - finalUV.x) * featherSharpness) *
std.saturate((1 - finalUV.y) * featherSharpness) *
std.saturate(finalUV.y);
const influence = std.saturate((1 - lightDir.y) * 2) * uvEdgeFeather;
return std.mix(
d.vec3f(1),
std.mix(shadowColor, d.vec3f(1), std.saturate(data.x * contrast + shadowOffset)),
influence,
);
}
};
const calculateAO = (position: d.v3f, normal: d.v3f) => {
'use gpu';
let totalOcclusion = d.f32(0.0);
let sampleWeight = d.f32(1.0);
const stepDistance = AO_RADIUS / AO_STEPS;
for (let i = 1; i <= AO_STEPS; i++) {
const sampleHeight = stepDistance * d.f32(i);
const samplePosition = position.add(normal.mul(sampleHeight));
const distanceToSurface = getSceneDistForAO(samplePosition) - AO_BIAS;
const occlusionContribution = std.max(0.0, sampleHeight - distanceToSurface);
totalOcclusion += occlusionContribution * sampleWeight;
sampleWeight *= 0.5;
if (totalOcclusion > AO_RADIUS / AO_INTENSITY) {
break;
}
}
const rawAO = 1.0 - (AO_INTENSITY * totalOcclusion) / AO_RADIUS;
return std.saturate(rawAO);
};
const calculateLighting = (hitPosition: d.v3f, normal: d.v3f, rayOrigin: d.v3f) => {
'use gpu';
const lightDir = std.neg(lightUniform.$.direction);
const fakeShadow = getFakeShadow(hitPosition, lightDir);
const diffuse = std.max(std.dot(normal, lightDir), 0.0);
const viewDir = std.normalize(rayOrigin.sub(hitPosition));
const reflectDir = std.reflect(std.neg(lightDir), normal);
const specularFactor = std.max(std.dot(viewDir, reflectDir), 0) ** SPECULAR_POWER;
const specular = lightUniform.$.color.mul(specularFactor * SPECULAR_INTENSITY);
const baseColor = d.vec3f(0.9);
const directionalLight = baseColor.mul(lightUniform.$.color).mul(diffuse).mul(fakeShadow);
const ambientLight = baseColor.mul(AMBIENT_COLOR).mul(AMBIENT_INTENSITY);
const finalSpecular = specular.mul(fakeShadow);
return std.saturate(directionalLight.add(ambientLight).add(finalSpecular));
};
const applyAO = (litColor: d.v3f, hitPosition: d.v3f, normal: d.v3f) => {
'use gpu';
const ao = calculateAO(hitPosition, normal);
const finalColor = litColor.mul(ao);
return d.vec4f(finalColor, 1.0);
};
const rayMarchNoJelly = (rayOrigin: d.v3f, rayDirection: d.v3f) => {
'use gpu';
let distanceFromOrigin = d.f32();
let hit = d.f32();
for (let i = 0; i < 6; i++) {
const p = rayOrigin.add(rayDirection.mul(distanceFromOrigin));
hit = getMainSceneDist(p);
distanceFromOrigin += hit;
if (distanceFromOrigin > MAX_DIST || hit < SURF_DIST * 10) {
break;
}
}
if (distanceFromOrigin < MAX_DIST) {
return renderBackground(
rayOrigin,
rayDirection,
distanceFromOrigin,
std.select(d.f32(), 0.87, blurEnabledUniform.$ === 1),
).rgb;
}
return d.vec3f();
};
const renderPercentageOnGround = (hitPosition: d.v3f, center: d.v3f) => {
'use gpu';
const textWidth = 1.9;
const textHeight = 0.33;
if (
std.abs(hitPosition.x - center.x) > textWidth * 0.5 ||
std.abs(hitPosition.z - center.z) > textHeight * 0.5
) {
return d.vec4f();
}
const localX = hitPosition.x - center.x;
const localZ = hitPosition.z - center.z;
const uvX = (localX + textWidth * 0.5) / textWidth;
const uvZ = (localZ + textHeight * 0.5) / textHeight;
if (uvX < 0.0 || uvX > 1.0 || uvZ < 0.0 || uvZ > 1.0) {
return d.vec4f();
}
return std.textureSampleLevel(
rayMarchLayout.$.valueTexture,
filteringSampler.$,
d.vec2f(uvX, uvZ),
0,
);
};
const renderBackground = (
rayOrigin: d.v3f,
rayDirection: d.v3f,
backgroundHitDist: number,
offset: number,
) => {
'use gpu';
const hitPosition = rayOrigin.add(rayDirection.mul(backgroundHitDist));
const percentageSample = renderPercentageOnGround(
hitPosition,
d.vec3f(0, 0, 0),
);
let highlights = d.f32();
const highlightWidth = d.f32(1);
const highlightHeight = 0.2;
let offsetX = d.f32();
let offsetZ = d.f32(0.05);
const lightDir = lightUniform.$.direction;
const causticScale = 0.2;
offsetX -= lightDir.x * causticScale;
offsetZ += lightDir.z * causticScale;
const endCapX = slider.endCapUniform.$.x;
const sliderStretch = (endCapX + 1) * 0.5;
if (
std.abs(hitPosition.x + offsetX) < highlightWidth &&
std.abs(hitPosition.z + offsetZ) < highlightHeight
) {
const uvX_orig = ((hitPosition.x + offsetX + highlightWidth * 2) / highlightWidth) * 0.5;
const uvZ_orig = ((hitPosition.z + offsetZ + highlightHeight * 2) / highlightHeight) * 0.5;
const centeredUV = d.vec2f(uvX_orig - 0.5, uvZ_orig - 0.5);
const finalUV = d.vec2f(centeredUV.x, 1 - (std.abs(centeredUV.y - 0.5) * 2) ** 2 * 0.3);
const density = std.max(
0,
(std.textureSampleLevel(bezierTexture.$, filteringSampler.$, finalUV, 0).x - 0.25) * 8,
);
const fadeX = std.smoothstep(0, -0.2, hitPosition.x - endCapX);
const fadeZ = 1 - (std.abs(centeredUV.y - 0.5) * 2) ** 3;
const fadeStretch = std.saturate(1 - sliderStretch);
const edgeFade = std.saturate(fadeX) * std.saturate(fadeZ) * fadeStretch;
highlights = (density ** 3 * edgeFade * 3 * (1 + lightDir.z)) / 1.5;
}
const originYBound = std.saturate(rayOrigin.y + 0.01);
const posOffset = hitPosition.add(
d.vec3f(0, 1, 0).mul(offset * (originYBound / (1.0 + originYBound)) * (1 + randf.sample() / 2)),
);
const newNormal = getNormalMain(posOffset);
// Calculate fake bounce lighting
const jellyColor = jellyColorUniform.$;
const sqDist = sqLength(hitPosition.sub(d.vec3f(endCapX, 0, 0)));
const bounceLight = jellyColor.rgb.mul((1 / (sqDist * 15 + 1)) * 0.4);
const sideBounceLight = jellyColor.rgb
.mul((1 / (sqDist * 40 + 1)) * 0.3)
.mul(std.abs(newNormal.z));
const litColor = calculateLighting(posOffset, newNormal, rayOrigin);
const backgroundColor = applyAO(groundColorUniform.$.mul(litColor), posOffset, newNormal)
.add(d.vec4f(bounceLight, 0))
.add(d.vec4f(sideBounceLight, 0));
const textColor = groundTextColorUniform.$;
return d.vec4f(
std.mix(backgroundColor.rgb, textColor, percentageSample.x).mul(1.0 + highlights),
1.0,
);
};
const rayMarch = (rayOrigin: d.v3f, rayDirection: d.v3f, _uv: d.v2f) => {
'use gpu';
let totalSteps = d.u32();
let backgroundDist = d.f32();
for (let i = 0; i < MAX_STEPS; i++) {
const p = rayOrigin.add(rayDirection.mul(backgroundDist));
const hit = getMainSceneDist(p);
backgroundDist += hit;
if (hit < SURF_DIST) {
break;
}
}
const background = renderBackground(rayOrigin, rayDirection, backgroundDist, d.f32());
const bbox = getSliderBbox();
const zDepth = d.f32(0.25);
const sliderMin = d.vec3f(bbox.left, bbox.bottom, -zDepth);
const sliderMax = d.vec3f(bbox.right, bbox.top, zDepth);
const intersection = intersectBox(rayOrigin, rayDirection, sliderMin, sliderMax);
if (!intersection.hit) {
return background;
}
let distanceFromOrigin = std.max(d.f32(0.0), intersection.tMin);
for (let i = 0; i < MAX_STEPS; i++) {
if (totalSteps >= MAX_STEPS) {
break;
}
const currentPosition = rayOrigin.add(rayDirection.mul(distanceFromOrigin));
const hitInfo = getSceneDist(currentPosition);
distanceFromOrigin += hitInfo.distance;
totalSteps++;
if (hitInfo.distance < SURF_DIST) {
const hitPosition = rayOrigin.add(rayDirection.mul(distanceFromOrigin));
if (!(hitInfo.objectType === ObjectType.SLIDER)) {
break;
}
const N = getNormal(hitPosition, hitInfo);
const I = rayDirection;
const cosi = std.min(1.0, std.max(0.0, std.dot(std.neg(I), N)));
const F = fresnelSchlick(cosi, d.f32(1.0), d.f32(JELLY_IOR));
const reflection = std.saturate(d.vec3f(hitPosition.y + 0.2));
const eta = 1.0 / JELLY_IOR;
const k = 1.0 - eta * eta * (1.0 - cosi * cosi);
let refractedColor = d.vec3f();
if (k > 0.0) {
const refrDir = std.normalize(std.add(I.mul(eta), N.mul(eta * cosi - std.sqrt(k))));
const p = hitPosition.add(refrDir.mul(SURF_DIST * 2.0));
const exitPos = p.add(refrDir.mul(SURF_DIST * 2.0));
const env = rayMarchNoJelly(exitPos, refrDir);
const progress = hitInfo.t;
const jellyColor = jellyColorUniform.$;
const scatterTint = jellyColor.rgb.mul(1.5);
const density = d.f32(20.0);
const absorb = d.vec3f(1.0).sub(jellyColor.rgb).mul(density);
const T = beerLambert(absorb.mul(progress ** 2), 0.08);
const lightDir = std.neg(lightUniform.$.direction);
const forward = std.max(0.0, std.dot(lightDir, refrDir));
const scatter = scatterTint.mul(jellyScatterUniform.$ * forward * progress ** 3);
refractedColor = env.mul(T).add(scatter);
}
const jelly = std.add(reflection.mul(F), refractedColor.mul(1 - F));
const finalJelly = std.mix(background.rgb, jelly, jellyColorUniform.$.w);
return d.vec4f(finalJelly, 1.0);
}
if (distanceFromOrigin > backgroundDist) {
break;
}
}
return background;
};
const raymarchFn = tgpu.fragmentFn({
in: { uv: d.vec2f },
out: d.vec4f,
})(({ uv }) => {
randf.seed2(randomUniform.$.mul(uv));
const ndc = d.vec2f(uv.x * 2 - 1, -(uv.y * 2 - 1));
const ray = getRay(ndc);
const color = rayMarch(ray.origin, ray.direction, uv);
return d.vec4f(std.tanh(color.rgb.mul(1.3)), 1);
});
const fragmentMain = tgpu.fragmentFn({
in: { uv: d.vec2f },
out: d.vec4f,
})((input) => {
return std.textureSample(sampleLayout.$.currentTexture, filteringSampler.$, input.uv);
});
const rayMarchPipeline = root.createRenderPipeline({
vertex: common.fullScreenTriangle,
fragment: raymarchFn,
targets: { format: 'rgba8unorm' },
});
const renderPipeline = root.createRenderPipeline({
vertex: common.fullScreenTriangle,
fragment: fragmentMain,
targets: { format: presentationFormat },
});
let lastTimeStamp = performance.now();
let frameCount = 0;
const taaResolver = new TAAResolver(root, width, height);
function createBindGroups() {
return {
rayMarch: root.createBindGroup(rayMarchLayout, {
backgroundTexture: backgroundTexture.sampled,
valueTexture: valueTextureView,
}),
render: [0, 1].map((frame) =>
root.createBindGroup(sampleLayout, {
currentTexture: taaResolver.getResolvedTexture(frame),
}),
),
};
}
(canvas as any).onpaint = () => {
const sourceDict = { source: valueElement };
const destDict = {
destination: { texture: valueRawTexture },
width: width,
height: height
};
try {
(root.device.queue as any).copyElementImageToTexture(sourceDict, destDict);
} catch (e) {
// The copyElementImageToTexture API was recently changed to take two maps
// (see: https://github.com/WICG/html-in-canvas#idl-changes). This snippet
// supports the old syntax temporarily so that the demos do not break.
(root.device.queue as any).copyElementImageToTexture(
valueElement, width, height, { texture: valueRawTexture });
console.log('Note: using old copyElementImageToTexture API');
}
// TODO(pdr): Calculate this correctly using `getElementTransform`. For now,
// the transform is just hard-coded.
//const view = camera.view;
//const proj = camera.proj;
//const mvp = m.mat4.mul(proj, view, d.mat4x4f());
//const sliderWidth = sliderElement.clientWidth || (canvas.clientWidth * 0.75);
const sliderHeight = sliderElement.clientHeight || (canvas.clientHeight * 0.125);
let x = (canvas.width / devicePixelRatio) / 8;
let y = (canvas.height / devicePixelRatio) / 2 - (sliderHeight / 2);
sliderElement.style.transform = `translate(${x}px, ${y}px)`;
valueElement.style.transform = `translate(${x}px, ${y}px)`;
};
(canvas as any).requestPaint();
let bindGroups = createBindGroups();
let animationFrameHandle: number;
function render(timestamp: number) {
frameCount++;
camera.jitter();
const deltaTime = Math.min((timestamp - lastTimeStamp) * 0.001, 0.1);
lastTimeStamp = timestamp;
randomUniform.write(d.vec2f((Math.random() - 0.5) * 2, (Math.random() - 0.5) * 2));
const reduce = motionMedia.matches || transparencyMedia.matches;
if (reduce) {
currentMouseX = targetMouseX;
slider.restLen = Math.max(0.001, Math.abs(currentMouseX - slider.anchor[0])) / (slider.n - 1);
} else {
currentMouseX += (targetMouseX - currentMouseX) * 0.08;
slider.restLen = 1.9 / (slider.n - 1);
}
slider.setDragX(currentMouseX);
slider.update(deltaTime);
const currentFrame = frameCount % 2;
rayMarchPipeline
.withColorAttachment({
view: textures[currentFrame].sampled,
loadOp: 'clear',
storeOp: 'store',
})
.with(bindGroups.rayMarch)
.draw(3);
taaResolver.resolve(textures[currentFrame].sampled, frameCount, currentFrame);
renderPipeline
.withColorAttachment({ view: context })
.with(bindGroups.render[currentFrame])
.draw(3);
animationFrameHandle = requestAnimationFrame(render);
}
function handleResize() {
[width, height] = [canvas.width * qualityScale, canvas.height * qualityScale];
camera.updateProjection(Math.PI / 4, width, height);
textures = createTextures(root, width, height);
backgroundTexture = createBackgroundTexture(root, width, height);
taaResolver.resize(width, height);
frameCount = 0;
bindGroups = createBindGroups();
}
const resizeObserver = new ResizeObserver(() => {
handleResize();
});
resizeObserver.observe(canvas);
animationFrameHandle = requestAnimationFrame(render);
const hcMedia = window.matchMedia('(forced-colors: active)');
const darkMedia = window.matchMedia('(prefers-color-scheme: dark)');
const contrastMedia = window.matchMedia('(prefers-contrast: more)');
const motionMedia = window.matchMedia('(prefers-reduced-motion: reduce)');
const transparencyMedia = window.matchMedia('(prefers-reduced-transparency: reduce)');
const updateReducedFeatures = () => {
const reduce = motionMedia.matches || transparencyMedia.matches;
if (reduce) {
slider.damping = 1.0;
slider.archStrength = 0.0;
jellyScatterUniform.write(0.0);
} else {
slider.damping = 0.01;
slider.archStrength = 2.0;
jellyScatterUniform.write(JELLY_SCATTER_STRENGTH);
}
};
motionMedia.addEventListener('change', updateReducedFeatures);
transparencyMedia.addEventListener('change', updateReducedFeatures);
updateReducedFeatures();
const parseColor3 = (colorStr: string): d.Infer<typeof d.vec3f> => {
const match = colorStr.match(/rgba?\((\d+),\s*(\d+),\s*(\d+)/);
if (match) {
return d.vec3f(parseInt(match[1]) / 255, parseInt(match[2]) / 255, parseInt(match[3]) / 255);
}
return d.vec3f(1.0);
};
const parseColor4 = (colorStr: string): d.Infer<typeof d.vec4f> => {
const match = colorStr.match(/rgba?\((\d+),\s*(\d+),\s*(\d+)(?:,\s*([0-9.]+))?\)/);
if (match) {
const a = match[4] !== undefined ? parseFloat(match[4]) : 1.0;
return d.vec4f(parseInt(match[1]) / 255, parseInt(match[2]) / 255, parseInt(match[3]) / 255, a);
}
return d.vec4f(1.0, 1.0, 1.0, 1.0);
};
const updateColors = () => {
const style = getComputedStyle(sliderElement);
jellyColorUniform.write(parseColor4(style.color));
groundColorUniform.write(parseColor3(style.backgroundColor));
groundTextColorUniform.write(parseColor3(style.caretColor));
(canvas as any).requestPaint?.();
};
sliderElement.addEventListener('focus', updateColors);
sliderElement.addEventListener('blur', updateColors);
hcMedia.addEventListener('change', updateColors);
darkMedia.addEventListener('change', updateColors);
contrastMedia.addEventListener('change', updateColors);
updateColors();
export function onCleanup() {
sliderElement.removeEventListener('focus', updateColors);
sliderElement.removeEventListener('blur', updateColors);
hcMedia.removeEventListener('change', updateColors);
darkMedia.removeEventListener('change', updateColors);
contrastMedia.removeEventListener('change', updateColors);
motionMedia.removeEventListener('change', updateReducedFeatures);
transparencyMedia.removeEventListener('change', updateReducedFeatures);
cancelAnimationFrame(animationFrameHandle);
resizeObserver.disconnect();
root.destroy();
}

View file

@ -0,0 +1,85 @@
01. What information might this feature expose to Web sites or other parties,
and for what purposes is that exposure necessary?
A design requirement is to not expose any new security information, and to limit the amount of new privacy information (see: [Privacy-preserving painting](https://github.com/WICG/html-in-canvas?tab=readme-ov-file#privacy-preserving-painting)). For the purpose of enabling interactivity, this API will reveal form control rendering, scrollbar rendering, text selection, find-in-page selection, and the caret blink rate (all without revealing OS theme colors).
02. Do features in your specification expose the minimum amount of information
necessary to enable their intended uses?
Yes.
03. How do the features in your specification deal with personal information,
personally-identifiable information (PII), or information derived from
them?
Since the feature renders pixels from DOM elements into canvas, those pixels can now be accessed by script, so it is important that no PII is present in those pixels. Cross-origin information, visited link information, spellcheck information, and autofill previews must not be painted. Disabling painting of this information also prevents revealing invalidation information via the `paint` event. See [privacy-preserving-painting](https://github.com/WICG/html-in-canvas/tree/main?tab=readme-ov-file#privacy-preserving-painting) for additional details.
04. How do the features in your specification deal with sensitive information?
See answer above, the feature ensures no new security information is revealed, and limits new privacy information.
05. Do the features in your specification introduce new state for an origin
that persists across browsing sessions?
No.
06. Do the features in your specification expose information about the
underlying platform to origins?
Similar to #1, the painting of information revealing information about the underlying platform (e.g., form autofill) is disabled, but some new platform information is revealed for interactivity, such as the caret blink rate. See [privacy-preserving-painting](https://github.com/WICG/html-in-canvas/tree/main?tab=readme-ov-file#privacy-preserving-painting) for additional details.
8. Does this specification allow an origin to send data to the underlying
platform?
No.
9. Do features in this specification enable access to device sensors?
No.
10. Do features in this specification enable new script execution/loading
mechanisms?
No.
11. Do features in this specification allow an origin to access other devices?
No.
12. Do features in this specification allow an origin some measure of control over
a user agent's native UI?
No.
13. What temporary identifiers do the features in this specification create or
expose to the web?
None.
14. How does this specification distinguish between behavior in first-party and
third-party contexts?
There is no difference in behaviour.
15. How do the features in this specification work in the context of a browsers
Private Browsing or Incognito mode?
There is no difference in behaviour for these modes.
16. Does this specification have both "Security Considerations" and "Privacy
Considerations" sections?
The specification is still in progress. The privacy issues have been highlighted in the explainer.
17. Do features in your specification enable origins to downgrade default
security protections?
No.
18. How does your feature handle non-"fully active" documents?
It only works in fully active documents.
19. What should this questionnaire have asked?
No suggestions.