← labopen splats →

Splats

Gaussian splatting, from scratch, on a phone. A first viewer written to learn from and to build on: no library, about 600 lines in four files, each doing one part of the job.

What a splat is

3D Gaussian Splatting (Kerbl et al., SIGGRAPH 2023) represents a scene as a large number of fuzzy, coloured ellipsoids, each a 3D Gaussian, rather than as triangles. A scene typically has a few hundred thousand to a few million of them. Each splat has:

The splats are found by optimisation. Training starts from a sparse point cloud from photogrammetry (COLMAP). It renders the splats, compares the result with the photographs, and adjusts every parameter by gradient descent, splitting splats where detail is missing and pruning those that contribute nothing. This viewer only renders splats; it doesn't train them.

How they're drawn (the four files)

  1. format.ts: the data. It reads .splat, the antimatter15 format: 32 bytes a splat, already decoded. It reads .ply, which the trainer writes: scales stored as logs, opacity before its sigmoid, colour as SH coefficients. It also generates a scene, garden, from shapes. The garden shows how surfaces are made: the ground and the stone are flat discs, each a splat squashed along its normal, and the grass is needles, each a splat stretched along one axis.
  2. sort.ts: the order. Splats are translucent and blended, so they must be drawn back to front, and the order changes as the view turns. The sort is a counting sort on a 16-bit depth along the view direction: linear time, about 12 ms for half a million splats on a laptop. It runs in a worker, so the page keeps drawing with the previous order. Only the view's direction matters, so it re-sorts only when the view turns.
  3. render.ts: the projection. All the splats live in one integer texture, 32 bytes each. Each is drawn as an instanced quad, in the sorted order. The vertex shader projects the 3D covariance to a 2D one on the screen, Σ' = J W Σ Wᵀ Jᵀ. W is the camera's rotation and J is the Jacobian of the perspective at the splat's centre, the local linear approximation that makes the projection affine. The shader adds 0.3 px² so no splat is thinner than a pixel, takes the eigenvectors of Σ' as the ellipse's axes, and spreads the quad to 3σ along them. The fragment shader is the Gaussian itself: opacity × exp(−r²/2), blended premultiplied.
  4. main.ts: the page: the orbit camera, the scenes, the controls.

The see buttons show the stages: ellipses draws each splat's 1σ and 2σ contours; footprints fills each quad solid, which shows the overdraw that costs most on a phone; centres shows the point cloud alone.

The research, briefly (late 2026)

Sources: kishimisu/Gaussian-Splatting-WebGL, catid/Mobile-GS, WebSplatter (2026), Portality, PlayCanvas splat formats, SOG open-sourced. Captured scenes are the examples from antimatter15's viewer, hosted on Hugging Face.

One Stone: splats as a drawing

?scene=stone (stone.ts). The idea is procedural drawing as a way to author a 3D world: each feature gets a way of drawing itself, and the Gaussian is what the marks are made of. The interesting decisions are where a mark belongs, what it describes, and when the drawing can do without it.

The stone is a cut stone of unequal geological planes on warm paper:

Its mesh, rays, faces and edges are all exact, from the planes.

Composed, not catalogued. An earlier version simplified by category ("the wash goes first"). Thinning evenly turned out to keep more of the planes than that did, and the sparse drawing became perimeter plus crack. So every mark now belongs to a stroke, every stroke to a feature (the seam, the ledge, the foot, the crown, a face), and each feature reduces within itself before it goes. Three states are composed deliberately (state buttons):

Tonal shapes. A face in shadow takes a shape anchored on an edge: the arris it shares with a lighter face (its terminator), or under the ledge, the lip. It's firm at the anchor (a dense band laid along it), softens away, and its far boundary is uneven (a slow noise along the anchor sets how far it reaches). The darkest ink is kept to narrow places: the crevice, the contact points.

The shadow is cast: the stone's outline is pushed along the light onto the ground, firm at the foot on the far side and fading along the light. The lit side of the base is only lightly and brokenly touched, and the darkest contact is at two or three points.

Contour marks sit along every arris, each knowing both its faces' normals. One is drawn only where one face turns toward you and the other away: the profile from this view, eased in as a face turns, broken into accents. The fracture belongs to the rock and the contour belongs to this view of the rock.

The brush develops a feature. Touch the seam and its branch, chips and moss come. Touch under the ledge and its underside develops. Touch a face and its tonal shape grows. Each feature comes back in its own order, heart first, so what appears has the feature's shape, not the brush's. Keep brushing to develop it further. It's on the stone, so it stays as you turn.

Renderer details that make it work: each corner of a mark's quad takes the depth of the surface it lies on, so broad marks aren't cut by their own face. The stone's exact mesh is the depth-only occluder. ?only=wash (or fracture, contour, stipple, shadow, lip, chip) shows one kind of mark alone.

Boards: node cells/lab/devtools/splats-stone.mjs [states|pairs|views|detail|occluder|brush] [wash|dry]. states is the main one: the three composed states from the reverse view (the projecting lip, the line-overhang-shadow relationship it's built around), then the hero and grazing views by the same rules.

Next: the Bamboo connection. A cliff face could give up its wash while a ledge keeps its lip and a fissure its line. A culm could keep its silhouette and nodes while its shaded body drops away.

Brushes: a shared medium

?scene=brushes (brush.ts, subjects.ts). One Stone showed the ideas; this generalises them into a medium that Taohua's lines and washes, Labyrinth's architecture, Bamboo's moving landscape and Autograph's gesture could all share:

scene structure → drawing instructions → strokes and washes → splats → image

A stroke and a wash are objects that keep their identity: a path, a gesture, a material, a place in a group. They become splats only when deposited, so a stroke can be shortened, re-pressured, regenerated at another density, or withdrawn with its group, and its gesture is kept. There are three operations:

  1. Stroke along a path. Support: on a surface (it lies flat and wraps over it) or in open space (round in section). Gesture: width, pressure along it, taper, breaks, tremor. Material: ink, dry (fragmenting where the pressure's light), wash-line.
  2. Wash within a region. Any patch of a surface that can sample itself, test a point and project onto itself (a plane polygon, a cylinder, a strip beside a curve). It's firm along an anchor edge, softens away, has an uneven reach, and leaves places open. Material: wash (discs) or dry (hatching along a direction).
  3. Group. A principal, its supports and its accents simplify together. The principal goes last, the wash withdraws toward its anchor, and the accents drop to the one that matters.

Each stroke names its heart (where its meaning lives), and reduces toward it.

The test is whether the same brushes, given only different paths, regions and anchors, can draw unlike subjects:

subjects.ts makes no marks itself. Board: node cells/lab/devtools/splats-brushes.mjs [wash|dry].

Composed: one small scene

?scene=composed (subjects.ts composedScene, edit.ts). A stone, three culms and an architectural corner, with a bank passing between them, drawn with the same brushes throughout:

?compose=0 (or the relate row) draws each object as if alone.

Three decisions, made separately. The first version scaled rank, width, opacity and wash depth together by one "importance", which weakened a secondary object every way at once and didn't protect the subject's identity. Now:

decisioncontrolshow
structurewhich marks must survive, together, to explain shape, contact and overlapeach object's minimum description: its structural members, ranked to survive the sparse state at full strength
emphasiswhere the strongest contrasts belongonly the contrast of material marks: lighter washes, thinner lines on lesser objects
material detailwhat can go freelyeverything else, ranked below all structure, lesser objects' first

The minimum descriptions are authored:

These follow the reference drawings (below).

Groups are kept as objects (group() returns members that point to their group, with its principal, supports, accents and minimum). Structural washes fall gently from their anchor, so a group's minimum appears and goes together.

Views (main.ts, each frame, eased so nothing pops). Values computed from the view open and close gates (render.ts: a feature with its top bit set is a gate), choosing among anchored alternatives without rebuilding marks:

Fixes from the review:

Material gestures:

Editing in the scene (the Bench's editor, generalised, edit.ts):

An edit deposits again only the stroke and its washes (2 of 168 elements).

Checked (splats.test.mjs):

Motion (node cells/lab/devtools/splats-motion.mjs): an orbit stepped 10° at a time, with frames rendered at each step so the eased gates move as they would.

configurationchange per 10° step (median)largestlargest / median
sparse3.44.01.16
resolved13.515.91.17
resolved, no halo13.716.11.17
sparse, no halo3.74.71.25

No step stands out, so nothing pops at this resolution. A 10° step is coarse, though: a small mark appearing could hide inside the camera's own motion. Finer steps across the gates' transitions would be a stricter test.

Edit latency (headless Chrome, software GL; upper bounds for the CPU side, not phone timings):

editdepositassemble and upload
seam4 ms21 ms
bank (its slope is a large region)100–200 msabout 15 ms

Reference drawings (devtools/splats/targets/composed-{sparse,intermediate,resolved}.jpg). Three studies of this scene, made as artistic targets. Because they're generated, their geometry is close to the scene's but not the same, so they're judged by eye for what survives together, not compared pixel by pixel or used as training pairs.

statethe targets keep
sparsethe stone's upper silhouette, broken and angular; its crack with a small dark wedge; its overhang's edge with a dark wedge of underside shadow; the front culm as one continuous side with compressed nodes; the back culms as single lines with node ticks; the corner as cornice edges, one arris and the niche's frame (no sill); one ground line running into the culm's foot
intermediatethe stone's full outline with its lower blocks; tone along edges; a broader underside shadow; culms with both sides and light tone; the cornice's thickness; the niche's inner shade
resolvedfull tone, moss, cast shadows on rising ground

What changed to follow them:

Board: node cells/lab/devtools/splats-targets.mjs [wash|dry]: each state beside its target, by ranks and by rules.

Gaps against the targets:

Boards: node cells/lab/devtools/splats-composed.mjs [wash|dry]:

Still open:

The pen: Autograph's hand and tool in the Splats brush

?scene=pen (pen.ts, penscene.ts; &part=word|board|culm|bank, &reference). Three experiments each had part of a drawing medium:

Until now the Splats brush's "pressure" was a prescribed width profile along a supplied path: convenient and controllable, but a mark didn't look as it did because of how it was made. This brings in Autograph's hand and pen, reused rather than copied (move(), body(), widths(), BROAD, LOUD, INK exported from Autograph, unchanged):

partresponsibilityhere
drawing ruleswhat to describe, which marks belong togetherBamboo (its bank-rim selection and dashes, unchanged)
hand (optional)a timed trajectory from a planAutograph's move() (speed by the two-thirds power law, rest at the ends, stops at cusps) and body() (a critically damped follower: hurried, it rounds corners)
toolmovement and carried state into materialthe pen: Autograph's rules for speed, broad nib, press, voice and ink; a nib-aligned Gaussian at every step
spatial supportwhere the page goes in spaceplaneMap, cylinderMap, groundMap: the nib keeps its angle to the surface
deposition and displaysplats, renderedSplats

How the pen keeps the tool's character in splats:

Astra's three questions, answered (splats.test.mjs, a CPU rasterisation at fine resolution):

questionresult
does the broad nib keep its directional character?56.9 mm moving across its edge, 5.0 mm along it: 11.4×
is the tone held without accidental darkening?peak darkness 0.85 whichever way it moves; a self-crossing 0.83 against the line's 0.87 (0.99 without the union)
does the gesture keep its character on a curved surface?wrapped round a cylinder: 56.9 mm and 5.0 mm, the same 11.4×
does ink behave?a long stroke goes from 0.84 to 0.35 as the pen runs dry; after a pause it's dipped (0.85)

On the boards (node cells/lab/devtools/splats-pen.mjs):

Gaps:

Washes as tonal shapes

Washes were cloudy stipple, not tone. A wash (not dry) now has a pooled edge. Along the firm side, a thin run of narrow marks sits a little inside the boundary, a touch darker: the rim where wet pigment dries. The body is laid with fewer, larger, steadier marks, darkest near the firm edge and fading inward, so it reads as one connected tone. Rare specks give granulation. edge: 'soft' turns the rim off. Washes now make up about 18.6k of the composed scene's 34.7k marks, up from about 24.6k in all.

The first pass was better but still mottled at viewing distance: the tone depended on where random marks clumped and left gaps. The body's marks are now spaced (blue noise: each kept a minimum distance from the others), at a steadier size, so a wash reads as one flat graded tone with its rim. That also cut the mark count back (the composed scene is about 38.7k with the additions below).

One brush for the One Stone

?scene=stone used to be drawn by about 230 lines of bespoke marks in stone.ts: its own seam, wedge, chips, moss, hatching, contours and shadow. Those are retired. The study is now the composed scene's stone, made of brush elements, its plain lines drawn by the pen in the stone's manner (stoneDrawing). Three things it lacked were added as brush elements, so the composed stone has them too:

The chips were dropped. stone.ts now holds only the stone's shape: its planes, mesh, ray, faces, arrises and tone.

A Bamboo passage: rock and bank (?scene=pen&part=rock)

This is one real Bamboo passage drawn only by the brush (rockbank.ts). The scene picks the biggest outcrop on Bamboo's slope (seed 1) with a bank rim within a few metres. The outcrop is a slab, about 2.3 × 2.0 × 1.2 m, with eight neighbours and fragments round it. Each of them is drawn as Bamboo made it:

The blocks' meshes and Bamboo's ground are the occluder. The ground is sampled finer than Bamboo's cells and set a little low, so it doesn't cut into the slope's wash. In all: 21 faces washed, 79 edges, 3 dashes, about 22k marks.

Gaps:

Measuring on a phone (&perf)

?scene=composed&perf, ?scene=mist&perf, ?scene=room&perf and ?scene=walk&perf run a fixed benchmark while the camera turns:

Each phase settles for 1.5 s, then records for 6 s: fps, p50 and p95 frame time, p50 and p95 draw time (each frame is waited for with a one-pixel read, so its cost shows whatever the display's rate), sort time, and policy time. Then it times ten seam edits (deposit and upload). A table appears with the GPU name and a copy button. It's also kept in localStorage (splats-perf) and on window.__perf. Headless Chrome uses software GL, so only a real device's numbers mean anything (devtools/splats-perf.mjs just checks that it runs).

A drawing policy: learning which marks to keep

policy.ts, ?scene=composed&policy=ranks|rules|learned, ?judge. The idea comes from the Mist experiment, generalised: use an expensive process offline to teach a compact model a useful decision. Here the decision is which marks best describe the scene at a given detail and view, decided at the level of groups, above deposition.

What a policy controls. For each group (a seam with its wedge, a cornice with its underside, a culm's sides), a treatment:

The brushes still make every mark, so identities, anchors and editability are untouched. The renderer reads each mark's group (a 24-bit index in the third texel) and the group's control (a texture of thresholds and strengths).

Its inputs, per group per view:

Policies, always compared at the same mark budget (the number of marks the ranks would keep at that detail):

policyhow it decides
ranksthe scene's own ranks against the global detail: no control
ruleshand-written rules over the inputs (structure keeps a floor; tone only as far as it faces you; hidden or tiny detail keeps less; lesser objects give way where they crowd; contacts stay strong), scaled to the budget
learneda small model (the group's features plus the treatment, 2 × 12 hidden units) scores each group's possible treatments; each group gets the best score less a price per mark, the price found to meet the budget, the rest filled greedily

All three draw through the same controls. Treatments are eased between frames, and the target is worked out every third frame.

The teacher is you. A good sparse drawing is an artistic judgement; training on the rank rules would only reproduce their limits.

What's checked so far:

Not yet:

Captured scenes in ink: the splats drawing themselves

Everything above lays strokes made by procedure. That uses Gaussians as stamps, not as what they are: a soft field fitted to a real thing, that a phone can capture. ?scene=nike&ink (or plush, train, a URL or your own file, with the ink button) draws a captured scene in ink from its own splats (ink.ts). Every splat stays where it was captured, with its own shape; what changes is what it means:

The shoe at half detail reads as a pen sketch from the front. The plush reads but is busy, because fur is noise at this scale. Both captures are only good from the front, inked or not.

Gaps:

A drawing fitted as a field (?scene=field)

Can a drawing be what splats are, a field fitted to pictures, rather than strokes placed by procedure? This is the first, bounded test, on the stone.

Result: on the held-out views, the mean squared error (× 1000) of the drawing's darkness is:

held-out error
blank paper26.3
constant opacity (baseline)4.45
view-dependent field3.52

That's 21% lower than the baseline. On the board (devtools/splats-field-board.mjs, the real renderer at twice the fitting resolution) the difference is the point of the experiment:

The renderer evaluates each disc's opacity from its terms every frame (mark kind 11). The terms are packed into the bits a drawing's feature and element ids use; the file stores 19 bytes a disc.

Gaps:

The whole composed scene as a field (?scene=field&field=composed)

This is the same fit at scale. There are 33,730 discs over every object (the occluder's triangles, each faced away from its object's middle) and the ground round the scene. The teachers are the composed scene at full detail, from 36 views orbiting the reference camera's target at its distance, at 192 × 128, with 12 views held out. The fit took 200 iterations, about 4 minutes.

error (× 1000)
held out18.8
training10.8
blank paper66.8

From views it never saw, it draws the stone with its washes, the culms with their nodes, the corner and the ground's lines. The gap between training and held-out error is wider than the stone's: thin culms and more occlusion, with the same number of views.

Gaps:

The reference studies as teachers: not as they are

The plan was to add the resolved reference study as a teacher from its own view, to see whether a drawing's character carries round the scene. It can't be done honestly with the studies as they are:

The studies are compositions after the scene, not drawings of it. Used as pixel teachers, they'd teach the field their marks in the wrong places.

What would work: a drawing made over the scene itself. devtools/splats-field-canvas.mjs renders the composed scene at the studies' size (1024 × 683) from the reference camera and a view either side, each with its camera (devtools/out/field/canvas-*.png, canvases.json). Redrawn in the studies' manner with the geometry kept exactly, for instance by an image model told to restyle without moving anything, a canvas becomes a teacher directly: splats-field-redrawn.py <image> ref, then splats-field-fit.mjs composed wash 200 --ref=redrawn-ref. One view redrawn tests whether the character carries to the views either side. Two or three redrawn test whether it carries round.

Drawn teachers: wireframes restyled by an image model

Since the studies can't teach, the scene supplied its own canvases. devtools/splats-field-wire.mjs draws the composed scene's real geometry from six views, at 1024 × 683, each with its camera:

An image model redrew each view in a style, told to keep every line where it is. Seven restyles came back, each a sheet of all six views: fine pen and wash, textured stone with strong cast shadows, three sumi-e brush executions (spare, bold, light), warm watercolour and engraved hatching. The author's preference is sumi-e; I agree it suits splats best, since its marks are blots, graded washes and dry-brush edges.

splats-field-sheets.py cuts each sheet's panels at its separators, masks the labels, and aligns each panel to its wireframe by a small scale and shift. After alignment, the panels match their wireframes at 0.34 to 0.77 (normalised correlation). The reference study managed 0.11 even after warping. The sparse brushes score lowest; that's sparsity, not misplacement (checked by overlay).

What the drawn teachers taught:

All errors below are held-out error on the right view, with the other five views of the same execution as teachers.

Measuring what drawn teachers share

The sumi comparisons above were all scored by pixel error, and their differences were a few percent. Before building on them, the measure itself was tested (devtools/splats-field-measure.py). The restyled sheets are independent drawings of the same geometry, so a measure is only worth optimising if it passes two tests: given one execution's drawing of a view, it finds the same view among another execution's six (chance 1 in 6), and it scores that view far nearer than blank paper.

measure, three sumi executionsfinds the same viewsame view ÷ blank paper
pixel error at 192 × 12836 of 361.00
tone: the same, blurred at three scales36 of 360.49
decisions: ink along each edge, tone on each face28 of 360.30

The fields scored again (splats-field-score.py), on the right view held out, each fitted to the other five views of its own execution at 384 × 256. Lower is better for tone and decisions; agreement is the correlation of decisions.

what drew the held-out viewsumi: tonedecisionsagreementbold: tonedecisionsagreementlight: tonedecisionsagreement
blank paper7.093273.781481.88166
another hand, same view2.11, 2.49147, 1090.51, 0.630.78 (light)280.850.78 (bold)280.85
field, one opacity a disc1.33680.741.67830.440.69630.59
field, view terms and fitted shapes1.45530.801.78760.520.62570.62

What follows:

Decisions as the unit: a rule, a table, and the brush

devtools/splats-decisions.py. Each teacher's view is about 59 decisions; three sumi executions of six views give 1,068.

Can a decision be predicted for a view it wasn't drawn from? Leaving one view out, all six in turn:

what predicts the held-out viewdecision erroragreement
blank paper235
a table: each entry's average over the other five views570.68
the table with a view term per entry580.69
a rule: one small network for every edge and face, from 25 features of its geometry and how it's seen620.64
a sibling hand's decisions for the same view360.82

The brush draws the decisions (policy.ts taught, ?scene=composed&policy=taught&teacher=sumi). Each group of the composed scene is matched to the teacher's entry of its own object and a kind it could be (a stroke to a crease, silhouette or line; a wash to a face), the nearest in the scene. The decision sets how much of the group is kept and how strongly. The ground's lines get none, since the teacher was shown no bank.

Scored on the right view, each table leaving it out (splats-taught-score.mjs, splats-field-score.py):

what drew itsumi: tonedecisionsagreementbold: tonedecisionsagreementlight: tonedecisionsagreement
blank paper7.093273.781481.88166
our best policy (rules or ranks)5.501170.593.281100.401.471030.46
taught, drawn by the brush5.06980.663.171210.491.431260.50
fitted field, shapes fitted1.45530.801.78760.520.62570.62

Next: close the loop on the brush. Measure the decisions of what it drew and adjust each group until they match the teacher's, instead of mapping by hand. Then washes and shadows carry their tone, and the score says whether crisp strokes with the right tone beat the smeared field.

Closing the loop on the brush, and checking the field against the renderer

The brush fitted to each teacher (devtools/splats-taught-fit.mjs). The scene is drawn with a treatment set from outside, from the five views other than the right; what it drew is measured as the teacher's drawings are; each group is adjusted until they match, ten rounds.

on the five fitted viewssumiboldlight
ink off by, before → after0.32 → 0.160.40 → 0.230.35 → 0.20
tone off by, before → after0.15 → 0.120.17 → 0.060.18 → 0.06

It stalls about halfway on ink. The composed scene has washes only where its own light put them and strokes only where its own design put them, so some of a teacher's tone has no mark to carry it (one stone face asks for 0.33 and gets 0.03 at full strength).

On the right view, which no fit saw:

taught brushsumi: tonedecisionsagreementbold: tonedecisionsagreementlight: tonedecisionsagreement
mapped by hand5.06980.663.171210.491.431260.50
fitted4.681370.442.53800.601.28760.55

The field against the real renderer (devtools/splats-field-browser.mjs). The fields' scores above were taken from the optimiser's own picture of the held-out view; the brush's from the browser. The same held-out sumi field, drawn by the real renderer from the same camera:

the held-out sumi fieldtonedecisionsagreementmean darkness
the optimiser's picture1.45530.800.043
the real renderer2.55600.830.024

The fitter and the renderer made to agree

A fitted field is only as good as the forward model it was fitted through. The check above found the browser laying about half the ink the optimiser believed. Five differences were found, one at a time, by drawing the same field both ways and comparing (devtools/splats-field-agree.py):

the held-out sumi field, refittedtonedecisionsagreementmean darkness
the optimiser's picture (384 wide)1.29470.840.0449
the real renderer, 384 wide1.32470.840.0431
the real renderer, 1024 wide1.32330.870.0432

Marks: a stroke or a wash as an organisation of splats (?scene=marks)

A fitted field draws a view well and has no marks in it: nothing to pick up, nothing that says "this is a stroke". The taught brush has marks and keeps what it learns outside the splats, in a table. The aim here is the thing between them: a stroke or a wash is an editable organisation of splats; a teacher can change both the organisation and its individual deposits.

The conservatory studies

Six studies for the next scene: a flooded conservatory overtaken by a willow, in three regions, each drawn dense and sparse. They are compositions, not views of any geometry here, so they can't teach a scene directly (as the earlier reference studies couldn't). Two things they do give:

A mark

A mark (client/splats/marks.ts) has a few parameters that organise its deposits, and each deposit has six small residuals of its own (ink, length, width, turn, two shifts).

Three ways of drawing a teacher's mark, with the same number of deposits

Each teacher mark is fitted through the renderer's forward model (devtools/splats-marks-fit.mjs):

Tone error / ink on paper the teacher left bare / dryness error (strokes; all lower is better):

teacher markdepositsindependentorganisedorganised, then depositsexisting brush
a root2940.23 / 0.3 / 010.45 / 10.3 / 960.70 / 1.7 / 20844.65 / 14.6 / 157
a second root2940.13 / 0.1 / 05.47 / 1.0 / 420.46 / 0.1 / 7722.02 / 28.7 / 123
a third root2170.38 / 0.3 / 07.12 / 2.0 / 290.55 / 0.5 / 11155.09 / 53.5 / 72
a branch1610.19 / 0.6 / 012.70 / 7.0 / 945.04 / 0.4 / 14619.17 / 25.5 / 82
a spray380.76 / 1.0 / 2110.42 / 25.6 / 923.54 / 7.7 / 12416.63 / 11.4 / 136
a second spray380.29 / 0.4 / 15.53 / 11.2 / 1081.85 / 3.2 / 1747.54 / 9.1 / 122
a canopy wash480.05 / 2.70.89 / 11.50.03 / 1.520.44 / 30.7
a larger wash960.26 / 1.15.12 / 6.00.35 / 1.519.17 / 1.3
a veil720.16 / 2.4 / 372.35 / 12.9 / 270.25 / 1.8 / 5020.27 / 19.3 / 117

A style, on another spine

The test a field can't take. A held-out teacher mark is drawn with the parameters learnt from another mark; only its pose is fitted (the points of its spine, and one number each for its ink, its width and its two ends). Beside it: a plain stroke posed the same way; the held-out mark's own organised fit; and independent deposits.

Tone error (strokes: / dryness / roughness errors):

held-out markthe learnt style, poseda plain stroke, posedits own organised fitindependent
a second root18.0 / 115 / 32533.1 / 223 / 8665.5 / 42 / 2600.1 / 0 / 35
a third root32.6 / 120 / 12284.9 / 130 / 3367.1 / 29 / 1630.4 / 0 / 26
a branch (another study)26.1 / 172 / 45238.7 / 267 / 65612.7 / 94 / 3260.2 / 0 / 29
a second spray6.5 / 152 / 38211.0 / 138 / 2945.5 / 108 / 1610.3 / 1 / 21
a larger wash7.17.95.10.3

Still a mark

In the page a fitted mark can be tapped and bent by the points of its spine; its deposits are laid again and come with it (a strand and its dabs move together). Independent deposits offer nothing to pick up.

Walking into a wash (?scene=walk)

From far off a teacher draws a canopy as a wash; close to, it's strands and leaves. Both are organisations of splats in the same volume (client/splats/canopy.ts): a wash of 30 large pale deposits, and 90 fronds (4,424 deposits now their leaves are formed, a few deposits a leaf: see One frond cluster, restored; 1,972 when a leaf was one dab).

Every 20 cm from 12 m to 0.3 m (devtools/splats-walk.mjs):

tone against the fronds, beyond 3.4 m: worst / meanlargest change beyond the walk's owndeposits at 12 m / at most
the wash alone21% / 15%none30 / 30
hand-over13% / 9%0.06 (where the walk's own is 123)30 / 4,454
swapped at once at 5 m20% / 11%22.7 (the walk's own there: 10.3)30 / 4,424

(devtools/splats-walk-board.py puts the board together.)

The room (?scene=room)

A flooded conservatory and a willow, built entirely of marks (client/splats/conservatory.ts): 1,383 of them, about 30,000 deposits laid. Three regions round one looped route, the willow in the middle of the loop: a root tunnel (a boardwalk under arching roots), an open pool (stepping stones, a veil, a far bridge) and a canopy (a deck under hanging fronds that hand over as above).

The room, taught from three drawn views

The room drawn plain from six views round the route was restyled by an image model, as the composed scene's wireframes were. All six line up with their underdrawings with no shift (blurred, they correlate at 0.78 to 0.90), and the teacher uses about half to two thirds of the underdrawings' ink. Three are taught from (the root tunnel, the open pool, the canopy). The three between them are reserved: never fitted to, only scored on.

The marks stay where the scene put them; their spines aren't moved. What is taught is how they are drawn, at four levels of freedom (devtools/splats-room-fit.mjs):

Tone error, mean of three views (blank paper: 74.7 on the views taught from, 71.6 on the reserved):

numbers taughton the views taught fromon the reserved viewsreserved: ink on bare paperreserved: ink against the teacher's
untaught063.633.081.6×1.71
a style a role32710.815.412.6×0.69
+ each mark's weight+ 4,1497.714.914.2×0.80
+ its deposits, bounded+ 183,0001.615.826.2×0.87
deposits set free(the same 183,000)0.825.796.3×1.43

Each reserved view on its own (between the tunnel and the pool / the pool and the canopy / the canopy and the tunnel): untaught 16.2 / 45.4 / 37.2; a style a role 10.8 / 14.9 / 20.7; + weights 9.5 / 13.3 / 21.8; + bounded deposits 9.1 / 17.3 / 20.9; set free 34.3 / 22.8 / 20.1.

What this does and doesn't show:

(These numbers are from before the room's solids hid anything: see the next section.)

The page draws the room as taught by default (a style a role and each mark's weight); &style=bench and &style=plain are the other two.

Identity and occlusion, and what they change

Three things a review of the room found, fixed before going on:

The room taught again, the same way, with its solids (tone error, mean of three views):

on the views taught fromon the reserved viewsreserved: ink on bare paper
untaught63.035.679.2
a style a role19.122.715.2
+ each mark's weight15.322.916.3
+ its deposits, bounded4.722.016.0
deposits set free2.425.238.4

The root: a limb as a loaded stroke

The room's limbs are its main subject, and after teaching they were still soft bands. Three materials are being taken one at a time, starting with the root.

The pool. Every limb the teacher drew is cut out of its restyle and straightened along the limb's own spine, at the restyle's full size (devtools/splats-pool.mjs, .py): 45 strokes of 24 limbs, 12 of them seen from more than one view. A strip runs a pixel a step along the limb, and across it from -2.5 to +2.5 of the limb's half-width.

Place by place doesn't work. The first comparison drew each limb four ways from each stroke's view, straightened it the same way, and matched its profile across the limb to the teacher's. Every way came out a grey band (profile error 88 to 110, against 125 as the room draws it). Along a limb the teacher's stroke wanders from the scene's spine by up to half the limb's width: a restyle keeps the composition, not each edge. A profile matched in the limb's frame rewards a pale hedge.

Read in the stroke's own frame, no correspondence is needed. At each step along a strip the dark band is found (nearest the last step's), and from it: how thick it is (in the limb's half-widths), how dark inside, how many pixels its ink takes to fall at its lower edge (firm) and its upper (feather), and how its darkness varies along it. Medians over the strokes from the views taught from / from the reserved views:

thickbodyfirm, pxfeather, pxapart from the teacher
the teacher1.00 / 1.080.81 / 0.816.3 / 4.56.9 / 7.6
as the room drew it (rows all round, inked beneath)0.83 / 0.920.63 / 0.631.9 / 2.09.7 / 14.24.4 / 6.0
the same, fitted here0.58 / 0.920.66 / 0.683.7 / 3.78.5 / 16.54.2 / 6.2
banded on its underside0.92 / 1.000.78 / 0.802.0 / 1.77.2 / 9.22.8 / 1.8
banded as seen0.92 / 1.080.79 / 0.812.7 / 3.15.6 / 7.02.2 / 1.1
a flat ribbon1.00 / 1.000.84 / 0.861.6 / 1.65.4 / 5.63.3 / 2.3
banded as seen, 2.5 times the deposits0.83 / 1.000.81 / 0.832.0 / 2.55.8 / 6.62.5 / 1.5

In the room (the default now; &roots=0 for the tubes), a limb is a dark stroke with a firm edge along its lower side from wherever it's seen, and the view between the canopy and the tunnel, where the nearest arch had gone blank, has its arch back.

Not done:

The second teachers: from the room's geometry

The first underdrawings sent for restyling were the room drawn in its own marks, and the teacher kept their choices: where a limb's ink sat, how soft a wash was. The second set carries none (devtools/splats-room-wires.mjs, .py; devtools/conservatory/wires2): the solid room's geometry as thin even lines with what's hidden left out. Each limb is its tube's outline; planks, posts, rails, strands, the veil and the far bridge are the lines they run along; a leaf is a tick where it is; a wash or mist is the dotted outline of its volume. Eight views: the six (three to teach from, three reserved), and two more reserved: back along the tunnel, and sideways from the stones.

The root, read again. The pool cut from the new restyles is 67 strokes of 24 limbs, 73% of their ink within the limb's band. It reads differently from the first:

A limb in parts, by its width on the screen (conservatory.ts, limbParts): far off, one loaded stroke; close to, a loaded stroke along one side, a broad dry sweep under it, and a line along each edge. The far one hands over to the near ones by the limb's own width on the screen, the same way a canopy's wash hands over to its fronds. The edge lines need no outline pass: each deposit lays its ink where the tube's surface is edge-on to the eye, which its view terms hold exactly.

Its style is fitted to the teacher's profiles across a limb, pooled over strokes by width class (devtools/splats-root2.mjs). Distance from the teacher's pooled profiles, strokes from the views taught from / from the five reserved views:

deposits drawn a limbtaught fromreserved
one band as seen (the first root)64036.861.1
in parts, by its width on the screen1,96322.217.2

The room taught again, from three views of the second teachers, scored on five reserved (tone error; the second teachers are sparse, so blank paper scores 27.0 on the views taught from and 24.2 on the reserved):

on the views taught fromon the reserved views
untaught (limbs in parts)89.499.9
a style a role, the limbs' ink left as the strokes taught it24.523.9
+ each mark's weight23.424.7
a style a role, the limbs' ink taught too17.718.8
+ each mark's weight13.920.3

One limb, three ways (?scene=limb)

The room's limbs read as shaded tubes: an even rim, a smooth gradient, the same from one end to the other. They were fitted to profiles across a limb, pooled over strokes, which throws away everything a stroke does along itself: where it's loaded, where it breaks, where it lifts. So one limb is taken out of the room (the root tunnel's third arch) and the teacher's strokes are read again as whole patches, along as well as across (devtools/splats-limb.mjs). At each step along a stroke the dark band is found, and what's kept is how it runs:

Three ways of laying the limb, each with the same number of deposits (2,484 for the median limb; 3,240 for this arch):

Read along the stroke, over the 31 strokes from the three views taught from (the last two rows: distance from the teacher over all the readings, there and on the 36 strokes from the other five views):

the teacherin partsa sweepwith accents
cover0.931.000.960.99
gap (px)14.30.014.13.7
thick0.540.410.480.46
swell1.210.370.890.70
press0.300.230.390.29
together0.35−0.040.480.40
firm (px)1.21.91.91.8
feather (px)2.95.26.76.0
rim (px)2.50.20.80.5
streak0.580.760.910.99
other0.730.730.400.78
distance, views taught from56.626.833.9
distance, the other five60.230.231.1

The room's limbs are unchanged: this is one limb. Tone at low resolution took no part in any of it.

The same limb, four renderers: splats made to compete

The sweep is a function worked out along a curve (a pressure, a band, bristles and their gaps) and then sampled into Gaussian deposits. Nothing in it is the deposits' own: limbSweep lays them with no residuals at all. So the last section showed a better brush model, not that splats are the way to draw it. Here the limb, its curve and the sweep's numbers are held fixed, and four renderers draw it:

Every number below is read from the real renderer, from the five views the teacher drew this limb in.

Where its ink is: squared error against the teacher's drawing, within the limb's width (×1000 a pixel):

the two views fitted tothe three not
bare paper98.198.7
the sweep, as deposits158.3179.2
the sweep, on a surface156.2167.8
deposits, fitted10.6107.2
the teacher's ink, on the surface13.6163.2

How its stroke runs: the readings along the stroke, on the teacher's five strips of this limb:

the teachersweep, depositssweep, surfacedeposits, fittedteacher's ink, surface
firm edge (px)1.62.41.52.41.5
soft edge (px)3.79.83.27.25.4
rim (px)3.40.73.10.82.7
thick0.320.380.320.350.37
body0.530.730.840.580.67
cover0.850.830.680.710.68
distance over all the readings114.283.963.855.5

What each fills, from the scene's camera far, medium and close (thousands of pixels of a 960 × 640 frame): the sweep's deposits 193, 531, 1,387 (their quads, ±3σ each way; 870, 804 and 490 of the 3,240 drawn at all); the fitted ones 250, 817, 1,679; the surface 7, 37, 75, each pixel once.

So, for a limb. The mark stays: its curve, its pressure, its identity, its edits. Splats don't earn their place drawing it. The surface draws the same gesture with the teacher's edges at a fraction of the fill, and takes a teacher's ink directly where splats need an optimiser to get the same. What this doesn't test is what a surface can't have: ink that stands off a surface or hangs in open space (fronds, a wash, mist), where every deposit is placed on its own. That's where splats still have to be tried against something.

What's weak in this: it's one limb and five strips. The teacher's drawings have other things crossing the limb, and they're painted on and fitted to with the rest. The specks are the surface's own addition, not the gesture's. And the cost on a phone isn't measured: ?scene=limb&perf draws each renderer close to and then 16 and 64 times over; ?scene=room&perf measures the room again, which now draws about 76,000 deposits where its last timing was taken at 23,000.

The room still draws its limbs in parts, as deposits.

The sweep on a bamboo culm (a first slice: Bamboo, ?brush=sweep)

Bamboo is the better place for this than the glasshouse: it already has solid things that hide what's behind them, light, wind, and a list of what deserves a mark. And the last section says how a mark on a thing with a surface should be drawn: on the surface. So the first slice isn't deposits at all. The sweep goes into Bamboo's own culm shader (client/bamboo/render.ts), which already works in the coordinates it needs: how far up the culm, how far along the internode, how far round, and each culm's own number. Behind a flag; without it Bamboo draws exactly as it did.

Four of Bamboo's pinned scenes, each as its board and in ink alone, with and without (devtools/bamboo-brush.mjs):

This is one of the four things the first comparison needs (a near culm, an overhead spray, a cut bank, a far stand). The other three aren't started. The far stand and the canopy's masses are where deposits would be tried: tone that sits in the air, handing over to drawn culms and leaves as they come near.

One frond cluster, restored

Taught by tone, the room's fronds faded almost out. That was the objective's doing: the teacher's leaves are where an image model put them, and at low resolution the only way to agree with leaves that aren't where yours are is to have none. So nothing is asked any more of where a leaf is. What the tone fit taught the strands and their leaves is withdrawn (room-taught.json: their roles' offsets zeroed, their 231 weights dropped), and the leaves are taught one at a time instead.

Reading the teacher's leaves (devtools/splats-leaves.mjs, .py). Where the room has fronds in a view, the restyle's ink is taken with its thin stems worn away, in connected pieces of a leaf's size. Each piece is measured on its own: its length and width (by the scale of the nearest strand), how pointed its ends are, how dark it is, its angle off straight down, how far the nearest other leaf is in leaf lengths, and how bunched they are against leaves scattered evenly. The room's own leaves are read the same way, from the real renderer.

Formed leaves (marks.ts, leaf; canopy.ts, LeafStyle). A leaf was one dab. It is now four deposits along its own axis, widest a third of the way out and narrowing to its tip, coming off its strand at an angle and turning round it leaf by leaf, three to a bunch. Each stays on the strand it was on: the scene's fronds are where they were. The style's numbers are moved toward the teacher's readings, each by the ratio of the teacher's reading to ours, three rounds, on the three views taught from:

leaves foundlength (cm)width (cm)ends against middledarknessangle off straight downto the nearest, in leaf lengthsbunched
the teacher4546.82.70.890.6220.7°1.380.62
a dab a leaf (as it was)1827.12.60.960.466.3°1.560.62
formed leaves3336.62.80.910.5818.7°1.360.53
the teacher, the other five views5147.83.00.880.6220.5°1.360.58
formed leaves, the other five views4977.42.90.920.5917.3°1.220.50

Air, water, cost

the room, as taught (22,780 deposits)draw time, median / 95th centile
the tunnel: still water / no water / air moving3 / 4 ms · 2 / 3 ms · 3 / 4 ms
the pool: still water / no water / air moving2 / 4 ms · 2 / 3 ms · 2 / 4 ms
the canopy: still water / no water / air moving4 / 5 ms · 3 / 4 ms · 4 / 5 ms
the canopy, drawn 2 / 4 / 8 times over7 / 8 ms · 8 / 11 ms · 8 / 11 ms
the canopy, drawn 16 times over14 / 18 ms (58 fps)
the walk (2,002 deposits)draw time, median / 95th centile
from 12 m / 5 m / 1.5 m1 / 2 ms · 2 / 3 ms · 2 / 2 ms
from 1.5 m, drawn 8 / 32 times over7 / 8 ms · 8 / 12 ms
from 1.5 m, drawn 128 times over20 / 23 ms (45 fps)

The Bench: one brush made dependable, one stroke made editable

?scene=bench (bench.ts; brush.ts). This answers a source review of the brush. Its five findings, all confirmed in the code, and what changed:

findingwhy it matterednow
element ids from a global counter; wash variation from that counterrebuilding a subject after visiting another scene could change itids hash a key: given, or the subject's scope and the element's place in it (stone/w3), or the element's content. Each element deposits from its own random stream, seeded by its id
wash reach and firm edge indexed by anchor sampleresampling a curve changed the washmeasured by arc length in scene units. Points are placed by true distance from the anchor polyline; the firm edge is laid every 12 mm
density added overlapping marks with no opacity compensationa quality setting darkened the drawingmarks scattered at d× the number are each 1/d as opaque, or wider where that can't be (faint). Strokes size marks to their spacing
feature ids in 4 bits, 16 uniformsa few objects exhaust ita 32-bit feature id per mark, its development read from a feature texture. Brushes gives each object its own range
contours carried their second normal in their colour, then forced to inka brush's contour colour was losta third texel per splat: support, second normal, feature, element. Contours keep their colour (Mist's stems are now light, as they asked to be)

Two renderer issues were fixed too:

Checked (devtools/splats.test.mjs, on a CPU rasterisation of the marks):

checkresult
a subject rebuilt after building other scenesidentical, mark for mark
a wash's tone at density 0.5 / 1 / 20.116 / 0.115 / 0.116
dry hatching's tone at density 0.5 / 1 / 20.021 / 0.023 / 0.023
a stroke's tone at density 0.5 / 1 / 20.0119 / 0.0116 / 0.0115
a straight anchor in 2 points vs 40same tone; layout correlation 1.000
an arc in 12 points vs 120tone 0.131 vs 0.128; layout correlation 0.997
a re-pressured strokesame id, its marks in the same places

The Bench shows the same stroke and wash on a plane (a board), a cylinder (a culm) and in open space (a volume wash round a free stroke). Each stroke is authored as control points; its wash is anchored to it.

An edit deposits again only the stroke and the wash anchored to it: 2 of 29 elements, about 400 of 4,600 marks, in 10–25 ms in headless Chrome. The rest come from the cache. The whole buffer is still re-uploaded and re-sorted; incremental upload is not done yet. Four edits undone restore the drawing exactly (the board checks this).

What it doesn't do yet:

Board: node cells/lab/devtools/splats-bench.mjs (bench-deposit.png: density and sides; bench-edit.png: a real tap and drag, dry, swell, undo).

Mist: a learned lighting function on splats

?scene=mist (mist.ts, mistnet.ts, devtools/splats-mist-train.mjs). After Charles Grassi's Neural Shaders, Explained With Clouds: a tiny network estimating the light a cloud scatters toward the camera from nearby density at several scales, density toward the sun and the view angle, while the renderer still does visibility and accumulation. (The article: <https://charlesgrassi.dev/blog/neural-volumetric-clouds/>.) The question here is whether a compact spatial asset (splats) plus a compact learned lighting function keeps more of an expensive scene's behaviour, within a phone's budget.

The scene is a procedural bank of mist between bamboo stems (the stems opaque in its density, so they shadow it), drawn with the brushes, under a moving sun.

The reference is an expensive light model, about 200 density evaluations per point:

The same splats (positions, sizes, opacity), seen from the same camera at the same resolution, are lit four ways:

methodwhere its light comes fromcost per splat
referencethe expensive model, on the CPU~200 density evaluations; seconds to update
bakedthe reference under one sun and view, frozennothing
analyticBeer–Lambert toward the sun from the six sun reads, with the phase, plus an ambient term; two weights fitted to the reference7 pyramid reads
neural13 → 16 → 16 → 2, 556 numbers, trained offline against the reference12 pyramid reads and ~500 multiply-adds

Following Grassi's method. The second version follows the capture of his article closely:

Trained on five banks × ten suns × three views (240,000 visible points). Measured where it wasn't trained, under suns in a held-out band of azimuth and elevation:

bakedanalyticneural
training banks, visible points: mean abs. error (mean light 0.59)0.3760.1340.084
unseen bank, visible points: mean abs. error (mean light 0.69)0.3980.1180.087
unseen bank, visible points: correlation0.330.910.93
unseen bank, the mist's splats: mean abs. error (mean light 0.59)0.3720.1280.086

Ablation. Each of the recipe's choices was undone in turn on the same features. Error is on visible points of the unseen bank:

recipemean abs. errorcorrelation
Grassi's (as above)0.0870.927
plain ReLU hidden0.0880.928
log1p target0.0880.928
one output (total light)0.0860.930
trained on splats, not visible points0.0870.923
all four undone (this experiment's first recipe)0.0890.928

What that says:

Performance, as the article reports it (laptop RTX 4080, cloud pass only):

Caveats:

?light=reference|baked|analytic|neural · ?az= ?el= (the sun) · ?bank=1|9 · ?diff (the error view). Board: node cells/lab/devtools/splats-mist.mjs; retrain: node cells/lab/devtools/splats-mist-train.mjs.

Where it could go

Roughly in order of what each teaches:

Flags

?scene=stone|brushes|composed|pen|field|bench|marks|walk|room|mist|garden|plush|nike|train · ?scene=marks&way=organised|organised+deposits|independent|style|unfitted|brush&only=<sheet> · ?scene=walk&d=<metres>&level=both|fronds|wash&auto · ?scene=room&view=tunnel|pool|canopy&s=<0…1>&style=taught|bench|plain&roots=0|band&light=<degrees>&mirror=0&moving&auto · ?field=stone|composed · ?flat (a fitted field's baseline) · ?compose=0 · ?halo=0 · ?halopx= · ?policy=ranks|rules|learned|taught · ?teacher=sumi|sumi-bold|sumi-light · ?judge · ?perf · ?ink (a captured scene in ink) · ?scene=pen&part=word|board|culm|bank|rock&reference · ?look=dry|wash · ?detail=0..1 · ?thin=even · ?only=wash · ?view=hero|grazing|reverse · ?orbit · ?detail=0.25|0.55|1 (sparse, intermediate, resolved) · ?brush · ?occluder=0 · ?clean (no controls) · ?url=… (any .splat or .ply CORS lets it read) · ?up=1 (for a file that's y-up) · ?mode=0..3 · ?size= · ?res= · ?count= (garden size) · ?noworkers · ?preview

Checks: node cells/lab/devtools/splats.test.mjs (formats, covariance, sort) and node cells/lab/devtools/splats-shot.mjs [garden plush nike train] (renders a board of each).

Every picture (42)

Splats

One stone

Brushes and the composed scene

A drawing policy

The pen

The bench and mist

Captured scenes in ink

Drawings fitted as fields

Marks as organised splats

Walking into a wash

The room

One limb, three ways

One limb, four renderers