Jekyll2024-01-24T23:20:51+00:00https://raphlinus.github.io/feed.xmlRaph Levien’s blogBlog of Raph Levien.
A note on Metal shader converter2023-06-12T18:05:42+00:002023-06-12T18:05:42+00:00https://raphlinus.github.io/gpu/2023/06/12/shader-converter<p>At WWDC, Apple introduced <a href="https://developer.apple.com/metal/shader-converter/">Metal shader converter</a>, a tool for converting shaders from DXIL (the main compilation target of HLSL in DirectX12) to Metal. While it is no doubt useful for reducing the cost of porting games from DirectX to Metal, I feel it does not move us any closer to a world of robust GPU infrastructure, and in many ways just adds more underspecified layers of complexity.</p>
<p>The specific feature I’m salty about is atomic barriers that allow for some sharing of work between threadgroups. These barriers are present in HLSL, and in fact have been since 2009, when <a href="https://en.wikipedia.org/wiki/Direct3D#Direct3D_11">Direct3D 11</a> and Shader Model 5 were first introduced. This barrier is not supported in Metal, and of the major GPU APIs, Metal is the only one that doesn’t support it. That holds back WebGPU’s performance (see <a href="https://github.com/gpuweb/gpuweb/discussions/3935">gpuweb#3935</a> for discussion), as WebGPU must be portable across the major APIs.</p>
<p>I’ve discussed the value of this barrier in my blog post <a href="https://raphlinus.github.io/gpu/2021/11/17/prefix-sum-portable.html">Prefix sum on portable compute shaders</a>, but I’ll briefly recap. Among other things, it enables a single-pass implementation of prefix sum, using a technique such as decoupled look-back or the <a href="https://dl.acm.org/doi/10.1145/2980983.2908089">SAM prefix sum</a> algorithm. A single-pass implementation can achieve the same throughput as memcpy, while a more traditional tree-reduction approach can at best achieve 2/3 that throughput, as it has to read the entire input in two separate dispatches. Further, tree reduction can actually be more complex to implement in practice, as the number of dispatches varies with the input size (it is typically <code class="language-plaintext highlighter-rouge">2 * ceil(log(n) / log(threadgroup size))</code>). Prefix sum, in turn, is an important primitive for advanced compute workloads. There are a number of instances of it in the <a href="https://github.com/linebender/vello">Vello</a> pipeline, and it’s also commonly used in stream compaction, decoding of variable length data streams, and compression.</p>
<p>I believe there are other important techniques that are similarly unlocked by the availability of these primitives. For example, Nanite’s advanced compute pipelines schedule work through job queues, and in general it is not possible to reliably coordinate work between different threadgroups (even within the same dispatch) without such a barrier.</p>
<h2 id="complexity-and-reasoning">Complexity and reasoning</h2>
<p>The GPU ecosystem exists at the knife edge of being strangled by complexity. A big part of the problem is that features tend to inhabit a quantum superposition of existing and not existing. Typically there is an anemic core, surrounded by a cloud of optional features. The Vulkan ecosystem is notorious for this: the <a href="https://vulkan.gpuinfo.org/listfeaturesextensions.php">extension list at vulkan.gpuinfo.org</a> currently lists 146 extensions.</p>
<p>The widespread use of shader translation makes the situation even worse. When writing HLSL that will be translated into other shader languages, it’s no longer sufficient to consider <a href="https://learn.microsoft.com/en-us/windows/win32/direct3dhlsl/d3d11-graphics-reference-sm5">Shader Model 5</a> to be a baseline, but rather the developer needs to keep in mind all the features that don’t translate to other languages. In some cases, the semantics change subtly (the rules for the various flavors “count leading zeros” when the input is 0 vary), and in other cases, like these device scoped barriers.</p>
<p>A separate category is things technically forbidden by the spec, but expected to work in practice. A good example here is the mixing of atomic and non-atomic memory operations (see <a href="https://github.com/gpuweb/gpuweb/issues/2229">gpuweb#2229</a>). The spirv-cross shader translation tool casts non-atomic pointers to atomic pointers to support this common pattern, which is technically undefined behavior in C++, but in practice lots of people would be unhappy if the Metal shader compiler did anything other than the reasonable thing. Since Metal’s semantics are based on C++, I’d personally love to see this resolved by adopting <a href="https://en.cppreference.com/w/cpp/atomic/atomic_ref">std::atomic_ref</a> from C++20 (Metal is still based on C++14). I’ll also note that the official Metal shader compiler tool generates <a href="https://gist.github.com/raphlinus/a8e0a3a3683127149b746eb37822bdc8">reasonable IR</a> for this pattern. It’s concerning that using open source tools such as spirv-cross triggers technical undefined behavior, but it’s probably not a big problem in practice.</p>
<p>I understand the incentives, but overall I find it disappointing that Metal chases shiny new features like ray-tracing, while failing to provide a solid, spec-compliant foundation for GPU compute.</p>
<h2 id="onward">Onward</h2>
<p>The Metal announcements from WWDC move us no closer to a world of robust GPU infrastructure. But there is much we can still do.</p>
<p>For one, there <em>is</em> a GPU infrastructure stack that is based on careful specification and conformance testing, and has two high quality, open source implementations enabling deployment to almost all reasonably current GPU hardware. I speak of course of WebGPU. It’s lacking the shiny features – raytracing, bindless, and cooperative matrix operations (marketed as “tensor cores” and quite important for maximum performance in AI workloads) – but what is there should work.</p>
<p>For two, we can cheer on the work of Asahi Linux. They have recently announced <a href="https://asahilinux.org/2023/06/opengl-3-1-on-asahi-linux/">OpenGL 3.1 support</a> on Apple Silicon, and an intent to implement Vulkan. That work may be highly challenging, as obviously that implies implementing barriers which the Apple GPU engineers haven’t been able to manage. But they have done consistently impressive work so far, and I certainly hope they succeed. If nothing else, their work will result in much better public documentation of the hardware’s capabilities and limitations.</p>
<p>I have a recommendations for Apple as well. I hope that they document which HLSL features are expected to work and which are not. Currently in their documentation (which is admittedly beta), it just says “Some features not supported,” which I personally find not very useful. I would also like to give them credit for clarifying the <a href="https://developer.apple.com/metal/Metal-Shading-Language-Specification.pdf">Metal Shading Language Specification</a> with respect to the scope of the <code class="language-plaintext highlighter-rouge">mem_device</code> flag to <code class="language-plaintext highlighter-rouge">threadgroup_barrier</code>. It now says, “The flag ensures the GPU correctly orders the memory operations to device memory for threads in the threadgroup,” which to a very careful reader does indicate threadgroup scope and no guarantee at device scope. Previously it <a href="https://github.com/gpuweb/gpuweb/pull/2297">said</a> “Ensure correct ordering of memory operations to device memory,” which could easily be misinterpreted as providing a device scope guarantee.</p>
<p>I am optimistic in the long term about having really good, portable infrastructure for GPU compute, but it is clear that we have a long way to go.</p>At WWDC, Apple introduced Metal shader converter, a tool for converting shaders from DXIL (the main compilation target of HLSL in DirectX12) to Metal. While it is no doubt useful for reducing the cost of porting games from DirectX to Metal, I feel it does not move us any closer to a world of robust GPU infrastructure, and in many ways just adds more underspecified layers of complexity.Simplifying Bézier paths2023-04-18T13:07:42+00:002023-04-18T13:07:42+00:00https://raphlinus.github.io/curves/2023/04/18/bezpath-simplify<p>Finding the optimal Bézier path to fit some source curve is, surprisingly, not yet a completely solved problem. Previous posts have given good solutions for specific instances: <a href="https://raphlinus.github.io/curves/2021/03/11/bezier-fitting.html">Fitting cubic Bézier curves</a> primarily addressed Euler spirals (which are very smooth), while <a href="https://raphlinus.github.io/curves/2022/09/09/parallel-beziers.html">Parallel curves of cubic Béziers</a> rendered parallel curves. In this post, I describe refinements of the ideas to solve the much more general problem of simplifying arbitrary paths. Along with the theoretical ideas, a reasonably solid implementation is landing in <a href="https://github.com/linebender/kurbo">kurbo</a>, a Rust library for 2D shapes.</p>
<p>The techniques in this post, and code in kurbo, come close to producing <em>the</em> globally optimum Bézier path to approximate the source curve, with fairly decent performance (as well as a faster option that’s not always minimal in the number of segments), though as we’ll see, there is considerable subtlety to what “optimum” means. Potential applications are many:</p>
<ul>
<li>Simplification of Bézier paths for more efficient publishing or editing</li>
<li>Conversion of fonts into cubic outlines</li>
<li>Tracing of bitmap images into vectors</li>
<li>Rendering of offset curves</li>
<li>Conversion from other curve types including NURBS, piecewise spiral</li>
<li>Distortions and transforms including perspective transform</li>
</ul>
<p>While the primary motivation is simplification of an existing Bézier path into a new one with fewer segments, the techniques are quite general. One of the innovations is the <code class="language-plaintext highlighter-rouge">ParamCurveFit</code> trait, which designed for efficient evaluation of any smooth curve segment for curve fitting. Implementations of this trait are provided for parallel curves and path simplification, but it is intentionally open ended and can be implemented by user code.</p>
<p>This work is an opportunity to revisit some of the topics from my <a href="https://levien.com/phd/thesis.pdf">thesis</a>, where I also explored systematic search for optimal Bézier fits. The new work is several orders of magnitude faster and more systematic about not getting stuck in local minima.</p>
<h2 id="the-paramcurvefit-trait">The ParamCurveFit trait</h2>
<p>At its best, a trait in Rust is a way to model some fact about the problem, then serves as an interface or abstraction boundary between pieces of code. The <code class="language-plaintext highlighter-rouge">ParamCurveFit</code> trait models an arbitrary curve for the purpose of generating a Bézier path approximating that source curve. The key insight is to sample both the position and derivative of the source curve for given parameters, and there is also a mechanism for detecting cusps (particularly important for parallel curves).</p>
<p>The curve fitting process uses those position and derivative samples to compute candidate approximation curves, then measure the error between source curve and approximation.</p>
<p>Two implementations are provided: parallel curves of cubic Béziers, and arbitrary Bézier paths for simplification. Implementing the trait is not very difficult, so it should be possible to add more source curves and transformations, taking advantage of powerful, general mechanisms to compute an optimized Bézier path.</p>
<p>The core cubic Bézier fitting algorithm is based on measuring the area and moment of the source curve. A default implementation is provided by the <code class="language-plaintext highlighter-rouge">ParamCurveFit</code> trait implementing numerical integration based on derivatives (Green’s theorem), but if there’s a better way to compute area and moment, source curves can provide their own implementation.</p>
<p>For Bézier paths, an efficient analytic implementation is possible. No blog post of mine is complete without some reference to a monoid, and this one does not disappoint. It’s relatively straightforward to compute area and moments from a cubic Bézier using symbolic evaluation of Green’s theorem. The area and moments of a <em>sequence</em> of Bézier segments is the sum of the individual values for each segment. Thus, we precompute the prefix sum of the areas and moments for all segments in a path, then can query an arbitrary range in O(1) time by taking the difference between the end point and the start point.</p>
<h2 id="error-metrics">Error metrics</h2>
<p>The problem of finding an optimal Bézier path approximation is always relative to some error metric. In general, an optimal path is one with a minimal number of segments while still meeting the error bound, and a minimal error compared to other paths with the same number of segments. A curve fitting algorithm will evaluate many error metrics, at least one for each candidate approximation to determine whether it meets the error bound. The simplest technique subdivides in half when the bound is not met, but a more sophisticated approach attempts to optimize the positions of the subdivision points as well.</p>
<p>The main error metric used for curve fitting is <a href="https://en.wikipedia.org/wiki/Fr%C3%A9chet_distance">Fréchet distance</a>. Intuitively, it captures the idea of the maximum distance between two curves while also preserving orientation of direction along a path (the related Hausdorff metric does not preserve orientation and so a curve with a sharp zigzag may have a small Hausdorff metric measured against a straight line). The difference is shown in the following illustration:</p>
<p><img src="/assets/hausdorf_vs_frechet.svg" alt="Comparison of Hausdorff and Fréchet distance metrics" class="center" /></p>
<p>Computing the exact Fréchet distance between two curves is not in general tractable, so we have to use approximations. It is important for the approximation to not underestimate, as this will yield a result that exceeds the error bound.</p>
<p>The classic <a href="https://ieeexplore.ieee.org/iel5/38/4055906/04055919">Tiller and Hanson</a> paper on parallel curves proposed a practical, reasonably accurate, and efficient error metric to approximate Fréchet distance. It samples the source curve at n points, and for each point casts a ray along the normal from the point, detecting intersections with the cubic approximation. The maxium distance from a source curve point to the corresponding intersection is the metric.</p>
<p>Unfortunately, there is a case Tiller-Hanson handles poorly, and equally unfortunately, it does come up when doing simplification of arbitrary paths. Consider a superellipse shape, approximated (badly) by a cubic Bézier with a loop. The rays strike only parts of the approximating curve and miss the loop entirely.</p>
<p><img src="/assets/simplify-t-h.svg" alt="Failure of Tiller-Hanson metric" /></p>
<p>Increasing the sampling density helps a little but doesn’t guarantee that the approximation will be well covered. Indeed, it is the high curvature in the source curve that makes this coverage more uneven.</p>
<p>A more robust metric is to parametrize the curves by arc length, and measure the distance between samples that share a corresponding portion of the total arc length. This ensures that all parts of both curves are considered. When curves are close (which is a valid assumption for curve fitting), it closely approximates Fréchet distance, though potentially can overestimate (because nearest points don’t necessarily coincide with arc length parametrization) and underestimate due to sampling.</p>
<p><img src="/assets/simplify-arc.svg" alt="Arc length parametrization is much more accurate" /></p>
<p>However, arc length parametrization is considerably slower (about a factor of 10) because it requires inverse arc length computations. The approach implemented in <a href="https://github.com/linebender/kurbo/pull/269">kurbo#269</a> is to classify whether the source curve is “spicy” (considering the deltas between successive normal angles) and use the more robust computation only in those cases.</p>
<h2 id="finding-subdivision-points">Finding subdivision points</h2>
<p>An extremely common approach is adaptive subdivision. Compute an approximation, evaluate the error metric, and subdivide in half when it is exceeded. This approach is simple, robust, and performant (the total number of evaluations is within a factor of 2 of the number of segments). However, it tends to produce results with more segments than absolutely needed. In the limit, it tends to be about 1.5 times the optimum.</p>
<p>Fancier techniques try to optimize the subdivision points to reduce the number of segments. That is equivalent to dividing the source curve into ranges such that each range is just barely below the threshold; ironically, it basically amounts to <em>maximizing</em> the error metric right up to the constraint.</p>
<p>One technique is given in section 9.6.4 of my <a href="https://levien.com/phd/thesis.pdf">thesis</a>. Basically you start on one end and for each segment find a subdivision point from the last point to one that’s just barely under the error threshold. Under the assumption that errors are monotonic (which is not always going to be the case), this finds the global minimum number of segments needed. The last segment will have an error well below the threshold. Then, another search finds the minimum error for which this process yields the same number of segments. Again, if error is monotonic, the result is the Fréchet distance of all segments being equal, which is (at least roughly) equivalent to the overall Fréchet distance being minimized.</p>
<p>For smooth source curves, monotonic error is a reasonable assumption. Even so, the above technique seems to work fairly robustly, producing fewer segments than simple adaptive subdivision, though it is somewhere around 50x slower.</p>
<p>There may be scope to improve this optimization process further. Crates like <a href="https://crates.io/crates/argmin">argmin</a> implement general purpose multidimensional optimization algorithms, and it’s worth exploring whether those could produce equally good results with fewer evaluations.</p>
<h2 id="bumps">Bumps</h2>
<p>Something unexpected arose during testing: the resulting “optimum” simplified paths had bumps. Obviously at first I thought this was some kind of failure of the algorithm, but now I think something more subtle is happening.</p>
<p>The solution below is with an error tolerance of 0.15, which produces a total of 43 segments:</p>
<p><img src="/assets/simplify-bump.svg" alt="Simplified path showing a bump" /></p>
<p>Near the bottom is a bump. It almost looks like a discontinuity (and other similar examples even more so), but on closer examination it is simply one very long and one very short control arm (ie distance between control point and corresponding endpoint), which creates a high degree of curvature variation:</p>
<p><img src="/assets/simplify-bump-zoom.png" alt="Closeup of bump in previous illustration" /></p>
<p>The underlying problem is that Fréchet distance optimizes only for a distance metric, and does not by itself guarantee a low angle (or curvature) error. In many cases, these objectives are not in tension – the curve that minimizes distance error also smoothly hugs the source curve. But there are cases where there is in fact a tradeoff, and when such tradeoffs exist, agressively optimizing for one causes the other to suffer.</p>
<p>For this particular range of the test data, I believe the Fréchet-minimizing cubic Bézier approximation does indeed exhibit the bump; tweaking the parameters will certainly improve smoothness, but also result in a greater distance from the source curve.</p>
<p>This state of affairs is not acceptable in most applications. It is evidence that Fréchet does not capture all aspects of the objective function. Section 9.2 of the thesis discusses this point.</p>
<p><img src="/assets/simplify-distance-error.png" alt="Figure 9.2 from the thesis: Two circle approximations with equal distance error" /></p>
<p>Given that aggressively optimizing Fréchet may give undesirable results, what is to be done? The most systematic approach would be to design an error metric that takes both distance and angle into account, and calibrate it to correlate strongly with perception. That is perhaps a tall order, requiring research to properly establish tuning parameters, and likely with complexity and runtime performance implications to evaluate. Even so, I think it should be pursued.</p>
<p>A simpler approach, implemented in the current code, is to recognize that these bumpy cubic Béziers can be recognized by their parameters; when the distance from the endpoints to the control points roughly equal to the chord length, then curves can have cusps or nearly so. These are the δ values from the core quartic-based curve fitting solver; a simple approach would be to simply exclude curves with δ values greater than some threshold (0.85 works well), and this also improves performance by decreasing the number of error evaluations needed, but a more sophisticated approach is to multiply the error by a penalty factor for larger δ. One advantage of the latter approach is that it is still possible to fit every actual exact Bézier input.</p>
<p>Applying this tweak gives a much smoother result, though it does require one more segment (44 rather than 43 previously).</p>
<p><img src="/assets/simplify-smooth.svg" alt="Smoother simplified path" /></p>
<p>Another point to make at this point is that the number of path segments needed scales very gently as the error bound is tightened; changing the threshold from 0.15 to 0.05 increases the number of segments only from 44 to 60. In the limit, the error scales as O(n^6). This extremely fast convergence is one reason that cubic Béziers can be considered a universal representation of curved paths; while some other representation such as NURBS may be able to represent conic sections exactly, any such smooth curve can be approximated to an extremely tight tolerance using a modest number of cubic segments. The path simplification technique in this blog (and in kurbo) gives a practical way to actually attain this degree of accuracy.</p>
<p><img src="/assets/simplify-smooth-0_05.svg" alt="Smooth simplified path, error 0.05" /></p>
<p>And taking the error down to 0.01 requires only 90 segments. This creates an extremely close fit to the source curve, still without requiring an excessive number of segments.</p>
<p><img src="/assets/simplify-smooth-0_01.svg" alt="Smooth simplified path, error 0.01" /></p>
<p>These are vector images, so please feel free to open them in a new tab and zoom in to inspect them more carefully. You should see all solutions display an impressive amount of control over the curvature variation afforded by cubic Béziers. The code is <a href="https://github.com/linebender/kurbo/pull/269">available</a>, so also feel free to experiment with it with your own test data and for your own applications. I’m especially interested in cases where the algorithm doesn’t perform well; it hasn’t been carefully validated yet.</p>
<h2 id="low-pass-filtering">Low pass filtering</h2>
<p>The “best” curve depends on the use case. When the source curve is an authoritative source of truth, then making each segment G1 continuous with it at the endpoints is reasonable. However, when the source curve is noisy, perhaps because it’s derived from a scanned image or digitizer samples, then an optimum simplified path may have angle deviations relative to the source that make the overall curve smoother.</p>
<p>I haven’t been working from noisy data and haven’t done experiments, but I do suggest a possibility: use a global optimizing technique such as that provided by <a href="https://crates.io/crates/argmin">argmin</a>, and jointly optimize both the location of the subdivision points and a delta to be applied to the angle (equally on both sides of a subdivision, so the resulting curve remains G1). Another possibility is to explicitly apply a low-pass filter, tuned so the amount of smoothing is consistent with the amount of simplification. In any case, using the existing code with no further tuning may yield less than optimum results.</p>
<h2 id="discussion">Discussion</h2>
<p>The current code in kurbo is likely considerably better than what’s in your current drawing tool, but curve fitting remains work in progress. The core primitive feels solid, but applying it might require different tuning depending on the specifics of the use case. I invite collaboration along these lines.</p>
<p>Many of the points in this blog were discussed in Zulip threads, particularly <a href="https://xi.zulipchat.com/#narrow/stream/260979-kurbo/topic/curve.20offsets">curve offsets</a> and <a href="https://xi.zulipchat.com/#narrow/stream/260979-kurbo/topic/More.20thoughts.20on.20cubic.20fitting">More thoughts on cubic fitting</a>. The Zulip is a great place to discuss these ideas, and we welcome newcomers and people bringing questions.</p>
<p>Thanks to Siqi Wang for insightful questions and making test data available.</p>Finding the optimal Bézier path to fit some source curve is, surprisingly, not yet a completely solved problem. Previous posts have given good solutions for specific instances: Fitting cubic Bézier curves primarily addressed Euler spirals (which are very smooth), while Parallel curves of cubic Béziers rendered parallel curves. In this post, I describe refinements of the ideas to solve the much more general problem of simplifying arbitrary paths. Along with the theoretical ideas, a reasonably solid implementation is landing in kurbo, a Rust library for 2D shapes.Moving from Rust to C++2023-04-01T13:00:42+00:002023-04-01T13:00:42+00:00https://raphlinus.github.io/rust/2023/04/01/rust-to-cpp<p><strong>Note to readers: this is an April Fools post, intended to satirize bad arguments made against Rust. I have mixed feelings about it now, as I think people who understood the context appreciated the humor, but some people were confused. That’s partly because it’s written in a very persuasive style, and I mixed in some good points. For a <em>good</em> discussion of C++ safety in particular, see JF Bastien’s talk <a href="https://www.youtube.com/watch?v=Gh79wcGJdTg">Safety and Security: The Future of C++</a>.</strong></p>
<p>I’ve been involved in Rust and the Rust community for many years now. Much of my work has been related to creating infrastructure for building GUI toolkits in Rust. However, I have found my frustrations with the language growing, and pine for the stable, mature foundation provided by C++.</p>
<h2 id="choice-of-build-systems">Choice of build systems</h2>
<p>One of the more limiting aspects of the Rust ecosystem is the near-monoculture of the Cargo build system and package manager. While other build systems are used to some extent (including <a href="https://mmapped.blog/posts/17-scaling-rust-builds-with-bazel.html">Bazel</a> when integrating into larger polyglot projects), they are not well supported by tools.</p>
<p>In C++, by contrast, there are many choices of build systems, allowing each developer to pick the one best suited for their needs. A very common choice is CMake, but there’s also Meson, Blaze and its variants as well, and of course it’s always possible to fall back on Autotools and make. The latter is especially useful if we want to compile Unix systems such as AIX and DEC OSF/1 AXP. If none of those suffice, there are plenty of other choices including SCons, and no doubt new ones will be created on a regular basis.</p>
<p>We haven’t firmly decided on a build system for the Linebender projects yet, but most likely it will be CMake with plans to evaluate other systems and migrate, perhaps Meson.</p>
<h2 id="safety">Safety</h2>
<p>Probably the most controversial aspect of this change is giving up the safety guarantees of the Rust language. However, I do not think this will be a big problem in practice, for three reasons.</p>
<p>First, I consider myself a good enough programmer that I can avoid writing code with safety problems. Sure, I’ve been responsible for some CVEs (including <a href="https://nvd.nist.gov/vuln/detail/CVE-2016-2414">font parsing code in Android</a>), but I’ve learned from that experience, and am confident I can avoid such mistakes in the future.</p>
<p>Second, I think the dangers from memory safety problems are overblown. The Linebender projects are primarily focused on 2D graphics, partly games and partly components for creating GUI applications. If a GUI program crashes, it’s not that big a deal. In the case that the bug is due to a library we use as a dependency, our customers will understand that it’s not our fault. Memory safety bugs are not fundamentally different than logic errors and other bugs, and we’ll just fix those as they surface.</p>
<p>Third, the C++ language is evolving to become safer. We can use modern C++ techniques to avoid many of the dangers of raw pointers (though string views can outlive their backing strings, and closures as used in UI can have very interesting lifetime patterns). The <a href="https://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines">C++ core guidelines</a> are useful, even if they’re honored in the breach. And, as discussed in the next section, the language itself can be expected to improve.</p>
<h2 id="language-evolution">Language evolution</h2>
<p>One notable characteristic of C++ is the rapid adoption of new features, these days with a new version every 3 years. C++20 brings us modules, an innovative new feature, and one I’m looking forward to actually being implemented fairly soon. Looking forward, C++26 will likely have stackful coroutines, the ability to embed binary file contents to initialize arrays, a safe range-based for loop, and many other goodies.</p>
<p>By comparison, the pace of innovation in Rust has become more sedate. It used to be quite rapid, and async has been a major effort, but recently landed features such as generic associated types have more of the flavor of making combinations of existing features work as expected rather than bringing any fundamentally new capabilities; the forthcoming “type alias impl trait” is similar. Here, it is clear that Rust is held back by its commitment to not break existing code (enacted by the edition mechanism), while C++ is free to implement backwards compatibility breaking changes in each new version.</p>
<p>C++ has even more exciting changes to look forward to, including potentially a new syntax as proposed by Herb Sutter <a href="https://github.com/hsutter/cppfront">cppfront</a>, and even the new C++ compatible Carbon language.</p>
<p>Fortunately, we have excellent leadership in the C++ community. Stroustrup’s <a href="https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2023/p2739r0.pdf">paper on safety</a> is a remarkably wise and perceptive document, showing a deep understanding of the problems C++ faces, and presenting a compelling roadmap into the future.</p>
<h2 id="community">Community</h2>
<p>I’ll close with some thoughts on community. I try not to spend a lot of time on social media, but I hear that the Rust community can be extremely imperious, constantly demanding that projects be rewritten in Rust, and denigrating all other programming languages. I will be glad to be rid of that, and am sure the C++ community will be much nicer and friendlier. In particular, the C++ subreddit is well known for their <a href="https://www.reddit.com/r/cpp/comments/128xpps/comment/jekwww0">sense of humor</a>.</p>
<p>The Rust community is also known for its code of conduct and other policies promoting diversity. I firmly believe that programming languages should be free of politics, focused strictly on the technology itself. In computer science, we have moved beyond prejudice and discrimination. I understand how some identity groups may desire more representation, but I feel it is not necessary.</p>
<h2 id="conclusion">Conclusion</h2>
<p>Rust was a good experiment, and there are aspects I will look back on fondly, but it is time to take the Linebender projects to a mature, production-ready language. I look forward to productive collaboration with others in the C++ community.</p>
<p>We’re looking for people to help with the C++ rewrite. If you’re interested, please join our <a href="https://xi.zulipchat.com">Zulip instance</a>.</p>
<p>Discuss on <a href="https://news.ycombinator.com/item?id=35400047">Hacker News</a> and <a href="https://old.reddit.com/r/rust/comments/128m5wn/moving_from_rust_to_c/">/r/rust</a>.</p>Note to readers: this is an April Fools post, intended to satirize bad arguments made against Rust. I have mixed feelings about it now, as I think people who understood the context appreciated the humor, but some people were confused. That’s partly because it’s written in a very persuasive style, and I mixed in some good points. For a good discussion of C++ safety in particular, see JF Bastien’s talk Safety and Security: The Future of C++.Requiem for piet-gpu-hal2023-01-07T17:12:42+00:002023-01-07T17:12:42+00:00https://raphlinus.github.io/rust/gpu/2023/01/07/requiem-piet-gpu-hal<p><a href="https://github.com/linebender/vello">Vello</a> is a new GPU accelerated renderer for 2D graphics that relies heavily on compute shaders for its operation. (It was formerly known as piet-gpu, but we renamed it recently because it is no longer based on the [Piet] render context abstraction, and has been substantially rewritten). As such, it depends heavily on having good infrastructure for writing and running compute shaders, consistent with its goals of running portably and reliably across a wide range of GPU hardware. Unfortunately, there hasn’t been a great off-the-shelf solution for that, so we’ve spent a good part of the last couple years building our own hand-rolled GPU abstraction layer, piet-gpu-hal.</p>
<p>During that time, the emerging <a href="https://www.w3.org/TR/webgpu/">WebGPU</a> standard, along with the allied <a href="https://www.w3.org/TR/WGSL/">WGSL</a> shader language, showed promise for providing such infrastructure, but it has seemed immature, and also has limitations that would have prevented us from doing experiments involving advanced GPU features. This motivated continued work on developing our own abstraction layer, so much so that the <a href="https://github.com/linebender/vello/blob/main/doc/vision.md#why-not-wgpu">piet-gpu vision</a> document has a section entitled “Why not wgpu?”.</p>
<p>Recently, however, we switched <a href="https://github.com/linebender/vello">Vello</a> to using <a href="https://wgpu.rs">wgpu</a> for its GPU infrastructure and deleted piet-gpu-hal entirely. This was the right decision, but there was something special about piet-gpu-hal and I hope its vision is realized some day. This post talks about the reasons for the change and the tradeoffs involved.</p>
<p>In the process of doing piet-gpu-hal, we learned a <em>lot</em> about portable compute infrastructure. We will apply that knowledge to improving WebGPU implementations to better suit our needs.</p>
<h2 id="goals">Goals</h2>
<p>The goals of piet-gpu-hal were admirable, and I believe there is still a niche for something that implements them. Essentially, it was to provide a lightweight yet powerful runtime for running compute shaders on GPU. Those goals are fulfilled to a large extent by WebGPU, it’s just that it’s not quite as lightweight and not quite as powerful, but on the other hand the developer experience is much better.</p>
<p>The most important goal, especially compared with WebGPU implementations, is to reduce the cost of shader compilation by precompiling them to the intermediate representation (IR) expected by the GPU, rather than compiling them at runtime. This avoids anywhere from about 1MB to about 20MB of binary size for the shader compilation infrastructure, and somewhere around 10ms to 100ms of startup time to compile the shaders.</p>
<p>An additional goal was to unlock the powerful capabilities present on many (but not all) GPUs by runtime query. For our needs, the most important ones are subgroups, descriptor indexing, and detection of unified memory (avoiding the need for separate staging buffers). However, for the coming months we are prioritizing getting things working well for the common denominator, which is well represented by WebGPU, as it’s generally possible to work around the lack of such features by doing a bit more shuffling around in memory.</p>
<h2 id="implementation-choices">Implementation choices</h2>
<p>In this section, I’ll talk a bit about implementation choices made in piet-gpu-hal and their consequences.</p>
<h3 id="shader-language">Shader language</h3>
<p>After considering the alternatives, we landed on GLSL as the authoritative source for writing shaders. HLSL was in the running, and it seems to be the most popular choice (primarily in the game world) for writing portable shaders, but ultimately GLSL won because it is capable of expressing <em>all</em> of what Vulkan can do. In particular, I wanted to experiment with the Vulkan memory model.</p>
<p>Another choice considered seriously was <a href="https://github.com/EmbarkStudios/rust-gpu">rust-gpu</a>. That looks promising, and it has many desirable properties, not least being able to run the same code on CPU and GPU, but just isn’t mature enough. Hopefully that will change. I think porting Vello to it would be an interesting exercise, and would shake out many of the issues needing to be solved to use it in production.</p>
<p>Another appealing choice would be <a href="https://www.circle-lang.org/">Circle</a>, a C++ dialect that targets compute shaders among other things. It might have made it over the edge if it were released as open source; someone should buy out Sean’s excellent work and relicense it.</p>
<h3 id="ahead-of-time-compilation">Ahead of time compilation</h3>
<p>From the GLSL source, we also had a pipeline to compile this to shader IR. For all targets, the first step was <a href="https://github.com/KhronosGroup/glslang">glslangValidator</a> to compile the GLSL to SPIR-V. And for Vulkan, that was sufficient.</p>
<p>For DX12, the next step was <a href="https://github.com/KhronosGroup/SPIRV-Cross">spirv-cross</a> to convert the SPIR-V to HLSL, followed by <a href="https://github.com/microsoft/DirectXShaderCompiler">DXC</a> to convert this to DXIL. We really wanted to use DXC, especially over relying on the system provided shader compiler (some arbitrary version of FXC), both to access advanced features in Shader Model 6 such as wave operations, and also because FXC is somewhat buggy and we have evidence it will cause problems. In fact, I see the current reliance on FXC to be one of the biggest risks for WebGPU working well on Windows. (In the longer term, it is likely that both Tint and naga will compile WGSL directly to DXIL and perhaps DXBC, which will solve these problems, but will take a while)</p>
<p>For Metal, we used spirv-cross to convert SPIR-V to Metal Shading Language. We intended to go one step further, to AIR (also known as metallib), using the <a href="https://developer.apple.com/documentation/metal/shader_libraries/building_a_library_with_metal_s_command-line_tools">command-line tools</a>, but we didn’t actually do that, as it would have made setting up the CI more difficult.</p>
<p>All this was controlled through a simple hand-rolled ninja file. At first, we ran this by hand and committed the results, but it was tricky to make sure all the tools were in place (which would have been considerably worse had we required both DXC with the <a href="https://www.wihlidal.com/blog/pipeline/2018-09-16-dxil-signing-post-compile/">signing DLL</a> in place and the Metal tools), and it also resulted in messy commits, prone to merge conflicts. Our solution to this was to run <a href="https://github.com/linebender/vello/blob/480e5a5e2fb1ed5c38da083bfa00c1ae6b9b2486/doc/shader_compilation.md">shader compilation in CI</a>, which was better but still added considerably to the friction, as it was easy to be in the wrong branch for committing PRs.</p>
<h3 id="gpu-abstraction-layer">GPU abstraction layer</h3>
<p>The core of piet-gpu-hal was an abstraction layer over GPU APIs. Following the tradition of gfx-hal, we called this a HAL (hardware abstraction layer), but that’s not quite accurate. Vulkan, Metal, and DX12 are all hardware abstraction layers, responsible among other things for compiling shader IR into the actual ISA run by the hardware, and dispatching compute and other work.</p>
<p>The design was based loosely on gfx-hal, but more tuned to our needs. To summarize, gfx-hal was a layer semantically very close to Vulkan, so that the Vulkan back-end was a very thin layer, and the Metal back-end resembled MoltenVK, with the DX12 back-end also simulating Vulkan semantics in terms of the underlying API. This ultimately wasn’t a very satisfying approach, and for wgpu was <a href="https://gfx-rs.github.io/2021/08/18/release-0.10.html#pure-rust-graphics">deprecated in favor of wgpu-hal</a>.</p>
<p>We only implemented the subset needed for compute, which turned out to be a significant limitation because we found we also needed to run the rasterization pipeline for some tasks such as image scaling. The need to add new functionality to the HAL was also a major friction point. We <em>were</em> able to add features like runtime query for advanced capabilities such as subgroup size control, and a polished timer query implementation on all platforms (the latter is still not quite there for wgpu). Overall it worked pretty well.</p>
<p>I make one general observation: all these APIs and HALs are very object oriented. There are objects representing the adapter, device, command lists, and resources such as buffers and images. Managing these objects is nontrivial, especially because the lifetimes are complex due to the asynchronous nature of GPU work submission; you can’t destroy a resource until all command buffers referencing it have completed. These patterns are fairly cumbersome, and especially don’t translate easily to Rust.</p>
<p>For others seeking to build cross-platform GPU infrastructure, I suggest exploring a more declarative, data-oriented approach. In that approach, build a declarative render graph using simple value types as much as possible, then write specialized engines for each back-end API. This should lead to more straightforward code with less dynamic dispatching, and also resolve the need to find common-denominator abstractions. We are moving in this direction in Vello, and may explore it further as we encounter the need for native back-ends.</p>
<h2 id="webgpu-pain-points">WebGPU pain points</h2>
<p>Overall the task of porting piet-gpu to WGSL went well, but we ran into some issues. We expect these to be fixed and improved, but at the moment the experience is a bit rough. In particular, the demo is not running on DX12 at all, either in Chrome Canary for the WebGPU version or through wgpu. The main sticking point is uniformity analysis, which only very recently has a good solution in WGSL (<a href="https://github.com/gpuweb/gpuweb/pull/3586">workgroupUniformLoad</a>) and the implementation hasn’t fully landed.</p>
<h2 id="collaboration-with-community">Collaboration with community</h2>
<p>A major motivation for switching to WGSL and WebGPU is to interoperate better with other projects in that ecosystem. We already have a demo of <a href="https://github.com/linebender/vello/tree/main/examples/with_bevy">Bevy interop</a>, which is particularly exciting.</p>
<p>One such project is <a href="https://github.com/wgsl-analyzer/wgsl-analyzer">wgsl-analyzer</a>, which gives us interactive language server features as we develop WGSL shader code: errors and warnings, inline type hints, go to definition, and more. Thanks especially to Daniel McNab, they’ve been super responsive to our needs and we maintain a working configuration. I strongly recommend using such tools; the old way sometimes felt like submitting shader code on a stack of punchcards to the shader compiler.</p>
<p>Obviously another major advantage is deploying to the web using WebGPU, which we have working in at least prototype form. With an <a href="https://groups.google.com/a/chromium.org/g/blink-dev/c/VomzPhvJCxI/m/SUhU9Z0vAgAJ">intent to ship</a> from Chrome and active involvement from Mozilla and Apple, the prospect of WebGPU shipping in usable state on real browsers seems close.</p>
<h2 id="precompiled-shaders-on-roadmap">Precompiled shaders on roadmap?</h2>
<p>The goals of a lightweight runtime for GPU compute remain compelling, and in our roadmap we plan to return to precompiled shaders so it is possible to use Vello without a need to carry along runtime shader compilation infrastructure, and also to more aggressively exploit advanced GPU features. There are two avenues we are exploring.</p>
<p>One is to add support for precompiled shaders to either wgpu or wgpu-hal. We have an <a href="https://github.com/gfx-rs/wgpu/issues/3103">issue</a> with some thoughts. The advantage of this approach is that it potentially benefits many applications developed on WebGPU, for example native binary distributions of games. If it is possible to identify all <a href="https://therealmjp.github.io/posts/shader-permutations-part1/">permutations</a> of shaders which will actually be used, and bundle the IR.</p>
<p>The other is to add additional native renderer back-ends. The new architecture is much more declarative and less object oriented, so the module that runs the render graph can be written directly in terms of the target API rather than going through an abstraction layer. If there is a desire to ship Vello for a targeted application (for example, animation playback on Android), this will be the best way to go.</p>
<h2 id="other-gpu-compute-clients">Other GPU compute clients</h2>
<p>One thing we were watching for was whether there was any interest in using piet-gpu-hal for other applications. Matt Keeter did some <a href="https://github.com/mkeeter/fidget/blob/1b41b6b8e4bdb017e2ca28c151391a4a080b581a/jitfive/src/metal.rs">experimentation</a> with it, but otherwise there was not much interest. Sometimes the “portable GPU compute” space feels a bit lonely.</p>
<p>An intriguing potential application space is machine learning. It would be an ambitious but doable project to get, say, Stable Diffusion running on portable compute using either piet-gpu-hal or something like it, so that very little runtime (probably less than a megabyte of code) would be required. Related projects include <a href="https://kompute.cc/">Kompute.cc</a>, which runs machine learning workloads but is Vulkan only, and also <a href="https://google.github.io/mediapipe/">MediaPipe</a>.</p>
<p>One downside to trying to implement machine learning workloads in terms of portable compute shaders is that it doesn’t get access to neural accelerators such the <a href="https://github.com/hollance/neural-engine">Apple Neural Engine</a>. When running in native Vulkan, you <em>may</em> get access to <a href="https://www.khronos.org/assets/uploads/developers/presentations/Cooperative_Matrix_May22.pdf">cooperative matrix</a> features, which on Nvidia are branded “tensor cores,” but for the most part these are proprietary vendor extensions and it is not clear if and when they might be exposed through WebGPU. Even so, at least on Nvidia hardware it seems likely that using these features can unlock very high performance.</p>
<p>Going forward, one approach I find particularly promising for running machine learning is <a href="https://github.com/webonnx/wonnx">wonnx</a>, which implements the ONNX spec on top of WebGPU. No doubt in the first release, performance will lag highly tuned native implementations considerably, but once such a thing exists as a viable open source project, I think it will be improved rapidly. And WebGPU is not standing still…</p>
<h2 id="beyond-webgpu-10">Beyond WebGPU 1.0</h2>
<p>WebGPU 1.0 is basically a “least common denominator” of current GPUs. This has advantages and disadvantages. It means that there are fewer choices (and fewer permutations), and that code written in WGSL can run on all modern GPUs. The downside is that there are a number of features that most (but not all) current GPUs have that can speed things further, and those features are not available.</p>
<p>The likely path through this forest is to define extensions to WebGPU. These can start as privately implemented extensions, running in native, then, hopefully based on that experience, proposed for standardization on the web. They would be optional, meaning that more shader permutations will need to be written to make use of them when available, or fall back when not. One such extension, fp16 arithmetic, has already been standardized, though we don’t yet exploit it in Vello.</p>
<p>In Vello, we have identified three promising candidates for further extension.</p>
<p>First, <a href="https://chunkstories.xyz/blog/a-note-on-descriptor-indexing/">descriptor indexing</a>, which is the ability to create an array of textures, then have a shader dynamically access textures from that array. Without it, a shader can only have a fixed number of textures bound. To work around that limitation, we plan to have an image atlas, copy the source images into that atlas using rasterization stages, then access regions (defined by uv quads) from the single binding. We don’t expect performance to be that bad, as such copies are fairly cheap, but it does require extra memory and is not ideal. For fully native, the GPU world is moving to <a href="https://alextardif.com/Bindless.html">bindless</a>, which is popular in DX12, and the recent <a href="https://www.khronos.org/blog/vk-ext-descriptor-buffer">descriptor buffer</a> extension makes Vulkan work the same way. It is likely that WebGPU will standardize on something more like descriptor indexing than full bindless, because the latter is basically raw pointers and thus unsafe by design. In any case, see the <a href="https://github.com/linebender/vello/issues/176">image resources</a> issue for more discussion.</p>
<p>Second, device-scoped barriers, which unlock single-pass (decoupled look-back) prefix sum techniques. They are not present in Metal, but otherwise should be straightforward to add. I wrote about this in my <a href="https://raphlinus.github.io/gpu/2021/11/17/prefix-sum-portable.html">Prefix sum on portable compute shaders</a> blog post. In the meantime, we are using multiple dispatches, which is much more portable, but not quite as performant.</p>
<p>Third, subgroups, also known as SIMD groups, wave operations, and warp operations. Within a workgroup, especially for stages resembling prefix sum, Vello uses a lot of workgroup shared memory and barriers. With subgroups, it’s possible to reduce that traffic, in some cases dramatically. That should make the biggest difference on Intel GPUs, which have relatively slow shared memory.</p>
<p>Unfortunately, there is a tricky aspect to subgroups, which is that in most cases there is no control in advance over the subgroup size, the shader compiler picks it on the basis of a heuristic. The <a href="https://registry.khronos.org/vulkan/specs/1.3-extensions/man/html/VK_EXT_subgroup_size_control.html">subgroup size control</a> extension, which is mandatory in Vulkan 1.3, fixes this, and allows writing code specialized to a particular subgroup size. Otherwise, it will be necessary to write a lot of conditional code and hope that the compiler constant-folds the control flow based on subgroup size. Another challenge is that it is more difficult to test code for a particular subgroup size, in contrast to workgroup shared memory which is quite portable. More experimentation is needed to determine whether subgroups <em>without</em> the size control extension work well across a wide range of hardware.</p>
<p>And on the topic of that experimentation, it’s difficult to do so without adequate GPU infrastructure. I may find myself reaching for the archived version of piet-gpu-hal.</p>
<h2 id="conclusion">Conclusion</h2>
<p>Choosing the right GPU infrastructure depends on the goals, as sadly there is not yet a good consensus choice for a GPU runtime suitable for a wide variety of workloads including compute. For the goals of researching the cutting edge of performance, hand-rolled infrastructure was the right choice, and piet-gpu-hal served that well. For the goal of lowering the friction for developing our engine, and also interoperating with other projects, WebGPU and wgpu are a better choice. Our experience with the port suggests that the performance and features are good enough, and that it is a good experience all-around.</p>
<p>We hope to make Vello useful enough to use in production within the next few months. For many applications, WebGPU will be an appropriate infrastructure. For others, where the overhead of runtime shader compilation is not acceptable, we have a path forward but will need to consider alternatives. Either ahead-of-time shader compilation can be retrofitted to wpgu, or we will explore a more native approach.</p>
<p>In any case, we look forward to productive development and collaboration with the broader community.</p>Vello is a new GPU accelerated renderer for 2D graphics that relies heavily on compute shaders for its operation. (It was formerly known as piet-gpu, but we renamed it recently because it is no longer based on the [Piet] render context abstraction, and has been substantially rewritten). As such, it depends heavily on having good infrastructure for writing and running compute shaders, consistent with its goals of running portably and reliably across a wide range of GPU hardware. Unfortunately, there hasn’t been a great off-the-shelf solution for that, so we’ve spent a good part of the last couple years building our own hand-rolled GPU abstraction layer, piet-gpu-hal.Raph’s reflections and wishes for 20232022-12-31T14:44:42+00:002022-12-31T14:44:42+00:00https://raphlinus.github.io/personal/2022/12/31/raph-2023<p>This post reflects a bit on 2022 and contains wishes for 2023. It mixes project and technical stuff, which I write about fairly frequently, and more personal reflections, which I don’t.</p>
<h2 id="reflections-on-2022">Reflections on 2022</h2>
<p>Overall 2022 was a good year for me, though it was stressful and challenging in some ways. A lot of the stuff I did I’ve talked about in this blog, but there are a few other things, and quite a few more things in the pipeline. It was not a year of shipping anything big, and that is one thing I’d like to see different in 2023.</p>
<p>What I do straddles a number of more traditional categories. I do research, but most of my output is blog posts and code, not academic papers. My area of focus is primarily 2D graphics and GUI infrastructure, and these are to a large extent neglected step-children of academic computer science - there aren’t conferences or journals that specialize in the topic, nor are there decent textbooks. As a result, knowledge tends to be somewhat arcane, and just as much “lore” shared among practitioners as a literature. Even so, I find the field intellectually very stimulating and consider this odd situation to be an opportunity.</p>
<p>My portfolio of projects is very ambitious, as it basically includes an entire Rust UI toolkit plus a lot of the supporting technologies for that. It’s arguably too much for one person to take on, but I’m trying my best to make it work, largely by fostering a community around the projects. In late 2022, a big step was setting up weekly office hours, one hour a week, where we check in and discuss the various projects. I think that’s working well and look forward to continuing it.</p>
<h2 id="on-happiness">On happiness</h2>
<p>I’m not one of those “quantified self” people, but I have noticed that my happiness tends to correlate pretty directly with how much code I’m writing. I’m sure some of that is simply because if there’s stressful stuff going on that gets in the way of coding, that makes me unhappy, but obviously I just really enjoy it.</p>
<p>In particular, I love solving deep puzzles, and I find plenty of opportunity for that. An especially enjoyable track is adapting algorithms to be massively parallel so they run efficiently on GPUs (especially compute). Sometimes that leads to friction; my projects often have a “rocket science” nature to them, making them hard to contribute to.</p>
<p>Aside from puzzle-solving, which is largely a solitary activity, I also like the aspects of teaching, getting people up to speed, especially in topics that are not easily accessible through a standard computer science education or textbooks. Writing, including this blog, is a big part of that.</p>
<p>To a large extent, I will try to make time in 2023 to focus on these things I really enjoy, that I find make me happy. I am extremely fortunate that my paying job, as a research software engineer on Google Fonts, lets me do that.</p>
<h2 id="vello">Vello</h2>
<p>Of the various projects in flight (and there are many!), one is clearly rising to the top of the stack: Vello, the GPU-accelerated 2D renderer. We made a lot of progress last year, but it’s still not quite ready to ship. Part of it is that it’s trying to solve a very hard problem, and part of it is that some solid GPU infrastructure needs to be in place for it to fly. A lot of time and energy last year went into GPU infrastructure in various ways, both pushing piet-gpu-hal forward, and then doing a full rewrite into WGSL and WebGPU (<a href="https://github.com/raphlinus/raphlinus.github.io/issues/86">writeup to come</a>).</p>
<p>For those who might be confused by the name, Vello is a rename of piet-gpu but still fundamentally the same project. The old name didn’t really fit, as “Piet” refers to a trait/method abstraction for the traditional 2D graphics API, and we’re moving away from that. The new name is intended to evoke both vellum (parchment, as in books and manuscripts) and velocity.</p>
<p>It is clear to me that there is strong demand for a good, cross-platform 2D rendering engine. There are a few other interesting things going on the space, but I’ll be honest, what I see, I <em>want</em> to compete with. There are big missing pieces (the most important by far is the ability to import images), but I see a pretty clear path to getting all that done.</p>
<p>So this is by far the biggest goal for the year: get Vello to a usable state. That involves shipping a crate, doing a proper writeup, which will probably be a 20-30 page self-standing report, and doing quantitative performance evaluation against other renderers.</p>
<h2 id="xilem">Xilem</h2>
<p>Another major initiative is Xilem. This was originally just a reactive layer, intended to be generic over the underlying widget tree, but is emerging as an umbrella for the larger project.</p>
<p>My hypothesis is (still) that Xilem is the best known reactive architecture for Rust. If I am correct, that means it is more concise, more ergonomic, more efficient, and better integrated with async than comparable work on Dioxus, Sycamore, pax-lang, and the Elm variants. That would be an exciting result, in which case I hope and expect some interesting systems will be built on the architecture. If I am wrong (which is entirely possible), then those other reactive approaches are all likely good enough for practical use, and have more evidence behind them than Xilem currently does. I feel strongly that it is worthwhile to test the hypothesis, and the only way to do that is to try to build real systems on the architecture. That experience will also likely drive improvements to the Xilem architecture, and I expect we’ll learn something interesting in any case. [Note: this paragraph is a rewrite of the original; see <a href="https://github.com/raphlinus/raphlinus.github.io/pull/88">raphlinus#88</a> for the diff]</p>
<p>Progress on Xilem will have a considerably different texture than Vello. I hope that a huge fraction of the actual implementation work will be done by the community. Most of that work is most decidedly <em>not</em> rocket science, as I expect lots of it to be straightforward adaptation of the existing Druid widget set into the new architecture (and I’m making certain decisions explicitly to make that easier). I’m trying hard not to <a href="https://devblogs.microsoft.com/oldnewthing/20091201-00/?p=15843">lick the cookie</a> too much, and encourage other people to take on subprojects. I’m also trying to foster a culture where everyone in the community feels empowered to review PR’s and keep things moving, as having that block on me has been difficult.</p>
<p>One thing I am looking forward to working on is immutable data structures for efficiently (sparsely) diffing collections, which will hopefully realize the promise of my <a href="https://www.youtube.com/watch?v=DSuX-LIAU-I">RustLab 2020 talk</a>. This is a problem I feel has never been fully solved in UI toolkits - either you use complex and fragile mechanisms to incrementally update the UI, or you end up diffing the whole collection every time - and I believe using solid computer science to solve it, plus good Rust API design, would be very satisfying.</p>
<p>I consider Xilem to be more speculative and riskier than Vello. The extent to which it succeeds is largely based on how well the community can organize around getting the work done; if it gated on me, it’s a good question when it would all get done, especially with other projects competing for my attention.</p>
<p>The best place to learn more about Xilem is my <a href="https://www.youtube.com/watch?v=zVUTZlNCb8U">High Performance Rust UI</a> talk. I go over the goals and motivations of the project, and there’s good Q&A at the end. There’s a bit more about the Rust language aspects, particuarly trying to do ergonomic API design, in my <a href="https://youtu.be/Phk0C-kLlho">RustLab 2022 keynote</a>. In any case, I will be writing more.</p>
<p>Obviously Xilem depends on having good 2D rendering, so clearly time spent getting Vello production-ready contributes toward the overall success of the project.</p>
<p>To the people who want a good GUI toolkit they can use right now: I apologize, and ask for patience. In the mean time, you might check out Iced, Egui, or Slint. Those are all pretty good right now, and continually improving. I personally find the end goal of Xilem to be extremely compelling, but even in the best case it will take a while to get there.</p>
<h2 id="curves-and-other-research">Curves and other research</h2>
<p>Over the holiday break, I let my mind roam more freely than usual, and I found myself coming back to various problems in curves. Probably the most satisfying work was refining the Bézier curve fitting ideas (see <a href="https://github.com/linebender/kurbo/pull/230">kurbo#230</a> for the code, hopefully writeup coming before too long). I also have what I consider a very promising idea to <a href="https://github.com/linebender/spline/issues/26">improve hyperbeziers</a>, which have been on the back burner for a couple years, and an idea for <a href="https://github.com/raphlinus/raphlinus.github.io/issues/79">robust boolean path operations</a>.</p>
<p>But probably the juiciest bit of work will be perfecting the path geometry parts in Vello. In particular, I have a fairly compelling prototype of a combined flatten + offset operation in terms of <a href="https://raphlinus.github.io/curves/2021/02/19/parallel-curves.html">Euler spirals</a>, which I feel can be implemented efficiently on GPU as a compute shader and integrated into the Vello pipeline. That will improve handling of strokes (to handle all the join/cap options), and also serve as the basis for stem darkening of font outlines. Fun fact: the <a href="http://raphlinus.github.io/curves/2022/09/09/parallel-beziers.html">parallel curve</a> work I did a few months ago was motivated by trying to get a good GPU implementation, but ultimately I believe that particular work is best suited for creative vector graphics applications, while the Euler spiral approach is better suited for GPU.</p>
<h2 id="non-goals">Non-goals</h2>
<p>A year ago, I said only half as a joke, that my main New Years resolution for 2022 would be to not write a shader language. I’m pleased to report that I have succeeded. I will renew that vow for next year as well.</p>
<p>In many ways, it would make sense to start designing a shader language. My process for writing compute shaders is basically to design them in a fictional high level language with operations like prefix sum, stream compaction, etc., then translate by hand into the much lower level WGSL (formerly GLSL). If a good high level language existed (Futhark is the closest of what I’ve seen so far, but rust-gpu is also a contender), it would in theory streamline that work and let me write more efficiently.</p>
<p>But I think the advice in <a href="https://blog.dhsdevelopments.com/dont-write-a-programming-language">Don’t write a programming language</a> is pretty sound. A programming language is a multi-year endeavor, and with a poor chance of success even in the most favorable conditions. In addition, I’ve found that everything in GPU land is at least 5 times harder than it is on the CPU. Case in point, I had trouble even getting Futhark to run on the GPU hardware of my main development machines, as it doesn’t have any compute shader back-ends, only OpenCL (which is basically dead) and CUDA (not much help on AMD or Apple M1 hardware). And that’s a relatively successful effort with over 8 years of experience!</p>
<p>I am also unlikely to write papers for academic journals and conferences. Perhaps it’s sour grapes, but I haven’t had a good experience, and feel that the (nontrivial) time and effort doesn’t really pay off. I’ll continue to use this blog, and publication of open source software, as the primary way I communicate research results. That said, I had one good experience of being asked to collaborate on an ASPLOS paper, and I remain open to collaborations. For someone with an incentive to publish academically, I think there’s quite a bit of raw material to work with.</p>
<h2 id="conclusion">Conclusion</h2>
<p>I really feel that a lot of the research effort of the past year and beyond is poised to pay off in the next. Most especially, I am excited about shipping Vello, as I think that will advance the state of 2D rendering and also points the way for doing other interesting things in GPU compute. I also really look forward to working deeply with the community forming around Xilem to push that forward and hopefully make it reality.</p>
<p>There’s lots more that could be said, especially about the many individual subprojects, but in this reflection I hoped to give a general overview.</p>
<p>Discussion on <a href="https://www.reddit.com/r/rust/comments/zzw2wr/raphs_reflections_and_wishes_for_2023/">/r/rust</a>.</p>
<p>In any case, I wish everybody a happy and healthy 2023.</p>This post reflects a bit on 2022 and contains wishes for 2023. It mixes project and technical stuff, which I write about fairly frequently, and more personal reflections, which I don’t.Minikin retrospective2022-11-08T18:40:42+00:002022-11-08T18:40:42+00:00https://raphlinus.github.io/text/2022/11/08/minikin<p>There’s a lot of new interest in open source text layout projects, including <a href="https://github.com/pop-os/cosmic-text">cosmic-text</a>, and the <a href="https://github.com/dfrg/parley">parley</a> work being done by Chad Brokaw. I’m hopeful we’ll get a good Rust solution soon.</p>
<p>I encourage people working on text to study existing open source code bases, as much of the knowledge is arcane, and there is unfortunately no good textbook on the subject. Obviously browser engines facilitate extremely sophisticated layout, but because of their complexity and the constraints of HTML and CSS compatibility, they may be hard going. The APIs of <a href="https://learn.microsoft.com/en-us/windows/win32/directwrite/direct-write-portal">DirectWrite</a> and <a href="https://developer.apple.com/documentation/coretext">Core Text</a> are also worthy of study, but unfortunately their implementations remain closed.</p>
<p>An interesting codebase is <a href="https://android.googlesource.com/platform/frameworks/minikin/">Minikin</a>. It powers text layout in the Android framework. I wrote the original version starting back around 2013, but it is since maintained by the very capable Android text team.</p>
<p>A bit of additional context. The design of Minikin was constrained by compatibility with the existing Android text API (mostly exposed through Java, though these days there is a nontrivial <a href="https://developer.android.com/ndk/reference/group/font">NDK surface</a> as well). There is a higher level layer, responsible for rich text representation, editing, and other things, while Minikin is the lower level that powers that and does shaping, itemization, hit testing, and so on. (See <a href="https://raphlinus.github.io/text/2020/10/26/text-layout.html">Text layout is a loose hierarchy of segmentation</a> if these terms are unfamiliar). For the most part, the interface between the higher and lower levels is <em>runs</em> of text. Line breaking is an interesting case where the responsibility for crossing levels is shared across the layer boundary. For the most part, the higher level iterates its own rich text representation and hands runs to Minikin.</p>
<p>A special feature of Minikin is its optimized line breaking algorithm, strongly inspired by Knuth-Plass. This was motivated largely by the need to handle small screens better, but has some tweaks to make it even better for mobile. The heuristics try not to place a word by itself on the last line, and in general try to balance line lengths. Line breaking generally follows ICU rules, but does special case email addresses and URLs, as those are very common on mobile and the ICU rules work poorly.</p>
<p>There’s also fun stuff in there for grapheme boundaries, deciding between color and text emoji presentation forms, and calculating exactly what should be deleted on backspace (the logic for that is surprisingly complicated and as far as I know has not been written down anywhere).</p>
<p>Much of the internal structure of Minikin is dedicated to itemization, which is done largely on the basis of the cmap coverage of the fonts in the fallback chain. Doing that query font-by-font for every character would be expensive, so there are fancy bitmap acceleration structures. A good, general, cross-platform way to do itemization and fallback is a hard problem, but I think this solution works well for Android’s specific needs.</p>
<p>While there’s unfortunately no excellent documentation on Minkin’s internals, there are some resources, and part of the purpose of this blog is to point people to them. I’ve just gotten permission to publish the <a href="/assets/Project_Minikin.pdf">Project Minikin slide deck</a> from 2013 (PDF), which explains the motivation and some of the early design ideas. There’s also an <a href="https://lwn.net/Articles/662569/">lwn article</a> which goes into some detail, largely based on my 2015 ATypI talk (<a href="https://www.youtube.com/watch?v=L8LD0BM-Vjk">video</a>, <a href="https://docs.google.com/presentation/d/1-b1loWe23QNk0ydrmEBG31iVABYWSpE28JmdXDHs73E/edit">slides</a>).</p>
<p>Flutter’s <a href="https://github.com/flutter/flutter/issues/35994">LibTxt</a> was originally based on Minikin, but no doubt has diverged considerably since then, as they have fairly different requirements and of course are not bound by compatibility with Android.</p>
<p>But if you have deep interest in this, I recommend studying the code, as that’s the ultimate source of truth. It’s extremely fortunate that Android’s open source development model gives us access to it!</p>
<p>If you have questions, let me know (an issue on this repo is a good way, or you can ask on Mastodon), and I’ll do my best to answer.</p>There’s a lot of new interest in open source text layout projects, including cosmic-text, and the parley work being done by Chad Brokaw. I’m hopeful we’ll get a good Rust solution soon.Parallel curves of cubic Béziers2022-09-09T17:45:42+00:002022-09-09T17:45:42+00:00https://raphlinus.github.io/curves/2022/09/09/parallel-beziers<!-- I should figure out a cleaner way to do this include, rather than cutting and pasting. Ah well.-->
<script type="text/x-mathjax-config">
MathJax.Hub.Config({
tex2jax: {
inlineMath: [['$', '$']]
}
});
</script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js?config=TeX-AMS-MML_HTMLorMML" type="text/javascript"></script>
<style>
svg {
touch-action: pinch-zoom;
overflow: visible;
}
svg .handle {
pointer-events: all;
}
svg .handle:hover {
r: 6;
}
svg .quad {
stroke-width: 1.5px;
stroke: #222;
}
svg .hull {
stroke: #a6c;
}
svg .approx_handle {
stroke: #444;
}
svg .polyline {
}
svg text {
font-family: sans-serif;
}
svg .button {
fill: #aad;
stroke: #44f;
}
svg .button:hover {
fill: #bbf;
}
svg text {
pointer-events: none;
}
svg #grid line {
stroke: #e4e4e4;
}
svg .band {
fill: #fda;
opacity: 0.3;
}
img {
margin: auto;
margin: auto;
display: block;
}
input#d {
width: 300px;
}
input#tol {
width: 4em;
}
input#alg {
width: 4em;
}
.controls {
display: grid;
grid-template-columns: repeat(3, max-content);
column-gap: 20px;
margin-bottom: 15px;
}
</style>
<svg id="s" width="700" height="500">
<g id="grid"></g>
</svg>
<div class="controls">
<div>Distance</div>
<div>Accuracy</div>
<div>Method</div>
<div><input type="range" min="1" max="100" value="40" id="d" /></div>
<div><input type="button" id="tol" value="1" /></div>
<div><input type="button" id="alg" value="Fit" /></div>
</div>
<script>
// Copyright 2022 Google LLC
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
// https://www.apache.org/licenses/LICENSE-2.0
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
const svgNS = "http://www.w3.org/2000/svg";
class Point {
constructor(x, y) {
this.x = x;
this.y = y;
}
lerp(p2, t) {
return new Point(this.x + (p2.x - this.x) * t, this.y + (p2.y - this.y) * t);
}
dist(p2) {
return Math.hypot(p2.x - this.x, p2.y - this.y);
}
hypot2() {
return this.x * this.x + this.y * this.y;
}
hypot() {
return Math.sqrt(this.hypot2());
}
dot(other) {
return this.x * other.x + this.y * other.y;
}
cross(other) {
return this.x * other.y - this.y * other.x;
}
plus(other) {
return new Point(this.x + other.x, this.y + other.y);
}
minus(other) {
return new Point(this.x - other.x, this.y - other.y);
}
atan2() {
return Math.atan2(this.y, this.x);
}
}
class Affine {
constructor(c) {
this.c = c;
}
apply_pt(p) {
const c = this.c;
const x = c[0] * p.x + c[2] * p.y + c[4];
const y = c[1] * p.x + c[3] * p.y + c[5];
return new Point(x, y);
}
apply_cubic(cu) {
const c = this.c;
const new_c = new Float64Array(8);
for (let i = 0; i < 8; i += 2) {
new_c[i] = c[0] * cu.c[i] + c[2] * cu.c[i + 1] + c[4];
new_c[i + 1] = c[1] * cu.c[i] + c[3] * cu.c[i + 1] + c[5];
}
return new CubicBez(new_c);
}
static rotate(th) {
const c = new Float64Array(6);
c[0] = Math.cos(th);
c[1] = Math.sin(th);
c[2] = -c[1];
c[3] = c[0];
return new Affine(c);
}
}
// Compute an approximation to int (1 + 4x^2) ^ -0.25 dx
// This isn't especially good but will do.
function approx_myint(x) {
const d = 0.67;
return x / (1 - d + Math.pow(Math.pow(d, 4) + 0.25 * x * x, 0.25));
}
// Approximate the inverse of the function above.
// This is better.
function approx_inv_myint(x) {
const b = 0.39;
return x * (1 - b + Math.sqrt(b * b + 0.25 * x * x));
}
class QuadBez {
constructor(x0, y0, x1, y1, x2, y2) {
this.x0 = x0;
this.y0 = y0;
this.x1 = x1;
this.y1 = y1;
this.x2 = x2;
this.y2 = y2;
}
to_svg_path() {
return `M${this.x0} ${this.y0} Q${this.x1} ${this.y1} ${this.x2} ${this.y2}`
}
eval(t) {
const mt = 1 - t;
const x = this.x0 * mt * mt + 2 * this.x1 * t * mt + this.x2 * t * t;
const y = this.y0 * mt * mt + 2 * this.y1 * t * mt + this.y2 * t * t;
return new Point(x, y);
}
eval_deriv(t) {
const mt = 1 - t;
const x = 2 * (mt * (this.x1 - this.x0) + t * (this.x2 - this.x1));
const y = 2 * (mt * (this.y1 - this.y0) + t * (this.y2 - this.y1));
return new Point(x, y);
}
weightsum(c0, c1, c2) {
const x = c0 * this.x0 + c1 * this.x1 + c2 * this.x2;
const y = c0 * this.y0 + c1 * this.y1 + c2 * this.y2;
return new Point(x, y);
}
subsegment(t0, t1) {
const p0 = this.eval(t0);
const p2 = this.eval(t1);
const dt = t1 - t0;
const p1x = p0.x + (this.x1 - this.x0 + t0 * (this.x2 - 2 * this.x1 + this.x0)) * dt;
const p1y = p0.y + (this.y1 - this.y0 + t0 * (this.y2 - 2 * this.y1 + this.y0)) * dt;
return new QuadBez(p0.x, p0.y, p1x, p1y, p2.x, p2.y);
}
}
const GAUSS_LEGENDRE_COEFFS_8 = [
0.3626837833783620, -0.1834346424956498,
0.3626837833783620, 0.1834346424956498,
0.3137066458778873, -0.5255324099163290,
0.3137066458778873, 0.5255324099163290,
0.2223810344533745, -0.7966664774136267,
0.2223810344533745, 0.7966664774136267,
0.1012285362903763, -0.9602898564975363,
0.1012285362903763, 0.9602898564975363,
];
const GAUSS_LEGENDRE_COEFFS_8_HALF = [
0.3626837833783620, 0.1834346424956498,
0.3137066458778873, 0.5255324099163290,
0.2223810344533745, 0.7966664774136267,
0.1012285362903763, 0.9602898564975363,
];
const GAUSS_LEGENDRE_COEFFS_16_HALF = [
0.1894506104550685, 0.0950125098376374,
0.1826034150449236, 0.2816035507792589,
0.1691565193950025, 0.4580167776572274,
0.1495959888165767, 0.6178762444026438,
0.1246289712555339, 0.7554044083550030,
0.0951585116824928, 0.8656312023878318,
0.0622535239386479, 0.9445750230732326,
0.0271524594117541, 0.9894009349916499,
];
const GAUSS_LEGENDRE_COEFFS_24_HALF = [
0.1279381953467522, 0.0640568928626056,
0.1258374563468283, 0.1911188674736163,
0.1216704729278034, 0.3150426796961634,
0.1155056680537256, 0.4337935076260451,
0.1074442701159656, 0.5454214713888396,
0.0976186521041139, 0.6480936519369755,
0.0861901615319533, 0.7401241915785544,
0.0733464814110803, 0.8200019859739029,
0.0592985849154368, 0.8864155270044011,
0.0442774388174198, 0.9382745520027328,
0.0285313886289337, 0.9747285559713095,
0.0123412297999872, 0.9951872199970213,
];
const GAUSS_LEGENDRE_COEFFS_32 = [
0.0965400885147278, -0.0483076656877383,
0.0965400885147278, 0.0483076656877383,
0.0956387200792749, -0.1444719615827965,
0.0956387200792749, 0.1444719615827965,
0.0938443990808046, -0.2392873622521371,
0.0938443990808046, 0.2392873622521371,
0.0911738786957639, -0.3318686022821277,
0.0911738786957639, 0.3318686022821277,
0.0876520930044038, -0.4213512761306353,
0.0876520930044038, 0.4213512761306353,
0.0833119242269467, -0.5068999089322294,
0.0833119242269467, 0.5068999089322294,
0.0781938957870703, -0.5877157572407623,
0.0781938957870703, 0.5877157572407623,
0.0723457941088485, -0.6630442669302152,
0.0723457941088485, 0.6630442669302152,
0.0658222227763618, -0.7321821187402897,
0.0658222227763618, 0.7321821187402897,
0.0586840934785355, -0.7944837959679424,
0.0586840934785355, 0.7944837959679424,
0.0509980592623762, -0.8493676137325700,
0.0509980592623762, 0.8493676137325700,
0.0428358980222267, -0.8963211557660521,
0.0428358980222267, 0.8963211557660521,
0.0342738629130214, -0.9349060759377397,
0.0342738629130214, 0.9349060759377397,
0.0253920653092621, -0.9647622555875064,
0.0253920653092621, 0.9647622555875064,
0.0162743947309057, -0.9856115115452684,
0.0162743947309057, 0.9856115115452684,
0.0070186100094701, -0.9972638618494816,
0.0070186100094701, 0.9972638618494816,
];
const GAUSS_LEGENDRE_COEFFS_32_HALF = [
0.0965400885147278, 0.0483076656877383,
0.0956387200792749, 0.1444719615827965,
0.0938443990808046, 0.2392873622521371,
0.0911738786957639, 0.3318686022821277,
0.0876520930044038, 0.4213512761306353,
0.0833119242269467, 0.5068999089322294,
0.0781938957870703, 0.5877157572407623,
0.0723457941088485, 0.6630442669302152,
0.0658222227763618, 0.7321821187402897,
0.0586840934785355, 0.7944837959679424,
0.0509980592623762, 0.8493676137325700,
0.0428358980222267, 0.8963211557660521,
0.0342738629130214, 0.9349060759377397,
0.0253920653092621, 0.9647622555875064,
0.0162743947309057, 0.9856115115452684,
0.0070186100094701, 0.9972638618494816,
];
function tri_sign(x0, y0, x1, y1) {
return x1 * (y0 - y1) - y1 * (x0 - x1);
}
// Return distance squared
function line_nearest_origin(x0, y0, x1, y1) {
const dx = x1 - x0;
const dy = y1 - y0;
let dotp = -dx * x0 - dy * y0;
let d_sq = dx * dx + dy * dy;
if (dotp <= 0) {
return x0 * x0 + y0 * y0;
} else if (dotp >= d_sq) {
return x1 * x1 + y1 * y1;
} else {
const t = dotp / d_sq;
const x = x0 + t * (x1 - x0);
const y = y0 + t * (y1 - y0);
return x * x + y * y;
}
}
class CubicBez {
/// Argument is array of coordinate values [x0, y0, x1, y1, x2, y2, x3, y3].
constructor(coords) {
this.c = coords;
}
static from_pts(p0, p1, p2, p3) {
const c = new Float64Array(8);
c[0] = p0.x;
c[1] = p0.y;
c[2] = p1.x;
c[3] = p1.y;
c[4] = p2.x;
c[5] = p2.y;
c[6] = p3.x;
c[7] = p3.y;
return new CubicBez(c);
}
p0() {
return new Point(this.c[0], this.c[1]);
}
p1() {
return new Point(this.c[2], this.c[3]);
}
p2() {
return new Point(this.c[4], this.c[5]);
}
p3() {
return new Point(this.c[6], this.c[7]);
}
to_svg_path() {
const c = this.c;
return `M${c[0]} ${c[1]}C${c[2]} ${c[3]} ${c[4]} ${c[5]} ${c[6]} ${c[7]}`
}
weightsum(c0, c1, c2, c3) {
const x = c0 * this.c[0] + c1 * this.c[2] + c2 * this.c[4] + c3 * this.c[6];
const y = c0 * this.c[1] + c1 * this.c[3] + c2 * this.c[5] + c3 * this.c[7];
return new Point(x, y);
}
eval(t) {
const mt = 1 - t;
const c0 = mt * mt * mt;
const c1 = 3 * mt * mt * t;
const c2 = 3 * mt * t * t;
const c3 = t * t * t;
return this.weightsum(c0, c1, c2, c3);
}
eval_deriv(t) {
const mt = 1 - t;
const c0 = -3 * mt * mt;
const c3 = 3 * t * t;
const c1 = -6 * t * mt - c0;
const c2 = 6 * t * mt - c3;
return this.weightsum(c0, c1, c2, c3);
}
// quadratic bezier with matching endpoints and minimum max vector error
midpoint_quadbez() {
const p1 = this.weightsum(-0.25, 0.75, 0.75, -0.25);
return new QuadBez(this.c[0], this.c[1], p1.x, p1.y, this.c[6], this.c[7]);
}
subsegment(t0, t1) {
let c = new Float64Array(8);
const p0 = this.eval(t0);
const p3 = this.eval(t1);
c[0] = p0.x;
c[1] = p0.y;
const scale = (t1 - t0) / 3;
const d1 = this.eval_deriv(t0);
c[2] = p0.x + scale * d1.x;
c[3] = p0.y + scale * d1.y;
const d2 = this.eval_deriv(t1);
c[4] = p3.x - scale * d2.x;
c[5] = p3.y - scale * d2.y;
c[6] = p3.x;
c[7] = p3.y;
return new CubicBez(c);
}
area() {
const c = this.c;
return (c[0] * (6 * c[3] + 3 * c[5] + c[7])
+ 3 * (c[2] * (-2 * c[1] + c[5] + c[7]) - c[4] * (c[1] + c[3] - 2 * c[7]))
- c[6] * (c[1] + 3 * c[3] + 6 * c[5]))
* 0.05;
}
chord() {
return new Line([this.c[0], this.c[1], this.c[6], this.c[7]]);
}
deriv() {
const c = this.c;
return new QuadBez(
3 * (c[2] - c[0]), 3 * (c[3] - c[1]),
3 * (c[4] - c[2]), 3 * (c[5] - c[3]),
3 * (c[6] - c[4]), 3 * (c[7] - c[5])
);
}
// A pretty good algorithm; kurbo does more sophisticated error analysis.
arclen(accuracy) {
return this.arclen_rec(accuracy, 0);
}
arclen_rec(accuracy, depth) {
const c = this.c;
const d03x = c[6] - c[0];
const d03y = c[7] - c[1];
const d01x = c[2] - c[0];
const d01y = c[3] - c[1];
const d12x = c[4] - c[2];
const d12y = c[5] - c[3];
const d23x = c[6] - c[4];
const d23y = c[7] - c[5];
const lp_lc = Math.hypot(d01x, d01y) + Math.hypot(d12x, d12y)
+ Math.hypot(d23x, d23y) - Math.hypot(d03x, d03y);
const dd1x = d12x - d01x;
const dd1y = d12y - d01y;
const dd2x = d23x - d12x;
const dd2y = d23y - d12y;
const dmx = 0.25 * (d01x + d23x) + 0.5 * d12x;
const dmy = 0.25 * (d01y + d23y) + 0.5 * d12y;
const dm1x = 0.5 * (dd2x + dd1x);
const dm1y = 0.5 * (dd2y + dd1y);
const dm2x = 0.25 * (dd2x - dd1x);
const dm2y = 0.25 * (dd2y - dd1y);
const co_e = GAUSS_LEGENDRE_COEFFS_8;
let est = 0;
for (let i = 0; i < co_e.length; i += 2) {
const xi = co_e[i + 1];
const dx = dmx + dm1x * xi + dm2x * (xi * xi);
const dy = dmy + dm1y * xi + dm2y * (xi * xi);
const ddx = dm1x + dm2x * (2 * xi);
const ddy = dm1y + dm2y * (2 * xi);
est += co_e[i] * (ddx * ddx + ddy * ddy) / (dx * dx + dy * dy);
}
const est3 = est * est * est;
const est_gauss8_err = Math.min(est3 * 2.5e-6, 3e-2) * lp_lc;
let co = null;
if (Math.min(est3 * 2.5e-6, 3e-2) * lp_lc < accuracy) {
co = GAUSS_LEGENDRE_COEFFS_8_HALF;
} else if (Math.min(est3 * est3 * 1.5e-11, 9e-3) * lp_lc < accuracy) {
co = GAUSS_LEGENDRE_COEFFS_16_HALF;
} else if (Math.min(est3 * est3 * est3 * 3.5e-16, 3.5e-3) * lp_lc < accuracy
|| depth >= 20)
{
co = GAUSS_LEGENDRE_COEFFS_24_HALF;
} else {
const c0 = this.subsegment(0, 0.5);
const c1 = this.subsegment(0.5, 1);
return c0.arclen_rec(accuracy * 0.5, depth + 1)
+ c1.arclen_rec(accuracy * 0.5, depth + 1);
}
let sum = 0;
for (let i = 0; i < co.length; i += 2) {
const xi = co[i + 1];
const wi = co[i];
const dx = dmx + dm2x * (xi * xi);
const dy = dmy + dm2y * (xi * xi);
const dp = Math.hypot(dx + dm1x * xi, dy + dm1y * xi);
const dm = Math.hypot(dx - dm1x * xi, dy - dm1y * xi);
sum += wi * (dp + dm);
}
return 1.5 * sum;
}
inv_arclen(s, accuracy) {
if (s <= 0) {
return 0;
}
const total_arclen = this.arclen(accuracy);
if (s >= total_arclen) {
return 1;
}
// For simplicity, measure arclen from 0 rather than stateful delta.
const f = t => this.subsegment(0, t).arclen(accuracy) - s;
const epsilon = accuracy / total_arclen;
return solve_itp(f, 0, 1, epsilon, 1, 2, -s, total_arclen -s);
}
find_offset_cusps(d) {
const q = this.deriv();
// x'' cross x' is a quadratic polynomial in t
const d0x = q.x0;
const d0y = q.y0;
const d1x = 2 * (q.x1 - q.x0);
const d1y = 2 * (q.y1 - q.y0);
const d2x = q.x0 - 2 * q.x1 + q.x2;
const d2y = q.y0 - 2 * q.y1 + q.y2;
const c0 = d1x * d0y - d1y * d0x;
const c1 = 2 * (d2x * d0y - d2y * d0x);
const c2 = d2x * d1y - d2y * d1x;
const cusps = new CuspAccumulator(d, q, c0, c1, c2);
this.find_offset_cusps_rec(d, cusps, 0, 1, c0, c1, c2);
return cusps.reap();
}
find_offset_cusps_rec(d, cusps, t0, t1, c0, c1, c2) {
cusps.report(t0);
const dt = t1 - t0;
const q = this.subsegment(t0, t1).deriv();
// compute interval for ds/dt, using convex hull of hodograph
const d1 = tri_sign(q.x0, q.y0, q.x1, q.y1);
const d2 = tri_sign(q.x1, q.y1, q.x2, q.y2);
const d3 = tri_sign(q.x2, q.y2, q.x0, q.y0);
const z = !((d1 < 0 || d2 < 0 || d3 < 0) && (d1 > 0 || d2 > 0 || d3 > 0));
const ds0 = q.x0 * q.x0 + q.y0 * q.y0;
const ds1 = q.x1 * q.x1 + q.y1 * q.y1;
const ds2 = q.x2 * q.x2 + q.y2 * q.y2;
const max_ds = Math.sqrt(Math.max(ds0, ds1, ds2)) / dt;
const m1 = line_nearest_origin(q.x0, q.y0, q.x1, q.y1);
const m2 = line_nearest_origin(q.x1, q.y1, q.x2, q.y2);
const m3 = line_nearest_origin(q.x2, q.y2, q.x0, q.y0);
const min_ds = z ? 0 : Math.sqrt(Math.min(m1, m2, m3)) / dt;
//console.log('ds interval', min_ds, max_ds, 'iv', t0, t1);
let cmin = Math.min(c0, c0 + c1 + c2);
let cmax = Math.max(c0, c0 + c1 + c2);
const t_crit = -0.5 * c1 / c2;
const c_at_t = (c2 * t_crit + c1) * t_crit + c0;
if (t_crit > 0 && t_crit < 1) {
let c_at_t = (c2 * t_crit + c1) * t_crit + c0;
cmin = Math.min(cmin, c_at_t);
cmax = Math.max(cmax, c_at_t);
}
const min3 = min_ds * min_ds * min_ds;
const max3 = max_ds * max_ds * max_ds;
// TODO: signs are wrong, want min/max of c * d
// But this is a suitable starting place for clipping.
if (cmin * d > -min3 || cmax * d < -max3) {
//return;
}
const rmax = solve_quadratic(c0 * d + max3, c1 * d, c2 * d);
const rmin = solve_quadratic(c0 * d + min3, c1 * d, c2 * d);
let ts;
// TODO: length = 1 cases. Also maybe reduce case explosion?
if (rmax.length == 2 && rmin.length == 2) {
if (c2 > 0) {
ts = [rmin[0], rmax[0], rmax[1], rmin[1]];
} else {
ts = [rmax[0], rmin[0], rmin[1], rmax[1]];
}
} else if (rmin.length == 2) {
if (c2 > 0) {
ts = rmin;
} else {
ts = [t0, rmin[0], rmin[1], t1];
}
} else if (rmax.length == 2) {
if (c2 > 0) {
ts = [t0, rmax[0], rmax[1], t1];
} else {
ts = rmax;
}
} else {
const c_at_t0 = (c2 * t0 + c1) * t0 + c0;
if (c_at_t0 * d < -min3 && c_at_t0 * d > -max3) {
ts = [t0, t1];
} else {
ts = [];
}
}
for (let i = 0; i < ts.length; i += 2) {
const new_t0 = Math.max(t0, ts[i]);
const new_t1 = Math.min(t1, ts[i + 1]);
if (new_t1 > new_t0) {
if (new_t1 - new_t0 < 1e-9) {
cusps.report(new_t0);
cusps.report(new_t1);
} else if (new_t1 - new_t0 > 0.5 * dt) {
const tm = 0.5 * (new_t0 + new_t1);
this.find_offset_cusps_rec(d, cusps, new_t0, tm, c0, c1, c2);
this.find_offset_cusps_rec(d, cusps, tm, new_t1, c0, c1, c2);
} else {
this.find_offset_cusps_rec(d, cusps, new_t0, new_t1, c0, c1, c2);
}
//console.log('iv', new_t0, new_t1);
}
}
cusps.report(t1);
//console.log(rmax);
//console.log(rmin);
//console.log('ts:', ts);
}
/*
// This is a brute-force solution; a more robust one is started above.
// Output is a partition of (0..1) into ranges, with signs.
find_offset_cusps(d) {
const result = [];
const n = 100;
const q = this.deriv();
// x'' cross x' is a quadratic polynomial in t
const d0x = q.x0;
const d0y = q.y0;
const d1x = 2 * (q.x1 - q.x0);
const d1y = 2 * (q.y1 - q.y0);
const d2x = q.x0 - 2 * q.x1 + q.x2;
const d2y = q.y0 - 2 * q.y1 + q.y2;
const c0 = d1x * d0y - d1y * d0x;
const c1 = 2 * (d2x * d0y - d2y * d0x);
const c2 = d2x * d1y - d2y * d1x;
let ya;
let last_t;
let t0 = 0;
for (let i = 0; i <= n; i++) {
const t = i / n;
const ds2 = q.eval(t).hypot2();
const k = (((c2 * t + c1) * t) + c0) / (ds2 * Math.sqrt(ds2));
const yb = k * d + 1;
if (i != 0) {
if (ya >= 0 != yb >= 0) {
let tx = (yb * last_t - ya * t) / (yb - ya);
const iv = {'t0': t0, 't1': tx, 'sign': Math.sign(ya)};
result.push(iv);
t0 = tx;
}
}
ya = yb;
last_t = t;
}
const last_iv = {'t0': t0, 't1': 1, 'sign': Math.sign(ya)};
result.push(last_iv);
return result;
}
*/
// Find intersections of ray from point p with tangent d
intersect_ray(p, d) {
const c = this.c
const px0 = c[0];
const px1 = 3 * c[2] - 3 * c[0];
const px2 = 3 * c[4] - 6 * c[2] + 3 * c[0];
const px3 = c[6] - 3 * c[4] + 3 * c[2] - c[0];
const py0 = c[1];
const py1 = 3 * c[3] - 3 * c[1];
const py2 = 3 * c[5] - 6 * c[3] + 3 * c[1];
const py3 = c[7] - 3 * c[5] + 3 * c[3] - c[1];
const c0 = d.y * (px0 - p.x) - d.x * (py0 - p.y);
const c1 = d.y * px1 - d.x * py1;
const c2 = d.y * px2 - d.x * py2;
const c3 = d.y * px3 - d.x * py3;
return solve_cubic(c0, c1, c2, c3).filter(t => t > 0 && t < 1);
}
}
class CuspAccumulator {
constructor(d, q, c0, c1, c2) {
this.d = d;
this.q = q;
this.c0 = c0;
this.c1 = c1;
this.c2 = c2;
this.t0 = 0;
this.last_t = 0;
this.last_y = this.eval(0);
this.result = [];
}
eval(t) {
const ds2 = this.q.eval(t).hypot2();
const k = (((this.c2 * t + this.c1) * t) + this.c0) / (ds2 * Math.sqrt(ds2));
return k * this.d + 1;
}
report(t) {
const yb = this.eval(t);
const ya = this.last_y;
if (ya >= 0 != yb >= 0) {
// More wired: use ITP
let tx = (yb * this.last_t - ya * t) / (yb - ya);
const iv = {'t0': this.t0, 't1': tx, 'sign': Math.sign(ya)};
this.result.push(iv);
this.t0 = tx;
}
this.last_t = t;
this.last_y = yb;
}
reap() {
const last_iv = {'t0': this.t0, 't1': 1, 'sign': Math.sign(this.last_y)};
this.result.push(last_iv);
return this.result;
}
}
class Line {
/// Argument is array of coordinate values [x0, y0, x1, y1].
constructor(coords) {
this.c = coords;
}
area() {
return (this.c[0] * this.c[3] - this.c[1] * this.c[2]) * 0.5;
}
}
function copysign(x, y) {
const a = Math.abs(x);
return y < 0 ? -a : a;
}
function solve_quadratic(c0, c1, c2) {
const sc0 = c0 / c2;
const sc1 = c1 / c2;
if (!(isFinite(sc0) && isFinite(sc1))) {
const root = -c0 / c1;
if (isFinite(root)) {
return [root];
} else if (c0 == 0 && c1 == 0) {
return [0];
} else {
return [];
}
}
const arg = sc1 * sc1 - 4 * sc0;
let root1 = 0;
if (isFinite(arg)) {
if (arg < 0) {
return [];
} else if (arg == 0) {
return [-0.5 * sc1];
}
root1 = -.5 * (sc1 + copysign(Math.sqrt(arg), sc1));
} else {
root1 = -sc1;
}
const root2 = sc0 / root1;
if (isFinite(root2)) {
if (root2 > root1) {
return [root1, root2];
} else {
return [root2, root1];
}
}
return [root1];
}
// See kurbo common.rs
function solve_cubic(in_c0, in_c1, in_c2, in_c3) {
const c2 = in_c2 / (3 * in_c3);
const c1 = in_c1 / (3 * in_c3);
const c0 = in_c0 / in_c3;
if (!(isFinite(c0) && isFinite(c1) && isFinite(c2))) {
return solve_quadratic(in_c0, in_c1, in_c2);
}
const d0 = -c2 * c2 + c1;
const d1 = -c1 * c2 + c0;
const d2 = c2 * c0 - c1 * c1;
const d = 4 * d0 * d2 - d1 * d1;
const de = -2 * c2 * d0 + d1;
if (d < 0) {
const sq = Math.sqrt(-0.25 * d);
const r = -0.5 * de;
const t1 = Math.cbrt(r + sq) + Math.cbrt(r - sq);
return [t1 - c2];
} else if (d == 0) {
const t1 = copysign(Math.sqrt(-d0), de);
return [t1 - c2, -2 * t1 - c2];
} else {
const th = Math.atan2(Math.sqrt(d), -de) / 3;
const r0 = Math.cos(th);
const ss3 = Math.sin(th) * Math.sqrt(3);
const r1 = 0.5 * (-r0 + ss3);
const r2 = 0.5 * (-r0 - ss3);
const t = 2 * Math.sqrt(-d0);
return [t * r0 - c2, t * r1 - c2, t * r2 - c2];
}
}
// Factor a quartic polynomial into two quadratics. Based on Orellana and De Michele
// and very similar to the version in kurbo.
function solve_quartic(c0, c1, c2, c3, c4) {
// This doesn't special-case c0 = 0.
if (c4 == 0) {
return solve_cubic(c0, c1, c2, c3);
}
const a = c3 / c4;
const b = c2 / c4;
const c = c1 / c4;
const d = c0 / c4;
let result = solve_quartic_inner(a, b, c, d, false);
if (result !== null) {
return result;
}
const K_Q = 7.16e76;
for (let i = 0; i < 2; i++) {
result = solve_quartic_inner(a / K_Q, b / (K_Q * K_Q), c / (K_Q * K_Q * K_Q),
d / (K_Q * K_Q * K_Q * K_Q), i != 0);
if (result !== null) {
for (let j = 0; j < result.length; j++) {
result[j] *= K_Q;
}
return result;
}
}
// Really bad overflow happened.
return [];
}
function eps_rel(raw, a) {
return a == 0 ? Math.abs(raw) : Math.abs((raw - a) / a);
}
function solve_quartic_inner(a, b, c, d, rescale) {
let result = factor_quartic_inner(a, b, c, d, rescale);
if (result !== null && result.length == 4) {
let roots = [];
for (let i = 0; i < 2; i++) {
const a = result[i * 2];
const b = result[i * 2 + 1];
roots = roots.concat(solve_quadratic(b, a, 1));
}
return roots;
}
}
function factor_quartic_inner(a, b, c, d, rescale) {
function calc_eps_q(a1, b1, a2, b2) {
const eps_a = eps_rel(a1 + a2, a);
const eps_b = eps_rel(b1 + a1 * a2 + b2, b);
const eps_c = eps_rel(b1 * a2 + a1 * b2, c);
return eps_a + eps_b + eps_c;
}
function calc_eps_t(a1, b1, a2, b2) {
return calc_eps_q(a1, b1, a2, b2) + eps_rel(b1 * b2, d);
}
const disc = 9 * a * a - 24 * b;
const s = disc >= 0 ? -2 * b / (3 * a + copysign(Math.sqrt(disc), a)) : -0.25 * a;
const a_prime = a + 4 * s;
const b_prime = b + 3 * s * (a + 2 * s);
const c_prime = c + s * (2 * b + s * (3 * a + 4 * s));
const d_prime = d + s * (c + s * (b + s * (a + s)));
let g_prime = 0;
let h_prime = 0;
const K_C = 3.49e102;
if (rescale) {
const a_prime_s = a_prime / K_C;
const b_prime_s = b_prime / K_C;
const c_prime_s = c_prime / K_C;
const d_prime_s = d_prime / K_C;
g_prime = a_prime_s * c_prime_s - (4 / K_C) * d_prime_s - (1. / 3) * b_prime_s * b_prime_s;
h_prime = (a_prime_s * c_prime_s - (8 / K_C) * d_prime_s - (2. / 9) * b_prime_s * b_prime_s)
* (1. / 3) * b_prime_s
- c_prime_s * (c_prime_s / K_C)
- a_prime_s * a_prime_s * d_prime_s;
} else {
g_prime = a_prime * c_prime - 4 * d_prime - (1. / 3) * b_prime * b_prime;
h_prime = (a_prime * c_prime + 8 * d_prime - (2. / 9) * b_prime * b_prime) * (1. / 3) * b_prime
- c_prime * c_prime
- a_prime * a_prime * d_prime;
}
if (!isFinite(g_prime) && isFinite(h_prime)) {
return null;
}
let phi = depressed_cubic_dominant(g_prime, h_prime);
if (rescale) {
phi *= K_C;
}
const l_1 = a * 0.5;
const l_3 = (1. / 6) * b + 0.5 * phi;
const delt_2 = c - a * l_3;
const d_2_cand_1 = (2. / 3) * b - phi - l_1 * l_1;
const l_2_cand_1 = 0.5 * delt_2 / d_2_cand_1;
const l_2_cand_2 = 2 * (d - l_3 * l_3) / delt_2;
const d_2_cand_2 = 0.5 * delt_2 / l_2_cand_2;
let d_2_best = 0;
let l_2_best = 0;
for (let i = 0; i < 3; i++) {
const d_2 = i == 1 ? d_2_cand_2 : d_2_cand_1;
const l_2 = i == 0 ? l_2_cand_1 : l_2_cand_2;
const eps_0 = eps_rel(d_2 + l_1 * l_1 + 2 * l_3, b);
const eps_1 = eps_rel(2 * (d_2 * l_2 + l_1 * l_3), c);
const eps_2 = eps_rel(d_2 * l_2 * l_2 + l_3 * l_3, d);
const eps_l = eps_0 + eps_1 + eps_2;
if (i == 0 || eps_l < eps_l_best) {
d_2_best = d_2;
l_2_best = l_2;
eps_l_best = eps_l;
}
}
const d_2 = d_2_best;
const l_2 = l_2_best;
let alpha_1 = 0;
let beta_1 = 0;
let alpha_2 = 0;
let beta_2 = 0;
if (d_2 < 0.0) {
const sq = Math.sqrt(-d_2);
alpha_1 = l_1 + sq;
beta_1 = l_3 + sq * l_2;
alpha_2 = l_1 - sq;
beta_2 = l_3 - sq * l_2;
if (Math.abs(beta_2) < Math.abs(beta_1)) {
beta_2 = d / beta_1;
} else if (Math.abs(beta_2) > Math.abs(beta_1)) {
beta_1 = d / beta_2;
}
if (Math.abs(alpha_1) != Math.abs(alpha_2)) {
let a1_cands = null;
let a2_cands = null;
if (Math.abs(alpha_1) < Math.abs(alpha_2)) {
const a1_cand_1 = (c - beta_1 * alpha_2) / beta_2;
const a1_cand_2 = (b - beta_2 - beta_1) / alpha_2;
const a1_cand_3 = a - alpha_2;
a1_cands = [a1_cand_3, a1_cand_1, a1_cand_2];
a2_cands = [alpha_2, alpha_2, alpha_2];
} else {
const a2_cand_1 = (c - alpha_1 * beta_2) / beta_1;
const a2_cand_2 = (b - beta_2 - beta_1) / alpha_1;
const a2_cand_3 = a - alpha_1;
a1_cands = [alpha_1, alpha_1, alpha_1];
a2_cands = [a2_cand_3, a2_cand_1, a2_cand_2];
}
let eps_q_best = 0;
for (let i = 0; i < 3; i++) {
const a1 = a1_cands[i];
const a2 = a2_cands[i];
if (isFinite(a1) && isFinite(a2)) {
const eps_q = calc_eps_q(a1, beta_1, a2, beta_2);
if (i == 0 || eps_q < eps_q_best) {
alpha_1 = a1;
alpha_2 = a2;
eps_q_best = eps_q;
}
}
}
}
} else if (d_2 == 0) {
const d_3 = d - l_3 * l_3;
alpha_1 = l_1;
beta_1 = l_3 + Math.sqrt(-d_3);
alpha_2 = l_1;
beta_2 = l_3 - Math.sqrt(-d_3);
if (Math.abs(beta_1) > Math.abs(beta_2)) {
beta_2 = d / beta_1;
} else if (Math.abs(beta_2) > Math.abs(beta_1)) {
beta_1 = d / beta_2;
}
} else {
// No real solutions
return [];
}
let eps_t = calc_eps_t(alpha_1, beta_1, alpha_2, beta_2);
for (let i = 0; i < 8; i++) {
if (eps_t == 0) {
break;
}
const f_0 = beta_1 * beta_2 - d;
const f_1 = beta_1 * alpha_2 + alpha_1 * beta_2 - c;
const f_2 = beta_1 + alpha_1 * alpha_2 + beta_2 - b;
const f_3 = alpha_1 + alpha_2 - a;
const c_1 = alpha_1 - alpha_2;
const det_j = beta_1 * beta_1 - beta_1 * (alpha_2 * c_1 + 2 * beta_2)
+ beta_2 * (alpha_1 * c_1 + beta_2);
if (det_j == 0) {
break;
}
const inv = 1 / det_j;
const c_2 = beta_2 - beta_1;
const c_3 = beta_1 * alpha_2 - alpha_1 * beta_2;
const dz_0 = c_1 * f_0 + c_2 * f_1 + c_3 * f_2 - (beta_1 * c_2 + alpha_1 * c_3) * f_3;
const dz_1 = (alpha_1 * c_1 + c_2) * f_0
- beta_1 * (c_1 * f_1 + c_2 * f_2 + c_3 * f_3);
const dz_2 = -c_1 * f_0 - c_2 * f_1 - c_3 * f_2 + (alpha_2 * c_3 + beta_2 * c_2) * f_3;
const dz_3 = -(alpha_2 * c_1 + c_2) * f_0
+ beta_2 * (c_1 * f_1 + c_2 * f_2 + c_3 * f_3);
const a1 = alpha_1 - inv * dz_0;
const b1 = beta_1 - inv * dz_1;
const a2 = alpha_2 - inv * dz_2;
const b2 = beta_2 - inv * dz_3;
const new_eps_t = calc_eps_t(a1, b1, a2, b2);
if (new_eps_t < eps_t) {
alpha_1 = a1;
beta_1 = b1;
alpha_2 = a2;
beta_2 = b2;
eps_t = new_eps_t;
} else {
break;
}
}
return [alpha_1, beta_1, alpha_2, beta_2];
}
function depressed_cubic_dominant(g, h) {
const q = (-1. / 3) * g;
const r = 0.5 * h;
let phi_0;
let k = null;
if (Math.abs(q) >= 1e102 || Math.abs(r) >= 1e164) {
if (Math.abs(q) < Math.abs(r)) {
k = 1 - q * (q / r) * (q / r);
} else {
k = Math.sign(q) * ((r / q) * (r / q) / q - 1);
}
}
if (k !== null && r == 0) {
if (g > 0) {
phi_0 = 0;
} else {
phi_0 = Math.sqrt(-g);
}
} else if (k !== null ? k < 0 : r * r < q * q * q) {
const t = k !== null ? r / q / Math.sqrt(q) : r / Math.sqrt(q * q * q);
phi_0 = -2 * Math.sqrt(q) * copysign(Math.cos(Math.acos(Math.abs(t)) * (1. / 3)), t);
} else {
let a;
if (k !== null) {
if (Math.abs(q) < Math.abs(r)) {
a = -r * (1 + Math.sqrt(k));
} else {
a = -r - copysign(Math.sqrt(Math.abs(q)) * q * Math.sqrt(k), r);
}
} else {
a = Math.cbrt(-r - copysign(Math.sqrt(r * r - q * q * q), r));
}
const b = a == 0 ? 0 : q / a;
phi_0 = a + b;
}
let x = phi_0;
let f = (x * x + g) * x + h;
const EPS_M = 2.22045e-16;
if (Math.abs(f) < EPS_M * Math.max(x * x * x, g * x, h)) {
return x;
}
for (let i = 0; i < 8; i++) {
const delt_f = 3 * x * x + g;
if (delt_f == 0) {
break;
}
const new_x = x - f / delt_f;
const new_f = (new_x * new_x + g) * new_x + h;
if (new_f == 0) {
return new_x;
}
if (Math.abs(new_f) >= Math.abs(f)) {
break;
}
x = new_x;
f = new_f;
}
return x;
}
// For testing.
function vieta(x1, x2, x3, x4) {
const a = -(x1 + x2 + x3 + x4);
const b = x1 * (x2 + x3) + x2 * (x3 + x4) + x4 * (x1 + x3);
const c = -x1 * x2 * (x3 + x4) - x3 * x4 * (x1 + x2);
const d = x1 * x2 * x3 * x4;
const roots = solve_quartic(d, c, b, a, 1);
return roots;
}
// See common.rs in kurbo
function solve_itp(f, a, b, epsilon, n0, k1, ya, yb) {
const n1_2 = Math.max(Math.ceil(Math.log2((b - a) / epsilon)) - 1, 0);
const nmax = n0 + n1_2;
let scaled_epsilon = epsilon * Math.exp(nmax * Math.LN2);
while (b - a > 2 * epsilon) {
const x1_2 = 0.5 * (a + b);
const r = scaled_epsilon - 0.5 * (b - a);
const xf = (yb * a - ya * b) / (yb - ya);
const sigma = x1_2 - xf;
const delta = k1 * (b - a) * (b - a);
const xt = delta <= Math.abs(x1_2 - xf) ? xf + copysign(delta, sigma) : x1_2;
const xitp = Math.abs(xt - x1_2) <= r ? xt : x1_2 - copysign(r, sigma);
const yitp = f(xitp);
if (yitp > 0) {
b = xitp;
yb = yitp;
} else if (yitp < 0) {
a = xitp;
ya = yitp;
} else {
return xitp;
}
scaled_epsilon *= 0.5
}
return 0.5 * (a + b);
}
function ray_intersect(p0, d0, p1, d1) {
const det = d0.x * d1.y - d0.y * d1.x;
const t = (d0.x * (p0.y - p1.y) - d0.y * (p0.x - p1.x)) / det;
return new Point(p1.x + d1.x * t, p1.y + d1.y * t);
}
class CubicOffset {
constructor(c, d) {
this.c = c;
this.q = c.deriv();
this.d = d;
}
eval_offset(t) {
const dp = this.q.eval(t);
const s = this.d / dp.hypot();
return new Point(-s * dp.y, s * dp.x);
}
eval(t) {
return this.c.eval(t).plus(this.eval_offset(t));
}
eval_deriv(t) {
const dp = this.q.eval(t);
const ddp = this.q.eval_deriv(t);
const h = dp.hypot2();
const turn = ddp.cross(dp) * this.d / (h * Math.sqrt(h));
const s = 1 + turn;
return new Point(s * dp.x, s * dp.y);
}
// Compute area and x moment
calc() {
let arclen = 0;
let area = 0;
let moment_x = 0;
const co = GAUSS_LEGENDRE_COEFFS_32;
for (let i = 0; i < co.length; i += 2) {
const t = 0.5 * (1 + co[i + 1]);
const wi = co[i];
const dp = this.eval_deriv(t);
const p = this.eval(t);
const d_area = wi * dp.x * p.y;
arclen += wi * dp.hypot();
area += d_area;
moment_x += p.x * d_area;
}
return {'arclen': 0.5 * arclen, 'area': 0.5 * area, 'mx': 0.5 * moment_x };
}
sample_pts(n) {
const result = [];
let arclen = 0;
// Probably overkill, but keep it simple
const co = GAUSS_LEGENDRE_COEFFS_32;
const dt = 1 / (n + 1);
for (let i = 0; i < n; i++) {
for (let j = 0; j < co.length; j += 2) {
const t = dt * (i + 0.5 + 0.5 * co[j + 1]);
arclen += co[j] * this.eval_deriv(t).hypot();
}
const t = dt * (i + 1);
const d = this.eval_offset(t);
const p = this.c.eval(t).plus(d);
result.push({'arclen': arclen * 0.5 * dt, 'p': p, 'd': d});
}
return result;
}
rotate_to_x() {
const p0 = this.c.p0().plus(this.eval_offset(0));
const p1 = this.c.p3().plus(this.eval_offset(1));
const th = p1.minus(p0).atan2();
const a = Affine.rotate(-th);
const ct = CubicBez.from_pts(
a.apply_pt(this.c.p0().minus(p0)),
a.apply_pt(this.c.p1().minus(p0)),
a.apply_pt(this.c.p2().minus(p0)),
a.apply_pt(this.c.p3().minus(p0))
);
const co = new CubicOffset(ct, this.d);
return {'c': co, 'th': th, 'p0': p0};
}
// Error evaluation logic from Tiller and Hanson.
est_cubic_err(cu, samples, tolerance) {
let err = 0;
let tol2 = tolerance * tolerance;
for (let sample of samples) {
let best_err = null;
// Project sample point onto approximate curve along normal.
let samples = cu.intersect_ray(sample.p, sample.d);
if (samples.length == 0) {
// In production, if no rays intersect we probably want
// to reject this candidate altogether. But we sample the
// endpoints so you can get a plausible number.
samples = [0, 1];
}
for (let t of samples) {
const p_proj = cu.eval(t);
const this_err = sample.p.minus(p_proj).hypot2();
if (best_err === null || this_err < best_err) {
best_err = this_err;
}
}
err = Math.max(err, best_err);
if (err > tol2) {
break;
}
}
return Math.sqrt(err);
}
cubic_approx(tolerance, sign) {
const r = this.rotate_to_x();
const end_x = r.c.c.c[6] + r.c.eval_offset(1).x;
const metrics = r.c.calc();
const arclen = metrics.arclen;
const th0 = Math.atan2(sign * r.c.q.y0, sign * r.c.q.x0);
const th1 = -Math.atan2(sign * r.c.q.y2, sign * r.c.q.x2);
const ex2 = end_x * end_x;
const ex3 = ex2 * end_x;
const cands = cubic_fit(th0, th1, metrics.area / ex2, metrics.mx / ex3);
const c = new Float64Array(6);
const cx = end_x * Math.cos(r.th);
const sx = end_x * Math.sin(r.th);
c[0] = cx;
c[1] = sx;
c[2] = -sx;
c[3] = cx;
c[4] = r.p0.x;
c[5] = r.p0.y;
const a = new Affine(c);
const samples = this.sample_pts(10);
let best_c = null;
let best_err;
let errs = [];
for (let raw_cand of cands) {
const cand = a.apply_cubic(raw_cand);
const err = this.est_cubic_err(cand, samples, tolerance);
errs.push(err);
if (best_c === null || err < best_err) {
best_err = err;
best_c = cand;
}
}
//console.log(errs);
if (best_c === null) {
return null;
}
return {'c': best_c, 'err': best_err};
}
cubic_approx_other(conf, sign) {
let c;
if (conf.method == 'T-H') {
c = this.tiller_hanson();
} else if (conf.method == 'Shape') {
c = this.shape_control();
}
if (c === null) {
return null;
}
const samples = this.sample_pts(10);
const err = this.est_cubic_err(c, samples, conf.tolerance);
return {'c': c, 'err': err};
}
cubic_approx_seq(conf, sign) {
let approx;
if (conf.method == 'Fit') {
approx = this.cubic_approx(conf.tolerance, sign);
} else {
approx = this.cubic_approx_other(conf, sign);
}
if (approx !== null && approx.err <= conf.tolerance) {
return [approx.c];
} else {
const co0 = this.subsegment(0, 0.5);
const co1 = this.subsegment(0.5, 1);
const seq0 = co0.cubic_approx_seq(conf, sign);
const seq1 = co1.cubic_approx_seq(conf, sign);
return seq0.concat(seq1);
}
}
subsegment(t0, t1) {
const cu = this.c.subsegment(t0, t1);
return new CubicOffset(cu, this.d);
}
tiller_hanson() {
const q = this.c.deriv();
const d0 = this.eval_offset(0);
const d1 = this.eval_offset(1);
const p0 = this.c.p0().plus(d0);
const p3 = this.c.p3().plus(d1);
const c_p1 = this.c.p1();
const c_p2 = this.c.p2();
const d12 = c_p2.minus(c_p1);
const s = this.d / d12.hypot();
const pm = new Point(c_p1.x - s * d12.y, c_p1.y + s * d12.x);
const pm2 = new Point(c_p2.x - s * d12.y, c_p2.y + s * d12.x);
const p1 = ray_intersect(p0, q.eval(0), pm, d12);
const p2 = ray_intersect(p3, q.eval(1), pm, d12);
return CubicBez.from_pts(p0, p1, p2, p3);
}
shape_control() {
const c = this.c.c;
const q = this.c.deriv();
const p0 = this.c.p0().plus(this.eval_offset(0));
const p3 = this.c.p3().plus(this.eval_offset(1));
const p = this.eval(0.5);
const a11 = c[2] - c[0];
const a12 = c[4] - c[6];
const a21 = c[3] - c[1];
const a22 = c[5] - c[7];
const b1 = (8. / 3) * (p.x - 0.5 * (p0.x + p3.x));
const b2 = (8. / 3) * (p.y - 0.5 * (p0.y + p3.y));
const det = a11 * a22 - a12 * a21;
if (det == 0) {
return null;
}
const a = (b1 * a22 - a12 * b2) / det;
const b = (a11 * b2 - b1 * a21) / det;
const p1 = new Point(p0.x + a * a11, p0.y + a * a21);
const p2 = new Point(p3.x + b * a12, p3.y + b * a22);
return CubicBez.from_pts(p0, p1, p2, p3);
}
}
function cubic_seq_to_svg(cu_seq) {
const c0 = cu_seq[0].c;
let str = `M${c0[0]} ${c0[1]}`;
for (cu of cu_seq) {
const ci = cu.c;
str += `C${ci[2]} ${ci[3]} ${ci[4]} ${ci[5]} ${ci[6]} ${ci[7]}`;
}
return str;
}
function cubic_seq_to_svg_handles(cu_seq) {
let str = '';
for (cu of cu_seq) {
const ci = cu.c;
str += `M${ci[0]} ${ci[1]}L${ci[2]} ${ci[3]}M${ci[4]} ${ci[5]}L${ci[6]} ${ci[7]}`;
}
return str;
}
/// Returns an array of candidate cubics matching given metrics.
function cubic_fit(th0, th1, area, mx) {
//console.log(th0, th1, area, mx);
const c0 = Math.cos(th0);
const s0 = Math.sin(th0);
const c1 = Math.cos(th1);
const s1 = Math.sin(th1);
const a4 = -9
* c0
* (((2 * s1 * c1 * c0 + s0 * (2 * c1 * c1 - 1)) * c0 - 2 * s1 * c1) * c0
- c1 * c1 * s0);
const a3 = 12
* ((((c1 * (30 * area * c1 - s1) - 15 * area) * c0 + 2 * s0
- c1 * s0 * (c1 + 30 * area * s1))
* c0
+ c1 * (s1 - 15 * area * c1))
* c0
- s0 * c1 * c1);
const a2 = 12
* ((((70 * mx + 15 * area) * s1 * s1 + c1 * (9 * s1 - 70 * c1 * mx - 5 * c1 * area))
* c0
- 5 * s0 * s1 * (3 * s1 - 4 * c1 * (7 * mx + area)))
* c0
- c1 * (9 * s1 - 70 * c1 * mx - 5 * c1 * area));
const a1 = 16
* (((12 * s0 - 5 * c0 * (42 * mx - 17 * area)) * s1
- 70 * c1 * (3 * mx - area) * s0
- 75 * c0 * c1 * area * area)
* s1
- 75 * c1 * c1 * area * area * s0);
const a0 = 80 * s1 * (42 * s1 * mx - 25 * area * (s1 - c1 * area));
//console.log(a0, a1, a2, a3, a4);
let roots;
const EPS = 1e-12;
if (Math.abs(a4) > EPS) {
const a = a3 / a4;
const b = a2 / a4;
const c = a1 / a4;
const d = a0 / a4;
const quads = factor_quartic_inner(a, b, c, d, false);
/*
const solved = solve_quartic(a0, a1, a2, a3, a4);
for (let x of solved) {
const y = (((a4 * x + a3) * x + a2) * x + a1) * x + a0;
console.log(x, y);
}
*/
roots = [];
for (let i = 0; i < quads.length; i += 2) {
const c1 = quads[i];
const c0 = quads[i + 1];
const q_roots = solve_quadratic(c0, c1, 1);
if (q_roots.length > 0) {
roots = roots.concat(q_roots)
} else {
// Real part of pair of complex roots
roots.push(-0.5 * c1);
}
}
} else {
// Question: do we ever care about complex roots in these cases?
if (Math.abs(a3) > EPS) {
roots = solve_cubic(a0, a1, a2, a3)
} else {
roots = solve_quadratic(a0, a1, a2);
}
}
const s01 = s0 * c1 + s1 * c0;
//console.log(roots);
const cubics = [];
for (let d0 of roots) {
let d1 = (2 * d0 * s0 - area * (20 / 3.)) / (d0 * s01 - 2 * s1);
if (d0 < 0) {
d0 = 0;
d1 = s0 / s01;
} else if (d1 < 0) {
d0 = s1 / s01;
d1 = 0;
}
if (d0 >= 0 && d1 >= 0) {
const c = new Float64Array(8);
c[2] = d0 * c0;
c[3] = d0 * s0;
c[4] = 1 - d1 * c1;
c[5] = d1 * s1;
c[6] = 1;
cubics.push(new CubicBez(c));
}
}
return cubics;
}
// One manipulable cubic bezier
class CubicUi {
constructor(ui, pts) {
this.ui = ui
this.pts = pts;
this.curve = ui.make_stroke();
this.curve.classList.add("quad");
this.hull = ui.make_stroke();
this.hull.classList.add("hull");
this.handles = [];
for (let pt of pts) {
this.handles.push(ui.make_handle(pt));
}
}
onPointerDown(e) {
const pt = this.ui.getCoords(e);
const x = pt.x;
const y = pt.y;
for (let i = 0; i < this.pts.length; i++) {
if (Math.hypot(x - this.pts[i].x, y - this.pts[i].y) < 10) {
this.current_obj = i;
return true;
}
}
return false;
}
onPointerMove(e) {
const i = this.current_obj;
const pt = this.ui.getCoords(e);
this.pts[i] = pt;
this.handles[i].setAttribute("cx", pt.x);
this.handles[i].setAttribute("cy", pt.y);
}
getCubic() {
const p0 = this.pts[0];
const p1 = this.pts[1];
const p2 = this.pts[2];
const p3 = this.pts[3];
let c = new Float64Array(8);
c[0] = p0.x;
c[1] = p0.y;
c[2] = p1.x;
c[3] = p1.y;
c[4] = p2.x;
c[5] = p2.y;
c[6] = p3.x;
c[7] = p3.y;
return new CubicBez(c);
}
update() {
const cb = this.getCubic();
const pts = this.pts;
this.curve.setAttribute("d", cb.to_svg_path());
const h = `M${pts[0].x} ${pts[0].y}L${pts[1].x} ${pts[1].y}M${pts[2].x} ${pts[2].y}L${pts[3].x} ${pts[3].y}`;
this.hull.setAttribute("d", h);
}
}
class OffsetUi {
constructor(id) {
const n_cubics = 2;
this.root = document.getElementById(id);
this.root.addEventListener("pointerdown", e => {
this.root.setPointerCapture(e.pointerId);
this.onPointerDown(e);
e.preventDefault();
e.stopPropagation();
});
this.root.addEventListener("pointermove", e => {
this.onPointerMove(e);
e.preventDefault();
e.stopPropagation();
});
this.root.addEventListener("pointerup", e => {
this.root.releasePointerCapture(e.pointerId);
this.onPointerUp(e);
e.preventDefault();
e.stopPropagation();
});
document.getElementById('d').addEventListener('input', e => this.update());
document.getElementById('tol').addEventListener('click', e => this.click_tol());
document.getElementById('alg').addEventListener('click', e => this.click_alg());
window.addEventListener("keydown", e => this.onKeyDown(e));
const pts_foo = [new Point(67, 237), new Point(374, 471), new Point(321, 189), new Point(633, 65)];
this.cubic_foo = new CubicUi(this, pts_foo);
this.xs = [200, 600];
this.quad = this.make_stroke();
this.quad.classList.add("quad");
this.approx_offset = this.make_stroke();
this.approx_handles = this.make_stroke();
this.approx_handles.classList.add("approx_handle");
this.n_label = this.make_text(500, 55);
this.type_label = this.make_text(90, 55);
this.type_label.setAttribute("text-anchor", "middle");
this.thresh_label = this.make_text(210, 55);
this.pips = [];
this.method = 'Fit';
this.grid = 20;
this.tolerance = 1;
this.renderGrid(true);
this.update();
this.current_obj = null;
}
getCoords(e) {
const rect = this.root.getBoundingClientRect();
const x = e.clientX - rect.left;
const y = e.clientY - rect.top;
return new Point(x, y);
}
onPointerDown(e) {
const pt = this.getCoords(e);
const x = pt.x;
const y = pt.y;
if (this.cubic_foo.onPointerDown(e)) {
this.current_obj = 'cubic';
return;
}
}
onPointerMove(e) {
// Maybe use object oriented dispatch?
if (this.current_obj == 'cubic') {
this.cubic_foo.onPointerMove(e);
this.update();
}
const pt = this.getCoords(e);
}
onPointerUp(e) {
this.current_obj = null;
}
onKeyDown(e) {
if (e.key == 's') {
this.method = "sederberg";
this.update();
} else if (e.key == 'r') {
this.method = "recursive";
this.update();
} else if (e.key == 'a') {
this.method = "analytic";
this.update();
} else if (e.key == 'w') {
this.method = "wang";
this.update();
}
}
renderGrid(visible) {
let grid = document.getElementById("grid");
//this.ui.removeAllChildren(grid);
if (!visible) return;
let w = 700;
let h = 500;
for (let i = 0; i < w; i += this.grid) {
let line = document.createElementNS(svgNS, "line");
line.setAttribute("x1", i);
line.setAttribute("y1", 0);
line.setAttribute("x2", i);
line.setAttribute("y2", h);
grid.appendChild(line);
}
for (let i = 0; i < h; i += this.grid) {
let line = document.createElementNS(svgNS, "line");
line.setAttribute("x1", 0);
line.setAttribute("y1", i);
line.setAttribute("x2", w);
line.setAttribute("y2", i);
grid.appendChild(line);
}
}
make_handle(p) {
const circle = this.plot(p.x, p.y, "blue", 4);
circle.classList.add("handle");
return circle;
}
make_stroke() {
const path = document.createElementNS(svgNS, "path");
path.setAttribute("fill", "none");
path.setAttribute("stroke", "blue");
this.root.appendChild(path);
return path;
}
make_clip_path(id) {
const clip_path = document.createElementNS(svgNS, "clipPath");
clip_path.setAttribute("id", id)
const path = document.createElementNS(svgNS, "path");
this.root.appendChild(clip_path);
clip_path.appendChild(path);
return path;
}
make_text(x, y) {
const text = document.createElementNS(svgNS, "text");
text.setAttribute("x", x);
text.setAttribute("y", y);
this.root.appendChild(text);
return text;
}
plot(x, y, color = "black", r = 2) {
let circle = document.createElementNS(svgNS, "circle");
circle.setAttribute("cx", x);
circle.setAttribute("cy", y);
circle.setAttribute("r", r);
circle.setAttribute("fill", color)
this.root.appendChild(circle);
return circle;
}
click_tol() {
const vals = [1, 0.1, 0.01, 0.001, 1e9, 10];
let tol = 1;
for (let i = 0; i < vals.length - 1; i++) {
if (this.tolerance == vals[i]) {
tol = vals[i + 1];
break;
}
}
this.tolerance = tol;
document.getElementById('tol').value = tol == 1e9 ? '\u221e' : `${tol}`;
this.update();
}
click_alg() {
let alg = 'Fit';
if (this.method == 'Fit') {
alg = 'T-H';
} else if (this.method == 'T-H') {
alg = 'Shape';
}
this.method = alg;
document.getElementById('alg').value = alg;
this.update();
}
update() {
for (let pip of this.pips) {
pip.remove();
}
this.pips = [];
const cb = this.cubic_foo.getCubic();
const conf = {
'd': document.getElementById('d').value,
'tolerance': this.tolerance,
'method': this.method,
};
const cusps = cb.find_offset_cusps(conf.d);
this.cubic_foo.update();
const c_off = new CubicOffset(cb, conf.d);
//console.log(c_off.sample_pts(10));
/*
const approx = c_off.cubic_approx();
const c = approx.c;
this.approx_offset.setAttribute('d', approx.c.to_svg_path());
this.type_label.textContent = `${approx.err}`;
const z = c.c;
const h = `M${z[0]} ${z[1]}L${z[2]} ${z[3]}M${z[6]} ${z[7]}L${z[4]} ${z[5]}`;
this.approx_handles.setAttribute('d', h);
*/
let seq = [];
for (let cusp of cusps) {
const co_seg = c_off.subsegment(cusp.t0, cusp.t1);
seq = seq.concat(co_seg.cubic_approx_seq(conf, cusp.sign));
}
this.approx_offset.setAttribute('d', cubic_seq_to_svg(seq));
this.approx_handles.setAttribute('d', cubic_seq_to_svg_handles(seq));
this.type_label.textContent = `subdivisions: ${seq.length}`;
}
}
new OffsetUi("s");
</script>
<p>The problem of <a href="https://en.wikipedia.org/wiki/Parallel_curve">parallel</a> or offset curves has remained challenging for a long time. Parallel curves have applications in 2D graphics (for drawing strokes and also adding weight to fonts), and also robotic path planning and manufacturing, among others. The exact offset curve of a cubic Bézier can be described (it is an analytic curve of degree 10) but it not tractable to work with. Thus, in practice the approach is almost always to compute an approximation to the true parallel curve. A single cubic Bézier might not be a good enough approximation to the parallel curve of the source cubic Bézier, so in those cases it is sudivided into multiple Bézier segments.</p>
<p>A number of algorithms have been published, of varying quality. Many popular algorithms aren’t very accurate, yielding either visually incorrect results or excessive subdivision, depending on how carefully the error metric has been implemented. This blogpost gives a practical implementation of a nearly optimal result. Essentially, it tries to find <em>the</em> cubic Bézier that’s closest to the desired curve. To this end, we take a curve-fitting approach and apply an array of numerical techniques to make it work. The result is a visibly more accurate curve even when only one Bézier is used, and a minimal number of subdivisions when a tighter tolerance is applied. In fact, we claim $O(n^6)$ scaling: if a curve is divided in half, the error of the approximation will decrease by a factor of 64. I suggested a previous approach, <a href="https://raphlinus.github.io/curves/2021/02/19/parallel-curves.html">Cleaner parallel curves with Euler spirals</a>, with $O(n^4)$ scaling, in other words only a 16-fold reduction of error.</p>
<p>Though there are quite a number of published algorithms, the need for a really good solution remains strong. Some really good reading is the <a href="https://github.com/paperjs/paper.js/issues/371">Paper.js issue</a> on adding an offset function. After much discussion and prototyping, there is still no consensus on the best approach, and the feature has not landed in Paper.js despite obvious demand. There’s also some interesting discussion of stroking in an <a href="https://github.com/googlefonts/colr-gradients-spec/issues/276">issue in the COLRv1 spec repo</a>.</p>
<h2 id="outline-of-approach">Outline of approach</h2>
<p>The fundamental concept is <em>curve fitting,</em> or finding the parameters for a cubic Bézier that most closely approximate the desired curve. We also employ a sequence of numerical techniques in support of that basic concept:</p>
<ul>
<li>Finding the cusps and subdividing the curve at the cusp points.</li>
<li>Computing area and moment of the target curve
<ul>
<li>Green’s theorem to convert double integral into a single integral</li>
<li>Gauss-Legendre quadrature for efficient numerical integration</li>
</ul>
</li>
<li>Quartic root finding to solve for cubic Béziers with desired area and moment</li>
<li>Measure error to choose best candidate and decide whether to subdivide
<ul>
<li>Cubic Bézier/ray intersection</li>
</ul>
</li>
</ul>
<p>Each of these numeric techniques has its own subtleties.</p>
<h2 id="cusp-finding">Cusp finding</h2>
<p>One of the challenges of parallel curves in general is <em>cusps.</em> These happen when the curvature of the source curve is equal to one over the offset distance. Cubic Béziers have fairly complex curvature profiles, so there can be a number of cusps - it’s easy to find examples with four, and it wouldn’t be surprising to me if there were more. By contrast, Euler spirals have simple curvature profiles, and the location of the cusp is extremely simple to determine.</p>
<p>The general equation for curvature of a parametric curve is as follows:</p>
\[\kappa = \frac{\mathbf{x}''(t) \times \mathbf{x}'(t)}{|\mathbf{x}'(t)|^3}\]
<p>The cusp happens when $\kappa d + 1 = 0$. With a bit of rewriting, we get</p>
\[(\mathbf{x}''(t) \times \mathbf{x}'(t))d + |\mathbf{x}'(t)|^3 = 0\]
<p>As with many such numerical root-finding approaches, missing a cusp is a risk. The approach <em>currently</em> used in the code in this blog post is a form of interval arithmetic: over the (t0..t1) interval, a minimum and maximum value of $|\mathbf{x}’|$ is computed, while the cross product is quadratic in t. Solving that partitions the interval into ranges where the curvature is definitely above or below the threshold for a cusp, and a (hopefully) smaller interval where it’s possible.</p>
<p>This algorithm is robust, but convergence is not super-fast - it often hits the case where it has to subdivide in half, so convergence is similar to a bisection approach for root-finding. I’m exploring another approach of computing bounding parabolas, and that seems to have cubic convergence, but is a bit more complicated and fiddly.</p>
<p>In cases where you <em>know</em> you have one simple cusp, a simple and generic root-finding method like ITP (about more which below) would be effective. But that leaves the problem of detecting when that’s the case. Robust detection of possible cusps generally also gives the locations of the cusps when iterated.</p>
<h2 id="computing-area-and-moment-of-the-target-curve">Computing area and moment of the target curve</h2>
<p>The primary input to the curve fitting algorithm is a set of parameters for the curve. Not control points of a Bézier, but other measurements of the curve. The position of the endpoints and the tangents can be determined directly, which, just counting parameters, leaves two free. Those are the area and x-moment. These are generally described as integrals. For an arbitrary parametric curve (a family which easily includes offsets of Béziers), Green’s theorem is a powerful and efficient technique for approximating these integrals.</p>
<p>For area, the specific instance of Green’s theorem we’re looking for is this. Let the curve be defined as x(t) and y(t), where t goes from 0 to 1. Let D be the region enclosed by the curve. If the curve is closed, then we have this relation:</p>
\[\iint_D dx \,dy = \int_0^1 y(t)\, x'(t)\, dt\]
<p>I won’t go into the details here, but all this still works even when the curve is open (one way to square up the accounting is to add the return path of the chord, from the end point back to the start), and when the area contains regions of both positive and negative signs, which can be the case for S-shaped curves. The x moment is also very similar and just involves an additional $x$ term:</p>
\[\iint_D x \, dx \,dy = \int_0^1 x(t)\, y(t)\, x'(t)\, dt\]
<p>Especially given that the function being integrated is (mostly) smooth, the best way to compute the approximate integral is <a href="https://en.wikipedia.org/wiki/Gauss%E2%80%93Legendre_quadrature">Gauss-Legendre quadrature</a>, which has an extremely simple implementation: it’s just the dot product between a vector of weights and a vector of the function sampled at certain points, where the weights and points are carefully chosen to minimize error; in particular they result in zero error when the function being integrated is a polynomial of order up to that of the number of samples. The JavaScript code on this page just uses a single quadrature of order 32, but a more sophisticated approach (as is used for arc length computation) would be to first estimate the error and then choose a number of samples based on that.</p>
<p>Note that the area and moments of a cubic Bézier curve can be efficiently computed analytically and don’t need an approximate numerical technique. Adding in the offset term is numerically similar to an arc length computation, bringing it out of the range where analytical techniques are effective, but fortunately similar numerical techniques as for computing arc length are effective.</p>
<h3 id="refinement-of-curve-fitting-approach">Refinement of curve fitting approach</h3>
<p>The basic approach to curve fitting was described in <a href="https://raphlinus.github.io/curves/2021/03/11/bezier-fitting.html">Fitting cubic Bézier curves</a>. Those ideas are good, but there were some rough edges to be filled in and other refinements.</p>
<p>To recap, the goal is to find the closest Bézier, in the space of all cubic Béziers, to the desired curve (in this case the parallel curve of a source Bézier, but the curve fitting approach is general). That’s a large space to search, but we can immediately nail down some of the parameters. The endpoints should definitely be fixed, and we’ll also set the tangent angles at the endpoints to match the desired curve.</p>
<p>One loose end was the solving technique. My prototype code used numerical methods, but I’ve now settled on root finding of the quartic equation. A major reason for that is that I’ve found that the quartic solver in the <a href="https://cristiano-de-michele.netlify.app/publication/orellana-2020/">Orellana and De Michele</a> paper works well - it is fast, robust, and stable. The JavaScript code on this page uses a fairly direct implementation of that (which I expect may be useful for other applications - all the code on this blog is licensed under Apache 2, so feel free to adapt it within the terms of that license).</p>
<p>Another loose end was the treatment of “near misses.” Those happen when the function comes close to zero but doesn’t quite cross it. In terms of roots of a polynomial, those are a conjugate pair of complex roots, and I take the real part of that as a candidate. It would certainly be possible to express this logic by having the quartic solver output complex roots as well as real ones, but I found an effective shortcut: the algorithm actually factors the original quartic equation into two quadratics, one of which always has real roots and the other some of the time, and finding the real part of the roots of a quadratic is trivial (it’s just -b/2a).</p>
<p>Recently, Cem Yuksel has proposed a variation of Newton-style <a href="http://www.cemyuksel.com/research/polynomials/">polynomial solving</a>. It’s likely this could be used, but there were a few reasons I went with the more analytic approach. For one, I want multiple roots and this works best when only one is desired. Second, it’s hard to bound a priori the interval to search for roots. Third, it’s not easy to get the complex roots (if you did want to do this, the best route is probably deflation). Lastly, the accuracy numbers don’t seem as good (the Orellana and De Michele paper presents the results of very careful testing), and in empirical testing I have found that accuracy in root finding is a real problem that can affect the quality of the final results. A Rust implementation of the Orellana and De Michele technique clocks in at 390ns on an M1 Max, which certainly makes it competitive with the fastest techniques out there.</p>
<p>The last loose end was the treatment of near-zero slightly negative arm lengths. These are roots of the polynomial but are not acceptable candidate curves, as the tangent would end up pointing the wrong way. My original thought was to clamp the relevant length to zero (on the basis that it is an acceptable curve that is “nearby” the numerical solution), but that also doesn’t give ideal results. In particular, if you set one length to zero and set the other one based on exact signed area, the tangent at the zero-length side might be wrong (enough to be visually objectionable). After some experimentation, I’ve decided to set the other control point to be the intersection of the tangents, which gets tangents right but possibly results in an error in area, depending on the exact parameters. The general approach is to throw these as candidates into the mix, and let the error measurement sort it out.</p>
<h3 id="error-measurement">Error measurement</h3>
<p>A significant amount of total time spent in the algorithm is measuring the distance between the exact curve and the cubic approximation, both to decide when to subdivide and also to choose between multiple candidates from the Bézier fitting. I implemented the technique from Tiller and Hanson and found it to work well. They sample the exact curve at a sequence of points, then for each of those points project that point onto the approximation along the normal. That is equivalent to computing the intersection of a ray and a cubic Bézier. The maximum distance between the projected and true point is the error. This is a fairly good approximation to the <a href="https://en.wikipedia.org/wiki/Fr%C3%A9chet_distance">Fréchet distance</a> but significantly cheaper to compute.</p>
<p>Computing the intersection of a ray and a cubic Bézier is equivalent to finding the root of a cubic polynomial, a challenging numerical problem in its own right. In the course of working on this, I found that the cubic solver in kurbo would sometimes report inaccurate results (especially when the coefficient on the $x^3$ term was near-zero, which can easily happen when cubic Bézier segments are near raised quadratics), and so implemented a <a href="https://github.com/linebender/kurbo/pull/224">better cubic solver</a> based on a blog post on <a href="https://momentsingraphics.de/CubicRoots.html">cubics by momentsingraphics</a>. That’s still not perfect, and there is more work to be done to arrive at a gold-plated cubic solver. The Yuksel <a href="http://www.cemyuksel.com/research/polynomials/">polynomial solving</a> approach might be a good fit for this, especially as you only care about results for t strictly within the (0..1) range. It might also be worth pointing out that the fma instruction used in the Rust implementation is not available in JavaScript, so the accuracy of the solver here won’t be quite as good.</p>
<p>The error metric is a critical component of a complete offset curve algorithm. It accounts for a good part of the total CPU time, and also must be accurate. If it underestimates true error, it risks letting inaccurate results slip through. If it overestimates error, it creates excessive subdivision. Incidentally, I suspect that the error measurement technique in the Elber, Lee and Kim paper (cited below) may be flawed; it seems like it may overestimate error in the case where the two curves being compared differ in parametrization, which will happen commonly with offset problems, particularly near cusps. The Tiller-Hanson technique is largely insensitive to parametrization (though perhaps more care should be taken to ensure that the sample points are actually evenly spaced).</p>
<h3 id="subdivision">Subdivision</h3>
<p>Right now the subdivision approach is quite simple: if none of the candidate cubic Béziers meet the error bound, then the curve is subdivided at t = 0.5 and each half is fit. The scaling is n^6, so in general that reduces the error by a factor of 64.</p>
<p>If generation of an absolute minimum number of output segments is the goal, then a smarter approach to choosing subdivisions would be in order. For absolutely optimal results, in general what you want to do is figure out the minimum number of subdivisions, then adjust the subdivision points so the error of all segments are equal. This technique is described in section 9.6.3 of my <a href="https://levien.com/phd/thesis.pdf">thesis</a>. In the limit, it can be expected to reduce the number of subdivisions by a factor of 1.5 compared with “subdivide in half,” but not a significant improvement when most curves can be rendered with one or two cubic segments.</p>
<h2 id="evaluation">Evaluation</h2>
<p>Somebody evaluating this work for use in production would care about several factors: accuracy of result, robustness, and performance. The interactive demo on this page speaks for itself: the results are accurate, the performance is quite good for interactive use, and it is robust (though I make no claims it handles all adversarial inputs correctly; that always tends to require extra work).</p>
<p>In terms of accuracy of result, this work is a dramatic advance over anything in the literature. I’ve implemented and compared it against two other techniques that are widely cited as reasonable approaches to this problem: <a href="https://math.stackexchange.com/questions/465782/control-points-of-offset-bezier-curve">Tiller-Hanson</a> and the “shape control” approach of Yang and Huang. For generating a single segment, it can be considerably more accurate than either.</p>
<p><img src="/assets/parallel-compare.png" width="870" alt="comparison against other approaches" /></p>
<p>In addition to the accuracy for generating a single line segment, it is interesting to compare the scaling as the number of subdivisions increases, or as the error tolerance decreases. These tend to follow a power law. For this technique, it is $O(n^6)$, meaning that subdividing a curve in half reduces the error by a factor of 64. For the shape control approach, it is $O(n^5)$, and for Tiller-Hanson is is $O(n^2)$. That last is a surprisingly poor result, suggesting that it is only a constant factor better than subdividing the curves into lines.</p>
<p><img src="/assets/parallel-scaling.svg" width="683" alt="chart showing scaling behavior" /></p>
<p>The shape control technique has good scaling, but stability issues when the tangents are nearly parallel. That can happen for an S-shaped curve, and also for a U with nearly 180 degrees of arc.</p>
<p>The Tiller-Hanson technique is geometrically intuitive; it offsets each edge of the control polygon by the offset amount, as illustrated in the diagram below. It doesn’t have the stability issues with nearly-parallel tangents and can produce better results for those “S” curves, but the scaling is much worse.</p>
<p><img src="/assets/parallel-tiller-hanson.png" width="560" height="250" alt="diagram showing Tiller-Hanson technique" /></p>
<p>Regarding performance, I have preliminary numbers from the JavaScript implementation, about 12µs per curve segment generated on an M1 Max running Chrome. I am quite happy with this result, and of course expect the Rust implementation to be even faster when it’s done. There are also significant downstream performance improvements from generating highly accurate results; every cubic segment you generate has some cost to process and render, so the fewer of those, the better.</p>
<p>I haven’t implemented all the techniques in the Elber, Lee and Kim paper, but it is possible to draw some tentative conclusions from the literature. I expect the Klass technique (and its numerical refinement by Sakai and Suenaga) to have good scaling but relatively poor acccuracy for a single segment. The Klass technique is also documented to have poor numerical stability, thanks in part to its reliance on Newton solving techniques. The Hoschek and related (least-squares) approaches will likely produce good results but are quite slow (the Yang and Huang paper reports an eye-popping 49s for calculating a simple case with .001 tolerance, of course on older hardware).</p>
<p>The Euler spiral technique in my previous blog post will in general produce considerably more subdivision (with $O(n^4)$ scaling), but perhaps it would be premature to write it off completely. Once the curve is in piecewise Euler spiral form, a result within the given error bounds can be computed directly, with no need to explicitly evaluate an error metric. In addition, the cusps are located robustly with trivial calculation. That said, getting a curve <em>into</em> piecewise Euler spiral form is still challenging, and my prototype code uses a rather expensive error metric to achieve that.</p>
<h2 id="discussion">Discussion</h2>
<p>This post presents a significantly better solution to the parallel curve problem than the current state of the art. It is accurate, robust, and fast. It should be suitable to implement in interactive vector graphics applications, font compilation pipelines, and other contexts.</p>
<p>While parallel curve is an important application, the curve fitting technique is quite general. It can be adapted to generalized strokes, for example where the stroke width is variable, path simplification, distortions and other transforms, conversion from other curve representations, accurate plotting of functions, and I’m sure there are other applications. Basically the main thing that’s required is the ability to evaluate area and moment of the source curve, and ability to evaluate the distance to that curve (which can be done readily enough by sampling a series of points with their arc lengths).</p>
<p>This work also provides a bit of insight into the nature of cubic Bézier curves. The $O(n^6)$ scaling provides quantitative support to the idea that cubic Bézier curves are extremely expressive; with skillful placement of the control points, they can extremely accurately approximate a wide variety of curves. Parallel curves are challenging for a variety of reasons, including cusps and sudden curvature variations. That said, they do require skill, as geometrically intuitive but unoptimized approaches to setting control points (such as Tiller-Hanson) perform poorly.</p>
<p>There’s clearly more work that could be done to make the evalation more rigorous, including more optimization of the code. I believe this result would make a good paper, but my bandwidth for writing papers is limited right now. I would be more than open to collaboration, and invite interested people to get in touch.</p>
<p>Thanks to Linus Romer for helpful discussion and refinement of the polynomial equations regarding quartic solving of the core curve fitting algorithm.</p>
<p>Discuss on <a href="https://news.ycombinator.com/item?id=32784491">Hacker News</a>.</p>
<h2 id="references">References</h2>
<p>Here is a bibliography of some relevant academic papers on the topic.</p>
<ul>
<li><a href="https://www.sciencedirect.com/science/article/abs/pii/0010448583900192">An offset spline approximation for plane cubic splines</a>, Klass, 1983</li>
<li><a href="https://ieeexplore.ieee.org/iel5/38/4055906/04055919">Offsets of Two-Dimensional Profiles</a>, Tiller and Hanson, 1984</li>
<li><a href="https://www.sciencedirect.com/science/article/abs/pii/0167839687900021">High Accuracy Geometric Hermite Interpolation</a>, de Boor, Höllig, Sabin, 1987 (<a href="https://minds.wisconsin.edu/bitstream/handle/1793/58822/TR692.pdf">PDF cache</a>)</li>
<li><a href="https://www.sciencedirect.com/science/article/abs/pii/0010448588900061">Optimal approximate conversion of spline curves and spline approximation of offset curves</a>, Hoschek and Wissel, 1988 (<a href="http://www.norbert-wissel.de/Diplom.pdf">PDF cache</a>)</li>
<li><a href="https://link.springer.com/chapter/10.1007/978-4-431-68456-5_17">A New Shape Control and Classification for Cubic Bézier Curves</a>, Yang and Huang, 1993 (<a href="https://github.com/paperjs/paper.js/files/752955/A.New.Shape.Control.and.Classification.for.Cubic.Bezier.Curves.pdf">PDF cache</a>)</li>
<li><a href="https://ieeexplore.ieee.org/document/586019/">Comparing offset curve approximation methods</a>, Elber, Lee, Kim, 1997 (<a href="http://3map.snu.ac.kr/mskim/ftp/comparing.pdf">PDF cache</a>)</li>
<li><a href="https://www.tandfonline.com/doi/abs/10.1080/00207160108805122">Cubic spline approximation of offset curves of planar cubic splines</a>, Sakai and Suenaga, 2001 (<a href="https://www.kurims.kyoto-u.ac.jp/~kyodo/kokyuroku/contents/pdf/1198-30.pdf">PDF cache</a>)</li>
<li><a href="https://dl.acm.org/doi/10.1145/3386241">Boosting Efficiency in Solving Quartic Equations with No Compromise in Accuracy</a>, Orellana and De Michele, 2020 (<a href="https://cristiano-de-michele.netlify.app/publication/orellana-2020/">PDF cache</a>)</li>
<li><a href="https://dl.acm.org/doi/10.1145/3543865">High-Performance Polynomial Root Finding for Graphics</a>, Yuksel, 2022 (<a href="http://www.cemyuksel.com/research/polynomials/polynomial_roots_hpg2022.pdf">PDF cache</a>)</li>
</ul>Advice for the next dozen Rust GUIs2022-07-15T17:53:42+00:002022-07-15T17:53:42+00:00https://raphlinus.github.io/rust/gui/2022/07/15/next-dozen-guis<p>A few times a week, someone asks on the <a href="https://discord.com/channels/273534239310479360/441714251359322144">#gui-and-ui channel</a> on the Rust Discord, “what is the best UI toolkit for my application?” Unfortunately there is still no clear answer to this question. Generally the top contenders are egui, Iced, and Druid, with <a href="https://slint-ui.com/">Slint</a> looking promising as well, but web-based approaches such as <a href="https://tauri.app/">Tauri</a> are also gaining some momentum, and of course there’s always the temptation to just build a new one. And every couple or months or so, a post appears with a new GUI toolkit.</p>
<p>This post is something of a sequel to <a href="https://raphlinus.github.io/rust/druid/2019/10/31/rust-2020.html">Rust 2020: GUI and community</a>. I hope to offer a clear-eyed survey of the current state of affairs, and suggestions for how to improve it. It also includes some lessons so far from Druid.</p>
<p>The motivations for building GUI in Rust remain strong. While Electron continues to gain momentum, especially for desktop use cases, there is considerable desire for a less resource-hungry alternative. That said, a consensus has not emerged what that alternative should look like. In addition, unfortunately there is fragmentation at the level of infrastructure as well.</p>
<p>Fragmentation is not entirely a bad thing. To some extent, specialization can be a good thing, resulting in solutions more adapted to the problem at hand, rather than a one-size-fits-all approach. More importantly, the diversity of UI approaches is a rich ground for experimentation and exploration, as there are many aspects to GUI in Rust where we still don’t know the best approach.</p>
<h2 id="a-large-tradeoff-space">A large tradeoff space</h2>
<p>One of the great truths to understand about GUI is that there are few obviously correct ways to do things, and many, many tradeoffs. At present, this tradeoff space is very <em>sensitive,</em> in that small differences in requirements and priorities may end up with considerably different implementation choices. I believe this is a main factor driving the fragmentation of Rust GUI. Many of the tradeoffs have to do with the extent to use platform capabilities for various things (of which text layout and compositing are perhaps the most intersting), as opposed to a cross-platform implementation of those capabilities, layered on top of platform abstractions.</p>
<h3 id="a-small-rant-about-native">A small rant about native</h3>
<p>You’ll see the word “native” used quite a bit in discussions about UI, but I think that’s more confusing than illuminating, for a number of reasons.</p>
<p>In the days of Windows XP, “native” on Windows would have had quite a specific and precise meaning, it would be the use of <a href="https://docs.microsoft.com/en-us/windows/win32/controls/window-controls">user32 controls</a>, which for the most part created a HWND per control and used GDI+ for drawing. Thanks to the strong compatibility guarantees of Windows, such code will still run, but it will look ancient and out of place compared to building on other technology stacks, also considered “native.” In the Windows 7 era, that would be <a href="https://devblogs.microsoft.com/oldnewthing/20050211-00/?p=36473">windowless controls</a> drawn with Direct2D (this was the technology used for Microsoft Office, using the internal DirectUI toolkit, which was not available to third party developers, though there were other windowless toolkits such as WPF). Starting with Windows 8 and the (relatively unsuccessful) <a href="https://en.wikipedia.org/wiki/Metro_(design_language)">Metro design language</a>, many of the elements of the UI came together in the compositor rather than being directly drawn by the app. As of Windows 10, Microsoft started pushing <a href="https://en.wikipedia.org/wiki/Universal_Windows_Platform">Universal Windows Platform</a> as the preferred “native” solution, but that also didn’t catch on. As of Windows 11, there is a new <a href="https://docs.microsoft.com/en-us/windows/apps/design/signature-experiences/design-principles">Fluent design language</a>, natively supported only in WinUI 3 (which in turn is an evolution of UWP). “Native” on Windows can refer to all of these technology stacks.</p>
<p>On Windows in particular, there’s also quite a bit of confusion about UI functionality that’s directly provided by the platform (as is the case for user32 controls and much of UWP, including the Windows.UI namespace), vs lower level capabilities (Direct2D, DirectWrite, and DirectComposition, for example) provided to a library which mostly lives in userspace. The new WinUI (and Windows App SDK more generally) is something of a hybrid, with this distinction largely hidden from the developer. An advantage of this hybrid approach is that OS version is abstracted to a large extent; for example, even though UWP proper is Windows 10 and up only, it is possible to use the <a href="https://platform.uno/">Uno platform</a> to deploy WinUI apps on other systems, though it is a very good question to what extent that kind of deployment can be considered “native.”</p>
<p>On macOS the situation is not quite as chaotic and fragmented, but there is an analogous evolution from Cocoa (AppKit) to SwiftUI; going forward, it’s likely that new capabilities and evolutions of the design language will be provided for the latter but not the former. There was a similar deprecation of <a href="https://en.wikipedia.org/wiki/Carbon_(API)">Carbon</a> in favor of Cocoa, many years ago. (One of the main philosophical differences is that Windows maintains backwards compatibility over many generations, while Apple actually deprecates older technology)</p>
<p>The idea of abstracting and wrapping platform native GUI has been around a long time. Classic implementations include wxWidgets and Java AWT, while a more modern spin is React Native. It is very difficult to provide high quality experiences with this approach, largely because of the impedance mismatch between platforms, and few successful applications have been shipped this way.</p>
<p>The meaning of “native” varies from platform to platform. On macOS, it would be extremely jarring and strange for an app not to use the system menubar, for example (though it would be acceptable for a game). On Windows, there’s more diversity in the way apps draw menus, and of course Linux is fragmented by its nature.</p>
<p>Instead of trying to decide whether a GUI toolkit is native or not, I recommend asking a set of more precise questions:</p>
<ul>
<li>What is the binary size for a simple application?</li>
<li>What is the startup time for a simple application?</li>
<li>Does text look like the platform native text?</li>
<li>To what extent does the app support preferences set at the system level?</li>
<li>What subset of expected text control functionality is provided?
<ul>
<li>Complex text rendering including BiDi</li>
<li>Support for keyboard layouts including “dead key” for alphabetic scripts</li>
<li>Keyboard shortcuts according to platform human interface guidelines</li>
<li>Input Method Editor</li>
<li>Color emoji rendering and use of platform emoji picker</li>
<li>Copy-paste (clipboard)</li>
<li>Drag and drop</li>
<li>Spelling and grammar correction</li>
<li>Accessibility (including reading the text aloud)</li>
</ul>
</li>
</ul>
<p>If a toolkit does well on all these axes, then I don’t think it much matters it’s built with “native” technology; that’s basically internal details. That said, it’s also very hard to hit all these points if there is a huge stack of non-native abstractions in the way.</p>
<h2 id="on-winit">On winit</h2>
<p>All GUIs (and all games) need a way to create windows, and wire up interactions with that window - primarily drawing pixels into the window and dealing with user inputs such as mouse and keyboard, but potentially a much larger range. The implementation is platform specific and involves many messy details. There is one very popular crate for this function – <a href="https://github.com/rust-windowing/winit">winit</a> – but I don’t think a consensus, at least yet, so there are quite a few other alternatives, including the <a href="https://github.com/tauri-apps/tao">tao</a> fork of winit used by Tauri, druid-shell, baseview (which is primarily used in audio applications because it supports the VST plug-in case), and handrolled approaches such as the one used by <a href="https://github.com/makepad/makepad">makepad</a>.</p>
<p>I would describe the tension this way (perhaps not everyone will agree with me). The <em>stated</em> scope of winit is to create a window and leave the details of what happens inside that window to the application. For some interactions (especially GPU rendering, which is well developed in Rust space), that split works well, but for other interactions it is not nearly as clean. In practice, I think winit has evolved to become quite satisfactory for game use cases, but less so for GUI. Big chunks of functionality, such as access to native menus, are missing (the main motivation behind the tao fork, and the reason <a href="https://github.com/iced-rs/iced/pull/1047">iced doesn’t support system menus</a>), and keyboard support is <a href="https://github.com/rust-windowing/winit/issues/753">persistently below</a> what’s needed for high quality text input in a GUI.</p>
<p>I think resolving some of this fragmentation is possible and would help move the broader ecosystem forward. For the time being, the Druid family of projects will continue developing druid-shell, but is open to collaboration with winit. One way to frame this is that the extra capabilities of druid-shell serve as a set of requirements, as well as guidance and experience how to implement them well.</p>
<p>In the meantime, which windowing library to adopt is a tough choice, and I wouldn’t be surprised to see yet another one pop up if the application has specialized requirements. Consider the situation in C++ world: for games, both <a href="https://www.glfw.org/">GLFW</a> and <a href="https://www.libsdl.org/">SDL</a> are good choices, but both primarily for games. Pretty much every serious UI toolkit has its own platform abstraction layer; while it would be possible to use something like SDL for more general purpose GUI, it wouldn’t be a great fit.</p>
<p>Advice: think through the alternatives when considering a windowing library. After adopting one, learn from the others to see how yours might be improved. Plan on dedicating some time and energy into improving this important bit of infrastructure.</p>
<h3 id="tradeoff-use-of-system-compositor">Tradeoff: use of system compositor</h3>
<p>One of the more difficult, and I think underappreciated tradeoffs is the extent to which the GUI toolkit relies on the system compositor. See <a href="https://raphlinus.github.io/ui/graphics/2020/09/13/compositor-is-evil.html">The compositor is evil</a> for more background on this, but here I’ll revisit the issue from the point of view of a GUI toolkit trying to navigate the existing space.</p>
<p>All modern GUI platforms have a compositor which composites the (possibly alpha-transparent) windows from the various applications running on the system. As of Windows 8, Wayland, and Mac OS 10.5, the platform exposes richer access to the compositor, so the application can provide a tree of composition surfaces, and update attributes such as position and opacity (generally providing an additional animation API so the compositor can animate transitions without the application having to provide new values every frame).</p>
<p>If the GUI can be decomposed into the schema supported by the compositor, there are significant advantages. For one, it is decidedly the most power-efficient method to accomplish effects such as scrolling. The system pays the cost of the compositor anyway (in most cases), so any effects that can be done by the compositor (including scrolling) are “for free.”</p>
<p>As an illustration of how a Rust UI app may depend on the compositor, see the <a href="https://github.com/robmikh/minesweeper-rs">Minesweeper using windows-rs</a> demo. Essentially all presentation is done using the compositor rather than a drawing API (this is why the numbers are drawn using dots rather than fonts). This sample application depends on the <a href="https://docs.microsoft.com/en-us/uwp/api/windows.ui?view=winrt-22621">Windows.UI</a> namespace to be provided by the operating system, so will only run on Windows 10 (build 1803 or later).</p>
<p>All that said, there are significant <em>disadvantages</em> to the compositor as well. One is cross-platform support and compatibility. There is currently no good cross-platform abstraction for the compositor (<a href="https://github.com/pcwalton/planeshift">planeshift</a> was an attempt, but is abandoned). Further, older systems (Windows 7 and X11) cannot rely on the compositor, so there has to be a compatibility path, generally with degraded behavior.</p>
<p>There are other more subtle drawbacks. One is a lowest-common-denominator approach, emphasizing visual effects supported by the compositor, especially cross-platform. As just one example, translation and alpha fading is well-supported, but scaling of bitmap surfaces comes with visual degradation, compared with re-rendering the vector original. There’s also the issue of additional RAM usage for all the intermediate texture layers.</p>
<p>Perhaps the biggest motivation to use the compositor extensively is stitching together diverse visual sources, particularly video, 3D, and various UI embeddings including web and “native” controls. If you want a video playback window to scroll seamlessly and other UI elements to blend with it, there is essentially no other game in town. These embeddings were declared as out of scope for Druid, but people request them often.</p>
<p>Building a proper cross-platform infrastructure for the compositor is a huge and somewhat thankless task. The surface area of these interfaces is large, I’m sure there are lots of annoying differences between the major platforms, and no doubt there will need to be a significant amount of compatibility engineering to work well on older platforms. Browsers have invested in this work (in the case of Safari without the encumbrance of needing to be cross-platform), and this is actually one good reason to use Web technology stacks.</p>
<p>Advice: new UI toolkits should figure out their relationship to the system compositor. If the goal is to provide a truly smooth, native integration of content such as video playback, then they must invest in the underlying mechanisms, much as browsers have.</p>
<h3 id="tradeoff-platform-text-rendering">Tradeoff: platform text rendering</h3>
<p>One of the most difficult aspects of building UI from scratch is getting text right. There are a lot of details to text rendering, but there’s also the question of matching the system appearance and being able to access the system font fallback chain. The latter is especially important for non-Latin scripts, but also emoji. Unfortunately, operating systems generally don’t have good mechanisms for enumerating or efficiently querying the system fonts. Either you use the built-in text layout capabilities (which means having to build a lowest common denominator abstraction on top of them), or you end up replicating all the work, and finding heuristics and hacks to access the system fonts without running into either correctness or efficiency problems.</p>
<p>There’s so much work involved in making a fully functional text input box that it is something of a litmus test for how far along a UI toolkit has gotten. Rendering is only part of it, but there’s also IME (including the emoji picker), copy-paste (potentially including rich text), access to system text services such as spelling correction, and one of the larger and richer subsurfaces for integrating accessibility.</p>
<p>Again, browsers have invested a massive amount of work into getting this right, and it’s no one simple trick. Druid, by comparison, <em>does</em> use the system text layout capabilities, but we’re seeing the drawbacks (it tends to be slow, and hammering out all the inconsistencies between platforms is annoying to say the least), so as we go forward we’ll probably do more of that ourselves.</p>
<p>Over the longer term, I’d love to have Rust ecosystem infrastructure crates for handling text well, but it’s an uphill battle. Just how to design the interface abstraction boundaries is a hard problem, and it’s likely that even if a good crate was published, there’d be resistance to adoption because it wouldn’t be trivial to integrate. There are thorny issues such as rich text representation, and how the text layout crate integrates with 2D drawing.</p>
<p>Advice: figure out a strategy to get text right. It’s not feasible to do that in the short term, but longer term it is a requirement for “real” UI. Potentially this is an area for the UI toolkits to join forces as well.</p>
<h2 id="on-architecture">On architecture</h2>
<p>One constant I’ve found is that the developer-facing architecture of a UI toolkit needs to evolve. We don’t have a One True architecture yet, and in particular designs made in other languages don’t adapt well to Rust.</p>
<p>Druid itself has had three major architectures: an intial attempt at applying ECS patterns to UI, the current architecture relying heavily on lenses, and the <a href="https://raphlinus.github.io/rust/gui/2022/05/07/ui-architecture.html">Xilem</a> proposal for a future architecture. In between were two explorations that didn’t pan out. Crochet was an attempt to provide an immediate mode API to applications on top of a retained mode implementation, and lasagna was an attempt to decouple the reactive architecture from the underlying widget tree implementation.</p>
<p>There are a number of triggers that might motivate large scale architectural changes in GUI toolkits. Among them, support for multi-window, accessibility, virtualized scrolling (and efficient large containers in general), async.</p>
<h3 id="the-crochet-experiment">The crochet experiment</h3>
<p>Now is a good time to review an architectural experiement that ultimately we decided not to pursue. The <a href="https://raphlinus.github.io/rust/druid/2020/09/25/principled-reactive-ui.html">crochet prototype</a> was an attempt to emulate immediate mode GUI on top of a retained mode widget tree. The theory was that immediate mode is easier for programmers, while retained mode has implementation advantages including making it easier to do rich layouts. There were other goals, including facilitating language bindings (for langages such as Python) and also better async integration. Language bindings were a pain point in the existing Druid architecture.</p>
<p>Ultimately I think it would be possible to build UI with this architecture, but there were a number of pain points, so I don’t believe it would be a good experience overall. One of the inherent problems of an immediate mode API is what I call “state tearing.” Because updates to app state (from processing events returned by calls to functions representing widgets) are interleaved with rendering, the rendering of any given frame may contain a mix of old and new state. For some applications, when continuously updating at a high frame rate, an extra frame of latency may not be a serious issue, but I consider it a flaw. I had some ideas for how to address this, but it basically involves running the app logic twice.</p>
<p>There were other ergonomic paper cuts. Because Rust lacks named and optional parameters in function calls, it is painful to add optional modifiers to existing widgets. Architectures based on simple value types for views (as is the case in the in the greater React family, including Xilem) can just use variations on the fluent pattern, method calls on those views to either wrap them or set optional parameters.</p>
<p>Another annoyingly tricky problem was ensuring that begin-container and end-container calls were properly nested. We experimented with a bunch of different ways to do try to enforce this nesting at compile time, but none were fully satisfying.</p>
<p>A final problem with emulating immediate mode is that the architecture tends to thread a mutable context parameter through the application logic. This is not especially ergonomic (adding to “noise”) but perhaps more seriously effectively enforces the app logic running on a single thread.</p>
<p>Advice: of course try to figure out a good architecture, but also plan for it to evolve.</p>
<h2 id="accessibility">Accessibility</h2>
<p>It’s fair to say that a UI toolkit cannot be considered production-ready unless it supports accessibility features such as support for screen readers for visually impaired users. Unfortunately, accessibility support has often taken the back seat, but fortunately the situation looks like it might improve. In particular, the <a href="https://github.com/AccessKit/accesskit">AccessKit</a> project hopes to provide common accessibility infrastructure to UI toolkits in the Rust ecosystem.</p>
<p>Doing accessibility <em>well</em> is of course tricky. It requires architectural support from the toolkit. Further, platform accessibility capabilities often make architectural assumptions about the way apps are built. In general, they expect a retained mode widget tree; this is a significant impedance mismatch with immediate mode GUI, and generally requires stable widget ID and a diffing approach to create the accessibility tree. For the accessibility part (as opposed to the GPU-drawn part) of the UI, it’s fair to say pure immediate mode cannot be used, only a hybrid approach which resembles emulation of an immediate mode API on top of a more traditional retained structure.</p>
<p>Also, to provide a high quality accessibility experience, the toolkit needs to export fine-grained control of accesibility features to the app developer. Hopefully, generic form-like UI can be handled automatically, but for things like custom widgets, the developer needs to build parts of the accessibility tree directly. There are also tricky interactions with features such as virtualized scrolling.</p>
<p>Accessibility is one of the great strengths of the Web technology stack. A lot of thought went into defining a cross-platform abstraction which could actually be implemented, and a lot of users depend on this every day. AccessKit borrows liberally from the Web approach, including the implementation in Chromium.</p>
<p>Advice: start thinking about accessibility early, and try to build prototypes to get a good understanding of what’s required for a high quality experience.</p>
<h2 id="what-of-druid">What of Druid?</h2>
<p>I admit I had hopes that Druid would become the popular choice for Rust GUI, though I’ve never explicitly had that as a goal. In any case, that hasn’t happened, and now is a time for serious thought about the future of the project.</p>
<p>We now have clearer focus that Druid is primarily a research project, with a special focus on high performance, but also solving the problems of “real UI.” The research goals are longer term; it is <em>not</em> a project oriented to shipping a usable toolkit soon. Thus, we are making some changes along these lines. We hope to get <a href="https://github.com/linebender/piet-gpu">piet-gpu</a> to a “minimum viable” 0.1 release soon, at which point we will be switching drawing over to that, as opposed to the current strategy of wrapping platform drawing capabilities (which often means that drawing is on the CPU). We change the reactive architecture to Xilem.</p>
<p>Assuming we do a good job solving these problems, over time Druid might evolve into a toolkit usable for production applications. In the meantime, we don’t want to create unrealistic expectations. The primary audience for Druid is people learning how to build UI in Rust. This post isn’t the appropriate place for a full roadmap and vision document, but I expect to be writing more about that in time.</p>
<h2 id="conclusion">Conclusion</h2>
<p>I don’t want to make too many predictions, but I am confident in asserting that there will be a dozen new UI projects in Rust in the next year or two. Most of them will be toys, though it is entirely possible that one or more of them will be in support of a product and will attract enough resources to build something approaching a “real” toolkit. I do expect fragmentation of infrastructure to continue, as there are legitimate reasons to choose different approaches, or emphasize different priorities. It’s possible we never get to a “one size fits all” solution for especially thorny problems such as window creation, input (including keyboard input and IME), text layout, and accessibility.</p>
<p>Meanwhile we will be pushing forward with Druid. It won’t be for everyone, but I am hopeful it will move the state of Rust UI forward. I’m also hopeful that the various projects will continue to learn from each other and build common ground on infrastructure where that makes sense.</p>
<p>And I remain very hopeful about the potential for GUI in Rust. It seems likely to me that it will be the language the next major GUI toolkit is written in, as no other language offers the combination of performance, safety, and high level expressiveness. All of the issues in this post are problems to be solved rather than obstacles why Rust isn’t a good choice for building UI.</p>
<p>Discuss on <a href="https://news.ycombinator.com/item?id=32112846">Hacker News</a> and <a href="https://old.reddit.com/r/rust/comments/vzz4mt/advice_for_the_next_dozen_rust_guis/">/r/rust</a>.</p>A few times a week, someone asks on the #gui-and-ui channel on the Rust Discord, “what is the best UI toolkit for my application?” Unfortunately there is still no clear answer to this question. Generally the top contenders are egui, Iced, and Druid, with Slint looking promising as well, but web-based approaches such as Tauri are also gaining some momentum, and of course there’s always the temptation to just build a new one. And every couple or months or so, a post appears with a new GUI toolkit.Xilem: an architecture for UI in Rust2022-05-07T15:17:42+00:002022-05-07T15:17:42+00:00https://raphlinus.github.io/rust/gui/2022/05/07/ui-architecture<p>Rust is an appealing language for building user interfaces for a variety of reasons, especially the promise of delivering both performance and safety. However, finding a good <em>architecture</em> is challenging. Architectures that work well in other languages generally don’t adapt well to Rust, mostly because they rely on shared mutable state and that is not idiomatic Rust, to put it mildly. It is sometimes asserted for this reason that Rust is a poor fit for UI. I have long believed that it is possible to find an architecture for UI well suited to implementation in Rust, but my previous attempts (including the current <a href="https://github.com/linebender/druid">Druid</a> architecture) have all been flawed. I have studied a range of other Rust UI projects and don’t feel that any of those have suitable architecture either.</p>
<p>This post presents a new architecture, which is a synthesis of existing work and a few new ideas. The goals include expression of modern reactive, declarative UI, in components which easily compose, and a high performance implementation. UI code written in this architecture will look very intuitive to those familiar with state of the art toolkits such as SwiftUI, Flutter, and React, while at the same time being idiomatic Rust.</p>
<p>The name “Xilem” is derived from <a href="https://en.wikipedia.org/wiki/Xylem">xylem</a>, a type of transport tissue in vascular plants, including trees. The word is spelled with an “i” in several languages including Romanian and Malay, and is a reference to <a href="https://xi-editor.io/">xi-editor</a>, a starting place for explorations into UI in Rust (now on hold).</p>
<p>Like most modern UI architectures, Xilem is based on a <em>view tree</em> which is a simple declarative description of the UI. For incremental update, successive versions of the view tree are <em>diffed,</em> and the results are applied to a widget tree which is more of a traditional retained-mode UI. Xilem also contains at heart an incremental computation engine with precise change propagation, specialized for UI use.</p>
<p>The most innovative aspect of Xilem is event dispatching based on an <em>id path,</em> at each stage providing mutable access to app state. A distinctive feature is Adapt nodes (an evolution of the lensing concept in Druid) which facilitate composition of components. By routing events <em>through</em> Adapt nodes, subcomponents have access to a different mutable state reference than the parent.</p>
<h2 id="a-quick-tour-of-existing-architectures">A quick tour of existing architectures</h2>
<p>This architecture is designed to address limitations and problems with the existing state of the art, both the current <a href="https://github.com/linebender/druid">Druid</a> architecture and other attempts. It’s a little hard to understand some of the motivation without some knowledge of those architectures. That said, a full survey of reactive UI architectures would be quite a long work; this section can only touch the highlights.</p>
<p>The existing Druid architecture has some nice features, but we consistently see people struggle with common themes.</p>
<ul>
<li>There is a big difference between creating static widget hierarchies and dynamically updating them.</li>
<li>The app data must have a <a href="https://docs.rs/druid/latest/druid/trait.Data.html">Data</a> bound, which implies cloning and equality testing. Interior mutability is effectively forbidden.</li>
<li>The “lens” mechanism is confusing and it is not easy to implement complex binding patterns.</li>
<li>We never figured out how to integrate async in a compelling way.</li>
<li>There is an environment mechanism but it is not efficient and doesn’t support fine-grained change propagation.</li>
</ul>
<p>Another common architecture is immediate mode GUI, both in a relatively pure form and in a modified form. It is popular in Rust because it doesn’t require shared mutable state. It also benefits from overall system simplicity. However, the model is oversimplified in a number of ways, and it is difficult to do sophisticated layout and other patterns that are easier in retained widget systems. There are also numerous papercuts related to sometimes rendering stale state. (I experimented with a retained widget backend emulating an immediate mode API in the “crochet” architecture experiment and concluded that the result was not compelling). The popular <a href="https://github.com/emilk/egui">egui</a> crate is solidly an implementation of immediate mode, and <a href="https://github.com/makepad/makepad">makepad</a> is also based on it, though it differs in some important ways.</p>
<p>A particularly common architecture for UI in Rust is <a href="https://guide.elm-lang.org/architecture/">The Elm Architecture</a>, which also does not require shared mutable state. Rather, to support interactions from the UI, gestures and other related UI actions creates <em>messages</em> which are then sent to an <code class="language-plaintext highlighter-rouge">update</code> method which takes central app state. <a href="https://iced.rs/">Iced</a>, <a href="https://github.com/antoyo/relm">relm</a>, and <a href="https://github.com/vizia/vizia">Vizia</a> all use some form of this architecture. Generally it works well, but the need to create an explicit message type and dispatch on it is verbose, and the Elm architecture does not support cleanly factored components as well as some other architectures. The <a href="https://guide.elm-lang.org/webapps/structure.html">Elm documentation</a> specifically warns against components, saying, “actively trying to make components is a recipe for disaster in Elm.”</p>
<p>Lastly, there are a number of serious attempts to port React patterns to Rust, of which I find <a href="https://dioxuslabs.com/">Dioxus</a> most promising. These rely on interior mutability and other patterns that I think adapt poorly to Rust, but definitely represent a credible alternative to the ideas presented here. I think we will have to build some things and see how well they work out.</p>
<h2 id="synchronized-trees">Synchronized trees</h2>
<p>The Xilem architecture is based around generating trees and keeping them synchronized. In that way it is a refinement of the ideas described in my previous blog post, <a href="https://raphlinus.github.io/ui/druid/2019/11/22/reactive-ui.html">Towards a unified theory of reactive UI</a>.</p>
<p>In each “cycle,” the app produces a view tree, from which rendering is derived. This tree has fairly short lifetime; each time the UI is updated, a new tree is generated. From this, a widget tree is built (or rebuilt). The view tree is retained only long enough to assist in event dispatching and then be diffed against the next version, at which point it is dropped. The widget tree, by contrast, persists across cycles. In addition to these two trees, there is a third tree containing <em>view state,</em> which also persists across cycles. (The view state serves a very similar function as React hooks)</p>
<p>Of existing UI architectures, the view tree most strongly resembles that of SwiftUI - nodes in the view tree are plain value objects. They also contain callbacks, for example specifying the action to be taken on clicking a button. Like SwiftUI, but somewhat unusually for UI in more dynamic languages, the view tree is statically typed, but with a typed-erased escape hatch (Swift’s AnyView) for instances where strict static typing is too restrictive.</p>
<p>The Rust expression of these trees is instances of the <code class="language-plaintext highlighter-rouge">View</code> trait, which has two associated types, one for view state and one for the associated widget. The state and widgets are <em>also</em> statically typed. The design relies <em>heavily</em> on the type inference mechanisms of the Rust compiler. In addition to inferring the type of the view tree, it also uses associated types to deduce the type of the associated state tree and widget tree, which are known at compile time. In almost every other comparable system (SwiftUI being the notable exception), these are determined at runtime with a fair amount of allocation, downcasting, and dynamic dispatch.</p>
<h2 id="a-worked-example">A worked example</h2>
<p>We’ll use the classic counter as a running example. It’s very simple but will give insight into how things work under the hood. For people who want to follow along with the code, check the idiopath directory of the <a href="https://github.com/linebender/druid/pull/2183">idiopath branch</a> (in the Druid repo); running <code class="language-plaintext highlighter-rouge">cargo doc --open</code> there will reveal a bunch of Rustdoc.</p>
<p>Here’s the application.</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">fn</span> <span class="nf">app</span><span class="p">(</span><span class="n">count</span><span class="p">:</span> <span class="o">&</span><span class="k">mut</span> <span class="nb">u32</span><span class="p">)</span> <span class="k">-></span> <span class="k">impl</span> <span class="n">View</span><span class="o"><</span><span class="nb">u32</span><span class="o">></span> <span class="p">{</span>
<span class="nf">v_stack</span><span class="p">((</span>
<span class="nd">format!</span><span class="p">(</span><span class="s">"Count: {}"</span><span class="p">,</span> <span class="n">count</span><span class="p">),</span>
<span class="nf">button</span><span class="p">(</span><span class="s">"Increment"</span><span class="p">,</span> <span class="p">|</span><span class="n">count</span><span class="p">|</span> <span class="o">*</span><span class="n">count</span> <span class="o">+=</span> <span class="mi">1</span><span class="p">),</span>
<span class="p">))</span>
<span class="p">}</span>
</code></pre></div></div>
<p>This was carefully designed to be clean and simple. A few notes about this code, then we’ll get in to what happens downstream to actually build and run the UI.</p>
<p>This function is run whenever there are significant changes (more on that later). It takes the current app state (in this case a single number, but in general app state can be anything), and returns a view tree. The exact type of the view tree is not specified, rather it uses the <a href="https://doc.bccnsoft.com/docs/rust-1.36.0-docs-html/edition-guide/rust-2018/trait-system/impl-trait-for-returning-complex-types-with-ease.html">impl Trait</a> feature to simply assert that it’s something that implements the View trait (parameterized on the type of the app state). The full type happens to be:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">VStack</span><span class="o"><</span><span class="nb">u32</span><span class="p">,</span> <span class="p">(),</span> <span class="p">(</span><span class="nb">String</span><span class="p">,</span> <span class="n">Button</span><span class="o"><</span><span class="nb">u32</span><span class="p">,</span> <span class="p">(),</span> <span class="p">{</span><span class="n">anonymous</span> <span class="n">function</span> <span class="n">of</span> <span class="k">type</span> <span class="k">for</span> <span class="o"><</span><span class="nv">'a</span><span class="o">></span> <span class="nf">FnMut</span><span class="p">(</span><span class="o">&</span><span class="nv">'a</span> <span class="k">mut</span> <span class="nb">u32</span><span class="p">)</span> <span class="k">-></span> <span class="p">()}</span><span class="o">></span><span class="p">)</span><span class="o">></span>
</code></pre></div></div>
<p>For such a simple example, this is not too bad (other than the callback), but would get annoying quickly.</p>
<p>Another observation is that three nodes of this tree implement the View trait: VStack and its two children String and Button. Yes, that’s an ordinary Rust string, and it implements the View trait. So will colors and shapes (following SwiftUI; implementation is planned in the prototype but not complete).</p>
<p>The view tree returned by the app logic can be visualized as a block diagram:</p>
<p><img src="/assets/xilem_view.svg" alt="Box diagram showing view hierarchy" /></p>
<p>The type of the associated widget is <code class="language-plaintext highlighter-rouge">widget::VStack</code>. Purely as a pragmatic implementation detail, we’ve chosen to type-erase the children of containers. Among other things, this avoids excessive monomorphization. If we wanted fully-typed widget trees, the architecture would support that, and indeed an earlier prototype chose that approach.</p>
<h3 id="identity-and-id-paths">Identity and id paths</h3>
<p>A specific detail when building the widget tree is assigning a <em>stable identity</em> to each view. These concepts are explained pretty well in the <a href="https://developer.apple.com/videos/play/wwdc2021/10022/">Demystify SwiftUI</a> talk. As in SwiftUI, stable identity can be based on <em>structure</em> (views in the same place in the view tree get to keep their identity across runs), or an <em>explicit key.</em> To illustrate the latter, assume a list container, and that two of the elements in the list are swapped. That might play an animation of the visual representations of those two elements changing places.</p>
<p>The idea of assigning stable identities is quite standard in declarative UI (it’s also present in basically all non-toy immediate mode GUI implementations), but Xilem adds a distinctive twist, the use of <em>id path</em> rather than a single id. The id path of a widget is the sequence of all ids on the path from the root to that widget in the widget tree. Thus, the id path of the button in the above is <code class="language-plaintext highlighter-rouge">[1, 3]</code>, while the label is <code class="language-plaintext highlighter-rouge">[1, 2]</code> and the stack is just <code class="language-plaintext highlighter-rouge">[1]</code>.</p>
<p>To continue our running example, given the view tree, the build step produces an associated view state tree as well as a widget tree. Here, we’ve annotated the view tree with ids, and also shown how ids and id paths are stored in the newly built structures. Most importantly, the view state for the VStack contains the ids of the child nodes, and the Button widget contains its id path, which is <code class="language-plaintext highlighter-rouge">[1, 3]</code>.</p>
<p><img src="/assets/xilem_expanded.svg" alt="Box diagram showing view hierarchy" /></p>
<p>As it turns out, neither text labels nor buttons require any special view state other than what’s retained in the widget, so those view states are just (), the unit type. Of course, other view nodes may require more associated state, in which case the type would be something other than the unit type.</p>
<details>
<summary>Advanced topic: does id identify view or widget?</summary>
In writing this explanation, I realized there is some ambiguity whether the id identifies a node in the view tree, or one in the widget tree. In many cases, there is a 1:1 correspondence between the two, so the distinction is not important. In addition, it's certainly possible to construct a widget tree so that the id of each node is provided during the method call that constructs that widget, and if so, it's more than reasonable to use the Xilem ids as generated during the reconciliation process. However, that's not required, and the identity of widgets could be from a disjoint space from Xilem ids assigned to views.
Further, it's possible to have a view node that doesn't directly correspond to a widget. That happens with holders of async futures, environment getters, and potentially other nodes that have reason to receive events. In those cases, ids are associated with views, not widgets. It's important to be clear, though, that the *lifetime* of an id is persistent across multiple cycles, while the lifetime of nodes in the view tree is shorter.
Thus, the most accurate description of Xilem ids is this: they are *persistent* identifiers that correspond to either structural or explicit identity of nodes in the view tree.
</details>
<h3 id="event-propagation">Event propagation</h3>
<p>Now let’s click that button. Raw platform events such as mouse movement and keystrokes are handled by the UI infrastructure, and don’t necessarily result in events propagated to the app logic. For example, the mouse can hover over the button (changing the appearance to a hover state), or the window can be resized, and that will all be handled without involving the app. But clicking the button <em>does</em> generate an event to be dispatched to the app.</p>
<p>Obviously the goal will be to run that callback and increment the count, but the details of how that happens are subtly different than most declarative UI systems. Probably the “standard” way to do this would be to attach the callback to the button, and have it capture a reference to the chunk of state it mutates. Again, in most declarative systems but not Xilem, setting the new state would be done using some variant of the <a href="https://en.wikipedia.org/wiki/Observer_pattern">observer pattern</a>, for example some kind of <code class="language-plaintext highlighter-rouge">setState</code> or other “setter” function to not only update the value but also notify downstream dependencies that it had changed, in this case re-rendering the label.</p>
<p>This standard approach works poorly in Rust, though it can be done (see in particular the <a href="https://dioxuslabs.com/">Dioxus</a> system for an example of one of the most literal transliterations of React patterns, including observer-based state updating, into Rust). The problem is that it requires <em>shared mutable</em> access to that state, which is clunky at best in Rust (it requires interior mutability). In addition, because Rust doesn’t have built-in syntax for getters and setters, invoking the notification mechanism also requires some kind of explicit call (though perhaps macros or other techniques can be used to hide it or make it less prominent).</p>
<details>
<summary>Advanced topic: comparison with Elm</summary>
The observer pattern is not the *only* way event propagation works in declarative UI. Another very important and influential pattern is [The Elm Architecture], which, being based on a pure functional language, also does not require shared mutable state. Thus, it is also used successfully as the basis of several Rust UI toolkits, notably Iced.
In Elm, app state is centralized (this is also a fairly popular pattern in React, using state management packages such as Redux), and events are given to the app through an `update` call. Dispatching is a three-stage process. First, the user defines a *message* type enumerating the various actions that are (globally) possible to trigger through the UI. Second, the UI element *maps* the event type into this user-defined type, identifying which action is desired. Third, the `update` method dispatches the event, delegating if needed to a child handler. Some people like the explicitness of this approach, but it is unquestionably more verbose than a single callback that manipulates state directly, as in React or SwiftUI.
</details>
<p>So what does Xilem do instead? The view tree is also parameterized on the <em>app state,</em> which can be any type. This idea is an evolution of Druid’s existing architecture, which also offers mutable access to app state to UI callbacks, but removes some of the limitations. In particular, Druid requires app state to be clonable and diffable, a stumbling block for many new users.</p>
<p>When an event is generated, it is annotated with the path of the UI element that originated it. In the case of the button, <code class="language-plaintext highlighter-rouge">[1, 3]</code>, and this is sent to the app logic for dispatching. In the case of a button click, there is no <em>payload,</em> but for a slider it would be the numeric slider value, or for a text input it would be the string.</p>
<p>Event dispatching starts at the root of the view tree, and calls to the <code class="language-plaintext highlighter-rouge">event</code> method on the <code class="language-plaintext highlighter-rouge">View</code> trait also contain a mutable borrow of the app state. In this example, the root view node is VStack. It examines the id path of the event, consults its associated view state, and decides that the event should be dispatched to the second child, as the id of that child (3) matches the corresponding id in the id path of the event. It recursively calls <code class="language-plaintext highlighter-rouge">event</code> on that child, which is the button. That is a <em>leaf node,</em> meaning there are no further ids in the id path, and the button handles that event by calling its callback, passing in the mutable borrow of app state that was propagated through the <code class="language-plaintext highlighter-rouge">event</code> traversal. That callback in turn just increments the count value.</p>
<p>Note that the UI framework retains the view tree a “fairly short” time - long enough to do any event dispatching that’s needed, and also for diffing (see below), but not longer than that.</p>
<h3 id="re-rendering">Re-rendering</h3>
<p>After clicking the button and running the callback, the app state consists of the number 1, formerly 0. The app logic function is run, producing a new view tree, and this time the string value is “Count: 1” rather than “Count: 0”. The challenge is then to update the widget tree with the new data.</p>
<p>As is completely standard in declarative UI, it is done by diffing the old view tree against the new one, in this case calling the <code class="language-plaintext highlighter-rouge">rebuild</code> method on the <code class="language-plaintext highlighter-rouge">View</code> trait. This method compares the data, updates the associated widget if there are any changes, and also traverses into children. The view tree is retained <em>just</em> long enough to do event propagation and to be diffed against the next iteration of the view tree, at which point it is dropped. At any point in time, there are at most two copies of the view tree in existence.</p>
<p>In the simplest case, the app builds the full view tree, and that is diffed in full against the previous version. However, as UI scales, this would be inefficient, so there are <em>other</em> mechanisms to do finer grained change propagation, as described below.</p>
<h2 id="components">Components</h2>
<p>The above is the basic architecture, enough to get started. Now we will go into some more advanced techniques.</p>
<p>It would be very limiting to have a single “app state” type throughout the application, and require all callbacks to express their state mutations in terms of that global type. So we won’t do that.</p>
<p>The main tool for stitching together components is the <code class="language-plaintext highlighter-rouge">Adapt</code> view node. This node is so named because it adapts between one app state type and another, using a closure that takes mutable access to the parent state, and calls into a child (through a “thunk”, which is just a callback for continuing the propagation to child nodes) with a mutable reference to the child state. We can then define a “component” in the Xilem world as a body of code that outputs a view tree with a different app state type than its parent component.</p>
<p>In the simple case where the child component operates independently of the parent, the adapt node is a couple lines of code. It is also an attachment point for richer interactions - the closure can manipulate the parent state in any way it likes. Event handling callbacks of the child component are also allowed to return an arbitrary type (unit by default), for propagation of data upward in the tree, including to parent components.</p>
<p>In Elm terminology, the Adapt node is similar to <a href="https://package.elm-lang.org/packages/elm/html/latest/Html#map">Html map</a>, though it manipulates mutable references to state, as opposed to being a pure functional mapping between message types. It is also quite similar to the “lens” concept from the existing Druid architecture, and has some resemblance to <a href="https://developer.apple.com/documentation/swiftui/binding">Binding</a> in SwiftUI as well.</p>
<h2 id="finer-grained-change-propagation-memoizing">Finer grained change propagation: memoizing</h2>
<p>Going back to the counter, every time the app logic is called, it allocates a string for the label, even if it’s the same value as before. That’s not too bad if it’s the only thing going on, but as the UI scales it is potentially wasted work.</p>
<p>Ron Minsky has <a href="https://signalsandthreads.com/building-a-ui-framework/#1523">stated</a> “hidden inside of every UI framework is some kind of incrementalization framework.” Xilem unapologetically contains at its core a lightweight change propagation engine, similar in scope to the attribute graph of SwiftUI, but highly specialized to the needs of UI, and in particular with a lightweight approach to <em>downward</em> propagation of dependencies, what in React would be stated as the flow of props into components.</p>
<p>In this particular case, that incremental change propagation is best represented as a <em>memoization</em> node, yet another implementation of the View trait. A memoization node takes a data value (which supports both <code class="language-plaintext highlighter-rouge">Clone</code> and equality testing) and a closure which accepts that same data type. On rebuild, it compares the data value with the previous version, and only runs the closure if it has changed. The signature of this node is very similar to <a href="https://guide.elm-lang.org/optimization/lazy.html">Html.Lazy</a> in Elm.</p>
<p>Comparing a number is extremely cheap (especially because all this happens with static typing, so no boxing or downcasting is needed), but the cost of equality comparison is a valid concern for larger, aggregate data structures. Here, immutable data structures (adapted from the existing Druid architecture) can work very well.</p>
<p>Let’s say there’s a parent object that contains all the app state, including a sizable child component. The type would look something like this:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nd">#[derive(Clone)]</span>
<span class="k">struct</span> <span class="n">Parent</span> <span class="p">{</span>
<span class="n">stuff</span><span class="p">:</span> <span class="n">Stuff</span><span class="p">,</span>
<span class="n">child</span><span class="p">:</span> <span class="nb">Arc</span><span class="o"><</span><span class="n">Child</span><span class="o">></span><span class="p">,</span>
<span class="p">}</span>
</code></pre></div></div>
<p>And at the top of the tree we can use a memoize node with type <code class="language-plaintext highlighter-rouge">Arc<Parent></code>, and the equality comparison <a href="https://doc.rust-lang.org/std/sync/struct.Arc.html#method.ptr_eq">pointer equality</a> on the <code class="language-plaintext highlighter-rouge">Arc</code> rather than a deep traversal into the structure (as might be the case with a derived <code class="language-plaintext highlighter-rouge">PartialEq</code> impl). The child component attaches with both a Memoize and an Adapt node.</p>
<p>The details of the Adapt node are interesting. Here’s a simple approach:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nf">adapt</span><span class="p">(</span>
<span class="p">|</span><span class="n">data</span><span class="p">:</span> <span class="o">&</span><span class="k">mut</span> <span class="nb">Arc</span><span class="o"><</span><span class="n">Parent</span><span class="o">></span><span class="p">,</span> <span class="n">thunk</span><span class="p">|</span> <span class="n">thunk</span><span class="nf">.call</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span> <span class="nn">Arc</span><span class="p">::</span><span class="nf">make_mut</span><span class="p">(</span><span class="n">data</span><span class="p">)</span><span class="py">.child</span><span class="p">),</span>
<span class="nf">child_view</span><span class="p">(</span><span class="o">...</span><span class="p">)</span> <span class="c">// which has Arc<Child> as its app data type</span>
<span class="p">)</span>
</code></pre></div></div>
<p>Whenever events propagate into the child, <code class="language-plaintext highlighter-rouge">make_mut</code> creates a copy of the parent struct, which will then not be pointer-equal to the version stored in the memoize node. If such events are relatively rare, or if they nearly always end up mutating the child state, then this approach is reasonable. However, it is possible to be even finer grain:</p>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nf">adapt</span><span class="p">(</span>
<span class="p">|</span><span class="n">data</span><span class="p">:</span> <span class="o">&</span><span class="k">mut</span> <span class="nb">Arc</span><span class="o"><</span><span class="n">Parent</span><span class="o">></span><span class="p">,</span> <span class="n">thunk</span><span class="p">|</span> <span class="p">{</span>
<span class="k">let</span> <span class="k">mut</span> <span class="n">child</span> <span class="o">=</span> <span class="n">data</span><span class="py">.child</span><span class="nf">.clone</span><span class="p">();</span>
<span class="n">thunk</span><span class="nf">.call</span><span class="p">(</span><span class="o">&</span><span class="k">mut</span> <span class="n">child</span><span class="p">),</span>
<span class="k">if</span> <span class="o">!</span><span class="nn">Arc</span><span class="p">::</span><span class="nf">ptr_eq</span><span class="p">(</span><span class="o">&</span><span class="n">data</span><span class="py">.child</span><span class="p">,</span> <span class="o">&</span><span class="n">child</span><span class="p">)</span> <span class="p">{</span>
<span class="nn">Arc</span><span class="p">::</span><span class="nf">make_mut</span><span class="p">(</span><span class="n">data</span><span class="p">)</span><span class="py">.child</span> <span class="o">=</span> <span class="n">child</span><span class="p">;</span>
<span class="p">}</span>
<span class="p">},</span>
<span class="nf">child_view</span><span class="p">(</span><span class="o">...</span><span class="p">)</span> <span class="c">// which has Arc<Child> as its app data type</span>
<span class="p">)</span>
</code></pre></div></div>
<p>This logic propagates the change up from the child object to the parent object in the app state <em>only if</em> the child state has actually changed.</p>
<p>The above illustrates how to make the pattern work for structure fields (and is very similar to the “lensing” technique in the existing Druid architecture), but similar ideas will work for collections. Basically you need immutable data structures that support pointer equality and cheap, sparse diffing. I talk about that in some detail in my talk <a href="https://www.youtube.com/watch?v=DSuX-LIAU-I">A Journey Through Incremental Computation</a> (<a href="https://docs.google.com/presentation/d/1opLymkreSTFfxygjzSLYI_uH7j1YFfE6DLl8RfCiw7E/edit">slides</a>), with a focus on text layout and a list component. Also note that the <a href="https://docs.rs/druid-derive/latest/druid_derive/">druid_derive</a> crate automates generation of these lenses for Druid, and no doubt a similar approach would work for adapt/memoize in Xilem. For now, though, I’m seeing how far we can get just using vanilla Rust and not relying on macros. I think all this is a fruitful direction for future work.</p>
<p>Also to note: while immutable data structures work <em>well</em> in the Xilem architecture, they are not absolutely required. The <code class="language-plaintext highlighter-rouge">View</code> trait itself can be implemented by anyone, as long as the <code class="language-plaintext highlighter-rouge">build</code>, <code class="language-plaintext highlighter-rouge">rebuild</code>, and <code class="language-plaintext highlighter-rouge">event</code> methods have correct implementation; change propagation is especially the domain of the <code class="language-plaintext highlighter-rouge">rebuild</code> method.</p>
<h2 id="type-erasure">Type erasure</h2>
<p>This section is optional but contains some interesting bits on advanced use of the Rust type system.</p>
<p>Having the view tree (and associated view state and widget tree) be fully statically typed has some advantages, but the types can become quite large, and there are cases where it is important to <em>erase</em> the type, providing functionality very similar to <a href="https://developer.apple.com/documentation/swiftui/anyview">AnyView</a> in SwiftUI.</p>
<p>For a simple trait, the standard approach would be <code class="language-plaintext highlighter-rouge">Box<dyn Trait></code>, which boxes up the trait implementation and uses dynamic dispatch. However, this approach will not work with Xilem’s View trait, because that trait is not <a href="https://huonw.github.io/blog/2015/01/object-safety/">object-safe</a>. There are actually two separate problems - first, the trait has associated types, and second, the <code class="language-plaintext highlighter-rouge">rebuild</code> method takes the previous view tree for diffing purposes as a <code class="language-plaintext highlighter-rouge">Self</code> parameter; though the contents of the view trees might differ, the type remains constant.</p>
<p>Fortunately, though simply <code class="language-plaintext highlighter-rouge">Box<dyn View></code> is not possible due to the View trait not being object-safe, there is a pattern (due to David Tolnay) for <a href="https://github.com/dtolnay/erased-serde#how-it-works">type erasure</a>. You can look at the code for details, but the gist of it is a separate trait (<code class="language-plaintext highlighter-rouge">AnyView</code>) with <code class="language-plaintext highlighter-rouge">Any</code> in place of the associated type, and a blanket implementation (<code class="language-plaintext highlighter-rouge">impl<T> AnyView for T where T: View</code>) that does the needed downcasting.</p>
<p>The details of type erasure in Xilem took a fair amount of iteration to get right (special thanks to Olivier Faure and Manmeet Singh for earlier prototypes). An <a href="https://github.com/linebender/druid/pull/1669">earlier iteration</a> of this architecture used the Any/downcast pattern everywhere, and I also see that pattern for associated state in <a href="https://github.com/audulus/rui">rui</a> and <a href="https://docs.rs/iced_pure/latest/iced_pure/">iced_pure</a>, even though the main view object is statically typed.</p>
<p>In the SwiftUI community, <code class="language-plaintext highlighter-rouge">AnyView</code> is <a href="https://www.swiftbysundell.com/articles/avoiding-anyview-in-swiftui/">frowned on</a>, but it is still useful to have it. While <code class="language-plaintext highlighter-rouge">impl Trait</code> is a powerful tool to avoid having to write out explicit types, it doesn’t work in all cases (specifically as the return type for trait methods), though there is <a href="https://github.com/rust-lang/rust/issues/63063">work to fix that</a>. There is an additional motivation for type erasure, namely language bindings.</p>
<h3 id="language-bindings-for-dynamic-languages">Language bindings for dynamic languages</h3>
<p>As part of this exploration, I wanted to see if Python bindings were viable. This goal is potentially quite challenging, as Xilem is fundamentally a (very) strongly typed architecture, and Python is archetypally a loosely typed language. One of the limitations of the existing Druid architecture I wanted to overcome is that there was no satisfying way to create Python bindings. As another negative data point, no dynamic language bindings for SwiftUI have emerged in the approximately 3 years since its introduction.</p>
<p>Yet I was able to create a fairly nice looking proof of concept for Xilem Python bindings. Obviously these bindings rely heavily on type erasure. The essence of the integration is <code class="language-plaintext highlighter-rouge">impl View for PyObject</code>, where the main instance is a Python wrapper around <code class="language-plaintext highlighter-rouge">Box<dyn AnyView></code> as stated above. In addition, <code class="language-plaintext highlighter-rouge">PyObject</code> serves as the type for both app state and messages in the Python world; an <code class="language-plaintext highlighter-rouge">Adapt</code> node serves to interface between these and more native Rust types. Lastly, to make it all work we need <code class="language-plaintext highlighter-rouge">impl ViewSequence for PyTuple</code> so that Python tuples can serve as the children of containers like VStack, for building view hierarchies.</p>
<p>I should emphasize, this is a <a href="https://github.com/linebender/druid/pull/2185">proof of concept</a>. To do a polished set of language bindings is a fairly major undertaking, with care needed to bridge the impedance mismatch, and especially to provide useful error messages when things go wrong. Even so, it seems promising, and, if nothing else, serves to demonstrate the flexibility of the architecture.</p>
<h2 id="async">Async</h2>
<p>The interaction between async and UI is an extremely deep topic and likely warrants a blog post of its own. Even so, I wanted to explore it in the Xilem prototype. Initial prototyping indicates that it can work, and that the integration can be fine grained.</p>
<p>Async and change propagation for UI have some common features, and the Xilem approach has parallels to <a href="https://rust-lang.github.io/async-book/08_ecosystem/00_chapter.html">Rust’s async ecosystem</a>. In particular, the id path in Xilem is roughly analogous to the “waker” abstraction in Rust async - they both identify the “target” of the notification change.</p>
<p>In fact, in the prototype integration, the waker provided to the Future trait is a thin wrapper around an id path, as well as a callback to notify the platform that it should wake the UI thread if it is sleeping. Somewhat unusually for Rust async, each <code class="language-plaintext highlighter-rouge">View</code> node holding a future calls <code class="language-plaintext highlighter-rouge">poll</code> on it itself; in some respects, a future-holding view is like a tiny executor of its own. A UI built with Xilem does not provide its own reactor, but rather relies on existing work such as <a href="https://tokio.rs/">tokio</a> (which was used for the prototype).</p>
<p>We refer the interested reader to <a href="https://github.com/linebender/druid/pull/2184">the prototype code</a> for more details. Clearly this is an area that deserves to be explored much more deeply.</p>
<h2 id="environment">Environment</h2>
<p>A standard feature of declarative UI is a dynamically scoped <em>environment</em> or <em>context</em> in which child nodes can retrieve information, usually in some form of key/value format, from ancestors somewhere higher up in the tree. Most Rust UI architectures have this (including existing Druid), but you should ask: is there fine-grained change propagation, or does it have to rebuild all children when <em>any</em> environment key changes?</p>
<p>We have a design for this, not fully implemented yet. The <code class="language-plaintext highlighter-rouge">Cx</code> structure threaded through the reconciliation process gains an environment, which is a map from key to (subscribers, value) tuples. The subscribers field is a set of id paths (of child nodes). There are then two <code class="language-plaintext highlighter-rouge">View</code> nodes, one for setting an environment value, which is then available to descendants, and one for retrieving that value. Let’s look at the <code class="language-plaintext highlighter-rouge">build</code> and <code class="language-plaintext highlighter-rouge">rebuild</code> methods for both of those nodes, the setter first.</p>
<p>The associated state of the setter is the value and the subscriber set. On <code class="language-plaintext highlighter-rouge">build</code> it creates that state, with the subscriber set empty. On <code class="language-plaintext highlighter-rouge">rebuild</code> it compares the new value with that stored in the state (these values must be equality-comparable), and, if they differ, sends a notification to each of the subscribers in the subscriber set, using the id path to dispatch those notifications. In both cases, it recurses to the child node, then pops the key from the environment stack in <code class="language-plaintext highlighter-rouge">Cx</code> before it returns (here, “pop” means either removing the key from the environment map, or setting the value to what it was before traversing into the setter node, depending on that previous value).</p>
<p>In the <code class="language-plaintext highlighter-rouge">build</code> case, it fetches the value and subscriber set from the environment map, and <em>adds</em> itself to that subscriber set. In either the <code class="language-plaintext highlighter-rouge">build</code> or <code class="language-plaintext highlighter-rouge">rebuild</code> case, it retrieves the value and calls the closure for its child view tree, passing in the value retrieved from the environment. Note that the “subscriber set” concept is an adaptation of the classic observer pattern to the Xilem architecture.</p>
<p>One reason this hasn’t been prototyped is that the implementation details are fairly different depending on whether reconciliation can be multithreaded (a note on that below). It’s possible to do but requires an immutable map data structure to avoid high cloning cost, as well as interior mutability for the subscription set.</p>
<p>We have a similar prototype of a <code class="language-plaintext highlighter-rouge">useState</code> mechanism, but not enough data on its usefulness to say for sure whether it carries its weight. Such a mechanism does not seem to be needed in the Elm architecture.</p>
<h2 id="other-topics">Other topics</h2>
<p>So far, I haven’t deeply explored styling and theming. These operations also potentially ride on an incremental change propagation system, especially because dynamic changes to the style or theme may propagate in nontrivial ways to affect the final appearance.</p>
<p>Another topic I’m <em>very</em> interested to explore more fully is accessibility. I <em>expect</em> that the retained widget tree will adapt nicely to accessibility work such as Matt Campbell’s [AccessKit], but of course you never know for sure until you actually try it.</p>
<p>An especially difficult challenge in UI toolkits is sparse scrolling, where there is the illusion of a very large number of child widgets in the scroll area, but in reality only a small subset of the widgets outside the visible viewport are materialized. I am hopeful that the tight coupling between view and associated widget, as well as a lazy callback-driven creation of widgets, will help with this, but again, we won’t know for sure until it’s built.</p>
<p>Another very advanced topic is the ability to exploit parallelism (multiple threads) to reduce latency of the UI. The existing Druid architecture threads a mutable context to almost all widget methods, basically precluding any useful parallelism. In the Xilem architecture, creation of the View tree itself can easily be multithreaded, and I <em>think</em> it’s also possible to do multithreaded reconciliation. The key to that is to make the <code class="language-plaintext highlighter-rouge">Cx</code> object passed to the <code class="language-plaintext highlighter-rouge">build</code> and <code class="language-plaintext highlighter-rouge">rebuild</code> methods <code class="language-plaintext highlighter-rouge">Clone</code>, which I think is possible. Again, actually realizing performance gains from this approach is a significant challenge.</p>
<h2 id="prospects">Prospects</h2>
<p>The work presented in this blog post is conceptual, almost academic, though it is forged from attempts to build real-world UI in Rust. It comes to you at an early stage; we haven’t <em>yet</em> built up real UI around the new architecture. Part of the motivation for doing this writeup is so we can gather feedback on whether it will actually deliver on its promise.</p>
<p>One way to test that would be to try it in other domains. There are quite a few projects that implement reactive UI ideas over a TUI, and it would also be interesting to try the Xilem architecture on top of Web infrastructure, generating DOM nodes in place of the associated widget tree.</p>
<p>I’d like to thank a large number of people, though of course the mistakes in this post are my own. The Xilem architecture takes a lot of inspiration from Olivier’s <a href="https://github.com/PoignardAzur/panoramix">Panoramix</a> and Manmeet’s <a href="https://github.com/Maan2003/olma">olma</a> explorations, as well as Taylor Holliday’s <a href="https://github.com/audulus/rui">rui</a>. Jan Pochyla provided useful feedback on early versions, and conversations with the entire Druid crew on <a href="https://xi.zulipchat.com/">xi.zulipchat.com</a> were also informative. Ben Saunders provided valuable insight regarding Rust’s async ecosystem.</p>
<p>Discuss on <a href="https://news.ycombinator.com/item?id=31297550">Hacker News</a> and <a href="https://www.reddit.com/r/rust/comments/ukk1p4/xilem_an_architecture_for_ui_in_rust/">/r/rust</a>.</p>Rust is an appealing language for building user interfaces for a variety of reasons, especially the promise of delivering both performance and safety. However, finding a good architecture is challenging. Architectures that work well in other languages generally don’t adapt well to Rust, mostly because they rely on shared mutable state and that is not idiomatic Rust, to put it mildly. It is sometimes asserted for this reason that Rust is a poor fit for UI. I have long believed that it is possible to find an architecture for UI well suited to implementation in Rust, but my previous attempts (including the current Druid architecture) have all been flawed. I have studied a range of other Rust UI projects and don’t feel that any of those have suitable architecture either.piet-gpu progress: clipping2022-02-24T19:33:42+00:002022-02-24T19:33:42+00:00https://raphlinus.github.io/rust/graphics/gpu/2022/02/24/piet-gpu-clipping<p>Recently piet-gpu has taken a big step towards realizing its <a href="https://github.com/linebender/piet-gpu/blob/master/doc/vision.md">vision</a>, moving the computation of path clipping from partially being done on the CPU to entirely being done on the GPU.</p>
<p>This post explains how that works. It’s been quite a journey - I actually started a draft of this more than a year ago. Since then, I’ve had to come up with a fundamental new parallel algorithm. It’s finally done. I think clipping is very much at the heart of what makes piet-gpu different than other 2D rendering engines.</p>
<h2 id="what-is-clipping">What is clipping?</h2>
<p>The basic idea of clipping is simple, but it has a profound impact on the overall imaging model. For one, it induces a <em>tree structure,</em> while drawing without clipping can be seen as a linear sequence of layers. There are different ways of implementing pure vector clipping, but the one I’ve gone with generalizes to <em>blending,</em> which is next.</p>
<p>Clipping with a vector path affects the drawing of all <em>child objects</em> of the clip node. Points inside the path are drawn, and points outside the path are not.</p>
<p><img src="/assets/clip_demo.svg" alt="Example of clipping" /></p>
<p>For pure vector path clipping, the effective clip applied to each (leaf) drawing node is the intersection of all clip paths up the tree. Thus, one viable implementation is to do this path intersection in vector space. However, boolean path operations are tricky to implement, and only CPU implementations are known; moving them to the GPU would be difficult, to say the least.</p>
<p>Rather, we treat (antialiased) clipping as a <em>blend</em> operation – in fact, it is an instance of the the <a href="https://www.w3.org/TR/compositing-1/#porterduffcompositingoperators_srcin">Porter-Duff “source in”</a> composition operator. Conceptually, the children of a clip are rendered into a temporary, intially clear buffer, the clip mask is rendered into an alpha channel, then the temporary buffer is composited on the background, the alpha channel multiplied by the mask. This approach can be done fully on the GPU, and the blending operations generalize.</p>
<p>Clips can nest arbitrarily; a clip node can be the child of another clip node. Thus, a full 2D scene is effectively a <em>tree,</em> which profoundly affects the way drawing is done.</p>
<h2 id="bounding-boxes">Bounding boxes</h2>
<p>A common technique in 2D rendering is to assign a <em>bounding box</em> to each drawing operation. The bounding box is an enclosing rectangle (hopefully tight), so that drawing operations affect only pixels inside the rectangle. For rendering any subregion of the target surface, any objects whose bounding box does not intersect that bounding box may be ignored. Piet-gpu uses bounding boxes extensively to organize parallel drawing, first by 256x256 pixel bins, then 16x16 pixel tiles. Bounding boxes exploit the fact that most 2D drawing is <em>sparse,</em> in that most draw objects don’t touch most tiles.</p>
<p>Bounding boxes of course interact with clipping. Each clip node in the tree has a bounding box, computed from the corresponding clip path. Then, the bounding boxes for all descendants in the tree is intersected with that bounding box. Another way of phrasing this is that the clipped bounding box for a draw object is the object’s own bounding box intersected with the bounding boxes of all clip paths in the path from that node to the root of the tree.</p>
<p>Computing these bounding boxes is less work than rendering the objects, but is not free. In particular, we’d like to do that work on the GPU rather than the CPU.</p>
<h2 id="per-tile-optimization">Per-tile optimization</h2>
<p>Things get even more interesting with per-tile optimization. The piet-gpu rendering pipeline is organized as a series of stages, finally writing a per-tile command list (sometimes called “tape”) for each 16x16 tile in the target. The last stage is “fine rasterization,” which plays these commands for all pixels in the tile. Within a tile there is no control flow; all commands are evaluated for all pixels.</p>
<p>In the general case, clip is rendered as follows. The fine rasterizer maintains a bunch of per-pixel state, notably a current pixel and a blend stack. The current pixel is initially clear (alpha = 0) or a background color, and ordinary draw objects are composited onto it (generally using <a href="https://www.w3.org/TR/compositing-1/#porterduffcompositingoperators_srcover">Porter-Duff “over”</a>). A <code class="language-plaintext highlighter-rouge">BeginClip</code> operation pushes the current pixel on a <em>blend stack.</em> The children of the clip are rendered, compositing into the current pixel. Then, at a matching <code class="language-plaintext highlighter-rouge">EndClip</code>, the clip mask is rendered into an alpha mask, that’s composited with the current pixel using “source in” (basically multiplying the alpha with the mask), and that result is composited (Porter-Duff “over”) with the top of the blend stack (which is popped), becoming the new current pixel.</p>
<p>Much of the time, we don’t need the general case, however. Piet-gpu rendering works by 16x16 pixel tiles. Earlier stages in the pipeline computes what should happen within a tile, and the final stage (fine rasterization) performs a sequence of drawing operations for all pixels in the tile. This gives us the opportunity to do some optimization.</p>
<p>In particular, zooming into a single tile, a clip path may be in one of 3 states: zero coverage, partial coverage, or full coverage. Partial coverage only happens with the clip path intersects the tile. Full coverage is for tiles entirely within the clip path, and zero coverage is for tiles outside the clip path.</p>
<p><img src="/assets/clip_tiles.svg" alt="Diagram showing zero, partial, and full path coverage by tile" /></p>
<p>The mask computation and compositing only needs to happen for the partial coverage tiles (shown as gray in the above figure). The others can be rendered much more efficiently. Zero coverage tiles suppress the rendering of child nodes, basically from the <code class="language-plaintext highlighter-rouge">BeginClip</code> to the corresponding <code class="language-plaintext highlighter-rouge">EndClip</code>. And full coverage tiles are basically a no-op; the mask need not be rendered, and child nodes are rendered as if there were no clip in effect.</p>
<h2 id="gpu-computation-of-clip-bounding-boxes">GPU computation of clip bounding boxes</h2>
<p>A previous iteration had accounting of bounding boxes done by CPU. A major goal of piet-gpu is to move as much computation as possible to the GPU.</p>
<p>The fundamental task is assigning to each draw object a bounding box rectangle that incorporates the intersection of all enclosing clips. A linearized representation of the scene will have a <code class="language-plaintext highlighter-rouge">BeginClip</code> element and an <code class="language-plaintext highlighter-rouge">EndClip</code> element. The <code class="language-plaintext highlighter-rouge">BeginClip</code> will also reference a path, and that path has an associated bounding box.</p>
<p>Here’s what that calculation looks like, as a sequential algorithm:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">stack</span> <span class="o">=</span> <span class="p">[</span><span class="n">viewport_bbox</span><span class="p">]</span>
<span class="k">for</span> <span class="n">element</span> <span class="ow">in</span> <span class="n">scene</span><span class="p">:</span>
<span class="k">if</span> <span class="n">element</span> <span class="ow">is</span> <span class="n">BeginClip</span><span class="p">(</span><span class="n">path</span><span class="p">):</span>
<span class="n">stack</span><span class="p">.</span><span class="n">push</span><span class="p">(</span><span class="n">intersect</span><span class="p">(</span><span class="n">stack</span><span class="p">.</span><span class="n">last</span><span class="p">(),</span> <span class="n">path</span><span class="p">.</span><span class="n">bbox</span><span class="p">()))</span>
<span class="n">element</span><span class="p">.</span><span class="n">effective_bbox</span> <span class="o">=</span> <span class="n">stack</span><span class="p">.</span><span class="n">last</span><span class="p">()</span>
<span class="k">elif</span> <span class="n">element</span> <span class="ow">is</span> <span class="n">EndClip</span><span class="p">:</span>
<span class="n">element</span><span class="p">.</span><span class="n">effective_bbox</span> <span class="o">=</span> <span class="n">stack</span><span class="p">.</span><span class="n">last</span><span class="p">()</span>
<span class="n">stack</span><span class="p">.</span><span class="n">pop</span><span class="p">()</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">element</span><span class="p">.</span><span class="n">effective_bbox</span> <span class="o">=</span> <span class="n">intersect</span><span class="p">(</span><span class="n">stack</span><span class="p">.</span><span class="n">last</span><span class="p">(),</span> <span class="n">element</span><span class="p">.</span><span class="n">bbox</span><span class="p">())</span>
</code></pre></div></div>
<p>As a sequential algorithm, this is very straightforward, almost trivial. There’s a stack of bounding boxes, and the size of that stack is bounded by the maximum nesting depth. The cost to processing each element is O(1) as well. The only problem is, you really, really don’t want to run a sequential algorithm on a GPU.</p>
<p>That raises a question: is there a way to make a parallel version of this algorithm, one that runs efficiently on actual GPU hardware? That question has been a major obsession of mine for at least a year. And I am pleased to say, the answer is yes. I’ve done it.</p>
<p>The core of my solution is what I call the stack monoid, which is a variant on the well-studied parentheses matching problem. I blogged about an earlier version in <a href="https://raphlinus.github.io/gpu/2021/05/13/stack-monoid-revisited.html">stack monoid revisited</a>. Since then, I’ve made an improved version, with almost 5x peak performance and better portability as well. I’m not going to go into the details in this blog post, rather I will just say the solution is available by magic, and focus on the application to the 2D rendering problem.</p>
<p><img src="/assets/stack_monoid_parent_tree.svg" alt="Diagram showing parent relationships in a tree" /></p>
<p>Basically, we use the result of parentheses matching for two things. First, each <code class="language-plaintext highlighter-rouge">EndClip</code> is able to access the same path and bounding box data as the corresponding <code class="language-plaintext highlighter-rouge">BeginClip</code>. In particular, that lets us do the per-tile optimization in coarse rasterization efficiently, as that shader doesn’t need to maintain significant state. Second, it computes the intersection of all clip bounding boxes on the path to the root. Rectangle intersection is, thankfully, a monoid, so it is possible</p>
<h3 id="stream-compaction">Stream compaction</h3>
<p>This section is a detail that can be skipped, but may be of interest to people writing fancier GPU algorithms.</p>
<p>The <a href="https://raphlinus.github.io/rust/graphics/gpu/2020/06/12/sort-middle.html">original piet-gpu design</a> used an “array of structures” approach to scene description, in particular a single array with fixed size elements, each of which was a tagged union of various drawing element types, including path segments. Processing this array basically requires a large switch statement to deal with the variants in the union. I had contemplated doing a stack monoid over this array, but was very worried about the performance cost of computing the stack monoid for every element in this array. I now have a <em>very</em> fast stack monoid implementation, but even so have reworked the architecture so this cannot be a problem.</p>
<p>The new architecture (described in some detail in the <a href="https://github.com/linebender/piet-gpu/issues/119">new element processing pipeline</a> issue) is more of a “structure of arrays” approach, which is extremely popular in the graphics and game world due to its performance advantage. Every major datatype gets its own stream. Further, as much of the logic for that type gets moved into its own shader dispatch, which works in bulk on only that type of object, with no big switch statement. To stitch these together, we use a bunch of indices into these streams, which are computed using prefix sum of the counts.</p>
<p><img src="/assets/clip_aos_vs_soa.svg" alt="Diagram of "array of structs" vs "struct of arrays" approach" /></p>
<p>Specifically, the draw object stage does a stream compaction and writes a <em>clip stream,</em> which is an array of just the <code class="language-plaintext highlighter-rouge">BeginClip</code> and <code class="language-plaintext highlighter-rouge">EndClip</code> objects. A clip index is an index into this stream. At the same time, it assigns a clip index to each draw object. A sequence of draw objects enclosed by the same clip all have the same clip index. In the above diagram, the “clips” array represents the clip stream written by the draw object stage. The arrows associating the different parts of the scene together are also computed in the draw object stage using prefix sum.</p>
<p>The clip stage then does parenthesis matching and bbox intersection of the clips in the clip stream. When it’s done, it assigns a bounding box to each object in the clip stream, intersecting the bounding boxes of the clip paths that have already been computed by the path processing stage, to produce clip bounding boxes. It also sets the path in <code class="language-plaintext highlighter-rouge">EndClip</code> to refer to the same path as the corresponding <code class="language-plaintext highlighter-rouge">BeginClip</code>.</p>
<p>Thus, the work of the clip stage is proportional to the number of clips in the scene, not to the total number of objects. It would take an enormous number of clips for this work to show up to any significant amount in profiles. We used similar stream compaction techniques to move to a more compact <a href="https://github.com/linebender/piet-gpu/blob/master/doc/pathseg.md">path encoding</a>, and I plan to apply it to other parts of the pipeline as well.</p>
<h2 id="clipping-and-scrolling">Clipping and scrolling</h2>
<p>One application of clipping is to define the viewport of a scrolled view in a UI. This can be represented in piet-gpu as a clip node with a transform node as a direct child, then the scrolled contents as children of the transform node. The translation associated with the transform node controls the scroll position (this architecture could do scaling as well).</p>
<p>A design goal of piet-gpu is the most of these contents can be encoded <em>once</em> and retained, so a new scene with a different scroll position could reassembled with very little work CPU-side. On the GPU side, there would be a fairly small amount of work to compute clip bounding boxes, which would be able to cull objects quite early in the pipeline,</p>
<p>Obviously this approach will work for moderate scrolling, where it is practical to have all the resources resident on the GPU. For huge scrolled windows, some virtualization is needed, with resources swapped in and out as they scroll into and out of view. Even so, this is an appealing direction to explore, as smooth scrolling is still a challenge for UI toolkits.</p>
<h2 id="related-work">Related work</h2>
<p>This work is perhaps most similar to <a href="https://w3.impa.br/~diego/projects/GanEtAl14/">Massively Parallel Vector Graphics</a>. We both represent the scene as a flattened tree, and allow arbitrary nesting depth. However, their tree algorithm is much more simplistic: for a nesting depth of n, they do n scans, each addressing one level of nesting. This work uses a new algorithm that allows arbitrary nesting depth with no slowdown. (GPU tree algorithms with a work factor proportional to the depth of the tree are not unusual; for example)</p>
<p>In a more traditional GPU renderer, the general way to do blends is to allocate a temporary texture, render into that, and then composite by drawing a quad into the render target, sampling from the intermediate texture. GPUs are very highly optimized for this sort of work, with hardware support for texture sampling and “raster ops” for compositing, but even so it requires traffic to main memory. I believe it’s faster not to have to do the work at all.</p>
<p>The techniques here are similar to those in Matt Keeter’s <a href="https://www.mattkeeter.com/research/mpr/">Massively Parallel Rendering of Complex Closed-Form Implicit Surfaces</a>. Clipping is basically the same as intersection in constructive geometry (whether 2D or 3D), and that paper uses similar techniques to optimize a tape, taking advantage of algebraic simplifications on a per-region basis. Those techniques are more general, while this work is more specialized to 2D rendering tasks.</p>
<p>It’s fairly dated by now, but Adam Langley’s blog post on <a href="http://neugierig.org/software/chromium/notes/2010/07/clipping.html">clipping in Chromium</a> makes for interesting reading. The main problem being discussed is “conflation artifacts,” which are not fully addressed by the alpha-channel compositing approach to clipping, but even so it remains the standard technique, largely because it’s more or less mandated by the W3C <a href="https://www.w3.org/TR/compositing-1/">compositing and blending</a> spec and the <a href="https://html.spec.whatwg.org/multipage/canvas.html#drawing-model">HTML canvas drawing model</a>.</p>
<p>Do you maintain a 2D renderer with interesting clip support? Are there good writeups somewhere I’ve missed? Let me know, and I’ll be happy to add links.</p>
<h2 id="next-steps">Next steps</h2>
<p>There are some things missing from this blog post, notably performance numbers. Right now, my focus in piet-gpu is to get the architecture right. It feels like that’s converging, many of the hard problems are being solved.</p>
<p>Now that the infrastructure for clipping is in place, blending should be relatively straightforward. Most of what needs to happen is additional blending logic in the fine rasterization, plus of course plumbing the relevant metadata through the pipeline. That, plus radial and sweep gradients, are the two major pieces need to support <a href="https://github.com/googlefonts/colr-gradients-spec">COLRv1 emoji</a>, the next big milestone.</p>Recently piet-gpu has taken a big step towards realizing its vision, moving the computation of path clipping from partially being done on the CPU to entirely being done on the GPU.