Wave 2 is progressing steadily though I think it’s approaching an inflection point. January was very challenging balancing out working on XAML literals alongside my contract work, but I still feel like I’m converging on a release. I want to share more insights into what’s going on behind the scenes, so I wrote this post. It’s really a chance for me to just nerd out about design so if it doesn’t excite you that’s ok. I find it cool because I’m a language design nerd. To appreciate some of the recent design changes and their implications you need some background on compilation that I’m going to try to sum up. A few years back I wrote a primer on Roslyn and how compilers work. Folks have been asking me to pick that series back up (and I know I’ll need to to involve more folks in mod development) so think of the first part of this post as a draft of doing so.

A quick review of compilation

At a high level, compiling a VB.NET program is broken into 4 phases:

• Syntax—what you type
• Semantics—what it means
• Lowering—reducing high-level VB constructs like loops and Await into low-level CLR constructs like Goto (it’s almost all gotos at the CLR level)
• Emit—writing the actual bytes that the CLR will load and run to an EXE or DLL file

The scope and separation between the first two are what’s relevant for this post.

Syntactic analysis means taking a sequence of characters literally one character at a time and assigning those characters to structure based on the set of rules that most people intuitively think of as “the language”. Syntactical analysis itself happens in two “phases”. I put that in quotes because in a textbook college-level computer science class you might think of them as “phases” that happen one after the other but technically in Roslyn they happen at the same time and are very interconnected. Those phases are scanning (also called lexing, or lexical analysis, or tokenizing) and parsing.

Anyway, if you’re reading this blog you’re almost certainly a VB.NET developer so I can just describe these concepts in terms of VB without straining analogies to natural languages like English.

Scanning

Given the string of characters: Await A.B(C) ‘ Comment, scanning chops the string up into a sequence of 7 tokens:

• Await
• A
• .
• B
• (
• C
• )

Scanning of course uses whitespace to separate tokens but whitespace (in VB) isn’t itself a token (other than newlines). Comments are, like whitespace, not tokens, they’re just something the scanner knows to skip over.

Parsing

The parser takes that flat sequence of 7 tokens and builds a tree structure

• ExpressionStatement
  • AwaitExpression
    • AwaitKeyword: Await
    • InvocationExpression
      • MemberAccessExpression
        • Identifier: `A`
        • Dot: `.`
        • Identifer: `B`
      • ArgumentList
        • OpenParenthesis: `(`
        • Argument
          • Identifier: `C`
        • CloseParenthesis: `)`

So, in this example the scanner decides that A, B, and C are identifiers (as opposed to keywords, punctuation, operators, or literals) but not what they refer to. The parser is what uses those tokens to build structure of expressions and statements (and declarations): C is an argument to an invocation, B is (presumably) a named member of A, etc. But again, parsing doesn’t decide what any of those identifiers refer to. Maybe none of those identifiers refer to anything and the program is an error. Maybe A is a namespace and B is a function in a module in that namespace. Maybe A is a local variable, B is a property that returns a collection, and C is being passed to the default property of that collection. The parser doesn’t determine that, that’s what Semantic Analysis does.

Interdependence

Again, in an academic context lexical analysis and parsing are very neatly separated but in practice in VB.NET the parser directs the scanner with context that only it knows about. For example, the parser knows whether it’s parsing inside of an Async method and that’s what determines whether Await is a keyword or an identifier (and thus whether Await (x) is a parenthesized expression being awaited or something called Await being invoked with argument x).

This familiarity between the parser and the scanner are what make it almost trivial to embed languages like XML, JSON, and the String.Format syntax for interpolated strings into VB; the parser is what knows whether it’s “in” an XML literal, JSON literal, or interpolated string and thus instructs the scanner to tokenize characters appropriate to that context.

Token values

Part of scanning also involves assigning what in Roslyn we call the value of a token. So identifier A, escaped identifier [A], suffixed identifier A% all have the value “A”—that’s the name that identifier declares or refers to. Decimal integer literal 10, hexadecimal literal &H0A, and binary literal &B1010 all have the value of integer 10. The value is part of the token, is what’s used by the compiler during semantic analysis, and is separate from text of a token. Another more relevant example is the string literal ”5’8””” which contains an escaped quotation mark, the value of that string is 5’8” what would be written to the console if you printed that string token. Likewise the value for XML text tokens in an XML literal replace XML character and entity references like &, &, & with the character &.

Immutability

Now, the last thing to keep in mind about the phases of compilation is that the output of parsing is an immutable syntax tree—a hierarchical data structure that cannot change once created and is fed into semantic analysis (along with project references) to determine the meaning of a program. There’s limited examples of parsing re-interpreting a scanned token in the moment (e.g. around contextual keywords) but that’s still in the midst of parsing. Once parsed there’s no going back during semantic analysis to decide that the developer typed something else.

OK. I think that’s enough of a review for the rest of this post.

A XAML Literal is a semantic distinction, not a syntactic one

With that understanding, note that whether or not an XML literal in VB represents a XAML literal is a semantic distinction. XAML is XML and syntactically (to the parser) a XAML literal will look like any other XML literal. It’s during semantic analysis that the XML namespaces in scope (which, remember, may be imported at the file or project level) are examined and checked against the available runtime extensions to determine what that XML translates to. Whether that’s an XElement instance, a System.Windows.Controls.StackPanel instance, or a instance of a class from an entirely different library is all determined semantically.

Markup extensions

This is all mostly good and well until you come to implement markup extensions because markup extensions have their own syntax.

In my original prototypes, I didn’t implement any special handling of markup extensions in the compiler. There were just hooks for extensions to process attribute values before assigning final values to members and those extensions could inspect XAML values for things they knew about like StartsWith(”{Binding “) and do whatever they wanted without the compiler having to know anything about it.

There are a several problems with this approach.

Markup extensions can be nested

OK, so one problem that comes up is that markup extensions aren’t just a pattern of simple values, they can be nested:

Text=”{Binding Name, Source={StaticResource Company}}”

But this isn’t the end of the world. Yes, it means the any implemented extension has to be able to recursively parse and process all of the supported markup extensions recursively. Annoying, but doable.

Markup extensions can be user-defined

Supporting all of the familiar markup extensions like Binding, StaticResource, DynamicResource, etc, would be very tedious on its own. But then you have to consider that XAML allows anyone to define their own markup extensions. And it’s not unreasonable to expect that if a user has defined such an extension that they’d be able to use them in a XAML literal just as they would in WPF, Silverlight, or WinUI.

So, ok, the extensions can try to parse and resolve custom binding extensions. Since these extensions will likely be mapped to arbitrary XML/XAML namespaces rather than in the default namespace it does mean the compiler must have a way to supply the XAML namespace mappings in scope for the extensions to do this but for other scenarios (type converters) the compiler would have to do that anyway so it’s not the end of the world. It does mean a lot of parsing, converting, and reflection at runtime which is a perf hit but again, it’s doable.

Users should expect a rich IDE experience typing markup extensions

If you were typing a XAML literal what kind of editor support would you expect? Completion on XAML namespace prefixes? Markup extension type names? Maybe data-binding property names? What about colorization within the attribute?

Technically the Visual Studio and Roslyn APIs do supply enough hooks that someone could, with a lot of work, implement these things over an opaque string (e.g. RegEx syntax colorization). But…

Markup extensions can have errors

Since all of this handling of markup extensions is deferred until runtime through extension helpers, when something goes wrong the way it would naturally be surfaced to a developer is at runtime—which stinks. What if you mistype a markup extension name—wouldn’t you want a compile-time squiggle for that? Of course you would! And then what about syntax errors—if you fail to close a markup extension you’d expect to hear about that early too.

But that’s where things get tricky! Because as I mentioned a while back, a XAML literal is a semantic distinction. In other words, …

<TextBox Text=”{Binding Name” />

…is an error if and only if that XML element expression is also a XAML literal. It would be too grievous a breaking change if otherwise valid XML literals all of a sudden started reporting syntax or semantic errors based on the fact that they would be erroneous XAML.

At first glance this made me think that compile-time error reporting should be left to an analyzer rather than the compiler because the compiler has a higher bar for correctness than an analyzer, which can only report warnings which can be ignored. Now you have a situation where the XAML provider extension is reparsing attributes values at runtime, and the IDE and analyzers are each reparsing attribute values as you type which really erodes the promise of Roslyn. So, eventually I decided that I wanted to do some analysis of markup extensions in the compiler.

Now that the compiler is processing markup extensions, it has to be decided how this information is represented. Naively, you’d add to the Roslyn Syntax API several nodes modeling markup extension syntax like you would when adding any new syntax to the language. But once again, remember that we only know that an XML literal represents a XAML expression at binding time—during semantic analysis. It would be … problematic for the syntax API to report that something “looks like” a special kind of XAML attribute. But, maybe not too crazy. By analogy, the compiler already has the concept of “structured trivia”. This technique allows the compiler to attach structure to a comment in a way that sits beneath its normal representation and this is what is used for XML documentation comments—interested parties can inquire on whether a comment has structure and other parties can treat it as just a comment. Maybe I could do something like that with the XML text tokens of an XML literal but I’m extremely hesitant as while structured trivia (comments) are a concept in the Syntax API, there’s no such analog for a potentially structured token.

Circling back to the question of how errors are represented, …

<TextBox Text=”{Binding Name” />

In our example we only want to report a syntax error here for a XAML literal, not a regular XML literal. But as I mentioned, once created a syntax tree is immutable. We can’t decide later that it has syntax errors or that it does not have syntax errors. So this is the cleverness that I propose:

We parse the markup extension but throw away any errors we find. If there were no errors we produce a valid WellformedMarkupExtensionSyntax node and if there are any errors we produce a valid MalformedMarkupExtensionSyntax node. In either event we don’t report ANY errors during parsing. When it comes time to bind a MarkupExtensionSyntax, if it’s a WellformedMarkupExtensionSyntax AND contained in a semantic XAML literal we bind it using XAML rules. If it’s a MalformedMarkupExtensionSyntax (inside a XAML literal) we reparse the entire expression but reporting this time reporting “syntax” errors alongside semantic errors. If it’s not a XAML literal we ignore the structure and just use XML value of the text in a regular XML literal.

Voila! Now we can have our cake and eat it too. We can report “syntax” errors for markup extensions but only inside XAML literals, and we can provide rich structure to the IDE and analyzers that they don’t have to recreate for themselves, and XAML provider extensions don’t have to do almost anything because the compiler is handling the semantic analysis of the markup extension (which is a XAML concept by the way, so it’s entirely reasonable that the compiler would do this). This is great because the actual rules for parsing and binding “text” inside a markup extension are arduous enough that to force anyone else to re-implement them would simply be mean spirited.

Spoiler: This isn’t what I ended up doing.

Markup extensions II

As I indicated in my last post, this whole thing has been part of the process of embracing the scope of the core feature of “Wave 2” as a true to name XAML literal, not just XAML-like enhancements to XML literals. This shift in mindset both frees me and binds me to lean on the formal XAML specification. For the most part this has been liberating and the XAML spec is actually a bit of a fun read. It describes a type system for XAML without actually depending on that type system being implemented by the CLR or any specific XAML vocabulary such as WPF, with notions such as “assignable to the x:MarkupExtension XamlType” rather than the CLR notion of inheritance from the WPF MarkupExtension class. It describes relationships between “types” and “members” and lookup without depending on the concept of a CLR property or classes or even static typing per se. If this doesn’t sound like a fun read to you, that’s normal. I am not. Not anymore.

So, there I am reading the XAML specification and for the most part everything is making sense and on page 72, in Section 8.6.3 “Member Node Creation from an XML:attribute”, I come across this one line:

Let attributeText be the string xmlAttribute[normalized value].

…and my head explodes.

The normalized value. Not the text as written. That’s when it clicks that the processing of attributes isn’t an alternate interpretation of XML but a subsequent step. In other words, in XAML (and you can verify this in the Visual Studio XAML editor right now), all of these (and more) are equivalent and perfectly valid markup extension usages:

• {Binding Name}
• {Binding Name&#x7D
• {Binding Name}

So it’s not enough to simply look at the text as we’re scanning an XML text token and check for a { character, one has to first tokenize the entire attribute value as XML—then retokenize the normalized value (the value after all XML character references and entities have been resolved). And there’s a further layer to this because markup extension syntax itself has an escape syntax where the value of {}{Binding} (or e.g. {}{Binding}) is not a markup extension but whatever follows the {} (however they’re encoded). And I hear you right now thinking “Who cares, though? You’re never going to run into this in the wild” and you’re right but we’re doing this The Right Way™️ so… in for a penny…

While at first this realization may seem like a complication, it actually simplified the design in that it left me firmly resolved that all computation of XAML “values” should happen entirely in semantic analysis. That is to say that once again parsing will be completely pure of any involvement of the rules of XAML or markup extensions or markup extension escapes and will purely handle the interpretation of text as valid XML. This is great for all existing code that handles XML literals as no special handling would be needed to undo pre-emptive XAMLfication of any speculative WellformedMarkupExtensionSyntax, MalformedMarkupExtensionSyntax, EscapedXamlAttributeTextSyntax, or anything else like that.

Designing an API for efficient tooling

As I mentioned above, the XAML specification provides a whole type system describing XamlType and XamlMember instances so it’s very natural to represent these concepts as first class in the compiler internals and to expose them through an API so that the IDE and analyzers can ask questions like “What type does this element construct?” and “What member does this attribute initialize?”; reasonable questions for providing a rich editing experience with completion and quick info, etc.

However, the Roslyn APIs are designed such that semantic questions are normally asked in relation to syntax nodes that very efficiently convey which span of text is being asked about and with XAML literal translation happening entirely in semantic analysis there is now a need for some way to ask these questions about offsets in an attribute’s text (and that’s a simplification). Long story short, I’m creating a pseudo-syntax API that captures the markup extension syntax so that tools can still be written efficiently. It’s very consistent with the true syntax API as I feel that’s ideal for API consumers. Having said that, I do think that XAML code editing tooling will have to wait for a future minor ModVB release because I need to get this feature out into your hands for testing/feedback ASAP. Maybe I’ll enlist help in building the code editing tooling from a community member.

Separations from and Interactions with top-level code

A few more design points…

InitializeComponent

In my earlier prototypes I planned to have the binding of top-level expressions special-case XML literals to support the common case of a XAML window or root object being initialized by a single literal. However, with this revision I’ve decided to move this behavior from being a special case of top-level code to being inherent behavior of a XAML literal expression statement inside a Sub (a method which does not return a value) whose name matches some configurable identifier such as InitializeComponent. This means that whether as a top-level expression statement, an expression statement in a top-level method, or an expression statement in a normal method in a normal class XAML-based initialization will work as expected. Aside from being a little cleaner this design is critical to enable testing of the XAML initialization scenario in isolation without top-level code, which is not implemented yet.

Name and x:Name/Code-behind

Another use case which is essential to reproducing the experience we’re all familiar with from WPF is backing field generation and wire-up for named controls. As it stands, the design of XAML literals is that using the x:Name directive (defined by the XAML spec) or setting the equivalent member (also a specified concept) on a XAML object will lookup a field or property on the specified x:Class and initialize it with that object. It does not cause that field or property to be created.

Part of the reason for this is because of a separation of subphases within Semantic Analysis—XAML literals are just normal expressions and are bound in the context of a method. Method body binding happens after the phase of semantic analysis that processes declarations. Of course, it would place undue burden on users if they were required to defined all their backing fields for named controls manually so the tension right now is between having these declarations created as a part of building declarations from top-level code (in contrast to the previous heading regarding InitializeComponent methods), or generating the fields explicitly through a source generator. The generator approach is closer to how WPF works today but the problem is to correctly determine which fields are generated and even their types requires semantic information. For example, if I assign a name to a control inside of a ControlTemplate or DataTemplate, Style, ResourceDictionary, or other deferred content that name technically exists within a separate namescope (A XAML spec concept). Only named objects within the root namescope should have fields generated. Which classes create their own namescopes is semantic knowledge which incurs a heavier performance penalty when using a source generator because the compilation must first be built up to the point where that information is available in order for a generator to ask about it.

Of course, a source generator could be hard coded with this knowledge (unlike the compiler) on a per-XAML vocabulary basis. One could even squint if the generator created too many fields in this case. But the alternative is to allow the compiler itself to do just enough semantic analysis of XAML types while building the declaration tree to create these fields. Both solutions can be elegant if you throw enough code at them and I’m currently trying to resolve this tension, though it won’t hold up testing/stabilization of the XAML literal feature because of my original design point—the XAML literal need only find the fields and initialize them, it takes no dependency on how those fields or properties are defined.

Another scenario that benefits from this separation, sort of, is the use of a XAML literal as an expression within a method. Having the literal take no responsibility for generating any new symbols means that a) I don’t need to design what to generate in such a use case, and b) it’s straight forward to design backing such objects with local variables.

The long tail of XAML scenarios

I hinted at the special handling of deferred content such as ControlTemplate and DataTemplate instances in the previous heading. There’s more testing and implementation to be done there beyond just namescopes. For example, .NET MAUI handles these very differently from WPF or Avalonia UI so I have to implement and test both approaches.

Another trailing work item I need to implement is ambient properties, which are a WPF concept rather than a XAML concept but an essential one for styling. I’m tempted to defer supporting this until later but styling is a pretty central WPF use case. Either way I think this scenario defines the cut line for wrapping the initial implementation of XAML literals. There is bound to be a long tail of minute technology specific scenarios the community will find when trying to port examples of different technologies to use ModVB and I’ll simply have to address those bugs as they come up.

Once and only once I feel confident in the stability of this subset of XAML literal capabilities will I move on to the implementation of top-level code. I expect a lot more stability bugs to come out of that feature than XAML literals and I want to make sure I’m not fighting two fires at once by mixing the two. I do think my original design for top-level code is pretty elegant and still the correct way to go and should be fairly quick to implement so fingers crossed Wave 2 should release this month. I already know supplemental features and tooling will come in a future wave.

Wrap-up

Thanks for listening to a rambling nerd to the bitter end. My next post will be a short personal update talking about what it’s been like juggling 4 projects in the month of January and other stuff.

Regards,

-ADG

This blog post was brought to you in part with the support of my wonderful Patreon community (which you can join). Special thanks to my Lunch tier patrons and higher: Dr. Donna M., Alex T., Kirill O., Neal G., Govert D., Dr. Anne J., Taiwo A., Jonathon M., Peter B., Kevin R., and Paul C.