Module: lib/compiler/parser/tokenizer

A quote-aware character scanner for Avenx component sources.

Why this exists

Declaration extraction used to be a set of regular expressions:

content.match(/<state\s+([\s\S]*?)\s*\/>/)
template.replace(/<state.*? \/>/g, '')

The two patterns above disagree with each other. The first matches a <state> tag written across several lines; the second does not, because . excludes newlines. A multi-line declaration was therefore read correctly and stripped incorrectly, so it survived into the template and was rendered as a literal element wrapping the whole component. The documented JSDoc-annotated form

<state
  /** @type {number} *\/
  count="0"
/>

hit exactly that path, with no diagnostic.

The failure is not in those two patterns. It is that a regular expression cannot describe a tag: it cannot know that a > inside a quoted attribute value is not the end of the tag, and it cannot know that the < in a < b inside an <action> body is not the start of one. Tightening the patterns moves the failure rather than removing it.

This module scans instead. It walks the source one character at a time, tracks quoting, and yields tags with exact source offsets — which is what lets the compiler remove a declaration by slicing the range it actually occupied rather than by matching a shape it hopes the declaration has.

What it is not

It is not a spec-compliant HTML parser and does not need to be. Avenx templates are authored, not scraped: there is no error recovery for mis-nested tags here, because module:lib/compiler/parser/htmlParser handles tree building. This layer answers one question — where does this tag begin and end in the source — and answers it exactly.

Source:

Members

(static, constant) RAW_TEXT_TAGS :Set:.<string:>

Tags whose content is opaque to the tag scanner.

The body of an <action> is JavaScript. It routinely contains <, > and <=, none of which start a tag, and it may contain a string literal holding markup. Scanning inside one produces phantom tags, so the scanner jumps from the open tag straight to the matching close tag and hands back the span between them verbatim.

Type:
  • Set:.<string:>
Source:

Methods

(static) createLineIndex(source) → {Object}

Builds a line/column index over a source string.

The naive getLineAndColumn recomputes the line by counting newlines from offset zero on every call, which is quadratic across a file's worth of declarations. This precomputes the newline offsets once so each lookup is a binary search.

Parameters:
Name Type Description
source string

The source text.
Source:
Returns:

A locator.

Type
Object

(static) findRawTextClose(source, from, lowerName) → {Object|null}

Finds the close tag that terminates a raw-text element.

Nesting is counted so that a nested <resource> inside a <resource> body does not close the outer one early. Matching is case-insensitive because HTML tag names are.

Parameters:
Name Type Description
source string

The source text.
from number

Offset just past the opening tag's >.
lowerName string

The lowercased tag name to close.
Source:
Returns:

Offsets of the close tag, or null.

Type
Object | null

(static) findTagEnd(source, start) → {number}

Finds the offset of the > that closes the tag beginning at start.

Quoting is tracked so that a > inside an attribute value — <a title="a > b"> — does not terminate the scan. This is the single behaviour a regular expression cannot express, and the reason this function exists.

Parameters:
Name Type Description
source string

The source text.
start number

Offset of the <.
Source:
Returns:

Offset of the closing >, or -1 if the tag is unterminated.

Type
number

(static) parseAttributeRegion(source, start, end) → {Object}

Parses an attribute list into names, values and source offsets.

Valueless attributes are reported separately from attributes whose value is the string "true". <action name="x" atomic> and <action name="x" atomic="true"> mean the same thing, but <contract static="false" /> does not mean <contract static />, so the distinction has to survive parsing rather than be reconstructed later.

Parameters:
Name Type Description
source string

The full source text.
start number

Offset where the attribute region begins.
end number

Offset where the attribute region ends (exclusive).
Source:
Returns:

The parsed attributes.

Type
Object

(static) scanTags(source, wanted) → {Array:.<ScannedTag:>}

Scans a source string for top-level tags whose name is in wanted.

Only tags at the top level of the source are reported. A declaration nested inside markup is not a declaration, and reporting one would let a literal <state> written inside a <pre> block silently become component state.

Raw-text tags (see RAW_TEXT_TAGS) have their body captured verbatim: the scanner skips from the opening tag to the matching close tag without interpreting anything in between, so an action body containing a < b, => or a string holding markup survives untouched.

Parameters:
Name Type Description
source string

The component source.
wanted Set:.<string:>

Lowercased tag names to report.
Source:
Returns:

The tags found, in source order.

Type
Array:.<ScannedTag:>

(static) stripRanges(source, ranges) → {string}

Removes a set of source ranges, returning the remaining text.

Ranges are removed by offset rather than by pattern, which is the whole point of scanning: whatever the scanner decided a declaration occupied is exactly what leaves the template, with no second, differently-shaped pattern that can disagree with the first.

Parameters:
Name Type Description
source string

The source text.
ranges Array:.<{start:: number:, end:: number:}>

Ranges to remove.
Source:
Returns:

The source with those ranges elided.

Type
string

(inner) at(offset) → {Object}

Resolves a source offset to a 1-based line and column.

Parameters:
Name Type Description
offset number

Absolute character offset.
Source:
Returns:

The position.

Type
Object

(inner) isSpace(ch) → {boolean}

Whether a character is HTML whitespace.

Parameters:
Name Type Description
ch string

A single character.
Source:
Returns:

True for space, tab, carriage return or newline.

Type
boolean

(inner) isTagNameChar(ch) → {boolean}

Characters that may continue a tag name.

Parameters:
Name Type Description
ch string

A single character.
Source:
Returns:

True when the character can continue a tag name.

Type
boolean

(inner) isTagNameStart(ch) → {boolean}

Characters that may begin a tag name.

@ is included because Avenx directives are spelled <@for>, <@css />, <@suspense>; uppercase is meaningful because a capitalised tag is a component reference rather than an element.

Parameters:
Name Type Description
ch string

A single character.
Source:
Returns:

True when the character can start a tag name.

Type
boolean

Type Definitions

ScannedTag

Type:
  • object
Properties:
Name Type Description
name string

The tag name exactly as written.
lowerName string

The tag name lowercased, for matching.
attrs Object:.<string:, string:>

Attribute values by name.
valueless Set:.<string:>

Names of attributes written without a value.
attrOffsets Object:.<string:, {start:: number:, end:: number:, valueStart:: number:}>

Source offsets per attribute.
selfClosing boolean

Whether the tag was written <x />.
start number

Offset of the opening <.
end number

Offset just past the tag, or past </name> for a raw-text tag with a body.
openEnd number

Offset just past the opening tag's >.
body string | null

Verbatim body text for a raw-text tag, else null.
bodyStart number

Offset where the body begins, or -1.
Source: