# Strings and Text
## Introduction
### Strings
A string is a sequence of characters such as `"Hello, ๐!"` or `"Simplify(๐จโ๐ ร โก๏ธ) โ ๐จโ๐ค"`.
In the Compute Engine, strings are composed of encoding-independent Unicode
characters and provide access to those characters through a variety of Unicode
representations.
Strings are **not handled as collections**. This is because the concept of a
โcharacterโ is inherently ambiguous: a single user-perceived character (a
**grapheme cluster**) may consist of multiple **Unicode scalars** (code
points), and those scalars may in turn be represented differently in various
encodings: UTF-8, UTF-16, or UTF-32.
For example:
- The grapheme `รฉ` can be represented as one Unicode scalar (`U+00E9`) or
two scalars (`U+0065` + `U+0301`, i.e. `e` + combining acute).
- The emoji `๐จโ๐` is a grapheme cluster made of multiple scalars:
`[U+1F468, U+200D, U+1F680]`.
In UTF-8, it's encoded as the byte sequence:
`[240, 159, 145, 168, 226, 128, 141, 240, 159, 154, 128]`
In UTF-16, it's encoded as the code units:
`[55357, 56457, 8205, 55357, 56960]`
```live
const s = ce.string("Hello, ๐!");
console.info(ce.function("Utf8", [s]).evaluate().json);
```
To avoid confusion and ensure consistent behavior, strings are **not accessed directly** as collections of characters. Instead, they must be **explicitly converted** either to a sequence of **grapheme clusters** (what users perceive as individual characters), or to a sequence of **Unicode scalars** (code points). For encoding-level operations (such as manipulating UTF-8 or UTF-16), strings must be converted to their encoded form, as **Unicode scalars are not encodings**. This distinction matters because a single grapheme cluster may be composed of multiple scalars, and each scalar may map to different byte representations depending on the encoding.
### Annotated Expressions
An **annotated expression** is an expression that carries additional visual or
semantic metadata that is not material to the interpretation of an expression
such as text color and size or other typographic variations, a tooltip or a hyperlink
data to link to a web page.
While annotated expressions can be applied to strings, they can also
be used to annotate mathematical expressions, such as variables, operators, or
functions, to provide additional context or visual emphasis.
For example, an annotated expression can be used to highlight a specific
part of a mathematical expression:
```json example
["Equal",
"circumference",
["Multiply", 2, ["Annotated", "Pi", {"color": "blue"}], "r"]
]
// โ Pi (in blue)
```
which would correspond to the LaTeX expression:
```latex
\mathrm{circumference} = 2 \cdot \textcolor{blue}{\pi} \cdot r
```
Annotated expressions are similar to attributed strings in other systems.
When `\textcolor` (or `\color`) wraps a bare **operator** rather than an operand
โ for example `x \textcolor{red}{=} y` โ the operator is parsed as usual
(`["Equal", "x", "y"]`) and the color is dropped: MathJSON has no way to attach
an `Annotated` wrapper to a lone operator glyph. Coloring an operand (such as
`\textcolor{red}{x + 1}`) still produces an `Annotated` expression.
### Text Expressions
A `["Text"]` expression is a sequence of strings, annotated expressions or
other `["Text"]` expressions. It is used to represent formatted text content,
for example from a LaTeX expression like `\text{Hello \mathbf{world}}`.
What would happen if you used a string expression instead of a text expression?
The arguments of a `["String"]` expression get converted to their string
representation, then joined together with no spaces. The text representation
of an annotated expression is the name of the expression, not its formatted
version. For example, `["Annotated", "world", {"dict": {"color": "blue"}}]` would
be serialized to LaTeX as `\mathrm{Annotated}(\text{world}, {color \to "blue"})`, which is not what you want.
The arguments of a `["Text"]` expression remain a sequence of elements. When
serialized to LaTeX, the elements are serialized to appropriate LaTeX commands
to preserve their formatting and structure.
```js example
const stringExpr = ce.expr([
"String",
"Hello",
["Annotated", "world", {dict: {"color": "blue"}}]
]);
console.info(stringExpr.latex);
// โ "\text{Hello $\mathrm{Annotated}(\text{world}, {color: "blue"})$}"
const textExpr = ce.expr([
"Text",
"Hello",
["Annotated", "world", {dict: {"color": "blue"}}]
]);
console.info(textExpr.latex);
// โ "\text{Hello \textcolor{blue}{world}}"
```
## Functions
..._strings_: string
Concatenate strings. In Epsil, the `<>` operator constructs `StringJoin`.
```json example
["StringJoin", "hello", " ", "world"]
// โ "hello world"
```
any*
A string created by joining its arguments. The arguments are converted to
their default string representation.
```json example
["String", "Hello", ", ", "๐", "!"]
// โ "Hello, ๐!"
["String", 42, " is the answer"]
// โ "42 is the answer"
```
any, _format_:string?
Convert the argument to a string, using the specified _format_.
| _format_ | Description |
| :--- | :--- |
| `utf-8` | The argument is a list of UTF-8 code points |
| `utf-16` | The argument is a list of UTF-16 code points |
| `unicode-scalars` | The argument is a list of Unicode scalars (same as UTF-32) or a single Unicode scalar |
If no _format_ is specified, the default is `unicode-scalars`.
For example:
```json example
["StringFrom", ["List", 240, 159, 148, 159], {str: "utf-8"}]
// โ "Hello"
["StringFrom", ["List", 55357, 56607], {str: "utf-16"}]
// โ "\u0048\u0065\u006c\u006c\u006f"
["StringFrom", 128287]
// โ "๐"
["StringFrom", ["List", 127467, 127479]]
// โ "๐ซ๐ท"
```
string
Return a list of UTF-8 code points for the given _string_.
**Note:** The values returned are UTF-8 bytes, not Unicode scalar values.
```json example
["Utf8", {str: "Hello"}]
// โ ["List", 72, 101, 108, 108, 111]
["Utf8", {str: "๐ฉโ๐"}]
// โ ["List", 240, 159, 145, 169, 226, 128, 141, 240, 159, 142, 147]
```
**To create a string from UTF-8 code points**, use the `["StringFrom", _list_, "utf-8"]` function.
**See also**: [`Utf16`](#utf16), [`UnicodeScalars`](#unicodescalars) and [`Characters`](#characters).
string
Return a list of UTF-16 code points for the given _string_.
**Note:** The values returned are UTF-16 code units, not Unicode scalar values.
```json example
["Utf16", {str: "Hello"}]
// โ ["List", 72, 101, 108, 108, 111]
["Utf16", {str: "๐ฉโ๐"}]
// โ ["List", 55357, 56489, 8205, 55356, 57235]
```
**To create a string from UTF-16 code units**, use the `["StringFrom", _list_, "utf-16"]` function.
**See also**: [`Utf8`](#utf8), [`UnicodeScalars`](#unicodescalars) and [`Characters`](#characters).
string
A **Unicode scalar** is any valid Unicode code point, represented as a number
between `U+0000` and `U+10FFFF`, excluding the surrogate range
(`U+D800` to `U+DFFF`). In other words, Unicode scalars correspond exactly to
UTF-32 code units.
This function returns the sequence of Unicode scalars (code points) that make
up the string. Note that some characters perceived as a single visual unit
(grapheme clusters) may consist of multiple scalars. For example, the emoji
๐ฉโ๐ is a single grapheme but is
composed of several scalars.
```json example
["UnicodeScalars", {str: "Hello"}]
// โ [72, 101, 108, 108, 111]
["UnicodeScalars", {str: "๐ฉโ๐"}]
// โ [128105, 8205, 127891]
```
**To create a string from Unicode scalars**, use the `["StringFrom", _list_, "unicode-scalars"]` function.
**See also**: [`Utf8`](#utf8), [`Utf16`](#utf16), and [`Characters`](#characters).
string
A **grapheme cluster** is the smallest unit of text that a reader perceives
as a single character. It may consist of one or more **Unicode scalars**
(code points).
For example, the character **รฉ** can be a single scalar (`U+00E9`) or a
sequence of scalars (**e** `U+0065` + **combining acute** `U+0301`),
but both form a single grapheme cluster.
Here, **NFC** (Normalization Form C) refers to the precomposed form of characters, while **NFD** (Normalization Form D) refers to the decomposed form where combining marks are used.
Similarly, complex emojis (๐ฉโ๐, ๐ซ๐ท)
are grapheme clusters composed of multiple scalars.
The exact definition of grapheme clusters is determined by the Unicode Standard
([UAX #29](https://unicode.org/reports/tr29/)) and may evolve over time as new
characters, scripts, or emoji sequences are introduced. In contrast, Unicode
scalars and their UTF-8, UTF-16, or UTF-32 encodings are fixed and stable across Unicode versions.
The table below illustrates the difference between grapheme clusters and Unicode scalars:
| String | Grapheme Clusters | Unicode Scalars (Code Points) |
|:-------------|:--------------------|:------------------------------------|
| `รฉ` (NFC) | `["รฉ"]` | `[233]` |
| `eฬ` (NFD) | `["รฉ"]` | `[101, 769]` |
| `๐ฉโ๐` | `["๐ฉโ๐"]` | `[128105, 8205, 127891]` |
This function splits a string into grapheme clusters โ the user-perceived
"characters" of the string:
```json example
["Characters", "Hello"]
// โ ["H", "e", "l", "l", "o"]
["Characters", "๐ฉโ๐"]
// โ ["๐ฉโ๐"]
["UnicodeScalars", "๐ฉโ๐"]
// โ [128105, 8205, 127891]
```
For more details on how grapheme cluster boundaries are determined,
see [Unicodeยฎ Standard Annex #29](https://unicode.org/reports/tr29/).
**Synonym**: `GraphemeClusters` โ the original name of this function, kept
for compatibility.
**See also**: [`StringSplit`](#stringsplit), [`Utf8`](#utf8), [`Utf16`](#utf16), and [`UnicodeScalars`](#unicodescalars).
stringstring, _separator_:string
Split a string into a list of substrings.
With no _separator_, the string is split on runs of whitespace, and empty
parts are dropped. Whitespace is defined as the code points with the Unicode
`White_Space` property (`U+0009`โ`U+000D`, `U+0020`, `U+0085`, `U+00A0`,
`U+1680`, `U+2000`โ`U+200A`, `U+2028`, `U+2029`, `U+202F`, `U+205F`,
`U+3000`) โ the definition does not depend on the host's interpretation of
`\s`.
With a _separator_ string, the string is split on each occurrence of the
separator, and empty parts are kept.
```json example
["StringSplit", {str: "the quick brown fox"}]
// โ ["the", "quick", "brown", "fox"]
["StringSplit", {str: "a,b,,c"}, {str: ","}]
// โ ["a", "b", "", "c"]
```
**To split into individual characters**, use [`Characters`](#characters).
**See also**: [`Characters`](#characters).
_value_:integer_value_:integer, _base_:integer
Represent an _integer_ in a specific _base_, such as hexadecimal or binary.
If no _base_ is specified, use base-10. _base_ should be an integer from 2 to 36.
`BaseForm` evaluates to the numeric value it represents, so based numerals
participate in arithmetic:
```json example
["BaseForm", 23, 2]
// โ 23
```
**Parsing.** A numeral with an integer-literal subscript base, e.g. `10111_2` or
`2748_{16}`, parses to `BaseForm`, provided every digit is valid for the base
(otherwise it stays an inert `Subscript`):
```javascript
ce.parse('10111_2').json;
// โ ["BaseForm", 23, 2]
ce.parse('1011_2 \\cdot 101_2').evaluate();
// โ 55
ce.parse('11_8 - 3_8 = 6_8').evaluate();
// โ "True"
ce.parse('19_2').json;
// โ ["Subscript", 19, 2] (9 is not a valid base-2 digit)
```
A **symbol** subscript base, e.g. `161_b`, parses to `BaseForm` of the digit
polynomial in that base, so base equations can be solved symbolically:
```javascript
ce.parse('161_b').json;
// โ ["BaseForm", ["Add", ["Power", "b", 2], ["Multiply", 6, "b"], 1], "b"]
ce.parse('161_b + 134_b = 315_b').solve('b');
// โ [0, 8]
```
**Serialization.** `BaseForm` round-trips through LaTeX as `value_{base}`:
```javascript
ce.box(['BaseForm', 23, 2]).latex;
// โ "10111_{2}"
ce.box(['BaseForm', 42, 16]).latex;
// โ "\mathrm{2a}_{16}"
```
_expr__expr_, _delim_
Visually group expressions with an open delimiter, a close delimiter
and separators between elements of the expression.
When serializing to LaTeX, render _expr_ wrapped in delimiters.
The `Delimiter` function is **inert** and the value of a `["Delimiter", _expr_]` expression is `expr`.
_expr_ is a function expression, usually a `["Sequence"]`. It should
not be a symbol or a number.
_delim_ is an optional string:
- when it is a single character it is a separator
- when it is two characters, the first is the opening delimiter and the second is the closing delimiter
- when it is three characters, the first is the opening delimiter, the second is the separator, and the third is the closing delimiter
The delimiters are rendered to LaTeX.
The open and close delimiters are a single character, one of: `()[]{}<>|โโโโโโโโโโฐโฑ"`. The open and close delimiters do not have to match.
For example, `"')]'"` is a valid delimiter.
If an open or close delimiter is `.`, it is ignored.
The separator delimiter is also a single character, one of `,;.&:|-` or `U+00B7` (middle dot), `U+2022` (bullet) or `U+2026` (ellipsis).
If no _delim_ is provided, a default delimiter is used based on
the type of _expr_:
- `["Sequence"]` -> `(,)`
- `["Tuple"]`, `["Single"]`, `["Pair"]`, `["Triple"]` -> `(,)`
- `["List"]` -> `[,]`
- `["Set"]` -> `{,}`
_width_
When serializing to LaTeX, `width` is the dimension of the spacing, in 1/18 em.
The `Spacing` function is **inert** and the value of a `["Spacing", _expr_]` expression is `expr`.
_expr_:expression, dictionary
`Annotated(expr, attributes)` is an expression that behaves exactly like `expr`,
but carries **visual or semantic metadata** as an attribute dictionary.
The attributes have no effect on evaluation. This function is inert โ it
evaluates to its first argument.
The `attributes` dictionary may include:
* Visual style hints (e.g. `weight: "bold"`, `color: "blue"`)
* Semantic metadata (e.g. `tooltip`, `language`, `link`)
Use `Annotated` when you want to attach presentational or semantic
information to an expression **without affecting its evaluation or identity**.
This is useful for rendering, tooltips, highlighting, etc.
The following keys are applicable to math expressions:
- `mathStyle` = `"compact"` or `"normal"`. The `"compact"` style is used for inline math expressions, while the `"normal"` style is used for display math expressions.
- `scriptLevel` = `0`, `1`, or `-1`, `+1`. The script level is used to
determine the size of the expression in relation to the surrounding text.
A script level of `0` is normal size, `1` is smaller, and `2` is even smaller.
The following keys are applicable to text content:
- `weight` a string, one of `"normal"`, `"bold"`, `"bolder"`, `"light"`
- `style` a string, one of `"normal"`, `"italic"`, `"oblique"`
- `language` a string indicating the language of the expression, e.g. `"en"` (English), `"fr"` (French), `"es"` (Spanish)
The following keys are applicable to both math expressions and text content:
- `color` a color name or hex code
- `backgroundColor` a color name or hex code for the background color
- `tooltip` a string to be displayed as a tooltip when the expression is hovered over
- `link` a URL to be followed when the expression is clicked
- `cssClass` a string indicating the CSS class to be applied to the expression
- `cssId` a string indicating the CSS id of the expression
The `Annotated` function is **inert** and the value of a `["Annotated", expr]` expression is `expr`.
Read more about formatting of **matrices** and **vectors**