String Internals

How native byte storage shapes the lifetime, mutation, Unicode, culture, formatting, and interop behavior behind familiar string APIs.

Representation in one sentence

A Sharp Runtime string is normally an owning std::string containing UTF-8 bytes; System::String supplies static compatibility algorithms over it, and System::Text supplies builders and encoding conversions.

Why the runtime uses std::string

std::string integrates directly with native libraries, streams, containers, operating-system adapters, and the C++ toolchain. It provides deterministic storage ownership, small-string optimization where the standard library chooses it, move semantics, contiguous bytes, and familiar performance characteristics. The runtime avoids allocating a wrapper object for every textual value.

The tradeoff is semantic visibility. A .NET string is an immutable UTF-16 managed reference. An std::string is a mutable native UTF-8 byte value. Sharp Runtime helper methods can reproduce useful API shapes, but they cannot erase that difference without replacing the representation and breaking native interoperability.

Object layout, ownership, and movement

The std::string object owns its current buffer. Copy construction copies the value; move construction transfers the native state according to the standard library. Destruction releases the buffer. A direct member shares its lifetime with the enclosing object; a returned string is a new owning value.

Pointers, references, iterators, and string_view into a string can be invalidated by mutation or destruction. Capacity growth can move the buffer. A c_str() pointer is for a bounded native call, not a durable handle. Store the string owner, not its transient data pointer, when a callback or asynchronous operation must retain text.

Value semantics versus reference semantics

Assigning one string to another copies a value. It does not create two references to one immutable CLR object. That usually makes local code simpler, but it changes identity-oriented patterns:

• There is no meaningful string reference equality.
• Copying a large string may allocate unless a move or optimization applies.
• Mutating one copy does not change another.
• Passing const std::string& borrows one owner; passing by value creates or receives an independent value.
• A returned substring owns its bytes and does not keep the source alive.

Null, empty, and optional

A string value is always present, although it may contain zero bytes. System::String::IsNullOrEmpty accepts a reference and therefore only tests emptiness. This is a source-port convenience, not a hidden nullable representation.

Choose std::optional<std::string> when absent and empty are different values. Choose a pointer only when the API actually shares or borrows an object. At a C boundary, a null char* may still be meaningful, but convert it according to the boundary contract instead of adopting a global null-to-empty rule.

UTF-8 storage model

The runtime’s documented convention is UTF-8 for ordinary text. ASCII remains a one-byte subset, while other scalars use multiple bytes. The representation does not automatically validate every string mutation: native code can construct malformed UTF-8, split a sequence, or append arbitrary bytes.

System::String helper architecture

The class deletes construction and destruction and exposes only static methods. The compiled implementation lives in Core.Base. Template-free helper bodies are linked from the core archive; the values passed in and returned remain std::string.

The helper covers classification, prefix/suffix and containment, comparisons, forward/backward search, split, substring, insertion/removal/replacement, trim, padding, case conversion, concatenation, join, byte-array conversion, creation, formatting, hashing, and no-op interning. This is a curated behavioral surface, not a proxy that forwards unknown methods to a CLR.

Comparison internals

Case-sensitive comparison uses the native byte representation. CompareOrdinal is explicitly byte-by-byte. For a comparison mode, a small classifier tests whether the enum is one of the three ignore-case values. If so, both inputs are transformed one byte at a time with std::tolower; otherwise the case-sensitive path runs.

Consequently, CurrentCulture, InvariantCulture, and Ordinal do not select distinct engines. The three ignore-case names also share one engine. UTF-8 multi-byte sequences are not Unicode case-folded. Native C locale can affect classification for individual bytes. Protocol code should prefer explicit ASCII/byte rules; user-language text needs a stronger globalization layer.

Hashing internals

String::GetHashCode invokes std::hash<std::string>, widens the result to a fixed 64-bit value, XOR-folds its upper and lower halves, and returns a 32-bit signed value. Widening before the shift avoids an invalid full-width shift on wasm32 and other 32-bit targets.

The standard does not promise the same hash across library implementations, processes, or build configurations. Hashing must use the same equality policy as its container. A bytewise hash is incompatible with a custom Unicode-insensitive comparer unless the comparer supplies a matching hash.

Substring and view behavior

Substringstd::string::substr. The result is an owning copy. It does not share source storage, so source mutation or destruction cannot invalidate the returned substring. The cost is an allocation/copy for non-empty values under normal implementations.

When a hot native path only needs a short-lived view, std::string_view can avoid copying, but it introduces a borrow. Do not return a view to a local string, retain one across owner mutation, or confuse its byte positions with Unicode characters.

Mutation and helper immutability

The String helper returns new values for trim, case, replacement, substring, insertion, removal, padding, and formatting. This resembles .NET call sites and prevents the helper from mutating its input reference. The underlying value remains mutable through std::string, and callers may intentionally use native in-place operations.

Document which layer an API promises. “String is immutable” is false for the representation; “this helper does not mutate its input” is precise. A callback that receives const std::string& still borrows storage whose owner may mutate it later.

StringBuilder internals

System::Text::StringBuilder owns an internal std::string buffer. It supports append operations for strings, C strings, bytes, booleans, integers and floating values; lines; insertion; removal; replacement; formatted append; joins; copying; length and capacity; indexing; and conversion back to a string.

All positions and lengths are storage bytes. For the UTF-8 text éA, length is three. Removing one byte at offset one removes only the second byte of é and leaves malformed UTF-8. This is a deliberate current contract documented in the header, not a Unicode character editor.

Builder references and data assumptions follow native mutation rules. Reallocation can move its buffer. ToString returns an owning copy, which remains valid after the builder changes or is destroyed.

Encoding architecture

System::Text::Encoding is an abstract base with factories for UTF-8, ASCII, UTF-16 little endian, UTF-16 big endian, UTF-32, UTF-7, and Latin-1. GetBytes encodes a native string; GetString decodes a byte range. Raw-pointer overloads validate null and signed index/count but cannot validate the caller’s actual buffer capacity because the pointer carries no size.

The output of every decode is the runtime’s UTF-8 std::string. Current GetCharCount therefore reports the number of UTF-8 output bytes, not the UTF-16 count implied by .NET naming. Encoding factories return shared mutable instances: changing a fallback on one affects other callers and can race concurrent conversion. Treat fallback mutation as process-shared configuration until this limitation changes.

Normalization and graphemes

StringNormalizationExtensions::IsNormalized currently always returns true and Normalize returns its input unchanged for every requested normalization form. This is correct for ASCII and insufficient for decomposed or compatibility Unicode sequences. The implementation contains no ICU-style normalization tables.

Similarly, code-point iteration does not solve user-perceived character segmentation. Combining marks, emoji sequences, flags, and variation selectors require grapheme rules. Use an appropriate Unicode library when normalization, collation, casing, display width, or grapheme slicing is a correctness requirement.

Formatting internals

The core formatter parses indexed replacement fields into a small internal argument variant for supported primitive/string types. Integer and floating formatters interpret a bounded specifier set through native stream and numeric formatting. Overloads construct one to three argument arrays and feed the shared parser.

This design is explicit and linkable without CLR reflection, but it cannot discover an arbitrary object’s format provider or enumerate managed interfaces. Adding a new supported argument kind requires a concrete overload, renderer, tests for valid/invalid specifiers, and documentation of culture behavior.

Native interop boundaries

• C APIs: c_str() supplies a null-terminated view for the duration of a bounded call. Embedded NUL bytes still terminate many C APIs early.
• POSIX paths: bytes are passed according to the host filesystem convention; valid UTF-8 is an application policy, not a kernel guarantee.
• Windows wide APIs: convert between UTF-8 and UTF-16 explicitly. Do not cast byte storage to wchar_t*.
• Binary protocols: keep byte vectors/spans distinct from text so invalid encoding is not silently normalized.
• Async calls: retain an owning string or copy the bytes if the native operation outlives the call.

Interning and identity

The current Intern and IsInterned helpers return the string value without maintaining a pool. There is no canonical storage address, identity comparison, or lifetime extension. If a native application needs symbol interning, it must own and synchronize an explicit pool with well-defined invalidation.

Porting decision matrix

Invariants worth testing in a port

• Every stored offset has an explicit unit: byte, UTF-16 unit, scalar, or grapheme.
• Malformed UTF-8 and embedded NUL behavior are tested at external boundaries.
• Case-insensitive lookup uses a comparer and hash with the same reduction.
• No view or C pointer survives mutation or destruction of its owner.
• Normalization-sensitive identifiers use a real normalization policy.
• Format strings and supported argument types are covered by exact tests rather than inferred from .NET.