actrone_memory.tokens module

Token counting for the context budget, kept in step with the TypeScript library.

The default is a dependency-free heuristic, about 4 characters per token (the well-known rough GPT ratio), identical to the TypeScript heuristicTokenCounter, so both libraries allocate a budget the same way and neither touches the network. Exact cl100k_base counts are opt-in through MemoryConfig(token_counter="tiktoken") and the [tiktoken] extra.

actrone_memory.tokens.TokenCounter

Returns how many tokens a text consumes.

alias of Callable[[str], int]

actrone_memory.tokens.heuristic_token_counter(text)[source]

Estimate tokens as ceil(len(text) / 4).

Never returns less than 1 for non-empty text, so a turn or memory always consumes some budget.

Parameters:

text (str)

Return type:

int

actrone_memory.tokens.tiktoken_token_counter()[source]

Return an exact cl100k_base counter.

The encoding is loaded here, once, rather than on every call. tiktoken downloads the encoding file on first use unless TIKTOKEN_CACHE_DIR already holds it, so this is the one opt-in path in the library that may reach the network.

Raises:

ConfigurationError – If tiktoken is not installed or the encoding cannot be loaded.

Return type:

Callable[[str], int]

actrone_memory.tokens.build_token_counter(kind)[source]

Return the counter MemoryConfig.token_counter names.

Raises:

ConfigurationError – If kind is “tiktoken” and tiktoken cannot be loaded.

Parameters:

kind (Literal['heuristic', 'tiktoken'])

Return type:

Callable[[str], int]