actrone_memory.tokens module¶
Token counting for the context budget, kept in step with the TypeScript library.
The default is a dependency-free heuristic, about 4 characters per token (the well-known rough GPT
ratio), identical to the TypeScript heuristicTokenCounter, so both libraries allocate a budget
the same way and neither touches the network. Exact cl100k_base counts are opt-in through
MemoryConfig(token_counter="tiktoken") and the [tiktoken] extra.
- actrone_memory.tokens.TokenCounter¶
Returns how many tokens a text consumes.
- actrone_memory.tokens.heuristic_token_counter(text)[source]¶
Estimate tokens as
ceil(len(text) / 4).Never returns less than 1 for non-empty text, so a turn or memory always consumes some budget.
- actrone_memory.tokens.tiktoken_token_counter()[source]¶
Return an exact
cl100k_basecounter.The encoding is loaded here, once, rather than on every call. tiktoken downloads the encoding file on first use unless
TIKTOKEN_CACHE_DIRalready holds it, so this is the one opt-in path in the library that may reach the network.- Raises:
ConfigurationError – If tiktoken is not installed or the encoding cannot be loaded.
- Return type: