Token counting seam. The default is a dependency-free heuristic (~4 characters per token, the well-known rough GPT ratio). Callers that need exact counts can pass a real tokenizer (e.g. js-tiktoken) via MemoryConfig.tokenCounter.
js-tiktoken
Token counting seam. The default is a dependency-free heuristic (~4 characters per token, the well-known rough GPT ratio). Callers that need exact counts can pass a real tokenizer (e.g.
js-tiktoken) via MemoryConfig.tokenCounter.