Token view
Live token view: typed text splits into tinted spans and a strip of pixels where new tokens drop in, using a rough tokenizer.
Installation
pnpm dlx shadcn@latest add @lumesec/px-tokenizeFirst timeRegister the @lumesec registry once, or install by URL+
Adds @lumesec to your components.json:
pnpm dlx shadcn@latest registry add @lumesec=https://elements.lumesec.ai/r/{name}.jsonOr skip the setup and install by URL:
npx shadcn@latest add https://elements.lumesec.ai/r/px-tokenize.jsonUsage
import { PxTokenize } from "@/components/lumesec/px-tokenize";
export function Example() {
return (
<PxTokenize />
);
}Behaviour
Type and the text splits into tokens, shown as tinted spans and as a strip of pixels where new tokens drop in. The tokenizer is a rough approximation.
API reference
Events
Events bubble and cross the shadow boundary unless the description says otherwise.
changeReact proponChangeDetail{ tokens: number }Fires on every edit with the token count. It does not fire when the element first connects.
Accessibility
- The textarea has
aria-label="Text to tokenize". - The tinted token spans are
aria-hidden; the pixel strip hasrole="img"with anaria-labelgiving the token count. - The token, character and ratio summary is an
aria-live="polite"region. Token highlighting on hover is pointer-only. - Reduced motion: new tokens appear on the pixel strip in place instead of dropping in, and the summary text swaps without rolling.
Theming
The element reads your shadcn theme tokens through its shadow root, so light and dark follow your theme. The accent comes from --lumesec. See Theming.
This component reads --border, --card, --foreground, --lumesec and --muted-foreground.
To restyle only LumeSec elements, set the matching --ui-* overrides: --ui-accent, --ui-border, --ui-fg, --ui-muted and --ui-surface.
Notes
- The tokenizer is a regular-expression approximation for illustration: words, groups of up to three digits and single symbols, with words longer than six letters split into chunks and common suffixes. Counts will not match a real model tokenizer.
- Text comes only from the built-in textarea, which starts with a sample sentence. The strip's height grows with the number of tokens.
Related
Was this page helpful?