Installs into .claude/skills of the current project.
Are you the author of Tokenizer Picker?
Add the live security badge to your README. It updates with every re-scan.
[](https://www.skillsdirectory.com/skills/anubhavg-icpl-tokenizer-picker)
---
name: tokenizer-picker
description: Pick a tokenization approach for a new language model or text pipeline. Use when you need help with tokenizer picker.
license: CC-BY-NC-SA-4.0
phase: 5
lesson: 04
metadata:
version: 1.0.0
tags: [nlp, tokenization, embeddings]
---
Given a task and dataset description, you output:
1. Tokenization strategy (word-level, BPE, WordPiece, SentencePiece, byte-level BPE). One-sentence reason.
2. Vocabulary size target. English-only LM: 32k. Multilingual: 64k-100k. Code: 50k-100k.
3. Library call with the exact training command. Name the library (Hugging Face `tokenizers`, `sentencepiece`). Quote arguments.
4. One reproducibility pitfall. Tokenizer-model mismatch is the single most common silent production bug. Name which tokenizer pairs with which pretrained checkpoint and warn against swapping.
Refuse to recommend training a custom tokenizer when the user is fine-tuning a pretrained LLM (the fine-tune must use the pretrained tokenizer). Refuse to recommend word-level tokenization for any production inference path. Flag non-English or multi-script corpora as needing SentencePiece with byte fallback.
