Skip to content
Navigation Menu
Toggle navigation
Sign in
Appearance settings
Platform
AI CODE CREATION
GitHub Copilot
Write better code with AI
GitHub Spark
Build and deploy intelligent apps
GitHub Models
Manage and compare prompts
MCP Registry
New
Integrate external tools
DEVELOPER WORKFLOWS
Actions
Automate any workflow
Codespaces
Instant dev environments
Issues
Plan and track work
Code Review
Manage code changes
APPLICATION SECURITY
GitHub Advanced Security
Find and fix vulnerabilities
Code security
Secure your code as you build
Secret protection
Stop leaks before they start
EXPLORE
Why GitHub
Documentation
Blog
Changelog
Marketplace
View all features
Solutions
BY COMPANY SIZE
Enterprises
Small and medium teams
Startups
Nonprofits
BY USE CASE
App Modernization
DevSecOps
DevOps
CI/CD
View all use cases
BY INDUSTRY
Healthcare
Financial services
Manufacturing
Government
View all industries
View all solutions
Resources
EXPLORE BY TOPIC
AI
Software Development
DevOps
Security
View all topics
EXPLORE BY TYPE
Customer stories
Events & webinars
Ebooks & reports
Business insights
GitHub Skills
SUPPORT & SERVICES
Documentation
Customer support
Community forum
Trust center
Partners
View all resources
Open Source
COMMUNITY
GitHub Sponsors
Fund open source developers
PROGRAMS
Security Lab
Maintainer Community
Accelerator
GitHub Stars
Archive Program
REPOSITORIES
Topics
Trending
Collections
Enterprise
ENTERPRISE SOLUTIONS
Enterprise platform
AI-powered developer platform
AVAILABLE ADD-ONS
GitHub Advanced Security
Enterprise-grade security features
Copilot for Business
Enterprise-grade AI features
Premium Support
Enterprise-grade 24/7 support
Pricing
state:open label:"tokenization"
Search code, repositories, users, issues, pull requests...
Search syntax tips
Provide feedback
Saved searches
Use saved searches to filter your results more quickly
Sign in
Sign up
Appearance settings
Resetting focus
You signed in with another tab or window.
Reload
to refresh your session.
You signed out in another tab or window.
Reload
to refresh your session.
You switched accounts on another tab or window.
Reload
to refresh your session.
Dismiss alert
{{ message }}
microsoft
/
vscode
Public
Notifications
You must be signed in to change notification settings
Fork
39.6k
Star
184k
Code
Issues
5k+
Pull requests
1.9k
Actions
Projects
Models
Wiki
Security and quality
23
Insights
Additional navigation options
Code
Issues
Pull requests
Actions
Projects
Models
Wiki
Security and quality
Insights
Terminal window cannot be opened
#313570 ·
fuentco
opened
on Apr 30, 2026
28
Issues
Search Issues
state
:
open
label
:
"tokenization"
state:open label:"tokenization"
Search
Labels
Milestones
New issue
Search results
Open
Closed
Can't hightlight shell code correctly
bug
Issue identified by VS Code Team member as probable bug
Issue identified by VS Code Team member as probable bug
grammar
Syntax highlighting grammar
Syntax highlighting grammar
languages-basic
Basic language support issues
Basic language support issues
tokenization
Text tokenization
Text tokenization
Status: Open.
#301808
In microsoft/vscode;
·
z7z8th
opened
on Mar 15, 2026
Large tooltip when not tokenized on long lines makes code hard to read
bug
Issue identified by VS Code Team member as probable bug
Issue identified by VS Code Team member as probable bug
tokenization
Text tokenization
Text tokenization
Status: Open.
#232298
In microsoft/vscode;
·
themehdev
opened
on Oct 25, 2024
Syntax highlighting sometimes gets out of sync
bug
Issue identified by VS Code Team member as probable bug
Issue identified by VS Code Team member as probable bug
tokenization
Text tokenization
Text tokenization
Status: Open.
#211369
In microsoft/vscode;
·
DanTup
opened
on Apr 25, 2024
·
Backlog
Cannot read properties of undefined (reading 'compileAG')
bug
Issue identified by VS Code Team member as probable bug
Issue identified by VS Code Team member as probable bug
error-telemetry
VS Code - Issues generated by telemetry
VS Code - Issues generated by telemetry
tokenization
Text tokenization
Text tokenization
Status: Open.
#193107
In microsoft/vscode;
·
vscodenpa
opened
on Sep 14, 2023
·
Backlog
Please give extensions access to native AST tokens (At least partially)
feature-request
Request for new features or functionality
Request for new features or functionality
tokenization
Text tokenization
Text tokenization
Status: Open.
#177452
In microsoft/vscode;
·
leodevbro
opened
on Mar 17, 2023
·
Backlog
Incorrect highlighting of tsx code inside Markdown code blocks (bracket token exclusion rules of embedded grammars are not considered)
bracket-pair-colorization
bug
Issue identified by VS Code Team member as probable bug
Issue identified by VS Code Team member as probable bug
tokenization
Text tokenization
Text tokenization
Status: Open.
#165232
In microsoft/vscode;
·
behrangsa
opened
on Nov 2, 2022
·
Backlog
Use the Flyweight pattern to encode tokens
feature-request
Request for new features or functionality
Request for new features or functionality
tokenization
Text tokenization
Text tokenization
under-discussion
Issue is under discussion for relevance, priority, approach
Issue is under discussion for relevance, priority, approach
Status: Open.
#165231
In microsoft/vscode;
·
hediet
opened
on Nov 2, 2022
·
Backlog
Consider supporting embedded language grammars when using semantic tokens
feature-request
Request for new features or functionality
Request for new features or functionality
semantic-tokens
Semantic tokens issues
Semantic tokens issues
tokenization
Text tokenization
Text tokenization
Status: Open.
#163292
In microsoft/vscode;
·
DanTup
opened
on Oct 11, 2022
·
Backlog
WebAssembly is not defined
bug
Issue identified by VS Code Team member as probable bug
Issue identified by VS Code Team member as probable bug
edge
Issues running VSCode in Web on Edge
Issues running VSCode in Web on Edge
error-telemetry
VS Code - Issues generated by telemetry
VS Code - Issues generated by telemetry
grammar
Syntax highlighting grammar
Syntax highlighting grammar
tokenization
Text tokenization
Text tokenization
Status: Open.
#158208
In microsoft/vscode;
·
vscodenpa
opened
on Aug 15, 2022
Reuse tokens when a new line is inserted
feature-request
Request for new features or functionality
Request for new features or functionality
tokenization
Text tokenization
Text tokenization
Status: Open.
#146996
In microsoft/vscode;
·
hediet
opened
on Apr 7, 2022
·
Backlog
Allow setting font weights or semibold/thin/heavy... in fontStyle in tokenColors
feature-request
Request for new features or functionality
Request for new features or functionality
tokenization
Text tokenization
Text tokenization
Status: Open.
#145018
In microsoft/vscode;
·
bboysnick5
opened
on Mar 14, 2022
·
Backlog
getTokenInformationAtPosition
API causes too much traffic and blocks the renderer process
api
bug
Issue identified by VS Code Team member as probable bug
Issue identified by VS Code Team member as probable bug
tokenization
Text tokenization
Text tokenization
under-discussion
Issue is under discussion for relevance, priority, approach
Issue is under discussion for relevance, priority, approach
Status: Open.
#99356
In microsoft/vscode;
·
alexdima
opened
on Jun 4, 2020
·
Backlog
You can’t perform that action at this time.