What are Tokens in Python

Python Tokens are the smallest meaningful units in a program. In python, all the statements and instructions in a program are constructed using tokens. The various python tokens includes:-


Keywords are reserved words that have special meaning in python. In the latest version of python, there are about 35 keywords, some of them are, if, else, while, break etc. We cannot use these words as variable name, function name, class name or any other entities in our code.

If we try to use these keywords as our value variable name or any other random purposes we will get an error.

Here is the list of all the python keywords.

ifelsewhileFalse Trueawaitimport
Nonebreak exceptin raiseclassfinally
isreturnand continueforlambdatry
notwith asyncyieldfinallyelifor
Python Keywords


Identifiers are names that you can give to any Variables, function, class etc. As we have discussed from the beginning, python is case-sensitive language, so there are some rules that you need to keep in mind before naming identifiers. Here are the rules of identifier.

  1. An Identifier can only contain Alpha-numeric characters (A-Z, a-z & 0-9) and underscores (_).
  2. Python is also a case-sensitive, so A & a are two different identifiers.
  3. Digits from 0-9 can include in an identifier, but it cannot start with them.
  4. Any special character like (@, $, %) and a blank space are not allowed in an identifier.
  5. An Identifier cannot be any of the Reserved words (python keywords).

Here is an example of some valid and unvalid identifiers.

crus4, crus_4, crus_

#unvalid identifiers
crus@, 4crus, crus 4


These types of tokens are used to perform operations on operands. There are various types of operators in python, which includes: arithmetic operators (+, -, *, / etc) assignment operators ( =, +=, -=, etc) comparison operators ( ==, !=, >, <, etc.) and more.


Literals are actually the values that are used in the code. Let me explain you with this example:


name = "john"
age = 19
Gender = M

In an above code “john19 and M are all Literals. Furthermore “john” is an example of String Literal, 19 is an example of integer literal and M is an example of character literal.

Punctuation Symbols

Punctuation symbols are the symbols or characters that serve specific purposes in the python syntax. Some of the examples of punctuators are (‘(‘ and ‘)‘), commas (,), colons (‘:‘), semicolons (;), periods (.), and brackets ([, ]).

Share This Post!

What are Tokens in Python