What are Tokens in Python

---

Python Tokens are the smallest meaningful units in a program. In python, all the statements and instructions in a program are constructed using tokens. The various python tokens includes:-

Keywords

Keywords are reserved words that have special meaning in python. In the latest version of python, there are about 35 keywords, some of them are, if, else, while, break etc. We cannot use these words as variable name, function name, class name or any other entities in our code.

If we try to use these keywords as our value variable name or any other random purposes we will get an error.

Here is the list of all the python keywords.

if	else	while	False	True	await	import
None	break	except	in	raise	class	finally
is	return	and	continue	for	lambda	try
as	def	from	nonlocal	assert	del	global
not	with	async	yield	finally	elif	or

Identifiers

Identifiers are names that you can give to any Variables, function, class etc. As we have discussed from the beginning, python is case-sensitive language, so there are some rules that you need to keep in mind before naming identifiers. Here are the rules of identifier.

1. An Identifier can only contain Alpha-numeric characters (A-Z, a-z & 0-9) and underscores (_).
2. Python is also a case-sensitive, so A & a are two different identifiers.
3. Digits from 0-9 can include in an identifier, but it cannot start with them.
4. Any special character like (@, $, %) and a blank space are not allowed in an identifier.
5. An Identifier cannot be any of the Reserved words (python keywords).

Here is an example of some valid and unvalid identifiers.

#valid_identifiers
crus4, crus_4, crus_

#unvalid identifiers
crus@, 4crus, crus 4

Operators

These types of tokens are used to perform operations on operands. There are various types of operators in python, which includes: arithmetic operators (+, -, *, / etc) assignment operators ( =, +=, -=, etc) comparison operators ( ==, !=, >, <, etc.) and more.

Literals

Literals are actually the values that are used in the code. Let me explain you with this example:

Example

name = "john"
age = 19
Gender = M

In an above code “john” 19 and M are all Literals. Furthermore “john” is an example of String Literal, 19 is an example of integer literal and M is an example of character literal.

Punctuation Symbols

Punctuation symbols are the symbols or characters that serve specific purposes in the python syntax. Some of the examples of punctuators are (‘(‘ and ‘)‘), commas (,), colons (‘:‘), semicolons (;), periods (.), and brackets ([, ]).

---

Share This Post!