Tokens or Lexical Unit in Python


☞The smallest individual unit in a program is known as token.

☞Tokens used in python are : Keywords, Identifiers, Literals, Punctuators, and Operators


Identifiers:-

☞Identifiers are programmer-defined names given to the various program elements such as variables, functions, objects, classes, lists, dictionaries, etc.

Examples of legal identifiers: age, _value, __1_value

Examples of illegal identifiers: 123abc, -salary, hello.file

Rules for naming identifiers :-

➺It may contain digits, letters and underscore.

➺It must begin with a letter or underscore but not a digit.

➺Python is case sensitive.


Literals or Constants:-

☞The data items which never change their value throughout the program run.

☞There are several kinds of literals : Integer literals, Floating or Real literals, Complex literals, String literals, Boolean literals, Special literal [None]


Integer literals :

➺They are whole numbers without any fractional part.

➺An integer literal must have at least one digit and must not contain any decimal point.

➺It may contain either + or - sign. A number with no sign is assumed as positive.

➺There are three types of integer literals

  1. Decimal (base 10):-starts with a digit other than 0. E.g. 24
  2. Octal (base 8):-starts with a digit 0o (digit zero followed by letter o) but can’t contain digits 8,9. E.g. 0o30
  3. Hexadecimal (base 16):-starts with a 0X or 0x but can’t contain letter G-Z. E.g. 0x18

Floating or Real literals:

➺Numbers which are having the fractional part are referred to as floating literals or real literals.

➺It may be a positive or negative number.

➺A number with no sign is assumed to be a positive number.

➺Example : 2.0, 17.5, -0.00256

➺A real literal in exponent form consists of two parts: mantissa and exponent

➺Example : 0.147 x 108 = 0.147E08

➺Part appearing before E is mantissa and after E is the exponent.


Complex Literals:

➺It is of the form a + bj, where a and b are floating or real j represents √(-1) which is an imaginary number.

➺a is the real part and b is imaginary


String Literals:

➺It is a sequence of characters surrounded by single or double-quotes.

➺Example : ‘a’, ‘23’, “abc”, “23”, ‘1-h-4j-i’, '''abc''', """abd"""


Boolean Literal:

➺It is True or False.


Special Literal [None]:

➺It is used to indicate an absence of value.

➺It is also used to indicate the end of lists in Python.

➺The None value in Python means “There is no useful information” or “There’s nothing here”.

NOTE :- Python can also store literal collections, in the form of tuples and lists, etc.

Keywords :-

☞Keywords are reserved words that convey a special meaning to the interpreter.

☞There are 33 keywords in Python 3.

☞['False', 'None', 'True', 'and', 'as', 'assert', 'break', 'class', 'continue', 'def', 'del', 'elif', 'else', 'except', 'finally', 'for', 'from', 'global', 'if', 'import', 'in', 'is', 'lambda', 'nonlocal', 'not', 'or', 'pass', 'raise', 'return', 'try', 'while', 'with', 'yield']

Note : 'False', 'None', 'True' starts with capital letter.

Punctuators :-

☞Punctuators are symbols that are used in programming languages to organize sentence structures, and indicate the rhythm and emphasis of expressions, statements, and program structure.

☞The following characters are used as punctuators:

[ ] Brackets These indicates single and multidimensional array subscripts
() Parenthesis These indicate function calls and function parameters.
{ } Braces Indicate the start and end of compound statements.
; Semicolon This is a statement terminator.
, Comma It is used as a separator.
: Colon It indicates a labeled statement
= Equal to It is used as an assigning operator.

Operators:-

☞An operator is a symbol or character which triggers some operation (computation) on its operands.

☞Types of operators:- Arithmetic Operators, Relational Operators, Logical Operators, Bitwise Operators, Assignment Operators, Identity Operators, Membership Operators