1. Introduction to Tensors

import numpy as np
import tensorflow as tf

Basic

Tensors are multi-dimensional arrays with a uniform type (called a dtype).
All tensors are immutable like Python numbers and strings: you can never update the contents of a tensor, only create a new one.

# Python number
num = 123
# Tensorflow Tensor
tensor = tf.constant([1, 2])

# Before num address
print("Before num address: ", id(num))
# Before tensor address
print("Before tensor address: ", id(tensor))

print("-"*40)
num += 123
tensor += tf.constant([1, 2])

# After num address
print("After num address: ", id(num))
# After tensor address
print("After tensor address: ", id(tensor))

Before num address:  140723504451664
Before tensor address:  1614066461864
----------------------------------------
After num address:  140723504455600
After tensor address:  1614066461512

Tensor to Numpy

print("1st method: ",np.array(tensor))
print("2nd method: ", tensor.numpy())

1st method:  [2 4]
2nd method:  [2 4]

Basic math of Tensor

a = tf.constant([[1, 2],
                 [3, 4]])
b = tf.constant([[1, 2],
                 [3, 4]])

print("Add: \n", tf.add(a, b))
print("Multiply: \n ", tf.multiply(a, b))
print("Matmul: \n ", tf.matmul(a, b))

c = tf.constant([[4.0, 5.0],
                 [10.0, 1.0]])

print("Max: \n ", tf.reduce_max(c))
print("Argmax: \n ", tf.argmax(c))
print("Softmax: \n ", tf.nn.softmax(c))

Add: 
 tf.Tensor(
[[2 4]
 [6 8]], shape=(2, 2), dtype=int32)
Multiply: 
  tf.Tensor(
[[ 1  4]
 [ 9 16]], shape=(2, 2), dtype=int32)
Matmul: 
  tf.Tensor(
[[ 7 10]
 [15 22]], shape=(2, 2), dtype=int32)
Max: 
  tf.Tensor(10.0, shape=(), dtype=float32)
Argmax: 
  tf.Tensor([1 0], shape=(2,), dtype=int64)
Softmax: 
  tf.Tensor(
[[2.6894143e-01 7.3105854e-01]
 [9.9987662e-01 1.2339458e-04]], shape=(2, 2), dtype=float32)

Shapes

Tensor have shapes. Some vocabulary:

• Shape: The length (number of elements) of each of the axes of a tensor
• Rank: Number of tensor axes. A scalar has rank 0, a vector has rank 1, a matrix is rank 2
• Axis or Dimension: A particular dimension of a tensor
• Size: The total number of items in the tensor, the product shape vector

tensor = tf.zeros([3, 2, 4, 5])

print("Type: ", tensor.dtype)
print("Axes: ", tensor.ndim)
print("Shape: ", tensor.shape)
print("First axis's shape: ", tensor.shape[0])
print("Last axis's shape: ", tensor.shape[-1])
print("Total number of elements: ", tf.size(tensor))

Type:  <dtype: 'float32'>
Axes:  4
Shape:  (3, 2, 4, 5)
First axis's shape:  3
Last axis's shape:  5
Total number of elements:  tf.Tensor(120, shape=(), dtype=int32)

Indexing

Single-axis indexing

• indexes start at 0
• negative indices count backwards from the end
• colons, :, are used for sliced: start:stop:step

tensor = tf.constant([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10])
print("Tensor: ", tensor.numpy())
print("-"*40)
print(f"First: ", tensor[0].numpy())
print("Second: ", tensor[1].numpy())
print("Last: ", tensor[-1].numpy())
print("-"*40)
print("Everything: ", tensor[:].numpy())
print("Before 4: ", tensor[:4].numpy())
print("After 4: ", tensor[4:].numpy())
print("From 2 to 7: ", tensor[2:7].numpy())
print("Even item: ", tensor[::2].numpy())
print("Reversed: ", tensor[::-1].numpy())

Tensor:  [ 0  1  2  3  4  5  6  7  8  9 10]
----------------------------------------
First:  0
Second:  1
Last:  10
----------------------------------------
Everything:  [ 0  1  2  3  4  5  6  7  8  9 10]
Before 4:  [0 1 2 3]
After 4:  [ 4  5  6  7  8  9 10]
From 2 to 7:  [2 3 4 5 6]
Even item:  [ 0  2  4  6  8 10]
Reversed:  [10  9  8  7  6  5  4  3  2  1  0]

Multi-axis indexing

Higher rank tensors are indexied by passing multiple indices.

tensor = tf.constant([[1.0, 2.0, 3.0],
                      [4.0, 5.0, 6.0],
                      [7.0, 8.0, 9.0]])

print("\nTensor: \n ", tensor.numpy())
print("\n1st item in 1st col: ", tensor[1, 1].numpy())
print("\n2nd row: ", tensor[1, :].numpy())
print("\n2nd col: ", tensor[:, 1].numpy())
print("\nLast row: ", tensor[-1, :].numpy())
print("\n1st item in last col: ", tensor[0, -1].numpy())
print("\nSkip the 1st row: \n ", tensor[1:, :].numpy())

Tensor: 
  [[1. 2. 3.]
 [4. 5. 6.]
 [7. 8. 9.]]

1st item in 1st col:  5.0

2nd row:  [4. 5. 6.]

2nd col:  [2. 5. 8.]

Last row:  [7. 8. 9.]

1st item in last col:  3.0

Skip the 1st row: 
  [[4. 5. 6.]
 [7. 8. 9.]]

Maniplulating Shapes

Reshaping a tensor is of great utility. You can reshape a tensor a new shape. The tf.reshape operation is fast and cheap as the underlying data does not need to be duplicated.
The data maintains its layout in memory and a new tensor is created, with the requested shape, pointing to the same data.

tensor = tf.constant([
  [[0, 1, 2, 3, 4],
   [5, 6, 7, 8, 9]],
  [[10, 11, 12, 13, 14],
   [15, 16, 17, 18, 19]],
  [[20, 21, 22, 23, 24],
   [25, 26, 27, 28, 29]]])

print("\nTensor: \n ", tensor)
print("\nReshape tensor by [30, 1]: \n", tf.reshape(tensor, [-1]))
print("\nReshape tensor by [3*2, 5]: \n", tf.reshape(tensor, [3*2, 5]))
print("\nReshape tensor by [3, 2*5]: \n", tf.reshape(tensor, [3, -1]))

Tensor: 
  tf.Tensor(
[[[ 0  1  2  3  4]
  [ 5  6  7  8  9]]

 [[10 11 12 13 14]
  [15 16 17 18 19]]

 [[20 21 22 23 24]
  [25 26 27 28 29]]], shape=(3, 2, 5), dtype=int32)

Reshape tensor by [30, 1]: 
 tf.Tensor(
[ 0  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
 24 25 26 27 28 29], shape=(30,), dtype=int32)

Reshape tensor by [3*2, 5]: 
 tf.Tensor(
[[ 0  1  2  3  4]
 [ 5  6  7  8  9]
 [10 11 12 13 14]
 [15 16 17 18 19]
 [20 21 22 23 24]
 [25 26 27 28 29]], shape=(6, 5), dtype=int32)

Reshape tensor by [3, 2*5]: 
 tf.Tensor(
[[ 0  1  2  3  4  5  6  7  8  9]
 [10 11 12 13 14 15 16 17 18 19]
 [20 21 22 23 24 25 26 27 28 29]], shape=(3, 10), dtype=int32)

Reshaping will "work" for any new shape with the same total number of elements, but it will not do anything useful if you do not respect the order of the axes. Swapping axes in tf.reshape does not work; you need tf.transpose for that.

# You can't reorder axes with reshape
print("\nReshape tensor by [2, 3, 5]: ", tf.reshape(tensor, [2, 3, 5]))

# This is a mess
print("\nReshape tensor by [5, 6]: ", tf.reshape(tensor, [5, 6]))

Reshape tensor by [2, 3, 5]:  tf.Tensor(
[[[ 0  1  2  3  4]
  [ 5  6  7  8  9]
  [10 11 12 13 14]]

 [[15 16 17 18 19]
  [20 21 22 23 24]
  [25 26 27 28 29]]], shape=(2, 3, 5), dtype=int32)

Reshape tensor by [5, 6]:  tf.Tensor(
[[ 0  1  2  3  4  5]
 [ 6  7  8  9 10 11]
 [12 13 14 15 16 17]
 [18 19 20 21 22 23]
 [24 25 26 27 28 29]], shape=(5, 6), dtype=int32)

More on dtypes

To inspect a tf.Tensors data type use the Tensor.dtype property. When creating a tf.Tensor from a Python object you may optionally specify the datatype. If you don't, TensorFlow chooses a datatype that can represent your data. TensorFlow converts Python integers to tf.int32 and Python floating numbers to tf.float32. Otherwise TensorFlow uses the same rules NumPy uses when coverting to arrays.

f64_tensor = tf.constant([2.2, 3.3, 4.4], dtype=tf.float64)
f16_tensor = tf.cast(f64_tensor, dtype=tf.float16)
u8_tensor = tf.cast(f16_tensor, dtype=tf.uint8)
print("float64 tensor :", f64_tensor)
print("float32 tensor :", f16_tensor)
print("uint8 tensor :", u8_tensor)

float64 tensor : tf.Tensor([2.2 3.3 4.4], shape=(3,), dtype=float64)
float32 tensor : tf.Tensor([2.2 3.3 4.4], shape=(3,), dtype=float16)
uint8 tensor : tf.Tensor([2 3 4], shape=(3,), dtype=uint8)

Broadcasting

Under certain conditions, smaller tensors are "stretched" automatically to fit larger tensors when running combined operations on them. Most of the time, broadcasting is both time and space efficient, as the broadcast operation never materializes the expanded tensors in memory.

x = tf.constant([1, 2, 3])
y = tf.constant(2)
z = tf.constant([2, 2, 2])

print("Multiply x and scalar 2: ", tf.multiply(x, 2))
print("Multiply x and y: ", tf.multiply(x, y))
print("Multiply x and z: ", tf.multiply(x, z))

Multiply x and scalar 2:  tf.Tensor([2 4 6], shape=(3,), dtype=int32)
Multiply x and y:  tf.Tensor([2 4 6], shape=(3,), dtype=int32)
Multiply x and z:  tf.Tensor([2 4 6], shape=(3,), dtype=int32)

x = tf.reshape(x, [3, 1])
y = tf.range(1, 5)

print("Multiply x and y: \n", tf.multiply(x, y))

Multiply x and y: 
 tf.Tensor(
[[ 1  2  3  4]
 [ 2  4  6  8]
 [ 3  6  9 12]], shape=(3, 4), dtype=int32)

tf.convert_to_tensor

Most ops, like tf.matmul ad tf.reshape take arguments of class tf.Tensor. However, you'll notice in the above case, Python objects shaped like tensors are accepted.

 

Most, but not all, ops call convert_to_tensor on non-tensor arguments. There is a registry of conversions, and most object classes like NumPy's ndarray, TensorShape, Python lists, and tf.Variable will all convert automatically.

Ragged Tensors

A tensor with variable numbers of elements along some axis is called "ragged". Use tf.ragged.RaggedTensor for ragged data.

list = [
    [0, 1, 2, 3],
    [4, 5],
    [6, 7, 8],
    [9]]

try:
  tensor = tf.constant(list)
except Exception as e:
  print(f"{type(e).__name__}: {e}")

tensor = tf.ragged.constant(list)
print("ragged tensor: ", tensor)
print("ragged tensor shape: ", tensor.shape)

ValueError: Can't convert non-rectangular Python sequence to Tensor.
ragged tensor:  <tf.RaggedTensor [[0, 1, 2, 3], [4, 5], [6, 7, 8], [9]]>
ragged tensor shape:  (4, None)

String Tensors

tf.string is a dtype, which is to say you can represent data as strings (variable-length byte arrays) in tensors.

 

The strings are atomic and cannot be indexed the way Python strings are. The length of the string is not of the axes of the tensor.

s = "abcedef"
print("s[1]: ", s[1])

tensor = tf.constant("Gray wolf")
try:
    print("tensor[1]: ", tensor[0])
except Exception as e:
  print(f"{type(e).__name__}: {e}")

s[1]:  b
InvalidArgumentError: Index out of range using input dim 0; input has only 0 dims [Op:StridedSlice] name: strided_slice/

#split
tensor = tf.constant(["Gray wolf",
                      "Quick brown fox",
                      "Lazy dog"])
print("tensor: ", tensor)
print("tensor[0] split: ", tf.strings.split(tensor[0], sep=" "))
print("tensor split: ", tf.strings.split(tensor))

tensor:  tf.Tensor([b'Gray wolf' b'Quick brown fox' b'Lazy dog'], shape=(3,), dtype=string)
tensor[0] split:  tf.Tensor([b'Gray' b'wolf'], shape=(2,), dtype=string)
tensor split:  <tf.RaggedTensor [[b'Gray', b'wolf'], [b'Quick', b'brown', b'fox'], [b'Lazy', b'dog']]>

#to_number
tensor = tf.constant("1 10 100")
print("tensor to_number", tf.strings.to_number(tf.strings.split(tensor, sep=" ")))

tensor to_number tf.Tensor([  1.  10. 100.], shape=(3,), dtype=float32)

tensor = tf.constant("Duck")

print("Byte char bytes: ", tf.strings.bytes_split(tensor))
print("Bytes: char values", tf.io.decode_raw(tensor, tf.uint8))

Byte char bytes:  tf.Tensor([b'D' b'u' b'c' b'k'], shape=(4,), dtype=string)
Bytes: char values tf.Tensor([ 68 117  99 107], shape=(4,), dtype=uint8)

tensor = tf.constant("안녕 친구야")
unicode_char = tf.strings.unicode_split(tensor, "UTF-8")
unicode_value = tf.strings.unicode_decode(tensor, "UTF-8")

print("\nUnicode string bytes: \n", tensor)
print("\nUnicode char bytes: \n", unicode_char)
print("\nUnicode char values; \n", unicode_value)

Unicode string bytes: 
 tf.Tensor(b'\xec\x95\x88\xeb\x85\x95 \xec\xb9\x9c\xea\xb5\xac\xec\x95\xbc', shape=(), dtype=string)

Unicode char bytes: 
 tf.Tensor(
[b'\xec\x95\x88' b'\xeb\x85\x95' b' ' b'\xec\xb9\x9c' b'\xea\xb5\xac'
 b'\xec\x95\xbc'], shape=(6,), dtype=string)

Unicode char values; 
 tf.Tensor([50504 45397    32 52828 44396 50556], shape=(6,), dtype=int32)

Spare Tensors

TensorFlow supports tf.sparse.SparseTensor and related operations to store sparse data efficiently.

tensor = tf.sparse.SparseTensor(indices=[[0, 0], [1, 2]],
                                values=[1, 2],
                                dense_shape=[3, 4])
print("\nsparse tensor: \n", tensor)
print("\nsparse tensor to dense tensor: \n", tf.sparse.to_dense(tensor))

sparse tensor: 
 SparseTensor(indices=tf.Tensor(
[[0 0]
 [1 2]], shape=(2, 2), dtype=int64), values=tf.Tensor([1 2], shape=(2,), dtype=int32), dense_shape=tf.Tensor([3 4], shape=(2,), dtype=int64))

sparse tensor to dense tensor: 
 tf.Tensor(
[[1 0 0 0]
 [0 0 2 0]
 [0 0 0 0]], shape=(3, 4), dtype=int32)