TINYTORCH PYTHON CODE
import numpy as np import math #Scaled Dot Attention x = Tensor(np.random.rand(4,4)) w_q = Tensor(np.random.rand(4,4)) w_k = Tensor(np.random.rand(4,4)) w_v = Tensor(np.random.rand(4,4)) Q = w_q.matmul(x) K = w_k.matmul(x) V = w_v.matmul(x) box("Q", Q, "1") box("K", K, "22") box("V", V, "3") k_transpose = K.transpose() scores = Q.matmul(k_transpose) scaling_factor = 1/math.sqrt(x.shape[-1]) scaled_scores = scores * scaling_factor softmax = Softmax() softmaxed_scores = softmax.forward(scaled_scores, dim=-1) attn_scores = softmaxed_scores.matmul(V) box("attn_scores", [k_transpose, scores, scaled_scores, softmaxed_scores, attn_scores], "4")
Run
CONSOLE OUTPUT
✕
PDF REF
Load PDF
◀
- / -
▶
-
100%
+
Load a PDF to view it here.
Edit Note
Supports Markdown and LaTeX: use $...$ for inline and $$...$$ for display math
Cancel
Delete
Save