Machine learning phases of matter: the Ising model
Goal. Reproduce the Ising-model results of J. Carrasquilla and R. G. Melko, Machine learning phases of matter, arXiv:1605.01735 (Nature Physics 13, 431 (2017)).
A neural network is shown raw spin configurations and is told only whether each one was sampled below or above $T_c$. Nothing about the Hamiltonian, the energy, or the magnetization is given. We ask three questions:
- Can it classify configurations it has never seen?
- What did it learn?
- Can it find $T_c$ of a model it was not trained on?
| Step | Content |
|---|---|
| 1 | Monte Carlo: generate spin configurations |
| 2 | Data set: configurations and labels |
| 3 | Feed-forward neural network (FNN) in NumPy |
| 4 | Training step by step, validation, and test |
| 5 | Inside the network: it learned the magnetization |
| 6 | Toy model with three hidden neurons |
| 7 | Finite-size scaling and $T_c$ |
| 8 | Triangular lattice: a model the network has never seen |
Lines marked # TRY are meant to be changed.
import os, time
import numpy as np
import matplotlib.pyplot as plt
from numba import njit, prange
Tc_square = 2 / np.log(1 + np.sqrt(2)) # exact (Onsager), in units of J/k_B
Tc_triangular = 4 / np.log(3) # exact, triangular lattice
BLUE, RED, ORANGE = "#1f4e9c", "#c0392b", "#f39c12" # low-T, high-T, exact Tc
plt.rcParams.update({"figure.dpi": 110, "font.size": 11,
"axes.spines.top": False, "axes.spines.right": False})
print(f"Tc (square) = {Tc_square:.4f}")
print(f"Tc (triangular) = {Tc_triangular:.4f}")
1. Monte Carlo: spin configurations
$$H = -J\sum_{\langle ij\rangle}\sigma_i\sigma_j,\qquad \sigma_i=\pm1,\qquad J=k_B=1 .$$
The sampler is the single-spin-flip Metropolis algorithm from the Ising notebooks. A spin configuration is an $L\times L$ matrix of $\pm1$ with periodic boundaries.
The option triangular=True adds the two diagonal neighbors $(i+1,j+1)$ and $(i-1,j-1)$.
Each spin then has six neighbors: this is the triangular lattice stored in the same
$L\times L$ matrix. We need it only in Section 8.
@njit
def sweep(spins, T, triangular):
# One Metropolis sweep: L*L single-spin-flip attempts.
L = spins.shape[0]
for _ in range(L * L):
i = np.random.randint(L)
j = np.random.randint(L)
neighbors = (spins[(i - 1) % L, j] + spins[(i + 1) % L, j]
+ spins[i, (j - 1) % L] + spins[i, (j + 1) % L])
if triangular:
neighbors += spins[(i + 1) % L, (j + 1) % L] + spins[(i - 1) % L, (j - 1) % L]
dE = 2 * spins[i, j] * neighbors
if dE <= 0 or np.random.random() < np.exp(-dE / T):
spins[i, j] = -spins[i, j]
@njit(parallel=True)
def generate(L, temperatures, n_chains, n_per_chain, n_equil, stride, triangular, seed):
# Returns configurations with shape (n_T, n_chains * n_per_chain, L, L).
# Every chain is independent. Half of them start all-up, half all-down.
n_T = len(temperatures)
out = np.empty((n_T, n_chains * n_per_chain, L, L), dtype=np.int8)
for idx in prange(n_T * n_chains): # chains run in parallel
t, c = idx // n_chains, idx % n_chains
np.random.seed(seed + idx)
T = temperatures[t]
spins = np.full((L, L), 1 if c % 2 == 0 else -1, dtype=np.int8)
for _ in range(n_equil): # equilibrate
sweep(spins, T, triangular)
for k in range(n_per_chain): # save one configuration every `stride` sweeps
for _ in range(stride):
sweep(spins, T, triangular)
out[t, c * n_per_chain + k] = spins
return out
What the configurations look like
Below $T_c$ the spins order (mostly one color). Above $T_c$ they are disordered. Near $T_c$ there are domains of all sizes, and the two phases are hard to tell apart by eye.
L_show = 40 # TRY: 20, 60
T_show = np.array([1.0, 1.8, 2.2, 2.4, 2.8, 3.5]) # TRY: other temperatures
gallery = generate(L_show, T_show, 2, 1, 2000, 1, False, 7) # 2 chains: one starts up, one down
fig, axes = plt.subplots(2, len(T_show), figsize=(13, 4.6))
for col, T in enumerate(T_show):
for row in range(2):
ax = axes[row, col]
ax.imshow(gallery[col, row], cmap="gray", vmin=-1, vmax=1)
ax.set_xticks([]); ax.set_yticks([])
for side in ax.spines.values():
side.set_visible(True); side.set_linewidth(3)
side.set_color(BLUE if T < Tc_square else RED)
axes[0, col].set_title(f"T = {T:.1f}", color=BLUE if T < Tc_square else RED)
fig.suptitle(f"Square lattice, L = {L_show}. Blue frame: T < Tc (label 'low T'). "
f"Red frame: T > Tc (label 'high T'). Tc = {Tc_square:.3f}", y=1.0)
plt.tight_layout(); plt.show()
Generate the data
Sizes $L = 10, 20, 30, 40, 60$ as in the paper. For each size and each of 40 temperatures:
- 250 configurations for training/validation (square lattice),
- 250 configurations from separate chains for testing (square lattice),
- 250 configurations of the triangular lattice (used only in Section 8).
The first run takes about one to a few minutes, depending on the number of CPU cores.
The result is saved to ising_ml_data.npz and reused afterwards.
sizes = [10, 20, 30, 40, 60]
temps = np.linspace(1.0, 3.5, 40) # square lattice
temps_tri = np.linspace(2.5, 5.0, 40) # triangular lattice
n_chains, n_per_chain = 10, 25 # 10 chains x 25 = 250 configurations per temperature
n_equil, stride = 2000, 40 # sweeps
data_file = "ising_ml_data.npz"
recompute = False # Reuse saved data; set True to regenerate after changing sampling parameters.
if os.path.exists(data_file) and not recompute:
data = dict(np.load(data_file))
print("Loaded", data_file)
else:
data = {}
for L in sizes:
t0 = time.time()
args = (n_chains, n_per_chain, n_equil, stride)
data[f"train_{L}"] = generate(L, temps, *args, False, 1_000_000 + 1000 * L)
data[f"test_{L}"] = generate(L, temps, *args, False, 2_000_000 + 1000 * L)
data[f"tri_{L}"] = generate(L, temps_tri, *args, True, 3_000_000 + 1000 * L)
print(f"L = {L:2d} done in {time.time() - t0:5.1f} s", flush=True)
np.savez_compressed(data_file, **data)
print("train_30 shape:", data["train_30"].shape, "= (temperatures, samples, L, L)")
The physicist's answer: the order parameter
A physicist would compute the magnetization per spin, $m = \frac1N\sum_i\sigma_i$. Left: $m$ of every test configuration ($L=30$). Right: $\langle|m|\rangle$ for all sizes, with Onsager's exact result for $L\to\infty$.
Keep this picture in mind. The network is not given $m$.
fig, (a, b) = plt.subplots(1, 2, figsize=(11, 3.8))
conf = data["test_30"]
m = conf.reshape(len(temps), -1, 30 * 30).mean(axis=2) # (n_T, n_samples)
jitter = np.random.default_rng(0).uniform(-0.02, 0.02, m.shape)
a.scatter(temps[:, None] + jitter, m, s=2, alpha=0.25, color="k")
a.axvline(Tc_square, color=ORANGE, lw=2, label="exact $T_c$")
a.set_xlabel("T"); a.set_ylabel("m of one configuration"); a.set_title("L = 30"); a.legend()
for L, color in zip(sizes, plt.cm.viridis(np.linspace(0, 0.9, len(sizes)))):
conf = data[f"test_{L}"]
abs_m = np.abs(conf.reshape(len(temps), -1, L * L).mean(axis=2)).mean(axis=1)
b.plot(temps, abs_m, "o-", ms=3, color=color, label=f"L = {L}")
T_fine = np.linspace(1.0, Tc_square - 1e-6, 400)
b.plot(T_fine, (1 - np.sinh(2 / T_fine) ** -4) ** 0.125, "k--", label="Onsager")
b.axvline(Tc_square, color=ORANGE, lw=2)
b.set_xlabel("T"); b.set_ylabel(r"$\langle |m| \rangle$"); b.legend(fontsize=9)
plt.tight_layout(); plt.show()
2. Data set: inputs and labels
- Input $x$: one configuration flattened to a vector of $N=L^2$ numbers $\pm1$.
- Label $y$:
[1, 0]if it was sampled at $T<T_c$ (low T),[0, 1]if $T>T_c$ (high T).
The label uses the known $T_c$ of the square lattice. That is the only physics input.
def make_dataset(conf, temperatures, Tc):
n_T, n_samples, L, _ = conf.shape
X = conf.reshape(n_T * n_samples, L * L).astype(np.float32) # one row = one configuration
# X
# for the first temperature point
# X[0,:] = [+1, -1, +1, ..., -1] # length of each row = L*L
# X[1,:] = [-1, -1, -1, ..., -1]
# ...
# X[n_samples-1,:] = [configuration of spins for each sample]
# X[n_samples,:] = [configuration of spins for the second temperature point]
# ..
T = np.repeat(temperatures, n_samples) # temperature of each row
Y = np.column_stack([T < Tc, T > Tc]).astype(np.float32) # [low T, high T]
# we want to assign the label for each x
# Y[0,:] = [1, 0] # low T
# Y[1,:] = [1, 0] # low T
# ...
# Y[n_samples-1,:] = [1, 0] # low T
# if your second temperature point is above the Tc
# Y[n_samples,:] = [0, 1] # high T
# ...
# Y[n_samples+n_samples-1,:] = [0, 1] # high T
return X, Y, T
X, Y, T = make_dataset(data["train_30"], temps, Tc_square)
print("X:", X.shape, " Y:", Y.shape)
k = 5010 # TRY: another row (250 rows per temperature)
fig, (a, b) = plt.subplots(2, 1, figsize=(11, 2.6), height_ratios=[4, 1])
a.axis("off")
ins = a.inset_axes([0.0, 0.0, 0.2, 1.0]); ins.imshow(X[k].reshape(30, 30), cmap="gray", vmin=-1, vmax=1); ins.set_xticks([]); ins.set_yticks([])
for side in ins.spines.values(): side.set_visible(True)
a.text(0.24, 0.5, f"T = {T[k]:.2f}\n\nflatten → x (900 numbers)\n\nlabel y = {Y[k].astype(int)} "
f"({'low T' if Y[k, 0] else 'high T'})", va="center", fontsize=12)
b.imshow(X[k][None, :], cmap="gray", aspect="auto", vmin=-1, vmax=1); b.set_yticks([])
for side in b.spines.values(): side.set_visible(True); b.set_xlabel("input index (row k of X)")
plt.show()
3. Training, in pictures
Predict → compare with the answer → adjust the weights → repeat. The spin pictures and labels stay fixed.
Run the setup cell once; its full implementation is optional reading.
Network setup: full NumPy code (optional reading; run once in the notebook)
def initialize_net(n_in, n_hidden=100, seed=0):
rng = np.random.default_rng(seed)
return {"W1": rng.normal(0, 1 / np.sqrt(n_in), (n_in, n_hidden)).astype(np.float32),
"b1": np.zeros(n_hidden, np.float32),
"W2": rng.normal(0, 1 / np.sqrt(n_hidden), (n_hidden, 2)).astype(np.float32),
"b2": np.zeros(2, np.float32)}
def sigmoid(z):
return np.exp(-np.logaddexp(0, -z)) # 1 / (1 + exp(-z)), without overflow
def log_softmax(z):
shifted = z - z.max(axis=1, keepdims=True) # same probabilities, safer exponentials
return shifted - np.log(np.exp(shifted).sum(axis=1, keepdims=True))
def softmax(z):
return np.exp(log_softmax(z))
def forward(net, X):
h = sigmoid(X @ net["W1"] + net["b1"])
out = softmax(h @ net["W2"] + net["b2"])
return h, out
def accuracy(net, X, Y):
# argmax chooses the column with the larger probability: 0 = low T, 1 = high T.
return np.mean(forward(net, X)[1].argmax(axis=1) == Y.argmax(axis=1))
def loss_parts(net, X, Y, lam=0.0):
h = sigmoid(X @ net["W1"] + net["b1"])
log_p = log_softmax(h @ net["W2"] + net["b2"])
ce = -np.mean(np.sum(Y * log_p, axis=1))
penalty = 0.5 * lam * (np.sum(net["W1"] ** 2) + np.sum(net["W2"] ** 2))
return float(ce), float(ce + penalty)
def gradients(net, x, y, lam=0.0):
h, out = forward(net, x) # 1. Predict with current parameters.
d2 = (out - y) / len(x) # 2. d(loss)/d(output logits), (B, 2)
d1 = (d2 @ net["W2"].T) * h * (1 - h) # 3. Chain rule, (B, n_hidden)
return {"W2": h.T @ d2 + lam * net["W2"], # Sum contributions from all rows.
"b2": d2.sum(axis=0),
"W1": x.T @ d1 + lam * net["W1"],
"b1": d1.sum(axis=0)}
def train(X, Y, X_val, Y_val, n_hidden=100, epochs=60,
batch=100, lr=1e-3, lam=1e-2, seed=0):
rng = np.random.default_rng(seed)
net = initialize_net(X.shape[1], n_hidden, seed)
m = {k: np.zeros_like(w) for k, w in net.items()}
v = {k: np.zeros_like(w) for k, w in net.items()}
history = {k: [] for k in ["train_ce", "val_ce", "objective", "train_acc", "val_acc"]}
step = 0
def record():
# Evaluation only: these calls never change parameters.
ce, objective = loss_parts(net, X, Y, lam)
history["train_ce"].append(ce)
history["val_ce"].append(loss_parts(net, X_val, Y_val)[0])
history["objective"].append(objective)
history["train_acc"].append(accuracy(net, X, Y))
history["val_acc"].append(accuracy(net, X_val, Y_val))
record() # Epoch 0: random, untrained network.
for epoch in range(epochs):
order = rng.permutation(len(X))
for start in range(0, len(X), batch):
rows = order[start:start + batch]
grad = gradients(net, X[rows], Y[rows], lam) # Training data only.
step += 1
for k in net:
m[k] = 0.9 * m[k] + 0.1 * grad[k]
v[k] = 0.999 * v[k] + 0.001 * grad[k] ** 2
m_hat = m[k] / (1 - 0.9 ** step)
v_hat = v[k] / (1 - 0.999 ** step)
net[k] -= lr * m_hat / (np.sqrt(v_hat) + 1e-8)
record()
return net, {k: np.array(values) for k, values in history.items()}
def train_all_sizes(epochs=60, batch=100, lr=1e-3, **settings):
nets, history, test, validation = {}, {}, {}, {}
for L in sizes:
t0 = time.time()
conf = data[f"train_{L}"]
# generate() stores each chain's samples contiguously along axis 1.
cut = (n_chains - 2) * n_per_chain
X, Y, _ = make_dataset(conf[:, :cut], temps, Tc_square)
X_val, Y_val, _ = make_dataset(conf[:, cut:], temps, Tc_square)
validation[L] = (X_val, Y_val)
test[L] = make_dataset(data[f"test_{L}"], temps, Tc_square)
nets[L], history[L] = train(X, Y, X_val, Y_val, epochs=epochs, batch=batch, lr=lr, **settings)
test_acc = accuracy(nets[L], test[L][0], test[L][1]) # Final evaluation, no update.
print(f"L = {L:2d}: train {history[L]['train_acc'][-1]:.3f}, "
f"validation {history[L]['val_acc'][-1]:.3f}, test {test_acc:.3f}"
f" ({time.time() - t0:.1f} s)")
return nets, history, test, validation
def plot_training_history(history, L_curve=30):
curves = history[L_curve]
epoch_axis = np.arange(len(curves["train_ce"])) # Includes epoch 0.
fig, (a, b) = plt.subplots(1, 2, figsize=(11, 3.8))
for prefix, label, style, color in [("train", "training", "-", BLUE),
("val", "validation", "--", RED)]:
a.plot(epoch_axis, curves[f"{prefix}_ce"], style, color=color, label=label)
b.plot(epoch_axis, curves[f"{prefix}_acc"], style, color=color, label=label)
a.set_xlabel("epoch"); a.set_ylabel("cross-entropy (without L2)"); a.legend()
b.set_xlabel("epoch"); b.set_ylabel("accuracy"); b.set_ylim(0, 1.02); b.legend()
fig.suptitle(f"L = {L_curve}: learning from random initialization")
plt.tight_layout(); plt.show()
# Plotting helpers are included in the notebook's optional setup cell.
from matplotlib.patches import Rectangle, FancyArrowPatch
def diagram_canvas(width=12, height=4):
fig, ax = plt.subplots(figsize=(width, height))
ax.set(xlim=(0, 12), ylim=(0, height))
ax.axis('off')
return fig, ax
def diagram_arrow(ax, start, end, color='black', style='-', rad=0):
arrow = FancyArrowPatch(start, end, arrowstyle='-|>', mutation_scale=18,
linewidth=1.7, color=color, linestyle=style,
connectionstyle=f'arc3,rad={rad}')
ax.add_patch(arrow)
def plot_training_loop(spins, probabilities, label):
fig, ax = diagram_canvas(height=4.2)
ax.text(0.2, 3.83, '1 INPUT', fontsize=16, weight='bold')
ax.imshow(spins, extent=(0.25, 1.8, 1.65, 3.2), cmap='gray', vmin=-1, vmax=1)
ax.add_patch(Rectangle((0.25, 1.65), 1.55, 1.55, fill=False, edgecolor='black', linewidth=1.2))
for g in range(1, spins.shape[0]):
pos = 1.55 * g / spins.shape[0]
ax.plot([0.25 + pos, 0.25 + pos], [1.65, 3.2], color='0.65', lw=0.4)
ax.plot([0.25, 1.8], [1.65 + pos, 1.65 + pos], color='0.65', lw=0.4)
ax.text(1.0, 1.25, 'One spin picture', ha='center', fontsize=14)
diagram_arrow(ax, (1.95, 2.45), (3.0, 2.45))
ax.text(3.25, 3.83, '2 PREDICT', fontsize=16, weight='bold')
ax.add_patch(Rectangle((3.1, 1.65), 2.0, 1.55, fill=False, linewidth=1.5))
ax.text(4.1, 2.68, 'Network', ha='center', fontsize=18)
ax.text(4.1, 2.08, 'weights + biases', ha='center', fontsize=13)
diagram_arrow(ax, (5.25, 2.45), (6.0, 2.45))
ax.text(6.2, 3.83, '3 COMPARE', fontsize=16, weight='bold')
for i, (name, p, color) in enumerate(zip(['Low T', 'High T'], probabilities, [BLUE, RED])):
y = 2.88 - 0.78 * i
ax.text(6.15, y, name, fontsize=13, va='center')
ax.add_patch(Rectangle((7.05, y - 0.18), 2.1, 0.36, fill=False, linewidth=0.8))
ax.add_patch(Rectangle((7.05, y - 0.18), 2.1 * p, 0.36, color=color))
ax.text(9.3, y, f'{p:.2f}', fontsize=15, va='center')
correct = int(np.argmax(label))
loss = -np.log(probabilities[correct])
ax.text(10.7, 3.0, 'Answer', fontsize=13, ha='center')
ax.text(10.7, 2.5, ['Low T', 'High T'][correct], fontsize=17, ha='center', weight='bold')
ax.text(8.4, 1.38, f'Loss = {loss:.3f}', fontsize=16, ha='center')
# Feedback reaches the parameters; the picture and label are unchanged.
ax.plot([8.4, 8.4, 4.1], [1.1, 0.55, 0.55], color='#8B0029', lw=1.7)
diagram_arrow(ax, (4.1, 0.55), (4.1, 1.6), color='#8B0029')
ax.text(6.3, 0.08, '4 Compute gradients, then update weights', ha='center',
fontsize=14, color='#8B0029')
ax.set_aspect('auto')
plt.tight_layout(); plt.show()
def plot_loss_curve(p_correct):
with plt.rc_context({'font.family': 'serif', 'font.size': 13}):
fig, ax = plt.subplots(figsize=(9.6, 3.8))
p = np.linspace(0.025, 1.0, 300)
ax.plot(p, -np.log(p), color='black', lw=2)
points = [(0.1, 'Confident, wrong', (52, 20)),
(0.5, 'Uncertain', (-25, 50)),
(0.9, 'Confident, correct', (-15, 55))]
for x, label, offset in points:
ax.scatter(x, -np.log(x), s=55, color=BLUE, zorder=3)
ax.annotate(f'{label}\np = {x:.1f}, loss = {-np.log(x):.2f}',
(x, -np.log(x)), xytext=offset, textcoords='offset points',
ha='center', fontsize=12,
arrowprops={'arrowstyle': '-', 'color': 'black'})
ax.scatter(p_correct, -np.log(p_correct), marker='*', s=190, color='#8B0029',
edgecolors='black', linewidths=0.5, zorder=5,
label=f'TRY value: p = {p_correct:g}')
ax.set(xlim=(0, 1.02), ylim=(-0.12, 4.0),
xlabel='Probability assigned to the correct answer', ylabel='Loss = −log(p)')
ax.text(0.98, 0.94, 'Lower is better', transform=ax.transAxes,
ha='right', va='top', fontsize=14)
ax.legend(loc='lower left', fontsize=11, frameon=False)
plt.tight_layout(); plt.show()
def plot_score_update(before, after, gradient, learning_rate):
with plt.rc_context({'font.family': 'serif', 'font.size': 13}):
fig, ax = plt.subplots(figsize=(10, 4.3))
left, right = min(-1.0, before - 0.5), max(1.8, after + 0.7)
s = np.linspace(left, right, 300)
ax.plot(s, np.logaddexp(0, -s), 'k-', lw=2, label='Loss for a low-T example')
y0, y1 = np.logaddexp(0, -before), np.logaddexp(0, -after)
tangent = np.linspace(before - 0.45, before + 0.85, 40)
ax.plot(tangent, y0 + gradient * (tangent - before), 'k--', lw=1.3,
label=f'Local slope = gradient = {gradient:.2f}')
ax.scatter([before, after], [y0, y1], s=70, c=[BLUE, '#8B0029'], zorder=4)
ax.annotate(f'Before\np(low T) = {sigmoid(before):.3f}\nloss = {y0:.3f}',
(before, y0), xytext=(-100, -15), textcoords='offset points',
ha='center', fontsize=12, arrowprops={'arrowstyle': '-', 'color': BLUE})
ax.annotate(f'After\np(low T) = {sigmoid(after):.3f}\nloss = {y1:.3f}',
(after, y1), xytext=(90, 40), textcoords='offset points',
ha='center', fontsize=12, arrowprops={'arrowstyle': '-', 'color': '#8B0029'})
ax.annotate('', (after, 0.07), (before, 0.07),
arrowprops={'arrowstyle': '-|>', 'color': '#8B0029', 'lw': 2})
ax.set(xlim=(left, right), ylim=(-0.04, max(1.5, float(y0) + 0.3)),
xlabel='Adjustable score', ylabel='Loss')
ax.legend(loc='upper right', frameon=False, fontsize=11)
fig.text(0.52, 0.02,
f'new score = old score − learning rate × gradient'
f' {before:g} − {learning_rate:g} × ({gradient:g}) = {after:g}',
ha='center', fontsize=12)
fig.tight_layout(rect=(0, 0.08, 1, 1)); plt.show()
def plot_repeated_updates(losses, before, after, labels):
with plt.rc_context({'font.family': 'serif', 'font.size': 13}):
fig, (a, b) = plt.subplots(1, 2, figsize=(11, 3.8))
steps = np.arange(len(losses))
a.plot(steps, losses, 'o-', color='black', ms=5)
a.set(xlabel='Number of updates', ylabel='Average loss on 8 pictures')
a.text(0.95, 0.93, f'{losses[0]:.3f} → {losses[-1]:.3f}',
transform=a.transAxes, ha='right', va='top', fontsize=14)
picks = [0, len(labels) - 1]
classes = labels.argmax(axis=1)[picks]
p0 = before[picks, classes]; p1 = after[picks, classes]
x = np.arange(2)
bars0 = b.bar(x - 0.18, p0, 0.34, label='Before', color='white', edgecolor='black')
bars1 = b.bar(x + 0.18, p1, 0.34, label='After', color='#8B0029', edgecolor='black')
b.bar_label(bars0, fmt='%.2f', padding=4, fontsize=11)
b.bar_label(bars1, fmt='%.2f', padding=4, fontsize=11)
b.set(xticks=x, xticklabels=['One low-T picture', 'One high-T picture'],
ylim=(0, 1.08), ylabel='Probability of the correct answer')
b.legend(loc='upper left', frameon=False, fontsize=11)
plt.tight_layout(); plt.show()
def plot_batches(X, Y):
fig, ax = diagram_canvas(height=3.7)
ax.text(6, 3.35, 'Example: 12 pictures, batch size 4', ha='center', fontsize=18)
rows = np.random.default_rng(5).choice(len(X), 12, replace=False)
for batch_index in range(3):
left = 0.28 + 4.0 * batch_index
ax.text(left + 1.6, 2.87, f'Batch {batch_index + 1}', ha='center', fontsize=14)
for j in range(4):
k = rows[4 * batch_index + j]
x = left + j * 0.85
ax.imshow(X[k].reshape(10, 10), cmap='gray', vmin=-1, vmax=1,
extent=(x, x + 0.72, 1.74, 2.46))
ax.add_patch(Rectangle((x, 1.74), 0.72, 0.72, fill=False,
edgecolor='black', linewidth=0.8))
label = 'L' if Y[k, 0] else 'H'
ax.text(x + 0.36, 1.5, label, ha='center', fontsize=11,
color=BLUE if label == 'L' else RED)
ax.plot([left, left, left + 3.27, left + 3.27],
[1.24, 1.12, 1.12, 1.24], 'k-', lw=1.3)
diagram_arrow(ax, (left + 1.63, 1.07), (left + 1.63, 0.65))
ax.text(left + 1.63, 0.42, f'Update {batch_index + 1}', ha='center', fontsize=14)
ax.text(6, -0.08, 'ONE EPOCH = all 12 pictures used once = 3 updates',
ha='center', fontsize=16, weight='bold')
ax.text(6, -0.58, 'Next epoch: shuffle pictures AND their labels, then repeat.',
ha='center', fontsize=14)
ax.set_ylim(-0.85, 3.7); ax.set_aspect('auto')
plt.tight_layout(); plt.show()
def plot_data_roles():
fig, ax = diagram_canvas(height=4.4)
rows = [(3.5, 'TRAINING', '8,000 pictures', 'Change the weights', '#8B0029', '-'),
(2.1, 'VALIDATION', '2,000 pictures', 'Check progress; choose settings', BLUE, '--'),
(0.7, 'TEST', '10,000 new pictures', 'Final check after choosing settings', 'black', '--')]
for y, name, count, action, color, style in rows:
for j in range(3):
ax.add_patch(Rectangle((0.2 + j * 0.1, y - 0.18 + j * 0.1), 0.5, 0.5,
facecolor='white', edgecolor=color, linewidth=1.2))
ax.text(1.05, y + 0.15, name, fontsize=15, weight='bold', color=color)
ax.text(1.05, y - 0.32, count, fontsize=12)
diagram_arrow(ax, (4.0, y), (5.6, y), color=color, style=style)
ax.text(6.0, y + 0.1, action, va='center', fontsize=15)
ax.text(6.0, 3.04, 'Weights change only in this row.', fontsize=12, color='#8B0029')
ax.text(6.0, 1.64, 'Measure loss / accuracy; no weight update.', fontsize=12)
ax.text(6.0, 0.24, 'Measure loss / accuracy; no weight update.', fontsize=12)
plt.tight_layout(); plt.show()
3.1 One training step
$$h=\operatorname{sigmoid}(xW_1+b_1),\qquad p=\mathrm{softmax}(hW_2+b_2).$$ $$\operatorname{sigmoid}(z)=\frac{1}{1+e^{-z}},\qquad \mathrm{softmax}(z)_c=\frac{e^{z_c}}{\sum_{c'}e^{z_{c'}}}.$$
| Symbol | Meaning |
|---|---|
| $x$ | One spin picture flattened into a row of $\pm1$ values. |
| $W_1,b_1$; $W_2,b_2$ | Weights (multipliers) and biases (additive offsets) for spins → hidden layer; hidden layer → outputs. |
| $h$; $p=(p_{\mathrm{low}},p_{\mathrm{high}})$ | Hidden-neuron activations; the two predicted class probabilities. |
| $z$; $z_c$ | Scalar input for sigmoid, or a vector of scores for softmax; its class-$c$ component. |
| $c,c'$ | Class indices: low T or high T. The sum over $c'$ includes both classes. |
The known label judges the prediction. Only weights and biases change.
X_example, Y_example, _ = make_dataset(data["train_10"], temps, Tc_square)
example_net = initialize_net(n_in=100, n_hidden=3, seed=7)
hidden, prediction = forward(example_net, X_example[0:1])
plot_training_loop(X_example[0].reshape(10, 10), prediction[0], Y_example[0])
3.2 Cross-entropy loss
$$\mathcal L_{\mathrm{CE}}=-\frac1B\sum_{i=1}^{B}\sum_c y_{ic}\log p_{ic}, \qquad \ell_i=-\log p_{i,\mathrm{correct}}.$$
| Symbol | Meaning |
|---|---|
| $B$; $i=1,\ldots,B$ | Number of pictures in a batch; index of one picture. |
| $y_{ic}$; $p_{ic}$ | Label: 1 if class $c$ is correct for picture $i$, otherwise 0; predicted probability of that class. |
| $\ell_i$; $\mathcal L_{\mathrm{CE}}$ | Loss of picture $i$; average cross-entropy over the batch. |
$p_{i,\mathrm{correct}}$ selects the correct class; $\log$ is the natural logarithm. The curve shows one picture's loss.
p_correct = 0.9 # TRY: 0.1, 0.5
loss = -np.log(p_correct)
plot_loss_curve(p_correct)
3.3 Gradient descent: one score update
$$\theta_{\mathrm{new}}=\theta-\eta\nabla_\theta\mathcal L, \qquad \theta=\{W_1,b_1,W_2,b_2\}.$$
| Symbol | Meaning |
|---|---|
| $\theta$; $\theta_{\mathrm{new}}$ | All current weights and biases; their values after one update. |
| $\mathcal L$; $\nabla_\theta\mathcal L$ | Loss to minimize (cross-entropy here); its derivatives with respect to every parameter. |
| $\eta>0$ | Learning rate: how far to step in the negative-gradient direction. |
First try a single adjustable score $s$. Let $q=p_{\mathrm{low}}$ be its low-T probability:
$$q=\operatorname{sigmoid}(s),\qquad \frac{\partial(-\log q)}{\partial s}=q-1.$$
$\partial/\partial s$ means the derivative with respect to $s$. The target is low T, so the desired probability is 1.
score_before = 0.0
learning_rate = 1.0 # TRY: 0.1 for a smaller step.
prediction = sigmoid(score_before)
gradient = prediction - 1.0 # Target: low T.
score_after = score_before - learning_rate * gradient
plot_score_update(score_before, score_after, gradient, learning_rate)
3.4 Backpropagation and repeated updates
Backpropagation applies the chain rule to a batch:
$$D_2=\frac{P-Y}{B},\qquad D_1=(D_2W_2^{\mathsf T})\odot H\odot(1-H),$$ $$\nabla_{W_2}\mathcal L_{\mathrm{CE}}=H^{\mathsf T}D_2, \qquad \nabla_{W_1}\mathcal L_{\mathrm{CE}}=X^{\mathsf T}D_1.$$
| Symbol | Meaning |
|---|---|
| $X,Y,P,H$ | Batch arrays: rows of spin inputs, labels, predicted probabilities, and hidden activations. |
| $D_2$; $D_1$ | Derivatives of $\mathcal L_{\mathrm{CE}}$ with respect to output scores before softmax; hidden weighted sums before sigmoid. |
| ${}^{\mathsf T}$; $\odot$ | Transpose (swap rows and columns); multiply corresponding entries. |
| $\nabla_{W_1}\mathcal L_{\mathrm{CE}},\nabla_{W_2}\mathcal L_{\mathrm{CE}}$ | Loss derivatives for each entry of the two weight matrices. |
Bias gradients sum $D_1,D_2$ over batch rows. Repeat the update ten times.
# Four low-T pictures and four high-T pictures.
x_batch = np.concatenate([X_example[:4], X_example[-4:]])
y_batch = np.concatenate([Y_example[:4], Y_example[-4:]])
demo = initialize_net(n_in=100, n_hidden=3, seed=7)
learning_rate = 0.1
before = forward(demo, x_batch)[1].copy()
losses = [loss_parts(demo, x_batch, y_batch)[0]]
for update in range(10):
grad = gradients(demo, x_batch, y_batch)
for name in demo:
demo[name] -= learning_rate * grad[name]
losses.append(loss_parts(demo, x_batch, y_batch)[0])
after = forward(demo, x_batch)[1]
plot_repeated_updates(losses, before, after, y_batch)
4. Training on the full data set
4.1 Batch and epoch
Batch: pictures used for one update. Epoch: one pass through all pictures.
plot_batches(X_example, Y_example) # Small 12-picture illustration.
Our run: 8,000 pictures ÷ 100 per batch = 80 updates per epoch.
4.2 Which pictures can change the weights?
Keep each Monte Carlo chain in one split, because nearby samples can be similar.
plot_data_roles()
4.3 Run training; watch both curves
$$\mathcal L=\mathcal L_{\mathrm{CE}}+ \frac{\lambda}{2}\left(\|W_1\|_F^2+\|W_2\|_F^2\right), \qquad \nabla_W\mathcal L=\nabla_W\mathcal L_{\mathrm{CE}}+\lambda W.$$
| Symbol | Meaning |
|---|---|
$\lambda$ (lam) |
Nonnegative strength of the weight penalty. |
| $W$; $\|W\|_F^2$ | Either $W_1$ or $W_2$; the sum of squares of all its entries (squared Frobenius norm). |
| $\mathcal L$ | Full loss minimized here: mean cross-entropy plus the weight penalty. |
Adam rescales gradient steps using past gradients (details in the appendix). Both plotted losses exclude the penalty.
epochs = 60 # TRY: 10
nets, history, test, validation = train_all_sizes(epochs=epochs, batch=100, lr=1e-3)
plot_training_history(history, L_curve=30)
Loss ↓, accuracy ↑. If only training improves while validation worsens, the model may be overfitting. Compare settings using validation, then test once.
What the network says about single configurations
Test configurations ($L=30$) at different temperatures. The bar under each one is the output of the network: blue = $O_\text{low T}$, red = $O_\text{high T}$.
L0 = 30 # the size analysed in detail
X0, Y0, T0 = test[L0]
h0, out0 = forward(nets[L0], X0)
m0 = X0.mean(axis=1) # magnetization of every test configuration
rng = np.random.default_rng(3) # TRY: another seed -> other samples
picks = [np.flatnonzero(np.isclose(T0, T))[rng.integers(250)] for T in temps[[0, 8, 14, 17, 19, 21, 24, 30, 39]]]
fig, axes = plt.subplots(2, len(picks), figsize=(13, 2.4), height_ratios=[8, 1])
for col, k in enumerate(picks):
axes[0, col].imshow(X0[k].reshape(L0, L0), cmap="gray", vmin=-1, vmax=1)
axes[0, col].set_title(f"T = {T0[k]:.2f}", fontsize=10, color=BLUE if T0[k] < Tc_square else RED)
axes[1, col].barh([0], [out0[k, 0]], color=BLUE); axes[1, col].barh([0], [out0[k, 1]], left=[out0[k, 0]], color=RED)
axes[1, col].set_xlim(0, 1); axes[1, col].set_xlabel(f"$O_{{low}}$ = {out0[k, 0]:.2f}", fontsize=9)
for ax in axes[:, col]:
ax.set_xticks([]); ax.set_yticks([])
for side in ax.spines.values(): side.set_visible(True)
plt.show()
Output and accuracy versus temperature (Fig. 1B, 1C of the paper)
Average the two outputs over all test configurations at the same temperature. Deep in either phase the network is certain. The two curves cross at 0.5 close to the exact $T_c$ (orange line), and the classification errors are concentrated there.
The paper quotes test accuracies of 94% for $L=10$ and 99% for $L=40$. The overall number depends on the temperature window, because all errors sit near $T_c$.
def output_vs_T(net, conf):
# average output over the configurations at each temperature -> (n_T, 2), and its standard error
n_T, n_samples, L, _ = conf.shape
out = forward(net, conf.reshape(n_T * n_samples, L * L).astype(np.float32))[1].reshape(n_T, n_samples, 2)
return out.mean(axis=1), out.std(axis=1) / np.sqrt(n_samples)
mean, err = output_vs_T(nets[L0], data[f"test_{L0}"])
correct = (out0.argmax(axis=1) == Y0.argmax(axis=1)).reshape(len(temps), -1).mean(axis=1)
fig, (a, b) = plt.subplots(1, 2, figsize=(11, 3.8))
a.errorbar(temps, mean[:, 0], err[:, 0], fmt="o-", ms=4, color=BLUE, label=r"$O_{\rm low\,T}$")
a.errorbar(temps, mean[:, 1], err[:, 1], fmt="s-", ms=4, color=RED, label=r"$O_{\rm high\,T}$")
a.axvline(Tc_square, color=ORANGE, lw=2, label="exact $T_c$"); a.axhline(0.5, color="0.7", lw=0.8)
a.set_xlabel("T"); a.set_ylabel("average output"); a.set_title(f"L = {L0}"); a.legend()
b.plot(temps, correct, "o-", ms=4, color="k"); b.axvline(Tc_square, color=ORANGE, lw=2)
b.set_xlabel("T"); b.set_ylabel("test accuracy"); b.set_ylim(0, 1.02)
b.set_title(f"overall test accuracy = {correct.mean():.3f}")
plt.tight_layout(); plt.show()
5. Inside the network: what did it learn?
Weights of the hidden neurons
Each hidden neuron has one weight per lattice site, so its weights can be drawn as an $L\times L$ picture. Red = positive, blue = negative, same color scale for all.
W1, b1 = nets[L0]["W1"], nets[L0]["b1"]
scale = np.abs(W1).max()
fig, axes = plt.subplots(3, 10, figsize=(13, 4.2))
for k, ax in enumerate(axes.flat): # first 30 of the 100 hidden neurons
ax.imshow(W1[:, k].reshape(L0, L0), cmap="bwr", vmin=-scale, vmax=scale)
ax.set_xticks([]); ax.set_yticks([]); ax.set_title(f"neuron {k}", fontsize=8)
fig.suptitle("Input weights of hidden neurons, drawn on the lattice", y=1.0)
plt.tight_layout(); plt.show()
mean_w = W1.mean(axis=0) # site-averaged weight of each neuron
# Test: throw away the site-to-site structure. Every neuron keeps only its average weight.
net_uniform = dict(nets[L0])
net_uniform["W1"] = np.tile(mean_w, (L0 * L0, 1))
print(f"test accuracy, trained weights : {accuracy(nets[L0], X0, Y0):.3f}")
print(f"test accuracy, site-averaged weights: {accuracy(net_uniform, X0, Y0):.3f}")
The pictures are noisy, but many have an overall tint: the weights of neuron $k$ share an average $\bar w_k$. Replacing every weight by this average does not reduce the test accuracy (printed above). The site-to-site structure is noise; only the average matters. With uniform weights,
$$\sum_i \sigma_i W_{1,ik} + b_k \;=\; N \bar w_k\, m + b_k ,$$
a linear function of the magnetization. The average part adds up over all $N$ sites, while the noise partly cancels. Check it directly: plot the input of each hidden neuron against $m$ for all test configurations.
z0 = X0 @ W1 + b1 # input of the hidden neurons, (n_configs, 100)
order = np.argsort(mean_w)
show = order[[0, 1, 2, 3, -4, -3, -2, -1]] # 4 most negative and 4 most positive neurons
fig, (a, b, c) = plt.subplots(1, 3, figsize=(13, 3.8))
for k in show:
color = RED if mean_w[k] > 0 else BLUE
a.scatter(m0, z0[:, k], s=1, alpha=0.15, color=color)
b.scatter(m0, h0[:, k], s=1, alpha=0.15, color=color)
a.set_xlabel("magnetization m"); a.set_ylabel("input of hidden neuron"); a.set_title("8 hidden neurons: input")
b.set_xlabel("magnetization m"); b.set_ylabel("activation h"); b.set_title("same neurons: activation")
sc = c.scatter(m0, out0[:, 0], s=3, c=T0, cmap="coolwarm", alpha=0.5)
c.set_xlabel("magnetization m"); c.set_ylabel(r"$O_{\rm low\,T}$"); c.set_title("output of every test configuration")
plt.colorbar(sc, ax=c, label="T")
plt.tight_layout(); plt.show()
print(f"correlation between O_low and |m|: {np.corrcoef(out0[:, 0], np.abs(m0))[0, 1]:.3f}")
Reading the figure.
- Left: the input of each hidden neuron is a straight line in $m$. Red neurons switch on for $m>0$, blue neurons for $m<0$.
- Right: the output depends on the configuration almost only through $|m|$. It is large when the state is polarized in either direction and small when $m\approx0$.
The network was never told about the magnetization. It discovered the order parameter.
6. Toy model: three hidden neurons (Fig. 1D–F of the paper)
If the network only needs $m$, three hidden neurons with a step activation $\Theta$ are enough. With one free parameter $0<\epsilon<1$:
$$W_1=\frac{1}{N(1+\epsilon)}\begin{pmatrix}1&\cdots&1\\-1&\cdots&-1\\1&\cdots&1\end{pmatrix},\quad b_1=\frac{\epsilon}{1+\epsilon}\begin{pmatrix}-1\\-1\\1\end{pmatrix} \;\Rightarrow\; W_1x+b_1=\frac{1}{1+\epsilon}\begin{pmatrix}m-\epsilon\\-m-\epsilon\\m+\epsilon\end{pmatrix}$$
$$W_2=\begin{pmatrix}2&1&-1\\-2&-2&1\end{pmatrix},\qquad b_2=0 .$$
| state | hidden $(h_1,h_2,h_3)$ | $W_2h$ | answer | | --- | --- | --- | --- | | $m>\epsilon$ (polarized up) | $(1,0,1)$ | $(1,-1)$ | low T | | $m<-\epsilon$ (polarized down) | $(0,1,0)$ | $(1,-2)$ | low T | | $|m|<\epsilon$ (unpolarized) | $(0,0,1)$ | $(-1,1)$ | high T |
The toy model says "low T" when $|m|>\epsilon$. There is nothing to train: only $\epsilon$ to choose.
def toy_model(X, eps):
N = X.shape[1]
W1 = np.array([np.ones(N), -np.ones(N), np.ones(N)]).T / (N * (1 + eps))
b1 = eps / (1 + eps) * np.array([-1, -1, 1])
W2 = np.array([[2, 1, -1], [-2, -2, 1]]).T
h = np.heaviside(X @ W1 + b1, 0)
return np.heaviside(h @ W2, 0) # columns: [O_low, O_high]
# choose epsilon on the training set
X, Y, T = make_dataset(data[f"train_{L0}"], temps, Tc_square)
eps_grid = np.linspace(0.02, 0.98, 97)
acc_eps = [np.mean(toy_model(X, e)[:, 0] == Y[:, 0]) for e in eps_grid]
eps_best = eps_grid[np.argmax(acc_eps)]
toy = toy_model(X0, eps_best) # evaluate on the test set
toy_mean = toy.reshape(len(temps), -1, 2).mean(axis=1)
toy_correct = (toy[:, 0] == Y0[:, 0]).reshape(len(temps), -1).mean(axis=1)
fig, (a, b, c) = plt.subplots(1, 3, figsize=(13, 3.6))
a.plot(eps_grid, acc_eps, "k-"); a.axvline(eps_best, color="0.5", ls="--")
a.set_xlabel(r"$\epsilon$"); a.set_ylabel("training accuracy"); a.set_title(rf"best $\epsilon$ = {eps_best:.2f}")
b.plot(temps, toy_mean[:, 0], "o-", ms=4, color=BLUE, label="toy $O_{low}$")
b.plot(temps, toy_mean[:, 1], "s-", ms=4, color=RED, label="toy $O_{high}$")
b.plot(temps, mean[:, 0], "-", color=BLUE, alpha=0.4, lw=3, label="trained FNN")
b.plot(temps, mean[:, 1], "-", color=RED, alpha=0.4, lw=3)
b.axvline(Tc_square, color=ORANGE, lw=2); b.set_xlabel("T"); b.set_ylabel("average output"); b.legend(fontsize=9)
c.plot(temps, toy_correct, "o-", ms=4, color="k", label=f"toy model ({toy_correct.mean():.3f})")
c.plot(temps, correct, "-", color="0.6", lw=3, label=f"trained FNN ({correct.mean():.3f})")
c.axvline(Tc_square, color=ORANGE, lw=2); c.set_xlabel("T"); c.set_ylabel("test accuracy"); c.set_ylim(0, 1.02); c.legend(fontsize=9)
plt.tight_layout(); plt.show()
A three-neuron model with one hand-chosen number does about as well as the trained network with 100 hidden neurons. For this problem the trained network is, in effect, a threshold on $|m|$.
7. Finite-size scaling and $T_c$ (Fig. 1a, 1b of the Nature Physics version)
Repeat the output-versus-temperature plot for all sizes. The crossover sharpens as $L$ grows. The temperature where the two outputs cross (both 0.5) is the network's estimate of $T_c$.
Near a continuous transition, quantities of a finite system depend on $T$ and $L$ only through $t\,L^{1/\nu}$ with $t=T-T_c$. For the 2D Ising model $\nu=1$. If the network output behaves like a physical observable, the curves for all sizes fall on one curve when plotted against $tL^{1/\nu}$.
def crossing(temperatures, mean):
# temperature where O_low - O_high changes sign (linear interpolation; the crossing nearest 0.5)
d = mean[:, 0] - mean[:, 1]
k = np.flatnonzero(d[:-1] * d[1:] < 0)
roots = temperatures[k] - d[k] * (temperatures[k + 1] - temperatures[k]) / (d[k + 1] - d[k])
return roots[0] if len(roots) else np.nan
nu = 1.0 # TRY: 0.5, 2
markers = ["x", "^", "o", "D", "s"]
colors = plt.cm.viridis(np.linspace(0, 0.9, len(sizes)))
fig, (a, b) = plt.subplots(1, 2, figsize=(12, 4))
Tc_est = {}
for L, mk, color in zip(sizes, markers, colors):
mean_L, err_L = output_vs_T(nets[L], data[f"test_{L}"])
Tc_est[L] = crossing(temps, mean_L)
for col in (0, 1):
a.plot(temps, mean_L[:, col], mk + "-", ms=4, lw=0.8, color=color, label=f"L = {L}" if col == 0 else None)
b.plot((temps - Tc_square) * L ** (1 / nu), mean_L[:, col], mk, ms=4, color=color)
a.axvline(Tc_square, color=ORANGE, lw=2); a.set_xlabel("T"); a.set_ylabel("average output"); a.legend(fontsize=9)
a.set_title("square lattice")
b.axvline(0, color=ORANGE, lw=2); b.set_xlim(-10, 10); b.set_xlabel(r"$t\,L^{1/\nu}$"); b.set_ylabel("average output")
b.set_title(rf"data collapse, $\nu$ = {nu:g}")
plt.tight_layout(); plt.show()
print("L crossing temperature")
for L in sizes:
print(f"{L:<4d} {Tc_est[L]:.3f}")
print(f"exact Tc = {Tc_square:.3f} (temperature grid spacing = {temps[1] - temps[0]:.3f})")
8. Triangular lattice: a model the network has never seen (Fig. 2 of the paper)
The triangular-lattice Ising ferromagnet has six neighbors per spin and a different critical temperature, $T_c = 4/\ln 3 \approx 3.641$.
Take the networks trained on the square lattice and, without any further training, show them triangular-lattice configurations. The labels are not used at all: we only look at where the two outputs cross. The paper reports a crossing at $T/J = 3.636$ for $L=30$.
fig, (a, b) = plt.subplots(1, 2, figsize=(12, 4))
Tc_tri_est = {}
for L, mk, color in zip(sizes, markers, colors):
mean_L, err_L = output_vs_T(nets[L], data[f"tri_{L}"]) # square-lattice network, triangular data
Tc_tri_est[L] = crossing(temps_tri, mean_L)
for col in (0, 1):
a.plot(temps_tri, mean_L[:, col], mk + "-", ms=4, lw=0.8, color=color, label=f"L = {L}" if col == 0 else None)
b.plot((temps_tri - Tc_triangular) * L ** (1 / nu), mean_L[:, col], mk, ms=4, color=color)
a.axvline(Tc_triangular, color=ORANGE, lw=2, label="exact $T_c$ (triangular)")
a.axvline(Tc_square, color=ORANGE, lw=1, ls=":", label="$T_c$ (square, used in training)")
a.set_xlim(2.1, 5.0); a.set_xlabel("T"); a.set_ylabel("average output"); a.legend(fontsize=8); a.set_title("triangular lattice")
b.axvline(0, color=ORANGE, lw=2); b.set_xlim(-10, 10); b.set_xlabel(r"$t\,L^{1/\nu}$"); b.set_ylabel("average output")
b.set_title(rf"data collapse, $\nu$ = {nu:g}")
plt.tight_layout(); plt.show()
print("L crossing (square) crossing (triangular)")
for L in sizes:
print(f"{L:<4d} {Tc_est[L]:.3f} {Tc_tri_est[L]:.3f}")
print(f"exact {Tc_square:.3f} {Tc_triangular:.3f}")
The network was trained with labels defined by $T_c\approx2.269$, yet it places the transition of the triangular lattice near $3.64$. It did not memorize a temperature. It learned to recognize order from the configuration itself, and that carries over to another lattice.
Summary
- A one-hidden-layer network classifies raw Ising configurations into low-T and high-T phases.
- Errors occur only near $T_c$, and the crossover sharpens with $L$ following finite-size scaling.
- The hidden neurons respond to the magnetization: replacing their weights by the site average keeps the accuracy.
- The same network locates $T_c$ of the triangular lattice without retraining.
Limit. This works because the Ising transition has a simple local order parameter. The paper goes on to phases without one (square ice, Ising gauge theory), where the fully connected network fails and a convolutional network is needed.
TRY
n_hidden=3intrain_all_sizes: how many hidden neurons are needed?lam=0(no regularization): compare training and validation loss/accuracy for $L=60$, and redraw the weight pictures.- Train on $T<1.8$ and $T>2.8$ only. Does the crossing still land near $T_c$?
- Shuffle the lattice sites of every configuration with one fixed permutation and retrain. Does the accuracy change? Why?
- Feed the antiferromagnet ($J<0$, staggered order) to the trained network. What does it answer, and why?
References
- J. Carrasquilla and R. G. Melko, Machine learning phases of matter, arXiv:1605.01735; Nature Physics 13, 431 (2017).
- L. Onsager, Phys. Rev. 65, 117 (1944): exact $T_c$ and magnetization of the square lattice.
- D. P. Kingma and J. Ba, Adam: a method for stochastic optimization.
- I. Goodfellow, Y. Bengio and A. Courville, Deep Learning, Chapter 6: feed-forward networks and backpropagation.
Optional appendix
Detailed equations and implementation notes
The network outputs model probabilities; they are not guaranteed to be calibrated. The one-score example isolates an update and does not itself learn spin patterns.
A. From spins to probabilities
A weight controls how strongly an input contributes. A bias adds an offset. For a batch of $B$ pictures, the network calculates
$$Z_1=XW_1+b_1,\qquad H=\operatorname{sigmoid}(Z_1),\qquad Z_2=HW_2+b_2,$$ $$\operatorname{sigmoid}(z)=\frac{1}{1+e^{-z}},\qquad P_{ic}=\frac{e^{Z_{2,ic}}}{\sum_{c'}e^{Z_{2,ic'}}}.$$
$Z_1,Z_2$ are batch arrays of weighted sums before sigmoid and softmax, respectively.
The second expression is softmax: it makes each row sum to one.
The entries of $Z_2$ are called logits (scores before softmax).
The one-score example used two logits [score, 0], whose low-T probability
is exactly sigmoid(score).
| Array | Shape |
|---|---|
X, Y |
(B, L*L), (B, 2) |
W1, b1 |
(L*L, n_hidden), (n_hidden,) |
h |
(B, n_hidden) |
W2, b2 |
(n_hidden, 2), (2,) |
out |
(B, 2) |
In NumPy, @ multiplies matrices and * multiplies corresponding entries.
Biases are added to every row. Random initial weights let hidden neurons start
differently. log_softmax computes log-probabilities without first rounding a
very small probability to zero; logaddexp avoids overflow in sigmoid.
B. Loss and backpropagation
With one-hot labels, cross-entropy selects the log-probability of the correct class. The full objective adds an $L_2$ penalty on weights, but not biases:
$$\mathcal L=-\frac1B\sum_{i,c}Y_{ic}\log P_{ic} +\frac\lambda2\left(\sum_{jk}W_{1,jk}^2+\sum_{kc}W_{2,kc}^2\right).$$
The penalty discourages large weights; it does not guarantee good predictions on
new data. loss_parts returns the data loss and the full objective separately.
The full objective is stored in history[L]["objective"].
For one low-T example with score $s$, $q=\operatorname{sigmoid}(s)$ and one-picture loss $\ell=-\log q$:
$$\frac{d\ell}{ds}=-\frac1q\,q(1-q)=q-1.$$
For a binary target $y=1$ (low T) or $y=0$ (high T), the derivative is $q-y$. For a batch with two softmax outputs, the corresponding derivative is $D_2=(P-Y)/B$. The chain rule propagates it back through the hidden layer:
$$D_1=(D_2W_2^\mathsf T)\odot H\odot(1-H),$$ $$\frac{\partial\mathcal L}{\partial W_2}=H^\mathsf T D_2+\lambda W_2, \qquad \frac{\partial\mathcal L}{\partial W_1}=X^\mathsf T D_1+\lambda W_1.$$
Here $\odot$ means elementwise multiplication. The bias gradients are the sums of
$D_2$ and $D_1$ over the batch rows. These are the lines in gradients.
All gradients are calculated before any parameter changes.
C. Adam
For each parameter, Adam tracks averages of the gradient $g_t$ and its square:
$$m_t=0.9m_{t-1}+0.1g_t,\qquad v_t=0.999v_{t-1}+0.001g_t^2,$$ $$\widehat m_t=\frac{m_t}{1-0.9^t},\qquad \widehat v_t=\frac{v_t}{1-0.999^t},$$ $$\theta_t=\theta_{t-1}-\eta\frac{\widehat m_t}{\sqrt{\widehat v_t}+10^{-8}}.$$
| Symbol | Meaning |
|---|---|
| $t$; $\theta_t$ | Update number (not epoch); parameters after that update. |
| $g_t$ | Gradient of the full loss at update $t$, for each parameter. |
| $m_t,v_t$ | Running averages of the gradient and its square. Adam's $m_t$ is unrelated to Ising magnetization. |
| $\widehat m_t,\widehat v_t$ | Averages corrected for starting from zero. |
| $10^{-8}$ | Small positive constant to keep the denominator away from zero. |
$\eta$ is the learning rate (lr).
Very small learning rates can be slow; very large ones can overshoot.
Mini-batch updates need not reduce the loss at every step. The loss can improve
without changing accuracy because probabilities can change without switching
which class has the larger probability.