Skip to content

ALS: improve CPU thread scaling in CG and loss calculation - #761

Open
riktor wants to merge 1 commit into
benfred:mainfrom
riktor:cg-blas1
Open

riktor wants to merge 1 commit into
benfred:mainfrom
riktor:cg-blas1

Conversation

@riktor

@riktor riktor commented Nov 8, 2026

Copy link
Copy Markdown

The CPU CG solver uses BLAS symv for its products with the Gram matrix
YtY: once per row for the initial residual, and once per CG step for A·p.
calculate_loss also calls symv once per row. With the pthreads OpenBLAS
build used in these benchmarks,
concurrent symv calls scale poorly beyond a few OpenMP threads. On last.fm,
CG was no faster with 40 threads than with 4.

This replaces each symv call with n BLAS level-1 axpy calls, where n
is the number of factors. The matrix-vector product is mathematically unchanged;
results differ only in floating-point rounding due to the changed summation order.

Seconds per ALS iteration on a dual-socket Xeon (20 cores per socket),
128 factors, cg_steps=3, one thread per physical core,
OPENBLAS_NUM_THREADS=1, before → after:

threads last.fm 360k ml20m
1 14.3 → 15.1 10.0 → 11.0
8 5.1 → 2.4 2.2 → 1.5
40 6.0 → 0.7 1.9 → 0.4

Single-threaded runs are about 10% slower (medians of five runs each:
last.fm 14.3 → 15.9 s, ml20m 10.3 → 11.1 s): without thread contention,
one symv call is faster than n axpy calls.

With the pthreads OpenBLAS build used in our benchmarks, the per-row symv
calls made by the CG solver (once per row for the initial residual, once
per CG step for A*p) and by calculate_loss stop scaling at a handful of
OpenMP threads: on last.fm, CG was no faster with 40 threads than with 4.
The level 1 routines do not have the problem, so the product with the Gram
matrix YtY is now n axpy calls.

Seconds per ALS iteration, 128 factors, cg_steps=3, one thread per physical
core on a dual-socket Xeon (20 cores per socket), OPENBLAS_NUM_THREADS=1,
before -> after:

    threads   last.fm 360k      ml20m
          1   14.3 -> 15.1      10.0 -> 11.0
          8    5.1 ->  2.4       2.2 ->  1.5
         40    6.0 ->  0.7       1.9 ->  0.4

Single-threaded runs are about 10% slower (medians of five runs each:
last.fm 14.3 -> 15.9 s, ml20m 10.3 -> 11.1 s): without thread contention,
one symv call is faster than n axpy calls. Results change only by floating
point rounding (summation order).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant