Count line of processor in /proc/cpuinfo on linux, that matches...

[sergey-dryabzhinsky]

cpu cores even in lxc containers
Address issue #4510

[sergey-dryabzhinsky]

Backup compression on proxmox3 server / debian7. Cpu: intel core i7 6850k 3.6 GHz, 12 cores.
W/o patch:

 time zstdmt -14 -vv -f vzdump-openvz-20001-2025_10_09-15_57_53.tar
*** Zstandard CLI (64-bit) v1.5.7, by Yann Collet ***
Note: 6 physical core(s) detected
Compressing with 6 worker threads
--zstd=wlog=22,clog=22,hlog=23,slog=5,mml=5,tlen=32,strat=6
--format=.zst --no-sparse --block-size=0 --memory=134217728 --threads=6 --content-size
Decompression will require 4194304 B of memory
vzdump-openvz-20001-2025_10_09-15_57_53.tar : 22.04%   (1803284480 B => 397499354 B, vzdump-openvz-20001-20
25_10_09-15_57_53.tar.zst)
vzdump-openvz-20001-2025_10_09-15_57_53.tar : Completed in 26.51 sec  (cpu load : 489%)
real    0m26.573s
user    2m8.388s
sys     0m1.313s

[sergey-dryabzhinsky]

With patch:

time zstdmt -14 -vv -f vzdump-openvz-20001-2025_10_09-15_57_53.tar
*** Zstandard CLI (64-bit) v1.5.7, by Yann Collet ***
Note: 12 physical core(s) detected
Compressing with 12 worker threads
--zstd=wlog=22,clog=22,hlog=23,slog=5,mml=5,tlen=32,strat=6
--format=.zst --no-sparse --block-size=0 --memory=134217728 --threads=12 --content-size
Decompression will require 4194304 B of memory
vzdump-openvz-20001-2025_10_09-15_57_53.tar : 22.04%   (1803284480 B => 397499354 B, vzdump-openvz-20001-2025_10_09-15_57_53.tar.zst)
vzdump-openvz-20001-2025_10_09-15_57_53.tar : Completed in 17.84 sec  (cpu load : 916%)
real    0m17.990s
user    2m42.266s
sys     0m1.353s

System load greater but process run faster.

[sergey-dryabzhinsky]

Second server: proxmox7, debian11, cpu intel xeon E3-1230 v6 @ 3.50GHz, 8 cores.
W/o patch:

 time zstdmt -14 -vv -f vzdump-openvz-60047-2023_08_18-18_01_53.tar
*** Zstandard CLI (64-bit) v1.5.5, by Yann Collet ***
Note: 4 physical core(s) detected
--zstd=wlog=22,clog=22,hlog=23,slog=5,mml=5,tlen=32,strat=6
--format=.zst --no-sparse --block-size=0 --memory=134217728 --threads=4 --content-size
Decompression will require 4194304 B of memory
vzdump-openvz-60047-2023_08_18-18_01_53.tar : 25.32%   (580116480 B => 146903865 B, vzdump-openvz-60047-2023_08_18-18_01_53.tar.zst)
vzdump-openvz-60047-2023_08_18-18_01_53.tar : Completed in 12.85 sec  (cpu load : 375%)
real    0m12.890s
user    0m47.804s
sys     0m0.433s

[sergey-dryabzhinsky]

With patch:

time zstdmt -14 -vv -f vzdump-openvz-60047-2023_08_18-18_01_53.tar
*** Zstandard CLI (64-bit) v1.5.7, by Yann Collet ***
Note: 8 physical core(s) detected
Compressing with 8 worker threads
--zstd=wlog=22,clog=22,hlog=23,slog=5,mml=5,tlen=32,strat=6
--format=.zst --no-sparse --block-size=0 --memory=134217728 --threads=8 --content-size
Decompression will require 4194304 B of memory
vzdump-openvz-60047-2023_08_18-18_01_53.tar : 25.01%   (580116480 B => 145070412 B, vzdump-openvz-60047-2023_08_18-18_01_53.tar.zst)
vzdump-openvz-60047-2023_08_18-18_01_53.tar : Completed in 9.25 sec  (cpu load : 673%)
real    0m9.279s
user    1m1.568s
sys     0m0.703s

[sergey-dryabzhinsky]

Failed action not related

[Cyan4973]

If you prefer zstdmt or -T0 to count logical cores as opposed to physical ones, there is already the --auto-threads=logical command.

[sergey-dryabzhinsky]

@Cyan4973 Thanx, I'll check it

[sergey-dryabzhinsky]

That works:

 zstdmt -vv --auto-threads=logical /dev/zero
*** Zstandard CLI (64-bit) v1.5.7, by Yann Collet ***
Note: 8 logical core(s) detected
Compressing with 8 worker threads
--zstd=wlog=21,clog=16,hlog=17,slog=1,mml=5,tlen=0,strat=2
--format=.zst --no-sparse --block-size=0 --memory=134217728 --threads=8 --content-size
zstd: /dev/zero is not a regular file -- ignored

Question: why it is not default behaviour?

[Cyan4973]

It's a question of choice.
The original choice was: one thread per physical code.
I already explained in another thread the reasoning leading to that choice.

Now, this is not immutable, we could change it if we feel that's a better option.
This is less a question of code than a question of properly documenting why a different default would be better.

[sergey-dryabzhinsky]

Maybe this behaviour could be changed at compile time to let distributors choose what they want?

[Cyan4973]

Maybe this behaviour could be changed at compile time to let distributors choose what they want?

A compilation variable would do the trick

[sergey-dryabzhinsky]

I added compilation "variable" through cflags -DZSTD_PARSE_FULL_CPUINFO , so my code stay disabled by default. I'll add other PR with another compile var later

[sergey-dryabzhinsky]

Work still in progress.

[sergey-dryabzhinsky]

All done I think.

[sergey-dryabzhinsky]

Not shure how to add this to docs.

[sergey-dryabzhinsky]

Ready for review