Cool! Looking forward to using your stuff on my own ALIX. Hope your patch is included into OpenSSL soon.

@Flo: Filed a bug at openssl, offering the patch for inclusion: Ticket 2554.

After some discussion with Markus via e-mail, it seems that some combinations of hardware (in my case the Kirkwood 88F6281) and drivers (the stock kernel drivers from Marvell) have trouble performing SSL when the SHA-1 offloading is enabled. By removing the line

  !ENGINE_set_digests (e, af_alg_digests))

(which is line 77 in the current revision) OpenSSL will use software processing to determine the hashes. This can work around issues in the hashing offload, and probably will get you better performance anyway. (Hardware offloading of hashes bottlenecks on context switching overhead and you can usually do it faster in software.)

Works great on a Seagate Dockstar (Kirkwood 88F6281)! Also thanks to Adam for his hint.

Some decryption tests with 8192 byte block size:

Software: openssl speed -evp aes-256-cbc -decrypt

“Doing aes-256-cbc for 3s on 8192 size blocks: 3021 aes-256-cbc's in 2.95s”

AF_ALG: openssl speed -evp aes-256-cbc -engine af_alg -decrypt

“Doing aes-256-cbc for 3s on 8192 size blocks: 7134 aes-256-cbc's in 0.03s”

I am using the latest kernel from here: http://sheeva.with-linux.com/sheeva/

Thank you very much for doing this!

@Julius: You missed the -elapsed flag for openssl.
If you want real numbers, run:

for i in {128,192,256}; do
openssl speed aes-$i-cbc -elapsed;
openssl speed aes-$i-cbc -elapsed -engine af_alg;
done

Simply paste the results enclosed in <code>tags</code>.

Hey there,

I am trying your engine (with an Geode AES on Alix 3D3). It segfaults when running the openssl speed test.

Any idea why?

Thanks Flo

No idea, might get one if you come back with a gdb backtrace and some basic details covering your openssl version and your commands args though.

Hi Markus,

I am using a current Debian 6.0 with a custom kernel 2.6.39.1 and “OpenSSL 0.9.8o 01 Jun 2010”. The syslog says:

 openssl[13308]: segfault at 8 ip 083116d5 sp bfe8a23c error 4
 openssl[13328]: segfault at 92f9d0f ip 09279690 sp bfba3adc error 4

Command line parameters as you describe them:

openssl speed -evp aes-128-cbc -engine af_alg

which dies with this message:

can't use that engine
27363:error:260B806D:engine routines:ENGINE_TABLE_REGISTER:init failed:eng_table.c:161:
Doing aes-128-cbc for 3s on 16 size blocks: zsh: segmentation fault  openssl speed -evp aes-128-cbc -engine af_alg

Thanks! Flo

Custom kernel?

Do you have

CONFIG_CRYPTO_USER_API=m
CONFIG_CRYPTO_USER_API_HASH=m
CONFIG_CRYPTO_USER_API_SKCIPHER=m

and af_alg loaded - or static?

The message you receive indicates there is a problem loading the engine, which makes it likely your custom kernel lacks the af_alg module.

That was it, thank you very much. Maybe you can include that in the README file (even if it is obvious).

[…] this one […]

do you know why is this engine still not available in openssl mainline?

Interesting! Have you tried to test this openssl (aes-128-cbc) boost in openvpn?

I tried the OCF Linux patch on Alix. Actually openssl showed me a boost in keys/sec : from ~ 8000 to ~ 30/40000 but throughput on the tunnel link didn't change so much and in some cases it decreased a bit.

could any of you try to setup a openvpn link with: cipher AES-128-CBC engine af_alg

regards.

Try disabling compression, and use a auth cheaper than sha1.

Adjust your config, and retry.

auth none
#comp-lzo 

The code compiles and I've copied it into /usr/lib/arm-linux-gnueabi/openssl-1.0.0/engines for my setup (Debian Wheezy, Marvell Kirkwood, openssl 1.0.0g).

The af_alg module is loaded into memory before I run the tests.

When I run openssl speed -evp aes-128-cbc -engine af_alg -elapsed I get exactly the same numbers as without af_alg, and the mv_crypto line in /proc/interrupts does not increase when running it. What could be blocking the use of the hardware crypto engine in my setup?

Did you adjust the openssl.cnf?
http://src.carnivore.it/users/common/af_alg/about/

+
+openssl_conf = openssl_def
+
+[openssl_def]
+engines = openssl_engines
+
+[openssl_engines]
+af_alg = af_alg_engine
+
+[af_alg_engine]
+default_algorithms = ALL
+CIPHERS=aes-128-cbc
+DIGESTS=sha1 

Ok that fixed it. Thanks!

And do you know how to specify multiple ciphers in openssl.cnf? I've tried various ways, but never manage to get openssl to recognize the whole list for acceleration with af_alg

@Markus: If it works, please provide numbers for every cipher/digest you can run hardware accelerated to compare.

Multiple ciphers/digests have to be space separated, currently af_alg only supports the ciphers aes-(128|192|256)-cbc and the digest(s) sha1.

I think your Marvell Kirkwood would do aes-128-cbc, des and md5,sha1 in hardware, you can use sha1 and aes-128-cbc.

I get a 'Segmentation fault' when I add multiple ciphers space separated and run openssl

Further more, sha / sha1 do not seem to be hardware accelerated through af_alg on my setup (exact same numbers as without af_alg).

This is a table I made with running openssl 1.0.0g with af_alg on my Kirkwood 1.2GHz CPU (Linux kernel 3.2.0)

aes-128-cbc           8762.28k    10663.36k    11279.53k    11448.32k    11466.07k
aes-128-cbc af_alg     375.19k     1455.87k     4933.80k    12133.72k    21433.00k
aes-128-cbc cryptodev  504.81k     1946.54k     5390.34k    14308.69k    22850.22k
aes-192-cbc           7686.01k     9116.78k     9564.84k     9698.65k     9721.17k
aes-192-cbc af_alg     379.30k     1480.85k     4973.82k    12056.58k    20957.87k
aes-256-cbc           6829.21k     8008.51k     8357.63k     8452.10k     8462.34k
aes-256-cbc af_alg     377.40k     1473.45k     4977.15k    11856.55k    20384.43k
aes-256-cbc cryptodev  506.12k     1951.06k     5345.37k    13790.89k    21624.15k

I've included the results I get with cryptodev API too, which is converted to CryptoAPI using the cryptodev-linux driver.

Hi,

good news first, the segfault for multiple ciphers/digests is fixed, seems I never tested multiple ciphers/digests as I've got only hw acceleration for aes-128-cbc myself.

From your numbers, it is hard to say if using the Kirkwood hardware acceleration is worth it.

For aes-128-cbc, you'd need a blocksize of at least 1024 bytes to have a benefit, else the software implementation would be faster, of course taking more cpu cycles. I'd assume https would benefit in most cases, but I doubt ssh would, where scp could if the files are large.

For sha1, I'd expect af_alg to be slower for small block sizes too. You could test if the af_alg sha1 is used, by adding a printf(“test\n”); into af_alg_sha1_update. If there is no flood of messages when running

openssl speed -evp sha1 -engine af_alg -elapsed

it is not used, and I'd have a look on the default_algorithms in openssl.cnf, should be set to ALL.

Thanks! multiple ciphers now work fine on my setup.

I have noticed that I get a segfault when I try to set the aes-128-ecb cipher in the af_alg section in openssl.cnf though.

Your advice to modify af_alg to show usage for the sha1 digest shows that it indeed is using af_alg. I don't know why the kernel doesn't use mv_cesa to calculate these digests though, as I'm sure mv_cesa supports sha1.

Here are the figures for the digests in openssl on my kirkwood 1.2 GHz machine:

type                  16 bytes     64 bytes    256 bytes   1024 bytes   2048 bytes
sha1                  2759.27k     8620.76k    21044.14k    32818.52k    39171.41k
sha1 af_alg*           205.66k      801.11k     3037.44k     9924.95k    28715.69k
sha1 cryptodev*        132.97k      525.38k     2024.79k     6966.95k    25187.67k
md5                   3560.50k    12510.21k    35354.62k    65143.81k    86349.14k
md5 af_alg*           1759.56k     6565.93k    21600.26k    50325.16k    82173.95k
md5 cryptodev*         135.01k      536.23k     2106.62k     7799.81k    37830.66k

The * denotes that no hardware acceleration was used; I suppose this shows the speed of the digests implemented in openssl vs. those in the linux kernel 3.2.0 + overhead of userspace crypto API

on another note, I've noticed that nginx cannot start when my af_alg_engine section contains anything in openssl.cnf.

I get the following error messages when trying to start nginx:

3059086544:error:260BC066:engine routines:INT_ENGINE_CONFIGURE:engine configuration error:eng_cnf.c:204:section=af_alg_engine, name=CIPHERS, value=aes-128-cbc aes-192-cbc aes-256-cbc
3059086544:error:0E07606D:configuration file routines:MODULE_RUN:module initialization error:conf_mod.c:235:module=engines, value=openssl_engines, retcode=-1

changing the amount of ciphers in the section doesn't make any difference. Only when commenting out both the CIPHERS= and default_algorithms= parameters nginx can start properly.

@Frank:

I have noticed that I get a segfault when I try to set the aes-128-ecb cipher in the af_alg section in openssl.cnf though.

If you provide invalid parameters, the module returns an error code, currently openssl does not exit graceful for such errors but segfaults, nothing I could fix.

For your number, remember md5 is not supported by the openssl af_alg code currently, which is why I'd expect the af_alg md5 numbers to match the native openssl numbers. The (not existing) performance of cryptodev/sha1 is somewhat surprising too, as it outscales af_alg for ciphers.

@Frank: Did you provide a ssl_engine parameter in ngnix config? Should not be required, af_alg will be used where possible given the configuration - and if the software using openssl honors the openssl.cnf. If the problem persists after removing the ssl_engine directive from ngnix configuration, I'd need some more ngnix context for the error from the logs, so I can figure out which part of the ssl startup fails here.

Hi,

I did not set ssl_engine in nginx config, but doing so doesn't seem to make a difference in observed behaviour.

How can I provide you with the extra information?

I've narrowed down the problem to setting any CIPHER= preventing nginx from starting (same result when testing the nginx configuration with nginx -t)

The nginx bug is a nice one, while I can explain why it happens and how to fix, I'm not sure who's bug it actually is, openssl, nginx or mine.

Problems is, simple: nginx calls ngx_ssl_init which calls OPENSSL_config directly. af_alg relies on the algorithms being available already at load-time, so it can map names like aes-128-cbc to the proper NID using openssl. Unfortunately nginx does not call OpenSSL_add_all_algorithms before but after calling OPENSSL_config, therefore the algorithm name mapping facilities of openssl are not initialized when af_alg uses them, and fails at it. Fix is trivial, add a OpenSSL_add_all_algorithms(); on top of af_alg_ctrl. But I'm not sure if this is the right place to fix, it could be be fixed in nginx or even OpenSSL too. I think fixing in af_alg is appropriate, as it would initialize resources it will need later on.

Thanks for feedback.

Hi, I'm trying to get AF_ALG working with my Kirkwood platform (Pogoplug v2) with Linux 3.1.10, I've downloaded and compiled af_alg as described here → http://src.carnivore.it/users/common/af_alg/about/ I've also added appropriate lines to /etc/ssl/openssl.cnf:

#oid_file               = $ENV::HOME/.oid
oid_section             = new_oids

openssl_conf = openssl_def

[openssl_def]
engines = openssl_engines

[openssl_engines]
af_alg = af_alg_engine

[af_alg_engine]
default_algorithms = ALL
CIPHERS=aes-128-cbc
DIGESTS=sha1

unfortunately it doesn't work, and OpenSSH breaks on loading:

:: Starting Secure Shell Daemon                                          [BUSY] Auto configuration failed
1078957292:error:260BC066:engine routines:INT_ENGINE_CONFIGURE:engine configuration error:eng_cnf.c:204:
1078957292:error:0E07606D:configuration file routines:MODULE_RUN:module initialization error:conf_mod.c:235:module=engines, value=openssl_engines, retcode=-1

google doesn't help much either :/ thank you for any and all help

Same problem as outlined above, fixed in git.

I can confirm the version in git to work properly now. Many thanks!

@Frank: Thank You I'll try that later today.

Markus,

I am confused by this statement:

“Hardware accelerated cryptography as provided by VIA Padlock1) and Intel AES-NI2) can be accessed from userspace directly, so you do not need AF_ALG at all…”

How is this done from user space?

Thanks, Amit

VIA Padlock & Intel AES-NI are CPU extensions, which can be used to execute machine code doing cryptography in hardware from userspace. You simply execute the crypto cpu instruction. Well it is not that simple, but it can be done from userspace, and openssl provides engines to do so.

For VIA Padlock, refer to VIA PadLock - Wicked Fast Encryption (Michal Ludvig) as a short introduction how access it from userspace.

AMD Geode crypto is not accessible from userspace, as access is done via DMA (I think), so the only way to access it is via the kernel.

So the kernel can interface Geode crypto, and exports this capability via the af_alg interface to userspace. The af_alg plugin to openssl accesses this kernel interface and makes the accelerated cryptography -which is not accessible from userspace directly- available to your userspace then.

AMD Geode is not the only hardware cryptography which is not accessible from userspace. Marvell hardware crypto embodied in multiple Marvell ARM cores is not accessible from userspace as well.

Markus,

Thanks for clearing that up. That makes sense.

Another question is that, since these are CPU instructions, what is the use of the *padlock-aes* and *padlock-sha* kernel drivers? Are there any benefits to loading these modules?

Thanks.

The kernel needs cryptography for itself for various tasks. Having padlock-aes as kernel module (and a capable cpu) allows the kernel to use hardware acceleration too - very useful for e.g. hard disk encryption via dm-crypt.

Hi, nicely done! I'm trying to get things work, but they won't :(

I'm running an alix 2d board with an AMD Geode LX CPU. Debian Squeeze with backported kernel: 3.2.0-0.bpo.2-486

If I try running your module I'll get:

root@alix:~# openssl speed -evp aes-128-cbc -engine af_alg -elapsed
Error configurin[22881.543786] openssl[810]: segfault at b77b899e ip b7661ae5 sp bfcdd6c0 error 7g OpenSSL
810:e in libcrypto.so.0.9.8[b761a000+129000]rror:260AC089:en
gine routines:INT_CTRL_HELPER:invalid cmd name:eng_ctrl.c:134:
810:error:260AB089:engine routines:ENGINE_ctrl_cmd_string:invalid cmd name:eng_ctrl.c:316:
810:error:260BC065:engine routines:INT_ENGINE_CONFIGURE:engine configuration error:eng_cnf.c:204:section=af_alg_engine, name=CIPHER, value=aes-128-cbc
810:error:0E07606D:configuration file routines:MODULE_RUN:module initialization error:conf_mod.c:235:module=engines, value=openssl_engines, retcode=-1
Speicherzugriffsfehler

I'll think I don't have the module af_alg. Can you tell me where to find it or how it's called? ATM I've set up a virtual Debian Environment and downloaded the kernel build chain for crosscompiling the required modules.

810:error:260BC065:engine routines:INT_ENGINE_CONFIGURE:engine configuration error:eng_cnf.c:204:section=af_alg_engine, name=CIPHER, value=aes-128-cbc

I think you have an error in your config, you name the key CIPHER where CIPHERS would have been appropriate.

The resulting segfault is openssl invalid error handling for such case.

So, check your config. For the module itself, lsmod | grep af_alg modprobe af_alg lsmod | grep af_alg

Found the module and included it in /etc/modules. You were right, was a spelling typo. And now it works!

openssl speed -evp aes-128-cbc -elapsed -engine af_alg 
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-128-cbc        541.34k     2064.32k     7452.54k    21461.73k    44265.16k


openssl speed -evp aes-128-cbc -elapsed 
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-128-cbc       4816.32k     6985.52k     7902.42k     8184.16k     8262.88k

I used the bellow but don't have any performance increase?

# lsmod
Module                  Size  Used by
i2c_dev                12891  0 
cpuid                  12506  0 
lm90                   17151  0 
af_alg                 12844  0 
tun                    17639  2 
raid1                  25968  1 
md_mod                 85049  2 raid1
ecb                    12649  0 
scx200_acb             12571  0 
cs5535_mfgpt           12569  0 [permanent]
i2c_core               19022  3 i2c_dev,lm90,scx200_acb
cryptd                 14049  0 
aes_i586               16608  0 
aes_generic            37066  1 aes_i586
geode_aes              12862  0 
geode_rng              12436  0 
rng_core               12550  1 geode_rng
cs5535_mfd             12512  0 
ext4                  300227  2 
mbcache                12810  1 ext4
jbd2                   51388  1 ext4
crc16                  12327  1 ext4
dm_mod                 56751  6 
usb_storage            34910  2 
uas                    12975  0 
sd_mod                 35060  6 
crc_t10dif             12332  1 sd_mod
ata_generic            12439  0 
pata_cs5536            12559  0 
ohci_hcd               21855  0 
pata_amd               13076  1 
libata                124073  3 ata_generic,pata_cs5536,pata_amd
ehci_hcd               34912  0 
usbcore               103418  5 usb_storage,uas,ohci_hcd,ehci_hcd
scsi_mod              134556  4 usb_storage,uas,sd_mod,libata
via_rhine              22182  0 
mii                    12595  1 via_rhine
usb_common             12338  1 usbcore

# tail -n 13  /etc/ssl/openssl.cnf 

openssl_conf = openssl_def

[openssl_def]
engines = openssl_engines

[openssl_engines]
af_alg = af_alg_engine

[af_alg_engine]
default_algorithms = ALL
CIPHERS=aes-128-cbc
DIGESTS=sha1 

# openssl speed -evp aes-128-cbc -elapsed -engine af_alg 
engine "af_alg" set.
You have chosen to measure elapsed time instead of user CPU time.
To get the most accurate results, try to run this
program when this computer is idle.
Doing aes-128-cbc for 3s on 16 size blocks: 1024896 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 64 size blocks: 327443 aes-128-cbc's in 3.01s
Doing aes-128-cbc for 3s on 256 size blocks: 88929 aes-128-cbc's in 3.01s
Doing aes-128-cbc for 3s on 1024 size blocks: 22653 aes-128-cbc's in 3.00s
Doing aes-128-cbc for 3s on 8192 size blocks: 2838 aes-128-cbc's in 3.00s
OpenSSL 0.9.8o 01 Jun 2010
built on: Sun Jan 22 12:57:18 UTC 2012
options:bn(64,32) md2(int) rc4(idx,int) des(ptr,risc1,16,long) aes(partial) blowfish(idx) 
compiler: gcc -fPIC -DOPENSSL_PIC -DZLIB -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DL_ENDIAN -DTERMIO -O3 -march=i586 -Wa,--noexecstack -g -Wall -DOPENSSL_BN_ASM_PART_WORDS -DOPENSSL_IA32_SSE2 -DSHA1_ASM -DMD5_ASM -DRMD160_ASM -DAES_ASM
available timing options: TIMES TIMEB HZ=100 [sysconf value]
timing function used: ftime
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-128-cbc       5458.83k     6969.19k     7570.94k     7724.50k     7739.31k

git clone http://src.carnivore.it/users/common/af_alg
cd af_alg
gcc -O3 -Wall -fPIC -c -o e_af_alg.o e_af_alg.c
gcc -shared -Wl,-soname,libaf_alg.so -lcrypto -o libaf_alg.so e_af_alg.o
cp libaf_alg.so /usr/lib/ssl/engines
chmod 644 /usr/lib/ssl/engines/libaf_alg.so
ls -hal /usr/lib/ssl/engines/

Don’t know whats wrong

Config looks valid, but does not look like af_alg is used by openssl properly at all.

Try adding some printfs to the init functions to figure out if the af_alg engine initialises correctly.

@root: Could you provide a pastbin with the e_af_alg.c you want me to try? I use an LX800 with http://www.pcengines.ch/alix6e2.htm Debian 6.0 OpenSSL 0.9.8o 01 Jun 2010

Hi,

thank you very much Markus! It works great here on my Alix 3d2.

@Jelle: at first i added the openssl config options to the end of the openssl.cnf like you and it didn't work … until i inserted them in the right spot. See the readme for the lines before and after them.

I have patched my kernel (2.6.32.13) with patch published here http://lwn.net/Articles/410536/.

I have made the .so library of your code set it to my openssl. I try to test with openssl speed, but I see that in af_alg_aes_init_key that bind fails with (No such file or directory) error.

Do you have any hint?

I have patched my kernel (2.6.32.13) with patch published here http://lwn.net/Articles/410536/.

I have made the .so library of your code set it to my openssl. I try to test with openssl speed, but I see that in af_alg_aes_init_key that bind fails with (No such file or directory) error.

Do you have any hint?