padding and buffer size for Cryptoapi CryptEncrypt()

[a***@yahoo.com]

I'm writing encryption/decryption functions using RSA keys (2048) and
RC2 algorithm, but decryption exits with NTE_BAD_DATA. Strangely
enough, it works ok if I decrypt a message from my own computer to my
own public key and everything worked fine with the Base Provider.
Maybe I should size the encryption buffer to take account of the
padding, but I can't understand what these MDSDN excerpts mean:

-----------------
In Windows 2000, the Microsoft® Enhanced RSA Provider supports direct
encryption with RSA public keys and decryption with RSA private keys.
The encryption uses PKCS #1 Type 2 padding. On decryption, this padding
is verified. The length of plaintext data that can be encrypted with a
call to CryptEncrypt with an RSA key is the length of the key modulus
minus eleven bytes. The eleven bytes is the chosen minimum for PKCS #1
padding.

...The length of ciphertext data to be decrypted must be the same
length as the modulus of the RSA key used to decrypt the data. If the
ciphertext has zeros in the most significant bytes, these bytes must be
included in the input data buffer and in the input buffer length...
-----------------

Any suggestions about how I should size my decryption (encryption)
buffer, assuming that it _is_ indeed a padding problem?

Thanks!

Andrew

[lelteto]

I assume you don't want to encrypt DATA with your RSA key. Here's what you
normally do:

on one side
1. You generate a RANDOM SESSION KEY for RC2 (CryptGenKey)
2. You encrypt (wrap) the session key with your RSA public key
(CryptExportKey)
3. You encrypt your data with the session key (CryptEncrypt)
now you send both the exported session key AND the encrypted data

on the receiving side
1. You decrypt the session key with the RSA private key (CryptImportKey)
2. You decrypt the data with the recovered RC2 session key (CryptDecrypt)

Of course, there is a bit more (eg. normally the sender gets the receiver's
public key from a certificate which needs to be imported. Note that the
sender can work with a CRYPT_VERIFYCONTEXT context while the receiver would
open the container which has its private key.

Also, above I assumed both sides use CAPI. If not, you would need to do some
byte-swapping (as CAPI works little endian; everybody else works in big
endian = network byte order).

Laszlo Elteto
SafeNet, Inc.

Post by a***@yahoo.com
I'm writing encryption/decryption functions using RSA keys (2048) and
RC2 algorithm, but decryption exits with NTE_BAD_DATA. Strangely
enough, it works ok if I decrypt a message from my own computer to my
own public key and everything worked fine with the Base Provider.
Maybe I should size the encryption buffer to take account of the
-----------------
In Windows 2000, the Microsoft® Enhanced RSA Provider supports direct
encryption with RSA public keys and decryption with RSA private keys.
The encryption uses PKCS #1 Type 2 padding. On decryption, this padding
is verified. The length of plaintext data that can be encrypted with a
call to CryptEncrypt with an RSA key is the length of the key modulus
minus eleven bytes. The eleven bytes is the chosen minimum for PKCS #1
padding.
....The length of ciphertext data to be decrypted must be the same
length as the modulus of the RSA key used to decrypt the data. If the
ciphertext has zeros in the most significant bytes, these bytes must be
included in the input data buffer and in the input buffer length...
-----------------
Any suggestions about how I should size my decryption (encryption)
buffer, assuming that it _is_ indeed a padding problem?
Thanks!
Andrew

[a***@yahoo.com]

Thanks for clearing up the confusion. This is in fact what I am doing:
I have functions that worked fine the complete encrypt/decrypt process
(CAPI both sides) using the Base CSP. I made minor modifications to use
the Enhanced CSP with longer keys, and I must have messed up some other
detail, as encryption/decryption will only work on the same machine,
when I feed the encryption with its own public key. However, the
decryption by a different PC running the same software will fail with
NTE_BAD_DATA.

Andrew

Post by lelteto
I assume you don't want to encrypt DATA with your RSA key. Here's what you
on one side
1. You generate a RANDOM SESSION KEY for RC2 (CryptGenKey)
2. You encrypt (wrap) the session key with your RSA public key
(CryptExportKey)
3. You encrypt your data with the session key (CryptEncrypt)
now you send both the exported session key AND the encrypted data
on the receiving side
1. You decrypt the session key with the RSA private key (CryptImportKey)
2. You decrypt the data with the recovered RC2 session key (CryptDecrypt)
Of course, there is a bit more (eg. normally the sender gets the receiver's
public key from a certificate which needs to be imported. Note that the
sender can work with a CRYPT_VERIFYCONTEXT context while the receiver would
open the container which has its private key.
Also, above I assumed both sides use CAPI. If not, you would need to do some
byte-swapping (as CAPI works little endian; everybody else works in big
endian = network byte order).
Laszlo Elteto
SafeNet, Inc.

[a***@yahoo.com]

I've found out that everything works as long as the two users that
exchange files use the same os i.e. XP <> XP, but not, for example XP
<> 2000. I wish I knew why: maybe different CSP versions are
incompatible?

Andrew

[lelteto]

You may run into the known KEY SIZE issue. You should set the key size
EXPLICITLY (CryptSetKeyParame with KP_EFFECTIVE_KEYLEN). Hopefully that may
fix your problem.

Laszlo Elteto
SafeNet, Inc.

Post by a***@yahoo.com
I've found out that everything works as long as the two users that
exchange files use the same os i.e. XP <> XP, but not, for example XP
<> 2000. I wish I knew why: maybe different CSP versions are
incompatible?
Andrew

[a***@yahoo.com]

Thank you, you hit the nail on the head!
I explicitly set CryptSetKeyParamer with a 128 bit key for encryption
and decryption and messages are encoded/decoded ok from/to XP/2000 and
XP/XP.
Andrew

Post by lelteto
You may run into the known KEY SIZE issue. You should set the key size
EXPLICITLY (CryptSetKeyParame with KP_EFFECTIVE_KEYLEN). Hopefully that may
fix your problem.
Laszlo Elteto
SafeNet, Inc.

Post by a***@yahoo.com
I've found out that everything works as long as the two users that
exchange files use the same os i.e. XP <> XP, but not, for example XP
<> 2000. I wish I knew why: maybe different CSP versions are
incompatible?
Andrew