Mfcttrf

I would like to audit how an implementation of an encryption algorithm is really performed with the following given data of the problem:

• the encryption mechanism is reversible,


• the algorithm is pretended to be AES256, but might be implemented correctly or not, or worse something else,


• the key is not known,


• I don't have access to the source code.

At least I would like to be able to detect that the file is not encrypted.
Next I would like to be able to detect that the RNG is running in a tiny set or not really random.

In a first approach I thought to make an analysis of the randomness quality of the encrypted file: average value + standard deviation (with a tool like ent).
But I immediatly thought of artificial files with a perfect average value
and standard deviation which are perfectly regular and not the result of any encryption.
Then my first approach is wrong.

The environment in which I will make this audit is a Unix one.

Unless the file has a plaintext header which indicates that it has been encrypted, there is no way to distinguish ciphertext from uniform random data. You can heuristically guess that a file is encrypted if it has absolutely no structure and appears completely random, but you cannot definitively prove it.

Any cipher whose output could be distinguished from random would be considered broken.

If you can't get access to the key for at least some sample uses, there's no way to be sure. For example, it's impossible to distinguish AES-128 from AES-256 if you don't have access to the key. That's true of any encryption method: without knowing the key, you can't distinguish the ciphertext from random data of the same length.

A professional auditor would normally get access to some test keys, if they can't normally access those keys through some administration interface.

You can make a statistical test, but all this will tell you is that the encryption is not completely botched or skipped.

If your vendor is not completely dishonest and they claim to have used AES, they probably did use AES. A far more common problem than not using AES is using AES wrong. Here too, you can't be sure that they got it right, but you can at least check for some common problems.

Check how the length of the ciphertext varies depending on the length of the plaintext. The ciphertext should include an initialization vector and an authentication tag, each of which normally adds 16 bytes. If the ciphertext is not 32 bytes larger than the plaintext, something is probably wrong, but there are cases where it can be ok (e.g. for disk encryption where the sector number is used to build unique IVs and no threat requires authentication).

Pass the same inputs in different conditions and make sure that the resulting ciphertexts are completely different. If you can arrange to encrypt multiple messages with the same key, make sure that identical messages result in completely different ciphertexts. This validates that initialization vectors are generated, if not correctly, then at least non-stupidly.

If there's a way for you to submit modified ciphertexts, do that and check that they are rejected with a generic “invalid ciphertext” error, rather than a specific error due to invalid content. This validates that authenticated encryption is used. There are threat models where it's ok not to have authentication, but you need to tread very carefully.

Two things that you definitely cannot know by looking at the output is whether the keys are generated and stored securely. (As opposed to e.g. using a non-cryptographic random generator to generate keys, and writing a copy of the secret keys to an unprotected location.) You can only audit this by looking at the behavior of the system.

In addition to what the other answers have stated, "proper" encryption using AES-256 (block mode choice aside) can still allow backdoors, such as by maliciously choosing IVs/nonces. Phil Rogaway and others discuss this in more detail in their paper "Security of Symmetric Encryption against Mass Surveillance" (abstract available here).