Bouncy Castle is throwing "mac check in OCB failed" when trying to decrypt an AES message

Answer #1 100 %

There are two issues, one is a bug in your Kotlin code, the other is a library bug:

Bug in Kotlin code

While PyCryptodome processes ciphertext and tag separately, BC/Kotlin expects the concatenation of both in the order: ciphertext|tag.
Therefore the line encryptedMessage += tag must be added in the Kotlin code:

fun decryptAesMessage2(sharedKey: String, encryptedMessageData: Map): ByteArray {
    var encryptedMessage = encryptedMessageData["encryptedMessage"]!!.utf8Base64Decode()
    var tag = encryptedMessageData["tag"]!!.utf8Base64Decode()
    encryptedMessage += tag // Fix
    var nonce = encryptedMessageData["nonce"]!!.utf8Base64Decode()

    var key = KeyParameter(sharedKey.toByteArray(Charsets.UTF_8))
    var params = AEADParameters(key, tag.size*8, nonce)
    var cipher = OCBBlockCipher(AESEngine(), AESEngine())

    cipher.init(false, params)

    val out = ByteArray(cipher.getOutputSize(encryptedMessage.size))
    var offset = cipher.processBytes(encryptedMessage, 0, encryptedMessage.size, out, 0)
    offset += cipher.doFinal(out, offset) // Throwing exception here

    return out

Test: Below, identical test data is successfully decrypted using the Python code and the fixed Kotlin code:


encrypted_message = {
  'encryptedMessage': 'LzoelJ9Nv4cruj0JUlxFrNR+mqyO2rvwqDHYwnj0OkvJ+BBvug+ORYVkxA==', 
  'tag': 'hl56drXePWiLkVavVwF3/w==', 
  'nonce': b64encode(b'012345678901').decode()
dt = decrypt_aes_message('01234567890123456789012345678901', encrypted_message)
print(dt) # The quick brown fox jumps over the lazy dog


val encrypted_message = mutableMapOf()
encrypted_message["encryptedMessage"] = "LzoelJ9Nv4cruj0JUlxFrNR+mqyO2rvwqDHYwnj0OkvJ+BBvug+ORYVkxA=="
encrypted_message["tag"] = "hl56drXePWiLkVavVwF3/w=="
encrypted_message["nonce"] = Base64.getEncoder().encodeToString("012345678901".toByteArray(Charsets.UTF_8))
val dt = decryptAesMessage2("01234567890123456789012345678901", encrypted_message)
println(String(dt, Charsets.UTF_8)) // The quick brown fox jumps over the lazy dog

Library bug

Another problem is that both implementations produce different results for a nonce length of 120 bits (15 bytes), the maximum allowed nonce length for OCB (see RFC 7253, 4.2. Encryption: OCB-ENCRYPT):

The following test uses a fixed key and plaintext and compares the results between the Python and BC/Kotlin code for nonce lengths 13, 14, and 15 bytes:

Key: b'01234567890123456789012345678901'
Plaintext: b'testmessage'

Nonce Länge     13 bytes                                                    14 bytes                                                    15 bytes
Nonce           b'0123456789012'                                            b'01234567890123'                                           b'012345678901234'
Python (ct|tag) 0xb35a69a245ab18fe3b6bae38b179c2a43b341f67c0451256b76bd7    0xff9be97fcb6e1ac57e6997bc3e84598a83ab70947ccac500fcf75e    0xff9be97fcb6e1ac57e6997bc3e84598a83ab70947ccac500fcf75e
BC/Kotlin       0xb35a69a245ab18fe3b6bae38b179c2a43b341f67c0451256b76bd7    0xff9be97fcb6e1ac57e6997bc3e84598a83ab70947ccac500fcf75e    0xa4355068324065f2ad194b058bdb86caa67c225b99021dbd588034

The Python implementation returns the same ciphertext/tag for nonce lengths 14 and 15, while the results for BC/Kotlin differ. This indicates a bug in the Python implementation.

Unfortunately, RFC 7253, Appendix A. Sample Results only provides test vectors whose nonces are all 12 bytes in size, so the bug cannot be assigned more clearly.

I.e. if you use a 15 bytes nonce, both implementations are not compatible, where the problem is most likely caused by the Python implementation.


Analysis of linked example:

You are affected by both bugs. According to your comment, you have already fixed the bug in the Kotlin code (concatenation of ciphertext and tag). Since the nonce you used in the example is 15 bytes (knQgYf1MsOs8smx9GtWM corresponds Base64 decoded to 0x92742061fd4cb0eb3cb26c7d1ad58c) the bug described in the second part of my answer is the cause of the problem. This bug is not in your code, but in one of the libraries, most likely in the Python library. Therefore you cannot fix it (at least not without greater effort).


As shown in the above test, the Python code for a nonce length of 15 bytes seems to simply ignore the 15th byte, i.e. if you use the 14 bytes nonce knQgYf1MsOs8smx9GtU= (0x92742061fd4cb0eb3cb26c7d1ad5) in your Python code, the Python code returns the same ciphertext and the same tag as with the 15 bytes nonce knQgYf1MsOs8smx9GtWM, which is why decryption with the 14 bytes nonce is also possible:

key = "f009Cip5hM4Obbb6E2MT5npJBHlc82vD"
message_data = {"encryptedMessage":"XMQx/xbVVTbMdpMiTXVp5XPICm11Vw2pgALpVI0NgbdqLLmikhPuu9M+qQzyOVZlZZBRlscijpyAZDsLGcTSPP54O35oKNp//PuOrWsN/ZZMkCByKCSBysJLRiZV1OjZDg01gi5/nYNbUgGGd8uRGKfBaKjjXngZ1J89GOvDeWPQcjbfbdzd9w+jbZGZ5jnAIChOL1Uqohf+6KHtjR/H06fFTHwB1abzAQrGbCNBNXBmN9+zEu7Auy3NPWKrZ+SL5Nk=","tag":"ZcqXSBqYU5TjgdMC+bMeUQ==","nonce":"knQgYf1MsOs8smx9GtU="}

decrypted_message = decrypt_aes_message (key, message_data)
print (decrypted_message) # https://app.passiv.com/snapTrade/redeemToken?token=v9uJsXYsi%2B6s9kyohisc6DFntJ/yD6m/2zhmO5xp6Vmezcyi8nwx63YtkqnnaogZvFmqs7L99EtZ0mxN9mAQTNoThHj3GaypXXUdiQIzig%3D%3D&clientId=SCANZ&broker=ALPACA

If you use this 14 bytes nonce knQgYf1MsOs8smx9GtU= in the Kotlin code, the decryption is also successful. This is the workaround for this specific example!

The general workaround, as long as the library bug is not fixed, is not to use a 15 bytes nonce, but only a maximum 14 bytes nonce!

On this website there are also testvectors with 15 bytes nonces listed. As expected, these are not met by the PyCryptodome implementation. I have filed an issue on PyCryptodome based on one of the 15 bytes nonce test vectors: Issue #664.

You’ll also like:

© 2023 CodeForDev.com -