RSA algorithm in C: Decryption and mathematical formula