At the simplest level: An encryption algorithm doesn’t concern itself with whether or not the right key was provided*. It takes a key, some encrypted data, and then spits out whethever the math says it should.

Programmers will build on top of an algorithm to add more complex security features. An example of which might be “Tell the user whether or not their key actually worked” or “Tell the user if someone tampered with the encrypted data”.

The actual implementation of these security features is different for every situation, and can get quite complex.

Here is a very simple example of what someone might do:

1. Take the data that someone wants to encrypt eg “hello world”
2. Put a known constant value at the beginning of the data eg using the constant “sentinel” with “hello world” becomes “sentinelhello world”
3. Encrypt everything together

Then when decrypting, you look for the word “sentinel” at the beginning, and then spit back everything after that. If the word “sentinel” isn’t the first thing you see, then you know the key is incorrect.

In the case of AES algorithm, it has a special way of padding the plaintext before encrypting. If the padding doesn’t show up after decrypting, then the key is incorrect.

* A general statement, not necessarily representative of common encryption algorithms

Encryption, in regards to computers, is a massive topic so I’ll take two individual examples:

• A file/disk is encrypted and the user wants to access it - Because the file is encrypted, the OS by itself, cannot open it and display the contents unencrypted. A software utility will challenge the user for the password, passphrase, or key. That will be stored in RAM in the computer (were the contents disappear when the computer is turned off). The decryption is done, and usually the utility discards the key. Another way this can be done is if the key is already stored elsewhere in the computer box. Such as an additional tiny computer called TPM (Trusted Platform Module). These have very little computing power, but the main computer can ask for the TPM to decrypt something so the main computer never knows the key.

• You want to decrypt the contents of a secure website being served to you. This is an example of asymmetric cryptography. There is one key that can encrypt and decrypt contents. This is called a Private Key. There is another key that can only decrypt those same contents. This is called a Public Key. The web site operator will use their private key (in an SSL cert) to encrypt the website contents. When your computer downloads the encrypted contents, theres a note telling where the computer can retrieve the Public Key. Your compute downloads the Public Key, and decrypts the website.

I mean, do you know the Caesar cipher? It needs a key, the key just being how many letters to shift by. Modern symmetrical keys are the same, just much more sophisticated so clever Greeks can’t beat them.

The code is never encrypted while running in anything common. Stuff is just encrypted on the hard drive or SSD before it’s loaded, and then erased in it’s decrypted state when you lock or shut down your device. The code to decrypt the code is also not encrypted, which is fine because it’s super standardised anyway.

Cybersecurity engineer here, these are all great answers I just wanted to add a couple tidbits.

• Where are the keys stored: In enterprise level security, you never want to store your key in the same system you are securing, so you normally use a “Key manager” or an “HSM” (Hardware security module) These are hardened appliances dedicated to generate keys and store them, then other devices that then use protocols such as KMIP or API calls to retrieve them remotely. The key is encrypted in transfer and never stored permanently in the client that is being encrypted. There’s also a key encryption mechanism … but let’s not goo too far into it. I’ll just mention that there’s usually a data encryption key (DEK) and a Key Encryption Key (KEK) and the DEK is never exposed. HSMs themselves are basically physical key managers that have a tamper proof crypto module built in. Think of them as TPM2 on steroids. If you want to read more about these guys check out the FIPS specification that they’re built to. https://en.wikipedia.org/wiki/FIPS_140-3

• How does the OS know which key to use: It doesn’t There is usually overlay software than handles this part, typically called an “encryption agent” that runs in the kernel space. Even tho most software is based on open source, encryption agents are usually very complex and secretive. My company’s filesystem encryption agent for example embeds 4Kb of metadata on every file with the name of the key that was used to encrypt it to prevent double encryption and help in file restoration. When a file request comes into the OS, we are loaded into the kernel and use a “filter driver” to decide if the file is encrypted or not, and if it’s encrypted, we route the file request to a virtual file system that then pipes it to our software for processing ACLs and decryption. If you want to learn more you can check out this brochure. https://cpl.thalesgroup.com/encryption/transparent-encryption

Happy to answer any questions.