From Computer History Wiki
A stack is a area of storage which is used to temporarily hold data in a 'push-down' fashion; i.e. data can be pushed onto the stack, and then popped off of it (in the reverse order from which it was pushed) when it is needed. (E.g. a 'push' of item A, and then B, would then be followed by a 'pop' of B, then A.)
Stacks are usually stored in main memory, but in some modern machines with many registers, the top of the stack (or a cache of the top) may be held in them.
Modern CPUs generally dedicate one register to be a stack pointer, to point to the top of the stack; and instructions usually have addressing modes which allow source operands to be popped from the stack, and the destination of the result to be a push onto the stack.