A register is a special small memory unit; usually within a CPU, i.e. not part of the machine's main memory, although device controllers usually include registers too. For the ones which are part of the CPU, access to them is very fast - usually faster than main memory.
The first computers did not include many registers in the CPU, since each register needs separate hardware, and that was a scarce resource early on. Many machines only had a single register available for doing computations with, the accumulator; these machines were typically a load-store architecture.
Later on, hardware was not so much of an issue, but computers still tend to have somewhat limited numbers of programmer-visible registers, because they are named by fields in instructions, and bits in the instruction word are (and always will be) a limited resource. (Register renaming was developed to allow more registers to be used, without being able to explicitly name them.)