A register is a special storage location; usually within a CPU, i.e. not part of the machine's main memory, although device controllers usually include registers too. Since they are part of the CPU, access to them is very fast - usually faster than main memory.
The first computers did not include many registers in the CPU, since each register needs separate hardware, and that was a scare resource early on. Many machines only had a single register available for doing computations with, the accumulator; these machines were typically a load-store architecture.
Later on, hardware was not so much of an issue, but computers still tend to have somewhat limited numbers of registers, because they are named by fields in instructions, and bits in the instruction word are (and always will be) a limited resource.