Do the speakers in a surround system need to be in a set?

Do I have to buy a set for 5.1 surround audio, or can I just buy 5 speakers and one subwoofer separately to potentially save money and link them to my AVR?