Of relative size and origin

Origin: Bricks and blocks are manufactured whereas stones and rocks are not; pretty obvious if you speak English.

Size: Relative to both categories. In the first: bricks are small enough to be picked up by the average hand: whereas two hands would usually be used to lift a block; now apply this test respectively to stones and rocks.

Simples!