Two of my favourite albums EVER are collections of the band's non-album single releases (like the A and B sides).

They are:

Les Savy Fav - Inches
Superchunk - Seed Toss

What others are out there, and are any of them as brilliant as the above? (I doubt it)