A new data resource for measuring the accuracy of metagenomic binning methods, created by in vitro-simulation of a metagenomic community, can be used to complement previous in silico benchmark studies. In constructing a synthetic community and sequencing its metagenome, researchers from the University of California Davis and DOE's Joint Genome Institute encountered several sources of observation bias that likely affect most metagenomic experiments to date and present challenges for comparative metagenomic studies. The researchers summarize that DNA preparation methods have a particularly profound effect implying that samples prepared with different protocols are not suitable for comparative metagenomics.