Which is more repeatable but less representative. IMO a non-shared EC2 instance is probably the way to benchmark (if you can drive enough load to saturate it).
Hmm. Some cloud provider ought to provide (for a premium price) guaranteed-identical non-shared configurations... including small groups of machines with uncontended, identical cross-connects.
(I know it's then close to 'dedicated' hosting, but people running benchmarks also want the quick setup and discard of virtualized instances. This would be a hybrid offering that ensures their cloud services is always chosen for such benchmarking comparisons. Of course this offering would not be good for cross-cloud comparisons, because it's not representative of their usual offerings.)
