I want to performance test (aka bake-off) MySQL server rpm against some other forks such as Percona server, MariaDB, and possibly some others. I’m hoping that by asking this question I can better understand the methodology behind setting up a proper performance test. I planned on using sysbench to run my actual test, but I’m open to anything.
- What steps should I be taking to
ensure that the test results in a
apples-to-apples comparison and that
only the RDBMS is the variant?
- Where do I get started?
- How do I evaluate results?
- What advice can you give me?
Here is the first thing I thought about
Setup 3 DB servers (identical HW/OS configs) each installed with
Get a Fourth Server installed with MONyog (eval version lasts 30 days)
Register the 3 DB servers in MONyog
Use SysBench against all 3 DB Servers at your discretion
This is just a basic outline. Believe me, you can use Monyog right out of the box. Check out my testimonial at MonYog’s Website (Look for my name on the page). I eventaully convinced my company to purchase the Utlimate Version outright after my testimonial, which they watched with me as it happened.
Caveat : Do no get Monyog until you have the 3 DB servers and Sysbench Ready to get the best 30-day usage of MonYog.
UPDATE 2011-06-20 13:18 EDT
- Queries, Questions
- SELECTs ((Com_select + Qcache_hits) / Questions)
- INSERTs ((Com_insert + Com_replace) / Questions)
- UPDATEs (Com_update / Questions)
- DELETEs (Com_delete / Questions)
Many of these are default graphs in MONyog’s Monitors/Advisors Section, and more graphs can be created to check other desired MySQL metrics. MONyog also lets you export the graphs as CSV data.
UPDATE 2011-10-12 14:17 EDT
- MySQL 4.1
- MySQL 5.0
- MySQL 5.1 (with built-in InnoDB)
- MySQL 5.1 with InnoDB-plugin
- MySQL 5.5
- MySQL 5.6
All tests were performed with MySQL unconfigured. The results?
- MySQL 4.1 performs the best single-threaded
- MySQL 5.1 with InnoDB plug-in scales on multiple cores better than 5.1 InnoDB built, 5.5 and 5.6
What do I get out of this? Your must configure MySQL 5.5/5.6 to get multicore enhancements engaged.
This should be very helpful to you as this is the defacto industry benchmark for databases, TPC-C, http://www.tpc.org/tpcc/spec/tpcc_current.pdf
Even if you do not use execute this benchmark, the information contained within the test definition document should provide you with tons of insight into setting up your own benchmark for testing your database options.