Magento Enterprise Edition (EE) 1.13 Benchmarking Guide header

Magento 1.x Software Support Notice

For Magento Commerce 1, Magento is providing software support through June 2020. Depending on your Magento Commerce 1 version, software support may include both quality fixes and security patches. Please review our Magento Software Lifecycle Policy to see how your version of Magento Commerce 1 is supported.

For Magento Open Source 1.5 to 1.9, Magento is providing software security patches through June 2020 to ensure those sites remain secure and compliant. Visit our information page for more details about our software maintenance policy and other considerations for your business.

Magento Enterprise Edition (EE) 1.13 Benchmarking Guide

Customer Needs and Expectations

The focus of the Magento Enterprise Edition 1.13 release is performance and scalability. The benchmarking results presented here demonstrate that we have addressed the following concerns:

  1. The eCommerce landscape is changing, and merchants must provide customers with an online shopping experience that meets their performance expectations
  2. As merchants grow their businesses and increasingly larger enterprise merchants are adopting Magento, Magento Enterprise Edition must scale to handle increased traffic volume and larger catalogs

Magento's Plan

We identified the following areas for enhancement in Magento Enterprise Edition 1.13:

  • Improve indexing for catalogs of all sizes
  • Make re-indexing operations invisible to the shopper by executing them in the background
  • Improve full page caching support
  • Reduce page load times for key shopping flows (checkout)
  • Enable merchants to serve heavier traffic volume without need to purchase additional hardware

What Magento Accomplished

Magento Enterprise Edition 1.13 delivers the following improvements over Magento Enterprise Edition 1.12:

  • Incremental re-indexing has been introduced with Magento Enterprise Edition 1.13, and scenarios where full-re-indexing was required have been limited. This means that operations that previously took hours can now be completed in minutes
  • 53% improvement in completion times for a full re-index with a 500,000 SKU catalog
  • 35% improvement in "Place Order" performance
  • 65% improvement in page load times across shopper flow pages that were tested.
  • 33% improvement in orders per day that can be processed on the same hardware configuration as Magento Enterprise Edition 1.12
  • 31% improvement in page views per day supported on the same hardware configuration on Magento Enterprise Edition 1.12

Multi-Node Deployment Topology

Before we introduce you to the benchmarks, an overview of our multi-node benchmarking facility is in order. We started with a basic multi-node cluster with load balancer and caching and separate DB node, and installed Apache. This is a familiar hardware configuration for the Magento developer community. Once provisioned, we installed both Magento Enterprise Edition 1.13 and Magento Enterprise Edition 1.12 on the cluster to compare their performance.

This is a representation of the multi-node Magento Enterprise Edition 1.12/1.13 installation used for performance testing. It consists of four physical nodes, three of which are virtualized.

We used HP ProLiant SL230s Gen8 servers with 8x16GB DDR3 and 2x Intel Xeon E5‐2660 CPUs for our benchmarking.

We used a physical disk of a usable 1.2 TB managed by a RAID10 controller. The load balancer we used is nginx.

The cluster resides in our Las Vegas data center, connected to the Internet via a gigabit connection. We tested the cluster using the popular Gatling suite from our engineering offices in Austin, TX.

Software Components

Scenarios

This section presents the merchant scenarios we simulated for our testing. These scenarios are based on real-world experience and industry standards with respect to shopper flows and catalog sizes.

Shopper Flows

We used real-world, established eCommerce metrics to simulate shopper flows.

  • 94% of eCommerce shoppers visited a storefront but did not purchase any products
  • 65% of shoppers added items to their carts but abandoned them
  • Of the 6% who did purchase products (otherwise known as the conversion rate), half checked out as guests without signing in, and half signed into the storefront.

Catalog

The catalogs we simulated also reflect real-world, established eCommerce experience.

  • We simulated small and medium-sized companies with a 50,000-item catalog with 27 product categories.
  • We simulated large companies 500,000 items in the catalog with 2,000 categories.
  • As many eCommerce sites offer different types of products, we specified physical (simple) products, virtual products, and downloadable products, with 60% of the products in each catalog being physical.
  • We benchmarked each catalog on one website/storefront.

Target Merchant Profile

The simulated merchant profile we benchmarked against represents the profile of our enterprise customers. We established these metrics based on the day-to-day experience of large merchants operating a successful eCommerce business. Again, we used what we consider to be a typical hardware and software configuration.

  • 50K visitors / day
  • 1M Page views / day
  • 18000 orders / day
  • 3000 orders during peak 4 hours
  • 1000 concurrent users
  • Standard HW/SW configuration

Benchmarking Results

These benchmarks were generated during extensive testing of our multi-node configuration running Magento Enterprise Edition 1.13 described above. Our configuration was modeled after what a commercial hosting partner would put into production, and the results reflect accurate gains over Magento Enterprise Edition 1.12.

The duration of the test sessions is 72 seconds, which significantly stresses the multi-node configuration.

Incremental Re-indexing

In Magento Enterprise Edition 1.12, any change to a product would result in a full re-index. Magento Enterprise Edition 1.13 introduces a new feature--incremental re-indexing. With incremental re-indexing, only those items that were changed or added will be re-indexed, reducing the processing time to a fraction of what was required before.

Take the example of a merchant with a catalog containing 500,000 products. In Magento Enterprise Edition 1.12, any change to a product would result in a full re-index operation. In Magento Enterprise Edition 1.13, incremental re-indexing means the merchant will only re-index items that were changed. The test focused on measuring the improvements provided by the incremental re-indexing feature in Magento Enterprise Edition 1.13. The table below compares improvements to common admin actions, such as changing a product description, prices or inventory.

Full Re-indexing

As part of the benchmarking effort, we also measured the improvements in the full re-indexing feature on Magento Enterprise Edition 1.13, where indexing a 500,000-item catalog was 53% faster than Magento Enterprise Edition 1.12. Faster re-indexing means less load on the system and that changes to the catalog propagate faster to the storefront.

  • 500,000 products
  • 2,000 categories
  • One store
  • One catalog update during test run

Individual Re-indexers

The Magento Enterprise Edition 1.13 indexer component contains a number of individual indexers such as Product Flat Data and Product Price. When a Magento admin changes the price of a product, it is only necessary to execute the Product Price indexer for pricing changes to propagate to the frontend. The completion times of these individual indexers were measured in the benchmark environment for Magento Enterprise Edition 1.13 and Magento Enterprise Edition 1.12.

This section presents the results for full re-index completion times for the individual indexers in Magento Enterprise Edition 1.13 compared to Magento Enterprise Edition 1.12.

  • URL Rewrite failed to run in Magento Enterprise Edition 1.12 but takes only 0.15 sec in Magento Enterprise Edition 1.13
  • Catalog description: 500,000 products, 2000 categories, one storefront

Page Load Times

When Magento Enterprise Edition 1.12 and Magento Enterprise Edition 1.13 were compared, with both running on our multi-node benchmarking configuration, Magento Enterprise Edition 1.13 loaded pages 65% faster than Magento Enterprise Edition 1.12.

Guest checkout and registered user checkout are two flows that are crucial to storefront operation. This section presents the results of page load time measurements for these two flows.

Guest Checkout Flow Page Load Times

In the guest checkout flow pages, Magento Enterprise Edition 1.13 provides a substantial decrease in load times over its predecessor, most of the time more than twice as fast.

Registered Checkout Flow Page Load Times

The bar chart below presents the improvements in page load times for registered checkout flow.

Page Views and Orders

In addition to page load times, the benchmark also focused on measuring throughput improvements, particularly page views per day and orders per day.

During our testing, which simulated a storefront running at peak hours, EE 1.13 executed 33% more orders and 31% more page views than Magento Enterprise Edition 1.12 on the multi-node benchmarking configuration. Notably, Magento Enterprise Edition 1.13 served 47K pages during the test run (10 minutes).

  • 10 minute session time

Observations

What we noted during the benchmarking tests:

  • The MySQL instance did not show any significant signs of CPU or I/O load during the tests.
    • The CPU was under 10% and no queries exceeded a 2-second threshold.
  • Redis and Memcached instances did not exceed a CPU load of 10% during the tests.
  • Web nodes showed high levels of CPU utilization under high load.
  • We anticipate achieving stable scaling by adding additional web nodes to the cluster until the services themselves begin to degrade.

Conclusion

Magento Enterprise Edition 1.13 was engineered for performance--and clearly delivers on the goal as measured by important metrics.

  • Indexing is improved to enable faster operations without impacting shopping experience. Merchant administrators can add and update products as needed while ensuring product URLs, promotions, navigational menus, and product search tools are always up to date.
  • The checkout process is improved by reducing page load times for browsing and placing orders. Faster checkout can significantly improve your customers’ shopping experience and customer satisfaction, and potentially improve your conversion rate.
  • Faster page load times means Magento Enterprise Edition 1.13 can support more page views per day and more orders per day, potentially increasing your conversion rate.

In addition, we have focused on providing these benefits without the need to upgrade existing hardware, improving your return on assets and investment.

Appendix

Page Load Times

This benchmark report presented analysis of the results produced by the Gatling load generator tool. In this appendix, you can see the actual results reported by Gatling. The results were obtained under the following test setup, which was designed to push Magento to its limit.
Concurrent session (users) 1000
Peak load duration 10 minutes
Session duration 72 seconds
CPU utilization 95%

CPU utilization is high because session duration was set to 72 seconds for the test. Session duration was set to a low value to increase the active load on Magento. A typical e-commerce site will see session durations of five minutes or more. Preliminary experiments by the benchmark team have confirmed that using a five-minute session duration reduces the CPU utilization to 50-60%.
Magento EE v1.13
Requests Total OK KO Min Max Mean Std Dev 95th Pct 99th Pct Req/s
Global Information0 62161 62087 74 80 12910 1735 1869 5780 8060 75
product page0 28573 28573 0 80 9390 709 996 2670 4920 34
Click add to cart1 10458 10458 0 440 7170 1868 1139 4010 4980 13
Click add to cart Redirect 12 10458 10458 0 960 12910 3816 2239 8050 9960 13
click checkout3 3486 3486 0 1100 10800 3459 1965 7130 8970 4
Registered Checkout:Login and Pick Address4 3366 3366 0 200 7380 1155 1482 4870 5900 4
Registered Checkout:Login and Pick Address Redirect 15 3366 3366 0 700 11440 2605 1662 5640 7710 4
Guest Checkout start6 120 120 0 360 2490 908 487 1860 2420 0
Registered:Pick Billing Address 17 120 120 0 390 4240 1457 974 3510 4210 0
Registered:Pick Billing Address 2-18 120 120 0 410 3690 1218 685 2460 3070 0
Registered:Pick Billing Address 2-29 120 120 0 400 3600 1201 714 2700 3510 0
Registered:Go to pick Shipping Method10 120 120 0 340 2900 1072 633 2330 2700 0
Guest:Pick Billing Address 111 120 120 0 610 4620 1688 941 3580 4440 0
Guest:Pick Billing Address 212 120 120 0 380 3530 1049 589 2050 2750 0
Guest:Go to pick Shipping Method13 120 120 0 300 2080 796 401 1530 1960 0
Set Shipping Method (Flatrate) 114 240 240 0 510 4830 1780 1066 3930 4720 0
Set Shipping Method (Flatrate): Goto Payment15 240 240 0 300 3150 905 516 1940 2300 0
Set Payment Method (Check/MO) 116 240 240 0 670 5290 1787 987 3660 4920 0
Set Payment Method (Check/MO) 217 240 240 0 300 2760 904 518 2010 2440 0
Set Payment Method (Check/MO) 318 220 220 0 370 8180 2696 1725 5810 7590 0
/checkout/../success/19 240 240 0 250 2830 931 583 1970 2530 0
/checkout/../success/ Redirect 120 74 0 74 1030 8350 2473 1620 5590 7310 0
Magento EE v1.12
Requests Total OK KO Min Max Mean Std Dev 95th Pct 99th Pct Req/s
Global Information0 48044 47887 157 40 40580 4985 3290 10650 13870 58
product page0 21327 21286 41 40 20040 4557 2820 9740 12350 26
Click add to cart1 8340 8324 16 40 12130 4216 2217 7520 9000 10
Click add to cart Redirect 12 8324 8320 4 40 16500 7022 3449 11780 13500 10
click checkout3 2780 2779 1 920 12350 6311 2829 9760 10950 3
Guest Checkout start4 90 90 0 390 3690 2103 795 3210 3410 0
Registered Checkout:Login and Pick Address5 2690 2690 0 180 19120 3298 4984 15470 16940 3
Guest:Pick Billing Address 16 90 90 0 600 7750 4348 1605 6430 7400 0
Registered Checkout:Login and Pick Address Redirect 17 2690 2688 2 40 24360 5433 3350 11780 14060 3
Guest:Pick Billing Address 28 90 90 0 520 8240 3412 1423 5480 6810 0
Guest:Go to pick Shipping Method9 90 90 0 310 4780 2334 888 3300 4450 0
Registered:Pick Billing Address 110 90 90 0 370 10090 3657 2453 8660 9600 0
Registered:Pick Billing Address 2-111 90 90 0 540 17670 4148 2741 10220 12510 0
Registered:Pick Billing Address 2-212 90 90 0 530 12640 3983 2411 8290 11590 0
Registered:Go to pick Shipping Method13 90 90 0 360 6290 2546 1401 5050 5980 0
Set Shipping Method (Flatrate) 114 180 180 0 450 7640 3926 1884 6550 7450 0
Set Shipping Method (Flatrate): Goto Payment15 180 180 0 380 6290 2451 1256 4270 4830 0
Set Payment Method (Check/MO) 116 180 180 0 570 8290 3923 1996 6460 7300 0
Set Payment Method (Check/MO) 217 180 180 0 330 5230 2318 1238 4210 4810 0
Set Payment Method (Check/MO) 318 180 180 0 940 40580 12673 7957 25910 29540 0
/checkout/../success/19 180 180 0 230 3330 1437 854 2870 3150 0
/checkout/../success/ Redirect 120 93 0 93 780 10470 5031 2251 7920 8880 0

Estimating the Number of Visitors

Gatling does not report the number of users that cycled through during the test. However, it is possible to estimate the number using the following formula: