STAC® World-Records, Junk Yard Dogs, HPC & AI on Wall Street

An All-Flash Array is a simple, reliable way to manage storage for high-frequency trading. But most arrays lack the punch to deliver the fastest tick analytics for a competitive advantage. Easy to say, not so easy to prove. A significant initiative at Pavilion right now is to substantiate our performance claims with public benchmarks.

On 10/14, the Securities Technology Analysis Center (STAC®) announced that Pavilion’s Hyperparallel Flash Array was faster in 8 of 17 mean-response time benchmarks than all other publicly disclosed All-Flash Arrays in the STAC-M3™ Antuco suite using kdb+ 3.6. That’s eight (8) STAC-M3™ Antuco AFA world records, including arrays using pricey, low capacity storage class memory (SCM). STAC members can see the full audited report here.

In fact, Pavilion’s HFA outperformed all publicly disclosed results in 4 of 17 mean-response time benchmarks (aka all world records for all storage architectures) for the M3 Antuco suite. That’s right; we outperformed scale-out file systems, NAS appliances, even Direct-Attached Storage (DAS) with SCM. And in most cases, the equipment we used for testing was significantly less expensive.

What’s even better, is that by using an AFA instead of DAS for these workloads, clusters can dramatically reduce the total flash they need to store their tick database. In DAS clusters, you need to have full copies of the database in every database node. For our test, with six nodes, that would require six times the storage of a shared AFA cluster. As the number of tick databases grows, or your clusters expand, the savings keep on multiplying.

We achieved these world records with what I call the Junkyard Dog.

Junkyard Dogs are rarely pedigreed, yet pack serious performance density. They are trustworthy, loyal, and easily trained.

With the Fall STAC Summits rapidly approaching and high demand for our arrays, we scoured the lab for spare parts not allocated to customer projects. We ended up with a hodge-podge of Toshiba® and Western Digital® NVMe SSDs of varying vintages. Layer in some older servers we “liberated” from the test lab, and we had our Junkyard dog. A dog that proved to be the fastest NVMe-oF array in the world (including in-server flash, SCM, and way faster servers) in four tests involving high-thread-count queries (i.e., lots of users, lots of models running at once). These were the multi-thread STATS-UI tests, which are high intensity read, and the analytics test STAC-M3.β1.10T.STATS-AGG.TIME.

That same setup beat all AFAs STAC has tested for four additional tests, giving us 8/17 best results. *All* STATS-UI tests, from 1-thread STAC-M3.ß1.1T.STATS-UI.TIME to 100-thread STAC-M3.ß1.100T.STATS-UI.TIME. In STAC-M3.v1.100T.VWAB-12D-NO.TIME we set a very high mark. This is an intensive, CPU-light test multi-day, multi-user test that shows how our hyperparallel architecture can accelerate real-world workloads. We also set records for STAC-M3.ß1.1T.MOHIBID.TIME, a light compute but IO-intensive workload, and STAC-M3.ß1.10T.VOLCURV.TIME, a post-trade analytics workload.

Aside from breaking numerous world records, our creation reminded us of the flexibility and resiliency of the Pavilion array, which is especially crucial for our OpenChoice customers, who may also leverage various types of media within the same chassis.

The take-away: junkyard dogs like to fight the bigger dogs, and many at a time.