People hate waiting. We’ve moved from it taking two weeks from mailing in your order from the Sears catalog to next-day shipping from online stores. Instead of driving to the video store and renting a movie on Friday night, we now instantly streaming movies over the internet to our phones and TVs. Even finding out the results of an election has changed from a week-long hand count to a declared winner on election night. All because people hate waiting.
Your applications hate waiting, too. The thing they wait most for: I/O. Waiting for I/O means that databases update slower, stock trades take longer, and effective costs increase since you can’t count on using all the cores and performance available in modern processors.
On Linux, the time a server is stuck waiting for I/O reads or writes is called “IOWAIT.” You can see it in the output of the “top” command as “wa”(wait):
| top - 14:24:37 up 48 days, 3:28, 2 users, load average: 6.54, 2.17, 0.94 |
| Tasks: 336 total, 1 running, 335 sleeping, 0 stopped, 0 zombie |
| %Cpu(s): 56.1 us, 0.0 sy, 0.0 ni, 3.9 id, 40.0 wa, 0.0 hi, 0.0 si, 0.0 st |
| MiB Mem: 15843.8 total, 156.1 free, 782.3 used, 14905.4 buff/cache |
| MiB Swap: 8068.0 total, 8054.2 free, 13.8 used. 14600.1 avail Mem |
| PID USER | NI | VIRT | RES SHR S | %CPU | %MEM | TIME+ | COMMAND | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 root | 0 | 42364 | 811488 | 8384 S | 55.3 | 21.7 | 02:56.3 | postgres | |||||||
| 2 root | 20 | 0 | 0 | 0 S | 0.7 | 0.5 | 00:01.5 | sysemd | |||||||
| 3 root | 0 | 0 | 0 | 0 I | 0.1 | 0.0. | 00:00.0 | rcu gp | |||||||
| 4 root | 0 | 0 | 0 | 0 I | 0 | 0.0. | 00:00.0 | rcu pagp |
The root cause of IOWAIT is simple: slow storage. Whether it’s a database is waiting to read a customer’s order history, or a drug discovery model waiting to store the results of its latest simulation before starting the next one, IOWAIT makes results take more time than they should because an application can’t work without its data.
If you had infinitely fast I/O, then IOWAIT would always be zero, and your processors would be used 100% of the time. We don’t quite have that yet, but the computer industry has been working on the problem since its inception. Storage moved from individual HDDs to SANs with dozens of drives to SSDs using 12G SAS, to NVMe SSDs using PCIe, and finally to NVMe over Fabrics (NVMe-oF) using latest-generation Ethernet and RDMA.
Pavilion isn’t waiting. With our hyperparallel architecture, built for NVMe and NVMe-oF from the ground up, we aim to make IOWAIT as irrelevant as renting videos from a store. Check us out today. Why wait?