I have been working on various storage devices for years. However, if you ask me about the differences between SAS SSD, SATA SSD, and NVMe SSD, well, I can’t tell. I just know that their performance varies.
I tried to look into these storage devices from time to time over the years. I have learned more and more terms like IDE, SCSI, PCIe, but I don’t know how to put these pieces together. Then I get interested in the block device layer on Linux these days, which requires some storage devices specific knowledge to understand. So I decide that this is a good chance to take the time and solve the puzzle.
I spent the past few days digging the web and built a Storage Architecture Model. This model serves as a framework to put storage devices related knowledge together, in the perspective of a database developer. However, I build this model based on my understanding, which is limited and may not be precise. So, I will be very grateful if you can correct me or provide more insight.
Before talking about the model, I need to give you some background knowledge first.
Let’s say you have a CPU and a disk. You want to let the CPU save your data to the disk, what do you need to do? Well, you need to connect the CPU with the disk so that they can communicate with each other at least. Maybe you can find some wires to tie them together, but that’s too hardcore. Besides, do you really know how to do that?
Forget it, let’s use a motherboard instead, which provides a CPU slot and some expansion slots for peripheral devices.
OK, now you may think that you can just insert your CPU and your disk to those slots, and then everything works, like magic. But things are never as simple as that. Do you know how many different kinds of CPUs, disks, and motherboards on the market? How can you know if your CPU, your disk, and your motherboard fit together?
Yes, we need to make some standards for them. Here is one thing I know about standards:
Right, there are a lot of standards out there but don’t worry, we don’t need to know all of them. According to my work experience on storage devices, we only need to pay attention to three standard families: SCSI, ATA, and PCI. I split each standard into three layers according to my understanding: Command Layer, Control Layer, and Transport Layer. My storage architecture model also includes a Storage Media Layer to complete the hardware-level description of storage devices. I’m going to talk about these layers one by one next, but I need to put an overview of these standards ahead so that I don’t need to explain these terms elsewhere.
SCSI Standards Architecture:
You can find more about SCSI standards from INCITS T10 Technical Committee.
- AT Atachment (ATA)
- Parallel AT Atachment (PATA)
- Serial AT Attachment (SATA)
- Advanced Host Controller Interface (AHCI)
ATA Standards Architecture:
You can find more about ATA standards from INCITS T13 Technical Committee.
- Peripheral Component Interconnect (PCI)
- PCI Express (PCIe)
- Non-Volatile Memory (NVM)
- NVM Express (NVMe)
You can find more about PCI standards from PCI-SIG.
Storage media are the physical materials to store data. We have machanical media (e.g. punched card), magnetic media (e.g. magnetic tape), and electrical media (e.g. flash memory). You can check this video about How Flash Memory Works.
HDD uses magnetic disks as its storage media and contains some electro-mechanical components. You can check this video about How HDD Works.
Command Layer defines the command set host systems use to control storage devices.
SCSI, ATA, and NVMe standards contain specifications about this layer.
SCSI splits its command set standards into two parts. The first part is SCSI Primary Commands (SPC), which defines commands that apply to all SCSI devices. The second part is device type specific command sets like SCSI Block Commands (SBC), which defines commands specific to SCSI block devices. So, for example, a SCSI block device needs to support both SPC and SBC.
ATA Command Set (ACS) defines commands to control ATA storage devices, including PATA and SATA devices. During years of evolution, ATA command set and SCSI command set share a lot of common, especially for block devices. SCSI also defines a SCSI/ATA Translation (SAT) standard to translate some SCSI commands to ATA commands.
NVMe command set defines the commands to control NVM devices. Before NVMe came up, people used SCSI and ATA command sets to control SSDs. But SCSI and ATA were primarily designed for HDDs and they can’t fully utilize the potential of SSDs. NVMe has been designed to capitalize on the low latency and internal parallelism of SSDs and has become the de facto standard for SSDs now.
People use different terms in different contexts to describe how host systems control storage devices.
In the software level, this layer acts as a programming interface. It might be referred to as:
- Register Interface: because it interacts with devices using registers
- Host Controller Interface (HCI): because it specifies the operation of host controllers
In the hardware level, this layer exists as a chip integrated on an expansion card or a motherboard (with a chipset called Southbridge, I/O Controller Hub or Platform Controller Hub, whatever). It might be referred to as:
- I/O Controller: because it controls I/O devices
- Host Controller: because it controls devices for host systems
- Host Bus Adapter (HBA): because it adapts devices to host buses.
These terms can be confusing, so I decide to name it Control Layer here based on what it does.
To understand this layer, you need to get some basic knowledge about CPU first. You can check this video to See How a CPU Works. A CPU can’t communicate with other devices directly because they don’t share the same physical and electrical characteristics. Instead, a CPU needs to access some specific addresses (or registers) mapping to a controller and then lets the controller deal with other devices for it. So, this layer defines the interface between a CPU and a controller, for example, to specifies how to put a command to a command queue.
ATA, AHCI, and NVMe standards contain specifications about this layer.
ATA defines the interface to interact with PATA devices over parallel buses. SATA devices use AHCI instead, which works as a PCI device. However, if some operating systems don’t support an AHCI driver, many motherboards also offer a “Legacy Mode” option, which makes SATA devices work like PATA devices on an ATA controller, but sacrifices support for some features of SATA.
NVMe defines the interface to interact with NVM devices over PCIe buses. Again, NVMe has been designed from the group up for SSDs since AHCI can’t deliver the optimal performance with SSDs. Check this comparison between NVMe and AHCI.
What about SCSI devices, you may ask. Well, I can’t find any public SCSI standard about this layer, so I think vendors don’t come up a common standard here. But I can find some vendor-specific drivers on Linux that implement the interfaces for SCSI devices.
Next, I will talk about some buses I mentioned above.
Most standards describe how storage devices communicate with each other in three layers:
- Physical Layer: defines the physical interconnect (e.g. cables, connectors) and electrical characteristics
- Data Link Layer: provides a reliable mechanism to exchange data, including error detection and frame transmission, etc
- Transport Layer: specifies the protocol (interaction) between two devices, including frame formats and how frames are processed, etc
I redefine the Transport Layer here to include all the three layers above. Because as an upper-level software developer, I don’t need to deal with all the details as long as these devices can talk to each other.
In computer architecture, people use the term bus to represent a communication system that transfers data between components. There are two kinds of buses: parallel and serial. Parallel buses transfer multiple bits at a time while serial buses transfer only one bit at a time. However, parallel buses are slower than serial buses because they operate at a much lower frequency.
SCSI, ATA, and PCI are three standard families that contain specifications about this layer. Each of them defines a parallel bus and a serial bus. Let’s put them into a table here:
|Parallel||SCSI Parallel Interface (SPI)||Parallel ATA (PATA)||Conventional PCI|
|Serial||Serial Attached SCSI (SAS)||Serial ATA (SATA)||PCI Express (PCIe)|
Nowadays, parallel buses are mostly replaced by serial buses, which are faster and simpler. These buses have evolved a few versions over the decades, providing higher and higher speed to suit the evolution of other devices.
Storage Architecture Model
Now we can put all the layers together and build our storage architecture model.
|Storage Media Layer||HDD: Magnetic Disk, SSD: NAND Flash, 3D XPoint|
That’s it, clean and compact. Next time you ask me what a SATA HDD or an NVMe SSD is, I can tell you loud and clear.
SATA HDD is a storage device, it accepts ATA commands, works with an AHCI controller, transfers data over a SATA bus, and stores data with magnetic disks.
NVMe SSD is a storage device, it accepts NVMe commands, works with an NVMe controller, transfers data over a PCIe bus, and stores data with NAND Flash or other electrical media.
But there is one more thing.
If you try to buy an NVMe SSD, you may find something like “NVMe PCIe M.2 SSD”.
If you follow me all along here, I think you already know what NVMe, PCIe, and SSD mean, but what is M.2? M.2 is a connector standard, it defines how to connect a device to a motherboard, like the shape of those golden pins on the right of the picture above.
There are a lot of connector standards, for example, U.2, mSAS, and mSATA. I don’t mention them until now to keep them out of the model so that they don’t muddy the water in the first place. You don’t need to pay much attention to them unless you are trying to buy an SSD that fits your motherboard. Again, as a software developer, who cares about the different shapes of these devices as long as they can be connected to the motherboard.