Pocketables SSD testing procedure and overview
We’ve covered several SSD’s so far some in several form factors as well as unique interfaces. But we’ve done so without much information on how or why we’ve selected the tests. So let’s take a look at that and bring some SSD comparison charts into the mix finally.
We keep our testbench consistent. All SATA and PCIe drives are tested on the same platform consisting of a i5-3470 and a Gigabyte Z77 mainboard. Intel Sata ports are used to keep the results repeatable on other systems as well. USB devices are tested on a Thinkpad S1 Yoga using Intel xHCI ports. This once again ensuring results can be reproduced on most modern systems.
What’s different between the interfaces
Modern SSD’s use a variety of interfaces to connect to the system. The latest and greatest of these is NVME over PCIe which allows absolutely staggering bandwidth. The fastest consumer NVME drives to date use PCIe 3.0 with a 4 lane interface. This allows a maximum theoretical bandwidth of 3.94GB/s although drives rarely reach this in practice. SATA is currently the most common interface we see on SSD’s. It allows a maximum of 600MB/s assuming no overhead beyond encoding but this is rarely the case and why drives like the Reactor hitting 550+ in some read tests are about as fast as SATA can get. USB3.0 is the slowest interface we see modern SSD’s attached to and is rated for 500MB/s post encoding. USB attached SSD’s have other considerations as well typically relying on a SATA to USB bridge and size constraints.
MLC TLC and SLC
There are 3 types of NAND available in drives today. SLC is the was one of the first available as the simplest. SLC stores a single bit of information per cell and offers the highest endurance but is at this point generally only available in enterprise type drives. Nothing we have tested to date is a SLC although that may change in the future. The bulk of the products we have tested to date are MLC. MLC is cheaper to produce than SLC memory and stores two bits per cell. This increase in density causes a decrease endurance and speed but also greatly decreases cost. The third and final is TLC or triple cell memory which increases bits per cell further to three. TLC memory is the slowest of the three and is capable of speeds below what mechanical hard drives offer sequentially in some instances. TLC based drives typically use a SLC cache to mask their low write performance and still offer impressive read speeds across the entire drive.
What is Queue depth
One term I throw around a fair bit in SSD reviews is “Queue depth”. At it’s simples this refers to the number of outstanding operations for a drive. For Sata based devices this is limited to a maximum of 32. With SSD’s employing parallelism at the device level higher queue depths translate to higher performance. Vendors like to quote QD32 specifically for sata devices as it usually yields the highest performance. It’s also the limit of the AHCI command set used by conventional SATA drives and something rarely reached in consumer workloads which is why we report results at multiple depths in some of our tests. Consumer workloads for reference rarely scale past QD4.
What about IOPS
The other fancy acronym that gets thrown around is IOPS. This one is dead simple and comes down to input output operations per second. IOPS are a lot simpler than they sound simply being the number of transfers per second a drive can handle. IOPS do need a size applied to them, typically we look at 4k random read and write IOPS as they’re the most taxing on a drives controller.
One of the key specifications we look at when examining drives is warranty and endurance. As a general rule we see consumer drives shipping with a 3 year warranty although less and less are stating an endurance rating. Simply put endurance is how much data can be written to a drive before it fails. Although we can’t test every drive till it dies we can look at a drive’s wear indicator and report what’s left after we’ve finished testing. None of our drive has reported a significant amount of wear and we usually write 5+TB during testing. For reference I’ve written 17TB to my laptop over the past year and a half or so. A more conventional consumer workload however is closer to 1-2TB/year or less.
Atto is always our first test with a new drive. This is a popular test with drive manufacturers. It responds to compression and most fully populated SSD’s perform quite well in it. Registration is unfortunately required to download it although it is available free of charge.
Looking at the drives we’ve tested the Accelsior E2 is a standout here although it’s important to remember it utilizes write compression. It’s also the only PCIe drive we’ve tested to date providing it a faster interface than the other drives. The Mushkin Reactor looks interesting in these charts as well showing consistent performance once it’s reached its peak compared to other drives.
Crystal Disk Mark
Crystal disk mark is another popular one with enthusiasts and SSD manufacturers alike. It’s available for download free and tests multiple access patterns by default. Crystal Disk mark defaults to testing random and sequential data both at QD1 and T1 as well as QD32 T1. SSD’s generally respond well to changes in Queue depth and usually post much higher results at QD32 although this is rarely seen outside of servers. Finally Crystal Disk Mark defaults to incompressible data. The use of incompressible data is important as some controllers such as those used in the Accelsior utilize write compression. Drives that don’t will post the same results between ATTO and Crystal disk mark and generally perform more predictably. Lets take a look at what we’ve tested to date.
That compression that I mentioned earlier is utilized by both the Ventura Ultra and the OWC Accelsior. Due to this they don’t perform as well here when it can’t be used. Both the Mushkin Reactor and Sandisk Ultra II perform excellently here. The Ultra II does take a leak in the 4k write test due to its use of an SLC cache.
Moving to read focused results nothing can quite keep up with the Accelsior sequentially. The SATA drives still have a strong showing here with the Reactor once again edging out the Ultra II.
For canned benchmarks ANVIL is a monster without question. It tests multiple queue depths, transfer sizes and is our first test to report IOPS. We use this as both a catch-all verification for our other tests and a benchmark in its own right. It’s results when combined with those of our previous tests give a feel for how a drive’s performance scales offering information at QD 4 and QD16 as well as QD1.
Taking a look at response time the results are rather unsurprising. The Accelsior is hampered by its use of two controllers and a raid controller. The Ventura Ultra is in a similar boat using a SATA to USB bridge chip hurts it in a test like this. Something to keep perspective with here however is that all of these response times are faster than spinning discs.
Anvil is the first test to give us any kind of results in IOPS and it’s something that SSD’s are generally quite good at. I mentioned earlier that 4k random IOPS were generally the hardest on a controller but this chart doesn’t quite tell the full story. The Sequential read and write testing anvil does is based on a 4Mb transfer size at that rate roughly 150iops is enough to saturate the interface.
Anvil also reports results in a much more familiar MB/s. We can see here that the anemic looking IOPS results for sequential data are actually approaching the interface limits. In fact the Accelsior thanks to it’s PCIe interface actually surpasses the maximum theoretical limit of SATA.
AS SSD is a catchall verification for our other tests. It doesn’t quite cover the variety of data types that ANVIL does but gives us IOPS in addition to MB/s and response times.
Looking at IOPS the chart is almost a duplicate of ANVIL’s. We see some different data with 16Mb as the selection for the sequential tests instead of the 4Mb used by ANVIL but the results are interface limiter as expected.
We also have results in MB/s as well, just like the previous IOPS results these generally corroborate what we’ve seen.
AS SSD is our first test that includes simulated real world workloads. These simulate moving real world data patterns around on the drives themselves. Results here show a strong preference for the SLC cache in the Sandisk Ultra II. The 120Gb Ventura Ultra takes a large hit in these tests
PCmark includes a storage benchmark that includes a wide range of real world data. This is used to gauge how an SSD will respond to real world workloads. The exact details of these are available in their technical guide. These tests were created by observing the data patterns that the applications use and then running through them in a controlled consistent manner.
In addition to it’s single score PC mark reports back the run time of all these tests. As we can see most of the SSD’s we’ve tested here with the exclusion of the USB attached Ventura Ultra perform similarly only separated by a few seconds. Overall ANY ssd is going to be an improvement from a spinning disc and these results demonstrate why.
We use a program called IOmeter to perform consistency testing. IOmeter is an older application originally written by Intel and with an original announcement at IDF in 1998 is the oldest tool in our test suite although the revision we use is the latest which was released in 2014. IOmeter gives us complete control to adjust anything with how a test is going to be performed. The test pattern we use is a 4k write with fully random data at a queue depth of 32. For consumer level drives(which is all we’ve tested to this point) we test for 3 hours which is enough to fill the drive completely and push it to what we call steady state. Once a drive has exhausted its spare area performance should be more or less the same from that point forward.
Taking a look at the drives we’ve tested is a bit of a surprise until we remember that the Sandisk Ultra II is a TLC based drive. TLC has a longer program/erase time leading to longer operations especially once a drive is out of spare area to work with.
High performance is great but consistency is important too. We take the average and divide it by the standard deviation over the same time period, a higher result in this metric is better and we can see that both Sandisk and Mushkin have done an excellent job here. The Accelsior unfortunately never stood a chance thanks to it’s multi controller design.
The final metric generally left towards the end of our review and is difficult to quantify as it can fluctuate rapidly with time. However it is useful to look at it during the review and periodically afterwards.
Looking at drives we’ve tested sofar the Sandisk Ultra II and Mushkin Reactor Generally offer the best pricing at a given capacity. This is unsurprising, the Ventura and Accelsior both have additional bridge chips that raise their respective prices. An interesting note here is the Reactor is clearly optimized for the 1tb price capacity where it’s cheaper than the Ultra II an oddity for MLC drives.
The drives we’ve tested so far are just a taste of what’s possible with NAND. The Reactor and Ultra II both showed us just what to expect out of consumer SATA drives and the OWC Accelsior gave us a taste of what moving to PCIe as an interface can do. The Ventura Ultra offers a unique USB offering in a nearly usb flash drive size form factor although there is a premium for this luxury. We will continue to use the performance chats that have been presented going forward for new reviews. Do you think one of our tests needs to be tweaked? Or did we completely miss something that you’re looking for when buying a drive? Let us know in the comments below or on Twitter, or Facebook. And don’t forget that you can support us on Patreon to help us continue to bring you high quality reviews as well as access to. early news and input on reviews to come.