Will QLC Flash Replace Hard Drives?

Published on searchstorage.techtarget.com. Read the full article and watch the video here.

00:00 Randy Kerns: Well, hi there, and welcome everyone to Session D-12, and the title of this is “Will QLC Flash Replace Hard Drives?” This is Randy Kerns, I’m the moderator, and we’ve got a prestigious panel here I’ll introduce in a minute.

Let me tell you a little bit more about what we’re going to talk about. We have watched QLC develop and great hopes for the technology from a lot of people, and some people said, “Oh no, this isn’t going to work.” And obviously, it’s very successful so far. So, this session is to explore a little bit about what we think is going to happen, and it’s because QLC is being used in application areas where large capacity disk drives have dominated. So, with our panel we’re going to discuss this and see where it’s going to go from here. Let me go down the list here and talk about some of our guys and ask them to tell me a little bit about where they’re interacting, or developing, or working on QLC type devices. First, we’ll talk with Roger Peene from Micron Technology. Roger?

01:14 Roger Peene: Hey Randy. Thanks. Yeah, this is Roger Peene. I actually manage the marketing and strategy for Micron Storage business unit. That involves the components, the NAND components themselves, but also SSDs for both the data center and the client. So, we develop a QLC device and then we also take that device and ship it into data center SSDs, SATA, NVMe, and then also a client. So, obviously, QLC is very, very relevant for us, especially as we have fabs pumping out QLC bits as we speak.

01:52 RK: Obviously, very critical for you. Alright, let’s move on to . . .

02:37 RK: Alright, let’s go to Jeff Denworth from Vast Data. Jeff?

02:43 Jeff Denworth: Yeah, hi. I’m Jeff Denworth, CMO and co-founder of Vast Data. In 2019, well actually, we started selling 2018, but in 2019, Vast came out of stealth mode, shipping the world’s first distributed file in object storage system, built entirely from QLC flash. We enabled that through Smart Controller software coupled with storage class memory and an NVMe fabric to build a new type of storage system that we think defies categorization. We call it Universal Storage, because you get more than just kind of archival capability out of the platform. You can use it for some of the highest performance applications in the world.

And so, yeah, we’ve been on a little bit of a tear since we started. Last year, we kind of announced that we were the fastest growing company in storage history. Appears that we are continuing that trend going into this year, and things seem to be going in the right direction. So, happy to be here.

03:45 RK: Alright. Thanks, Jeff. Shawn Rosemarin from Pure Storage. Shawn?

03:49 SR: Yeah. Thank you, Randy. So, look, at Pure Storage, I look after all the systems engineers globally, roughly 600 of the smartest people I get the privilege to work with. Pure Storage has been in business about 10 years. We were the first to bring an all-flash array to the market. We’ve been on a tear, I’ll use Jeff’s term, but been on a tear for a decade doing just that, and actually we’re the first to bring QLC optimized to an all-flash array just last year with the launch of FlashArray//C. Very proud to have been named Best in Show, The FlashArray//C, just yesterday. So, excited to be here. Thank you, Randy.

04:24 RK: Alright, Thanks, Shawn. Now let’s go to Ken Steinhardt from Infinidat.

04:28 KS: Hello, I’m Ken Steinhardt. I’m the field CTO at Infinidat. I suspect I may be potentially the lone contrarian on this panel in that we tend to be agnostic relative to the technologies and use DRAM and flash as well as conventional drives, and see a place for possibly all the technologies going forward, but QLC will potentially play a role in there as well going into the future.

04:51 RK: Always good to have a contrarian to make fun of. Alright. So, now let’s go to Tom Isakovich. Tom?

04:57 Thomas Isakovich: Alright, thanks, Randy. My name is Thomas Isakovich. I’m the founder of Nimbus Data. We have an all-flash platform. We also have high-capacity SSDs based on our own controller technology, and we’ve introduced into the market a 64-terabyte QLC SSD that’s actually in the same form factor and interface as a hard drive. And so, to the extent this panel is about whether QLC will replace hard drives, this product was designed expressly to do that by mimicking the hard drive size and interface. So, we’re big believers in QLC and looking forward to contributing.

05:36 RK: Okay, well, thanks everybody. Let’s get started with a very simple question, and then I’d like to ask you to give a short answer as to why for your answer. First one is, will QLC replace hard drives, and what is the single most important reason for that? So, let me just throw it open. Ken, since you’re the contrarian, what’s your thought?

06:00 KS: I will start. I would say that, that’s almost a bit of a presumptive question. It’s sort of like asking, who’s the greatest rock guitarist, and why did you say Eddie Van Halen?

[laughter]

06:09 KS: Certainly a great guitarist, but there’s also a Jimmy Hendrix, there’s an Eric Clapton that people have heard of, and there might even be people that you haven’t heard of, like a Gary Hoey or an Eric Gales or someone like that.

One of the assumptions, I think, about QLC is that the price delta has significantly eclipsed that of conventional hard drives. And relative to the traditional old enterprise class hard drives, the old 15K and 10K drives, I believe that’s probably true, and I think there’s probably a really good space for it there. But relative to nearline SAS, most are still projecting that the gap in cost has been, right now, about . . . For the last five years, going back about to 2015, about 9x relative to that class of technology. And Gartner, in their most recent forecast from September 30th, said that it’s still 9x today, and curiously, they’re forecasting that all the way through 2024, it’s still going to be 9x.

07:09 KS: So to me, the issue isn’t QLC or hard disks. I believe there’s going to be a place for both, as well as for even things like PLC to follow on, as well as for storage class memory, as well as for DRAM, as well as for hard disks. And there’s a lot of work that’s being done in . . . Even in the hard disk space that I think people are ignoring if they’re focused myopically just on the flash side of things; there’s technologies like shingled magnetic recording and heat-assisted and microwave-assisted and even dual actuator, where after a fairly quiet time, there’s a lot of exciting technology things going on in that space as well that I think gets swept under the carpet that are going to keep that technology very competitive in certain spaces for a while.

07:54 KS: So I don’t think it’s a question of QLC replacing hard disks, I think it’s a question of where will hard disks fit and where will QLC fit as well as these other technologies that in all likelihood there’s actually going to be a pretty bright future for all of them. It all comes down to the price and the capacity and the durability and the reliability and the performance. And ultimately, I don’t think consumers of the technology, whether they be individual or even the major storage vendors, ultimately are going to care as long as they hit those actual criteria for what people look for; performance, cost, scale,  et cetera, and that there’s a place for a whole lot of things, going forward.

08:32 RK: Okay. It’s hard to keep quiet, but I’m going to turn this [chuckle] over to Shawn, your comment.

08:39 SR: Okay, so I’m going to answer the question the way you asked it, “Will QLC flash replace hard drives?” Absolutely, there’s no doubt about it. Spinning disk is on a declining curve where it will only exist for the lowest-value data. That being said, the decline is a long tail, and it will be asymptotic, so there will . . . I don’t disagree with Ken that there might be some workloads, even out in the long tail, that will still sit on disk, but it will become less and less important.

Why? You asked for one reason, so I’m going to give you one reason, I’m going to side with physics here on latency. It’s the biggest advantage of flash. Yes, performance, but specifically latency and low latency. As drives get bigger and bigger and data gets bigger and bigger, we have no choice but to look for the technology that drives the lowest amount of latency into the system. So, I would say latency wins out here as the number one factor.

09:27 RK: That’s good. Thanks. Roger?

09:31 RP: Sure. I’m going to change the word, because I don’t think it replaces hard drives in the near term, I think it displaces hard drives in the near term. And we have to look at it from the perspective of the total cost of ownership. We always have to take into consideration all of the elements — including performance, including power cooling, including reliability — that all come into play into the total cost of ownership.

If we look at just the pure raw cost . . . Micron introduced a 176-layer NAND technology at the beginning of Flash Memory Summit on Monday, which gets us the most dense flash you can find on the planet right now, and so what’s happening is our costs are continuously coming down. And when it does, it tends to open up more usages to store data on QLC-based technology.

10:21 RP: Now, storing data inexpensively on M-Flash is always a good thing, but it’s also about access to it. And when you can get much better latency and much better performance on access to the data to pull it out and to read it, that’s where we think the flash technology really shines over hard drives. And so, I do agree with Shawn in that as an industry, we tend to overestimate front-end ramps and underestimate back-end tails. So, I believe that there will be a long tail of hard drives, of course, but over time, the displacement ratio between QLC technology and hard drives will continue to grow.

11:05 RK: Okay. I’ve got to make a comment here that I do end up, a lot of times, with customers, and they bring up cost differentials. And so, you’ve already brought up other factors: performance, longevity and all these other things. And I tell them, “If cost is your only parameter, you should be using tape.” And that tends to get the discussion going about what the other attributes are. Jeff, give me your thoughts here.

11:36 JD: The answer is QLC, and the reason is cost. And this component-level discussion that we’re having is extremely myopic, because you need a system to expose your data, and that system consists of a lot of different stuff, and you need a data center that can manage that data through it. And if you look at QLC, what it allows you to do is to eliminate a higher-level kind of applications of flash in your data center, so you get a savings there. The longevity is better, so you get a savings there.

The data reduction codes that could be implemented using QLC are monumentally better than what you could implement on a hard drive in a practical basis. And I believe that 9x, that cost delta that was referenced earlier, I believe was against the cost of the high-capacity hard drive versus the average cost of flash; and I can tell you, after shipping hundreds of petabytes of QLC, that those aren’t our experiences in terms of our cost of goods.

12:38 JD: So, we’re actively replacing not only spinning disk-based systems, but on some occasions, we’ve even replaced tape libraries already with the Vast Universal Storage system, and the only way you can do that is cost. That was the fundamental reason that we started this company was just to solve the cost problem, and if you get there, we ask customers all the time, “If you had the choice, would you buy a hard drive-based system?” Cost wasn’t the issue, and not a single customer has ever said, “Yeah, give me spinning rust.”

[laughter]

13:09 RK: Tom, please.

13:11 TI: Yeah. I got to get this back into the land of the real. Guys, there’s a 4x or 5x cost difference between hard drives and flash. It’s going to persist for years. This isn’t a winner-take-all game. It sort of like thinking everyone’s going to drive a Ferrari because they’re faster, so there won’t be any cargo planes or buses anymore.

Most of the capacities go to hyperscalers. These guys, they get bent out of shape over fractions of a penny. You’re talking about a media type that is seven cents more per gig than hard drives. It’s not a winner-take-all thing, and even if it could be, the supply isn’t there, the manufacturing isn’t there. It’s sort of like when people say, “Hey, the whole country needs to run on solar and wind.” It’s not possible to power this country on solar and wind. You can’t actually store all the data in the world if all you had was flash — 90% of the data in the world would have to disappear. So, it’s not a winner-take-all thing. I think, I agree with Roger. It’s going to selectively displace hard drives.

14:08 TI: You also have to think about this whole concept of data reduction. Data reduction is supposedly the great savior in making flash cheaper. The reality is that the data types that are emerging are less reducible over time to the extent that their unstructured content, media, stuff doesn’t reduce terribly well, so, you’re not going to get the same efficiencies that you got with, say, VDI and virtualization that made flash more effective when you could talk about a 10 to 1 data reduction. So, I think both technologies exist for a long time. Supply, cost, these are all important factors.

14:39 RK: Yep, and totally agree, And I think somebody hit this, is there’s some specific case discussions that have to be evaluated each time. Now, let me jump around a little bit. One of the things that I’ve seen is that, and a lot of people tend to misunderstand this a little bit, is that QLC SSDs really are not a drop-in replacement. There’s a lot of design work that has to go into this, and then the handling and the way the design’s done can really dictate the longevity of the devices.

So, do you think that the not being a drop-in replacement and having to design it is really going to delay more people bringing QLC to the market to make it a more competitive landscape for storage systems?

15:28 TI: Randy, I wanted to talk about this because we looked at this very closely. Given that we have a QLC SSD that’s in the 3.5-inch hard drive form factor, so it literally is physically designed to be exactly that, a drop-in replacement. And what we found is that, first, you got to worry about the mechanical aspect, you have to worry about the interface aspect, you have to worry about power and cooling. Can you get it to cool and power within the same envelope as a traditional hard drive?

But I think you’re more specifically talking about endurance and concerns over endurance regarding QLC. And one thing we found, which the hard drive guys don’t really talk about, but it’s in all their data sheets, Seagate, WD and so on, is that all of their drives servo motors are rated for 550 terabytes of writes per year. And based on the capacity of an 18-terabyte drive, that actually works out to something under 0.1 drive writes per day, which is actually much lower than the endurance of flash.

And so, when you look at the device, the hard drive guys do have a correct statement when they say that the media has higher endurance then QLC, but the motor has lower endurance than QLC. So, the device, as a whole, practically speaking, may have lower endurance than QLC.

16:46 RK: But the design issue, the question is really about having to change a design for a storage system. I’m going to get to Jeff in a minute, because his is a good example of design from the ground up to work with that type of device. Do you think that design change required delays more competitive systems?

17:09 TI: Yeah, I think you need to optimize the way the write patterns work in a storage array so that you’re minimizing the wear and tear on QLC, and that’s a new thing for QLC relative to, say, TLC or MLC. When we made those transitions in the industry, people didn’t really have to change their applications too much because we were still getting enough reliability. With QLC that’s not the case, and that’s why you have to position it into specific workloads or you have to tune the way the server or the storage array writes to the media in a specific way to avoid its limitations.

17:41 RK: Hey, Jeff, if you don’t mind, because you went down this road, and so did Shawn, so we’ll go to Shawn next.

17:49 JD: I think the barrier actually only gets worse over time, Randy, because you’ve got, now, kind of capabilities creeping in like Zoned Namespaces, and drive manufacturers are talking about indirection units in the 64 kilobytes range, just as like a base unit of bit management.

And so, if you think about the storage systems that have come up until now, the storage software has been conditioned to write in 4K sectors or 512 byte sectors, and the race block that we’re dealing with our systems today is measured at over a gigabyte. And if you write to this device at less than a gigabyte, what happens is that the internal garbage collection just creates perilous write amplification. So, I think it is a really big barrier. You can’t take these drives and just put them into any storage environment and expect them to always work. It will be an impediment to QLC, and ultimately also PLC adoption, and zone namespaces will be kind of verboten for almost everything.

18:53 RK: Okay. Shawn, please.

18:55 SR: Yeah, so I wanted just to answer your specific questions again, so yes, it’s very different. Yes, you cannot drag and drop, but I would also say that I give the kind of a corollary here that when we took SSDs and we dropped them into trays designed for traditional hard drives and we just continued to use those controllers, we lost a lot of efficiency, we lost a lot of optimization.

To use a non-disk drive example, we also kept mirrors and SLR cameras long past beyond when they were needed for digital because people just . . . It was just easier for the camera manufacturers to keep the frame the way it was, and then all of a sudden they shrunk, because we took the mirrors off, we didn’t need them.

19:30 SR: When you think about what we did at Pure, we brought all flash to an array, we used ROM NAND instead of SSDs, and we were able to tune that ROM NAND using direct flash modules.

When you look at QLC, there’s no doubt, and I agree with both Tom and Jeff, we have to treat QLC right. In fact, today we’re seeing a lot of other vendors front-ending QLC with TLC, with storage class memory, with DRAM, to hide and cache its issues. I promise that breaks under scale, it quickly destroys its cost advantage if I got to buy all these storage class memory just to make up for all of these caching issues and performance issues and garbage collection issues that I have in the QLC side. And so ultimately, you have to optimize the system to run QLC.

In fact, we had to launch a whole new architecture with FlashArray//C to optimize it to run QLC, and that allows us to take advantage of that architecture. Otherwise, you’re very quickly going to get in a position where you’re going to tell clients, “It doesn’t make sense for you to move to QLC because my architecture can’t do it cost efficient.” Does that make sense?

20:31 RK: It does. I want to ask Ken a question, then Roger, this is all your fault, and I’ll come to you next. Ken, now you stated you were trying to develop systems that you could interchange the technologies, I believe. So this does put a challenge in front of you, right?

20:53 KS: Well, actually, we view all of these technologies agnostically. And I want to agree with something that both Shawn and Roger said earlier, which is that latency matters. So the way that you get past the fact that HDD clearly has ridiculously slower latency than flash technologies or solid state is you put cache in front of them and you write really good software that intelligently figures out what to bring in there.

And to Jeff’s point earlier that nobody asked for spinning rust, I agree, they also never asked for QLC NAND flash, they asked for capacity, they asked for price, they asked for cost, they asked for reliability, they asked for stability. And whatever it is inside that cabinet, users don’t care what it is as long as it hits the criteria, from a business perspective, that they want to get.

21:35 KS: So, we are going to stay agnostic to it. We’d have obviously today DRAM, which is faster than everything from a media perspective, and if you get most of your cache hits to DRAM, you’re going to smoke an all-flash array in performance, even if, dare I say, your persistent data on the back end is conventional hard drives.

And today, just below that, is a secondary read cache of honest-to-goodness NAND flash, because that fits well. But for what those technologies will look like in the future, who cares if it becomes QLC or PLC, awesome. If that secondary level becomes storage class memory because the economics makes sense, great. I don’t think users care, I don’t think users should care, I don’t think that the underlying technology is what solves a customer problem, I think it’s meeting their requirements of the things that are much higher level.

22:23 RK: Alright, for somebody like me who’s been around for way too long and developed a number of systems, and in that time, anybody says smoke, it’s a bad thing. Okay, Roger.

22:34 RP: But since this is all my fault, I kind of view it like any new technology that comes into the market. We have to intelligently deploy it, and that’s in understanding of what the underlying limitations and benefits are for that technology. And what I really, really appreciate about the industry is the fact that we’re offering . . . Micron’s going to innovate on semiconductor physics, we’re going to innovate at the component level, and we’re going to do our best to put out the best QLC device out there.

But then, it’s the innovation of the companies that are on this panel that’s taking that base technology, innovating on top of it and making it better for the end customer and making its usage more applicable. So things like, whether it would be Zoned Namespaces, whether it would be compression, whether it would be techniques to reduce the I/O blender effect or even write throttling, these are all techniques that make QLC even better. And as we continue that innovation, again, I think that the technology will get deployed more broadly. QLC from our perspective is not new. The first QLC device that we put out was almost two years ago, and so we anticipate obviously supporting QLC for quite some time into the future.

23:54 RK: Let me make a comment, since I spend a lot of time with large end users, is that, what Ken said is right, what really matters are the economics and things like that, however, the marketing people have conditioned the customers about the technology.

24:13 KS: Very true.

24:13 RK: So, it always comes up. So, despite the reality, this is what’s most important, you’re going to end up having a technology discussion with the customers. They’re bright people, they’re interested, and they’ve heard a lot, and you just can’t keep the marketing in check to not talk about this because everybody’s proud of it, it’s unique. That’s a big deal.

24:38 KS: Right. And interestingly, if everything you have is a hammer, then everything . . . If all you have is a hammer, rather everything looks like a nail, doesn’t it?

24:44 RK: Right, right. Okay, so now . . .

24:46 JD: But I would also say that Ken mischaracterized what I said. I said, if a customer had a choice between flash or a hard disk, what would they choose? And they don’t choose hard disk. I didn’t say, does a customer want QLC? So, it’s important to understand what’s being said here.

25:02 RK: Yep, I don’t disagree. Again, I sit on the other side of the table sometimes from the salesman, on the customer’s side, and it’s pretty interesting listening to the conversations and the high degree of variability from different sales guys, it’s very interesting.

Okay, so, now Roger brought up the 176 layers. I thought this is a great segue in here in that the capacity increase vectors are quite different. And, you remember Ken talked about HAMR and MAMR and different things going on in the spinning disk area, but these vectors are diverging, and significantly greater increases are occurring in QLC technology, rather than what’s going on in the disk.

This is going to . . . I think somebody said, you don’t think this is going to change the cost profile. I think it will change the cost profile, but does that produce some type of opportunity to talk to the customers and show them that changing vector? And you think that’s important to them. Meaning that they’re going to look and anticipate, this is a technology that’s moving forward, we’re not near . . . Hard disk drives are not near as much increase in capacity and therefore the cost vector, certainly the performance, et cetera. So, the question is now, do you think that cost vector, that change is the most important thing we’re dealing with with QLC today. Roger?

26:39 RP: Yeah okay. So, QLC in flash technology is on a trend line and we continue to innovate, we continue to fit more die per wafer, we continue to fit more bits into a single piece of silicon.

But don’t forget, hard drives is also on a cost vector as well, that’s why we saw things like SMR drives come out. Now the rate of decline in terms of raw cost typically is greater on flash because there’s still a lot more room to go. Again, I think that it comes down to the economics, and I always go back to total cost of ownership. And the reason for that is because if you just do a simple dollar per gigabyte compare, you’re going to stay cheaper on hard drives for quite some time.

In fact, you’re going to stay cheaper on tape drives. It all comes down to the latency, the performance, as well as the cost. And when you look at all that, and the reliability by the way because that’s really important when you’re dealing with SSDs versus hard drives, I do think that we’re getting closer and closer to the realization of an all-flash data center.

27:50 RK: Okay.

27:50 SR: Okay Roger, let me jump on that if I can. So, I like the way you laid that out, very well put. I mean ultimately, it’s raw performance, latency and durability reliability, and in some cases, lower power consumption. I don’t know if you guys caught earlier this week, but obviously Samsung’s not here, so I can quote them. But their CEO did say that general SSDs allow . . . low power consumption SSDs consume 50% and 94% less power than HDDs respectively. Consequently, if all HDD storage and global data centers worldwide was replaced by low power consumption SSDs, we eliminate 41 million tons of carbon dioxide.

So, sustainability is a pretty big factor here, but getting back to the question at hand, I’d say at Pure we’ve seen that when flash gets to within two to three times the cost of disk for any segment, it’s a prime time for a switch. So, two to three times is what we look at, not the actual crossover of zero, but two to three times. And so, if I was kind of trying to make a prediction here of an over-under, I would say within three years is when that absolute crossover happens, and it will be a long tail. I love the way you put it, Roger. It’s not going to die, but it will just become more and more displaced over time.

29:05 RK: Alright, Tom?

29:08 TI: You know, we’re an all-flash, I actually just don’t agree with that, anyway. Despite being an all-flash company and not doing hard drives, I’ve been a quarter century in this too, and there’s no one size fits all. There’s going to be people that are 100% motivated by cost per bit, and even 2x to 3x, even though that’s a monumental improvement compared to 20x just a few years ago, it’s still 2x to 3x. And there are plenty of people in the hyperscale world that are going to buy capacity by the pound, by the bit at the cheapest price, so I think the tail for hard drives is going to be a long time.

I think one thing about the perspective, I think everyone here on this panel, with maybe the exception of Roger, is we’re all coming from an enterprise systems perspective. So, we’re thinking about system architecture things for enterprise storage, but so much of the capacity of being deployed by the hyperscale folks that have a completely different calculus than we do in terms of the value offered to a customer and the value to them. And so, they’re going to be driving HDDs for a long time. I love to see QLC everywhere, but I think they’re going to be around for quite a long time, both hard drives and QLC. I would at least say probably at least a decade, or perhaps even two decades.

30:21 RK: So, we’ve got just a little bit of time, Ken and Jeff please. Ken first, then Jeff.

30:29 KS: I’ll agree with what Tom said. I see the market the same way, that you’re fundamentally going to see the technologies fitting the business criteria that people look for. And to your point earlier, Randy, if the acceleration of that cost does drop faster then everybody wins. When you’re agnostic to the technology and something moves faster than it was thought to, you bring it into your product and your users and customers win. When it becomes a problem is when you have to, by definition, limit yourself to the technologies that you’re using. So, I see . . .That may sound, you know, something unexpected to hear me say, but if that happens, we win, and I think some of the other vendors here win as well.

31:09 RK: Jeff?

31:10 KS: And then ultimately our collective customers win.

31:12 RK: Actually, I think Roger wins the most because he’s got the bid.

31:16 RP: That’s right.

31:18 RK: Jeff, please?

31:20 JD: Yeah, so I think the cost inflection point for the 7200 RPM drive is closer than what people otherwise naturally think if you look at it at the system level. But the one question that . . . A conversation I had with Howard Marks, who’s on our team, a couple of weeks ago was around, is there some critical mass of manufacturing capability at the hard drive manufacturers that at some point as . . . You know, we’re watching volumes decline on an annualized basis, at what point do you get to a spot where it becomes like the LTO market, where there’s one company keeping it up and keeping a fab running and things like this. Our drive fabs are very expensive to maintain, so I don’t have the answer to that, but it’s an interesting question to me because there is a point where this may break.

32:10 RK: Yeah, we’ve got three disk drive manufacturers now, and a few of us go back to when there were over a hundred, so it’s quite the change.

Alright, so we’re passed time here, I do want to thank you all. We could talk for another hour, easily.

32:25 S?: Easily.

32:26 RK: You guys are all very interesting to speak with, so I appreciated this. Thank you very much for attending and thanks to everyone on Flash Memory Summit. Again, thanks again, everyone.

32:37 KS: Thanks Randy.