VSAN Technical Questions

With the introduction of VMware’s VSAN product (official GA was March 12 2014), I’ve been spinning some free cycles trying to understand the mechanics of this product.

The details of VSAN striping, provisioning, and cache behavior are hard to find. I have broken down the key unknowns into 3 general question subjects. Please provide any insights you may have, for sharing with the community.

Question Subject #1 – Stripe Layout and Thin Provisioning (Stripe Filling)

  • Layout on HDD: Do VSAN’s 1MB stripes correspond to 1MB of physically contiguous space on the HDDs (minimizing seek times) ?
  • Blocks within each stripe: Does each stripe contain an ordered and defined address range of logical blocks for the object? This subject is particularly relevant as VSAN supports thin-provisioning space efficiency.
  • If there are scenarios where logical fragmentation can occur (non-contiguous blocks within a stripe), does the VSAN implementation ever “defragment” them?  EMC has suggested thick-provisioning pooled LUNs on VNX, and iSCSI LUNs for Isilon, to avoid fragmentation resulting from thin provisioning.

Question Subject #2 – Block Size / SSD Read Cache Page Size

  • What is the block size of VSAN?  I estimate that it may be 8KB, but that could wrong.
  • Does VSAN cache hot blocks in the SSD read cache in full 1MB stripes? Or does it use a smaller, more finely granular, unit?

Question Subject #3 – SSD Read Cache Pre-Fetch / Pre-Warm

  • Writes to any object in the VSAN datastore are always staged in SSD (the SSD write cache essentially replaces the role of a battery-backed NVRAM). When destaging to disk, can VSAN let them remain in the SSD to effectively “pre-warm” that data in the SSD read cache?
  • Does VSAN have the ability to detect sequential read access to an object, and pre-fetch blocks of data that would likely be read next ?

Thank you for reading – looking forward to your insights!

Nutanix Virtual Computing Platform compared to VMware VSAN 1.0

Well written #Nutanix vs #vsan 1.0 post by Marcel Van Den Berg


This post is part of a series of posting on the VMworld 2013 announcements. See this post for an overview of what has been announced.

At VMworld 2013 VMware announced a new product named Virtual SAN or VSAN. VSAN allows to create a distributed storage out of local storage. VSAN is an additional solution in the range of hyperconverged infrastructure solutions. Typical for those solutions is that compute, memory, networking and storage are in the same box. These solutions are also called Datacenter in a Box.

Nutanix is the most mature player in the field of Datacenter in a Box. This post will compare Nutanix Virtual Computing Platform with VMware Virtual SAN 1.0.

The end of this long post has a feature compare matrix showing the most relevant features of both Nutanix VCP and VSAN 1.0

Introduction to converged storage and compute.

We are living in a world where over…

View original post 3,194 more words

Re-using NetApp ONTAP 8 drives with a ONTAP 7.X system

My company recently had a client who was re-using drives from an ONTAP 8.0.X system for a ONTAP 7.X system. Here are a few notes from that experience.

This is not as easy as it sounds. And that IS A GOOD THING. I love how NetApp systems can read any disk (a high level of backward and forward compatibility), yet draw a clear line in the sand of what they will and will not reformat. The ONTAP 7.X system will discover the 8.X disk, which uses a RAID format that 7.X is unfamiliar with, but it is smart enough to say “Look my friend, I know this is a NetApp disk, and yes I even know it’s a newer disk, but I’m not going to let you shoot yourself in the foot and reformat it. I (Mr. NetApp) will only allow you to do what I have been qualified and designed to do.

There are many times when IT engineer would like systems to be less restrictive and just blast things away. But having systems be intelligent enough to only do what they were designed to do is never a dangerous thing. Positive acknowledgement is a wonderful thing!

With the ONTAP behavior being the way it is, yes it is more work, however it is far safer and ensures that engineers know what they are doing and are fully aware of the consequences. Whatever data was on the disk, it will be absolutely 100% unrecoverable once it is consumed back into a ONTAP 7.X system.

One of the first clues that you will see when ONTAP 7.X discovers the 8.X disk is a message with the string “unsupported label version”. That the first clue. Surfing the web will reveal some instructions on fixing this problem (as shown below). However, don’t bother in this case as the following procedure will not work if the disk in from a later ONTAP release:

  1. Lookup the disks which show the error,
  2. Reboot the filer and press ctrlC,
  3. From the 1-5 menu, choose option 5 to boot into maintenance mode
  4. Clear the label on those disks by typing: label wipe <disk_ID>

Rebooting the NetApp filer with 7.X, and performing the above procedure, will not work. The error message that you will discover is something like the following (SN blanked out to protect the customer):

Thu Jun  2 04:07:27 GMT [raid.config.disk.bad.label.version:error]: Disk 0d.112 Shelf 7 Bay 0 [NETAPP   X292_HVIPC560F15 NA00] S/N [XXXXXXXX] has an unsupported label version.

The following is from (http://communities.netapp.com/thread/12897?tstart=2), and is a perfect summary of the situation:

Before we move disks from a 7.3 system (even if spares) to a 7.2 system (and similarly from 8.0 to 7.3) is to download and revert_to the older release prior to moving the shelf so the RAID is compatibile.  Newer ONTAP has a newer label and it isn’t understood from 7.3 to 7.2 even for just spare disks.  If you can’t do revert_to on the system that owned the disks, we use a test/dev/lab system to do the revert.  The other option is to upgrade the 3140 to 7.3 to work with this shelf.

So, the only way I know to make it work 1) put the shelf on a 7.3 system that you can revert_to 7.2    2) upgrade the 3140 to 7.3.  If you decide to stay on 7.2 and remove the shelf to do a revert, remember that there is no hot shelf removal, so you need to halt the 3140 prior to removing the shelf.

In summary –

  1. Put your 7.X disk onto a 8.X system
  2. Downgrade (revert_to) the entire NetApp system from ONTAP 8.X to 7.X, and the disks will be reverted with it.

If you plan on reverting disks often, it might be viable to keep an inexpensive (and small!) NetApp filer around that can quickly upgrade and revert ONTAP versions. Looks like I’ll be trying to provision such a resource myself.

Good luck with it.

Win or Lose, its all about the chase

Watching a Discovery HD show now on a car buff who loves to hunt down rare cars, usually flipping them for a profit (often doing restoring along the way). He appears to really like what he does, which I admire.

His little pitch with the intro is “Win or lose, its all about the chase”. He loves hunting down that rare bird.

This is good advice for all of you. Do what you love. And enjoy doing it, and know that it is the doing that is the reward. There will be victories, and there will be failures. Most of us actually fail far more often than we want to admit.

Thinking deeper still, this is the attitude that the best technologists in IT seem to have. Not the ones that just know how to brand themselves and land a big paycheck, I mean the ones who are doing the real work and bring about innovation. Dont get too caught up in rewards, or success and failure. If you enjoy what you do, and do it with passion, you will lead. And it will never get old.

OK, its 12:40am and I still got a lot of crammin to do for the IBM 000-119 exam. Thank you car dude.

ITs whats for breakfast

Sitting at a table here at the Arrow conference during breakfast, a topic came up about why a particular large provider of IT equiptment and services is having challenges growing in the storage industry. It was a great opportunity for me to provide feedback on the storage product line. Amongst things that I brought up were the relative boring-ness of many 0f their products, the lack of a simulator availble (i.e. a virtual machine), too much process involved in handling system issues, a difficult to navigate parter site… it brought about some rather long faces.

But what is the real core issue? I looked two tables away, and there it was.

What I saw was a table of the lead sales and architects for this product line. It was mostly men, mid-50s, all wearing thick sport jacketss. It was clear, their job was to meet clients, sell boxes, and expect people to just keep buying boxes. Make your margins, sell boxes, make a good paycheck, and eat more breakfast.

I hope in our lifetime we move beyond IT being a cash cow. Its disrepectful to the potential IT offers us, and is holding us back. Customers spent a lot of extra money on products (beyond R+D, manufacturing, and support) to pay for all those professional salaries and big breakfasts.

Grab some salt to go with those practice cert tests

Its 10:37pm and I’m at the Sheraton in Fort Worth, Texas. It is the Arrow “May Days” event this week, and there will be ample opportunity to attend tech sessions (big focus will be IBM, the main event sponsor), meet vendors, and take cert tests.

Before I get to the point of this post, does anybody know why there is not a CVS or Walgreens within 400 miles of downtown Fort Worth? Not even a McDonalds. Im a simple blue collar guy (well, I think of myself that way) and after a miserable plane ride here today all i wanted was a 1.49 McDonalds crispy chicken snack wrap to hold me til dinner. All I can find are pinky-up burger joints, bistros, theme restaurants… perhaps town planners were going for a particular look. I can respect that. But this IS the United States, and as a citizen I EXPECT my McDonalds. For heavens sake, at least put out some hotdog carts. I enjoy fine dining as much as anyone, but I was tired.. I had studying to do.. and not to be anti-social I didnt want to sit in a restaurant and order and wait for food. And about that Walgreens – for the past year, since going to Germany when I was with EMC, I have been a sparkling water junkie. You would think that downtown Fort Worth, in its attempt to be a convention hub (international hub I hope), they would appeal to the popular demand of sparkling water. But no sir, and that alone is a big strike I have with this town.

Well, now that we have established how biased the author gets when he cant get his sparkling water, let me state a recent observation. Be very, very careful when studying for techncial certification exams using on-line questions and answers. Some of the questions are clearly answered wrong. Not sure if this was intentionally leaked by the test authors, or maybe the individuals who stole the questions simply didnt know the material that well.

A great example is as follows, from IBM 000-118:

Which of the following IBM storage products allows for 960 TB (raw) in one standard IBM rack?

  • A – XIV system
  • B – DS5300
  • C – DS8800
  • D – Storwize V7000

The source of this questions says the answer is “A”. Well, until this week, my knowledge of XIV was very limited. But never trusting these things, I did my research and learned that the XIV (as of today – May 2011) can handle no more than 360 TB, and that is raw. After system overhead, and mirroring, realistically you cant get more than 170 TB.

Meanwhile, the V7000 can handle up to 9 expansion enclosures per controller enclosure. Each controller enclosure can be 2U and have up to 24 internal drives. The expansion enclosures (also 2U) can also have 24 drives each. That yields 10 enclosures, 24 drives each = 240 drives in 20U. With 2TB drives, that yields 480TB of raw storage with a V7000 in only 20U. In a 42U rack, you can then have two of these configurations and have a total of 960TB, with a couple of U to spare for your PDUs.

Knowing that the correct answer is “D” for the Storwize V7000, we now have to make a guess. Is the exam itself wrong? If I answer “D”, would I get the wrong answer? Should I answer “A” just to get it right?   OR – was the individual who stole this question and published it on the net careless or perhaps just not strong with IBM?

I respect cert tests, the effort that goes into them, and believe they provide the best current solution to a complicated problem (proving abilities). Just dont go out there and memorize the questions and answers. Study them, and make your time worthwhile. And please be careful, as the suggested correct answers may not be right.

How information technology is impacting your speed

One topic that is without question delightful is the impact of crowdsourcing. Perhaps Im using the word loosely, but to me it is the ability to rapidly acquire data points about so many people, on so many levels, and measure the once unmeasurable.

Crowdsourcing allows you to use your phone to know where traffic jams are on the highway, before you even get on it. The data comes direclty from other peeople’s phones (or some type of device that reports information). Crowdsourcing is how retailers can acquire data on how much of certain things to stock, with a level of precision that was unthinkable year ago. But these are silly, virtually meaningless examples when compared to what lies ahead.

The big one, we all know, will eventually be healthcare in the US, and using information technology to monitor the activity level, diet, and even prescription medication use of citizens to best distribute what is and always will be a limited pool of resources. In case you are wondering, no – I have no opinion on the “goodness” or “badness” of this. It is what it is, and from a cold calculating efficiency perspective of society, one cant help but say “its about time!”

The ability to measure that accurately, and with permission of the court (although legal or not finances will make it happen), is still several years away, perhaps more than a decade.

But worry not, the stepping stone of “how many 36W jeans to stock in June” to this is starting to take shape. On the news today in Massachusetts was a piece on an auto insurance company offering a discount IF you let your car usage activity be monitored. Speed, miles, location, etc. Its called Snapshot (yes the impact of IT is so great we are re-using storage management lingo!). Snapshot makes all the sense in the world. If you drive like a robot and obey all the rules, why on Earth should you pay more to subsidize some maniac like me?

Next step would be that so much financial burden for insurance falls on faster drivers, that they too in large numbers slow down. Anybody who doesnt want Snapshot in their car will be a suspect to be a crazy driver even if their record is clean.

The whole of society is working here to eliminate waste and improve efficiency. Years of police tickets and speed limits have had no impact since it was far too spotty. But now, the link of information technology, crowdsourcing, and the need for operational efficiency amongst the members of society will make this happen no matter how one feels about it. Its sad, and its scary, but its perfectly logical in a cold kind of way.

This is a topic we’ll be sure to discuss frequently – the use of information technology to bind individuals together for improved efficiency of society.