David Axler, Cloudian Sales Engineer
Erasure Coded Storage Policy for Multi-Cloud
In this blog, I will explain how Cloudian Hyperstore protects your data efficiently in a multicloud environment using Erasure Coding (EC) and specifically Distributed Erasure coding which offers resilience to a site/cloud failure
Cloudian HyperStore Supports Multiple flexible data protection policies
In the last blog I talked about RF which is the simplest mechanism for protecting your data by making multiple copies, here we are doing something slightly more complex by splitting data into multiple fragments and then computing additional parity fragments.
This data protection system is well-proven in enterprise storage. If you are familiar with traditional RAID6 you can think of EC as RAID6 across nodes rather than disks.
The big advantage over RF is storage efficiency. When discussing site-resilient designs RF3 is 33% efficient vs EC 7+5 which is 58%. In single site designs you can see efficiencies of 75%+ for EC.
In the video, I have chosen a distributed 5+4 because I only have 9 nodes available to me but this is still 55% efficient. In Cloudian terminology distributed means that the fragments are evenly distributed across 3 or more datacentres allowing for site resilience in case of a failure.
Data Durability
Along with efficiency, distributed Erasure coding allows for high data durability. With so many parity fragments multiple simultaneous failures across the system will have no effect on data availability. For example, in a common EC7+5 policy you can have 5 simultaneous failures across the system without impacting the ability to read it back again. This gives an effective data durability of 14 nines or 99.999999999999
Multiple EC or RF storage (data protection) policies can be implemented on the same cluster at the same time on a per-bucket basis, which ensures that your protection profile matches the specific business requirement for that data.
Configuring Consistency Settings for HyperStore Multi-Cloud with EC
Consistency settings allow fine-grained control of read/write behaviour. With Distributed EC, we have a lot fewer options to choose from as compared to RF with either All or Quorum. This is because strong consistency is a requirement for distributed EC. Below states the necessary configurations for writes to succeed.
All – All fragments must be written
Quorum – The system must succeed in writing K+1 fragments i.e. All the data and at least 1 parity fragment.
Other Storage Configuration Options
As with RF other storage options such as compression and encryption can also be specified, along with application-specific policies depending on the data being stored. As of HyperStore 7.5.1 you can now choose a Hybrid policy threshold whereby you can have RF and EC objects in the same bucket split depending on object size (RF is better for small objects).
Erasure coded storage policies are a very storage-efficient and durable way to store your data particularly when dealing with large objects (>1MB). Using Cloudian’s distributed erasure coding allows for resilience to Cloud outages. By using a Hybrid bucket you can get the best of each policy by using RF for small objects and EC for large objects automatically.
Learn more in this on-demand demo.
For more about Cloudian, visit Cloudian.com.
Or try a Free Trial!