Can Scale-Out Storage Also Scale-Down?

This is a guest post by Tim Wessels, the Principal Consultant at MonadCloud LLC.

Private cloud storage can scale-out to meet the demands for additional storage capacity, but can it scale-down to meet the needs of small and medium-sized organizations who don’t have petabytes of data?

The answer is, yes it can, and you should put cloud storage vendor claims to the test before making your decision to build a private storage cloud.

Scale-out cloud storage

A private storage cloud that can cost-efficiently store and manage data on a smaller scale is important if you don’t need petabyte-capacity to get started. A petabyte is a lot of data. It is equivalent to 1000 terabytes. If you have 10 or 100 terabytes of data to manage and protect, a scale-down private storage cloud is what you need to do that. And in the future, when you need additional storage capacity, you must be able to add it without having to rip-and-replace the storage you started with.

The characteristics of scale-down, private cloud storage make it attractive for organizations with sub-petabyte data storage requirements.

It's important for storage to be both scale-out and scale-down

You can start with a few storage servers and grow your storage capacity using a mix of storage servers and storage capacities from different manufacturers. A private storage cloud is storage server hardware agnostic so you can buy what you need when you need it.

Scale-down, private cloud storage should employ a “peer-to-peer” architecture, which means the same software elements are running on each storage server.

A “peer-to-peer” storage architecture doesn’t use complex configurations that require specialized and/or redundant servers to protect against a single point of failure. Complexity is not a good thing in data storage. After all, why would you choose a private cloud storage solution that is too complex for your needs?

Scale-down, private cloud storage should also be easy-to-use and easy-to-manage.

Easy-to-use means simple procedures to add, remove or replace storage servers. It also means using storage software with built-in intelligence that can protect your data and keep it accessible without a lot of fine tuning or tinkering to do it.

Easy-to-manage means you don’t need a dedicated storage administrator to keep your private cloud storage cluster running. An in-house computer systems administrator can do it or you can hire out administration to a managed services provider who can do it remotely.

So just how small is small when it comes to building your own private cloud storage? Small is a relative term, but a practical minimum from a hardware perspective would be about 10 terabytes of usable storage. There is nothing hard and fast about starting with 10 terabytes of usable storage, but once you start moving data into your private storage cloud, you should have an amount of usable storage that is appropriate for the uses you have in mind.

If you have never built your own private cloud storage, you will need to determine which private storage cloud vendor has a simple, easy-to-use and easy-to-manage, private cloud storage solution that will work for you.

The best way to help you make your decision is to conduct a Proof-of-Concept (POC) to determine which vendor will best meet your requirements for private cloud storage. Every vendor will tell you how easily their cloud storage scales out, but they may not mention if it can easily scale-down to meet the needs of organizations with sub-petabyte data storage requirements.

A Proof-of-Concept is not a whiteboard exercise or a slide presentation. A POC is done by having vendors showing you how their storage software running on their storage hardware or your storage hardware works. A vendor who cannot commit to a small-scale POC may not be a good fit for your requirements.

The applications you plan to use with your private storage cloud should also be included in your POC. If you are not writing your own applications, then it is important to consider the size of the application “ecosystem” supported by the storage vendors participating in your POC.

After ten years in the public cloud storage business, Amazon Web Services (AWS) has the largest “ecosystem” of third-party applications written to use their Simple Storage Service (S3). The AWS S3 Application Programming Interface (API) constitutes a de facto standard that every private storage cloud vendor supports to a greater or lesser degree, but only Cloudian guarantees that applications that work with AWS S3 will work with Cloudian HyperStore. The degree of AWS S3 API compliance among storage vendors is something you can test during your POC.

Running a POC will cost you some time and money, but it is a worthwhile exercise because storage system acquisitions have meaningful implications for your data. It is worth spending a small percentage of the acquisition cost on a POC in order to make a good decision.

The future of all data storage is being defined by software. Storage software running on off-the-shelf storage server hardware defines how a private storage cloud works. A software-defined private storage cloud gives you the features and benefits of large public cloud storage providers, but does it on your premises, under your control, and on a scale that meets your requirements. Scale-down private cloud storage is useful because it is where many small and medium-sized organizations need to start.

Tim Wessels is the Principal Consultant at MonadCloud LLC, which designs and builds private cloud storage for customers using Cloudian HyperStore. Tim is a Cloudian Certified Engineer and MonadCloud is a Preferred Cloudian Reseller Partner. You can call Tim at 978.413.0201, email twessels@monadcloud.net, tweet @monadcloudguy, or visit http://www.monadcloud.com

 

YOU MAY ALSO BE INTERESTED IN:

Object Storage vs File Storage: What’s the Difference?

New Use Cases for Smart Data and Deep Learning

In case you missed it, we recently announced a project with advertising giant Dentsu, QCT (Quanta Cloud Technology) Japan, and Intel Japan. Using deep learning analysis and Cloudian HyperStore’s smart data storage, we’re launching a billboard that can automatically recognize vehicles and display relevant ads.

The system has ‘seen’ 3,000-5,000 images per car so that it can distinguish all the various features of a particular car and identify the make, model, and year with an average 94% accuracy. For example, if someone is driving an older Mercedes, the billboard could advertise the latest luxury car. Or, if someone is driving a Prius, then the billboard could show eco-friendly products. It’s important to note that none of this data is stored – it is simply processed and then relayed into a relevant ad.

Cloudian and Dentsu use smart data for billboardsOur smart data system sifts through thousands of images to accurately identify vehicles

You can also turn to this piece from CNN Money to learn a bit more about the project. The first billboard will be up and running later this year in Tokyo.

Broader Potential for Innovative Technology

 

One of the reasons why this technology is possible is through the use of metadata. Typically, big data is just stored passively for future analysis. Because this data is unorganized and untagged, it requires a good amount of effort in order to discover and pull out specific information.

Object storage, on the other hand, can have metadata tags attached to them. We run the data through real-time classification and auto-recognition/discrimination, which means these metadata tags are attached on the fly. As a result, we use this ‘deep learning’ to turn big data into smart data.

How IoT and deep learning combine to make smart data

So what are the implications of this technology beyond advertising? There is potential for tremendous applications of deep learning in other fields, such as improved object recognition for self-driving cars, higher quality screening for manufacturing equipment, or even better tumor detection in MRIs.

Still skeptical? Sign up for a free trial and test out our smart data storage for yourself.

Shifting Technology Habits and the Growth of Object Storage

Technology is, for many of us, a vital and inextricable part of our lives. We rely on technology to look up information, keep in touch with friends and family, monitor our health, entertain ourselves, and much more.

space

However, technology wasn’t always so ubiquitous – it wasn’t too long ago that our wireless phones had limited features and even fewer users actually using these features. Here’s the breakdown from 2004, according to a study from the Yankee Group:

This means that just over 10 years ago, less than 50% of cell phones had internet access and less than 10% had cameras. Even with 50% of phones having internet access, only 15% of users took advantage of this feature.

pew research center

By contrast, look at this survey conducted by Pew Research in 2014:

Among the 18-29 age group, text messaging and internet are more frequently used features than phone calls, which is indicative of the tremendous shift in technology use over the past few years. This study doesn’t even cover a major feature that many users use their phones for: pictures. As younger users turn almost exclusively to smartphone cameras for their photos (and, of course, #selfies), they turn to photo-sharing sites to host and display their images.

Photos are just one type of the ever-growing deluge of unstructured data, though. For enterprises, unstructured data also includes emails, documents, videos, audio files, and more. In order for companies to cost-effectively store this data (while keeping it protected and backed up for end-users), many of them are starting to turn to object storage over traditional network-attached storage (NAS).

Some of the benefits of object storage include a lower total cost of ownership (TCO) and the ability to easily scale up as data needs grow. That by itself is not enough, though. With a solution like our very own HyperStore, in addition to the affordable price (as low as 1c per GB per month) and infinite scalability (from tens of terabytes to hundreds of petabytes), we offer easy management and access control, plus strong data protection with both erasure coding and replication settings. You can read about all of HyperStore’s features and benefits here.

Unstructured data use is only going to continue to grow. Smartphones and other data-intensive technologies will only become more prevalent, and you’ll want to be prepared to meet that growth. Learn more about Cloudian’s hardware and software solutions today.

Lenovo Solves Data Storage Needs with a New Appliance

As our lives become increasingly digital, we’ll generate more and more data. By current estimates, storage needs are doubling in size every two years. That means that by 2020, we will reach 44 zettabytes – or 44 trillion gigabytes – of data, with most of that growth as unstructured data for backups, archives, cloud storage, multimedia content, and file data. This growth in data is quickly outpacing IT budgets. It’s clear we need a new storage approach if we hope to keep up with this deluge of data.

Introducing a New Appliance by Lenovo and Cloudian

 

Lenovo, together with Cloudian, is attacking the $40B storage market with a new, innovative capacity storage appliance for low-cost, scalable storage which addresses 80% of customer’s data needs. We are proud to introduce the Lenovo DX8200C powered by Cloudian as the storage building block which can scale to this challenge and further drive datacenter efficiency and investment protection.

Lenovo DX8200C powered by CloudianThe Lenovo DX8200C powered by Cloudian is an affordable and scalable object storage solution.

Offered as part of Lenovo’s StorSelect software-defined storage program, this factory integrated appliance is built upon Lenovo’s industry-leading servers and features:

  • S3: S3 is the de facto cloud storage standard as stated by Gartner. Cloudian is the only native S3-compatible mass capacity storage solution on the market, enabling customers and partners to take advantage of the $38B AWS ecosystem
  • Affordability: Lower the total cost of ownership (TCO) to $0.1 per GB per month
  • Scalability: The flexible design allows you to start small and scale up to 112 TB of storage capacity per node
  • Security: Utilize always-on erasure coding and replication to ensure your data is protected
  • Simplicity: Single SKU for full appliance and support

The Lenovo DX8200C powered by Cloudian delivers a fully-integrated and ready-to-deploy capacity storage system, reducing risks and variables in the datacenter. Global support is provided by Lenovo’s top-rated support team.

Additionally, what sets this appliance apart from others is the use of Cloudian’s HyperStore storage platform, bringing with it a full host of key features, including:

 

In a news announcement today, David Lincoln, GM of the Data Center Group at Lenovo, stated that “the Cloudian HyperStore solution enables us to deliver leading innovative, software-defined storage capabilities to enterprises and service providers worldwide.”

Michael Tso, CEO and co-founder of Cloudian, reiterated this point by stating that “enterprises and value-added resellers (VARs) can maximize their business investment and revenue opportunities with this fully turnkey, channel-ready, 100 percent S3 object storage solution.”

With more and more industries requiring massive amounts of data to be stored, this partnership with Lenovo represents a vital next step – one where pre-loaded appliances make it easy for companies to both integrate with existing infrastructure and scale out for large deployments.

The Lenovo DX8200C powered by Cloudian will be available worldwide in the third quarter of 2016 but Lenovo and Cloudian are working closely together to address all customer needs in the meantime.

Start Small and Grow with Unlimited Scale

It seems that much of the current conversation around data revolves around how much of it there is and how much there will be in the coming years. While this macro level perspective is important and should help inform how data is stored, it’s also important to focus in on the micro level use cases.

space
Many companies tout that they can start big and go bigger. The issue with this approach is that it ignores a large swath of customer needs. What if you don’t need hundreds of TBs of storage immediately? What if you want to start small, but anticipate growth down the line?

Cloudian HyperStore 6.0

Scale as you grow with Cloudian HyperStore

 

Cloudian offers the flexibility to start small without sacrificing any of the robust features in our HyperStore operating environment. We offer both software and hardware solutions so you can start with as little as tens of TB of storage and scale up to hundreds of PBs.

Cloudian HyperStore can be deployed on off-the-shelf commodity hardware for 1c per GB per month, making it both easy and affordable to scale out as your data grows. As you add more data, HyperStore will automatically divert from highly used disks to less used disks to avoid imbalance. Of course, as you scale, security and data resiliency become more and more vital, which is why this smart disk balancing is only one part of the wider array of protection features in HyperStore.

Big protection for all your data

 

No matter how much data you’re storing, we’ve built in some of the most robust security features possible to protect your data. On a read request on your data, all replicas are checked and missing or out-of-date replicas are automatically updated or replaced. As a result, you don’t have to worry about restoring to outdated data.

The Cloudian Management Console lets you monitor your system’s health and get alerts when things are off. Be proactive by utilizing replication or erasure coding (or both!) to properly protect your data. Plus, spread your data out among geographically independent data centers as an added contingency against data loss. If you need to conduct a more granular check-up on your system, we’ve implemented an “object GPS” so you can quickly and easily locate any specific object within a given bucket.

As your organization grows, your access needs will change as well. HyperStore gives you multi-tenancy controls so that you can give role-based access to administrators and users.

From the very beginning, we believed strongly in providing customers with all the tools they needed to create the storage platform that works for them. In addition to the HyperStore software, we also have turnkey appliances that enable small deployments with the potential to scale up to many PBs.

Cloudian HyperStore Appliance 1500 The Cloudian HyperStore 1500 Appliance offers hot-swappable hardware, automated data tiering, and unlimited scale.

If you’d like to try Cloudian HyperStore for yourself, sign up for a free trial today.

Simplifying Enterprise Data Protection with Rubrik

Data center sprawl can be a real pain to manage, and it’s only made worse when dealing with legacy architecture. As data needs continue to grow exponentially, having an efficient and cost-effective data protection solution in place is more necessary than ever. If you’re still dealing with outdated hardware and software, then it will only become increasingly complex (and frustrating) to deal with data migration, backup, and recovery.

space

Working with Rubrik for Better Data Protection

 

To help address these pain points, Cloudian has partnered with Rubrik to bring simple, seamless, and secure backup and long-term data retention solutions to enterprises.

Rubrik acts as a sort of ‘time machine’ for VMs. Backup software, catalog management, replication, and deduplicated storage are all brought into a single appliance. As a result, you get incredible ease of use and infinite scalability. Rubrik is able to deliver near-zero recovery times without the need for rehydration.

How Rubrik and Cloudian work together

How Rubrik and Cloudian HyperStore work hand-in-hand to provide efficient and affordable data backup and protection.

How Cloudian Fits In

 

Rubrik serves as a smart on-ramp to Cloudian HyperStore – no additional software installations or plugins necessary to connect the two. This, in turn, lets IT automate backup, replication, and archival via a policy-based engine. By sending the deduplicated data to HyperStore, IT can save money on data transfer and storage costs.

Furthermore, Cloudian HyperStore is fully software-defined with no affinity to hardware. It is a scale-out, 100% native S3 object storage platform designed for large but flexible storage solutions – ideal for storing unstructured data and content. HyperStore is robust and durable, but also flexible, and we kept ease of management and usability in mind from the start. Additionally, HyperStore provides seamless tiering and replication to public cloud providers such as AWS, Glacier, and other S3-compatible endpoints, including tape.

You can read more about Cloudian HyperStore’s features and benefits here.

—

Having ironclad data protection in place is vital. Read more about how Rubrik and Cloudian can help in our joint solution brief.

Betting on Software-Defined Storage

Cloudian

Picking a company to advise is not always easy, but sometimes it just clicks – take, for example, my recent decision to join Cloudian’s advisory board. As outlined in my recent blog post, I look at several factors before deciding to advise a company:

  • The potential for growth
  • How well they know their target audience
  • The quality of the team
  • How passionate everyone is (not just the employees of the company, but also my own passion and excitement)

Of course, these are broad factors I always consider. When it came to Cloudian, I had plenty of other questions as well:

  • Does the product/service actually work?
  • Does it scale?
  • Does it save money?
  • Does it enable a more agile operations environment?

 

Everything I’ve heard from customers – and from the team, of course – indicates that the answer is a definite YES. But before I go into more detail on what Cloudian does, a bit of background on object storage.

Storage: Then and Now

 

Storage was long dominated by firms that did a good job protecting your data – and serving it up for the then dominant vertically scalable workloads – and that, in turn, locked you into their proprietary hardware and software, resulting in the largest margins in IT infrastructure. Back in 2008, many firms such as Intel and LSI and the whitebox server providers and entrepreneurs including myself thought: hmm, storage is taking more and more of the IT budget but is not keeping up with new application architectures, new patterns of data generation, and a new generation of storage managers. There has got to be a better way.

And that better way is now called software-defined storage.

Today, storage is much better than it was in 2008, with far better economics, business models that pass the benefits of flash and network improvements on to customers, and a shift towards scale-out, developer-friendly architectures. Much of this change has been forced by Amazon and S3, who set the bar quite high for easy-to-use, massively scalable, and comparatively less expensive storage.

How Cloudian Fits In

 

Cloudian provides on-premise software-defined storage at 1c/GB. This by itself does not set the company apart, but they made a smart move early on – they bet on Amazon’s S3 API. Instead of inventing another proprietary API in a sea of proprietary APIs, they decided to focus on S3 from day one. This gives them a unique offering – 100% S3-compliant storage that uses metadata in interesting and intelligent ways.

So if you and your developers and the software you are running can interact with S3 – it can interact with Cloudian. Cloudian’s management interfaces blow away the AWS storage GUI, incidentally – giving you a firm-wide view into your data, grouped into blobs, and the policies you’ve applied to these blobs. So your developers are happy because it scales and it is basically S3 out the front and your storage teams are happy because they retain the control and visibility they need to do their jobs. And that’s not to mention “native” multi-tenancy, which is one of many reasons service providers like NTT rely on Cloudian.

The results have been outstanding, with customers providing excellent reviews – and this growing word of mouth has led to a sales acceleration.

I went into the Cloudian offices recently to talk a bit more about my reasons for joining the advisory board, and you can watch the video here:

I’m excited to get a front-row seat as Cloudian continues to ramp up their momentum and grow, and I’m looking forward to getting to know everyone in this wonderful team of people.

Hypervisor Agnostic Cloud Storage for VDI Home Directories

Virtual Desktop Infrastructure (VDI) is, simply put, the process of running virtualized desktops for users within an organization using server-based computing. You have the option of running persistent or non-persistent virtual desktops.

If you are interested in knowing more about these types of deployments, you can find out more by using your favorite search engine. If you are like me and you just want to click on a link and have the information pop up on your screen, here is a blog post that explains VDI in more detail.

For example, a VDI deployment with Hyper-V can be explained with the diagram below. The RDP client logs in to the web browser server, then the RD Connection Broker server lists and orchestrates the virtual machines. The AD server authenticates access. The RD Session Host server redirects the RDP client to the right virtual machine and the RD Gateway server publishes and makes the VM available to the authorized user.

VDI deployment with Hyper-VVDI deployment with Hyper-V

Whether you are running persistent and/or non-persistent virtual desktop solutions, your organization will still require storage for your virtual desktop user home directories, etc. You can deploy additional LUN storage. However, with traditional LUN storage, you are likely to run through the following process:

    1. You, as the VDI administrator, will define the storage requirement for each user.
    2. You’ll submit the storage requirement request to your storage administrator.
    3. The storage administrator creates the LUN storage base on your requirements.
    4. You create your master image/golden image for your virtual desktops.
    5. You’ll provision the virtual desktops to your user when you are ready.
    6. You’ll run into late night support calls when your user runs out of disk space to store their files/presentations/media in their user home directories.
    7. You make a call to your storage administrator to provision more storage @ 1am in the morning.
    8. You and your storage administrator work through the wee hours on the support calls.
    9. Repeat steps 6 through step 8…monthly, weekly, or even daily!

 

A Better Solution for VDI Users with Cloud Storage

 

 

Did the previous process flow sound familiar? What if you and your storage administrator could host your own hypervisor agnostic on-premise cloud storage solution within your own firewalls for your virtual desktop users’ home directories? What if you, as the storage administrator, can easily increase the bucket capacity for each user when a support call comes in with a few clicks? All of this is possible because software-defined cloud storage solutions such as Cloudian are designed from the ground up to be:

    • Simple
    • Highly scalable
    • Always-on with secure and encrypted access

 

VDI deployment with cloud storageHyper-V VDI deployment with cloud storage for VDI users’ home directories

Instead of fielding late-night support calls, easily provision S3 buckets for each virtual desktop user and provide secure S3 portal access without missing a beat. With some on-premise cloud storage solutions, you can:

    • Use the inherent multi-tenancy feature to create and deploy storage for all VDI user home directories.
    • Use QoS to throttle each user’s PUTs and GETs.
    • Monitor per user usage and easily review reports for chargeback purposes.
    • Use replication or erasure coding on a per storage policy to ensure each group has the right data protection benefits.

 

To see how easily a virtual desktop user can use a secure cloud storage system as a home directory, watch the following video:

A Virtual Desktop user using Cloudian HyperStore Cloud Storage as his home directoryPer bucket and per user granularity is possible. It is because some cloud storage solutions are fully Amazon S3 compliant. Think about it – rather than creating additional LUN storage for your VDI users’ home directory requirements, I can simply create a master image that has the secure on-premise cloud portal for every group and every virtual desktop user defined in the web browser of the desktop master/golden image.

From that one golden image, I can deploy hundreds or thousands of virtual desktops that have a secure on-premise cloud storage solution for all my virtual desktop users. My virtual desktop users can easily access and use my deployed cloud storage as their home directories from any web browser. Best of all, many cloud storage solutions also support NFS/SMB/FTP via its native file access integration. This means you get the benefit of industry standard file protocols access without any third-party gateways!

With cloud storage solutions for user home directories and for file sharing purposes, we get the following benefits for virtual desktop deployments:

    • Simplified virtual desktop deployment with a hypervisor agnostic cloud storage solution. Minimize your late night “virtual desktop user is out of storage” support calls.
    • Scale-out and highly available home directories for virtual desktop user storage. Each storage bucket capacity is tunable for each virtual desktop user. There is no single point of failure with cloud storage.
    • Secure in-flight data and data-at-rest with AES-256 encryption.
    • Simplified and flexible data protection. Virtual desktop users can manage their own data protection and retention requirements.
    • Predictive analytics for storage planning. Use built-in analytics to manage your storage growth requirements.
    • Fully S3 compliant storage. This means you can support hundreds of S3 compatible applications using your very own on-premise cloud storage solution.
    • Software-defined. Use any x86 commodity server to deploy your own hybrid, private, and public cloud storage solution and minimize your operating cost.
    • Manage access and performance with QoS throttle. Easily manage user PUTs and GETs by using QoS throttles at the group or user level.

 

With the availability of secure cloud storage solutions for on-premise deployments, it’s a no-brainer to look into simplifying our lives so that we can get away from the repetitive user storage support calls in the middle of the night. Say goodbye to conversations such as:


“Help! I can’t save my executive briefing presentation and videos on my desktop. I need to securely share the presentation and videos with our Tokyo office. I am getting an error on my desktop pointing out that my e:\ drive and my user home directory is full. And NO, I cannot delete any of the existing files to make space for these new files and videos because I need all of it to be always accessible to me.”

To find out more about the advantages of cloud storage solutions, visit Cloudian.

Cheers,

Dominic

Reduce Media Production Workflows with Cloudian & QStar

The National Association of Broadcasters (NAB) attracts over 100,000 people every year from over 160 countries, with attendees gathering in Las Vegas to explore everything from broadcast to post-production to virtual reality and everything in between that has to do with media & entertainment (M&E). It’s an intense show, but totally worth it if you’re in the M&E space.

space

So what brought Cloudian to NAB last month? We provide scale-out solutions for software-defined object storage. Cloudian integrates with existing media asset management (MAM) solutions to simplify production workflows. To do this, we work with leading technology partners, such as QStar. QStar Archive Manager software provides a NAS-style archive gateway to Cloudian HyperStore’s fully S3-compatible object storage platform. Cloudian and QStar simplify the integration of self-managing, self-healing storage into existing MAM environments. QStar and Cloudian Check out this quick video we filmed live at NAB this year while catching up with QStar’s VP of Sales and Marketing, David Thomson:

If you want to learn more about the Cloudian and QStar integration, check out our webinar: Simplify MAM Environments with Active Archive and Scale-out Storage Until the next episode of Time Out with Tam! @Cloudian_Tam

#smartdatastorage

Reinventing Storage Administration at Scale with Cloudian HyperStore 6.0

Learn More About Cloudian HyperStore– Download White Paper

At Cloudian, we are continuously enhancing and evolving our flagship product HyperStore, a fully S3-compatible object storage technology. But even with all our rich history of innovative releases, I’m most proud of the upcoming Cloudian HyperStore 6.0 – we have risen to a new challenge…@scale operations. How do we help IT administrators manage Petabytes of storage and still have time for a cup of tea?

As data volumes start to outpace operating budgets and the headcount needed to manage them, today’s enterprises are turning to full-featured, low-cost software defined storage technologies with almost limitless scalability and accessibility. While the software can be deployed on low-cost commodity hardware, thereby reducing the total cost of ownership (TCO), the cost of managing storage infrastructures – managing operations, performance, protecting data and tuning workflows and processes – has remained fixed.

A 2016 Gartner report found that the TCO for a terabyte of traditional on-premises storage was $2009 per terabyte, with 62 percent of the costs resulting from hardware and software acquisition ($1245). The operational costs associated with this terabyte of storage were 26 percent of the total, or $511, with “other” costs adding up to 12 percent ($253).

With software-defined storage like Cloudian HyperStore, the TCO for a terabyte of storage now drops to $866 per year, and with costs as low as one cent per gigabyte per month, hardware and software acquisition are reduced to just 14 percent of the total costs, or $122.

Unfortunately, as reported by Gartner, the cost of operations for this model remains fixed at $511, which now represents 58 percent of total yearly storage costs, significantly outweighing hardware and software expense. It’s clear that to further drive costs from cloud storage, organizations will need to find ways to reduce management burden and operational costs.

Cloudian HyperStore 6.0 is designed to help operate and manage at scale. The new release simplifies and automates the operational management of cloud storage by creating new system management features, which scale to meet the requirements of multi-petabyte, multi-region storage deployments. It lowers the management costs and reduces administrative burden. HyperStore 6.0 advances the vision Cloudian has shared for Smart Data Storage by delivering fully integrated smart data operations within the platform for seamless and extremely cost-efficient management.

The basis for many of 6.0’s key features came from our customers. They shared their “wish list” of management tools and functionality that could streamline or eliminate the manual tasks that normally weigh down the storage admin in 100+ node deployments.

Introducing Storage Operations and Management @Scale

 

Cloudian HyperStore 6.0 provides new tools, user interfaces and automated features to scale management and operational efficiency across complex cloud storage infrastructures, lowering storage management costs for enterprises and streamlining operational tasks for storage administrators. The new 6.0 release can reduce operations management as nodes are added, helping lower the overall operational costs to businesses as data volumes continue to grow. HyperStore 6.0 also provides greater data durability with more continuous automated failure resolution, and delivers robust system tuning tools for proactive and low-cost management of system health and operational efficiency for data storage at petabyte scale.

Operations @scale. In Cloudian HyperStore 6.0, an entirely new operations console provides an instant 360-degree view into storage system performance, with an enhanced GUI that greatly increases the visibility of key data. The new operations console enables storage admins to view and manage hundred of nodes across multiple data centers and cloud environments in a single screen, while automating such operations as adding and removing nodes with the simple configuration of an IP address. And the user interface isn’t just to look at: Users can click on the data representations to drill down and access detailed information and take action directly from the interface screen. 6.0 also automates non-disruptive, node-by-node rolling technical patches and upgrades that are delivered via touchless distribution and management systems, saving time and resources for storage admins.

Cloudian HyperStore operations console HyperStore’s newly renovated operations console provides a 360-degree, drillable view of the health of your system

Durability @scale. HyperStore 6.0 also extends data durability with automated features that constantly evaluate data integrity – even data at rest – to ensure that storage within the system is always repaired and data is always verified. When the system identifies data failures, it proactively repairs and rebuilds them. The Cloudian HyperStore 6.0 system proactively monitors and scans workflows for I/O or disk failure, which Smart Redirect features can track and map to help trigger automatic repairs. The new release enables both local cache replication for data protection and Amazon S3 cross-region replication – one of Cloudian’s many rich S3 compatible features – for regulatory compliance and disaster recovery.

Cluster rebalance and data rebuild information The storage admin can view cluster rebalance and data rebuild information from HyperStore’s robust analytics capabilities

Tuning @scale. Cloudian HyperStore 6.0 simplifies system tuning to optimize the infrastructure for the highest level of system health and data protection. Visual Storage Analytics automatically identifies “hot spots” and rebalances workflows for storage I/O protection. It also can locate object parts using new “Object GPS” functionality that track data distribution with high granularity across nodes, racks and data centers around the globe.

HyperStore 6.0 operations console The HyperStore 6.0 operations console allows the storage admin to view real-time capacity consumption and system performance

As before, Cloudian continues to keep pace with incorporating changes to the AWS S3 API into HyperStore. Cloudian is the leading object storage vendor with 100 percent S3 compatibility, and the only vendor to offer an S3 guarantee.

To learn more about how Cloudian HyperStore 6.0 can help your company simplify and automate the operational management of your hybrid storage infrastructures, contact Cloudian at www.cloudian.com.

Also watch this blog over the next few weeks as we add more detailed posts about the features of 6.0.

Stay tuned,

Paul

S3 API & Extensions for Enterprise Object Storage

Amazon’s S3 API is the de-facto standard for object storage APIs. Having multiple service providers, software providers, and applications standardize on S3 has made it easier to interchange between them and rapidly stand up new uses for object storage. But there are different grades of S3 compatibility. Some software and solutions provide only the basic CRUD (create, remove, update, delete) functions. At the other end is Cloudian’s Hyperstore, committed to providing the highest fidelity S3 compatibility backed by a guarantee.

The S3 API is an HTTP/S REST API where all operations are via HTTP PUT, POST, GET, DELETE, and HEAD requests. Each object is stored in a bucket. Beyond the basic object CRUD operations provided by S3, there are many advanced APIs like versioning, multi-part upload, access control list, and location constraint. There are multiple options for encryption including (1) server-side encryption where the server manages encyrption keys, (2) server-side encyption with customer keys, and (3) client-side encryption where the data is encrypted/decrypted at the client side. Though no single S3 user is likely to use all of the advanced APIs, the union of APIs used by different users quickly covers them all. The table below highlights some advanced object storage APIs supported by S3:

S3 Feature Azure Google Cloud OpenStack Swift
Object versioning No Yes Yes
Object ACL No Yes No
Bucket Lifecycle Expiry No Yes Yes
Multi-object delete No Yes Yes
Server-side encryption No Yes Yes
Server-side encryption with customer keys No No No
Cross-region replication Yes No Yes
Website No No No
Bucket logging No No No
POST object No No No

Table 1 – Comparison of some S3 advanced object storage APIs[1]

S3 API compatibility is a prerequisite, but not sufficient to provide object storage for enterprises. There are 4 additional areas that Cloudian has added to make S3 object storage enterprise-ready.

 

  1. Software or Appliance, not a service.The software-only package includes a Puppet-based installer with a wizard-style interface. It runs on commodity software (CentOS/RedHat) and commodity hardware. The appliances come in a few fixed models ranging from 1U (24TB) to the FL3000 series of PB-scale in 8U form.
  1. APIs for all functions
    • Configuration
    • Multi-Tenancy: User/Tenant provisioning
    • Quality of Service (QoS)
    • Reporting
    • S3 Extensions: Compression, Metadata APIs, Per-bucket Protection Policies.

    Highlighting the per-bucket protection policies feature, each bucket can have its own protection policy. For example, a“UK3US2” policy can be defined as UK DC with 3 replicas and US DC with 2 replicas. Another example is a “ECk6m2” policy as DC1 with Erasure Coding with 6 data and 2 coding fragments. As buckets are created they can be assigned a policy.

Bucket
Figure 1 – Per-bucket protection policies example

  1. O&M tools to install, monitor, and manage.In addition to the installer, a single pane web-based Cloudian Management Console (CMC) does system administration from the perspective of the system operator, a tenant/group administrator, and a regular user. It’s used to provision groups and users, view reports, manage the cluster, and monitor the cluster.

Cloudian Management Console

Figure 2 – CMC dashboard

  1. Integration with Other Products
    • NFS/CIFS file interface
    • OpenStack, CloudPlatform
    • Tiering to any S3 system (public or private).
    • Active Directory, LDAP

The opportunity and use case for enterprises and object storage has never been more compelling. Amazon S3 API compatibility ensures full portability of already working applications. Using Cloudian’s HyperStore platform instead of AWS, enterprise data can be brought on-premise for better data security and manageability at lower cost. For STaaS providers, S3 API compatibility, backed by a full guarantee, provides the same benefits of a fully controlled storage platform, and opens up a large range of compatible applications. Beyond the S3 API, Cloudian is committed to providing all operations by API and has added APIs to make the platform enterprise-ready, including multi-tenancy.

If you would like a technical overview, you can check out this webinar I recently presented, “S3 Technical Deep Dive” and make sure to check out more information on our S3 Guarantee…we’ll run all your S3 Apps anytime and anywhere – Guaranteed!

– Gary

—
[1] References:
http://docs.openstack.org/developer/swift/#object-storage-v1-rest-api-documentation
https://cloud.google.com/storage/docs/xml-api-overview
https://msdn.microsoft.com/en-us/library/azure/dd135733.aspx

Categories