AWS re:Invent Attendees See Benefits of Hybrid Storage

AWS re:Invent was a fantastic show this year. The show has seen phenomenal growth, with over 32,000 attendees, up from 18,000 attendees last year.

AWS re:Invent

Many visitors were looking for solutions to let them integrate their on-premises operations with the cloud. By adopting a hybrid cloud storage approach, they would be able to capitalize on the scalability and cost of cloud storage when appropriate, while also maintaining the cost predictability and control of on-prem storage.

For these visitors, Cloudian proved to be the perfect fit. We provide 100% native Amazon S3 object storage, with automated tiering between the data center and the cloud. Our HyperStore solution is also available directly from AWS Marketplace, which means users can get all their usage and billing data within a single monthly invoice from AWS.

Steve Varner, Principal Data Engineer at Motorola Solutions, visited our booth and had this to say afterwards:

Steve Varner

Interested in learning more about Cloudian? Contact us or try it out for yourself.

 

5 Things You Need to Know About Hybrid Cloud Before You Start

You’ve probably heard the term hybrid cloud, but what is it, and what can it do for you?

It’s a particularly hot topic now because public cloud storage is growing in popularity. As it should be. Public cloud is inexpensive and solves real storage problems.

But public cloud is not for everyone and not for all data types. That’s where the hybrid cloud comes in. By combining public and on-prem storage into a single management pool, hybrid has the potential to deliver the best attributes of both worlds.

This blog series will give you industry perspective, tips, and tech background, so you can decide if hybrid cloud is right for you. We’ll give you the facts with no fluff. Let’s get started.

Here are five quick facts to put hybrid cloud in perspective.

  1. Hybrid cloud storage hype is real: 

Sometimes buzz is just noise, but with hybrid cloud the growing interest reflects real activity. A recent survey found that 68% of organizations said it’s in their deployment plans for the next two years. Typical motivations included:

    • Data Governance / Security: For data governance reasons, about half of organizations reported a need to keep some data on prem.
    • Cost: If data is frequently accessed, costs can quickly add up.
    • Performance: On prem applications may perform poorly when accessing data in the public cloud.
  1. Not all data will live in the public cloud
    Hybrid gives you the ability to keep sensitive data on-premises, rather than putting everything in the public cloud. In the survey, 59% of respondents agreed, stating that an average of 51% of their data needs to remain on prem. Typical use cases for hybrid cloud include:

    • Backup: Cut RTO by hours vs. either cloud or tape; save cost vs. conventional disk
    • Home directory: Keep frequently accessed files local, automatically tier cold files to the cloud
    • Compliance: Store sensitive file types on premise, automatically migrate others to the cloud
  1. Hybrid cloud offers better scalability than public cloud alone

Since a hybrid cloud connects public and private clouds, it provides a unified infrastructure lets you choose the most efficient infrastructure for specific data and workloads. For example, you can take advantage of the cost efficiency of Amazon’s S3 Infrequent Access tier or Amazon Glacier for backup or archive data, use Amazon S3 Standard to provide data access in different regions, while keeping the bulk of active data in a private cloud on-premises.

      1. A unified infrastructure also gives you a few benefits that public or private cloud alone don’t. Hybrid cloud eliminates silos of data, lets you keep data and applications on premises if you need to (usually for compliance or performance reasons), and is easier to manage than separate environments.
  1. Hybrid cloud saves cost

It’s usually true that cloud storage is less expensive than traditional SAN or NAS systems. Public cloud storage today costs as little as 0.4 cents per GB per month. But for frequently used data, access costs can add up. Hybrid cloud lets your store frequently accessed data locally, avoiding cloud data transfer charges. That data can be automatically tiered to the public cloud when it becomes cold. This lets you easily capitalize on the super-low cost of public cloud, while meeting your security, performance, and data governance objectives.

  1. Hybrid cloud will help you survive the data tsunami

The explosive growth of unstructured data is only going to accelerate. With more connected devices and the emerging internet of things (IoT), we’ll go from less than 2 billion devices in 2010 to more than 25 billion by 2020, according to analyst estimates.

That in turn is fueling massive data growth — from 4.4 zettabytes (ZB) in 2013 to 44 ZB in 2020 — much of it generated at the edge, not in the cloud. How will we manage, analyze and store all of that data? Implementing hybrid cloud now provides an architecture that can scale as we face a data tsunami over the next three years.

Next up, we’ll look at steps to get started with hybrid cloud.

Embracing Hybrid Storage

It’s no surprise that Amazon Web Services (AWS) is a dominant force when it comes to the public cloud – it’s a $10B a year business, with nearly 10% of Amazon’s Q2 net sales attributed to AWS.

AWS Q2 net sales

While AWS has been touting public cloud since its inception, only recently has it started to acknowledge the need for hybrid storage solutions. Why? Because it’s simply not realistic for many companies to move all their data to the public cloud.

Private vs. Public Cloud

 

A company may choose to stay with private, on-premises storage solutions if they have existing data centers already in place. Or they may prefer the enhanced performance and extra measure of control that comes with on-premises storage.

Nonetheless, public cloud storage has significant advantages. It’s easy to implement, scales on demand, and automates many of the data management chores.

Neither option is clearly better than the other – in fact, customers are spending more than ever on both private and public cloud solutions. IDC forecasts that total IT spending on cloud infrastructure will increase by 15.5% in 2016 to reach $37.1B.The bottom line is that companies need both on-prem and cloud solutions.

The Best of Both Worlds: Hybrid Storage

 

What’s needed is a solution that allows you to enjoy that advantages of both — the speed and control of on-prem and the on-demand scalability of cloud. And ideally, you’d get both within a single, simple management model.

That’s what Cloudian HyperStore is. It’s S3 cloud storage that physically sits in your data center. And, it looks and behaves exactly like Amazon S3 cloud storage, so your apps that work with Amazon will work with Cloudian. Best of all, you can manage the combined Cloudian + Amazon S3 storage pool as a single, limitlessly scalable storage environment.

Amazon Makes It Easy

 

Fortune summed up Amazon’s need for a hybrid compute model in their recent article, stating:

It’s become clear that AWS, which is the leader in public cloud, will have to address this issue of dealing with, if not embrace, customers’ on-premises computing.

Thankfully, in the storage world they’ve already addressed this by adding Cloudian HyperStore directly to the AWS Marketplace. We announced this last month, but it bears repeating because it’s an important step in AWS’s evolution.

The advantages in moving towards hybrid storage are numerous. Everything folds up to AWS, so even usage and billing from private cloud will be centralized in the monthly AWS invoices. More importantly, Cloudian HyperStore was built from day one to be fully S3 compatible, which ensures complete investment protection.

So if you’re debating between public and private cloud options for your company, remember that you can still get the best of both worlds. Check out Cloudian HyperStore for a better hybrid storage solution with AWS and Amazon S3.

AWS CLI with S3-Compatible Storage

There’ve been a lot of discussions about Amazon’s Simple Storage Service (S3) and Amazon Web Services (AWS). It seems to me that everyone is saying that they are Amazon S3-compatible or that they work with S3 storage. That makes me wonder, what is the best way to validate a solution or test it out to see if the storage solution will meet my object storage needs? Well, why not just use Amazon’s own S3APIs and AWS Command Line Interface (CLI)?

AWS CLI is a unified tool developed to help manage AWS services. I believe this is the best way to test out any solution that says they are an S3 compatible storage such as Cloudian HyperStore. So let’s hop on to it and get started. The following shows the steps on how to install and use AWS CLI with Cloudian HyperStore on your Linux server.

Prerequisite:

You will need to install PIP to simplify your AWS CLI installation, you can copy the following python script to your Linux server and it will help you install pip and awscli. The script is provided as-is but feel free to copy, modify and improve it to your liking.

import urllib

import os

PIP=’get-pip.py’

urllib.urlretrieve (“https://bootstrap.pypa.io/get-pip.py”, PIP)

os.system(“python get-pip.py”)

os.system(“pwd”)

os.system(“pip install awscli”)

Process:

  1. Download the following dc_getpip.py to your Linux server. The script has been tested on RHEL and CentOS. The Cloudian S3 region used in this example is s3-region.addomain.local
  2. Run python dc_getpip.py. This script will download pip and install AWS CLI for you.
  3. When the AWS CLI is successfully installed, continue with configuring AWS CLI with Cloudian HyperStore.
  4. Execute aws configure and provide the Cloudian credential along with the Cloudian S3 region information. For example:
  5. cd ~/./.aws because the config and the credential files for aws is located in your user directory. In this example, this is the root user directory.AWS CLI root user directory
  6. There are 2 files in .aws directory:
    1. config
    2. credentials
  7. Update the config file with the Cloudian region information. Include [cloudian] in your update.AWS CLI Cloudian regional information
  8. Update the credentials files with the Cloudian information, include [cloudian] in your update.AWS CLI credentials file
  9. Run the following aws command to validate connectivity to your Cloudian HyperStore cluster. Using s3 ls will list the buckets of the tenant that was configured.
    1. aws –profile=cloudian –endpoint-url=http://s3-region1.addomain.local s3 lsAWS CLI validate connectivity to Cloudian HyperStore cluster
    2. Replace s3-region1.addomain.local with your Cloudian region.
    3. You can use aws –profile=cloudian –endpoint-url=http://s3-region1.addomain.local s3 cp file s3://bucket to test upload to your s3 bucket.
  10. Your AWS CLI is successfully configured with Cloudian HyperStore S3.

 

If you are curious to learn more about S3, download Cloudian HyperStore’s community edition and validate the solution for yourself.

Learn more about hybrid cloud management here.

IBM Spectrum Protect with Amazon S3 Cloud Storage

IBM Spectrum Protect (formerly IBM Tivoli Storage Manager) solution provides the following benefits:

  • Supports software-defined storage environments
  • Supports cloud data protection
  • Easily integrates with VMware and Hyper-V
  • Enables data protection by minimizing data loss with frequent snapshots, replication, and DR management
  • Reduce the cost of data protection with built-in efficiencies such as source-side and target-side deduplication

IBM Spectrum Protect has also enhanced its offerings by providing support for Amazon S3 cloud storage (version 7.1.6 and later) and IBM Spectrum Protect version 7.1.6 was just released on June 17th, 2016. I was actually a little nervous and excited at the same time. Why? Because Cloudian HyperStore has a S3 guarantee. What better way to validate that guarantee than by trying a plug-and-play with a solution that has just implemented support for Amazon S3?

Overview of IBM Spectrum Protect with Amazon S3 cloud storage

And the verdict? Cloudian HyperStore configured as “Cloud type: Amazon S3” works right off the bat with IBM Spectrum Protect. You can choose to add a cloud storage pool from the V7.1.6 Operations Center UI or use the Command Builder. The choice is yours.

We’ll look at both the V7.1.6 Operations Center UI and the Command Builder to add our off-premise cloud storage.

NOTE: Cloudian HyperStore can be deployed as your on-premise S3 cloud storage but it has to be identified as an Amazon S3 off-premise cloud storage and you have to use a signed SSL certificate.

Here’s how you can add an Amazon S3 cloud storage or a Cloudian HyperStore S3 cloud storage into your IBM Spectrum Protect storage pool:

From the V7.1.6 Operations Center UI

 

From the V7.1.6 Operations Center console, select “+Storage Pool”.

Adding 'Storage Pool' to the IBM Spectrum Protect V7.1.6 Operations Center console

In the “Add Storage Pool:Identity” pop-up window, provide the name of your cloud storage and the description. In the next step of the “Add Storage Pool:Type”, select “Container-based storage:Off-premises cloud”.

IBM Spectrum Protect cloud storage description

Click on “Next” to continue. The next step in the “Add Storage Pool:Credentials” page is where it gets exciting. This is where we provide the information for:

  • Cloud type: Amazon S3 (Amazon S3 cloud type is also used to identify a Cloudian HyperStore S3)
  • User Name: YourS3AccessKey
  • Password: YourS3SecretKey
  • Region: Specify your Amazon S3 region (for Cloudian HyperStore S3, select “Other”)
  • URL: If you had selected an Amazon S3 region, this will dynamically update to the Amazon region’s URL. If you are using a Cloudian HyperStore S3 cloud storage, input the S3 Endpoint Access (HTTPS).

Complete the process by clicking on “Add Storage Pool”.

IBM Spectrum Protect

NOTE: Be aware that there is currently no validation performed to verify your entries when you click on “Add Storage Pool”. Your S3 cloud storage pool will be created. I believe the IBM Spectrum Protect group is addressing this with a validation process for the creation of a S3 cloud storage pool. I hope the step-by-step process that I have provided will help minimize errors with your Amazon S3 cloud storage pool setup.

From the V7.1.6 Operations Center Command Builder

 

From the V7.1.6 Operations Center Command Builder, you can use the following define stgpool command and you are done adding your off-premise S3 cloud storage pool:

  • define stgpool YourCloudName stgtype=cloud pooltype=primary cloudtype=s3 cloudurl=https://s3.cloudianstorage.com:443 access=readwrite encrypt=yes identity=YourS3AccessKey password=YourS3SecretKey description=”Cloudian”

NOTE: You can review the server instance dsmffdc log if there’s errors. It is located in the server instance directory. There’s also a probability that the signed SSL certificate might not be correct.

For example:

06-20-2016 11:58:26.150][ FFDC_GENERAL_SERVER_ERROR ]: (sdcloud.c:3145) com.tivoli.dsm.cloud.api.ProviderS3 handleException com.amazonaws.AmazonClientException Unable to execute HTTP request: com.ibm.jsse2.util.h: PKIX path building failed: java.security.cert.CertPathBuilderException: unable to find valid certification path to requested target
[06-20-2016 11:58:26.150][ FFDC_GENERAL_SERVER_ERROR ]: (sdcntr.c:8166) Error 2903 creating container ibmsp.a79378e1333211e6984b000c2967bf98/1-a79378e1333211e6984b000c2967bf98
[06-20-2016 11:58:26.150][ FFDC_GENERAL_SERVER_ERROR ]: (sdio.c:1956) Did not get cloud container. rc = 2903

 

Importing A Signed SSL Certificate

 

You can use the IBM Spectrum Protect keytool –import command to import the signed SSL certificate. However, before you perform the keytool import process, make a copy of the original Java cacerts.

The Java cacerts is located in IBM_Spectrum_Protect_Install_Path > TSM > jre > security directory.

You can run the command from IBM_Spectrum_Protect_Install_Path > TSM > jre > bin directory.
For example, on Windows:

    • ./keytool –import ../lib/security/cacerts –alias Cloudian –file c:/locationofmysignedsslcert/admin.crt

 

Enter the keystore password when prompted. If you haven’t updated your keystore password, the default is changeit and you should change it for production environments. When you are prompted to “Trust this certificate?”, input “yes”.

NOTE: Keep track of the “Valid from: xxxxxx” of your signed SSL certificate, you will have to import a new certificate when the current one expires.

By the way, if you encounter error “ANR3704E sdcloud.c(1636): Unable to load the jvm for the cloud storage pool on Windows 2012R2”, update the PATH environment variable on the Spectrum Protect Server:
IBM_Spectrum_Install_Path\Tivoli\TSM\jre\bin\j9vm and also set the JVM_LIB to jvm.dll.

Here’s what your Amazon S3 cloud storage type looks like from IBM Spectrum Protect V7.1.6 Operations Center console:

Operations Center console final result after adding Amazon S3 cloud storage to IBM Spectrum Protect V7.1.6

And you’re off! If you encounter any issues during this process, feel free to reach out to our support team.

You can also learn more by downloading our solution brief.

How-To: S3 Your Data Center

As the Storage Administrator or a Data Protection Specialist in your data center, you are likely looking for some alternative storage solution to help store all your big data growth needs. And with all that’s been reported by Amazon (stellar growth, strong quarterly earnings report), I am pretty sure their Simple Storage Service (S3) is on your radar. S3 is a secure, highly durable and highly scalable cloud storage solution that is also very robust. Here’s an API view of what you can do with S3:

S3 API view

As a user or developer, you can securely manage and access your bucket and your data, anytime and anywhere in the world where you have web access. As a storage administrator, you can easily manage and provision storage to any group and any user on always-on, highly scalable cloud storage. So if you are convinced that you want to explore S3 as a cloud storage solution, Cloudian HyperStore should be on your radar as well. I believe a solution that is easy to deploy and use helps accelerates the adoption of the technology. Here’s what you will need to deploy your own cloud storage solution:

  • Cloudian’s HyperStore Software – Free Community Edition
  • Recommended minimum hardware configuration
    • Intel-compatible hardware
    • Processor: 1 CPU, 8 cores, 2.4GHz
    • Memory: 32GB
    • Disk: 12 x 2TB HDD, 2 x 250GB HDD (12 drives for data, 2 drives for OS/Metadata)
    • RAID: RAID-1 recommended for the OS/Metadata, JBOD for the Data Drives
    • Network: 1x1GbE Port


You can install a single Cloudian HyperStore node for non-production purposes, but it is best practice to deploy a minimum 3-node HyperStore cluster so that you can use logical storage policies (replication and erasure coding) to ensure your S3 cloud storage is highly available in your production cluster. It is also recommended to use physical servers for production environments.

Here are the steps to set up a 3-node Cloudian HyperStore S3 Cluster:

  1. Use the Cloudian HyperStore Community Edition ISO for OS installation on all 3 nodes. This will install CentOS 6.7 on your new servers.
  2. Log on to your servers
    1. The default root password is password (Update your root access for production environments)
  3. Under /root, there are 2 Cloudian directories:
    1. CloudianTools
      1. configure_appliance.sh allows you to perform the following tasks:
        1. Change the default root password
        2. Change time zone
        3. Configure network
        4. Format and mount available disks for Cloudian S3 data storage
          1. Available disks that were automatically formatted and mounted during the ISO install for S3 storage will look similar to the following /cloudian1 mount:
            Format and mount available disks for Cloudian S3 data storage
    2. CloudianPackages
      1. Run ./CloudianHyperStore-6.0.1.2.bin cloudian_xxxxxxxxxxxx.lic to extract the package content from one of your nodes. This will be the Puppet master node.
        S3 Puppet master mode
      2. Copy sample-survey.csv survey.csv
        sample-survey.csv
      3. Edit the survey.csv file
        Edit survey.csv
        In the survey.csv file, specify the region, the node name(s), IP address(s), DC, and RAC of your Cloudian HyperStore S3 Cluster.

        NOTE: You can specify an additional NIC on your x86 servers for internal cluster communication.

      4. Run ./cloudianInstall.sh and select “Install Cloudian HyperStore”. When prompted, input the survey.csv file name. Continue with the setup.
        NOTE: If deploying in a non-production environment, it is possible that your servers (virtual/physical) may not have the minimum resources or a DNS server. You can run your install with ./cloudianInstall.sh dnsmasq force. Cloudian HyperStore includes an open source domain resolution utility to resolve all HyperStore service endpoints.
      5. v. In the following screenshot, the information that we had provided in the survey.csv file is used in the Cloudian HyperStore cluster configuration. In this non-production setup, I am also using a DNS server for domain name resolution with my virtual environment.Cloudian HyperStore cluster configuration
      6. Your Cloudian HyperStore S3 Cloud Storage is now up and running.
        Cloudian HyperStore S3 cloud storage
      7. Access your Cloudian Management Console. The default System Admin group user ID is admin and the default password is public.
        Cloudian Management Console
      8. Complete the Storage Policies, Group, and SMTP settings.
        Cloudian HyperStore - near final

Congratulations! You have successfully deployed a 3-node Cloudian HyperStore S3 Cluster.

Hypervisor Agnostic Cloud Storage for VDI Home Directories

Virtual Desktop Infrastructure (VDI) is, simply put, the process of running virtualized desktops for users within an organization using server-based computing. You have the option of running persistent or non-persistent virtual desktops.

If you are interested in knowing more about these types of deployments, you can find out more by using your favorite search engine. If you are like me and you just want to click on a link and have the information pop up on your screen, here is a blog post that explains VDI in more detail.

For example, a VDI deployment with Hyper-V can be explained with the diagram below. The RDP client logs in to the web browser server, then the RD Connection Broker server lists and orchestrates the virtual machines. The AD server authenticates access. The RD Session Host server redirects the RDP client to the right virtual machine and the RD Gateway server publishes and makes the VM available to the authorized user.

VDI deployment with Hyper-VVDI deployment with Hyper-V

Whether you are running persistent and/or non-persistent virtual desktop solutions, your organization will still require storage for your virtual desktop user home directories, etc. You can deploy additional LUN storage. However, with traditional LUN storage, you are likely to run through the following process:

    1. You, as the VDI administrator, will define the storage requirement for each user.
    2. You’ll submit the storage requirement request to your storage administrator.
    3. The storage administrator creates the LUN storage base on your requirements.
    4. You create your master image/golden image for your virtual desktops.
    5. You’ll provision the virtual desktops to your user when you are ready.
    6. You’ll run into late night support calls when your user runs out of disk space to store their files/presentations/media in their user home directories.
    7. You make a call to your storage administrator to provision more storage @ 1am in the morning.
    8. You and your storage administrator work through the wee hours on the support calls.
    9. Repeat steps 6 through step 8…monthly, weekly, or even daily!

 

A Better Solution for VDI Users with Cloud Storage

 

 

Did the previous process flow sound familiar? What if you and your storage administrator could host your own hypervisor agnostic on-premise cloud storage solution within your own firewalls for your virtual desktop users’ home directories? What if you, as the storage administrator, can easily increase the bucket capacity for each user when a support call comes in with a few clicks? All of this is possible because software-defined cloud storage solutions such as Cloudian are designed from the ground up to be:

    • Simple
    • Highly scalable
    • Always-on with secure and encrypted access

 

VDI deployment with cloud storageHyper-V VDI deployment with cloud storage for VDI users’ home directories

Instead of fielding late-night support calls, easily provision S3 buckets for each virtual desktop user and provide secure S3 portal access without missing a beat. With some on-premise cloud storage solutions, you can:

    • Use the inherent multi-tenancy feature to create and deploy storage for all VDI user home directories.
    • Use QoS to throttle each user’s PUTs and GETs.
    • Monitor per user usage and easily review reports for chargeback purposes.
    • Use replication or erasure coding on a per storage policy to ensure each group has the right data protection benefits.

 

To see how easily a virtual desktop user can use a secure cloud storage system as a home directory, watch the following video:

A Virtual Desktop user using Cloudian HyperStore Cloud Storage as his home directoryPer bucket and per user granularity is possible. It is because some cloud storage solutions are fully Amazon S3 compliant. Think about it – rather than creating additional LUN storage for your VDI users’ home directory requirements, I can simply create a master image that has the secure on-premise cloud portal for every group and every virtual desktop user defined in the web browser of the desktop master/golden image.

From that one golden image, I can deploy hundreds or thousands of virtual desktops that have a secure on-premise cloud storage solution for all my virtual desktop users. My virtual desktop users can easily access and use my deployed cloud storage as their home directories from any web browser. Best of all, many cloud storage solutions also support NFS/SMB/FTP via its native file access integration. This means you get the benefit of industry standard file protocols access without any third-party gateways!

With cloud storage solutions for user home directories and for file sharing purposes, we get the following benefits for virtual desktop deployments:

    • Simplified virtual desktop deployment with a hypervisor agnostic cloud storage solution. Minimize your late night “virtual desktop user is out of storage” support calls.
    • Scale-out and highly available home directories for virtual desktop user storage. Each storage bucket capacity is tunable for each virtual desktop user. There is no single point of failure with cloud storage.
    • Secure in-flight data and data-at-rest with AES-256 encryption.
    • Simplified and flexible data protection. Virtual desktop users can manage their own data protection and retention requirements.
    • Predictive analytics for storage planning. Use built-in analytics to manage your storage growth requirements.
    • Fully S3 compliant storage. This means you can support hundreds of S3 compatible applications using your very own on-premise cloud storage solution.
    • Software-defined. Use any x86 commodity server to deploy your own hybrid, private, and public cloud storage solution and minimize your operating cost.
    • Manage access and performance with QoS throttle. Easily manage user PUTs and GETs by using QoS throttles at the group or user level.

 

With the availability of secure cloud storage solutions for on-premise deployments, it’s a no-brainer to look into simplifying our lives so that we can get away from the repetitive user storage support calls in the middle of the night. Say goodbye to conversations such as:


“Help! I can’t save my executive briefing presentation and videos on my desktop. I need to securely share the presentation and videos with our Tokyo office. I am getting an error on my desktop pointing out that my e:\ drive and my user home directory is full. And NO, I cannot delete any of the existing files to make space for these new files and videos because I need all of it to be always accessible to me.”

To find out more about the advantages of cloud storage solutions, visit Cloudian.

Cheers,

Dominic

Data Availability & Data Protection for the IoT World

New York cityscape

New York, “The City That Never Sleeps”. A very fitting moniker for a city that is full of energy and excitement. Servers located in data centers all around the world are constantly crunching numbers and generating analytics in every financial institution in New York. Why are some of these servers located worldwide? Well, for a variety of reasons, but in my humble opinion, it is to ensure that data is always on and always available. After all, we are talking about billions of dollars in capital electronically managed by the New York Stock Exchange alone.

By 2020, it is predicted that there will be at least 20+ billion internet connected devices. As your business grows, so will the amount of data and storage that you will need. We’ll obviously need solutions to protect our data on-premise or in the cloud. A company that can make sure customers data is always on, secure, highly available, and also protected, rules the IoT WORLD.

modern storage requirementsBut in order to serve and protect your data for the always on, always available IoT world, what requirements should we take into account before deploying any data protection or storage solution? If you are a data protection geek, you’ll most likely see some of your requirements being listed on the right. If you are a data protection solutions provider, you guys definitely rock! Data protection solutions such as Commvault, NetBackup, Rubrik, Veeam, etc. are likely the solutions you have in-house to protect your corporate data centers and your mobile devices. These are software-defined and they are designed to be highly available for on-premise or in-the-cloud data protection.

What about storage? What would you consider? I am sure there are many well-known storage providers you can easily name. But with the new kids on the block disrupting the storage market, would lowering your operating costs ($0.005/GB per month) and meeting the above-listed requirements pique your interest?

Amazon S3 and Cloudian
Cloudian is a software-defined storage company. The solution is fully S3 compliant, which means that if you are familiar with Amazon S3, you’ll love the features that comes with this solution. If you are not, as a data protection geek with more than 15 years of experience, I invite you to give Cloudian HyperStore free trial a shot. The features and capabilities of Cloudian HyperStore as a scale-out storage solution with true multi-tenancy is pretty cool in my books. Imagine being able to deploy and grow storage as you need it for your corporate user home directories, backups, archiving, and even object storage for virtualization solutions (i.e. Red Hat OpenStack). The use cases for scale-out storage solutions are vast. There is no more hardware vendor lock-in as you can easily select between the options of a Cloudian HyperStore appliance or commodity servers to roll-your-own scale-out storage with Cloudian HyperStore software.

Imagine that you, as a storage administrator, can easily provide Storage as a Service (STaaS) to all your users. Take a look at the image below. The granular object level management that is available on a per user basis is pretty sweet. I can provide access to my files/objects with read and/or write permissions, with object level ACL and share the object via a public URL access.
Cloudian object level management

To top it all off, I can also limit the maximum number of downloads of that specific object that I want to share. As a service provider, you can also use the analytics inherent in the solution to implement chargeback to your customers on every account that you manage using Cloudian HyperStore smart storage solution.

Best of all, if you decide that you want to move your data to Amazon, use Cloudian Hyperstore’s built-in auto-tiering feature. Dynamically move your data to Amazon S3 if you choose to do so. You don’t have to take my word for it. Cloudian will provide you with a 45-day free trial. Try it out today.

S3 API & Extensions for Enterprise Object Storage

Amazon’s S3 API is the de-facto standard for object storage APIs. Having multiple service providers, software providers, and applications standardize on S3 has made it easier to interchange between them and rapidly stand up new uses for object storage. But there are different grades of S3 compatibility. Some software and solutions provide only the basic CRUD (create, remove, update, delete) functions. At the other end is Cloudian’s Hyperstore, committed to providing the highest fidelity S3 compatibility backed by a guarantee.

The S3 API is an HTTP/S REST API where all operations are via HTTP PUT, POST, GET, DELETE, and HEAD requests. Each object is stored in a bucket. Beyond the basic object CRUD operations provided by S3, there are many advanced APIs like versioning, multi-part upload, access control list, and location constraint. There are multiple options for encryption including (1) server-side encryption where the server manages encyrption keys, (2) server-side encyption with customer keys, and (3) client-side encryption where the data is encrypted/decrypted at the client side. Though no single S3 user is likely to use all of the advanced APIs, the union of APIs used by different users quickly covers them all. The table below highlights some advanced object storage APIs supported by S3:

S3 Feature Azure Google Cloud OpenStack Swift
Object versioning No Yes Yes
Object ACL No Yes No
Bucket Lifecycle Expiry No Yes Yes
Multi-object delete No Yes Yes
Server-side encryption No Yes Yes
Server-side encryption with customer keys No No No
Cross-region replication Yes No Yes
Website No No No
Bucket logging No No No
POST object No No No

Table 1 – Comparison of some S3 advanced object storage APIs[1]

S3 API compatibility is a prerequisite, but not sufficient to provide object storage for enterprises. There are 4 additional areas that Cloudian has added to make S3 object storage enterprise-ready.

 

  1. Software or Appliance, not a service.The software-only package includes a Puppet-based installer with a wizard-style interface. It runs on commodity software (CentOS/RedHat) and commodity hardware. The appliances come in a few fixed models ranging from 1U (24TB) to the FL3000 series of PB-scale in 8U form.
  1. APIs for all functions
    • Configuration
    • Multi-Tenancy: User/Tenant provisioning
    • Quality of Service (QoS)
    • Reporting
    • S3 Extensions: Compression, Metadata APIs, Per-bucket Protection Policies.

    Highlighting the per-bucket protection policies feature, each bucket can have its own protection policy. For example, a“UK3US2” policy can be defined as UK DC with 3 replicas and US DC with 2 replicas. Another example is a “ECk6m2” policy as DC1 with Erasure Coding with 6 data and 2 coding fragments. As buckets are created they can be assigned a policy.

Bucket
Figure 1 – Per-bucket protection policies example

  1. O&M tools to install, monitor, and manage.In addition to the installer, a single pane web-based Cloudian Management Console (CMC) does system administration from the perspective of the system operator, a tenant/group administrator, and a regular user. It’s used to provision groups and users, view reports, manage the cluster, and monitor the cluster.

Cloudian Management Console

Figure 2 – CMC dashboard

  1. Integration with Other Products
    • NFS/CIFS file interface
    • OpenStack, CloudPlatform
    • Tiering to any S3 system (public or private).
    • Active Directory, LDAP

The opportunity and use case for enterprises and object storage has never been more compelling. Amazon S3 API compatibility ensures full portability of already working applications. Using Cloudian’s HyperStore platform instead of AWS, enterprise data can be brought on-premise for better data security and manageability at lower cost. For STaaS providers, S3 API compatibility, backed by a full guarantee, provides the same benefits of a fully controlled storage platform, and opens up a large range of compatible applications. Beyond the S3 API, Cloudian is committed to providing all operations by API and has added APIs to make the platform enterprise-ready, including multi-tenancy.

If you would like a technical overview, you can check out this webinar I recently presented, “S3 Technical Deep Dive” and make sure to check out more information on our S3 Guarantee…we’ll run all your S3 Apps anytime and anywhere – Guaranteed!

– Gary

—
[1] References:
http://docs.openstack.org/developer/swift/#object-storage-v1-rest-api-documentation
https://cloud.google.com/storage/docs/xml-api-overview
https://msdn.microsoft.com/en-us/library/azure/dd135733.aspx

Cloudian HyperStore Integration with Symantec NetBackup

Starting with Symantec NetBackup 7.7, administrators will find an exciting new feature for cloud storage backup: Cloudian HyperStore®. The NetBackup Cloud Storage Connector enables the NetBackup software to back up data to and from Cloudian HyperStore straight out of the box without additional software installations or plugins. HyperStore is an option in the “Cloud Storage Server Configuration Wizard”. Users can simply add their S3 account information such as endpoint, access key, and secret key to begin the process of backing up their data to Cloudian HyperStore storage.

cloudian hyperstore 4000

Cloudian HyperStore and Symantec NetBackup together deliver the following benefits:

  • Enterprise-level backup
  • Complete integrated data center solution: computing, networking, and storage
  • Reduced total cost of ownership (TCO) that continues to improve as the solution scales out
  • Operational efficiency
  • Agility and scalability with the scale-out architectures of Cloudian HyperStore
  • Complete Amazon Simple Storage Service (S3) API–compatible geographically federated object storage platform
  • Enterprise-class features: multi-tenancy, quality of service (QoS), and dynamic data placement in a completely software-defined package
  • Policy-based tiering between on-premises hybrid cloud storage platform and any S3 API–compliant private or public cloud
  • Investment protection: mix and match different generations and densities of computing platforms to build your storage environment; more than 400 application vendors support S3

The seamless integration allows IT Departments to manage cloud storage for backup and recovery as easily as on-premise storage, but with lower costs. Finally, this integrated solution helps deliver an automated and policy-based backup and recovery solution. Organizations can also leverage the cloud as a new storage tier or as a secondary off-site location for disaster recovery.

For more information, please see the Symantec NetBackup and Cloudian HyperStore Solution Brief.

 

Categories