CF Object Storage

objectstorage2

A Cloud Foundry CLI Plugin for Working With Object Storage

A new plugin, CF Object Storage, is now available.  This plugin provides users with the ability to work with Object Storage directly from the Cloud Foundry CLI. Users can take a look at the containers and objects currently in object storage and access, create and modify them. OpenStack Object Storage has many great features, but they are not always the easiest to use especially when you want to upload very large files.   So the IBM jStart team has provided this CF CLI plugin to make uploading files and managing object storage contents easier.

In addition to standard objects, Object Storage has two very powerful types of aggregate files, Static Large Objects (SLOs) and Dynamic Large Objects (DLOs). You can find a ton of great information on SLOs and DLOs in this post. Both types of large objects use a manifest file to allow users to create large files out of small chunks. This is especially useful when trying to upload files greater than OpenStack’s maximum file size, 5GB. Currently, there is not much tooling around the use of large objects. Previously, users needed to upload each individual chunk via an HTTP request which can be quite tedious.

This plugin, CF Object Storage (CFOS), also includes commands to make working with large objects easier than ever. This new Cloud Foundry CLI plugin works on IBM Bluemix as well as on other Cloud Foundry based platforms.  CFOS provides Cloud Foundry CLI users with sixteen new commands for interacting with OpenStack Object Storage. Specifics on these commands can be found here. Now it is possible to fetch authentication credentials as well as upload DLOs, SLOs and standard objects from the CLI. This gets rid of the hassle of HTTP requests as well as the need to go through the long OpenStack authentication process. CFOS takes care of all of that for you.

Why Use Large Objects?

Great question, I’m glad you asked! In addition to uploading files that exceed the size limit, large objects have many other uses and benefits. But let’s start there. The main reason users will find themselves creating SLOs and DLOs is because their files are too large to upload directly. Both have the ability to break files into manageable chunks, the main difference being that DLOs provide the ability to add and remove chunks as desired. SLOs, on the other hand, must be provided with a user-generated manifest that defines exactly what chunks to include.

This strict manifest gives SLOs the advantage of being able to store their chunks across multiple containers. This allows us to exploit parallelism to upload and download large files faster. Because DLO manifests are actually zero-byte files that only define a container and filename prefix, all of their chunks must be located in the same container. This means users can upload chunks after the manifest is generated, but lose the ability to upload to or download from multiple containers at the same time.

How about a concrete example? We could use manifests to create sample subsets of large files without the need for redundant data uploads. This is quite useful when your files are large data sets that you may want to run some preliminary or test analysis on. Multiple manifests can take different sets of chunks (perhaps representing rows of a data set) and make different sample sets of varying sizes. Instead of having a bunch of large subsets of the same data, we just have a couple small manifest files.

You may also be wondering when to use a large object versus a regular object. You should only use a DLO when you plan on adding more data to the object after its initial upload. Generally, SLOs should only be used when uploading a file larger than 5GB, the maximum object size. There may be other more specialized use cases that could make SLOs useful with files of smaller sizes.

Installation and Usage

The CF Object Storage GitHub page contains detailed instructions for installation and usage of this plugin. The plugin is not currently supported in one of Cloud Foundry’s official repos, so it must be installed from binary. After downloading the binary from GitHub, running the following command from the binary’s directory will install CFLO.

Now you can use  cf os help SUBCOMMAND with any of the subcommands detailed on the GitHub page to view the usage details. Here are a few examples:

A sample run of  containers  may look something like this.

If you choose to create an SLO in your object storage, it may look something like this:

Note: This particular invocation of put-large-object  does not use the -s  flag, so the plugin uses the default chunk size, 1GB.

If you have a small file and would like to create a regular object, it could look like this:

Again, you can find CF Object Storage here. Feedback is much appreciated!

Ryan Soley

Ryan began working with the jStart team as a co-op in May 2016. He is currently a senior at Penn State University pursuing his Bachelor’s degree in Computer Science.

Ryan Soley
Ryan Soley