gsutil 3.0 release

There is a new gsutil release.

GsUtil is a python application to access Google Cloud Storage from the command line. It’s very useful to integrate it into your python projects and upload content to Google Cloud Storage.

I attach official post on Google Cloud Storage discuss.


A new major release of gsutil (v3.0) is available by zipfile or tarball.

This release provides significant functionality and performance enhancements, a new hierarchical file tree abstraction layer, and numerous bug fixes. However, it also changes the behavior of the ls command and the * wildcard, in ways that may require changes to scripts that depend on ls and * behavior. Please see the Release notes for full details; below are highlights of the new features and potentially required script changes.

New Features:

  • Built-in help for all commands and many additional topics (try “gsutil help”).
  • Support for copying data to/from bucket sub-directories (see “gsutil help cp”).
  • Support for renaming bucket sub-directories (see “gsutil help mv”).
  • Support for listing individual bucket sub-directories and for listing directories recursively (see “gsutil help ls”).
  • Support for Cross-Origin Resource Sharing (CORS) configuration (see “gsutil help cors”).
  • Multi-threading support for the setacl command (see “gsutil help setacl”).
  • Support for using the UNIX “file” command to do content type recognition as an alternative to filename extensions (see “gsutil help metadata”).
  • The gsutil update command is no longer beta/experimental.

As part of the bucket sub-directory support we changed the * wildcard to match only up to directory boundaries, and introduced the new ** wildcard to span directories the way * used to. We made this change both to be more consistent with how wildcards work in command interpreters (like bash), and to enable a variety of use cases for distributing large transfers across many machines. For example you can run the following commands on 3 machines:
gsutil cp -R gs://my_bucket/result_set_[0-3]* dir
gsutil cp -R gs://my_bucket/result_set_[4-6]* dir
gsutil cp -R gs://my_bucket/result_set_[7-9]* dir
and end up with all of the result_set_* directories nested under dir.

Script Changes You May Need to Make To Use gsutil 3.0:
If your script depends on listing the entire (flat) contents of a bucket using something like:
gsutil ls gs://my_bucket
you’ll need to change to use:
gsutil ls gs://my_bucket/**

If your script uses the * wildcard to name objects spanning directories, such as:
gsutil cp gs://mybucket/*.txt ./dir
(where you want to match objects several directories down from the top-level bucket), you’ll need to change to use:
gsutil cp gs://mybucket/**.txt ./dir

Mike Schwartz and the Google Cloud Storage Team


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s