Allow writing to S3 paths

[hunterowens]

It would be really great if to_(filetype) supported writing to S3.

Here is an example upload to s3 function that takes in a local file and places it on a s3 bucket.

def upload_to_s3(local_file_path, file_name, bucket_name, s3_directory):
    """
    Returns
    ----------
    Uploads local file to appropriate s3 key, and prints status 

    Parameters
    ----------
    local_file_path : str
        ex. 'my/local/path'
    file_name : str
        ex. 'cleaned_data.csv'
    bucket_name : str
        ex. 'dsapp-edu-data'
    s3_directory : str
        ex. 'NC-Cabarrus/cleaned_data'
    """

    def percent_cb(complete, total):
        """
        Helper function that prints progress
        """
        sys.stdout.write('.')
        sys.stdout.flush()

    conn = boto.connect_s3()
    bucket = conn.get_bucket(bucket_name)
    full_key_name = os.path.join(s3_directory, file_name)
    k = bucket.new_key(full_key_name)
    full_filepath = os.path.join(local_file_path, file_name)
    k.set_contents_from_filename(full_filepath, cb=percent_cb, num_cb=10)

    return None 

[TomAugspurger]

I'll try to get to this by 0.16

[andrewgiessel]

any PRs open on this? I thought of this too and think it'd be great. I can take a shot at it if not.

[TomAugspurger]

Go for it!

[TomAugspurger]

https://github.com/wrobstory/pgshift/blob/master/pgshift/pgshift.py#L99 could be a helpful template

[swarajban]

I also wanted the ability to write a DF as a CSV to S3 (acronym overload...), and wrote the following snippet:

# Write dataframe to buffer
csv_buffer = StringIO()
df.to_csv(csv_buffer, index=False)

# Upload CSV to S3
s3_key = 'test.csv'
s3_resource = aws_session.resource('s3')
s3_resource.Object(s3_bucket, s3_key).put(Body=csv_buffer.getvalue())

Hope this helps anyone trying to do the same thing. IMO, this features shouldn't be added to pandas. I think this snippet (or a better version) should simply be documented

Edit: This is using Python 3.5 & boto3. I'm sure a similar snippet will work for 2.7 or the old boto

[maximveksler]

Any hope of supporting writing to S3 for the new release? Now that parquet is supported this becomes double interesting.

[jreback]

@maximveksler this is not very hard to do as we already have a dep for S3 interactions with https://pypi.python.org/pypi/s3fs, and pyarrow also supports s3. want to do a pull request?

[maximveksler]

@jreback sure, please point me into some relevant locations and I'll PR gladly.

Would appreciate focus about what from s3fs and pyarrow I should be looking into for more details.