Blob download to provide InputStream #928

Activity

1. product-auto-label

   added

   api: storageIssues related to the googleapis/java-storage API.Issues related to the googleapis/java-storage API.

   on Jul 27, 2021
2. BenWhitehead

   added

   type: feature request‘Nice-to-have’ improvement, new feature or different behavior or design.‘Nice-to-have’ improvement, new feature or different behavior or design.

   on Jul 27, 2021
3. BenWhitehead

   self-assigned this

   on Jul 27, 2021
4. BenWhitehead commented on Jul 27, 2021

   

   BenWhitehead

   on Jul 27, 2021

   Collaborator

   More actions

   Hi @nanda-kumar-office1, Can you expand on what types of things you are hoping to accomplish via using an InputStream directly?

   Currently downloadTo transparently takes care of retrying a request that may have been interrupted, decompressing if there was any transport compression as well as moving the bytes to the provided OutputStream with minimal copying.
5. nanda-kumar-office1 commented on Jul 28, 2021

   

   nanda-kumar-office1

   on Jul 28, 2021

   Author

   More actions

   Thank you Ben for the response.
   downloadTo is perfect for Desktop kind of app, but on server side programming, we may have to set few headers before streaming the file content to user. Also, it may not be a good practice to pass servlet response handler all the way to the backend layer.
   Feel free if you have more question, I can get specific examples based on api that I had been working across other vendor apis too.
6. BenWhitehead commented on Oct 5, 2021

   

   BenWhitehead

   on Oct 5, 2021

   Collaborator

   More actions

   Unfortunately we haven't been able to get around to implementing this yet as we've ran into some potential issues needing to manage the streams that are passed out from the underlying rpc systems.

   For the interim, your best bet is to do something like the following where you leverage our resumable download reader:

       OutputStream outputStream = request.getOutputStream();
       Blob b = storage.get(BlobId.of("bucket", "blob"));
       try (InputStream inputStream = java.nio.channels.Channels.newInputStream(b.reader())) {
         com.google.common.io.ByteStreams.copy(inputStream, outputStream);
       }
7. nanda-kumar-office1 commented on Oct 6, 2021

   

   nanda-kumar-office1

   on Oct 6, 2021

   Author

   More actions

   Thank you Ben,
   During the course, I used the below workaround code that worked for us.

   BlobInfo blobInfo = BlobInfo.newBuilder(BlobId.of(bucketName, filePath)).build();
   URL url = storage.signUrl(blobInfo,5,TimeUnit.MINUTES,SignUrlOption.withV4Signature());

   InputSteam in = url.openStream()
8. BenWhitehead commented on Oct 6, 2021

   

   BenWhitehead

   on Oct 6, 2021

   Collaborator

   More actions

   Glad to hear you were able to figure out something that worked for you. One thing of note however, if you use the reader adapter I mentioned above you will receive our automatic retry for downloading of individual chunks. So if you have medium~ish size blobs (more than say 5mb), this might help in the reliability perception for your app.
9. nanda-kumar-office1 commented on Oct 6, 2021

   

   nanda-kumar-office1

   on Oct 6, 2021

   Author

   More actions

   Sure Ben, let me try it and update you.
   Thanks,
10. BenWhitehead commented on Feb 1, 2023

    

    BenWhitehead

    on Feb 1, 2023

    Collaborator

    More actions

    Apologies on the very late reply to this issue. Recently we've been able to clean up a large portion of the internal middleware of the client for the read path of objects. This allows storage.reader (and blob.reader) to provide bandwidth competitive with downloadTo[1] and providing transparent automatic retries - something downloadTo can't do. As such, the recommended path moving forward is to leverage the code shared in #928 (comment).

    [1] #1799
11. BenWhitehead

    closed this as completedon Feb 1, 2023