2

I am getting this error only on my local workstation and prod server. In Dev and Cert it is working fine.

local workstation - 20 GB memory, Win 7 64 bit, IIS Express, VS 2013
dev, cert & prod - 8 GB memory , 2008 R2 64 Bit, IIS 7.5

I have a web api (.net 4.0) which takes the incoming request body and uploads it to a storage server. configured web api as per this website.

I have these in my web.config

<system.webServer>
  <security>
    <requestFiltering>
      <requestLimits maxAllowedContentLength="2147483648" />
    </requestFiltering>
  </security>
</system.webServer>

<system.web>
  <httpRuntime maxRequestLength="2097152" />
</system.web>

I also have an implementation of IHostBufferPolicySelector which returns false for PUT & POST requests. so the request to this web api for PUt & POST are not buffered.

for any files < ~350 MB it is working fine. but web api is throwing out of memory exceptions when file size >= ~ 400 MB and this is happening only on Local workstation and Prod server.

Web Api controller calls below code to stream the request to the destination server

public async Task<HttpResponseMessage> StoreObjectAsync(Uri namespaceUrl, string userName, string password, string objectName, Stream objectContent, string contentType = "application/octet-stream", IDictionary<string, string> systemMetadata = null)
        {                
            Uri namespaceRootUrl = Utilities.GetNamespaceRootUrl(namespaceUrl);
            using (var request = new HttpRequestMessage() { Method = HttpMethod.Put })
            {
                request.RequestUri = Utilities.CreateRequestUri(namespaceRootUrl, objectName);

                request.Content = new StreamContent(objectContent);
                request.Content.Headers.ContentType = new MediaTypeHeaderValue(contentType);

                HttpResponseMessage response;
                response = await this.httpClient.SendAsync(request, HttpCompletionOption.ResponseHeadersRead);

                return response;
            }
        }

After doing some research online, i understand from this link & this link that HttpClient on .Net 4.0 buffers the request body and because of that behavior it seemed to me that it is throwing outofmemory exception

so I changed my code to below this time using HttpWebRequest using which I have the control to specify that request should be streamed but not buffered.

public async Task<HttpResponseMessage> StoreObjectAsync(Uri namespaceUrl, string userName, string password, string objectName, Stream content, long contentLength, string contentType = "application/octet-stream", IDictionary<string, string> systemMetadata = null)
        {            

            Uri namespaceRootUrl = Utilities.GetHCPNamespaceRootUrl(namespaceUrl);

            HttpWebRequest httpWebRequest = ((HttpWebRequest)WebRequest.Create(requestUri));           

            httpWebRequest.Method = "PUT";
            httpWebRequest.KeepAlive = true;

            httpWebRequest.AllowWriteStreamBuffering = false;
            httpWebRequest.ContentType = contentType;
            httpWebRequest.ContentLength = contentLength;


            using (Stream requestStream = await httpWebRequest.GetRequestStreamAsync())
            {
                await content.CopyToAsync(requestStream);
            }

            var webResponse = await httpWebRequest.GetResponseAsync();
            HttpWebResponse httpWebResponse = (HttpWebResponse)webResponse;
            Stream httpWebResponseContent = httpWebResponse.GetResponseStream();

            HttpResponseMessage response = new HttpResponseMessage()
            {
                StatusCode = httpWebResponse.StatusCode,
                ReasonPhrase = httpWebResponse.StatusDescription,
                Content = new StreamContent(httpWebResponseContent)
            };

            return response;
        }

Now it is working fine on my local machine. I am able to upload files around 1GB without any errors or memory exceptions. Havent pushed this to Prod yet.

But I still dont understand why the same code using HttpClient on .net 4.0 worked on Dev and Cert servers but not on Prod and my local.

please help me in understanding

How to find out why it worked on Dev and Cert?
What system/server configurations will affect the memory allocations to this api?

6
  • Why is your API controller issuing an upload request to another server? Is the out of memory exception for sure from your server or could the destination server be throwing that exception? HttpClient also has some default timeouts that should be adjusted when uploading files so you may want to make sure that is increased. Also, there will be timeouts for the request from the client to this controller. You have increased the payload size in the web.config but not the timeout.
    – ManOVision
    Commented May 4, 2016 at 1:58
  • thanks @ManOVision for taking time and looking into this. this API encapsulates different storage services. we have diff storage products from diff vendors and each of them have their own interfaces and security mechanisms. this api encapsulates all those details and provides an uniform interface,thats the reason for this api uploading sending request to another servers. I will look into timeout settings but does timeout setting has any affect on memory allocations and outofmemory exceptions?
    – iRamesh
    Commented May 4, 2016 at 14:52
  • Timeouts won't affect the memory, but you'll likely run into that as soon as the memory issue is solved. I'll try to look more into this tonight. We are doing a similar thing using an API controller to "proxy" the stream to Azure Storage much like you are doing. We have been uploading multiple GB sized files with no problems. I'll see what differences we have.
    – ManOVision
    Commented May 4, 2016 at 14:57
  • Since you have total control over your controllers, do you have to stay with .net 4? Or can you upgrade? I know 4 has some limitations.
    – ManOVision
    Commented May 4, 2016 at 14:59
  • it is very unfortunate that we need to use 4.0 because of some restrictions. the server that this API is on doesn't support 4.5. so need to live with this setup for atleast an year before we migrate this to 4.5. please do share any observations from your code.. that should be very helpful.. thanks again
    – iRamesh
    Commented May 4, 2016 at 15:17

0

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.