Be quite, the story is starting :)

Not Responding Story

I was working on the new version of our windows based product. Main change for this version, it is a multithread application. The previous version of the application was working on a single thread. There is a simple service based architecture on the product, application (client) calls a web service (server) then the web service responses the client request an so on.. For a single threaded application generally it works well, except the client connection issues (because of poor internet connection) or sometimes server has more load so replies comes a bit late and this is  causing to the client "Not Responding".. Not responding message is like a horror film scene for a developer :)

In fact, this problem is the main reason why we decided to pass a multithreaded interface. If you have "not responding" issues, that's mean your application is waiting for some response or in an operation on the main thread. So you need to another thread for your long processing operation. We do this with delegates. About this tech, check out here or here or just google it. A good coded threading application never says "Not Responding"..

But your request can say "Request time out". Another horror scene for a developer..

Request Timeout Story

Why? There is not an answer.. There should be many reasons like a slow running query, long running process, short timeout durations, wrong application configuration, firewall and others... If you do not know source of the real problem, this great tools can save your life;

  1. Fiddler  : http://fiddler2.com/

    This great http proxy tool shows all requests, responses and many other details about back of your application on runtime.
    Fiddler
     
  2. Wireshark : http://www.wireshark.org/

    If you need a professional network sniffer, you dont need more. You can see all TCP / UDP connections and all the steps for a connection life cycle. You can find more than you expect;


     
  3. TCPView : http://technet.microsoft.com/en-us/sysinternals/bb897437.aspx

    It is a good tool to see live connections on your computer. It is like auto refreshed netstat command, also a bit more. Also very easy to use.


     
  4. NetMon : http://blogs.technet.com/b/netmon/

    Another network capturer and protocol analyser tool.

 

After ads, back to the timeout problem. This tools helped me to find the real problem. there was not a slow SQL query or long running process. Client is calling web service methods in different threads but some of the requests not completing. Not certain methods and not in a certain time line. Wireshark showed me some bad TCP packets absolutely with expect-100 behaviour. So removed this header with this line.

System.Net.ServicePointManager.Expect100Continue = false;

This line must work before the web service proxy created. You can see more details about expect-100 from here. From other hand problem still continues and when i check the IIS logs, everything seems good and run times per each requesst under than one or two seconds. Then i asked to myself, does all connections arrives to the IIS ?

Half Open Connection Story

This is a long story for me.. I checked IIS logs again and then i see that some requests does not arrive to the server. I closed firewalls and then tried again but nothing changed. Another developer from our company said that, this is half open connection limit. I started to resource about this situation and made too many tests with watching all TCP actions with the above tools. I saw the half-open connections then decided to close any opened connection after needs done.

using (WebResponse webResponse = (HttpWebResponse)webRequest.GetResponse())
                {
                    using (Stream webStream = webResponse.GetResponseStream())
                    {
                        bmp = Image.FromStream(webStream) as Bitmap;
                        webStream.Close();
                    }
                    webResponse.Close();
                }

Actualy closing the response is required but i don't mean this, i mean;

System.Net.ServicePointManager.SetTcpKeepAlive(true, 1000, 500);

This code line must work before the web service proxy, like expect-100 code line. Firstly i tried to remove completely Keep-Alive header but according my stress tests, holding connection for a short time (in here one second) has better results.

Long story finished here, TCP connections closed quickly but still the problem continued.. I said long story because i spend too many time here..

Connection Limit Story

While this solution resource duration, i made many many test. Also i developed a web service connection creator tool. This tool calls same web method simultaneosly how many i wanted and download some pictures with the same time from same server. Then i see that after 10 opened connections on my comouter, 11 th connection waiting and waiting to complete other requests. Then it completes or it fails with a timeout issue. I think i found Chucky. After that is more easy. I have two choices;

  1. Increase the connection limit
  2. Decrease connection count to the server

I did both of these. Firstly i created new web methods to get more data in a single request, and then increased the connection limit with the line below

System.Net.ServicePointManager.DefaultConnectionLimit = 16;

Now i send less than 8 connections simultanously and it passed all my tests. Also i tried this with different computers (both 32 or 64 bit)

Also i am trying to change MaxConnectionPerServer registry key with code;

GetSetRegistry(Microsoft.Win32.Registry.CurrentUser,
                    @"Software\Microsoft\Windows\CurrentVersion\Internet Settings",
                    "MaxConnectionsPerServer",
                    32,
                    Microsoft.Win32.RegistryValueKind.DWord,
                    true);

 

public static string GetSetRegistry(RegistryKey key, string subKeyPath, string value, object data, RegistryValueKind valueKind, bool writable)
        {
            string retVal = string.Empty;
            try
            {
                RegistryKey subKey = key.OpenSubKey(subKeyPath, writable);
                if (subKey != null)
                {
                    if (writable)
                    {
                        subKey.SetValue(value, data, valueKind);
                    }
                    retVal = subKey.GetValue(value).ToString();
                }
            }
            catch (Exception ex)
            {
                LogWriter.WriteLog("GetSetRegistry", ex.ToString());
            }
            return retVal;
        }

By the way, MaxConnectionPerServer registry key is not stored in some computers. Check out Microsoft KB.

Now, multithreaded application works faster with full response for our users.

 

If you like this, follow my RSS channel!