Tips for Distributed Applications

Ingo Rammer gave an excellent presentation on Optimising Distributed Applications at DNUG yesterday. I'm going to apply all my limited skills at précis to summarise it here, so that it won't matter when I lose the scrappy bit of paper that I took notes on. The following may be incomprehensible if you weren't at the talk.

Preliminaries: Don't write distributed apps unless you have to! Seriously, modern databases can easily cope with 100s of concurrent connections; your desktop app may well just be able to connect straight to the database without a middle tier. This will cut out loads of complexity. It's practically heresy to suggest "optimising" 3-tier apps by deleting a tier, but you see his point.

Main Idea: Profile things in the following order: Network, Database Access, Memory Usage, In-Process Stuff. No point spending hours bit-twiddling if the real problem is that your app is thrashing the daylights out of the network.

1. Network Profiling. Get hold of Ethereal, TcpTrace and ProxyTrace, and use them. If trying to insert TcpTrace into a remoting conversation, tweak the server config file by pointing the machinename attribute of the channel element at TcpTrace.
   
   
2. Database Access. Sql Profiler is your friend. DataSets are, if not evil, then certainly to be avoided for high-volume transactional stuff. (For instance: copying a bunch of rows will produce many INSERT statements if you use a DataSet, but can be done with a single INSERT INTO ... SELECT FROM if you write the t-sql yourself.) Except in SQL Server 2005, where we have snapshot isolation, a SELECT statement will take out a shared lock -- although you can add WITH (NOLOCK) if you don't mind seeing uncommitted changes. This fact alone should make you think twice about using Identity Tables (imagine what happens when that lock escalates!); use Identity columns instead, it's what they're for. Lastly, locks in Enterprise Services will last longer than in regular ADO.NET, because the isolation level defaults to Serializable instead of Read Committed; if you don't need this, then change it.
   
   
3. Memory Allocation. Use CLR Profiler to find out what is allocating memory in your app. Remember to profile a release build! The goal is to allocate as little as possible (especially in ASP.NET, where allocating great wads of memory is likely to prompt a process recycle) and to avoid getting temporary objects into generation 2 of the GC. Again, DataSets seem to be much bigger than their alternatives.
   
   
4. Everything else. Now you can throw Compuware DevPartner, or RedGate ANTS, or Automated QA AQTime at the problem. You may need to try all three.
   

Tips:

• Look at Network Load Balancing. It's built in to Server 2003; you can just point at a bunch of other (not necessarily 2003) machines and say "make them a load-balancing cluster". No hardware or software requirements, but obviously you have to design with this architecture in mind, because of state-management and caching issues.
  
• The communication between Web Server and Web Services, if they are on different machines, will be limited by default to 2 concurrent connections (because the HTTP spec says so). You will probably want to fix this, which you can do on the web server by inserting something like
  

  <configuration>
     <system.net>
        <connectionManagement>
           <add name = "*" maxconnection = "99" />
        </connectionManagement>
     </system.net>
  </configuration>

  
  
• If you're using Integrated Windows authentication for your Web Service, every WS call will be rejected at first until a challenge-response is completed. This means your first call, with all its data, is pretty much a waste of time. Obvious ways around this include: having a "ping" method you call first, caching the proxy class, or calling PreAuthenticate. None of these will work (PreAuthenticate is good for Basic auth, though). Solution is either go for Basic over SSL -- not nearly as bad as it sounds, because once you have an authenticated connection you can just use it without further faffing around -- or, if you can, use the proxy's UnsafeAuthenticatedConnectionSharing property, and remember to also set the ConnectionGroupName so you're not sharing that connection with just anybody.
  

Lastly, a few random things: Use ZoomIn for demos when you can't increase the font size of a tool you want to demo. Use Notepad2 as a replacement for notepad; it does syntax highlighting and all sorts of good stuff. And subscribe to Ingo's Very Infrequent Newsletter on distributed app design.