Friday, 20 August 2010

Working WCF Peer Mesh Code

I've had a few requests for the working source code for the wcf peer mesh, those of you that have asked will be happy to hear i finally got around to digging out the code and cleaning it up a little, I have hosted on my new Open Source SVN Repository
http://svn2.xp-dev.com/svn/smpPeerMesh/trunk/PeerResolver1.0/
anyone who doesn't use svn can email me, and i'll zip up the files and send them out to you.
I've build this in VS 2010, but its .net 3.5 so should open in 2008 without any problems (again if problems give me a shout and i will rebuild in 08).
Hope its of some help, I have implemented this in a production environment to provide a wpf ui with live updates on a background worker thread.
To cover myself... I take no responsibilty for the use of this code or code written on the back of it.

Wednesday, 7 July 2010

IIS Wcf and msmq Load balancing proposal

So, the system I'm currently re-architecting is essentially a bunch of wcf (tcp and msmq) windows services sat on a single box and various types of client, for various reasons its approaching capacity.

We have been looking at various options to provide the magic, scalable available reliable platform. Some are obvious (and previously spoken about), and simply involve replacing certain heavy layers with simpler (designed for usee) push systems and efficient data access (don't get me started on loading the entire data set to get a customer name)...

Anyway, after a lot of research, looking at hardware and software options i think we have finally settled on an idea (at least until i have prototyped it all).

So couple of thoughts before i divulge. Firstly we only actually use netTcpBinding to provide simple binary data transport, which in fact is supported by http by specifying the binary transport in the binding. Http is significantly more flexible, and opens our options greatly. Tcp binding by definition doesn't strictly support per call load balancing, its a session transport protocol, and requires careful tweaking to provide something similar to good per call scaling. Finally after spending far too much time playing, IIS7 and Windows activation services for IIS hosted services is fiddly at best.

So http binding is the way to go, IIS support out of the box, and a lot of options opened up, so i started to look at IIS 7.5 and web farms. First thing i find is Shared Config and Content, store your content in a secure shared environment (obvious and supported in iis 6 with some playing). but the important bit is the shared config (there are some great videos out there demoing the tech, so get google-ing). Basically set your site/service up on your first (master) server. The goto Configuration Sharing (in iis manager) and export the config to a shared location (create a password as appropriate), then switch on shared config pointing at the export you created and check everything still work.
Now on to the interesting part... Go to your additional servers in the farm, in config sharing switch on sharing (pointing at your config again). IIS needs to restart, and on doing so you'll find all your sites / applications you set up on the master have been configured for you on the additional boxes.
Ok so that takes care of deployment, but what about load balancing, NLB doesn't really hack it for most uses, if your hammering the boxes, and cpu is high, you need to check your code and architecture (could be done better), its very easy and possible though to max out threads on a wcf service, especially if it has relatively long running processes, this is why we want a web farm, instance load balancing. Yes you can go for something like a Cisco ACE module for hardware balancing, but that has a big price tag attached to it. NLB isn't a true load balancer (and this bit is only my opinion) it hammers one box until that box is unhappy (very high load) before it moves over to the next in the cluster. In comes a much more sophisticated software solution from microsoft, Application Request Routing module for IIS (ARR). This only handle http url routing, hence http bindings for wcf, essentially you set up a new box (or active/passive failover cluster for safety). In iis manager install, then create your web farm, defining the incoming URL, and the target instances in the farm. Whats really impressive is the load balancing, you have some common options out of the box (such as round robin, weighted etc) or fully customised algorythms, the examples on the IIS site talk about geographic datacenter routing (ie you'll get routed to your nearest available datacenter).
All this true load balancing (thank you microsoft, i was starting to bang my head against a brick wall) is wonderful and all, but doesn't resolve our push requirements, i intend to rely heavily on MSMQ for this, (using durable messaging for improved safety). I've looked at nServiceBus and a couple of others, but to be honest its firstly another technology for me to guide my team through, and secondly i'm unsure how well proven it is at enterprise level. I don't really have particularily complex requirements, (its all in the high and mid level architecture) so I'm looking at remote routing, i've heard tell that this is only supported in the legacy MsmqIntegrationBinding, however it looks like an AD addressing issue which won't affect my project (You would have to try very very hard to convice me to use public queues with AD integration in MSMQ - its a nice idea but over complicates what is essentially - pass this message to that server/farm over there - i always know where i'm pointing to, or someone has designed something badly). So the plan is to host the queue on the routing server, and have service instances (IIS 7 with WAS msmq services under appfabric) pointing at the remote host. Now i have to be honest, not sure if this is going to work, i know msmq can handle remote hosts (with a small overhead during locking of a message for pickup - blocking other services picking up a message briefly), I just need to test this configuration (finger crossed).
Finally since i mentioned it already appfabric (specifically dublin components for now), this amongst many things provides improved was support, and importantly instance monitoring and logging (stunning configurable wcf logging can be used).
So i've gone on a lot, and nothing particularily concrete at this time, i will however post the results of my prototyping.

Thursday, 18 March 2010

wcf and iis7

There are many good posts out there on how to create a wcf service, so not going to go in any detail, suffice to say that a service contract, its implementation (I like these in there own assemblies, as it makes referencing easier) and a wcf web site with the appropriate service.svc file pointing at your implementation. Then its all config config config. although .net 4 is apparently moving away from the config dependancy i've not had chance to play.

The easiest option by far is to host an http binding service, but there are performance drawbacks, so look at the project its expected use before deciding on this. if you need to pass lots of data or you need speed go with tcp, and although you can load balance net.tcp its easier with http.

net.tcp and net.msmq are a little more involved, first make sure you have installed the non-http wcf component of .net in windows components, then check the nettcpactivator etc services are running, if you have already installed .net 4 you'll have to remove before doing the above as it causes problems. once these are running though its safe to reinstall. Now add the appropriate bindings (with port for tcp eg 9191:* and machine name with queue for msmq). all thats left is to add the protocols. you'll see many posts with command promt lines to copy edit paste and run. i think its far easier to open inetmgr, right click on the parent site, select advanced settings, then manage site and where it says http add ",net.tcp" and/or ",net.msmq" as appropriate (no spaces) then do the same for the app containing your service under the site. I also like to use the same credentials for the app pool and net activator services to avoid any possible issues.

That should be it, the activator services are the listeners and activate the service call when a request hits the server.

One quick note on msmq, you have to create the queue yourself, and unless you really need to advertise the queue on ad (and generally you don't) keep the queue private, the only difference between public and private is that public uses actve directory, if you plan to point your client endpoints at a server name or ip, public is a waste of resources and will hurt performance for no good reason. I have a theory aout load balancing msmq services in iis, and it involves using a horizontal pattern (lots of services on one machine pointing at a single queue, which works very well) with a vertical infrastructure (many machines) by pointing the nodes at a single queue on the primary server which should in theory work. I have a collegue to thank for the idea, and will post soon when i have tested it.

Tuesday, 9 March 2010

Load Underestimations

WCF services in an N-Tier service based system is very useful, but is dependant on appropriate design. Im currently in the process of re-designing a processing system. On initial deployment and for 18 months after the system could cope with the load. but not any more, and I'm faced with a full architecture change.

So a few thoughts on designing scalable systems, 1 if you have heavy critical processes in the chain consider the use of msmq with throttling to manage load on these processes. The same can be said for external system dependancies, technology like msmq can help.

Don't poll production databases to look for specific data. This is obvious i know, but I see so many ideas flawed by this approach. Try instead to use a push system to handle such requirements.

Load balancing, ideally design your architecture to allow load balancing, obviously small systems don't require this, but enterprise level applications should support it, even if its not going to be used straight away, at least its an option. Using IIS to host the services is an obvious way to do this.

Friday, 19 February 2010

MSMQ host in windows 7

I've recently had a problem running a wcf msmq host in windows 7, specifically to debug a new service. Turns out this was purely down to the installation of msmq. Unlike Windows XP, Windows 7 includes msmq out of the box, therefore generally a fresh windows install is not on a company domain at this point, and thats the problem. To create and manage a public queue on a domain, the msmq service needs to be integrated into AD (just making sure the check box is ticked isn't enough).

So if you are having trouble, goto windows components, uninstall msmq, check your logged into the domain and install again. final step is to make sure that the application in question has permissions to manage msmq, in vs debug, just run as administrator, otherwise the service user (not tested this yet though) should be fine.

Friday, 2 October 2009

Just read an interesting post on Scott Allen's blog, and the following go me thinking.... (http://odetocode.com/Blogs/scott/archive/2009/09/29/13248.aspx)

"The next time your user asks for something and your first impulse is to start assessing the risks involved – STOP.

Think about the possibilities first. Think about the value to the user. Could this request make the software great? If not, how could it be even better? What can you do in this area to empower the user? How can the software truly transform their job, or their business? What could you change about your technology to make this work? What could you change about your team to make this work? What could you change about your company to make this work? How many days would you really need to experiment with something new and different? Only after you understand the true potential of what is possible can you can put on your engineering hat and assess the risks.

We all work with limited resources and deadlines, but we often use these constraints as excuses instead of parameters in a formula for victory.

Only courage allows us to find the true potential of software. Only courage allows us to recognize our weaknesses and try something new. Only courage allows us to explore unfamiliar landscapes in the world of software development.Only courage will allow us to align our goals with the goals of the user, make our software great, and give us a shot at undisputed victory. "

This is a very good practice, and i agree completely except for one thing, users will and do ask for utterly rediculous things form time to time.

Cruise Control .net Project Dependency

Ok just a quick post, I'm NOT taking any credit for this, great blog post on handling Cruise Control .net project dependency (using this solution in 1.4.4)

http://richardsbraindump.blogspot.com/2007/07/integration-queues-in-cruisecontrolnet.html