Step 1 – Gathering dependencies

For this post I’m going to be using VStudio 2008. The primary reason is to show the explicit use of NuGet rather than depending on Visual Studio to do it for us. Now let’s get started. Start by creating a new project in Visual Studio, for this I’m going to use a simple command-line application named “SampleProtoRpc”.

After you have created the project, right-click the project and select “New Folder” and type the name “Depends”. Now visit the NuGet project page and download the “NuGet.exe Command Line bootstrapper”. It should be a single file, “NuGet.exe”. Place this file in the newly created “Depends” directory. From a command-prompt, run NuGet.exe to ensure that you are up and running.

Now right-click the project and select the “Properites” from the bottom. In the properties window, click the “Build Events” tab on the left. In the “Pre-build event command line:” text box enter the following text:

"$(ProjectDir)Depends\NuGet.exe" INSTALL Google.ProtocolBuffers.Rpc -OutputDirectory "$(ProjectDir)Depends" -ExcludeVersion -Version 1.11.1016.3

You can update the version to the latest by checking the current version at http://nuget.org/packages/Google.ProtocolBuffers.Rpc. The reason to use a fixed version is to prevent NuGet from constantly checking with the server to see if it has the latest version. By pinning the version number NuGet.exe will make a quick check and continue if it exists.

Click “Build” and view the Output window, it should contain something like the following text:

------ Build started: Project: SampleProtoRpc, Configuration: Debug Any CPU ------
"E:\Projects\Templates\SampleProtoRpc\Depends\NuGet.exe" INSTALL Google.ProtocolBuffers.Rpc -OutputDirectory "E:\Projects\Templates\SampleProtoRpc\Depends" -ExcludeVersion -Version 1.11.1016.3
Attempting to resolve dependency 'CSharpTest.Net.RpcLibrary (≥ 1.11.924.348)'.
Attempting to resolve dependency 'Google.ProtocolBuffers (≥ 2.4.1.473)'.
Successfully installed 'CSharpTest.Net.RpcLibrary 1.11.924.348'.
Successfully installed 'Google.ProtocolBuffers 2.4.1.473'.
Successfully installed 'Google.ProtocolBuffers.Rpc 1.11.1016.3'.
SampleProtoRpc -> E:\Projects\Templates\SampleProtoRpc\bin\Debug\SampleProtoRpc.exe
========== Build: 1 succeeded or up-to-date, 0 failed, 0 skipped ==========

Lastly we need to add the dependencies to the project. Right-click the “References” folder in the project and select “Add References…”. Click on the “Browse” tab in the resulting “Add Reference” dialog box. For each of the following files navigate to the directory and select the file:

• Depends\Google.ProtocolBuffers\lib\net20\Google.ProtocolBuffers.dll
• Depends\CSharpTest.Net.RpcLibrary\lib\net20\CSharpTest.Net.RpcLibrary.dll
• Depends\Google.ProtocolBuffers.Rpc\lib\net20\Google.ProtocolBuffers.Rpc.dll

Don’t worry about these being ‘net20′ assemblies, it will work fine in 3.5. If you insist upon native 4.0 images, the first two packages contain net40 versions; however, the Google.ProtocolBuffers.Rpc does not at this time. You will need to pull the source and build a 4.0 version for that library.

Step 2 – Defining a Protocol

Now that we have a project containing the correct dependencies we need to add a protocol definition file. This is a very easy format to write in, if you need help see the Google Protocol Buffers Language Guide. For now let’s get started by right-clicking the project, and selecting “Add” -> “New Item…”. Select the “General” tab on the left, and then pick the “Text File” option from the right-hand list. In the “Name” field, enter “Sample.proto” and click the “Add” button.

Once you have created the file, select “File” -> “Save As…” from the menu. Next to the “Save” button click on the drop-down arrow and choose “Save with Encoding”. Answer “Yes” when prompted to overwrite the file. From the “Encoding:” list, choose the option “US-ASCII – Codepage 20127″ near the end of the list and then click “OK”.

Now we can type a protocol buffer definition in this file. For now we are going to use the following:

package Sample;
option optimize_for = SPEED;

message MyRequest {
  required string message = 1;
}

message MyResponse {
}

service MyService {
  rpc Send (MyRequest) returns (MyResponse);
}

And this will be our first service. To generate the source, we are going to add another pre-build event. Right-click the project and select “Properties” again. On the “Build Events” pane add the following line:

"$(ProjectDir)Depends\Google.ProtocolBuffers\tools\ProtoGen.exe" --proto_path="$(ProjectDir)\" -output_directory="$(ProjectDir)\" -cls_compliance=false -service_generator_type=IRPCDISPATCH "$(ProjectDir)Sample.proto"

You should now be able to build the project successfully. Once completed, right-click the project and choose “Add” -> “Existing Item…”, then select the “Sample.cs” that should appear next to the “Sample.proto” file we created. Admittingly this is a crufty integration at best.

If you are not opposed to it, I would recommend using CmdTool.exe with a configuration similar to this example. CmdTool.exe is available in one of these downloads, you just download, run “CmdTool.exe Register” and save the configuration example in the same directory as the project. That’s about it, you now have all the source generated to do the work.

Step 3 – Implementing the Service

Before we can go much further we must implement our service interface. Right-click the project and add a new Class file so we can create our implementation, I just called mine “Implementation”. The interface has already been defined for us, it’s name will be our service’s name prepended with an ‘I’. Here is a first-pass implementation that simply writes the message to the console.

    class Implementation : IMyService
    {
        #region IMyService Members
        public MyResponse Send(MyRequest myRequest)
        {
            using (WindowsIdentity user = WindowsIdentity.GetCurrent())
            {
                Console.WriteLine("{0} says: {1}", user.Name, myRequest.Message);
                return MyResponse.DefaultInstance;
            }
        }
        #endregion
    }

Step 4 – Setting up the Listener

Time to start playing with our server-side listener. For this example we are going to allow lrpc, tcp, or named pipes. The static IID defines the interface we want to talk to. We can host several interfaces from this process, but this example will only use one. You will notice we create the generated server proxy “MyService.ServerStub” by handing it an implementation of the IMyService interface. This server stub can then be used to create the RpcServer instance. Once we add at least one protocol and call StartListening we are ready to receive calls. The setup is trivial, so without further explanation here is our new ‘Program’ class:

    class Program
    {
        static readonly Guid IID = Marshal.GenerateGuidForType(typeof(IMyService));

        static void Main(string[] args)
        {
            switch (args[0].ToLower())
            {
                case "listen":
                    {
                        using (RpcServer.CreateRpc(IID, new MyService.ServerStub(new Implementation()))
                            .AddAuthNegotiate()
                            .AddProtocol("ncacn_ip_tcp", "8080")
                            .AddProtocol("ncacn_np", @"\pipe\MyService")
                            .AddProtocol("ncalrpc", "MyService")
                            .StartListening())
                        {
                            Console.WriteLine("Waiting for connections...");
                            Console.ReadLine();
                        }
                        break;
                    }
            }
        }
    }

Step 4 – Sending a Message

Now that we have a working server we need to write a client. The reason for the Main() method above to switch on arg[0] for ‘listen’ is that we are going to use the same program for a client. The client case statement below adds support for an LRPC client call:

                case "send-lrpc":
                    {
                        using (MyService client = new MyService(
                            RpcClient.ConnectRpc(IID, "ncalrpc", null, "MyService")
                            .Authenticate(RpcAuthenticationType.Self)))
                        {
                            MyResponse response = client.Send(
                                MyRequest.CreateBuilder().SetMessage("Hello via LRPC!").Build());
                        }
                        break;
                    }

Once we have added this switch case to the Main routine we wrote we now run one process with the ‘listen’ argument, and another one with the ‘send-lrpc’ argument. We should see the following output in the server process:

Waiting for connections...
DOMAIN\user says: Hello via LRPC!

You may now create two additional case labels, one for “send-tcp”, and one for “send-np”. The only difference between them will be the parameters to the RpcClient.ConnectRpc() api. For TCP/IP we will use RpcClient.ConnectRpc(IID, “ncacn_ip_tcp”, @”localhost”, “8080″), and for named-pipes we would use RpcClient.ConnectRpc(IID, “ncacn_np”, @”\\localhost”, @”\pipe\MyService”). Go ahead and fill those in or not at your choosing.

Step 5 – Authentication

By default the RPC server will allow any user (even anonymous users) to connect. This may work for your needs, this may not. Usually you will want to impersonate the caller and then verify they have access to some resource or are a member of a specific group, etc. To do this in a generic way so that we do not have to place the impersonation code in each method we are going to implement the Google.ProtocolBuffers.IRpcServerStub interface. So let’s create a new class now called Impersonation and it’s going to look a lot like the following:

    class Impersonation : IRpcServerStub
    {
        private readonly IRpcServerStub _stub;

        public Impersonation(IRpcServerStub stub)
        {
            _stub = stub;
        }

        public IMessageLite CallMethod(string methodName, ICodedInputStream input, ExtensionRegistry registry)
        {
            using(RpcCallContext.Current.Impersonate())
            {
                return _stub.CallMethod(methodName, input, registry);
            }
        }

        public void Dispose()
        {
            _stub.Dispose();
        }
    }

Once that has been added we will update our server’s listen routine as follows:

                case "listen":
                    {
                        using (RpcServer.CreateRpc(IID, new Impersonation(new MyService.ServerStub(new Implementation())))
                            .AddAuthNegotiate()
                            ... etc ...

Now every call into every method of MyService implementation on the server will be impersonating the client user. The Rpc layer will also ensure that they are NOT an anonymous user.

Zipping it all up…

Server Options: There are numerous extensibility points on the server and client. There are a few worth covering here. The following is a brief outline of the most important configuration options.

• RpcServer.EnableMultiPart() – Allows unlimited message lengths to be received over tcp/np connections. By default Windows limits these to around 1mb. To circumvent this limitation the server and client can be configured to send messages in multiple parts. Both client and server must enable this for this to work, and doing so will enable server-side connection state.
• RpcServer.ExceptionDetails – An enumeration value that determines how much exception detail to return to the client. The default, FullDetails, returns all information in the exception including the stack trace.
• RpcServer.ExtensionRegistry – Allows registration of proto-buffer ‘extensions’ on both your service and on the transport messages defined in csharp_rpc_messages.proto. This can be used as a side-channel to flow information from the client to server and back again.
• RpcCallContext.Current

– This class provides context information about the caller, protocol, authentication, etc.

• RpcSession.EnableSessions()

– Enables session state, accessed via RpcCallContext.Session for the current call.

Client Options: The following controls the client options:

• RpcClient.EnableMultiPart() – Allows unlimited message lengths to be sent over tcp/np connections. By default Windows limits these to around 1mb. To circumvent this limitation the server and client can be configured to send messages in multiple parts. Both client and server must enable this for this to work, and doing so will enable server-side connection state.
• RpcClient.ExceptionTypeResolution – This enumeration controls the exception type resolution when an exception is returned from a server. The default, OnlyUseLoadedAssemblies, will only resolve types that are defined in assemblies that have already been loaded into the client process.
• RpcClient.ExtensionRegistry – Just as for the server, this allows proto-buffer ‘extensions’ to be registered and used when receiving response messages.
• RpcClient.CallContext – Provides access to the call context instance associated with this connection. Used with the extension registry this allows you to customize side-channel communications between the client and server.

Connection Caching
It should be noted that it is acceptable and generally useful to cache the RpcClient connection; however, you should be aware that a connection can be closed. RpcClient connections will not retry a failed call and will not attempt to reconnect to a server once disconnected. Due to this it is advisable that if you are caching client connections you should create an implementation of Google.ProtocolBuffers.IRpcDispatch. Delegate the actual RpcClient.ConnectRpc() call and configuration to this object so that it can reconnect at will. Finally use this object as as the parameter to the MyService() constructor instead of directly using the RpcClient.ConnectRpc() result.

Download the Sample RPC Project The project zip file is completely stand-alone. Just extract the contents and open the solution to build.

I completely agree with his assertion that, from an architecture point of view, the editing and publishing of content should be separated. I would however take the assertion much farther than that. Websites should NOT be capable of editing themselves. The mere idea of this is absurd IMHO. I’ve written CMS systems before back in the late 90′s, and even then it was obvious. You can not secure a self-editing website.

Why is a self-editing website a bad idea?

1. The group take down. To say most CMS systems have a vulnerability or two is putting it mildly. Attackers love to take these vulnerabilities and then proceed to use automated software to seek out sites using that CMS and exploit them. This allows them to inexpensively disperse malware to a large audience in a very short period of time. This, IMHO, is the worst thing about running a CMS solution. Nobody specifically targeted your site, it just happened to be running on software they knew how to attack. No provocation needed, you got taken down with 10,000 other unfortunate people.

2. It runs in the browser. The issue here is that some form of logon allows users to modify the content on the web server. This means that the user’s horribly insecure browser environment is entirely in control of ‘production’ content. Thus a simple XSS script, a malicious browser plugin, or other common vulnerability can allow an attacker to modify content. Browsers are the worst place to be editing content. Even with the advent of Windows Live Writer and other rich-client authoring tools you still occasionally need to log into the website. So these tools help, but they do not fix the problem.

3. Preview is not a preview. Most all the of the CMS systems out there will allow you to preview the content before publishing it. Most of them get it wrong. It seems CMS systems are more and more moving to a “wysiwyg” display editing where they modify the output HTML so that you can edit it, even in preview. This then gives you no assurance about how it will actually format and display since the authoring widgets on screen change the HTML being rendered. Furthermore while previewing a single page is possible many CMS systems will not allow you to preview entirely new sections and navigation elements. Lastly previewing an entire redesign of the site’s look-and-feel, navigation structure, etc is also not possible.

4. My web server runs DRY. CMS systems often fail to appropriately cache the rendered HTML. This produces lags in performance as you server must reprocess the same content against the template over and over again. I prefer my sever to run as DRY as possible, Don’t Repeat Yourself. There is just no point in reprocessing the content for every request.

5. User provided content. IMHO, user authored content does not belong on your sever. This is one of the driving factors behind #4 and is simply not necessary. Using Facebook or another discussion server is easy. If you need something more fancy that what is freely available, go build it. Stand up a completely different site on a different domain with a completely different authentication model. Users should never log in to your site.

6. XCopy backup and deployment. Asside from backup and deployment there is also the issue with applying a version control system to most CMS systems. This is one of my biggest pet-peeves with CMS systems. They absolutely love to rely on a database back-end. Although some newer CMS solutions can use embeded sql servers, most do not support it and this is not an option if you are farming the content across several servers. I suspect most CMS sites are not being backed up regularly and if the server is lost or it’s drive corrupted their likely to loose most if not all of their site.

What are my alternatives?

1. Find a better CMS. I’m not aware of a single CMS system in operation today that avoids the issues above. Please correct me in the comments if this is inaccurate, I’d love to know if one exists.

2. Using a CDN (Content Distribution Network). These are often very powerful tools and can be configured to avoid many of the issues mentioned above. If you are looking for one I would consider CloudFlare a viable starting point.

3. HttpClone or similar product. I’m sure there are other solutions that have similar capabilities, but honestly I love using HttpClone. I use WordPress on the back-end and have a deployment script that automates the process end-to-end. Whether I’m publishing the result to a test server or to production it’s relatively easy once you get it working. The hard part was the configuration of the crawler to identify content I wanted removed or changed, and indexing for search. Once that was complete I wrote a simple batch file to do the deployment that looks roughly like:

@ECHO OFF
HttpClone.exe crawlsite http://csharptest.net/
HttpClone.exe copysite http://csharptest.net/ http://csharptest.net /overwrite
HttpClone.exe optimize http://csharptest.net
HttpClone.exe index http://csharptest.net
HttpClone.exe addrelated http://csharptest.net
HttpClone.exe publish http://csharptest.net
mysqldump.exe -u root -ppassword --create-options --skip-extended-insert --databases csharptest --result-file=csharptest.sql

Basically what this does is crawls my locally running copy of this website (admin.csharptest.net) and captures the results. Then it crawls all the pages and changes references from admin.csharptest.net to csharptest.net overwriting the content that was previously there. Then it performs a series of steps: optimizing the content, creating the search index, and injecting related article links. Finally it packages and publishes all the content to the remote site, and then backs up the database. The entire site is instantly switched to the new content once it is ready. For small edits I can choose to publish the content directly to production, or more often I push to a local site to then verify the content package.

Obviously the most vulnerable part of the process is the code on the server that allows publication. This is why the entire thing requires the client and server to know each-other’s public key. They negotiate a session key, transfer the file, and sign/verify every request and response. This code uses the CSharpTest.Net.Crypto.SecureTransfer class from my library if you are interested in the details.

The benefit to both client and server using a public/private key is that an observer knowing only one of the two keys can learn very little about the content being transferred. It should be obvious that if an attacker obtains the servers private key they can replace the server (assuming some form of DNS poisoning or the like); however, they will not be able to then forward it to the actual server and still be able to read the content. Again it should be obvious that if someone were to obtain my client private key they can publish new or modified content to the server since this is the only form of authentication. I will add that even with my client private key they still can not upload anything that is executable on the server. This leaves my server secure and in-tact and all that is needed for me to recover is replacing the client key and republishing the content.

I wish the guys at WordPress or another CMS would just do this out of the box.

1. Items in a tree node must be presorted.
FALSE This is simply not true, a btree can decide to sort the items in a node, or to perform a linear search. Neither approach change more than the small constants in the big-O, it’s still O(n log(n)). The BPlusTree keeps it’s children sorted since the memory-move cost is nothing compared to the disk IO.

2. Split/Join of a node requires reorganizing it’s ancestors
FALSE While this is the most common approach it is also the most naive. Reorganization of the tree from the bottom-up is not only expensive but more complicated than the top-down approach. The BPlusTree uses a top-down reorg that says simply “If I’m inserting and this node is full, split” all the way down the tree. This means that at any given time I only need two nodes ‘locked’ for write, me and my parent. Once I’m sure that this node is not full I can release my parent’s lock and move down the tree. Not only does this strategy allow multiple concurrent writers, but it also prevents deadlocks. By using this top-down approach locks are always acquired at the root and traverse down so that a deadlock is simply impossible.

3. Nodes can have at most order-1 items
FALSE This is also a naive implementation detail where leaf nodes (or all nodes) have a vacant slot. It’s most commonly used when using a bottom-up reorganization so that you always have room to insert a split node. It is simply not necessary with either bottom-up or top-down reorganization, it’s just easier. BPlusTree uses all the space available in a node, only when inserting and already full will it split the node.

4. Nodes can have no fewer than order/2 items
FALSE The minimum items in a node is 1. When you calculate the lookup time with a minimum of 1 (let b = 1) you get O(log_bn) or O(n). So you never want a minimum value of 1, but you can do it. Based on my own testing the best value for the minimum node count in BPlusTree is 1/3 of the order (or max).

5. The order of a btree is the same at all levels
FALSE The BPlusTree supports having a different order for hierarchy nodes than that of it’s leaf nodes. The reason is simple. Disk-based B+Trees optimize disk access based on what will fit in a ‘page’. Using 8kb as a page size, at the leaf node this is 8192 / (sizeof(key) + sizeof(value)). At all other locations in the hierarchy this is 8192 / (sizeof(key) + c) where c is some constant size of data used to reference a child node. This means that if I am storing 1kb records with a 4-byte key I want no more than 8 records in a leaf, whereas I can have as many as 680 children in a non-leaf node. Now we can let b = 680 for O(log_bn), meaning we can find the leaf node of a 1,000,000 item tree in approximately 3 operations.

/// <summary>
/// Creates a pseudo-random password containing the number of character classes
/// defined by complexity, where 2 = alpha, 3 = alpha+num, 4 = alpha+num+special.
/// </summary>
public static string GeneratePassword(int length, int complexity)
{
    System.Security.Cryptography.RNGCryptoServiceProvider csp =
        new System.Security.Cryptography.RNGCryptoServiceProvider();
    // Define the possible character classes where complexity defines the number
    // of classes to include in the final output.
    char[][] classes =
    {
        @"abcdefghijklmnopqrstuvwxyz".ToCharArray(),
        @"ABCDEFGHIJKLMNOPQRSTUVWXYZ".ToCharArray(),
        @"0123456789".ToCharArray(),
        @" !""#$%&'()*+,./:;<>?@[\]^_{|}~".ToCharArray(),
    };

    complexity = Math.Max(1, Math.Min(classes.Length, complexity));
    if(length < complexity)
        throw new ArgumentOutOfRangeException("length");

    // Since we are taking a random number 0-255 and modulo that by the number of
    // characters, characters that appear earilier in this array will recieve a
    // heavier weight. To counter this we will then reorder the array randomly.
    // This should prevent any specific character class from recieving a priority
    // based on it's order.
    char[] allchars = classes.Take(complexity).SelectMany(c => c).ToArray();
    byte[] bytes = new byte[allchars.Length];
    csp.GetBytes(bytes);
    for (int i = 0; i < allchars.Length; i++)
    {
        char tmp = allchars[i];
        allchars[i] = allchars[bytes[i]%allchars.Length];
        allchars[bytes[i]%allchars.Length] = tmp;
    }

    // Create the random values to select the characters
    Array.Resize(ref bytes, length);
    char[] result = new char[length];

    while(true)
    {
        csp.GetBytes(bytes);
        // Obtain the character of the class for each random byte
        for (int i = 0; i < length; i++)
            result[i] = allchars[bytes[i]%allchars.Length];

        // Verify that it does not start or end with whitespace
        if (Char.IsWhiteSpace(result[0]) || Char.IsWhiteSpace(result[(length - 1) % length]))
            continue;

        string testResult = new string(result);
        // Verify that all character classes are represented
        if (0 != classes.Take(complexity).Count(c => testResult.IndexOfAny(c) < 0))
            continue;

        return testResult;
    }
}

Essentially this method starts by creating a randomly ordered set of characters to choose from. The reason this is randomly ordered is that we will offset into this array by taking a random number from 0-255 modulo the length of the array. Because of the modulo operation this can weight the first set of characters higher so we randomize it.

Once we have a randomly ordered set of characters to choose from we use the RNGCryptoServiceProvider.GetBytes to populate a byte array with random values. Each random byte will be used to select a character in the output array.

Lastly we make a few assertions about the result. Since a space ‘ ‘ can be a viable password character we make sure that the password neither starts with or ends with whitespace. Having whitespace at the beginning or end would not only be confusing, but some input forms will automatically remove the trailing whitespace. The second thing we verify is that each requested character class is represented by at least one character. This ensures that an alpha-numeric password will contain at least one number and one letter.

This brings us to the ‘how long of a password to use’ kind of question. The following is a chart to display how many password characters are required to produce an equivelent key in bit-length:

*56-bit keys are capable of being brute forced in just a few hours on current specialized hardware. Distributed systems are also capable of cracking 56-bit encryption. 128-bit keys are considered a minimum key space for today’s cryptographic algorithms.

The table is actually somewhat surprising to me in how little difference the added numeric and special characters make. To make a password as secure as the AES/128-bit algorithm you need 23 upper and lower case characters. Saving one keystroke by adding numbers to the mix just doesn’t seem all that worth while to me. When you look at most people using just 8-12 characters for what they consider a ‘good’ password it almost makes you laugh.

It seems to me that most password complexity rules requiring special characters and numbers make little sense. I guess it prevents dictionary attacks to a degree, but clearly the length of the password is much more important.

Prerequisites
Let’s start by fetching a copy of the protobuf-csharp-port binaries. We can manually download these and unpack them, use NuGet installed in VS2010, or download the NuGet Bootstrapper. I’m going to use the later approach and download NuGet.exe and run the following command:

C:\Projects\ProtoService>NuGet.exe install google.protocolbuffers -x
Successfully installed 'Google.ProtocolBuffers 2.4.1.473'.

Service Definition
With our only dependencies out of the way we are going to need to define some messages and a service. We will start with a new project in visual studio, for ease of demonstration I’ve created an ASP.NET project. To create our service definition we are going to create an empty text file and save it with a “.proto” extension. Be sure to use the File->Save As… menu on this text file, click the down arrow next to the Save button and select “Save with Encoding…”. Near the bottom choose “US-ASCII – Codepage 20127″. This is required by the protoc compiler as it does not support text BOM (Byte order mark). Now that we have a text file let’s create a message for the request and response and the service using the protobuffer definition language:

package ProtoService;

message SearchRequest {
  required string query = 1;
  optional int32 page_number = 2;
  optional int32 result_per_page = 3;
}
message SearchResponse {
  repeated group Result = 1 {
    required string url = 2;
    optional string title = 3;
    repeated string snippets = 4;
  }
}
message ErrorResult {
  required string error_text = 536870911; //max field-id
}
service SearchService {
  rpc Search (SearchRequest) returns (SearchResponse);
}

Code Generation
Now with this saved to “ProtoService.proto” we can run the ProtoGen.exe command-line tool we downloaded with NuGet earlier. ProtoGen will automatically detect that it has been given a “.proto” text file and run the protoc.exe compiler from the same directory as ProtoGen.exe. If you’re including files and setting options defined by google or the csharp port you will need to have the google directory from {Package}\content\protos copied to the location of your proto files. Since this is a stand-alone proto and not including others we don’t need to create a directory structure. Ready to build, let’s run ProtoGen now to create our generated code:

C:\Projects\ProtoService>Google.ProtocolBuffers\tools\ProtoGen.exe -service_generator_type=IRPCDISPATCH ProtoService.proto

The service_generator_type tells the ProtoGen what type of service classes/interfaces we are interested in, the value IRPCDISPATCH generates both interfaces and client/server stubs. There are lots of other options both for protoc and protogen, running ProtoGen.exe /? will list all of them. In addition this can be done directly from VStudio 2005~2010 via the CmdTool.exe integration described here for ProtoGen.exe.

Project References
Now we should find that ProtoService.cs has been created for us. Let’s now add this generated source file to our project and reference the two assemblies we need. Both of our required dependencies are located in Google.ProtocolBuffers\lib\net35, called Google.ProtocolBuffers.dll and Google.ProtocolBuffers.Serialization.dll. After we have added the two references and the generated source file we should able to compile the project. Note: if you get some warnings about CLSCompliant you can either attribute your project as CLSCompliant(true) or add the option “-cls_compliance=false” to the protogen.exe command line above.

Service Implementation
The first code we will write will be our service implementation. The code generator has defined an interface for us to implement called ISearchService. Let’s stub out that implementation now in a class called ServiceImplementation:

class ServiceImplmentation : ISearchService
{
    public SearchResponse Search(SearchRequest searchRequest)
    {
        // Create the response builder
        return SearchResponse.CreateBuilder()
            // Add a result to the response
            .AddResult(
                SearchResponse.Types.Result.CreateBuilder()
                .SetUrl("http://example.com")
                .Build()
                )
            // Build the result message
            .Build();
    }
}

Of course you’re service implementation will be a lot more complicated than this, but this will suffice for demonstration purposes. Go ahead and build your project and then let’s move on to creating the IIS handler.

IIS Handler
Our IHttpHandler implementation could be reduced to a single line call to HttpCallMethod if we chose. The following implementation adds handling of GET requests by parsing of uri query string values and some rudimentary exception handling.

Uri encoded requests are allowed for simple messages (non-nested simple types) and allow us to test right from a browser. This also allows javascript to use a GET request and pass parameters. The MIME type constant ‘ContentFormUrlEncoded’ is defined as “application/x-www-form-urlencoded” which is also the mime type used by HTML forms. This means that web clients can also simply post an HTML form to the service to execute a method, the constraint of simple types remains for forms as well.

class ServiceHandler : IHttpHandler
{
    public bool IsReusable { get { return true; } }
    public void ProcessRequest(HttpContext context)
    {
        MessageFormatOptions defaultOptions = new MessageFormatOptions();
        // Capture the request stream and content-type
        Stream requestStream = context.Request.InputStream;
        string requestType = context.Request.ContentType;

        if (context.Request.HttpMethod == "GET")
        {
            // If the call is an HTTP/GET, we will use URI encoding and the query string
            requestType = MessageFormatOptions.ContentFormUrlEncoded;
            requestStream = new MemoryStream(Encoding.UTF8.GetBytes(context.Request.Url.Query));
        }

        // Parse the HTTP accept header to determine the content-type of the response
        context.Response.ContentType = (context.Request.AcceptTypes ?? new string[0])
                                       .Select(m => m.Split(';')[0])
                                       .FirstOrDefault(m => defaultOptions.MimeInputTypes.ContainsKey(m))
                                       ?? defaultOptions.DefaultContentType;

        // Create the server-side stub to dispatch the call by method name
        using (IRpcServerStub stub = new SearchService.ServerStub(new ServiceImplmentation()))
        {
            try
            {
                // The URI's last path segment will be used for the method name
                string[] path = context.Request.Url.Segments;
                // Use the extension method defined in Google.ProtocolBuffers.Extensions to process
                // the request and write the response back to the client.
                stub.HttpCallMethod(
                        path[path.Length - 1],
                        defaultOptions,
                        requestType,
                        requestStream,
                        context.Response.ContentType,
                        context.Response.OutputStream
                    );
            }
            catch(Exception error)
            {
                // If something fails we will create an ErrorResult and serialze it with the requested
                // content-type obtained earlier while returning an HTTP 500 error.
                context.Response.StatusCode = (int)HttpStatusCode.InternalServerError;
                ErrorResult.CreateBuilder()
                    .SetErrorText(error.Message)
                    .Build()
                    .WriteTo(defaultOptions, context.Response.ContentType, context.Response.OutputStream);
            }
        }
    }
}

IIS Handler Configuration
The IIS 7x handler configuration is very straight-forward. We are binding the path to be a ‘child’ of our service description proto file “ProtoService.proto”. This, combined with the mimeMap below, allows the user to discover our service definition so that they can interact with it. So it’s time to get working, build the project and update the web.config with the following:

  <system.webServer>
    <staticContent>
      <mimeMap fileExtension=".proto" mimeType="text/plain"/>
    </staticContent>
    <handlers>
      <add name="SearchService" preCondition="integratedMode" verb="GET,POST" path="/ProtoService.proto/*"
           type="ProtoService.ServiceHandler, ProtoService, Version=1.0, Culture=neutral" />
    </handlers>
  </system.webServer>

Getting Results
Make sure you are running in IIS, this will not work in Cassini. Open up your browser (NOT IE) and enter the URL: http://localhost/protoservice.proto/Search?query=asdf You should see the following XML response:

<root>
  <result>
    <url>http://example.com</url>
  </result>
</root>

If you type something that doesn’t make sense, or generates an error, (ie. http://localhost/protoservice.proto/BadMethodName) you will see an error message like this:

<root>
<error_text>Method 'ProtoService.ISearchService.BadMethodName' not found.</error_text>
</root>

Client Proxy
Now that we have a working service, building a simple client proxy for C# binary protobuffers is really easy. First we need an implementation of the client proxy dispatch interface, IRpcDispatch. I’m going to use the WebClient here simply because it’s easy; however, production systems more often use the HttpWebRequest class.

class HttpProxy : IRpcDispatch
{
    readonly Uri _baseUri;
    public HttpProxy(Uri baseUri) { _baseUri = baseUri; }

    public TMessage CallMethod<TMessage, TBuilder>(string method, IMessageLite request, IBuilderLite<TMessage, TBuilder> response)
        where TMessage : IMessageLite<TMessage, TBuilder>
        where TBuilder : IBuilderLite<TMessage, TBuilder>
    {
        WebClient client = new WebClient();
        client.Headers[HttpRequestHeader.ContentType] = MessageFormatOptions.ContentTypeProtoBuffer;
        client.Headers[HttpRequestHeader.Accept] = MessageFormatOptions.ContentTypeProtoBuffer;
        byte[] result = client.UploadData(new Uri(_baseUri, method), request.ToByteArray());
        return response.MergeFrom(result).Build();
    }
}

Once we have this defined we can now instantiate and call the proxy.

SearchRequest result;
SearchResponse result;
using(SearchService svc = new SearchService(new HttpProxy(new Uri("http://localhost/protoservice.proto/"))))
    result = svc.Search(SearchRequest.CreateBuilder().SetQuery("bar").Build());

foreach (SearchResponse.Types.Result r in result.ResultList)
    Console.WriteLine(r.Url);

Alternative Client Formats
This proxy uses protobuffers but it could easily be adapted to use json or xml just by changing the content-type and and accept headers and serializing accordingly. To Serialize a protobuffer message as xml or json the following extensions can be used:

//XML
string xmlResult = client.UploadString(new Uri(_baseUri, method), request.ToXml());
return response.MergeFromXml(
    System.Xml.XmlReader.Create(new StringReader(xmlResult)))
    .Build();

//JSON
string jsonResult = client.UploadString(new Uri(_baseUri, method), request.ToJson());
return response.MergeFromJson(jsonResult).Build();

Lastly there are two more extension methods that can do this by simply providing a stream and a mime-type. This is demonstrated above in the catch block of our http handler. Here are the extension method prototypes that can be used:

public static void WriteTo(this IMessageLite message, MessageFormatOptions options, string contentType,
                           Stream output);

public static TBuilder MergeFrom<TBuilder>(this TBuilder builder, MessageFormatOptions options,
                                           string contentType, Stream input) where TBuilder : IBuilderLite;

Closing Remarks
I’m very biased here since I wrote most of this capability; however, I am constantly amazed at how easy protobuffers are to use. Google’s Protocol Buffers are very powerful and extremely fast. I’ve been using them for two years now and I can’t imagine writing a serialization or remoting solution without them.

Let’s face it nothing is going to be bullet proof in this arena, and certainly not the solution I’m going to suggest. Yet it is easy and will prevent someone from attaching a debugger after the program is running. I’m really not concerned with trying to prevent a debug-session from startup. Why? Because you can’t. The debugger can jump past any code you have that tries to verify that a debugger is not currently active. Besides this it provides me very little value, I want to protect an actively running process (a service). If the user has the rights to stop it in the first place (i.e. they are an admin) then there isn’t anything I can do to stop them from debugging.

What I want is to prevent someone from attaching a debugger to this service to protect potentially sensitive information. How? Well as it turns out the easiest way to prevent a debug session from starting is with a debug session. So what we need is to debug ourselves! Oh wait you can’t :( but what you can do is easily spawn another process to debug this process while we debug that new process. This reciprocal or circular debug session will prevent either process from being debugged. Further, any attempt to kill either process will immediately terminate the other process thus thwarting an effort to kill one debugger so that you can attach one.

To accomplish this in managed code we have two choices, either using the managed debugger API or the native win32 debugger API. It turns out that the managed debugger is excessively complicated and requires an extraordinary amount of COM code to pull it off (see the mdbg sample) So I chose to go with a raw win32 debug session and see if we could pull that off easily with a few PInvoke calls. Sure enough, this is really easy.

So let’s jump in and take a look at the debugging API calls we are going to need…

const int DBG_CONTINUE = 0x00010002;
const int DBG_EXCEPTION_NOT_HANDLED = unchecked((int)0x80010001);

enum DebugEventType : int
{
    CREATE_PROCESS_DEBUG_EVENT = 3, //Reports a create-process debugging event. The value of u.CreateProcessInfo specifies a CREATE_PROCESS_DEBUG_INFO structure.
    CREATE_THREAD_DEBUG_EVENT = 2, //Reports a create-thread debugging event. The value of u.CreateThread specifies a CREATE_THREAD_DEBUG_INFO structure.
    EXCEPTION_DEBUG_EVENT = 1, //Reports an exception debugging event. The value of u.Exception specifies an EXCEPTION_DEBUG_INFO structure.
    EXIT_PROCESS_DEBUG_EVENT = 5, //Reports an exit-process debugging event. The value of u.ExitProcess specifies an EXIT_PROCESS_DEBUG_INFO structure.
    EXIT_THREAD_DEBUG_EVENT = 4, //Reports an exit-thread debugging event. The value of u.ExitThread specifies an EXIT_THREAD_DEBUG_INFO structure.
    LOAD_DLL_DEBUG_EVENT = 6, //Reports a load-dynamic-link-library (DLL) debugging event. The value of u.LoadDll specifies a LOAD_DLL_DEBUG_INFO structure.
    OUTPUT_DEBUG_STRING_EVENT = 8, //Reports an output-debugging-string debugging event. The value of u.DebugString specifies an OUTPUT_DEBUG_STRING_INFO structure.
    RIP_EVENT = 9, //Reports a RIP-debugging event (system debugging error). The value of u.RipInfo specifies a RIP_INFO structure.
    UNLOAD_DLL_DEBUG_EVENT = 7, //Reports an unload-DLL debugging event. The value of u.UnloadDll specifies an UNLOAD_DLL_DEBUG_INFO structure.
}

[StructLayout(LayoutKind.Sequential)]
struct DEBUG_EVENT
{
    [MarshalAs(UnmanagedType.I4)]
    public DebugEventType dwDebugEventCode;
    public int dwProcessId;
    public int dwThreadId;
    [MarshalAs(UnmanagedType.ByValArray, SizeConst = 1024)]
    public byte[] bytes;
}

[DllImport("Kernel32.dll", SetLastError = true)]
static extern bool DebugActiveProcess(int dwProcessId);
[DllImport("Kernel32.dll", SetLastError = true)]
static extern bool WaitForDebugEvent([Out] out DEBUG_EVENT lpDebugEvent, int dwMilliseconds);
[DllImport("Kernel32.dll", SetLastError = true)]
static extern bool ContinueDebugEvent(int dwProcessId, int dwThreadId, int dwContinueStatus);
[DllImport("Kernel32.dll", SetLastError = true)]
public static extern bool IsDebuggerPresent();

The structure DEBUG_EVENT above is actually only 96 bytes in total and the bytes 12 to 96 are actually a union. I did not need any real details about the specifics so I chose to use slam a blob here and not worry about it. The byte array could be reduced in theory to 84 bytes but I really didn’t care since we are only going to create a single one of these. The rest is pretty strait-forward PInvoke junk, nothing fancy.

Using the methods above to debug a process looks something like the following code:

// Start a thread to perform the debug loop
new Thread(DebuggerThread) { IsBackground = true, Name = "DebuggerThread" }
    .Start(processId);
// Debugging thread main loop
static void DebuggerThread(object arg)
{
    DEBUG_EVENT evt = new DEBUG_EVENT();
    evt.bytes = new byte[1024];
    // Attach to the process we provided the thread as an argument
    if (!DebugActiveProcess((int)arg))
        throw new Win32Exception();

    while (true)
    {
        // wait for a debug event
        if (!WaitForDebugEvent(out evt, -1))
            throw new Win32Exception();
        // return DBG_CONTINUE for all events but the exception type
        int continueFlag = DBG_CONTINUE;
        if (evt.dwDebugEventCode == DebugEventType.EXCEPTION_DEBUG_EVENT)
            continueFlag = DBG_EXCEPTION_NOT_HANDLED;
        // continue running the debugee
        ContinueDebugEvent(evt.dwProcessId, evt.dwThreadId, continueFlag);
    }
}

Frankly I had no idea how trivial this was to do. Once I had this working I was able to quickly build a method that takes my program arguments and detects if it is a parent or child process and act appropriately. For the child we need only debug our parent process, for the parent we need to both spawn the child and debug it. Ideally this would be done as a single step since it is possible to start a process as a debugee; however, I didn’t bother to figure out how to do that. If you know and don’t mind sharing please drop a comment. Since I do know how to start a process and I know how to debug an active process I chose that approach. It doesn’t seem like it would be any more or less secure, but I could be wrong on that point. Anyway the following is what my process start-up looks like…

static void Main(string[] args)
{
    NativeDebug.DebugSelf(args);
    ... stuff ...
}
public static void DebugSelf(string[] args)
{
    Process self = Process.GetCurrentProcess();
    // Child process?
    if (args.Length == 2 && args[0] == "--debug-attach")
    {
        int owner = int.Parse(args[1]);
        Process pdbg = Process.GetProcessById(owner);
        new Thread(KillOnExit) { IsBackground = true, Name = "KillOnExit" }.Start(pdbg);
        //Wait for our parent to debug us
        WaitForDebugger();
        //Start debugging our parent process
        DebuggerThread(owner);
        //Now is a good time to die.
        Environment.Exit(1);
    }
    else // else we are the Parent process...
    {
        ProcessStartInfo psi =
            new ProcessStartInfo(Environment.GetCommandLineArgs()[0], "--debug-attach " + self.Id)
                {
                    UseShellExecute = false,
                    CreateNoWindow = true,
                    ErrorDialog = false,
                    WindowStyle = ProcessWindowStyle.Hidden
                };
        // Start the child process
        Process pdbg = Process.Start(psi);
        if (pdbg == null)
            throw new ApplicationException("Unable to debug");
        // Monitor the child process
        new Thread(KillOnExit) {IsBackground = true, Name = "KillOnExit"}.Start(pdbg);
        // Debug the child process
        new Thread(DebuggerThread) {IsBackground = true, Name = "DebuggerThread"}.Start(pdbg.Id);
        // Wait for the child to debug us
        WaitForDebugger();
    }
}
static void WaitForDebugger()
{
    DateTime start = DateTime.Now;
    while (!IsDebuggerPresent())
    {
        if ((DateTime.Now - start).TotalMinutes > 1)
            throw new TimeoutException("Debug operation timeout.");
        Thread.Sleep(1);
    }
}
static void KillOnExit(object process)
{
    ((Process)process).WaitForExit();
    Environment.Exit(1);
}

So we’ve written about 100 lines of code or so and spent almost no time at all getting it up and running, a good ROI for this kind of stuff. It worked quite well even running under the NETWORK SERVICE account as a service. It certainly stops me from debugging it although I’m certainly not what I’d call an accomplished hacker. Given the time and effort to put this together I’d have to call it a win for active debugger prevention. The down side of course is that there are now two processes running and trying to tell them apart is difficult at first launch.

If you’re going to do something like this, I would suggest adding an ‘if (!IsDebuggerPresent())’ to the else clause of our DebugSelf method above. This would allow you to launch with a debugger but not to attach one at a later time. Have fun with it and as with any code on this site, “Don’t blame me”. I didn’t make you use it ;)

My own intentions for this are to make a reasonable effort to secure a service running in a controlled but insecure environment. I would never ship something like this to a customer and hope you would not either. Preventing a consumer from accessing software they purchased is not at all what I’m after. IMHO if information is on their machine it is theirs to debug and view all they want.

One more thing, in case you are wondering this was not my idea. I read about doing this in a security article some years ago but I can’t seem to locate it.

So let’s get started, to do this we want to adjust our process’ access control list. The individual rights we can grant and deny are discussed on MSDN in the articled titled “Process Security and Access Rights“. Though most of that article is about creating a process with specific rights, in this case we want to modify the rights of our current process. To do this we are going to PInvoke GetKernelObjectSecurity to obtain the DACL (Discretionary Access Control List), modify it using the RawSecurityDescriptor, and finally write it back using the SetKernelObjectSecurity API. This should be somewhat familiar to those of you that followed that part of building our service.

Step 1 – Obtaining the process DACL

[DllImport("advapi32.dll", SetLastError = true)]
static extern bool GetKernelObjectSecurity(IntPtr Handle, int securityInformation, [Out] byte[] pSecurityDescriptor,
                                           uint nLength, out uint lpnLengthNeeded);

public static RawSecurityDescriptor GetProcessSecurityDescriptor(IntPtr processHandle)
{
    const int DACL_SECURITY_INFORMATION = 0x00000004;
    byte[] psd = new byte[0];
    uint bufSizeNeeded;
    // Call with 0 size to obtain the actual size needed in bufSizeNeeded
    GetKernelObjectSecurity(processHandle, DACL_SECURITY_INFORMATION, psd, 0, out bufSizeNeeded);
    if (bufSizeNeeded < 0 || bufSizeNeeded > short.MaxValue)
        throw new Win32Exception();
    // Allocate the required bytes and obtain the DACL
    if (!GetKernelObjectSecurity(processHandle, DACL_SECURITY_INFORMATION,
                                        psd = new byte[bufSizeNeeded], bufSizeNeeded, out bufSizeNeeded))
        throw new Win32Exception();
    // Use the RawSecurityDescriptor class from System.Security.AccessControl to parse the bytes:
    return new RawSecurityDescriptor(psd, 0);
}

And you thought that was going to be hard? Of course we also need to be able to save the DACL. So…

Part 2 – Updating the process DACL

[DllImport("advapi32.dll", SetLastError = true)]
static extern bool SetKernelObjectSecurity(IntPtr Handle, int securityInformation, [In] byte[] pSecurityDescriptor);

public static void SetProcessSecurityDescriptor(IntPtr processHandle, RawSecurityDescriptor dacl)
{
    const int DACL_SECURITY_INFORMATION = 0x00000004;
    byte[] rawsd = new byte[dacl.BinaryLength];
    dacl.GetBinaryForm(rawsd, 0);
    if (!SetKernelObjectSecurity(processHandle, DACL_SECURITY_INFORMATION, rawsd))
        throw new Win32Exception();
}

Cool, that was easy, so we are ready now right? Not quite, we still need to get the process handle. A simple thing for our own process:

Part 3 – Getting the current process

[DllImport("kernel32.dll")]
public static extern IntPtr GetCurrentProcess();

It just gets easier and easier. Since this is the current process there (AFAIK) is not any reason to duplicate handle and request additional permissions. This handle should have access to do anything. So the only thing left before we can modify the permissions is to define what those permissions are.

Part 4 – Process access rights

[Flags]
public enum ProcessAccessRights
{
    PROCESS_CREATE_PROCESS =0x0080, //	Required to create a process.
    PROCESS_CREATE_THREAD = 0x0002, //	Required to create a thread.
    PROCESS_DUP_HANDLE = 0x0040, //	Required to duplicate a handle using DuplicateHandle.
    PROCESS_QUERY_INFORMATION = 0x0400, //	Required to retrieve certain information about a process, such as its token, exit code, and priority class (see OpenProcessToken, GetExitCodeProcess, GetPriorityClass, and IsProcessInJob).
    PROCESS_QUERY_LIMITED_INFORMATION = 0x1000, //	Required to retrieve certain information about a process (see QueryFullProcessImageName). A handle that has the PROCESS_QUERY_INFORMATION access right is automatically granted PROCESS_QUERY_LIMITED_INFORMATION. Windows Server 2003 and Windows XP/2000:  This access right is not supported.
    PROCESS_SET_INFORMATION = 0x0200, //	Required to set certain information about a process, such as its priority class (see SetPriorityClass).
    PROCESS_SET_QUOTA = 0x0100, //	Required to set memory limits using SetProcessWorkingSetSize.
    PROCESS_SUSPEND_RESUME = 0x0800, //	Required to suspend or resume a process.
    PROCESS_TERMINATE = 0x0001, //	Required to terminate a process using TerminateProcess.
    PROCESS_VM_OPERATION = 0x0008, //	Required to perform an operation on the address space of a process (see VirtualProtectEx and WriteProcessMemory).
    PROCESS_VM_READ = 0x0010, //	Required to read memory in a process using ReadProcessMemory.
    PROCESS_VM_WRITE = 0x0020, //	Required to write to memory in a process using WriteProcessMemory.
    DELETE = 0x00010000, //	Required to delete the object.
    READ_CONTROL = 0x00020000, //	Required to read information in the security descriptor for the object, not including the information in the SACL. To read or write the SACL, you must request the ACCESS_SYSTEM_SECURITY access right. For more information, see SACL Access Right.
    SYNCHRONIZE = 0x00100000, //	The right to use the object for synchronization. This enables a thread to wait until the object is in the signaled state.
    WRITE_DAC = 0x00040000, //	Required to modify the DACL in the security descriptor for the object.
    WRITE_OWNER = 0x00080000, //	Required to change the owner in the security descriptor for the object.
    STANDARD_RIGHTS_REQUIRED = 0x000f0000,
    PROCESS_ALL_ACCESS = (STANDARD_RIGHTS_REQUIRED | SYNCHRONIZE | 0xFFF),//	All possible access rights for a process object.
}

Part 5 – Put it all together

// Get the current process handle
IntPtr hProcess = GetCurrentProcess();
// Read the DACL
var dacl = GetProcessSecurityDescriptor(hProcess);
// Insert the new ACE
dacl.DiscretionaryAcl.InsertAce(
    0,
    new CommonAce(
        AceFlags.None,
        AceQualifier.AccessDenied,
        (int)ProcessAccessRights.PROCESS_ALL_ACCESS,
        new SecurityIdentifier(WellKnownSidType.WorldSid, null),
        false,
        null)
);
// Save the DACL
SetProcessSecurityDescriptor(hProcess, dacl);

The ace we’ve added now will try to deny access to the “Everyone” group (aka WorldSid) to do anything with our process. We need not fear that we are interrupting an administrator’s ability to perform any action they desire. They will have any access required that’s why we call them administrators :) As I said in the beginning, we are not trying to stop Administrators, we just want to make sure they are an administrator if they plan to modify or kill this process.

We could of course be more discrete about the rights we are trying to deny, that is entirely up to you.

So let’s recap our goals for this project:

• Building a service that can also be used from the console
• Proper event logging of service startup/shutdown and other activities
• Allowing multiple instances by using command-line arguments
• Self installation of service and event log
• Proper event logging of service exceptions and errors
• Controlling of start-up, shutdown and restart options
• Handling custom service commands, power, and session events
• Customizing service security and access control

Yep, we are on the mark there and have demonstrated most of these in the past few posts. The goal not stated directly has always been to to create a new project template. We had a bit of cleaning up to do before we could deal with that. I’m not going to cover this clean up in the same level of detail as before, but here are a few words about how this has changed from the previous posts:

1. I’ve added an application icon and manifest. This brought to light that the generator for the win32 resources was not preserving the manifest, nor was it correctly locating the icon when I moved the resources file. To get this working I’ve built a custom version of the CSharpTest.Net.Generators executable.
2. For all the good it does, I’ve added an UnhandledException event listener on the AppDomain to attempt to log the exception. As you probably know this is unreliable at best ;)
3. I’ve split all the resources into three parts, messages that are written, exceptions that log, and normal non-logging string resources. These have been to a Resources folder and all resource/event specific stuff is now located there.
4. The remainder of the service stub class, “ServiceImplementation” has been filled out to include power and session events, custom commands, etc. The appropriate logging and plumbing has been introduced into the ServiceProcess class to support these new methods.
5. There are a couple of other small changes and always bugs, and I’ve had to fix a few from the previous posts.

The end result is a clean easy to use project template for Visual Studio 2008. This can easily be upgraded to 2010 if you desire. The project is currently using the 2.0 Framework to allow the greatest flexibility, but obviously once you create the project from the template it can be changed.

Download the Visual Studio Service Project Template To use the template place it in the following path for 2008: %UserProfile%\My Documents\Visual Studio 2008\Templates\ProjectTemplates\Visual C#

Once installed, create a new project, locate the template “ServiceTemplate” and create a project. Since the project name is used heavily throughout the template you should avoid names that contain non-alpha-numeric text or are longer than 64 characters. You can always rename the project file, assembly, and default namespace later, but start with a simple name. After the project has been created there are two things to do from the command line. Assuming a working directory at the root of the project the following commands should be run:

C:\Projects\MyService> Tools\CmdTool.exe register
C:\Projects\MyService> Tools\CmdTool.exe build MyService.csproj
Generating C:\Projects\MyService\Resources\Exceptions.resx
Generating C:\Projects\MyService\Resources\FormatString.resx
Generating C:\Projects\MyService\Resources\Messages.resx

Now you can build and run the project with the ‘install’ command and away you go. Modify the ServiceImplementation class to provide the service implementation. Adjust the default IsolateDomain, and ShadowCopy settings to your desired defaults. Don’t forget if you want to debug service start-up you can run the command “start {servicename} DEBUG!” and, if the service is not running, a debug break-point will be triggered.

It’s not often I have to create a windows service, yet each time it takes me several days to work all this out. I hope this will save you some time as well, enjoy.

PS: If you’ve missed any of the previous posts, here they are:

• Building a Windows Service – Part 1: Getting started
• Building a Windows Service – Part 2: Adding a Service to a Console Application
• Building a Windows Service – Part 3: Creating a Service Installer
• Building a Windows Service – Part 4: Extending the Service Installer
• Building a Windows Service – Part 5: Adding command-line installation
• Building a Windows Service – Part 6: Adding resources and event logging
• Building a Windows Service – Part 7: Finishing touches

So we have a working service it’s time to start adding some logging. Proper event logging is critical to monitoring services so we are not going to use .NET’s typical excuse for events. Rather we are going to build a ‘real’ message dll from our server, register it as a message event source, and have a real event log. Sounds crazy I know, but all the hard work has already been done by CSharpTest.Net.Generators.exe. We are also going to use CmdTool.exe to integrate integrate into Visual Studio.

The first thing we are going to need to do is to pull three binaries from my library (available here). The three we are after are CmdTool.exe, CSharpTest.Net.Generators.exe, and CSharpTest.Net.Library.dll which the Generators assembly depends upon. I’m going to just create a folder in the project directory called “Tools” and add these files there. If you’ve never used CmdTool before you will need to register it by running “CmdTool.exe Register” from an administrative command prompt.

Creating CmdTool.config
To use CmdTool for our resources we need a configuration file that tells it what to do. This is fairly trivial and is explained in some detailed here: “CmdTool.exe Configuration“. So we are going to copy/paste this default configuration and save it into a file called “CmdTool.config” in the root of our project. Here is our configuration file:

<?xml version="1.0" encoding="utf-8" ?>
<CmdTool xsi:noNamespaceSchemaLocation="http://csharptest.net/downloads/schema/CmdTool.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <match filespec="*.resx">
    <generator debug="false">
      <execute exe=".\Tools\CSharpTest.Net.Generators.exe" />
      <arg value="ResX"/>
      <arg value="-input=$(InputPath)"/>
      <arg value="-namespace=$(Namespace)"/>
      <arg value="-class=$(ClassName)"/>
      <arg value="-public=true"/>
      <arg value="-partial=true"/>
      <arg value="-test=true"/>
      <arg value="-sealed=true"/>
      <arg value="-base=System.ApplicationException"/>
      <arg value="-rxns=$(DefaultNamespace)"/>

      <std-output extension=".Designer.cs"/>
    </generator>
  </match>
</CmdTool>

Once our configuration is there we need to create standard Resources.resx file just as you always would. In order for CmdTool to find the configuration file above it must be in the same directory, or a parent directory, of the resource file. I’m just adding the Resources.resx file in the root of the project.

After we’ve added the Resources.resx file to the project we now need to modify the “Custom Tool” property for the solution item. Right-click on the resx file and select “Properties”. You should the option there, just type ‘CmdTool’ in the value next to Custom Tool. Make sure you’ve registered CmdTool or you will get an error. The end result should look like this:

Once this is changed, you’re output window should display “Generating C:\{some path}\Resources.resx” with the full path to the location of the resx file. Once that is working we already have a lot of bang for our buck. Creating a resource string like “String1″, “{0}” will now create a static method with the signature ‘string Resources.String1(object _0)’. That is neat, we can even create exceptions just by renaming “String1″ to “String1Exception” and presto-magic happens and you have a complete exception class. I can’t live without this thing, I’ve used it sparingly in the open but I use it extensively in my day job as well as private projects. We have over 250 custom exceptions, each have a unique HRESULT that can be written to the Event Log and monitored. This is what we are after, Adding support for the event logging part.

Before we go any farther with this you should be very familiar with what is an HRESULT, what it’s composite parts are, and what their constraints are. Most of this is documented in the help for Message Text Files which is exactly what we are going to be generating.

So let’s get started turning our resx into an event log writer. Step 1, we need to add some information, the easiest way is to just copy/paste the following settings into the “View Source” xml view:

  <assembly alias="mscorlib" name="mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" />
  <data name=".EventCategory" type="System.Int32, mscorlib">
    <value>1</value>
    <comment>General</comment>
  </data>
  <assembly alias="System" name="System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" />
  <data name=".EventSource" type="System.Uri, System">
    <value>ServiceTemplate/ServiceTemplate</value>
  </data>
  <data name=".AutoLog" type="System.Boolean, mscorlib">
    <value>True</value>
  </data>
  <data name=".AutoLogMethod" type="System.Uri, System">
    <value>EventLogging.WriteEvent</value>
  </data>
  <data name=".Facility" type="System.Int32, mscorlib">
    <value>256</value>
    <comment>General</comment>
  </data>
  <data name=".HelpLink" type="System.Uri, System">
    <value>http://example.com/?HResult={0:x8}</value>
  </data>
  <data name=".NextMessageId" type="System.Int32, mscorlib">
    <value>2</value>
  </data>
  <data name=".EventMessageFormat" xml:space="preserve">
    <value>
More Information: {1}
Excpetion Type: {2}
Base Type: {3}
Stack Trace: {4}
{5}</value>
  </data>

Once the above has been inserted into the resx file you can verify proper behavior by adding any resource string. When you save the resx file, a Comment value should magically appear, something like “#MessageId=1″. Do not change or remove, these ids should not be reused to avoid confusion with old event logs, the “.NextMessageId” controls the next id that will be assigned. Occasionally, if you’re custom exception crazy, you may run into source merge conflicts. At that point you need to make sure all messages have a unique id (it will fail if this is not the case) and that the “.NextMessageId” value is higher than all existing ids.

The “.AutoLogMethod” above, “EventLogging.WriteEvent” is to allow us to customize the values provided to the event log. This method is called when either a method is called to format a string, or when an exception is constructed (not thrown). Because we are augmenting the data in the message with the “.EventMessageFormat” above we are required to write this method. We could simply remove both the “.AutoLogMethod” and “.EventMessageFormat” and it would just create the log and write it for us. By customizing this however we can get some fixed fields on our exceptions that will help in monitoring and triggering of alerts. So here is our custom EventLogging class that will provide some fixed values and also provide a serialized exception as the binary data for the event.

public class EventLogging
{
    private static readonly Dictionary<string, EventLog> _logs = new Dictionary<string,EventLog>(StringComparer.Ordinal);

    public static void WriteEvent(string eventLog, string eventSource, int category, EventLogEntryType eventType, long eventId, object[] arguments, Exception error)
    {
        EventLog log;
        lock(_logs)
        {
            if(!_logs.TryGetValue(eventLog + '.' + eventSource, out log))
                _logs.Add(eventLog + '.' + eventSource, log = new EventLog(eventLog, ".", eventSource));
        }

        const int fixedCount = 6;
        object[] fullargs = new object[arguments.Length + fixedCount];
        int ix = 0;
        fullargs[ix++] = eventId.ToString("x8");
        fullargs[ix++] = error == null ? null : error.HelpLink;
        fullargs[ix++] = error == null ? null : error.GetType().ToString();
        fullargs[ix++] = error == null ? null : error.GetBaseException().GetType().ToString();
        fullargs[ix++] = error == null ? null : error.StackTrace;
        fullargs[ix++] = new StackTrace(3).ToString();
        Array.Copy(arguments, 0, fullargs, fixedCount, arguments.Length);

        byte[] data = null;
        if (error != null)
        {
            try
            {
                using (MemoryStream ms = new MemoryStream())
                {
                    BinaryFormatter ser = new BinaryFormatter();
                    ser.Serialize(ms, error);
                    data = ms.ToArray();
                }
            }
            catch { }
        }

        lock(log)
            log.WriteEvent(new EventInstance(eventId, category, eventType), data, fullargs);
    }
}

Now we can build again, and the biggest thing left is getting these .NET resources turning into Win32 message resources. We have two options here, option 1 is to produce a stand-alone message dll. This can work well in some cases, but I really want an all-in-one approach here so I’m going to embed the messages into this managed assembly. File versions, app icons, etc should survive; however, be aware that this tool builds a win32 resource that must later be bound to this project. To get the generator producing resources we just need to add a pre-build event.

$(ProjectDir)Tools\CSharpTest.Net.Generators.exe ProjectResX "$(ProjectPath)" /name=Messages\Resources "/tools=$(Registry:HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Microsoft SDKs\Windows@CurrentInstallFolder)Bin"

Here I’m using the Windows SDK versions of RC.exe and MC.exe which are both required. You could optionally change the tools path to another location if desired so long as both of these tools exist. After you define the pre-build event above build the project. Assuming that worked there is a new directory below the project file called “Messages”. This Messages directory contains two C# files we need to add to this project, Resources.Constants.cs, and Resources.InstallUtil.cs. Include these by clicking the “Show All Files” option and right-click to select “Include in Project”. Now we need to manually add an entry to our project. First right-click the project and select “Unload Project”. Then right-click again and select “Edit XXX.csproj” to view the file in XML. At the end of the first PropertyGroup element we need to add the following:

  <PropertyGroup>
    ...
    <Win32Resource>$(ProjectDir)Messages\Resources.res</Win32Resource>
  </PropertyGroup>

This can be done in the project properties dialog, however, if you have assigned an icon, manifest, or something it will be erased. This ensures those properties remain although you will not be able to edit them. Now rebuild the project. You can verify that the resources are properly being created by using the File -> Open -> File… command and selecting the .exe file you just built. You should see at least the resources depicted to the right.

Now that we’ve built it we can simply run the .exe file with the ‘install’ argument to register both the service and the event log/source. Before we do that it would be nice to have something in that event log to see. So let’s start with adding a resource string for service startup and shutdown. I’m going to start with the following messages in the resources.resx file:

ServiceStartFailedException
The service {0} failed to start.
#MessageId=1


ServiceStarted
The service {0} started successfully.
#MessageId=2


ServiceStopped
The service {0} has stopped.
#MessageId=3

Now we are going to update the service process to add the logging:

protected override void OnStart(string[] args)
{
    try
    {
        OnStop();
        _service = new ServiceImplementation();

        List allarguments = new List(_arguments);
        if (args != null && args.Length > 0)
            allarguments.AddRange(args);

        _service.Start(allarguments);
        Resources.ServiceStarted(ServiceName);
    }
    catch(Exception e)
    {
        throw new ServiceStartFailedException(ServiceName, e);
    }
}

protected override void OnStop()
{
    if (_service == null)
        return;
    try
    {
        _service.Stop();
        _service.Dispose();
        _service = null;
    }
    finally
    {
        Resources.ServiceStopped(ServiceName);
    }
}

Once this has been added we will build and run the following to install, start, and stop the service. After that we are going to the Win7/Win2008 command-line tool to export the log to xml so we can peek at the events.

ServiceTemplate.exe
net start ServiceTemplate
net stop ServiceTemplate
Wevtutil.exe qe ServiceTemplate /f:RenderedXml

The output of Wevtutil.exe should display something like the following:

<Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'>
  <System>
    <Provider Name='ServiceTemplate'/>
    <EventID Qualifiers='16640'>3</EventID>
    <Level>4</Level>
    <Task>1</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime='2011-11-17T22:10:47.000000000Z'/>
    <EventRecordID>3</EventRecordID>
    <Channel>ServiceTemplate</Channel>
    <Computer>HOSTNAME</Computer>
    <Security/>
  </System>
  <EventData>
    <Data>41000003</Data>
    <Data></Data>
    <Data></Data>
    <Data></Data>
    <Data></Data>
    <Data>
      at ServiceTemplate.Service.ServiceProcess.OnStop()
      at System.ServiceProcess.ServiceBase.DeferredStop()
      at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Int32 methodPtr, Boolean fExecuteInContext, Object[]& outArgs)
      at System.Runtime.Remoting.Messaging.StackBuilderSink.AsyncProcessMessage(IMessage msg, IMessageSink replySink)
      at System.Runtime.Remoting.Proxies.AgileAsyncWorkerItem.ThreadPoolCallBack(Object o)
      at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
      at System.Threading._ThreadPoolWaitCallback.PerformWaitCallbackInternal(_ThreadPoolWaitCallback tpWaitCallBack)
      at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(Object state)
    </Data>
    <Data>ServiceTemplate</Data>
  </EventData>
  <RenderingInfo Culture='en-US'>
    <Message>
      The service ServiceTemplate has stopped.

      More Information:
      Excpetion Type:
      Base Type:
      Stack Trace:
      at ServiceTemplate.Service.ServiceProcess.OnStop()
      at System.ServiceProcess.ServiceBase.DeferredStop()
      at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Int32 methodPtr, Boolean fExecuteInContext, Object[]& outArgs)
      at System.Runtime.Remoting.Messaging.StackBuilderSink.AsyncProcessMessage(IMessage msg, IMessageSink replySink)
      at System.Runtime.Remoting.Proxies.AgileAsyncWorkerItem.ThreadPoolCallBack(Object o)
      at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
      at System.Threading._ThreadPoolWaitCallback.PerformWaitCallbackInternal(_ThreadPoolWaitCallback tpWaitCallBack)
      at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback(Object state)
    </Message>
    <Level>Information</Level>
    <Task>General</Task>
    <Opcode></Opcode>
    <Channel></Channel>
    <Provider></Provider>
    <Keywords>
      <Keyword>Classic</Keyword>
    </Keywords>
  </RenderingInfo>
</Event>

Of course you may not want that much detail in your event log for informational messages like this one. The easiest way to achieve that is to separate the errors from the exceptions by creating another resx file with a different facility id (It can use the same event log/source, or a new one). For some uses this much detail is more than you want even for exceptions, but it should be obvious enough now how to modify this to prune out the data you don’t want. Our service is fairly complete now, it installs, run, logs, etc. We have a little more work to do in the service behavior itself to support some nice-to-have things like pause/continue, custom messages, etc.

Continued on “Building a Windows Service – Part 7: Finishing touches“