How to build a multi-protocol REST service with protocol buffers, xml, and json in 10 minutes or less using protobuf-csharp-port
In this post we are going to explore some great new features introduced in the latest release of the protobuf-csharp-port project. We are going to build both an IIS service to handle requests as well as a sample client. Let’s get started.
Prerequisites
Let’s start by fetching a copy of the protobuf-csharp-port binaries. We can manually download these and unpack them, use NuGet installed in VS2010, or download the NuGet Bootstrapper. I’m going to use the later approach and download NuGet.exe and run the following command:
C:\Projects\ProtoService>NuGet.exe install google.protocolbuffers -x Successfully installed 'Google.ProtocolBuffers 2.4.1.473'.
Service Definition
With our only dependencies out of the way we are going to need to define some messages and a service. We will start with a new project in visual studio, for ease of demonstration I’ve created an ASP.NET project. To create our service definition we are going to create an empty text file and save it with a “.proto” extension. Be sure to use the File->Save As… menu on this text file, click the down arrow next to the Save button and select “Save with Encoding…”. Near the bottom choose “US-ASCII – Codepage 20127″. This is required by the protoc compiler as it does not support text BOM (Byte order mark). Now that we have a text file let’s create a message for the request and response and the service using the protobuffer definition language:
package ProtoService;
message SearchRequest {
required string query = 1;
optional int32 page_number = 2;
optional int32 result_per_page = 3;
}
message SearchResponse {
repeated group Result = 1 {
required string url = 2;
optional string title = 3;
repeated string snippets = 4;
}
}
message ErrorResult {
required string error_text = 536870911; //max field-id
}
service SearchService {
rpc Search (SearchRequest) returns (SearchResponse);
}
Code Generation
Now with this saved to “ProtoService.proto” we can run the ProtoGen.exe command-line tool we downloaded with NuGet earlier. ProtoGen will automatically detect that it has been given a “.proto” text file and run the protoc.exe compiler from the same directory as ProtoGen.exe. If you’re including files and setting options defined by google or the csharp port you will need to have the google directory from {Package}\content\protos copied to the location of your proto files. Since this is a stand-alone proto and not including others we don’t need to create a directory structure. Ready to build, let’s run ProtoGen now to create our generated code:
C:\Projects\ProtoService>Google.ProtocolBuffers\tools\ProtoGen.exe -service_generator_type=IRPCDISPATCH ProtoService.proto
The service_generator_type tells the ProtoGen what type of service classes/interfaces we are interested in, the value IRPCDISPATCH generates both interfaces and client/server stubs. There are lots of other options both for protoc and protogen, running ProtoGen.exe /? will list all of them. In addition this can be done directly from VStudio 2005~2010 via the CmdTool.exe integration described here for ProtoGen.exe.
Project References
Now we should find that ProtoService.cs has been created for us. Let’s now add this generated source file to our project and reference the two assemblies we need. Both of our required dependencies are located in Google.ProtocolBuffers\lib\net35, called Google.ProtocolBuffers.dll and Google.ProtocolBuffers.Serialization.dll. After we have added the two references and the generated source file we should able to compile the project. Note: if you get some warnings about CLSCompliant you can either attribute your project as CLSCompliant(true) or add the option “-cls_compliance=false” to the protogen.exe command line above.
Service Implementation
The first code we will write will be our service implementation. The code generator has defined an interface for us to implement called ISearchService. Let’s stub out that implementation now in a class called ServiceImplementation:
class ServiceImplmentation : ISearchService
{
public SearchResponse Search(SearchRequest searchRequest)
{
// Create the response builder
return SearchResponse.CreateBuilder()
// Add a result to the response
.AddResult(
SearchResponse.Types.Result.CreateBuilder()
.SetUrl("http://example.com")
.Build()
)
// Build the result message
.Build();
}
}
Of course you’re service implementation will be a lot more complicated than this, but this will suffice for demonstration purposes. Go ahead and build your project and then let’s move on to creating the IIS handler.
IIS Handler
Our IHttpHandler implementation could be reduced to a single line call to HttpCallMethod if we chose. The following implementation adds handling of GET requests by parsing of uri query string values and some rudimentary exception handling.
Uri encoded requests are allowed for simple messages (non-nested simple types) and allow us to test right from a browser. This also allows javascript to use a GET request and pass parameters. The MIME type constant ‘ContentFormUrlEncoded’ is defined as “application/x-www-form-urlencoded” which is also the mime type used by HTML forms. This means that web clients can also simply post an HTML form to the service to execute a method, the constraint of simple types remains for forms as well.
class ServiceHandler : IHttpHandler
{
public bool IsReusable { get { return true; } }
public void ProcessRequest(HttpContext context)
{
MessageFormatOptions defaultOptions = new MessageFormatOptions();
// Capture the request stream and content-type
Stream requestStream = context.Request.InputStream;
string requestType = context.Request.ContentType;
if (context.Request.HttpMethod == "GET")
{
// If the call is an HTTP/GET, we will use URI encoding and the query string
requestType = MessageFormatOptions.ContentFormUrlEncoded;
requestStream = new MemoryStream(Encoding.UTF8.GetBytes(context.Request.Url.Query));
}
// Parse the HTTP accept header to determine the content-type of the response
context.Response.ContentType = (context.Request.AcceptTypes ?? new string[0])
.Select(m => m.Split(';')[0])
.FirstOrDefault(m => defaultOptions.MimeInputTypes.ContainsKey(m))
?? defaultOptions.DefaultContentType;
// Create the server-side stub to dispatch the call by method name
using (IRpcServerStub stub = new SearchService.ServerStub(new ServiceImplmentation()))
{
try
{
// The URI's last path segment will be used for the method name
string[] path = context.Request.Url.Segments;
// Use the extension method defined in Google.ProtocolBuffers.Extensions to process
// the request and write the response back to the client.
stub.HttpCallMethod(
path[path.Length - 1],
defaultOptions,
requestType,
requestStream,
context.Response.ContentType,
context.Response.OutputStream
);
}
catch(Exception error)
{
// If something fails we will create an ErrorResult and serialze it with the requested
// content-type obtained earlier while returning an HTTP 500 error.
context.Response.StatusCode = (int)HttpStatusCode.InternalServerError;
ErrorResult.CreateBuilder()
.SetErrorText(error.Message)
.Build()
.WriteTo(defaultOptions, context.Response.ContentType, context.Response.OutputStream);
}
}
}
}
IIS Handler Configuration
The IIS 7x handler configuration is very straight-forward. We are binding the path to be a ‘child’ of our service description proto file “ProtoService.proto”. This, combined with the mimeMap below, allows the user to discover our service definition so that they can interact with it. So it’s time to get working, build the project and update the web.config with the following:
<system.webServer>
<staticContent>
<mimeMap fileExtension=".proto" mimeType="text/plain"/>
</staticContent>
<handlers>
<add name="SearchService" preCondition="integratedMode" verb="GET,POST" path="/ProtoService.proto/*"
type="ProtoService.ServiceHandler, ProtoService, Version=1.0, Culture=neutral" />
</handlers>
</system.webServer>
Getting Results
Make sure you are running in IIS, this will not work in Cassini. Open up your browser (NOT IE) and enter the URL: http://localhost/protoservice.proto/Search?query=asdf You should see the following XML response:
<root>
<result>
<url>http://example.com</url>
</result>
</root>
If you type something that doesn’t make sense, or generates an error, (ie. http://localhost/protoservice.proto/BadMethodName) you will see an error message like this:
<root> <error_text>Method 'ProtoService.ISearchService.BadMethodName' not found.</error_text> </root>
Client Proxy
Now that we have a working service, building a simple client proxy for C# binary protobuffers is really easy. First we need an implementation of the client proxy dispatch interface, IRpcDispatch. I’m going to use the WebClient here simply because it’s easy; however, production systems more often use the HttpWebRequest class.
class HttpProxy : IRpcDispatch
{
readonly Uri _baseUri;
public HttpProxy(Uri baseUri) { _baseUri = baseUri; }
public TMessage CallMethod<TMessage, TBuilder>(string method, IMessageLite request, IBuilderLite<TMessage, TBuilder> response)
where TMessage : IMessageLite<TMessage, TBuilder>
where TBuilder : IBuilderLite<TMessage, TBuilder>
{
WebClient client = new WebClient();
client.Headers[HttpRequestHeader.ContentType] = MessageFormatOptions.ContentTypeProtoBuffer;
client.Headers[HttpRequestHeader.Accept] = MessageFormatOptions.ContentTypeProtoBuffer;
byte[] result = client.UploadData(new Uri(_baseUri, method), request.ToByteArray());
return response.MergeFrom(result).Build();
}
}
Once we have this defined we can now instantiate and call the proxy.
SearchRequest result;
SearchResponse result;
using(SearchService svc = new SearchService(new HttpProxy(new Uri("http://localhost/protoservice.proto/"))))
result = svc.Search(SearchRequest.CreateBuilder().SetQuery("bar").Build());
foreach (SearchResponse.Types.Result r in result.ResultList)
Console.WriteLine(r.Url);
Alternative Client Formats
This proxy uses protobuffers but it could easily be adapted to use json or xml just by changing the content-type and and accept headers and serializing accordingly. To Serialize a protobuffer message as xml or json the following extensions can be used:
//XML
string xmlResult = client.UploadString(new Uri(_baseUri, method), request.ToXml());
return response.MergeFromXml(
System.Xml.XmlReader.Create(new StringReader(xmlResult)))
.Build();
//JSON
string jsonResult = client.UploadString(new Uri(_baseUri, method), request.ToJson());
return response.MergeFromJson(jsonResult).Build();
Lastly there are two more extension methods that can do this by simply providing a stream and a mime-type. This is demonstrated above in the catch block of our http handler. Here are the extension method prototypes that can be used:
public static void WriteTo(this IMessageLite message, MessageFormatOptions options, string contentType,
Stream output);
public static TBuilder MergeFrom<TBuilder>(this TBuilder builder, MessageFormatOptions options,
string contentType, Stream input) where TBuilder : IBuilderLite;
Closing Remarks
I’m very biased here since I wrote most of this capability; however, I am constantly amazed at how easy protobuffers are to use. Google’s Protocol Buffers are very powerful and extremely fast. I’ve been using them for two years now and I can’t imagine writing a serialization or remoting solution without them.