Data Transfer Using Self Hosted WCF Service

Rohan Warang | February 18, 2009

Traditionally services are hosted in IIS. This article describes a method of self hosting and consuming a WCF service without the use of IIS.

The WCF service is declared and configured programmatically i.e without the use of configuration files. Most people would not advice hosting services programmatically. But I believe it is the most compact and easiest way of hosting a WCF service.

I have used two console application one to host the service and second to consume it. The host application demonstrates the use of properly configured self hosted WCF service to transfer large amount of data. However this configuration is not confined to self hosted services it can be applied to any WCF service to improve its performance.

Self Hosting

Every .net application (winforms, console or windows service) runs as a process in windows. Web applications on the other hand are hosted in the IIS service.

Every process can run multiple application domains. An application domain is a separation envelope that .net CLR uses to isolate assemblies, code and data.

We can have multiple ServiceHost instances in that application domain. This means we can host multiple WCF services from a winforms application, console application or a windows service, similar to the way IIS does.

The Service Contract and DataUploader Class

We first define the host application. The service contract describes the operations supported by the WCF service.

Show Code

[ServiceContract]
interface IDataUploader
{
    [OperationContract]
    void Upload(Stream data);
}

The service takes data in the form of a stream and writes it to file. A class DataUploader is declared which implements the service contract as shown below. This class actually processes the WCF request.

The incoming stream is read in small packets or buffers of size 1 MB. The buffer size can be adjusted according to your need. Larger buffers will consume more memory, but will read the stream in lesser iterations. So you can manipulate this size to optimize the performance depending on size of files to be transferred.

Instead of a stream a byte array could have been chosen. The only problem is the entire byte array will be cached in memory at both receiving and sending ends. For large files this would just cause memory out of bound exceptions. So instead we transfer a stream which is then pulls data at the receiving end.

Show Code

class DataUploader : IDataUploader
{
    public void Upload(Stream data)
    {
        using (FileStream fs =
            new FileStream(@"C:\UploadedFile.avi", FileMode.Create))
        {
            int bufferSize = 1 * 1024 * 1024; // 1 MB buffer
            byte[] buffer = new byte[bufferSize];
            int bytes;

            while ((bytes = data.Read(buffer, 0, bufferSize)) > 0)
            {
                fs.Write(buffer, 0, bytes);
                fs.Flush();
            }
        }
    }
}

Hosting the service

The service is hosted in a ServiceHost instance. The below mentioned code declares a ServcieHost. The ServiceHost constructor either takes a singleton object or a class type as parameter which in this case is DataUploader.

Show Code

ServiceHost host = new ServiceHost(typeof(DataUploader));

Adding Endpoints

Since the service is going to perform large amount of data transfer operations it is advisable to use a net tcp binding. Net tcp bindings use a tcp socket as a communication channel which is ideal for our DataUploader. But as discussed earlier since WCF can have multiple endpoints, any number of endpoints can be added to this ServiceHost instance.

Show Code

host.AddServiceEndpoint(typeof(IServiceTracker), new NetTcpBinding(),
new Uri("net.tcp://" + Dns.GetHostName() + ":5000/DataUploader"));

Dns.GetHostName() gets the name of the machine on which the service is hosted, 5000 is a TCP port. Any other port can be defined here. The hostname along with a TCP port identifies a TCP socket on your machine which will be used to communicate with this service. Multiple service endpoints cannot be declared on one socket.

However adding an endpoint as shown in the above line of code adds a net tcp binding with default settings. The default binding settings are not good enough for providing high performance. So instead we tweak the binding as shown below.

Show Code

NetTcpBinding binding = new NetTcpBinding();
binding.TransferMode = TransferMode.Streamed;
binding.ReceiveTimeout = TimeSpan.MaxValue;
binding.MaxReceivedMessageSize = long.MaxValue;

host.AddServiceEndpoint(typeof(IDataUploader), binding,
new Uri("net.tcp://" + Dns.GetHostName() + ":5000/DataUploader"));
host.Open();

The default TransferMode of any binding is buffered. In buffered mode the data is cached in memory until the whole message is transferred. In streamed mode only the headers are cached rest of the data is available as a stream. This enables large data to be transferred as a stream instead of chunks.

Either of the modes can be used depending on the data transferred. Buffered mode will hold the data in memory so for large data it is not advisable. Streaming is faster as it does not buffer the data and does not consume as much memory. But in case of network or channel failures the entire stream needs to be resent. Buffers on other hand are more reliable as only the chunk that is lost is resent.

So we set the ReceiveTimeout to its maximum value which prevents the channel from timing out for large file transfers.

MaxReceiveMessageSize is set to maximum to allow large message files to be transferred.

Service endpoint is added using this configured endpoint. The service contract is used to specify the type of contract implemented.

And finally host.open() is called to open for the service to start listening to the TCP socket.

Show Code

class Program
{
    static void Main(string[] args)
    {
        ServiceHost host = new ServiceHost(typeof(DataUploader));

        NetTcpBinding binding = new NetTcpBinding();
        binding.TransferMode = TransferMode.Streamed;
        binding.MaxReceivedMessageSize = int.MaxValue;
        binding.CloseTimeout = TimeSpan.MaxValue;

        host.AddServiceEndpoint(typeof(IDataUploader), binding,
        new Uri("net.tcp://" + Dns.GetHostName() + ":5000/DataUploader"));

        host.Open();

        Console.WriteLine("Service Hosted ...");
        Console.ReadKey();

        host.Close();
    }
}

Consuming the Service

Now we write a sample client for consuming the hosted service. The client simply uploads a file by calling the Upload method provided by the service.

Show Code

class Program
{
    static void Main(string[] args)
    {
        NetTcpBinding binding = new NetTcpBinding();
        binding.TransferMode = TransferMode.Streamed;
        binding.SendTimeout = TimeSpan.MaxValue;

        EndpointAddress address =
            new EndpointAddress("net.tcp://localhost:5000/DataUploader");

        ChannelFactory<IDataUploader> channel =
        new ChannelFactory<IDataUploader>(binding, address);

        IDataUploader uploader = channel.CreateChannel();
        uploader.Upload(File.Open(@"E:\InputFile.avi", FileMode.Open));

        ((IClientChannel)uploader).Close();

        Console.WriteLine("File Uploaded Successfully");
        Console.ReadKey();
    }
}

A ChannelFactory is declared to create channels to communicate with the host service. To create a channel factory we need to know the following things.

The service endpoint address
The type of binding
The service contract (the same one used for declaring the service)

The binding used is the same as client a net tcp binding in streaming mode. The SendTimeout needs to be set as maximum in this case to prevent channel timeout at the client end.

In the endpoint address replace localhost with the server IP or servername where service is hosted.

The channel.CreateChannel() method opens a new TCP channel of type IDataUploader which is the service contract. This channel is used to upload the file.

On successful uploading the channel is closed.

Metadata Exchange

This service cannot be consumed simply by adding reference from Visual Studio’s Add Service Reference option. The reason being this service is not discoverable since it does not have a metadata exchange binding.

This is a good security feature since it can be consumed only if you know the details about the service. However if you wish to make it discoverable all you have to do is specify a MetadataExchangeBindings binding for the service host.

Show Code

host.Description.Behaviors.Add(new ServiceMetadataBehavior());

Binding mexBinding = MetadataExchangeBindings.CreateMexTcpBinding();
host.AddServiceEndpoint(typeof(IMetadataExchange), mexBinding,
new Uri("net.tcp://localhost:5000/DataUploader/mex"));

Configuring WCF Throttling Behaviors

Download Sample
WCFStreamingSample.zip

Categories: Optimization
Tags: Microsoft, Volume, WCF
: Comments rss
: Trackback

17 Responses to “Data Transfer Using Self Hosted WCF Service”

catweazle says:

March 31, 2009 at 8:34 pm

very helpful article, thanks a lot !
I fully agree to the statement “Most people would not advice hosting services programmatically. But I believe it is the most compact and easiest way of hosting a WCF service.”
Richard Harrigan says:

April 12, 2009 at 12:08 am

Hi,

I downloaded your code and ran a test.

- I Changed from localhost:5000 to 5001 and recompiled.
- I started the StreamingServer.
- I Started StreamingClient.

Failed on No connection could be made because the target machine actively refused it 127.0.0.1:5001.

Any help would be greatly appreacted…I would like to use this if I can get past the security issue.

Thanks

Rich
Rohan Warang says:

April 12, 2009 at 12:13 am

@Richard Harrigan

Have you changed the port to 5001 in both the server and client? The ports have to be same.

Also make sure you run the server first and then the client.
Kris Senden says:

April 17, 2009 at 12:01 pm

Hi,

I took your example as a basis for my WCF services hosted in a Windows Service. The client which consumes the service requests a large number of records to the service, but unfortunately, everytime I try this I get the following exeception:

System.ServiceModel.CommunicationObjectAbortedException was unhandled
Message=”The socket connection has been disposed.”

I have changed the NetTcpBinding from both the service and the client to the following:

Service:
NetTcpBinding binding = new NetTcpBinding();
binding.TransferMode = TransferMode.Streamed;
binding.ReceiveTimeout = TimeSpan.MaxValue;
binding.SendTimeout = TimeSpan.MaxValue;
binding.CloseTimeout = TimeSpan.MaxValue;
binding.OpenTimeout = TimeSpan.MaxValue;
binding.MaxReceivedMessageSize = long.MaxValue;

Client:
NetTcpBinding binding = new NetTcpBinding();
binding.TransferMode = TransferMode.Streamed;
binding.ReceiveTimeout = TimeSpan.MaxValue;
binding.SendTimeout = TimeSpan.MaxValue;
binding.CloseTimeout = TimeSpan.MaxValue;
binding.OpenTimeout = TimeSpan.MaxValue;
binding.PortSharingEnabled = true;
binding.MaxReceivedMessageSize = long.MaxValue;

The first time I tried to do it I got the message that the maximum message size was exceeded and that I needed to change the value, which I did. However, I am really puzzled with this one and was hoping you could point me in the right direction?

Thanks,

Kris
Rohan Warang says:

April 17, 2009 at 6:32 pm

Hi, @Kris Senden

The exception System.ServiceModel.CommunicationObjectAbortedException is a very generic exception. You can not narrow it down to one solution. If you could provide me with more details I may be able to help you.

When you say “The client which consumes the service requests a large number of records to the service” does this mean you consume the service multiple times in a loop? In that case you can set throttling parameters to the service and enable more multiple concurrent calls. Refer Configuring WCF Throttling Behaviors for more details

The code you have written is ok and is not causing your error. Give me a bit more specific details and I will try to help you with your solution.

- Rohan
Tool 2358 says:

April 28, 2009 at 3:37 am

I like this explain, great.

But I have a question.

I use a file with 800kB when I got it from server, the packets are about 2MB… What are doing bad?

I use the same Transfer Mode (Buffered), NetTcP. MaxLenght for long.max…
binding.ReaderQuotas.MaxArrayLength = 2147483647;
binding.ReaderQuotas.MaxDepth = 2147483647;
binding.ReaderQuotas.MaxStringContentLength = 2147483647;
binding.ReaderQuotas.MaxBytesPerRead = 2147483647;
binding.ReaderQuotas.MaxNameTableCharCount = 2147483647;

Sorry for my bad english.

Thank you for your help.
Rohan Warang says:

April 28, 2009 at 11:37 pm

@Tool 2358

The binding.ReaderQuotas.MaxStringContentLength and binding.maxBufferSize should be of the same size, which I guess is true in your case.

The value you have set 2147483647 bytes comes to roughly 2MB. That is the reason why your packet sizes are 2MB even though your actual message is 800kB.

You could use Transfer mode streamed as I have in this article then you will not have to worry about ReaderQuotas.
Venky says:

June 17, 2009 at 5:56 pm

Excellent sample..Saved my day..Thanks!
Sharanya says:

September 3, 2009 at 3:13 am

Hello,
your sample code was very helpful.
But I am running into a few problems.

I got this error.

The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue. Local socket timeout was ‘00:04:59.9843759′.

Is this some port problem on my machine or some config error?
Sharanya says:

September 3, 2009 at 7:01 pm

Hello,
It worked for me.
I just had to change the timeout settings.

I have one more question.

Can we use http binding instead of tcp binding?
Because tcp binding can only be used for intranet purposes right?
Rohan Warang says:

September 3, 2009 at 10:11 pm

Actually TCP binding should work on internet as well. But you can test it out and let me know.
Vishal says:

November 25, 2009 at 9:18 pm

Hi

I am using your sample code.

But unable to get output. When i run the application, it runs without any error but just show the output window with no output.

When i tried debugging the application using breakpoints, the calling client code gets executed(i mean the channel get created) but the server code does not. Neither the file gets created nor data transfer.

Please let me know?
Aaron says:

December 29, 2009 at 5:26 am

Hi Rohan,

I have a similar issue as Kris Senden’s. What I am doing here is I want to retrieve a report based on a few filters. When I request a small amount of data in the report, it works fine. When I request a large amount of data, it gives the error “The socket connection has been disposed”. I have tried to set different attributes to big numbers. but all failed with the same error. Below is the whole stack trace. If you could shed some light on this, I would greatly appreciate it. Aaron

System.ServiceModel.CommunicationObjectAbortedException: The socket connection has been disposed. —> System.ObjectDisposedException: The socket connection has been disposed. Object name: ‘System.ServiceModel.Channels.SocketConnection’. — End of inner exception stack trace — Server stack trace: at System.ServiceModel.Channels.SocketConnection.ThrowIfClosed() at System.ServiceModel.Channels.SocketConnection.Read(Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout) at System.ServiceModel.Channels.DelegatingConnection.Read(Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout) at System.ServiceModel.Channels.DelegatingConnection.Read(Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout) at System.ServiceModel.Channels.PreReadConnection.Read(Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout) at System.ServiceModel.Channels.ConnectionStream.Read(Byte[] buffer, Int32 offset, Int32 count, TimeSpan timeout) at System.ServiceModel.Channels.ConnectionStream.Read(Byte[] buffer, Int32 offset, Int32 count) at System.ServiceModel.Channels.SingletonConnectionReader.SingletonInputConnectionStream.ReadCore(Byte[] buffer, Int32 offset, Int32 count) at System.ServiceModel.Channels.SingletonConnectionReader.SingletonInputConnectionStream.Read(Byte[] buffer, Int32 offset, Int32 count) at System.ServiceModel.Channels.MaxMessageSizeStream.Read(Byte[] buffer, Int32 offset, Int32 count) at System.ServiceModel.Channels.SingletonConnectionReader.Close(TimeSpan timeout) at System.ServiceModel.Channels.SingletonConnectionReader.DoneReceiving(Boolean atEof, TimeSpan timeout) at System.ServiceModel.Channels.SingletonConnectionReader.DoneReceiving(Boolean atEof) at System.ServiceModel.Channels.SingletonConnectionReader.SingletonInputConnectionStream.Close() at System.ServiceModel.Channels.DelegatingStream.Close() at System.Xml.XmlBufferReader.Close() at System.Xml.XmlBaseReader.Close() at System.Xml.XmlBinaryReader.Close() at System.Xml.XmlReader.Dispose(Boolean disposing) at System.Xml.XmlReader.System.IDisposable.Dispose() at System.ServiceModel.Dispatcher.OperationFormatter.DeserializeBodyContents(Message message, Object[] parameters, Boolean isRequest) at System.ServiceModel.Dispatcher.OperationFormatter.DeserializeReply(Message message, Object[] parameters) at System.ServiceModel.Dispatcher.ProxyOperationRuntime.AfterReply(ProxyRpc& rpc) at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc) at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs) at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation) at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message) Exception rethrown at [0]: at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg) at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type) at Mercer.HB.PA.ODP.WCFService.Contracts.IReportingService.GetSubmissionActivityReport(IEnumerable`1 mpis, String periodList, String submTypeList, String statusList, String startDate, String endDate, Int32 submID) at Mercer.HB.PA.ODP.Service.SystemReportService.GetSubmissionActivityReport(ISubmissionActivityView view, IEnumerable`1 MPIs, String periodList, String submTypeList, String statusList, String startDate, String endDate, Int32 submID) at Mercer.HB.PA.ODP.Presentation.SubmissionActivityPresenter.BuildReport(String mpis, String periodList, String submTypeList, String statusList, String startDate, String endDate, Int32 submID) at Mercer.HB.PA.ODP.Presentation.SubmissionActivityPresenter.GetReport() at DesktopModules_ODPSubmissionActivity_SubmissionActivity.btnGetReport_Click(Object sender, EventArgs e) at System.Web.UI.WebControls.Button.OnClick(EventArgs e) at System.Web.UI.WebControls.Button.RaisePostBackEvent(String eventArgument) at System.Web.UI.WebControls.Button.System.Web.UI.IPostBackEventHandler.RaisePostBackEvent(String eventArgument) at System.Web.UI.Page.RaisePostBackEvent(IPostBackEventHandler sourceControl, String eventArgument) at System.Web.UI.Page.RaisePostBackEvent(NameValueCollection postData) at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean includeStagesAfterAsyncPoint) — End of inner exception stack trace —
Aaron says:

December 29, 2009 at 5:31 am

Hi Rohan,

Continue from my last post, here is some additional info.

I am using NetTcp binding and have two Web servers behind a load balancer comminicate with one
Application server that hosts the wcf service using windows service.

Thanks
Aaron
Rohan Warang says:

December 29, 2009 at 11:25 am

Hi Aaron,
How are you requesting the report data. It is unclear from your stacktrace or maybe i’m just unable to read it well
The reason why I am asking this is because the streaming works only with serializable data types such as DataSets and not with DataTables.
Rohan Warang says:

December 29, 2009 at 11:33 am

Hi Vishal,
Sorry for the late reply, I was a bit caught up in work. I really cant say why you are not getting an output, you will need to debug and verify. To get debug on both server and client two separate debug instances have to be started. The easiest way to do that is to run server and client as disparate solutions.
Let me know if you find any specific problem, I will be glad to help you out.

- Rohan
Aaron says:

December 29, 2009 at 11:50 pm

Hi Rohan,

The report data is returned as an IEnumerable class object.
I still think it’s one of the config settings that throws the error since it works for small amount of data.

Thanks
Aaron

Csharp Code Samples