C# Read Lines from a transfer-encoding chunked HTTP stream

Published: 2015-04-17

David Edelstein

Using System.Net.HttpWebRequest to hit a remote server is easy, and typically you want to be notified when the response is ready before processing. However, couchdb has a continuous changes feed available that will leave the connection open. And send the header Transfer-Encoding: chunked

To look at the data exchanged in this HTTP response, use CometPeek in fiddler so that the streamed contents are easily viewed. In the response stream, each update encountered by the couchdb server will send one chunked reply prefixed by the content-length and terminted with \r\n\r\n

.Net’s HttpWebRequest consumes these messages transparently, so when you read from the stream directly you won’t see content length or the extra line breaks. Your code can simply read directly from the stream

Now on to a hard lesson learned when reading from this ResponseStream. In the callback to BeginGetResponse you will call EndGetResponse to get the response. Using that your code will GetResponseStream. With this response stream, you can simply ReadBytes while !endOfStream. And this all works with no extra wrestling. However, since couchdb conveniently sends single line messages, I wanted to use System.IO.StreamReader so that I could simply ReadLine each couchdb document. However! if the wrapped response.GetResponseStream() in a StreamReader gets to the end of the stream it actually closes the connection! To get around this, use the infrequently used (as far as I know) constructor for StreamReader that will allow you to ask it to keep the underlying stream open.

msdn

public StreamReader(
	Stream stream,
	Encoding encoding,
	bool detectEncodingFromByteOrderMarks,
	int bufferSize,
	bool leaveOpen
)

Here is some simple example usage (using still works correctly here, but is possibly unnecessary as we are trying to not close anything):

using (var reader = new StreamReader(response.GetResponseStream(), Encoding.UTF8, true, 1024, true))